Oculus Rift panorama with depth viewer

[Kajos]

Since we're adding demos now, I thought, why not Zoidberg?

So here's mine, a stereoscopic panorama viewer with support for 2d depth. I exaggerated the depth a little, but it is a proof of concept as of now. I'm planning to create a point and click adventure game and release the source of the engine for other purposes.

I created two binaries, which run on Java, LWJGL, one for anaglyphic support, the other for the Oculus Rift.

Click the mouse to switch fullscreen on/off.

I couldn't find a panorama image with depthmap, so this image has to do.

Feedback is most welcome!

Screenshot of anaglyphic version:


Anaglyphic version:
http://www.2shared.com/file/FTmSp8zd/anaglyph.html

Rift version:
http://www.2shared.com/file/4_LhJ2qO/rift.html

EDIT: This method is not a classy way to do this, I'm working on a better way atm.

[cybereality]

Cool. So where would you get images to use with this? Do you plan to make them yourself?

[Kajos]

Yes, that's going to be a tricky one. Finding panoramas with depthmaps is hard, because well, there are none really I think I'm going to snap my own panoramas and simply draw the depthmaps by hand.

Also, I forget to say, it has functionality built in to allow 3D models to integrate with the panorama image. Those are the spheres that are drawn on top. This way I can have nifty animations integrated.

[Fredz]

Doing 2D panoramas by hand would probably not give a very convincing result, it's very difficult to keep the focus point at the center of rotation and it'll be distorted because of the FOV of the cam. Creating a depthmap by hand to create a 3D conversion on top of this would only make it worse IMHO.

Maybe you should start with some high quality still renderings of nice scenes using a 3D modeler like Blender. Or try to take stills with a standard cam using the "Cha-Cha" method and custom optics (fisheye) to get a wide FOV.

But nice initiative anyway, a View-Master with 360° head tracking would be quite awesome.

What would be even nicer would be to find digital copies of 3D movies created for the Imax Solido (3D hemispheric screen with 180° HFOV and 123° VFOV).

EDIT : I found an example of a sequence of images filmed for the Imax Solido, I hope it can help :

[Kajos]

Doing 2D panoramas by hand would probably not give a very convincing result, it's very difficult to keep the focus point at the center of rotation and it'll be distorted because of the FOV of the cam. Creating a depthmap by hand to create a 3D conversion on top of this would only make it worse IMHO.

Yeah, I'll have to try it out sometime, see how handdrawing turns out. The effect won't be as good as "true" stereoscopic 3D, but might be the only solution, since scanning depth is almost impossible. Maybe when 2D depth scanning algorithms advance, but I don't see that happen soon. For the panorama photos I thought using a panorama cameraholder (the one that rotates with the lens as center point) and a fisheye camera. I've seen some pretty nice results with that.

Maybe you should start with some high quality still renderings of nice scenes using a 3D modeler like Blender. Or try to take stills with a standard cam using the "Cha-Cha" method and custom optics (fisheye) to get a wide FOV.

3D renderings would be the best option, but I suck at advanced 3D modelling I'll take a look if there's a free scene somewhere so I can atleast optimize my algorithms with that. Cha-cha method, or using multiple points for taking picture, like dual cameras like in the 3DS, doesn't really work with panoramas, since a 360 degree camera can only be at one place. Having two origin points for example creates a good effect of 3D for two slight angles, but mostly it sucks, so that's the main issue.

But nice initiative anyway, a View-Master with 360° head tracking would be quite awesome.

What would be even nicer would be to find digital copies of 3D movies created for the Imax Solido (3D hemispheric screen with 180° HFOV and 123° VFOV).

Movies with this technique would be awesome! Too bad material is lacking at the moment Also, you need quite the resolution, since you need to cover the whole sphere/panorama every frame. Cloudstreaming would come in handy here..

EDIT : I found an example of a sequence of images filmed for the Imax Solido, I hope it can help :

Cool, gonna look into that! I must be able to learn something from the tricks they use.

[Fredz]

Yeah, I'll have to try it out sometime, see how handdrawing turns out. The effect won't be as good as "true" stereoscopic 3D, but might be the only solution, since scanning depth is almost impossible. Maybe when 2D depth scanning algorithms advance, but I don't see that happen soon.

Depth extraction from photo is still not a resolved problem, but good progress has been made in this domain. Have a look here for state of the art :
http://vision.middlebury.edu/stereo/eval/

I've tried implementing something similar to study the quality of depth in 3D movies and games by using simpler algorithms (simple SAD with 3x3 windows), even if you want to create depthmaps by hand it would give you a good start with a first approximation. I've posted several pics of my results on this thread :
http://www.mtbs3d.com/phpBB/viewtopic.p ... 301#p68301

For the panorama photos I thought using a panorama cameraholder (the one that rotates with the lens as center point) and a fisheye camera. I've seen some pretty nice results with that.

Yes, I build one myself looking like this one :

I can't remember which tutorial I followed, it's been years ago, you may have a look at this page to grab some ideas :
http://www.peterloud.co.uk/nodalsamurai ... murai.html

The results were quite good (viewed with a Flash-based viewer I can't remember the name of) but there were always alignment problems which I think could ruin the effect in 3D.

3D renderings would be the best option, but I suck at advanced 3D modelling I'll take a look if there's a free scene somewhere so I can atleast optimize my algorithms with that.

A good start would probably be to use assets from open projects like "Elephants Dream", "Big Buck Bunny" or "Sintel" and rerender in 110°x90° FOV in 1280x800 SBS. Someone already did a stereoscopic render for the first two IIRC. Doing this with a wider FOV should prove a bit more complicated, but it sounds doable for the least efforts compared to starting a whole project on your own. Doing a complete video render would even be greater for sure.

Cha-cha method, or using multiple points for taking picture, like dual cameras like in the 3DS, doesn't really work with panoramas, since a 360 degree camera can only be at one place. Having two origin points for example creates a good effect of 3D for two slight angles, but mostly it sucks, so that's the main issue.

I think it would be exactly like what Imax Solido 3D movies look like, albeit they use only a 180° HFOV, but I think it would still be a nice start.

Movies with this technique would be awesome! Too bad material is lacking at the moment Also, you need quite the resolution, since you need to cover the whole sphere/panorama every frame.

I don't know what resolution they did use, but I did watch one such movie when I was younger (Echoes of the Sun) and I didn't found the resolution lacking. But it was some 20 years ago and not digital, so 1080p transfer would probably not be enough to keep the resolution intact. Maybe 4K or 8K would be needed.

Cool, gonna look into that! I must be able to learn something from the tricks they use.

The resolution is pretty much lacking on this one but I couldn't find anything better. If you find other images of this type don't hesistate to post them here.

The image comes from this site, it's an article from 1995 with a good amount of information about how the 3D wide FOV cameras were designed for the Imax Solido :
http://www.laserfocusworld.com/articles ... rding.html

And good luck for your project, not an easy task but extremly interesting...

[QuadDeuce]

This is such an exciting time, not just for gaming. The Rift is going to cause significant disruption in many traditional industries. Quality virtual reality is going to have a profound impact on the real estate profession, and I think will reinvigorate the panorama photograph profession. There are three magic keys to pulling this off,

1. HMD with quality head tracking (Rift)
2. computer controlled stereo panorama capture devices. (will need to modify to dual camera setup)
3. A hardware accelerated (very low latency) equirectangular panorama viewer with supported head tracking (yet to be developed)

So here is the concept: capture two identical 360x180 panoramas, using a dual cam stereo rig, 20+ cm of separation for landscape shots and less for interior photos. After performing the same post processing/leveling on both images, you open two instances of the viewer and output SBS for the Rift. Hopefully at some point one of the more popular viewers will support head tracking.

There were a few instances of this being pulled off during the last VR go around, I'm including some links below to what I found if anyone is interested. Also some links to some very cool anaglyph stereo panoramas.

http://www.gali-3d.com/archive/articles ... acking.php (an older iteration of what needs to be accomplished)
http://www.nodalninja.com/forum/showthr ... sheye-lens
http://www.indiavrtours.com/vrphotos/river_3d.html (really cool anaglyph outdoor pano)
http://www.outline.be/stereo/Guillemins3D_HD/ (train station anaglyph pano)
http://www.outline.be/stereo/StJacques3D_2012/ (church anaglyph pano, great DOF)
http://www.kolor.com/blog-en/2011/01/06 ... hamblenne/ (how the train pano was made)
http://tech.groups.yahoo.com/group/Pano ... r=1&tidx=1 (thread covering 3d pano creation)
http://www.icinema.unsw.edu.au/projects ... nstrators/ (inserting 3d objects into a pano)
http://game-multimedia.com/tut/img/stereo3d_pano.html (lots of topics covered)

[Kajos]

Using a depthmap is the only way it can work for video. As you see on the youtube video you posted, http://www.youtube.com/watch?v=EwzCMPeS6JE, he uses a dual camera setup. You can't do this with 360 degree videocameras, as is my understanding (please correct me if I'm wrong, but that seems obvious to me).

So there's that way, dual camera, and the way with depthmaps. I haven't heard of any video devices which can capture depth however, except the Kinect.
http://www.icinema.unsw.edu.au/projects ... nstrators/ this link shows 3D objects, but I think they're just pasted on. To properly inject 3D models, you also need a depthmap, so objects in the panorama can display themselves in front of the 3D model if needed.

Thanks for the links, they are quite informative. It's hard finding info on this.

[QuadDeuce]

True about video, one camera would occlude the FOV for the other. You get around that with photos by using a longer focal length and more images. I believe it takes 50+ images per camera to capture the stereo panos. 20+ cm of stereo separation would make it very difficult to build a rig, even one like the Ladybug systems.

[Kajos]

Do you happen to know any cheap cameras fit for panorama shooting?

[Fredz]

You could also limit yourself to 180° for videos, like for Omnimax and Imax Solido. Probably not as immersive as 360°, but it would be a lot easier to create content for this FOV. And generally when you watch a movie you are constantly looking at a point of interest in the center of the screen (be it flat or hemispheric). There was a 360° giant screen at Futuroscope some years ago but they removed it and kept only 180° displays, I guess they thought it was a better solution.

[Kajos]

The depthmap also allows for better parallax viewing.

I made a new demo, it has an panorama image that shows it off (the depthmap is terrible, it is handdrawn; took 15 minutes for it):

http://www.2shared.com/file/vEZqRbtA/depth2d_new.html

View with your favorite anaglyph glasses

It can still be improved, but I take my time

[cybereality]

@QuadDeuce: Those are some cool panoramics, thanks for posting.

[Fredz]

Created a start.sh and chmoded it +x :

Code: Select all

java -Djava.library.path=native/linux -jar evidence-all.jar

On a Debian/unstable amd64 it shows a black window then crashes.
Java version :

java version "1.6.0_24"
OpenJDK Runtime Environment (IcedTea6 1.11.3) (6b24-1.11.3-2)
OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)

Error log :

Code: Select all

Info log:
0:0(0): error: no matching function for call to `pow(vec4, float)'
0:0(0): error: candidates are: float pow(float, float)
0:0(0): error:                 vec2 pow(vec2, vec2)
0:0(0): error:                 vec3 pow(vec3, vec3)
0:0(0): error:                 vec4 pow(vec4, vec4)
0:0(0): error: no matching function for call to `pow(vec4, float)'
0:0(0): error: candidates are: float pow(float, float)
0:0(0): error:                 vec2 pow(vec2, vec2)
0:0(0): error:                 vec3 pow(vec3, vec3)
0:0(0): error:                 vec4 pow(vec4, vec4)
0:1(1088): error: type mismatch
0:1(1130): error: Operands to relational operators must be scalar and numeric
0:1(1130): error: if-statement condition must be scalar boolean
0:1(1181): error: type mismatch
0:1(1209): error: type mismatch
0:1(1335): error: type mismatch
0:1(1404): error: type mismatch
0:1(1441): error: Operands to relational operators must be scalar and numeric
0:1(1441): error: LHS of `&&' must be scalar boolean
0:1(1517): error: type mismatch
0:1(1594): error: Operands to relational operators must be scalar and numeric
0:1(1594): error: if-statement condition must be scalar boolean
0:1(1622): error: type mismatch
0:1(1754): error: type mismatch
0:1(1789): error: Operands to relational operators must be scalar and numeric
0:1(1789): error: loop condition must be scalar boolean
0:1(1848): error: type mismatch
0:1(1924): error: Operands to relational operators must be scalar and numeric
0:1(1924): error: if-statement condition must be scalar boolean
0:1(1952): error: type mismatch
0:1(2699): error: Could not implicitly convert operands to relational operator
0:1(2699): error: LHS of `||' must be scalar boolean
0:1(2715): error: Could not implicitly convert operands to relational operator
0:1(2731): error: Could not implicitly convert operands to relational operator
0:1(2731): error: RHS of `||' must be scalar boolean
0:1(2746): error: Could not implicitly convert operands to relational operator
0:1(2746): error: RHS of `||' must be scalar boolean

Fail reading fragment shading code
ERROR - sphereTexture: Invalid value

[Kajos]

Mmm. That's odd, can you try this one see what happens: http://www.2shared.com/file/pbCNIKii/latest.html

[Fredz]

Code: Select all

Info log:
0:1(1761): error: Could not implicitly convert operands to relational operator
0:1(1761): error: LHS of `&&' must be scalar boolean
0:1(2683): error: Could not implicitly convert operands to relational operator
0:1(2683): error: LHS of `||' must be scalar boolean
0:1(2699): error: Could not implicitly convert operands to relational operator
0:1(2715): error: Could not implicitly convert operands to relational operator
0:1(2715): error: RHS of `||' must be scalar boolean
0:1(2730): error: Could not implicitly convert operands to relational operator
0:1(2730): error: RHS of `||' must be scalar boolean

Fail reading fragment shading code
ERROR - sphereTexture: Invalid value

My GPU may not be up to the task, it's an integrated Intel HD on a SandyBridge MoBo.

[Kajos]

Yeah, I coded some later OpenGL calls into it, but nothing that cannot be fixed. Thanks for pointing them out.

http://www.2shared.com/file/mCYuf-ZG/morecomp.html I think I have all incompatibilities fixed now, but I'm not 100% sure.

[Fredz]

Nice, it does work now, very slow, but working.

But I don't have anaglyphic rendering, probably related to these errors about reading shading code :

Code: Select all

Fail reading fragment shading code
Fail reading vertex shading code
Fail reading fragment shading code
Loaded OBJ from file '/Data/IKARUGA/IKARUGA2.OBJ'
2152 vertices.
1688 normals.
1877 textures coordinates.
1774 faces.

Considering the small number of vertices and faces it should go faster too I guess, could it be related to the moving sphere in the center that seems to be rendered with environment mapping ?

It would be nice if you could add a start.sh file in the zip containing this line for Linux users too :

java -Djava.library.path=native/linux -jar evidence-all.jar

[Kajos]

http://www.2shared.com/file/yeveP2_P/dist.html There should be no errors now at all. If it still doesn't show the anaglyph, can you post a screenshot and make sure you drivers are up to date? Also what OpenGL is supported by your Intel chip?

[Fredz]

No message on the console this time, but still no anaglyph rendering.

glxinfo says the OpenGL version is 3.2 and according to Wikipedia my GPU should support SM 4.1 (DirectX 10.1 equivalent).

Screenshot :

Capture.png

[Kajos]

Mmm yeah it surely doesn't use the shader. However the FBO's are handled fine.

I specified the OpenGL version in the shader and removed another integer which should be a float.
It might work, else I suggest updating the drivers, since that is a common problem with OpenGL. I'll see if I can enable some more debugging output but I don't think so.

http://www.2shared.com/file/FrjJIJh4/opengl120.html

[Kajos]

Also, try running it with a different Java machine. I think LWJGL is especially designed towards Sun's.

EDIT: No OpenJDK should work fine as well.

[Kajos]

Here the new version; this one is heavily optimized by a factor of 10. http://www.2shared.com/file/WiO7_KDp/optimized.html

If anyone else experiences any problems with the anaglyph shader not showing, please let me know.

[Fredz]

It's much faster now but still not fluid, could you display some sort of FPS counter in a corner of the window ?

There is still no anaglyph rendering though, no message for it failing either.

Also could you make the window fullscreen by default instead of having to click twice inside it, and preferably using the current resolution of the desktop instead of the default 1280x720 if it's possible ?

Thanks.

I noticed a small rendering bug which was already present in the previous version, a column of several pixels wide is incorrectly rendered on the left side of the window, you can see it in the screenshot I posted.

[Kajos]

Okay, so I added a FPS counter and fullscreen is enabled by default now. I also removed the line on the left, but it did suggest to me that the shader is actually active but not working properly.

http://www.2shared.com/file/hPTyK1Ky/evidence_002.html

I think your Intel HD chip might not be up to par to run 60 fps, but I'll try to optimize as much as possible. Maybe you can tell me if it is your cpu thats maxing out by looking at the system performance? Because the cpu end is pretty basic stuff, so I would think not.

I changed the depthmap texture from 24 bit to the old default one (16 bit I think). This caused some problems with some cards.

I am now going some keystrokes for testing purposes for testing if the shader is active, depthmap is properly received, etc.

[Kajos]

Okay, so here a testing build: http://www.2shared.com/file/jfjTs8je/evidence_003.html

Hold T for approx 1 second and it will switch between a red shader, hold it again for 1 sec and it will display the depthmap of the models. Again will switch back to regular.

[Fredz]

Neither evidence_0.02.zip nor evidence_0.03.zip did start :

Code: Select all

Exception in thread "main" java.lang.NoClassDefFoundError: org/newdawn/slick/SlickException
Caused by: java.lang.ClassNotFoundException: org.newdawn.slick.SlickException
	at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
Could not find the main class: evidence.Evidence. Program will exit.

[Kajos]

Oh sorry! Forgot to include the slick library into the package. Apparently it gives some problems with Slick.

http://www.2shared.com/file/K38TRiAk/evidence-004.html
This one should work.

[Kajos]

Forget the other one please, I fixed a wrongly placed variable and possibly found where your intel hd card hiccups: http://www.2shared.com/file/_U7Xsile/evidence-005.html

[Fredz]

I got a white window this time, with this log :

Code: Select all

Info log:
0:58(9): error: `x' undeclared
0:58(9): error: Operands to arithmetic operators must be numeric
0:58(31): error: Operands to relational operators must be scalar and numeric
0:58(31): error: if-statement condition must be scalar boolean

Seems you are having a hard time with versioning, you could use semantic versioning : http://datasift.github.com/gitflow/Versioning.html

Basically :
Semantic versioning is a very simple scheme built around the X.Y.Z-buildNo concept (for example 2.6.0-2 or 2.6.0-SNAPSHOT20120501):
- Increment Z when you fix something
- Increment Y when you add a new feature
- Increment X when you break backwards-compatibility or add major features
- Use the buildNo to differentiate different builds off the same branch, and to differentiate between development, test and production builds.

[Fredz]

Using evidence-0.04.zip (and correcting the Evidence.jar.jar in start.sh) I got a normal rendering without anaglyph and the FPS showed 0 for some seconds, then 1.

My GPU is certainly not a top one, but I fail to see how rendering a skybox with several spheres can be so slow, when I'm able to play a recent game like Pro Evolution Soccer 2012 at a normal speed with full settings on using Wine.

[Kajos]

http://www.2shared.com/file/GX6zrg1J/ev ... -TEST.html

Yes, I have some difficulty with that, and it's probably also better if I switch to version control. So I'll do that once I get the time for it.

[Kajos]

My GPU is certainly not a top one, but I fail to see how rendering a skybox with several spheres can be so slow, when I'm able to play a recent game like Pro Evolution Soccer 2012 at a normal speed with full settings on using Wine.

Yes, that should not happen. However if it's something that weighing down you GPU, it's probably the stereoscopic shader. I saw your screenshot and concluded it was indeed active, but it tripped somewhere. I might have fixed it in the last build, don't know for sure. It's a bit trial and error since it's not giving any errors for me to investigate.

Can you tell me if your cpu is maxing out, or is it really just your gpu?

If you take the last build and hold T for a second or some, it should display a red screen. Hold it again for a second and it should display the depthmap of the spheres. Do you notice any sudden increase in performance when switched to these modes?

[Fredz]

Same result with this one, 0 FPS, no anaglyph and nothing happens when I press T. The first core is at 80% on average on the Java process, the second core is mostly idle. The CPU is an Intel Pentium G630 with two cores at 2.70GHz each.

For now I think you shouldn't waste more time to make your demo work on my machine, try to add features that everyone (and you) could use and then you may have a try at optimizing the code.

Good luck...

[Kajos]

Yes, I think this is not very beneficial. However, I did get to improve my code on some points, so that's cool. Thanks for your time testing!