The following article and video show that even the most demanding games of today can be played on any web-enabled device:
http://www.techcrunch.com/2009/06/22/exclusive-otoy-goes-mobile-turns-your-cell-phone-into-a-powerful-gaming-rig/
A portable game platform with dedicated game controls such as the PSP connected via WiFi to a 3G phone seems more comfortable and mobile than the setup in the video. In my opinion this is just a proof of concept and I think that the real crowd pleaser will be the photorealistic virtual world with LightStaged characters.
Tuesday, June 30, 2009
Tuesday, June 16, 2009
2 New OTOY video's
2 new video's showing the server side rendering in action and giving some specifics about the lag:
http://www.techcrunch.com/2009/06/16/videos-otoy-in-action-you-have-to-see-this/
The first video shows a guy playing Left 4 Dead and Crysis on his HD TV, hooked up to his laptop, which is connected to the OTOY server through broadband access. He can switch instantly between both games while playing, very impressive. According to the TechCrunch article, EA is partnering with OTOY.
The second video shows GTA4 being played in a web browser while running in the cloud. According to the tester, there is some lag, but it's very playable. Personally, I think GTA4 is not the best game to show off server side rendering because it runs on a terribly unoptimized, buggy and laggy engine. There's no way you can tell if the lag is due to the crappy engine or to the connection. Unfortunately, there's no info about the geographical distance between the player and the cloud. What it does show is that a GTA game with LightStage quality characters and environments could definitely be possible and playable when rendered in the cloud. In fact, I asked Jules this very question yesterday and he confirmed to me that it was indeed possible.
update: Many people cannot believe how OTOY can render so many instances per single GPU. I checked my notes and as Jules explained it to me, he can run 10 instances of a high-end game (like Crysis) and up to 100 instances of a low-end game per GPU. The GPU has a lot of "idle" and unused resources in between the rendering of frames for the same instance. OTOY efficiently uses this idle time to render extra instances. The games shown in the videos (Crysis, Left 4 Dead, GTA IV) are of course traditionally rendered. When using voxel ray tracing, OTOY scales even better.
OTOY can switch between rasterizing and voxel raycasting, because it uses a point cloud as input. Depending on the complexity, one is faster than the other. The scorpion demo for example (the Bug Snuff movie), was first rendered as voxels, but rasterizing it was faster. The Ruby demo from last year was completely raytraced (the voxel rendering is not limited to raycasting, but uses shadow rays and reflection rays as well, so it could be considered as true raytracing).
http://www.techcrunch.com/2009/06/16/videos-otoy-in-action-you-have-to-see-this/
The first video shows a guy playing Left 4 Dead and Crysis on his HD TV, hooked up to his laptop, which is connected to the OTOY server through broadband access. He can switch instantly between both games while playing, very impressive. According to the TechCrunch article, EA is partnering with OTOY.
The second video shows GTA4 being played in a web browser while running in the cloud. According to the tester, there is some lag, but it's very playable. Personally, I think GTA4 is not the best game to show off server side rendering because it runs on a terribly unoptimized, buggy and laggy engine. There's no way you can tell if the lag is due to the crappy engine or to the connection. Unfortunately, there's no info about the geographical distance between the player and the cloud. What it does show is that a GTA game with LightStage quality characters and environments could definitely be possible and playable when rendered in the cloud. In fact, I asked Jules this very question yesterday and he confirmed to me that it was indeed possible.
update: Many people cannot believe how OTOY can render so many instances per single GPU. I checked my notes and as Jules explained it to me, he can run 10 instances of a high-end game (like Crysis) and up to 100 instances of a low-end game per GPU. The GPU has a lot of "idle" and unused resources in between the rendering of frames for the same instance. OTOY efficiently uses this idle time to render extra instances. The games shown in the videos (Crysis, Left 4 Dead, GTA IV) are of course traditionally rendered. When using voxel ray tracing, OTOY scales even better.
OTOY can switch between rasterizing and voxel raycasting, because it uses a point cloud as input. Depending on the complexity, one is faster than the other. The scorpion demo for example (the Bug Snuff movie), was first rendered as voxels, but rasterizing it was faster. The Ruby demo from last year was completely raytraced (the voxel rendering is not limited to raycasting, but uses shadow rays and reflection rays as well, so it could be considered as true raytracing).
A quantum leap of faith
Believe it or not, but yesterday I was on the phone with Jules Urbach, the man himself behind OTOY and LightStage (I guess writing a blog does pay off ;-). He had offered me the opportunity to talk a bit about OTOY, LightStage, the Fusion Render Cloud and where things are heading. It was my first interview ever and I was just shooting one question after another. Too bad I was bloody nervous (I haven’t been that nervous since my last oral exam). Due to my nervousness and my limited understanding of graphics programming, I didn’t absorb a lot of the things he said, but I think I’ve got the bottom line. He answered a lot of my OTOY-related technical questions (unfortunately OTOY isn’t open source, so obviously he couldn’t answer every single one of my questions) and offered me some insight in the cloud computing idea. What follows is my own interpretation of the information that Jules gave me.
Just a couple of weeks ago, I was still wondering what the technical specifications of the next generation of consoles would be like. But after yesterday… frankly I don’t give a damn anymore. The promise of OTOY and server side rendering is even bigger than I initially thought. In fact it’s huge and that’s probably an understatement. In one interview, Jules said that it “is comparable to other major evolutions of film: sound, color, cinemascope, 70mm, THX, stereoscopic 3D, IMAX, and the like” I think it’s even bigger than that, and it has the potential to shake up and “transform” the entire video game industry.
Server side rendering opens up possibilities for game developers that are really hard to wrap your head around. Every game developer has learned to work inside the limitations of the hardware ( memory, polygon and texture budgets, limited number of lights, number of dynamic objects, scene size and so on). These budgets double in size only every 12 to 18 months. Now imagine that artists and level designers could make use of unlimited computational resources and no longer have to worry about technical budgets. They can make the scene as big as they want, with extreme detail (procedurally generated at the finest level) and with as much lighting information and texture layers as they desire. That’s exactly what server side rendering combined with OTOY’s voxel ray tracing might offer. It requires a shift in the minds of game developers and game publishers that could be considered a quantum leap of faith. The only limitation is their imagination (besides time and money of course), and anything that you see in offline rendered CG, could be possible in real-time. Jules is also working on tools to facilitate the creation of 3D environments and to keep development budgets reasonable. One of those tools is a portable LightStage, which is (as far as I understood) a cut down version of the normal LightStage that can be mounted onto a driving car and that can capture whole streets and cities and convert them into a 3D point cloud. It’s much better than LIDAR, because it captures lighting and texture information as well. Extremely cool if it works.
Because the server keeps the whole game scene in memory and because of the way that the voxel ray tracing works, OTOY and the render cloud can scale very easily to tens of thousands of users. Depending on the resolution, he can run 10 to 100 instances of a game scene on one GPU. And you can interconnect an unlimited number of GPU’s.
The best thing about the server side rendering idea is that every one is a winner: IHV’s, ISV’s, game publishers and most importantly the gamers themselves (for a number of reasons which I talked about in one of my previous posts).
In conclusion, I guess every PC gamer has dreamt at some point about a monster PC with terabytes of RAM and thousands of GPU’s working together, with a million unified shaders combined. Until recently, no one in their right mind would make such a monster, because economically it makes no sense to spend a huge load of cash on the development of a game that would make full use of such enormous horse power and could only be played by one person at a time. But with the rapid spreading of broadband internet access, suddenly a whole lot of people are able to play on that monster PC and it becomes economically viable to make such an extremely high quality game. I think OTOY will be the first to achieve this goal. Following the increasing trend of office applications being run in the cloud, server side rendering is going to be the next step in the evolution of the video game industry and it will make “client-side hardware” look like an outdated concept. Jules told me he thinks that in the future, the best looking games will be rendered server side and that there’s no way that expensive local hardware (on the client side) will be able to compete. I for one can’t wait to see what OTOY will bring in the near future.
Just a couple of weeks ago, I was still wondering what the technical specifications of the next generation of consoles would be like. But after yesterday… frankly I don’t give a damn anymore. The promise of OTOY and server side rendering is even bigger than I initially thought. In fact it’s huge and that’s probably an understatement. In one interview, Jules said that it “is comparable to other major evolutions of film: sound, color, cinemascope, 70mm, THX, stereoscopic 3D, IMAX, and the like” I think it’s even bigger than that, and it has the potential to shake up and “transform” the entire video game industry.
Server side rendering opens up possibilities for game developers that are really hard to wrap your head around. Every game developer has learned to work inside the limitations of the hardware ( memory, polygon and texture budgets, limited number of lights, number of dynamic objects, scene size and so on). These budgets double in size only every 12 to 18 months. Now imagine that artists and level designers could make use of unlimited computational resources and no longer have to worry about technical budgets. They can make the scene as big as they want, with extreme detail (procedurally generated at the finest level) and with as much lighting information and texture layers as they desire. That’s exactly what server side rendering combined with OTOY’s voxel ray tracing might offer. It requires a shift in the minds of game developers and game publishers that could be considered a quantum leap of faith. The only limitation is their imagination (besides time and money of course), and anything that you see in offline rendered CG, could be possible in real-time. Jules is also working on tools to facilitate the creation of 3D environments and to keep development budgets reasonable. One of those tools is a portable LightStage, which is (as far as I understood) a cut down version of the normal LightStage that can be mounted onto a driving car and that can capture whole streets and cities and convert them into a 3D point cloud. It’s much better than LIDAR, because it captures lighting and texture information as well. Extremely cool if it works.
Because the server keeps the whole game scene in memory and because of the way that the voxel ray tracing works, OTOY and the render cloud can scale very easily to tens of thousands of users. Depending on the resolution, he can run 10 to 100 instances of a game scene on one GPU. And you can interconnect an unlimited number of GPU’s.
The best thing about the server side rendering idea is that every one is a winner: IHV’s, ISV’s, game publishers and most importantly the gamers themselves (for a number of reasons which I talked about in one of my previous posts).
In conclusion, I guess every PC gamer has dreamt at some point about a monster PC with terabytes of RAM and thousands of GPU’s working together, with a million unified shaders combined. Until recently, no one in their right mind would make such a monster, because economically it makes no sense to spend a huge load of cash on the development of a game that would make full use of such enormous horse power and could only be played by one person at a time. But with the rapid spreading of broadband internet access, suddenly a whole lot of people are able to play on that monster PC and it becomes economically viable to make such an extremely high quality game. I think OTOY will be the first to achieve this goal. Following the increasing trend of office applications being run in the cloud, server side rendering is going to be the next step in the evolution of the video game industry and it will make “client-side hardware” look like an outdated concept. Jules told me he thinks that in the future, the best looking games will be rendered server side and that there’s no way that expensive local hardware (on the client side) will be able to compete. I for one can’t wait to see what OTOY will bring in the near future.
Saturday, June 6, 2009
Has it been 1 year already?
The Ruby city demo, the very reason why I started this blog, was first shown to the world on June 4, 2008. Check the date on this youtube video: http://www.youtube.com/watch?v=zq1KTtU8loM
One full year, I cannot believe it. AMD has released every Ruby demo to the public well within a year after the introduction of the hardware. It began with Ruby Double Cross on Radeon X800, then Ruby Dangerous Curves on X850, followed by Ruby The Assassin on X1800, and finally Ruby WhiteOut on HD 2800. So it made perfect sense that the Voxelized Ruby would come out within a few months after the unveiling of the Radeon 4870. Even Dave Baumann said on the Beyond3d forum that the demo would be released for the public to play with.
So what went wrong? Did ATI decide to hold back the demo or was it Jules Urbach? I think the intial plan was to release the demo at a certain point, but the voxel technology was not finished and took longer to develop than expected. To enjoy the demo at a reasonable framerate, it had to be run on two 4870 cards or on the dual 4870X2, so only the very high end of the consumer market would be able to run it. The previous Ruby demo's were made by RhinoFX, and this was the first time that OTOY made a Ruby demo. Either way, if AMD is making another Ruby demo (with or without OTOY, but I prefer with), it has to look better than the last one, and they better release it within a reasonable amount of time.
Something else that crossed my mind: OTOY is now being used to create a virtual world community (Liveplace/Cityspace) and I think OTOY's technology would be a perfect match for Playstation Home. Virtual world games are much more tolerant to lag than fast-paced shooters or racers, and I think that even a lag of 500 milliseconds would be doable. Imagine you're playing a game on your PS3. Once you're done playing, you automatically end up in Home, being rendered in the cloud. Sony has trademarked PS Cloud, ( http://www.edge-online.com/news/sony-trademarks-ps-cloud ) and I wouldn't be surprised if Sony moved the rendering for PS Home from the client to the server side sooner or later.
One full year, I cannot believe it. AMD has released every Ruby demo to the public well within a year after the introduction of the hardware. It began with Ruby Double Cross on Radeon X800, then Ruby Dangerous Curves on X850, followed by Ruby The Assassin on X1800, and finally Ruby WhiteOut on HD 2800. So it made perfect sense that the Voxelized Ruby would come out within a few months after the unveiling of the Radeon 4870. Even Dave Baumann said on the Beyond3d forum that the demo would be released for the public to play with.
So what went wrong? Did ATI decide to hold back the demo or was it Jules Urbach? I think the intial plan was to release the demo at a certain point, but the voxel technology was not finished and took longer to develop than expected. To enjoy the demo at a reasonable framerate, it had to be run on two 4870 cards or on the dual 4870X2, so only the very high end of the consumer market would be able to run it. The previous Ruby demo's were made by RhinoFX, and this was the first time that OTOY made a Ruby demo. Either way, if AMD is making another Ruby demo (with or without OTOY, but I prefer with), it has to look better than the last one, and they better release it within a reasonable amount of time.
Something else that crossed my mind: OTOY is now being used to create a virtual world community (Liveplace/Cityspace) and I think OTOY's technology would be a perfect match for Playstation Home. Virtual world games are much more tolerant to lag than fast-paced shooters or racers, and I think that even a lag of 500 milliseconds would be doable. Imagine you're playing a game on your PS3. Once you're done playing, you automatically end up in Home, being rendered in the cloud. Sony has trademarked PS Cloud, ( http://www.edge-online.com/news/sony-trademarks-ps-cloud ) and I wouldn't be surprised if Sony moved the rendering for PS Home from the client to the server side sooner or later.
Friday, June 5, 2009
A possible faster-than-light solution for the latency problem?
Interesting article: http://spectrum.ieee.org/computing/networks/game-developers-see-promise-in-cloud-computing-but-some-are-skeptical/0
I have totally embraced the cloud computing idea. I hope OTOY and OnLive can pull it off and create a paradigm shift from client to cloud rendering. The main problem seems to be lag. Apart from the extra lag introduced by encoding/decoding the video stream at the server/client side respectively, which should not be greater than a couple of milliseconds, there is lag due to the time that the input/video signal needs to travel the distance between client and server, which can amount to several tens to hundreds of milliseconds. This is due to the fact that information cannot travel faster than the speed of light (via photons or electromagnetic waves). Quantum physicists have discovered ways to do quantum computing and quantum encryption at 10.000 times the speed of light, but they all agreed that it was not possible to send information faster than lightspeed, because they could not control the contents of quantum entangled photons. But very recently, Graeme Smith, a researcher at IBM has proposed a way to "transmit large amounts of quantum information" described in the following paper, published in Feb, 2009:
http://domino.research.ibm.com/comm/research_people.nsf/pages/graemesm.Main.html/$FILE/NonPrivate.pdf
http://tech.slashdot.org/article.pl?sid=08/08/06/0043220
If his theory holds true, IBM or someone else could make a computer peripheral based on quantum technology (sort of like an Apple Airport) that can communicate large amounts of data instantaneously! Distance between client and cloud would no longer be a problem and transmission lag would be non-existent! It would make playing server side rendered games an almost lag-free experience and the ultimate alternative for costly, power hungry consoles.
I have totally embraced the cloud computing idea. I hope OTOY and OnLive can pull it off and create a paradigm shift from client to cloud rendering. The main problem seems to be lag. Apart from the extra lag introduced by encoding/decoding the video stream at the server/client side respectively, which should not be greater than a couple of milliseconds, there is lag due to the time that the input/video signal needs to travel the distance between client and server, which can amount to several tens to hundreds of milliseconds. This is due to the fact that information cannot travel faster than the speed of light (via photons or electromagnetic waves). Quantum physicists have discovered ways to do quantum computing and quantum encryption at 10.000 times the speed of light, but they all agreed that it was not possible to send information faster than lightspeed, because they could not control the contents of quantum entangled photons. But very recently, Graeme Smith, a researcher at IBM has proposed a way to "transmit large amounts of quantum information" described in the following paper, published in Feb, 2009:
http://domino.research.ibm.com/comm/research_people.nsf/pages/graemesm.Main.html/$FILE/NonPrivate.pdf
http://tech.slashdot.org/article.pl?sid=08/08/06/0043220
If his theory holds true, IBM or someone else could make a computer peripheral based on quantum technology (sort of like an Apple Airport) that can communicate large amounts of data instantaneously! Distance between client and cloud would no longer be a problem and transmission lag would be non-existent! It would make playing server side rendered games an almost lag-free experience and the ultimate alternative for costly, power hungry consoles.
Saturday, May 30, 2009
Ruby 2009 and the future of games
This presentation from Jules Urbach shows what the next Ruby demo could look like:
http://developer.amd.com/gpu_assets/ATIGPGPUComputingFusionRenderCloudGDC09CompatibilityMode.pdf
http://business.outlookindia.com/newolb/article.aspx?101987
However, OTOY CEO, Jules Urbach, is optimistic: "We can put any game on the cloud. Eventually, we can virtualise everything, including consoles, hardware and even Grand Theft Auto games. In three to five years, consoles will look different. Perhaps, consoles would be based on data on a cloud."The idea of server side rendering or cloud computing of games as proposed by OTOY, OnLive and others is really starting to grow on me. At first it didn't really seem that interesting or practical, but the more I think about it, the more I can see its potential. Some random thoughts:
Meanwhile, consumers across the US will test the prototype in the latter half of this year. Marketing plans, purchase points and so on would be developed later, depending on their verdict.
- No more console cycles: Gamers don't need to upgrade their hardware every 4-6 years and game developers don't have to frustratingly wait for the next round of consoles to be able to use new features such as DX11 compute shaders or tesselation
- Game developers can use the latest and greatest GPU's and algorithms instantly, without worrying about developing for a "common lowest denominator" (like Valve did with Half-life 2)
- Piracy that killed PC game development would be much harder (as it is now with Steam)
- Performance problems would be a thing of the past: just add a few hundred extra GPU's to the cloud and the framerate is buttersmooth again. There's no limit to the complexity and visual fidelity of a game: all depends on the willingness of the developer to invest in the server hardware
- no more costly multi-platform ports: the cloud harware is the only platform that needs to be targetted
- the cloud can be upgraded whenever newer CPU/GPU hardware becomes available, memory and SSD's can be added at any time
- It would also mean the end of the chicken and egg problem: small console install bases during the launch window scare developers away, but with server side rendering, everyone with a reasonably fast internet connection and a screen is a potential customer
- Most importantly: no more console wars (at least not of the hardware kind), no need to steal exclusives like Microsoft loves doing,
- Games will no longer be tied to one specific platform and will as a result be reviewed by press and gamers with less bias and have a better chance to sell
- no more hardware problems, red rings, repairs and warranty refunds on the client side
- very cheap, almost free "microconsoles", set top boxes, ... instead of $600 launch price which of course stiffles growth of install base
- the install base for cloud games already exists and is huge: all owners of Xbox 360, PS3, PC, Mac, set-top box with broadband access
- Games don't have to be bought in stores or downloaded and installed any more, but can be played from the moment the game is running in the cloud
- bigger potential for episodic game content (like Half-life 2 episodes on Steam)
- no more DVD royalties from developers to console makers (multiple DVD's for Rage on 360)
- no more limit on the size of the game content (if you want to make a 500GB game and the server has enough memory, go ahead)
- no more "console makers" in the classical sense: MS, Sony, Nintendo will not be making high-tech fully featured consoles anymore, but just simple set-top box like consoles or none at all. Their main focus will become online services instead of hardware. Of course Sony will still sell boatloads of TV's and Blu-ray players. Every electronics manufacturer that can make set-top boxes will be able to make a "next-gen" console.
- GPU makers nVidia and ATI will sell more GPUs to businesses (the render cloud owners, i.e. game publishers) and GPU sales to consumers will drastically decrease. This trend is already taking place: with the rapid death of the PC games market, less GPU's are sold to consumers directly and more to console manufacturers. On top of that, they will not be stuck with selling the same GPU for 6 years in a row.
In short, there are much less restrictions on game development if the cloud server is big enough.
The question that everyone seems to ask with respect to cloud rendering: what about lag? OTOY and OnLive seem to have the answer: new and vastly improved compression algorithms. Latency is reduced to a few milliseconds, not perceivable by the human eye. So I think that problem is solved. An unsolved question is how to make money of cloud games.
Obviously, if cloud rendering becomes mainstream, it will completely transform the current game console landscape. Every game publisher will be able to run its own game cloud (think about Valve and Steam but on a global scale) and offer their own online service with an online store, without having to pay royalties to console makers (just as it was in the Good Ol' PC days).
Saturday, April 4, 2009
More OTOY video's
Jules Urbach talking about cloud rendering:
http://neuronspark.com/videos/rendering-in-the-cloud/
An early concept video of LivePlace from March 2007, commented by Jules Urbach. Doesn't use voxel raycasting. Shows scenes created by JJ Palomo from 3DBlasphemy, currently at Big Lazy Robot VFX, and working together with OTOY:
http://metanomics.net/otoy
http://neuronspark.com/videos/rendering-in-the-cloud/
An early concept video of LivePlace from March 2007, commented by Jules Urbach. Doesn't use voxel raycasting. Shows scenes created by JJ Palomo from 3DBlasphemy, currently at Big Lazy Robot VFX, and working together with OTOY:
http://metanomics.net/otoy
Thursday, April 2, 2009
Voxels, id tech 6, stuff
Great video of Zelex (Jon Olick) graphics engineer and researcher at id Software, showing the sparse voxel octree technology for id Tech 6:
It's the übersexy Imrod! modelled in Z-Brush by Dmitry Parkin
http://www.youtube.com/watch?v=VpEpAFGplnI
There's a Siggraph presentation about SVO from Jon Olick as well
http://s08.idav.ucdavis.edu/olick-current-and-next-generation-parallelism-in-games.pdf
Gigavoxels: high quality sparse voxel octree raycasting from Cyril Crassin:
http://www.youtube.com/watch?v=HScYuRhgEJw
And an awesome blog about voxel rendering:
http://voxels.blogspot.com
It's the übersexy Imrod! modelled in Z-Brush by Dmitry Parkin
http://www.youtube.com/watch?v=VpEpAFGplnI
There's a Siggraph presentation about SVO from Jon Olick as well
http://s08.idav.ucdavis.edu/olick-current-and-next-generation-parallelism-in-games.pdf
Gigavoxels: high quality sparse voxel octree raycasting from Cyril Crassin:
http://www.youtube.com/watch?v=HScYuRhgEJw
And an awesome blog about voxel rendering:
http://voxels.blogspot.com
Ruby, Ruby, Rubaaaaaaaaaaayyyyy
So, almost 10 months have passed since the first unveiling of the Ruby/OTOY/Cinema 2.0/whatever demo. AMD still hasn't released the demo. For shame.
Since my last post in October, their was a little media fart about OTOY's server side rendering around CES09 in January. They're gonna build a 1 Petaflop supercomputer, built out of ATI cards.
Recently, OTOY showed up at GDC as well. They showed a completely lifelike CG model of Ruby's head, made with LightStage. Only the upper half for now. So damn sexy...
linky for the pic: http://www.pcper.com/comments.php?nid=6967 Skip to 50:00 and turn off the volume, cos the show host doesn't know crap about what he's talking about.
Monday, October 6, 2008
Dynamic voxels
In the past couple of weeks, I've learned that there are different methods to efficiently store voxels for GPU raycasting:
- Octrees
- Geometry images (Hoppe 2002, Carr et al., 2006)
- Spatial hashes
- Hybrid aceleration structures such as an octree with bricks (Crassin et al., 2008)
Jules Urbach said in an article on TechCrunch (http://www.techcrunch.com/2008/08/20/the-truth-behind-liveplaces-photo-realistic-3d-world-and-otoys-rendering-engine/):
Lightstage 5 is being used to capture performances of real actors in polygon based animations, which are then converted to voxels and stored in geometry maps (or geometry images, see A Brief Overview of Geometry Maps). So there is at least one way to render characters and dynamic objects through voxel raycasting, without the need for hybrid techniques. The paper from Carr et al. (Fast GPU Ray Tracing of Dynamic Meshes using Geometry Images) shows that "interactive" raycasting of dynamic objects is possible "at no extra cost". They use geometry maps to store triangles however instead of voxels. With this method, it's feasible to have extremely detailed characters raycasted in realtime.
Spatial hash maps could possibly be used as for dynamics as well.
In the TechCrunch article Jules Urbach gives some more info on the rendering methods behind OTOY and the Ruby voxel demo:
- Octrees
- Geometry images (Hoppe 2002, Carr et al., 2006)
- Spatial hashes
- Hybrid aceleration structures such as an octree with bricks (Crassin et al., 2008)
Jules Urbach said in an article on TechCrunch (http://www.techcrunch.com/2008/08/20/the-truth-behind-liveplaces-photo-realistic-3d-world-and-otoys-rendering-engine/):
We store voxel data in several ways, including geometry maps (see our Siggraph or Iceland presentations, where we show this method applied to the Ligthstage 5 structured light data, courtesy Andrew Jones ICT/Graphics lab)
Lightstage 5 is being used to capture performances of real actors in polygon based animations, which are then converted to voxels and stored in geometry maps (or geometry images, see A Brief Overview of Geometry Maps). So there is at least one way to render characters and dynamic objects through voxel raycasting, without the need for hybrid techniques. The paper from Carr et al. (Fast GPU Ray Tracing of Dynamic Meshes using Geometry Images) shows that "interactive" raycasting of dynamic objects is possible "at no extra cost". They use geometry maps to store triangles however instead of voxels. With this method, it's feasible to have extremely detailed characters raycasted in realtime.
Spatial hash maps could possibly be used as for dynamics as well.
In the TechCrunch article Jules Urbach gives some more info on the rendering methods behind OTOY and the Ruby voxel demo:
- The datasets from the BCN and Ruby city scenes contain up to 64 data layers per voxel, including diffuse albedo, fresnel reflectance values, irradiance data, UV coordinates (up to 8 sets), normals, and, for static scenes, look up vectors for 1-20 bounces of light from up to 252 evenly distributed viewpoints (it is important to note that this data is always 100% optional, as the raycaster can do this procedurally when the voxels are close and reflection precision is more important than speed; however, with cached reflectance data, you might see the scene rendering at 100s-1000s of fps when the scene isn’t changing).
- A note on raytracing vs. rasterization: amplifying the tree trunk in Fincher’s Bug Snuff demo to 28 million polys using the GPU tessellator turned out to be faster than rendering a 28 million voxel point cloud for this object. So there is a threshold where voxels become faster than rasterziation at about 100 million polys. At least in our engine, on R7xx GPUs, using full precision raycasting at 1280×720. Below that point, traditional rasterization using the GPU tessellator seems to be faster for a single viewport.
- The engine can convert a 1 million poly mesh into voxel data in about 1/200th second on R770 (60 fps on R600 and 8800 GTX). This is useful for baking dense static scenes that are procedurally generated once, or infrequently, on the GPU. That is why some of the OTOY demos require the GPU tessellator to look right.
- Hard shadows in OTOY were done using rasterization until we got R770 in May. Now hard shadows, like reflections, can be calculated using raycasting, although shadow masks are still very useful, and raycasting with voxel data can still give you aliasing.
- We can use the raycaster with procedurally generated data (perlin generated terrain or clouds, spline based objects etc.). At Jon Peddie’s Siggraph event, we showed a deformation applied in real time to the Ruby street scene. It was resolution independent, like a Flash vector object, so you could get infinitely close to it with no stair stepping effects, and likewise, the shadow casting would work the same way.
- The voxel data is grouped into the rough equivalent of ‘triangle batches’ (which can be indexed into per object or per material groups as well). This allows us to work with subsets of the voxel data in the much the same way we do with traditional polygonal meshes.
- The reflections in the march 2007 ‘Treo’ video are about 1/1000th as precise/fast as the raycasting we now use for the Ruby demo on R770/R700.
- One R770 GPU can render about 100+ viewports at the quality and size shown in the ‘Treo’ video. When scenes are entirely voxel based, the number of simultaneous viewports is less important than the total rendered area of all the viewports combined.- The server side rendering system is currently comprised of systems using 8x R770 GPUs ( 8 Gb VRAM, 1.5 Kw power per box).
The full Ruby demo: http://www.youtube.com/watch?v=sWgQp_LL-CgHigh quality download: http://blip.tv/file/get/Ubergizmo-AMDR700RubyDemo193.mov
Saturday, August 2, 2008
Voxel ray tracing vs polygon ray tracing
Carmack's thoughts about ray tracing:
- ray tracing in the classical sense, in which rays intersect with triangles, is far too expensive for use in games, even with next generation hardware
- the sparse voxel octree format permits unique geometry
Octrees can be used to accelerate ray tracing and store geometry in a compressed format at the same time.
Quote from a game developer (Rare) on the voxel octree:
Voxels have some very interesting benefits compared to polygons:
- It's a volumetric representation, so you can model very fine details and bumps, without the need for bump mapping. Particle effects like smoke, fire and foam can be efficiently rendered without using hacks. Voxels are also being used by some big Hollywood special effects studio's to render hair, fur and grass.
- id wants to use voxels to render everything static with real geometry without using normal maps.
- Voxels can store a color and a normal. For the renderer, textures and geometry are essentially the same.
- The position of the voxel is defined implicitely by the structure that is holding it (the octree). Here's the good part: this structure represents both the primitives that need to be intersected and the spatial division of these primitives. So, in contrast to triangle ray tracing which needs a separate spatial division structure (kd-tree, BVH, ...), voxels are right away structured in a grid or an octree (this does not mean that other structures can't be used as well). So for voxel ray tracing, octrees are perfect.
- Voxels are very cheap primitives to intersect, much cheaper than triangles. This is probably their biggest benefit when choosing between voxel and polygon ray tracing.
- A voxel octree permits a very natural multiresolution. There's no need to go deeper into the octree when the size of a pixel is larger than the underlying cell, so you don't have to display detail if it's not necessary and you don't streal in data that isn't visible either way.
- Voxels are extremely well suited for local effects (voxel ray casting). In contrast to triangle rasterization, there are no problems with transparency, refraction, ... There are also major benefits artwise: because voxels are volumetric, you can achieve effects like erosion, aging materials, wear and tear by simply changing the iso value.
- Ray casting voxels is much less sensitive to scene complexity than triangles
(partly translated from http://forum.canardplus.com/showpost.php?p=1257790&postcount=96)
Disadvantages of voxel ray tracing vs polygon ray tracing:
- Memory. Voxel data sets are huge relative to polygon data. But this doesn't have to be a problem, since all data can be streamed in. This does however create new challenges when the point of view changes rapidly and a lot of new data bricks have to be streamed in at once. Voxels sets have the benefit over polygons that voxel subsets can be loaded in, which permits some sort of progressive refinement. Other possible solutions are: using faster hard disks or solid state drives to accelerate the streaming, limiting depth traversal during fast camera movement or masking the streaming with motion blur or depth of field postprocessing.
- Animation of voxels requires specialized tools
- Disadvantages of ray tracing in general: dynamic objects require the octree to be updated in realtime. However, there are solutions for dynamic objects which don't require updating of the octree (such as building a deformation lattice around dynamic objects so that when you raycast into it bend the rays as it hits the deformation lattice). id Tech 6 plans to tackle the problem of having many dynamic objects with hybrid rendering.
More on dynamic raytracing:
Dynamic Acceleration Structures for Interactive Ray Tracing, Reinhard, E., Smits, B., and Hansen, C., in Proc. Eurographics Workshop on Rendering, pp. 299-306, June 2000. Summary: This system uses a grid data structure, allowing dynamic objects to be easily inserted or removed. The grid is tiled in space (i.e. it wraps around) to avoid problems with fixed boundaries. They also implement a hiearchical grid with data in both internal and leaf nodes; objects are inserted into the optimal level.
Towards Rapid Reconstruction for Animated Ray Tracing, Lext and Akenine-Moller, Eurographics 2001. Summary: Each rigid dynamic object gets its own grid acceleration structure, and rays are transformed into this local coordinate system. Surprisingly, they show that this scheme is not a big win for simple scenes, because in simple scenes it is possible to completely rebuild the grid each frame using only about a quarter of the runtime. But, this would probably not be true for a k-d or BSP tree.
Distributed Interactive Ray Tracing of Dynamic Scenes, Wald, Benthin, and Slusallek, Proc. IEEE Symp. on Parallel and Large-Data Visualization and Graphics (PVG), 2003. Summary: This system uses ray transformation (into object coordinate system) for rigid movement, and BSP rebuild for unstructured movement. A top-level BSP tree is rebuilt every frame to hold bounding volumes for the moving objects. Performance is still an issue for unstructured movement.
More info on id Tech 6 and voxel ray casting in the ompf thread
There are some interesting things to note in there:I think that ray tracing in the classical sense, of analytically intersecting rays with conventionally defined geometry, whether they be triangle meshes or higher order primitives, I’m not really bullish on that taking over for primary rendering tasks which is essentially what Intel is pushing. But, I do think that there is a very strong possibility as we move towards next generation technologies for a ray tracing architecture that uses a specific data structure, rather than just taking triangles like everybody uses and tracing rays against them and being really, really expensive. It
involves ray tracing into a sparse voxel octree which is essentially a geometric evolution of the mega-texture technologies that we’re doing today for uniquely texturing entire worlds. It’s clear that what we want to do in the following generation is have unique geometry down to the equivalent of the texel across everything.
- ray tracing in the classical sense, in which rays intersect with triangles, is far too expensive for use in games, even with next generation hardware
- the sparse voxel octree format permits unique geometry
Octrees can be used to accelerate ray tracing and store geometry in a compressed format at the same time.
Quote from a game developer (Rare) on the voxel octree:
Storing data in an octree is far more efficient than storing it using textures and polygons (it's basically free compression for both geometry and texture data). It's primarily cool because you stop traversing when the size of the pixel is larger than the projected cell, so you don't even need to have all your data in memory, but can stream it in on demand. This means that the amount of data truly is unlimited, or at least the limits are with the artists producing it. You only need a fixed amount of voxels loaded to view a scene, and that doesn't change regardless of how big the scene is. The number of voxels required is proportional to the number of pixels on the screen. This is true regardless of how much data you're rendering! This is not true for rasterization unless you have some magical per-pixel visibility and LOD scheme to cut down the number of pixels and vertices to process, which is impossible to achieve in practice. Plus ray casting automatically gives you exact information on what geometry needs to be loaded in from disk, so it's a "perfect" streaming system,
wheras with rasterization it would be very difficult to incrementally load a scene depending on what's visible (because you need to load the scene before you know what's visible!)
If you want to model micrometer detail, go ahead, it won't be loaded into memory until someone zooms in close enough to see it. Voxels that are not intersected can be thrown out of memory. Of course you would keep some sort of cache and throw things out on a least recently used basis, but since it's hierarchical you can just load in new levels in the hierarchy only when you hit them.
Voxels have some very interesting benefits compared to polygons:
- It's a volumetric representation, so you can model very fine details and bumps, without the need for bump mapping. Particle effects like smoke, fire and foam can be efficiently rendered without using hacks. Voxels are also being used by some big Hollywood special effects studio's to render hair, fur and grass.
- id wants to use voxels to render everything static with real geometry without using normal maps.
- Voxels can store a color and a normal. For the renderer, textures and geometry are essentially the same.
- The position of the voxel is defined implicitely by the structure that is holding it (the octree). Here's the good part: this structure represents both the primitives that need to be intersected and the spatial division of these primitives. So, in contrast to triangle ray tracing which needs a separate spatial division structure (kd-tree, BVH, ...), voxels are right away structured in a grid or an octree (this does not mean that other structures can't be used as well). So for voxel ray tracing, octrees are perfect.
- Voxels are very cheap primitives to intersect, much cheaper than triangles. This is probably their biggest benefit when choosing between voxel and polygon ray tracing.
- A voxel octree permits a very natural multiresolution. There's no need to go deeper into the octree when the size of a pixel is larger than the underlying cell, so you don't have to display detail if it's not necessary and you don't streal in data that isn't visible either way.
- Voxels are extremely well suited for local effects (voxel ray casting). In contrast to triangle rasterization, there are no problems with transparency, refraction, ... There are also major benefits artwise: because voxels are volumetric, you can achieve effects like erosion, aging materials, wear and tear by simply changing the iso value.
- Ray casting voxels is much less sensitive to scene complexity than triangles
(partly translated from http://forum.canardplus.com/showpost.php?p=1257790&postcount=96)
Disadvantages of voxel ray tracing vs polygon ray tracing:
- Memory. Voxel data sets are huge relative to polygon data. But this doesn't have to be a problem, since all data can be streamed in. This does however create new challenges when the point of view changes rapidly and a lot of new data bricks have to be streamed in at once. Voxels sets have the benefit over polygons that voxel subsets can be loaded in, which permits some sort of progressive refinement. Other possible solutions are: using faster hard disks or solid state drives to accelerate the streaming, limiting depth traversal during fast camera movement or masking the streaming with motion blur or depth of field postprocessing.
- Animation of voxels requires specialized tools
- Disadvantages of ray tracing in general: dynamic objects require the octree to be updated in realtime. However, there are solutions for dynamic objects which don't require updating of the octree (such as building a deformation lattice around dynamic objects so that when you raycast into it bend the rays as it hits the deformation lattice). id Tech 6 plans to tackle the problem of having many dynamic objects with hybrid rendering.
More on dynamic raytracing:
Dynamic Acceleration Structures for Interactive Ray Tracing, Reinhard, E., Smits, B., and Hansen, C., in Proc. Eurographics Workshop on Rendering, pp. 299-306, June 2000. Summary: This system uses a grid data structure, allowing dynamic objects to be easily inserted or removed. The grid is tiled in space (i.e. it wraps around) to avoid problems with fixed boundaries. They also implement a hiearchical grid with data in both internal and leaf nodes; objects are inserted into the optimal level.
Towards Rapid Reconstruction for Animated Ray Tracing, Lext and Akenine-Moller, Eurographics 2001. Summary: Each rigid dynamic object gets its own grid acceleration structure, and rays are transformed into this local coordinate system. Surprisingly, they show that this scheme is not a big win for simple scenes, because in simple scenes it is possible to completely rebuild the grid each frame using only about a quarter of the runtime. But, this would probably not be true for a k-d or BSP tree.
Distributed Interactive Ray Tracing of Dynamic Scenes, Wald, Benthin, and Slusallek, Proc. IEEE Symp. on Parallel and Large-Data Visualization and Graphics (PVG), 2003. Summary: This system uses ray transformation (into object coordinate system) for rigid movement, and BSP rebuild for unstructured movement. A top-level BSP tree is rebuilt every frame to hold bounding volumes for the moving objects. Performance is still an issue for unstructured movement.
Interactive Space Deformation with Hardware Assisted Rendering, IEEE Computer Graphics and Applications, Vol 17, no 6, 1997, pp. 66-77. Summary: Instead of deforming objects directly, this system deforms the space in which they reside (using 1-to-1 deformations). During raytracing, the rays are deformed into the object space instead of deforming the objects into the ray space. However, the resulting deformed rays are no longer straight, so they must be discretized into short line segments to perform the actual ray-object intersection tests.
Ray casting free-form deformed-volume objects, Haixin Chen, Jürgen Hesser, Reinhard Männer A collection of techniques is developed in this paper for ray casting free-form deformed-volume objects with high quality and efficiency. The known inverse ray deformation approach is combined with free-form deformation to bend the rays to the opposite direction of the deformation, producing an image of the deformed volume without generating a really deformed intermediate volume. The local curvature is estimated and used for the adaptive selection of the length of polyline segments, which approximate the inversely deformed ray trajectories; thus longer polyline segments can be automatically selected in regions with small curvature, reducing deformation calculation without losing the spatial continuity of the simulated deformation. We developed an efficient method for the estimation of the local deformation function. The Jacobian of the local deformation function is used for adjustments of the opacity values and normal vectors computed from the original volume, guaranteeing that the deformed spatial structures are correctly rendered. The popular ray casting acceleration techniques, like early ray termination and space leaping, are incorporated into the deformation procedure, providing a speed-up factor of 2.34-6.56 compared to the non-optimized case.
More info on id Tech 6 and voxel ray casting in the ompf thread
Carmack, id Tech 6, hybrid rendering
In his QuakeCon keynote, John Carmack explained that his next generation engine id Tech 6 would still be mainly a hybrid renderer:
Full voxel based games with many dynamic objects and characters are still too demanding for next generation technology. On the other hand, a scene with few dynamic objects (such as the Ruby demo) should be entirely possible with voxels only.
I can say with conviction at this point that the next generation games are still going to be predominantly polygon games. Even what we're looking at for id Tech 6 with all of this infinite geometry, voxelising everything, probably recursive automatic geometry generation - all of this is still going to be a hybrid approach. We hope that we can generate these incredible lush environments on there, but the characters are probably still going to be coming in as triangles over a skeleton there. There will probably be some interesting things tried with completely non-polygonal renderers, but the practical approach with games that look like the games we're doing now, but play better, probably will still have lots of polygons going on and these chips better be really good at that.
Full voxel based games with many dynamic objects and characters are still too demanding for next generation technology. On the other hand, a scene with few dynamic objects (such as the Ruby demo) should be entirely possible with voxels only.
Friday, August 1, 2008
id, Voxels and Ray Tracing
According to this article, the full Ruby demo will be shown to the public at Siggraph 2008.
My interest in this demo is, apart from the photorealistic quality, based on two facts: the GPU ray tracing and the voxel based rendering. Never before have I seen a raytraced (CPU or GPU) scene of this scope and quality in realtime. Urbach has stated in the video's that his raytracing algorithm is not 100% fully accurate, but nevertheless I think it looks absolutely amazing.
At Siggraph 2008, there will be a panel discussion on realtime ray tracing, where Jules Urbach will be the special guest. Hopefully, there will be more info on the ray tracing part then.
On to the voxels...
In March of this year, John Carmack stated in an interview that he was investigating a new rendering technique for his next generation engine (id Tech 6), which involves raycasting into a sparse voxel octree. This has spurred renewed interest in voxel rendering and parallels with the new Ruby demo are quickly drawn.
Today's GPU are already blazingly fast when it comes to polygon rendering and don't break a sweat in the multimillion triangle scenes of Crysis. So there must be a good reason why some developers are spending time and energy on voxel rendering. John Carmack explains it like this in the interview:
In the ompf thread, there are also a number of interesting links to research papers about voxel octree raycasting:
A single-pass GPU ray casting framework for interactive out-of-core rendering of massive volumetric datasets Enrico Gobbetti, Fabio Marton, and José Antonio Iglesias Guitián 2008
http://www.crs4.it/vic/cgi-bin/bib-page.cgi?id=
Interactive Gigavoxels, Cyril Crassin, Fabrice Neyret, Sylvain Lefebvre 2008
http://artis.imag.fr/Publications/2008/CNL08/
Ray tracing into voxel compressed into an octree http://www.sci.utah.edu/~wald/Publications/2007///MROct/download//mroct.pdf
The octree texture Sylvain Lefebvre
http://lefebvre.sylvain.free.fr/octreetex/
The difference between id Tech 6 and Otoy is the way the voxels are rendered: id's sparse voxel octree tech is about voxel ray casting (primary rays only), while Otoy does voxel raytracing, which allows for raytraced reflections and possibly even raytraced shadows and photon mapping.
My interest in this demo is, apart from the photorealistic quality, based on two facts: the GPU ray tracing and the voxel based rendering. Never before have I seen a raytraced (CPU or GPU) scene of this scope and quality in realtime. Urbach has stated in the video's that his raytracing algorithm is not 100% fully accurate, but nevertheless I think it looks absolutely amazing.
At Siggraph 2008, there will be a panel discussion on realtime ray tracing, where Jules Urbach will be the special guest. Hopefully, there will be more info on the ray tracing part then.
On to the voxels...
In March of this year, John Carmack stated in an interview that he was investigating a new rendering technique for his next generation engine (id Tech 6), which involves raycasting into a sparse voxel octree. This has spurred renewed interest in voxel rendering and parallels with the new Ruby demo are quickly drawn.
Today's GPU are already blazingly fast when it comes to polygon rendering and don't break a sweat in the multimillion triangle scenes of Crysis. So there must be a good reason why some developers are spending time and energy on voxel rendering. John Carmack explains it like this in the interview:
Jon Olick, programmer at id Software, provided some interesting details about the sparse voxel octree raycasting in this ompf thread. He will also give a talk on the subject at Siggraph.It’s interesting that if you look at representing this data in this particular sparse voxel octree format it winds up even being a more efficient way to store the 2D data as well as the 3D geometry data, because you don’t have packing and bordering issues. So we have incredibly high numbers; billions of triangles of data that you store in a very efficient manner. Now what is different about this versus a conventional ray tracing architecture is that it is a specialized data structure that you can ray trace into quite efficiently and that data structure brings you some significant benefits that you wouldn’t get from a triangular structure. It would be 50 or 100 times more data if you stored it out in a triangular mesh, which you couldn’t actually do in practice.
In the ompf thread, there are also a number of interesting links to research papers about voxel octree raycasting:
A single-pass GPU ray casting framework for interactive out-of-core rendering of massive volumetric datasets Enrico Gobbetti, Fabio Marton, and José Antonio Iglesias Guitián 2008
http://www.crs4.it/vic/cgi-bin/bib-page.cgi?id=
Interactive Gigavoxels, Cyril Crassin, Fabrice Neyret, Sylvain Lefebvre 2008
http://artis.imag.fr/Publications/2008/CNL08/
Ray tracing into voxel compressed into an octree http://www.sci.utah.edu/~wald/Publications/2007///MROct/download//mroct.pdf
The octree texture Sylvain Lefebvre
http://lefebvre.sylvain.free.fr/octreetex/
The difference between id Tech 6 and Otoy is the way the voxels are rendered: id's sparse voxel octree tech is about voxel ray casting (primary rays only), while Otoy does voxel raytracing, which allows for raytraced reflections and possibly even raytraced shadows and photon mapping.
Otoy, Transformers and Ray Tracing
After the Ruby/LightStage demo, 4 other video's appeared as part of an article about Otoy on TechCrunch. Urbach explains that he started experimenting with Renderman code on graphics hardware during the making of Cars in 2005. This work caught the interest from ILM, who gave Urbach the models from the Transformer movie to render in realtime. Urbach and his team made 4 commercials for the Transformer movie that were rendered and directed in realtime on graphics hardware. Afterwards, he was contacted by Sony to work on the Spiderman movie.
The 4 video's:
Video 1 OTOY Demo
This video shows short clips of realtime rendered Transformer sequences
Video 2 Jules Urbach explains OTOY's real-time graphics rendering
In this video, Urbach talks about his experiments with GPU ray tracing in 2005, the Transformers trailers and the voxel raytracing for the Ruby demo. For the tests with Cars in 2005, he was able to do "realtime raytraced reflections with up to 20 bounces of light". He also implemented some realtime global illumination technique. For the new Ruby demo, he is actually "raytracing the entire scene", and "not using the vertex pipeline anymore". Thanks to the voxel rendering "the level of detail becomes infinite".
Video 3 OTOY Graphics Rendered in the Browser
This video shows the server side rendering capabilities of Otoy. It shows Urbach interacting with scenes from the Transformers trailers, that are being rendered in realtime on his GPU servers and streamed over the net into the browser.
Urbach mentions "raytraced reflections on the windows". When he switches to nighttime, he says "in this particular demo, there's no baked lighting, nothing is precomputed", there are "hundreds of lights in the building rendered in realtime".
The demo runs on three graphics cards (3x RV770): one card renders the ILM Optimus, second card renders the G1 Optimus Prime and the third card renders the city and the raytraced reflections on the windows.
Video 4 Jules Urbach of OTOY Explains LightStage
Video about LightStage, slightly more elaborate than this one.
There is also a video of the full AMD Cinema 2.0 event in which Urbach talks a bit about ray tracing on GPU's (from 41:00 to 47:00) and goes a bit more in-depth during the Q&A session (from 72:00 to 88:00):
- Urbach has been talking to game publishers to start integrating the relighting part of Otoy in existing game engines
- Otoy can do full raytracing, but also supports hybrid rendering. It can convert any polygonal mesh to voxels
- The Ruby demo does not use any polygons, only voxels
- For games, Urbach thinks hybrid rendering will be the way to go "for a very long time"
- With this technology, game developers will require a different way of working. Basically they're saying that you can make a photorealistic game, but the workload on the artist side will be astronomous
- In 2005, Urbach started out writing approximations to Renderman code during the making of Cars. At the time, he used cheats for ray tracing and reflections. In three years, GPU’s have evolved so quickly that the latest hardware makes realtime ray tracing possible that is “99 % accurate”
- Voxel data sets are huge, but with voxel based rendering you can load only subsets of the voxel space, which is not possible with polygons. You can also choose which texture layers to load
- Compression and decompression of the voxel data is CPU bound. What takes 3 seconds to decompress on a CPU, can be done at a “thousand frames per second” on a GPU.
- What's interesting according to Urbach is that in 2005 he started out writing approximations to ray tracing, but the latest generation of hardware allows him to do ray tracing that gets really close to the 100% point
Urbach also showed another Otoy demo at the AMD event, called Bug Snuff. It shows a photorealistic scene with a scorpion, rendered in realtime and directed by David Fincher. Really impressive stuff!
Lastly, the ompf thread where it all started: http://ompf.org/forum/viewtopic.php?f=6&t=882
Thanks to all the ompf members and guests who participated and contributed to the thread.
The 4 video's:
Video 1 OTOY Demo
This video shows short clips of realtime rendered Transformer sequences
Video 2 Jules Urbach explains OTOY's real-time graphics rendering
In this video, Urbach talks about his experiments with GPU ray tracing in 2005, the Transformers trailers and the voxel raytracing for the Ruby demo. For the tests with Cars in 2005, he was able to do "realtime raytraced reflections with up to 20 bounces of light". He also implemented some realtime global illumination technique. For the new Ruby demo, he is actually "raytracing the entire scene", and "not using the vertex pipeline anymore". Thanks to the voxel rendering "the level of detail becomes infinite".
Video 3 OTOY Graphics Rendered in the Browser
This video shows the server side rendering capabilities of Otoy. It shows Urbach interacting with scenes from the Transformers trailers, that are being rendered in realtime on his GPU servers and streamed over the net into the browser.
Urbach mentions "raytraced reflections on the windows". When he switches to nighttime, he says "in this particular demo, there's no baked lighting, nothing is precomputed", there are "hundreds of lights in the building rendered in realtime".
The demo runs on three graphics cards (3x RV770): one card renders the ILM Optimus, second card renders the G1 Optimus Prime and the third card renders the city and the raytraced reflections on the windows.
Video 4 Jules Urbach of OTOY Explains LightStage
Video about LightStage, slightly more elaborate than this one.
There is also a video of the full AMD Cinema 2.0 event in which Urbach talks a bit about ray tracing on GPU's (from 41:00 to 47:00) and goes a bit more in-depth during the Q&A session (from 72:00 to 88:00):
- Urbach has been talking to game publishers to start integrating the relighting part of Otoy in existing game engines
- Otoy can do full raytracing, but also supports hybrid rendering. It can convert any polygonal mesh to voxels
- The Ruby demo does not use any polygons, only voxels
- For games, Urbach thinks hybrid rendering will be the way to go "for a very long time"
- With this technology, game developers will require a different way of working. Basically they're saying that you can make a photorealistic game, but the workload on the artist side will be astronomous
- In 2005, Urbach started out writing approximations to Renderman code during the making of Cars. At the time, he used cheats for ray tracing and reflections. In three years, GPU’s have evolved so quickly that the latest hardware makes realtime ray tracing possible that is “99 % accurate”
- Voxel data sets are huge, but with voxel based rendering you can load only subsets of the voxel space, which is not possible with polygons. You can also choose which texture layers to load
- Compression and decompression of the voxel data is CPU bound. What takes 3 seconds to decompress on a CPU, can be done at a “thousand frames per second” on a GPU.
- What's interesting according to Urbach is that in 2005 he started out writing approximations to ray tracing, but the latest generation of hardware allows him to do ray tracing that gets really close to the 100% point
Urbach also showed another Otoy demo at the AMD event, called Bug Snuff. It shows a photorealistic scene with a scorpion, rendered in realtime and directed by David Fincher. Really impressive stuff!
Lastly, the ompf thread where it all started: http://ompf.org/forum/viewtopic.php?f=6&t=882
Thanks to all the ompf members and guests who participated and contributed to the thread.
The Ruby demo continued
One week after the unveiling of the Ruby teaser, Jules Urbach recorded a new video, in which he gave a bit more info on the voxel based rendering that was used in the Ruby demo and announced his other company, LightStage LLC. LightStage is a technology used by Hollywood film studio's to capture lifelike 3D representations of actors, which can be relighted afterwards. It has been used in Spiderman 3 and will also be used to motion capture and digitalize the actress playing Ruby. Her digital facsimile will then replace the Ruby character from the teaser.
Video: http://youtube.com/watch?v=Bz7AukqqaDQ
Full transcript:
Video: http://youtube.com/watch?v=Bz7AukqqaDQ
Full transcript:
Hi, I’m Jules Urbach and this is a follow-up to the presentation we did a week ago at Cinema 2.0, the launch for the 770. What we are showing is a couple of new things that we weren’t able to talk about last week, that I think are really interesting. So we’re announcing today that we are developing LightStage. LightStage LLC is a separate company for capturing and rendering really high quality 3d characters and LightStage is an interesting technology that was developed at USC by Paul Debevec and Tim Hawkins, and it solves the problem of the uncanny valley as far as characters go. So we’re very pleased we’re able to announce that we can actually take this data and start working with it and applying it in our projects.
So, if you take a look at what we were doing years ago, to do characters and animation it was limited by the fact that our artists can only create so much detail in a head or a human form. And this is the normal map for that head and this is a really complex skin shader that we wrote to try to recreate what humans look like. And in a lot of ways both these problems go away with the LightStage.
The LightStage first of all can capture a real person. So, to generate this head, an artist doesn’t even sit there and sculpt it. We can essentially put a woman in the LightStage, which is a domed capture environment, and it captures all the surfaces and all the normals for it and all the details, including the hair. And it captures it in full motion, so you have to understand that, unlike traditional motion capture where there is either make-up applied to the face or dots put on the face, we just put somebody in the LightStage and they can do their lines and speak and it does full motion capture optically.
So it gets the full data set of essentially all the points in their face. And this is in fact the rendered version of LightStage. This is all the data that is captured on the LightStage accumulated. It’s not a photograph. This is a fully relit head, based on the model you just saw. And it’s obviously a lot of data, but the work that we announced last week with the GPU, compression / decompression, we’re going to apply that to LightStage (and have) data sets that we can start loading in real-time and rendering them, not just for film work but in games as well.
So the LightStage data, I think it really closes the uncanny valley. I mean particularly this kind of data where we have all the relighting information, stored for every single pixel, it’s exciting and it gives us really high quality characters that I think look completely real. And that’s, I think one of the things we showed last week at the Cinema 2.0 event was that we could do scenes and cities and things that look really good and this essentially gives us people. And it goes even further than that, but we will certainly be announcing more as we get further along working on LightStage.
I’m gonna show one more thing that we didn’t get a chance to really show last week as well, which is some of the real-time stuff, the voxel rendering. We basically had two separate demos, one for just showing the fact that we can look around the
voxelized scene and render it. And this one now, this demo, is a slight update of the original one, where I’m able to actually look around and essentially place voxels in
the scene, but also relight it as well.So this is part two of three that we’re releasing. This shows essentially us going through the scene, selecting an object and either rendering it through the full lighting pipeline or just doing the global illumination pass. Then we can also just use the normals to do full, totally new novel reflections on. But you can essentially see that the voxels, even in this fairly low resolution form, can capture lighting information and capture all these different details. And we can do that as we’re navigating through the scene.
So this is sort of part two of our Ruby real-time demo and part three of it is gonna show, as a next step, the full navigation through the voxel scenes. And that’s gonna be dependent on the compression we’re developing. Because right now, the reason why we are not loading the entire animation is that the frame data is about 700 megabytes for every frame. We can easily compress that down to 1/100th the size, we're looking to do about a 1000th the size. And then with that we’ll be able to load much larger voxel data sets and actually have you navigating pretty far throughout the scene and still keep the ray tracing and voxelization good enough that you don’t really see any sort of pixelized or voxelized data sets too closely.
So one more demo, that’s worth showing I think, is related to the LightStage. You can actually see that on the right here. And that basically is really a mesh that is generated from one person in LightStage. It doesn’t have all the LightStage data in there, but what you can see from this example is one reason why voxel rendering may be important. So this is really using a very simple polygonal mesh, evenso it’s about 32 million triangles, just to render the scene. So I’m gonna show the wireframe of it and you can see that the data is so dense, everything from the eyes to the eyelashes are all there, and we’re only really using a small subset of the point cloud that’s generated from the LightStage. So if we move to voxel rendering, which I’m planning to do for LightStage as soon as we’re done with the Ruby demo, we’ll be able to have voxelized assets rendering in realtime at mùch higher resolutions than this. And that’s gonna be giving us characters that look better than anything we can show in any of these videos. And we should have that ready probably before the end of the year, so it’s exciting stuff! Thank you for watching, hope you enjoyed the
demos.
The Ruby demo
To begin with... a picture says more than a thousand words
This is a picture from the Ruby demo. It looks photorealistic, but is realtime and interactive.
The Ruby demo was made by a company named Otoy. Jules Urbach, founder and CEO of Otoy, has given some info on the technology behind the Ruby demo in several video's on the net:
Video 1: http://www.youtube.com/watch?v=ROAJMfeRGD4&feature=related
In this video, Urbach says that the Ruby demo is rendered with voxel ray tracing. Otoy can also dynamically relight the scene.
The full transcript of the video:
This is a picture from the Ruby demo. It looks photorealistic, but is realtime and interactive.
A high quality video (720p) of the animation can be seen here
Low quality video hereThe Ruby demo was made by a company named Otoy. Jules Urbach, founder and CEO of Otoy, has given some info on the technology behind the Ruby demo in several video's on the net:
Video 1: http://www.youtube.com/watch?v=ROAJMfeRGD4&feature=related
In this video, Urbach says that the Ruby demo is rendered with voxel ray tracing. Otoy can also dynamically relight the scene.
The full transcript of the video:
The first time I saw this video, the words "ray tracing" and "voxels" immediately grabbed my attention. So I did some further research...Rick Bergman: So Jules, this is his creation. He's done a fantastic job with it. You're probably also thinking, well this is a video. He's gonna step you through and actually talk you some of the key features of this demo.
Jules Urbach: Thank you, Rick. What you're seeing here, is a frame from the animation you just saw, done Cinema 2.0 style. So the first thing you'll notice is that this isn't really just a video, we can look around, we can see the set that we've built, in fact it is, it's a set, you can see it's really ... When we first showed the clips of
what we were doing with this, some people thought the street scene was film, and it's not, it's a completely computer generated scene, created by our art team. And you can see here, this is the relighting portion of the rendering pipeline,this is really just a very early teaser, a preview of what we're doing with this Ruby demo.So you're seeing only the second half of the Cinema 2.0 rendering pipeline, the
relighting portion of it. I can drag the cursor over any object and I can sort of see the different layers that go into making it whether it's global illumination, photon maps, diffuse lighting or in this case, complete control over the scene and the reflections.And this is a really novel way of rendering graphics, we're not using any polygons. And the thing that makes this very different from just a simple relighting demo, is that every single pixel you're seeing in the scene has depth and it's essentially renderable as voxels.
We also have the capability of controlling every aspect of the exposure in the lighting pipeline, adding glares and glints to our satisfaction. And that makes a big impact in the rendering of any scene that we're doing.
So one of the things that is key to doing voxel based rendering is ray tracing, which I spoke about earlier, and the other element is compression, because these data sets are enormous. One of the things that's very exciting about the latest generation of hardware, coming from AMD, is that we can now write general purpose code, using CAL, that does wavelet compression. So we're able to compress these data sets, which are pretty massive, down to very reasonable components. And we think that we can stream those down and essentially give people, who have ever seen a video stream of that animation, essentially a fully navigable, relightable, completely interactive scene and that's the ... of 2.0 and we're very excited to be able to be part of that technology and that processing and bringing that to fruition.
Ruby, voxels and ray tracing
Intro
My name is Ray Tracey. I'm a graphics enthusiast with a passion for lifelike interactive graphics.
Recently, I was struck by a realtime demo of a photorealistic city scene that was shown at an AMD/ATI event. It looked like something that came straight out of my imagination. For years I have been wondering when interactive graphics would reach this level of quality. I was interested to say the least.
A week later, AMD revealed that this city scene was part of their new Ruby demo for the Radeon 4800 cards: they showed the same scene, but this time there was moving traffic and people. Ruby appeared, running for her life from a giant killer machine. Once again, I was in awe: it looked unbelievable, I didn't expect to see such graphics within the next three years on consumer hardware. So after I witnessed this graphical marvel, I decided to find out more about it. After several hours of Google'ing, I had come to the disheartening conclusion that there was almost no info on the demo to be found, except for some lousy quality youtube video's and a small PR paragraph on the AMD site. But I searched further... The small snippets of info that I did find, stimulated me to write this blog as a "resource" that bundles all the publicly available information on this technology and as a means for better understanding of the tech to anyone interested.
My name is Ray Tracey. I'm a graphics enthusiast with a passion for lifelike interactive graphics.
Recently, I was struck by a realtime demo of a photorealistic city scene that was shown at an AMD/ATI event. It looked like something that came straight out of my imagination. For years I have been wondering when interactive graphics would reach this level of quality. I was interested to say the least.
A week later, AMD revealed that this city scene was part of their new Ruby demo for the Radeon 4800 cards: they showed the same scene, but this time there was moving traffic and people. Ruby appeared, running for her life from a giant killer machine. Once again, I was in awe: it looked unbelievable, I didn't expect to see such graphics within the next three years on consumer hardware. So after I witnessed this graphical marvel, I decided to find out more about it. After several hours of Google'ing, I had come to the disheartening conclusion that there was almost no info on the demo to be found, except for some lousy quality youtube video's and a small PR paragraph on the AMD site. But I searched further... The small snippets of info that I did find, stimulated me to write this blog as a "resource" that bundles all the publicly available information on this technology and as a means for better understanding of the tech to anyone interested.
Subscribe to:
Posts (Atom)