Darkwind Media Blog

A while back I made a simple volumetric light demo in Luster. I just wanted to test the rendering technique. Colin then picked it up and decided he would make the light pulse to the beat of the music by doing some audio processing in Flash and sending commands back out to Luster. Turns out, it worked really well. I put in similar audio processing capabilities into the core Luster API and pretty soon that simple volumetric light demo became our first music visualizer.

This is the second visualizer we’ve made. This one focuses more heavily on particles, with a lot of action going on at once. There’s a vignette-style blur post-processor as well as healthy amounts of bloom. Almost everything in this application is affected by the music, including the camera movement, the particle motion, and the return of the spectrum targets from the first visualizer.

We haven’t uploaded this application to the beta tech demos section yet. Videos, screenshots, and the app itself will soon be on the Luster3D site. All music in the video is from Jason Mraz. Also, if you view the video on YouTube you can see the high-def version, which I recommend doing.

We recently finished up our skeletal animation tech demo, and I wanted to go into some of the details behind it.  One of the first things you’ll probably notice is that it’s not simply showing off the skeletal animation capabilities of Luster.  We ended up also using normal maps, scrolling UV texture animation, several shadow techniques and Flash/Luster communication.   And all of this runs in a web browser (this particular demo requires Shader Model 3).

The main questions we get are “How did you get the Arthas model?” and “How did you animate him?”  To answer the first question, we’ll go back to the beginning of the project.   We had been thinking about possible Luster demos for a while.  We wanted something that would not only show off the features of Luster, but get people talking about it.  Around this time everyone in the office had renewed their World of Warcraft accounts for a bit so we could play during our spare time.  Matt came up with the idea to use World of Warcraft models in Luster somehow.  I immediately opened up the WoW Model Viewer to see if there were any model export options.

Not only were we able to export models (with textures and skeletons intact), but the WoW Model Viewer could automatically grab a character’s gear from the Armory and dress the model up appropriately.  One of our initial tests was the get a character from the Armory and try to get it into Luster.  To do this we exported from the WoW Model Viewer in the Milkshape3D format.  This export also provided us with the textures for the model we exported.  Once in Milkshape we exported the model to FBX, which allowed us to get the model into Maya.  Getting the model into Maya was important for two reasons: 1) We could go from Maya to the Ogre3D Mesh format which is supported by Luster and 2) I actually know how to use Maya.

Once we figured out the pipeline, we scoured the WoW Model Viewer for more characters and models to import.  The sample application in Luster grew from one random character to having several characters from the Armory, Arthas, Illidan, a huge protodrake, Sylvanas and a boss from Naxxramas.  We added some stationary and dynamic lights and per-pixel lighting (with normal maps), which made the models look great, but something was definitely missing.  The models were too static.  They had no sense of life to them, no purpose.  It was at this point that we tried to extract the animation data from the WoW Model Viewer, but it doesn’t support animation exporting.  We could export different poses, like a single frame of an animation, but not the whole animation.  We did find some tools to extract the .anim files from World of Warcraft’s .MPQ content files, but these were not useful to us in their raw form.

Pretty disheartened by the fact that we couldn’t extract any animation data, I went into Maya to play around with the models we had.  After realizing that the models already had skeletons and the vertex/joint bindings were intact, I started animating an arm here and a head there.  It dawned on me that I could probably create my own animation on one of the characters and incorporate that into the application, maybe as an Easter egg of some sort.  I decided to use Arthas for a few reasons.  First, he is from the latest expansion and therefore has much more detail than many of the older models.  Second, he has some animated textures (his eyes and sword for instance) which look great.  Third, he just looks badass.  After the Arthas animation was complete, we decided to focus soley on that particular model since the animation process worked so well.

The controls for the animation were created in Adobe Flash.  The resulting .SWF file was imported into Luster and incorporated into the project.  The Flash movie and Luster communicate back and forth.  When you click on the play button, Flash tells Luster to start the animation.  Luster then tells Flash to update the scrollbar as the animation plays.

The finishing touches were to add a gradient background and some shadows.  I did the main ground shadows using a regular shadow mapping technique.  The self-shadow implementation was done by Brian and used Variance Shadow Maps (VSMs) to produce soft shadows.  The normal maps on the model were generated from the original textures using the NVIDIA Normal Map Filter Photoshop plugin.

-Chris

On May 2nd, during the ImagineRIT innovation festival, the vaudeville show for virtual theater was performed. We had 3 characters who could be dynamically assigned to two different real-time motion capture systems. The system then synchronized the production with >20 clients across the campus. The performance itself was controlled from the 3rd floor of the computing college in the Computer Science graphics lab. I acted as the “stage manager” with a special client that had access to activate staging cues for lighting, set changes, sound effects, etc. The director of the production was also in the graphics lab giving direction to the actors in the motion capture systems as well as myself for timing.

We are still compiling the different video feeds we had set up from that day. There was a video recorded from one of the labs that was linked into the performance. It was recorded by Andrew Brown, who wrote the plugin which output real-time motion capture data from AutoDesk’s MotionBuilder. Here’s the video from YouTube:

A new version of Luster (0.18.8) has been released. This new version has a few minor enhancements and a couple critical bug fixes for the nature plugin (terrain). This is the version that will run the Soarcraft demo. This player version is still part of our guarded Beta. If you’d like to see the power of Luster you can either email us, or jump over to the Beta sign-up page. After you sign up, we’ll be getting back to you with a link to download Luster and a few demos to play around with.

We’ve been continuing our work on our major contracts, forming new relationships and pushing Luster for our final release. There are a lot of useful enhancements coming with version 0.19. Our Mac version has been coming along nicely as well. At the same time we haven’t been neglecting the need for good demos to show you the power of Luster. Here’s a sneak peak of a demo showing the ease of importing assets, controlling animations, advanced lighting, and our revolutionary integration with Flash. We were able to take one of the many open source tools available and extract a World of Warcraft character model to create a custom animation. The full thing will be posted in a few days, so check back soon.

Another recent project done in Luster was World of Soarcraft. No, I did not come up with the name. This came out of a partnership between Vuzix and the Rochester Institute of Technology. Our task was to come up with a demo incorporating Vuzix head-mounted displays and 3D game technology before the Imagine RIT Innovation + Creativity Festival happening on May 2nd. We had about one week. Naturally, I turned to Luster as our 3D platform. I wrote a plugin for interfacing with the Vuzix display, connected a Wii remote for control and designed a simple 3D landscape to fly around in.

The system would be output both to a standard monitor and to the Vuzix display. This gave a more immersive feel to the demo. The best thing about the Vuzix displays are their built-in head tracking. Since they can track all 3 degrees of rotation they can be used to control what part of the 3D scene the user can see. As the user looks around, the scene changes to reflect the direction they are looking in the real world.

We aren’t yet sure if the project will continue into the summer. There is certainly a lot more than can be done with the system. Currently there is no gameplay. That would be an interesting component to add to the demo. Also, the flight is not terribly accurate (the physics of it) and so it would also be nice to make the flight mechanics a bit more realistic. As always, there will be updates when we know more.

Those that have followed the blog before will remember Virtual Theater. This is a grant project at RIT which is trying to create a system for theatrical performances that happen completely within a virtual performance space. The actors control virtual characters, performing on a virtual stage, in front of an audience that is watching from anywhere in the world through their computers.

Tomorrow, Virtual Theater’s Vaudeville performance will start rehearsals. This is a brand new show, and the first to be put on within the new Luster framework. Tomorrow we start hooking all of the motion capture systems up to the Luster clients and running through the script.

I haven’t gone into a lot of detail before, but I want to tell you that Virtual Theater actually demands a quite complicated network of systems. RIT currently has access to two motion capture systems, both located in separate buildings on campus. These systems must send their motion capture data in real-time to a server running AutoDesk’s MotionBuilder software. We have created a plugin for MotionBuilder which immediately sends out the real-time motion data to our client systems. Running along-side the motion capture server is the main Virtual Theater server. While the motion capture server sends out the real-time motion data, the general Virtual Theater server is responsible for funneling all other performance events to the clients (most of whom are audience members). Outside of the motion capture systems there are two other important tools for controlling the performance. The first is the Stage Manager. This system allows the director of the show to trigger predefined events (sounds, set changes, lighting events, curtain up and down, etc.) while the Puppet manager lets a controller animate a virtual character during the performance. Some characters won’t be controlled through motion capture, so the Puppet Manager can allow a person to move and animated the characters in a more traditional way.

Along with this new script comes a new set and cast of characters. Here is the set for this performance, in Luster.

In the final image you can see the panel accessible only to the Stage Manager. Those are Cues, set up in advance, that lets the director set off planned events. You can see that until the final script is coded into our special XML format the general lighting and curtain controls are what is available.

A few weeks ago a local architect from Rochester, NY contacted Darkwind looking for more information regarding Luster. He’s very interested in bringing people together in interactive virtual worlds. One of his current projects, the Baden Street Gaming Center, is available on Google’s 3D Warehouse. I decided to bring it into Luster to show him what Luster could do for him. Here’s what it looks like from the outside:

After only a few hours of work, I was able to create a simple guided tour that brings the user into the various rooms of the complex. The user can pause and play the tour at any time, as well as switch into the free roam mode. This mode allows the user to move freely throughout the scene to take a better look around.

As you can see, there are several wall-mounted televisions in the basement. With a few lines of code I was able to put a flash movie player on the screen in which the user can interact with (pause/play, seeking, volume control). I personally love this use of flash.

You may notice a flickering effect occuring on some of the walls. This effect is commonly referred to as z-fighting, and it’s due to the way the model was constructed. The model was not originally created in Sketchup. It was created in a more professional tool such as AutoCAD and imported into Sketchup. Sketchup isn’t exactly the best when it comes to importing other file formats, so problems like this are unfortunately somewhat common.

Recently, I’ve been playing around with different looks in Luster. I’ve been interested in Non-photorealistic rendering, which is a focus not on realism but on expression. There are many techniques that have been developed over the years, most often a technique called Cel Shading. This achieves a cartoonish look. Hatching has also been researched and presented in the past couple of years. To start my prototyping, I decided to first implement a hatched look similar to the ones that have been presented before. Below are my results.

I will be continuing to work on different styles. Our ultimate goal is soften this look into a colored pencil, and if possible eventually a painted style. Painterly rendering is one of the most difficult styles to mimic in real time. Also, within the medium of painting there are obviously a huge number of artistic styles to attempt to mimic, including movements such as Impressionism to even the style of individual painters such as Van Gogh. Obviously, believably rendering an animated scene in real time with these styles is a difficult task. I’ll post updates as we make progress.

After a long build-up the Darkwind Media team is now open for business. We have extensive interactive 3D and multimedia experience and can build a wide range of different interactive applications. We’ve built systems in the fields of education, architecture, medicine, and games. Our 3D system, Luster, enables us to build 3D applications at a blinding pace, with amazing visual and interactive quality. We also have extensive ActionScript 3 knowledge and have built numerous Flash systems.

You can check out our company website at www.darkwindmedia.com or browse through our Luster website at www.luster3d.com. We can help you build state-of-the-art interactive applications or get you started with your own 3D system by licensing Luster to embed into your own applications.