homepage   computer graphics  short film   photo gallery

                                        Stereoscopic Production Pipeline                                   

Creating first person stereoscopic shots using custom MEL scripts and mental ray shaders.

                                           Celambarasan Ramasamy

For creating realistic first person stereoscopic shots I implemented a mental ray stereoscopic shader that works closely with a MEL script. Together they work on top of motion capture data to create first person stereoscopic shots.

    The shader can render out the left/right eye images separately or directly render out an anaglyph version of the stereo image. The shader takes in a set of input parameters and uses them to create the projection parameters required for the left and right eye camera for each frame. When mental ray requests the shader to return the color for a pixel, in anaglyph mode, the shader traces a ray through the requested pixel in the left eye camera and then in the same call traces another ray through the corresponding pixel in the right eye camera. The green and blue channel values from the left eye is combined and placed in the red channel of the output color. Then the whole intensity of the red channel of the output pixel is bumped up using gamma correction. The green and blue channel from the right eye pixel is directly placed in the green and blue channels of the output color. This kind of encoding is called as Optimized Anaglyph encoding. This is just one of the methods to encode the image, there are other ways to distribute the left and right pixel intensities among three channels [Anaglyphs Method Comparison, 2005]. The left and right eye intensities of the anaglyphic image needs to be gamma corrected separately, as our perception of the red intensity is lesser than our perception of cyan (green and blue channels combined). Mental Ray renders the output image by requesting output color from the shader one pixel at a time. There is a link to my thesis at the bottom of the page. It contains the source code for both my stereoscopic shader and the MEL script.

3.1.1.1 Features

Other than creating stereoscopic images, this shader has a number of additional features that provides more creative control over the production.

 Focal plane depth of field:

 In addition to supporting conventional depth of field that controls the blurriness with depth, I also implemented a way to control the blurriness along the field of view of the camera. The blurriness across the field of view is controlled using a two circle setup where you give the radius of the two circles and the amount of blurriness at each circle. In Figure 6, the circle of confusion values v1 and v2 between the inner and the outer circles are interpolated to get the blurriness values at the various points on the image.    

                    Focal Plane Depth of Field

Spherical lens:

 The shader can also simulate spherical lens that’s used to create fish eye lens effect.

 Encoding Stereo Pixel Disparity:

 The shader also has the ability to encode either the horizontal or the vertical pixel disparity of the stereo image in the alpha channel. In other words, along with a color at each pixel, a disparity value is also stored. To enable mental ray to encode this value, one must turn on the 'Pass custom Alpha Channel' flag in 'Custom Entites' under the Mental Ray rendering tab.

The pixel disparity is encoded in a normalized scale between 0.0 and 1.0 to make the values resolution independent. The disparity values are encoded using the following logic. In the normalized scale, the extreme values of 0.0 and 1.0 are used to indicate an invalid pixel. The maximum allowable value that can be encoded is a pixel disparity of a quarter of the image resolution. An alpha value of 0.5 indicates no pixel disparity. The range 0.0 to 0.5 indicates that the left eye pixel is to the right of the corresponding right eye pixel in the stereo image. The range 0.5 to 1.0 indicates that the left eye pixel is to the left of the corresponding right eye pixel in the stereo image.

Figure 7 shows a stereoscopic frame and its corresponding horizontal disparity map. In the horizontal disparity map, because of the alpha encoding, objects at screen depth appear mid grey, objects that pop out of the screen appear darker while the objects that lie further back appear brighter. White indicates invalid disparity for the corresponding image pixels. The teddy bear and the stand appear darker than the rest of the image as they are at screen depth and thus the objects with the least stereoscopic depth in the image.

                                                                    A Stereo Frame and its corresponding horizontal disparity map

Since the shader renders stereoscopic images using a cross-eyed lens configuration. This setup introduces some vertical disparity in the stereo image due to perspective distortion between the two cameras, especially along the edges of the frame. Similar to the horizontal pixel disparity, the vertical pixel disparity in the stereo image can also be encoded in the alpha channel. An alpha value of 0.5 indicates no vertical pixel disparity. The range 0.0 to 0.5 indicates that the left eye pixel is to the top of the corresponding right eye pixel in the stereo image. The range 0.5 to 1.0 indicates that the left eye pixel is to the bottom of the corresponding right eye pixel in the stereo image. Figure 8 shows a stereo frame (top) and its corresponding vertical disparity map (middle). As seen in the figure, the vertical disparity varies across the different regions of the frame. The bottommost diagram shows the vertical disparity distribution in a frame that renders a point at infinity. The vertical disparity in the different regions of the frame is labeled according to the color code of the encoding that I implemented in the shader. The vertical disparity is maximum near the four corners of the frame and it is minimum near the center of the frame and along the vertical and horizontal axis cutting the frame.   

                                                   A Stereo frame and its corresponding vertical disparity map along with a diagram showing the distribution of vertical disparity in a frame.

Most of the shader parameters were not meant to be controlled manually. The original design plan for the shader was that only a few of the parameters, like the ones that control the quality of depth of field, were meant to be user controllable. The rest of the parameters were designed to be controlled by a Mel script.

3.1.2 Mel script

ctSimulateEye is the Mel script that updates the parameters of the stereoscopic shader every frame. The design of this Mel script is closely coupled with the motion capture data obtained from the actors’ head. The script can operate in two modes namely the 'Eye' mode and the 'Locked to Head' mode.

'Locked To Head' mode: In this mode the stereo cameras point in the same direction as the head. This mode has a handheld camera look that is very common in the first person shots in movies. 

                                Raw Motion Capture Data

                         Stereo Rig in 'Locked to Head' mode

'Eye' Mode: In 'Eye' mode the stereo cameras behave similar to the way in which human eyes behave on the head. In this mode the cameras fixate on a particular object in the scene even as the head of the actor moves around. This setup helps to recreate the visual field that the character is seeing as opposed to the visual world that he or she is experiencing [Lipton, 1982]. This setup also makes it possible to create nice eye shift transition effects.

Link To My Thesis    (Contains source code)