Compositing RichDirt with Photoshop + mental ray

This tutorial shows how to use RenderElements created by RichDirt in Photoshop. It uses the basics of compositing with Photoshop and mental ray.

The tutorial shows how to solve 4 major hurdles in the workflow, 2 in 3ds Max (Background, Exposure) and 2 in Photoshop (Invert, Masks from Alpha). Note that an identical tutorial exists for V-Ray, with a very similar approach, the hurdles and solutions are different.

What is the basic idea of compositing dirt ?

You split a rendering in several elements and then re-add these elements in Photoshop. The advantage is that you can easily control color and intensity of layers. Adding images in Photoshop is real time, you don´t need to re-render for 1 hour to get a change in your images. This can be up to 1000 times faster than simple re-rendering ! And it offers much more artistic freedom.

The tutorial is organized in 2 sections:

  • An Introduction to Compositing / Render Elements. No Dirt is used here. This is very useful if you haven´t done any kind of compositing before. (Experienced users simply read this section faster) The first section runs from number 1 to 5
  • Adding dirt - how to do this most efficiently ! This section runs from number 6 to 14



    • Introduction to Compositing with mental ray, 3ds Max Render Elements and Photoshop

    An image of our example scene - without dirt : It uses 3 concrete cubes on a floor. The size of the cubes is around 200 cm (80 inch). There is 1 mirror sphere to cover reflections. The background uses a procedural sky. Note that this image is shown with a gamma of 1.0.

    1) Where do I find the Render Elements (RE) ?

    They are found in the rendering dialog, in the ´render elements´ tab. Non-English versions of 3ds Max might have different translations, but the position of the tab is the same.

    We need to seperate lighting (illumination) from textures. Why ? We want to change the diffuse texture later !

    2) Add the following Render Elements

    Diffuse

    Lighting - this includes direct and indirect lighting

    Background (for the sky)

    Reflections

    3) Possible mistakes part I

    When using a mental ray physical sun and sky you will get color values above 1.0. For standard rendering this is handled by exposure control. Render Elements do not use exposure control, they -should- write the raw unclamped color. However 3ds Max with mental ray writes all Render Elements -clipped-, so all values above 1.0 are lost, even when you write to file formats like hdr or exr. To solve this we need to switch off the exposure control, and adjust brightness of sun and sky. Typical multipliers for the sun and sky are 0.02. Note that mental ray -can- write raw unclamped colors when the framebuffer is set to 32 bit float. (Renderer->Sampling Quality-> Frame Buffer Type: floating point (32 bit per channel). However the main image (beauty pass) would still run through exposure control - this would create a difference when compositing Render Elements. Also Photoshop doesn´t support certain operations on 32 bit buffers. So for learning it is better to switch off the exposure control.

    4) Possible mistakes part II

    You need to export images using gamma 1.0. Use RGB (do -not- use RGBA) Use 16 bit tif files everywhere. Why 16 bit and not 8 bit or 32 bit ? 8 bit does not have sufficient precision. Photoshop can not -invert- 32 buffers, and we will need the invert operation.

    The 4 Render Elements :


    5) Start Photoshop

    • Create a new output image, name this 'All Together' This will be used to add all layers. We will call it ´main image´ as well.
    • Set the Color 'mode' to 16 bit.
    • Add the 'Lighting' image ('normal' mode). The Lighting Render Element includes both global illumination and direct illumination.
    • Add the 'Diffuse' image ('multiply' mode) Now we have the image reconstructed. Background (sky) and glossy reflections are missing.
    • Add 'Background' image ( 'add' mode ) This isn´t as easy as it sounds. Unlike the VRayBackground RE the ´standard´ RE adds values also to areas that are covered by geometry. Simply adding this in Photoshop is then not possible.

      We need a mask that is white for the background and black whereever we have geometry. The alpha channel from any of the Render Elements would do this. Modify the 'Diffuse' RE, add alpha to it. Press the Render Element´s "..." button. The 'Setup' dialog is grayed out by default. To enable it pretend to save as .bmp. Then switch back to .tif. This enables the 'Setup'. In the setup dialog choose the 'Store Alpha Channel'. Make sure to specify gamma 1.0. Render again to get the image with RGBA.

      Next we need to get the alpha as a mask. Reopen the ´Diffuse´ RE in Photoshop. In the Layer dialog select the ´Channels´ tab.

      Switch off the RGB, Red, Green, Blue channels. Switch on the ´Layer Mask´ channel. Now you should see a nice mask. If this didn´t work - load Diffuse.tif again, Photoshop sometimes stores intermediate results. Also make sure you only have the alpha layer selected in the channels tab when you press control a and control c

      Control a + control c to copy this to the 'Background' image. In the 'Background' image change the mode to multiply.

      Inside the background image select the new mask layer and invert it ( we need the geometry to be black). Invert can be done with 'control + i', Then change the mode to 'multiply'. Black is 0, so multiplying with black removes the geometry.

      So far the background looks great, but we also have to copy it to the 'main image'. To do so we need to first collapse (= merge layers) the layer stack. Now we have a nice background that can be directly added to the main image ('All together').

    • Add reflections ( 'add' mode ). The reflections render element contains the glossy and mirror reflections.

        • Result - we get the exact same image as inside 3ds Max. So the compositing worked ! If you get a different image, check again that you exported the images from 3ds Max using a gamma of 1.0 and used 16 bit in Photoshop.

        Now let us start with the second section of this tutorial - how to add dirt. This is the fun part - no ´Background mask business to do ´!

        6) Go back to 3ds Max, add 3 "mr Shader Element" Render Elements

        a) Create the first mr Shader Element RenderElement

        b) Add a RichDirt texture to it.

        Drag the texture into the material editor to edit it. Use the 'Rich Building preset'.

        If this is too fast (not enough details in the tutorial - see the video tutorial how to create a single dirt Render Element. The video has more details.

        c) Create a second and third mr Shader Element RenderElement
        d) Add RichDirt with preset 'Splashes (variations)' and 'Building edge streaks'. Render - you now should get the following 3 images:




        8) Switch back to Photoshop

        • Add the ´Rich Building´ RE to the main image (´normal´ mode)
        • Add the ´Splashes´ RE to it (´multiply´ mode)
        • Add the ´Rich Building + Streaks´ RE (´multiply´ mode)

        This creates combined dirt. RichDirt uses black and white colors by default so multiplying layers is easy.

        Let us have a quick look at the delicious dirt:

        9) How to get rid of the ground plane / how to add a mask in Photoshop ?

        Dirt is added to -all- the geometry, including the ground plane. (Note that one can use RichDirt also as a texture, then each material can have different dirt. Per-material-textures can also write to a RenderElement, see the Reuse/Store tab of RichDirt´) When we later mix this with the diffuse texture we do -not- want dirt on the ground plane. So we need to mask it.

        In your renderings you might not have a ground plane. So why is this covered so much in this tutorial ? The ground plane is an example for all geometry that you want to keep free from dirt. When you have a large model 1000s of objects might need dirt and other 1000s of objects may not want dirt. The approach described here solves this for you.

        While handling the Background buffer is much easier in V-Ray creating a mask for the selected objects is easier with mental ray. mental ray supports the ´render only selected objects´. We use this to create a mask for the selected objects. - Add a second 'Diffuse' Render Element. - Name this ´myMask´, and set it to 16bit tif with rgba. We will need the alpha mask. - Switch off all other render elements so they are not touched. - Select the 3 cubes. - Hit Render with the 'green cube + teapot' toggle on.



        You should now get an image that looks like :

        To convert this into a mask do steps that are similar to the alpha mask for the background image. In Photoshop in the 'Layers' dialog select the 'Channels' tab. Switch off rgb, red,green,blue, and switch on 'alpha'.

        Now you have an image that can be used as mask.

        10) Copy the mask (control-a + control-c)

        a) Paste it into ´All together´ on -top- of the 3 dirt layers.
        b) Invert it using ´control I´.
        c) Set the mode to ´Lighter Color´.

        Why ´Lighter color´ ? The inverted mask is white for all objects that should not get dirt.

        11) Now select the 3 dirt layers + the mask, press ´Control-G´ to create a group.

        Name this group ´Dirt Effects´. ´myMask´ must be inside the group, so it is only applied to the dirt.
        The huge advantage of using a group is that we can control weights (opacity) of streaks, splashes and more effects directly. This is essential to get the big speed-up compared to re-rendering.

        12) Congratulations ! You have a full setup now.

        The fun part is to adjust the weights of streaks and splashes to get the exact amount of dirt you like.
        To make sure you see the full setup here is a snapshot of the layers in Photoshop. Check this when you think you missed a step somewhere in the tutorial.
        Also check if your layers are still in 16 bit. When converting a single layer to 8 bit Photoshop sometimes converts all layers to 16 bit.

        13) Aliasing

        Compositing in Photoshop is very fast, so why doesn´t everybody use it -all- the time ? At edges of objects (masks) one can easily get aliasing. To minimize this be sure

        • You have indeed antialiasing on in the renderer. High quality is required here. For the background (sky) there can be a gap at pixels that are half background and half object. Using ´lighter color´ instead of ´add´ can improve this.
        • Try a gauss filter with size 0.5 px to the selected objects mask. This can reduce most of the aliasing directly.
        • A brute force approach to solve aliasing, if you have enough processor power, is to render in larger resolution. Do the high resolution compositing in Photoshop and then reduce the image size.

        14) Creating colored dirt

        The compositing set up above uses multiplication, which makes things -darker-. This is often perfect. But sometimes you want dirt that has a different color, like greenish moot, or brown rust. To achieve this do the following steps in Photoshop

        • Duplicate the Dirt Effects group. (To duplicate select the group layer, and press right mouse, then choose ´duplicate group´). We need to duplicate it to avoid loosing it in the next step. Name this new group ´green dirt´.
        • Now convert this to a ´smart object´
        • Once it is a smart object Photoshop allows to apply ´Rasterize Layer´. We need this (group -> smart object - > rasterize layer) because Photoshop doesn´t allow changing colors on a group. Now change the color to a greenish tone Main menu -> Image -> Adjustments -> Hue/Saturation. Choose a green color here. Switch on the ´Colorize´ toggle in the Hue/saturation dialog, so the green is visible.

        The dirt group with a green tint

        Moot on concrete - gamma is set to 2.2 after compositing.

        Copyrights
        mental images and mental ray, are registered trademarks, and are trademarks of mental images GmbH, in the U.S.A. and/or other countries. V-Ray and the V-Ray logo are registered trademarks of Chaos Software Ltd. in Bulgaria and/or other countries. Autodesk and 3ds Max are registered trademarks or trademarks of Autodesk, Inc., and/or its subsidiaries and/or affiliates in the USA and/or other countries. finalRender is a registered trademark of cebas Computer in Germany.Adobe and Photoshop are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States and/or other countries . All other brand names, product names, or trademarks belong to their respective holders.