Part 1: Setting up the Masks

V-Ray 5 introduced a lot of interesting tools since the first release of the new VFB (Frame Buffer). Here we’d like to present the first of two articles in which we’ll go through the post-production workflow to use with V-Ray 5 VFB.

This first part is focused on the Masking tools and how to set up the masks that we’ll use later during the post-processing in the VFB (presented in the next days in the second article).

NOTE: the masking tools have been introduced in V-Ray 5 update 1 (Feb 23rd, 2021). Make sure to have the correct requirements if you want to try them by yourself.

Below there’s a render of the scene straight from the VFB, with no post-production. We are going to improved it by using only the tools in the V-Ray 5 VFB, and take advantage of the new masking tools:

NOTE: I’m working with both the V-Ray GPU and CPU engines: the steps are the same in both cases but there may be differences in the positions of some parameters.

I’m also using ACES throughout the workflow. If you don’t know how to use ACES in V-Ray 5, or you’re not sure how to work with it, check out the presentation “ACES Workflow with V-Ray 5” from the AcademyDays 2020. If you haven’t done it yet, you’ll need to register for the event. The registration is completely free.

Setting up the Masks

Now let’s jump into 3ds Max and see how to set up everything!

First, let’s open the VFB. Then to add a mask go to Create Layer and down below you’ll see 3 options:

  • Integer mask
  • MultiMatte mask
  • Cryptomatte mask
The mask elements become active only if there is a Color Correction layer or Folder selected.

All the mask types are generated by Render Elements.

In particular, integer mask depends on:

  • VRayMtlID
  • VRayObjectID
  • VRayRenderID

MultiMatte mask depends on:

  • MultiMatteElement

Cryptomatte mask depends on:

  • VRayCryptomatte

Here are a few notes before moving on:

  • Integer mask: the render elements VRayMtlID and VRayObjectID must be set with Output type to Integer in order to work (usually it’s set by default);
  • MultiMatteElement: you need to set the Red, Green and Blue IDs, either manually from the Render Element or by using the script IDTool.
  • VRayCryptomatte: it works only with the Image Sampler set to Bucket.

So, for this example, I’m gonna set the Image Sampler to Bucket and then I’m going to add the various Render Elements.

Now that we have all the Render Elements let’s take an overview to see what type of mask they generate.

→ VRayRenderID doesn’t need any setup.

→ VRayMtlID creates mattes based on the MaterialIDs property of the meshes. We can easily replace it with either MultiMatteElement or VRayCryptomatte.

→ VRayObjectID needs an ID > 0 applied through the Object Properties of the meshes. To do it, you can either use Right Click > Object Properties > Object ID or you can use the IDTool script (version for Corona Renderer here).

→ VRayCryptomatte will generate masks based on the ID type we set in the Render Element.

The options are:

  • Node name – mattes are created based on objects names
  • Node name with hierarchy – same as above but it takes into account the hierarchy of objects (like Groups)
  • Node material name – mattes are created based on the materials applied to the objects
  • Node user property – mattes are created based on the property in Node User Property
  • Layer name – mattes are created based on the names of the layers in the scene

You’ll notice that the VRayCryptomatte_Objects> and VRayCryptomatte_ObjectsHierarchy are the same, since I don’t have groups in the scene.

→ MultiMatteElement works similar to the VRayObjectID. It uses the ObjectID property of the meshes to generate masks stored in a Red, Green or Blue channel. In the same way as the VRayObjectID element, you can either use the IDTool script, do it from the Object Properties or from the parameters of the Render Element. You can also add multiple objects to the same ID, for example all the meshes named “glass”. Remember that since VRayObjectID and MultiMatteElement use both the Object ID property of the meshes, once you set the IDs, they’ll basically generate the same masks. The main difference is that the MultiMatteElement creates RGB masks that are easier to use in Photoshop.

We can also generate multiple RenderElement for VRayCryptomatte and MultiMatteElement and then change the properties to get different variations in the masks.

The masking tools can also be used to mask additional Render Elements, like VRayReflection, <VRayRefraction, VRaySpecular and so on. The masks are created and work in the same way, but with only one difference: the mask layer has to be applied to a Folder layer; it cannot be applied directly to a Render Element.

If you’re planning on using the Render Elements for the post-production in the VFB a good starting point is to add the VRayBackToBeauty. It will automatically create most of the Render Elements you need.

It also solves another problem: to work with the Render Elements in the VFB you need to be on the layer Source Composite, and with the VRayBackToBeauty we can quickly recreate the Beauty Pass.

To apply a mask, first we need to duplicate the Render Element and put it into a Folder, then assign the mask to the Folder in the same way we did for the other layers.

In the next article we’ll move entirely to V-Ray 5 VFB and see how to use different Color Correction layers and Render Elements together with the masking tools to enhance the render.

Thanks and stay tuned!

Share this