Notes On Implementing a Maya Viewport 2.0 Renderer Override
In October of 2014, I was working on a project in my free time that overrides Maya's Viewport 2.0 renderer. A viewport renderer override can be useful to render the scene externally, such as in a game engine. The advantages of this are that you can fully control the rendering so that the result in the viewport matches your game engine in real-time. From a productivity standpoint, this is an amazing plus. Other advantages include, but are not limited to, removing the limitation for the number of light sources natively available within Maya's built-in viewport renderers. Implementing such a project is no easy task, and it comes with many strange quirks and issues. I will discuss, on a high level, implementation of such a renderer, as well as share some notes on things I've learned throughout the project. Some things may be generic to the API whereas others may be specific to renderer overrides. Please note that this is not a tutorial, but is instead intended to provide a direction for you to follow.
What is it?
There are many kinds of Viewport 2.0 rendering-related overrides available, but we will be focusing on MRenderOverride. Simply put, an MRenderOverride will override all rendering per refresh in the viewport, or per-frame in batch rendering. Each MRenderOverride should contain one or more MRenderOperations. These operations are the heart of the override where the actual rendering is done. Again, in line with Maya's style, there are many variations of these operations - they can clear buffers, let Maya render all or part of the scene for you, render quads, present to the screen, and even run user-defined rendering code. It is possible to mix-and-match these operations as you please - you can have Maya render the HUD, and also run your own rendering code in the same frame. Through these methods, we are able to define our own rendering code, whether it be in the plugin or through an external renderer, and present that to Maya.
Why do it?
There are many possible reasons you would want to implement a complete rendering override for viewports within Maya. In the context of a game engine, it would provide artists with instant visual feedback about how their assets would look in the game, without having to resort to a potentially long Maya-to-game exporting process every time they make tweaks. Maya natively has a light limitation of eight, sometimes sixteen, active light sources in the viewport, depending on the GPU. This limitation can be removed by overriding the renderer, especially if a deferred path is used, as is more common in game engines now.
To create a custom Viewport 2.0 renderer override, you must create a class that derives from MRenderOverride, and register it using MRenderer's reigsterOverride function. An override of this kind expects the user to do everything manually, from clearing to presenting, but don't worry, as Maya provides quite a lot of functionality to help. When Maya calls this class to render a frame, it first calls the setup function. Here is where you can do initialization and updating. Note that this is called every frame, so ensure you check resources exist so that you do not recreate them. Next, the startOperationIterator function is called. Here, you can reset the method you use to track the currently active render operation. After this, renderOperation and nextRenderOperation are repeatedly called until the latter returns false. The order of operations depends on what pointer is returned from the former function. Finally, cleanup is called, and the frame is complete. Akin to startup, this is called every frame, so don't deallocate resources you want to use again here. It is a good idea to let Maya know which draw APIs you wish to support by overriding the supportedDrawAPIs function. You can also set a name to be displayed in the Maya UI using the uiName override.
To provide the renderer override with some operations, you must return pointers from the renderOperation function to instances of any class derived from MRenderOperation or any of its existing derived classes. For example, in my implementation, I created an array of four render operations. In order of execution, they were: KUserOp, which derives from MUserRenderOperation, allowing me to run custom rendering code; KBlitSharedTexturesOp, which derives from MQuadRender, which is used to blit render targets to Maya's internal buffers; MayaUIDraw, which derives from MSceneRender, drawing only the UI elements through filtering overrides; and finally, a standard MPresentTarget to present the buffers to the screen. From this, we can show that the flow of each frame is setup->KUserOp->KBlitSharedTexturesOp->MayaUIDraw->MPresentTarget->cleanup. Understanding the flow can be essential to debugging graphical oddities.
If you are going to be hooking in an external renderer, I strongly recommend that any render targets that will be used by Maya - that is, read from or drawn to within the plugin itself - should be generated in the plugin and have their native handles sent to the external renderer to ensure compatibility with Maya. In my implementation, I generate a color buffer in Maya that is then sent to my external renderer, where the result of a deferred rendering path is output to this render target. This shared render target is then blitted to Maya's internal buffers during the KBlitSharedTexturesOp operation.
The KUserOp operation, derived from MUserRenderOperation, has a function called execute, which will be called whenever Maya wants to run this operation. Within this function, you can run any custom rendering code. In my implementation, I call my external renderer's draw function, and when the operation is complete, the shared render target I previously mentioned will contain the result of my external renderer's draw. If you intend to parse and build scene data every frame for rendering, it is possible to do it here. In my implementation, however, I mirror Maya's DAG, which I will discuss later. If you modify GPU states by hand here, or anywhere else in the application, you must restore them to their pre-modified states. Maya's behavior is undefined if GPU states are unexpectedly changed.
The KBlitSharedTexturesOp operation, derived from MQuadRender, blits the shared render target previously mentioned to Maya's internal buffers. Prior to this operation, the shared render target contains the contents of my external renderer's output, and therefore, after this operation, Maya's internal buffers will contain the same contents. In my implementation, I override the shader using the shader function to a custom written shader. Note that Maya provides a lot of built-in shaders that are usable here, so it is a good idea to check them out before writing your own, as it could save you time. Additionally, I override the clear flags and depth-stencil state to ensure the quad clears and always renders by using the clearOperation and depthStencilStateOverride function overrides respectively.
The MayaUIDraw operation, derived from MSceneRender, lets Maya draw the scene, but filters out undesired objects by using the renderFilterOverride and objectTypeExclusions functions. In my implementation, I also override the clear flags using the clearOperation function. Be careful when overriding these clear flags, as you could mistakenly clear the buffers you have been writing to.
The MPresentTarget class simply presents the internal buffers to the screen. It is possible, like the other classes, to derive from this and customize the operation as you see fit, but in my implementation, I didn't require any modifications.
As briefly mentioned previously, I do not scrub the scene for data every frame, but instead, mirror Maya's DAG in a system I called KDag. The KDag is where Maya's scene is traversed, appended, and listened in on. It also acts as a translation layer between Maya and my external renderer. Both Maya and the external renderer should never know about each other internally. The high-level concept of KDag is actually very simple. It clones Maya DAG nodes that it cares about, and listens for changes to those nodes, and if necessary, updates the external renderer's information accordingly, acting as a translation layer between Maya and the external renderer. More specifically, when traversing the scene, different nodes are created based on an MObject's API type. The nodes are added appropriately to the mirrored version of the DAG, and are also added to two hashmaps - one that links to an MObject and one that links to the node in the KDag. Separating the cloned DAG nodes by the MObject's API type allows for a more manageable and object-oriented approach, rather than having lots of switch statements. Traversing the scene is done by using an instance of MItDependencyNodes. The advantage of using this class is that you can just reset the iterator to a new type and access every instance of it in the Maya scene. Listening for nodes being added or removed can be done using the addNodeAddedCallback and addNodeRemovedCallback callback functions respectively, found in MDGMessage. Listening for child added and removed events can be done using the addChildAddedCallback and addChildRemovedCallback callback functions respectively, found in MDagMessage. Listening for a specific MObject's attribute changes can be done using the addAttributeChangedCallback callback function found in MNodeMessage. In my implementation, I update the external renderer's resources when the attribute is changed, resulting in instant changes. The structure of your classes will change based on the implementation of the external renderer. In my implementation, I mirror MFnDirectionalLight, MFnPointLight, and MFnSpotLight nodes as KDagLight. shadingEngine nodes are mirrored along with their connected surfaceShader as KDagMaterial. MFnMesh nodes are cloned as KDagMesh. kPsdFileTexture and variations are mirrored as KDagTexture. MFnTransform nodes serve more than one purpose, depending on the child nodes below. If the children are only further transforms, it is classed as a transform, and in my case, is not sent to the external renderer, but may, depending on your configuration. Otherwise, if a node such as a light or mesh is detected as a child, the node becomes an Object, which is what my external renderer draws. Each of these nodes knows about the external renderer's equivalent object and updates it accordingly, bridging the gap between Maya and the external renderer, without both having to know about each other. Note that this will be different depending on the configuration of the external renderer in question and how it manages resources and objects.
Tips 'n' Tricks
If a buffer is used within Maya, create it in Maya
If a buffer is going to be used within the plugin by Maya - that is, read or modified by Maya - then create it within Maya and pass the native resource handle to your external application. Maya lets you specify every type you could ever require when creating textures or render targets. Creating them within Maya ensures compatibility. Feel free to generate and use any other intermediate textures or render targets as you see fit outside of Maya.
Know the order of your MRenderOperations inside and out
The order of your operations is defined in the MRenderOverride's renderOperation function. The order pointers are returned from this function defines the order in which the operations are run. This may seem obvious, but this order can be crucial to debugging why something doesn't appear on the screen. For example, if a HUD operation clears screen and is called after a user operation, you would not see any results from the user operation.
MRenderOverride's setup and cleanup functions are called every frame
Ensure you check for initialized values within setup function, as it is called every frame. Think of it as setting up the current frame, rather than a one-time initialization. Similarly for cleanup, don't delete resources you want to use the next frame.
Restore modified GPU states
Maya is extremely sensitive to GPU state modifications. If you change any state in your rendering code, make sure to set it back to the value before your rendering code was called. Maya's behavior is undefined with externally-modified GPU states.
printf sometimes doesn't print
I've tried debugging my code on multiple computers, and printf only seems to work in certain environments, whereas MGlobal's print functions worked fine. If you see no output when expected, try swapping from printf.
Sometimes MFn objects must be created from the MDagPath
There are some functions which require the MFn object to be created directly from a path rather than an object. I am unsure of why this is, but the documentation will let you know if this is the case. I've only come across it a handful of times, but it is worth keeping in mind if you get unexpected results from a function.
Add dump-to-file capabilities to your external renderer
Not sure why your rendered image isn't showing up in the viewport or if there's anything being drawn at all? Add functionality to save your render targets to file and you will save time.
Use Maya's plethora of built-in shaders
Maya has a somewhat large library of shaders accessible through the MShaderManager's getEffectsFileShader function. Have a browse through them, as it may save you writing some common shaders.
Visual Studio debugger is your best friend
Is Maya crashing? Attach Visual Studio's debugger and see why. Need to check you're correctly reading data from Maya? Visual Studio debugger, again. As with all branches of programming, having a debugger at hand saves so much time (and pain, and suffering). If you're developing on another platform, you'll have to delve into their specific debugging tools, sorry!