Rising Sun Pictures (RSP) offcially launched their Machine Learning (ML) solution Revize at SIGGRAPH Asia in Sydney. Leveraging ML technologies and advanced VFX workflows, Revize is being used for feature films, series, and entertainment projects at RSP. Revize is not one ML tool but a suite of ML tools developed and most significantly integrated with Nuke. The toolset has been used for various digital ML augmentations, notably face replacement, facial performance modification, de-aging, body replacements, and other actor adaptations. The Revize toolkit includes REVIZE™ Face and REVIZE™ Body, with future developments already in development.
The RSP team has over 300 employees, and the artists started using the first version of Revize in early 2018. The toolset has been used on key films such as Thor: Love and Thunder, Shang-Chi and the Legend of the Ten Rings, and Elvis, but one of the most interesting aspects is its integration into a Nuke pipeline.
Foundry has been deeply involved with bringing ML tools to compositing for several years. We first reported on this in May of 2019, when Foundry released their first ML-Server. Then, with Nuke 13, they introduced their machine learning toolset inside Nuke. With Nuke 13.1, this was extended so the Inference node could load third-party PyTorch models. Today, Nuke has both Copy-Cat for training and the Cattery, their library of open-source machine learning models converted to .cat files to run natively inside Nuke. A .cat file is the format that the Nuke Inference node can understand.
Based on this open approach to ML, RSP extended its ML framework and approach to allow for RSP-trained models to be brought into Nuke and for the comp artist to control the inference. Their Pytorch model is exported to a TorchScript .pt file that stores the model architecture along with the the trained weights and biases in a standalone format. The CatFileCreator node (NukeX and Nuke Studio only) creates custom .cat files from PyTorch TorchScript .pt files within Nuke. Users can import TorchScript files from GitHub, Model Zoo, or in the case of RSP, their own internal models and slot them into RSP comp pipeline. CatFileCreator nodes allows users to add their own ‘knobs’ to the .cat files that are automatically passed to NukeX’s Inference node so that the artist can have complete control over your pre-trained ML models. The Nuke artist can use this directly inside Nuke rather than just bringing in inferred outputs. This is exactly what any Nuke artist would want:- to have control and dramatically speed up the time to roundtrip any changes to the ML process.
This also scales the ML pipeline inside RSP; now, the ML team avoids ever becoming a bottleneck; rather, the new Nuke approach facilitates their expansion to allow the comp team to work individually and in parallel, not sequentially. “The manipulation and models are RSP proprietary,” explains Beveridge, “but you can load a model into Nuke and have the models showing up interactively.”
This does not remove the importance of data curation or the need for specific data acquisition. RSP likes to have special capture sessions with the actors wherever possible and as much training data as possible. However, the approach is much cheaper than a traditional CG pipeline. Compared to creating a digi-double or a hand-crafted CGI solution, the ML shots can be fraction of the price of traditional approaches.
The Revize ML pipeline respects the normal RSP color pipeline, and all of the ML tools match the normal VFX pipeline with Nuke 14.1. They have worked to make their tools as artist-friendly as possible, and not all their Revize tools are in Nuke. The ML team did use DeepFaceLab (DFL) tool early on, especially in some of their first feature films, but they have primarily moved on, and it is now just a legacy tool. DFL is also all implemented in TensorFlow, and Nuke is PyTorch, so regardless, DFL cannot be integrated into Nuke. “You can see people are using DFL right now even at the consumer level, but there is a big difference between doing that and working at the highest feature film level,” explains Beveridge. Also, some open-source tools are not designed to work at high resolution, for a half-height face on screen at 4K in HDR. “It is not as simple as using a SuperRes (ML tool) – and some of your areas of the face with harsh contours get super sharp so it doesn’t really hold up. So we have had to do a lot of work ourselves in this area.”
Similarly, the process is built to allow Directors to adjust the final output creatively. Unlike some approaches, the Nuke – RSP Revize approach inherently allows for artist intervention and creative tweaking and polishing.