My Work Explained:

Deep Learning for Visual Effects

November 2018, by Sam Hodge | Semi-technical | Visual Effects | Object Segmentation | Deep Learning

This blog post is part of the AI Collaborative Network "My Work Explained" series and is featured in Issue 1 of the #AICollaborative Network Newsletter. View a full copy of the Newsletter. Forward the Newsletter to a friend. Subscribe to receive the Network Newsletter in your inbox.

Kognat is a small Adelaide based company working on Deep Learning based tools for the Visual Effects industry.

Our first product automatically separates (or segments) different types of objects (people, cars, landscape etc.) in video footage.

Figure 1: Example of Kognat in action

In November 2017 I saw a video of object segmentation using a deep learning model. I was in awe, it was able to create masks automatically which would have been hours of labour per second of footage to do by hand. I knew that anybody with enough experience could access the source code and deploy the model. I also knew that visual effects pipelines are complicated and bound to a software stack that places importance on security and stability.

We took this set of open source tools, and turned them into an easy-to-install plugin for standard software packages used in the visual effects industry. This plugin can be installed within the company’s secure firewall, as it is vital to avoid any possibility of unfinished movie footage leaking.

As far as the end user is concerned it is just another node in the compositing package. They ask for a mask for “person” and it detects the pixels with “people” in them, or a specific “person” among the crowd.

Figure 2: Using the Kognot plugin

The system currently works for 80 different types of objects such as person, car, dog, etc.

The alpha testers have been very enthusiastic when they saw much time could be saved.

Our challenges are in resolution (movies are filmed at very high resolution) and flickering between frames. To address this, we are training new models for the specific use case of high resolution footage.

Additionally our customers want more types of objects detected, so we are creating our own tagged data to train on to address this. This brings the challenge of deciding what new types of object to tag, because tagging thousands of images of “house plant”, when our customers are interested in “building” would be a costly mistake. These challenges highlight the importance of customer relations and software engineering, not just the machine learning technology itself.

Sam Hodge

Sam has been a member of the Artificial Intelligence Collaborative Network since August 2018.

Sam is the Lead Software Developer in Pipeline at Rising Sun Pictures and Director at Kognat Software Pty Ltd.

He has been working in Visual Effects since 2003, with credits in films such as Harry Potter, X-Men and the Hunger Games - check out his show reel or IMDB profile. Get in touch with Sam through the Kognat contact page.