Thinkbox Software recently announced Deadline 6, the newest version of their render software, which is powered by MongoDB. Deadline is a render management tool, used heavily in the visual effects, broadcast and architectural industries. Thinkbox’s Deadline team chose MongoDB as their database of choice for its JSON-based document structure and speed at handling connections.
A bit of background on render management for those unfamiliar: when making a film, production teams create and render thousands to millions of individual frames for the layers of their shots. Computers process your image data for compositing blue screens, characters or rendering fire, smoke and other special effects. “There are a lot of products that make this happen,” says Chris Bond, founder and CEO of Thinkbox Software. “They all render, control and work differently depending on your OS platform.”
In the past, filmmakers or graphic specialists would manually split these jobs up between a number of machines to distribute the amount of data for processing jobs. There are thousands of frames in a shot, and thousands of shots in a movie. Some rendering processes could take 24 hours, while others could take 20 minutes. There wasn't the virtualization platform that could distribute these tasks and manage these processes, so they were typically done manually or executed by scripts. Deadline solves this problem as a management queue for rendering farms.
Deadline 6's Monitoring Dashboard
Deadline currently supports over 40 applications and operating systems across the board, and breaks up shots and frames into tasks across different machines. Many of this year’s summer movies such as Man of Steel, Iron Man 3, Oblivion, and 300: Rise of an Empire, are made with Deadline 6 and used MongoDB as the backend. The team switched to MongoDB in order to improve create a tool that could run fast while operating on tens of thousands of tasks at a time.
Up until Deadline 5, the team was using an XML, file-based storage system which was easy for rapid development, like JSON, and it was schema-less, so they didn't have to worry about tables. “Deadline was designed as a ‘serverless’ architecture.” said Ryan Russell, lead developer on Deadline 6. “Rather than having a central application telling its clients what to do, the clients would go to a central data store, based on the data in there, it would take the next step.” The client would go to the storage location where these "jobs" are housed and would pick up and frame and then pick up another frame. “It was a solid backend for our software, because Deadline could never ’go down’ unless our server hosting that data went down.”
But a file-based system is hard to scale and the team needed a way to help Deadline scale and hit higher performance numbers. When searching for a new solution, one of the developers stumbled across CouchDB. “It matched our document-based system, and when we investigated further, the team kept seeing MongoDB come up in searches and blog posts.” For them, the 1 to 1 XML to JSON mapping was a winning feature for both document stores, but to get a better understanding, they decided to build a prototype with each data store to see which one performed better. MongoDB won out in the end. “MongoDB handled hundreds of connections at the same time without any issue. CouchDB had a hard time because of the RESTful API overhead. The improved performance with MongoDB created such a great experience for the end-user. That was a huge thing for us.” Select clients recently went through a 6 month a beta program with Deadline 6, and after using it in production, were amazed at how much faster their system was with MongoDB as the backend.
In Deadline, MongoDB functions as a task queue, and controls the management of third party render processes. Each day, a large film set might render over 25-30 Terabytes of data. MongoDB handles the process management of each job and controls the data movement. To the team MongoDB “seems like something we can stand on and scale in many orders of magnitude more than we have before.”
Look out for a film powered by Deadline and MongoDB this summer.