Building a render farm can be difficult. You’ve got to get lots of machines and processes coordinated with users, permissions, licenses, file space and applications to make it all work. Jam it all together and you have something that starts to look like a workflow.
The Molecule’s first render farm attempted to use an rsh-based collection of scripts and html pages to accomplish this. It worked but had lots of problems. You could think of it as a Beowulf or Rocks style cluster, which essentially it was, with 60 machines shared over NFS.
In practice, this approach has problems. Ensuring consistency between the machines is difficult (which the Rocks system addresses with kickstart files), but it’s not always practical to completely format machines when a problem occurs, especially in an environment that’s not completely homogeneous (ie, even though they’re all running FC4 and at approximately the same speed, the bioses are different and ethernet device ordering is different, etc). Moreover, if a rendering problem does occur, locating which machine and its exact cause can prove to be a daunting task. It is difficult to track down the problem when frames are bad or missing, especially when each machine is rendering every 17th frame of a sequence! This is a Data Wrangler’s nightmare. Clearly a more intelligent dispatching system is required.
Over the last 6 months our render farm has gone through a full body remodeling. Over 100 processors are logged into the system at once. The new server, written in php, features a virtual file system that links together functions and resources into a read-write space that connects the interface to the back-end components, offering a pretty cool thin client set-up. Some other interesting and different features are the render farm’s graphical interface, written in Flash AS3 and containing 21,000 lines of code, and its node-based design allowing for the creation of dependency trees (a means by which processes can be organized). At present, there are 9 nodes available to the system- After Effects Degrain, After Effects, Frame Sequence to QuickTime, QuickTime to Frame Sequence, E-mail, Maya, Mental Ray, Shake, and Nuke.
It offers a multi-user common workspace, which allows several people to work on the same materials simultaneously from different terminals. With its new user-friendly format, the system shows rendering progress on the frame-level as well as on an overall-job level and the system is set up to allow e-mail alerts to both users and clients, notifying them that a job has finished. New applications can be added by subclassing a Process Class, which ends up being only about 200 new lines of code per application.
There are 2 really impressive new features of our Render Farm. The first is the ability to save groups of nodes into shots (otherwise stated after your nodes are rendered they don’t just disappear, they remain in their per-project, per-shot workspace). The second is the macro language we created that allows the creation of assets which respect the conventions of our workflow- i.e. naming conventions, file locations, etc.
We’ve just started using our new render farm and after a few more months of alpha- and beta-testing, and a visit from the digital exterminator, it will be ready for heavy production and full-time use!