|
Nuke has a really elegant way of handling this. You can localize any or all network footage in a project file to a predefined cache location. Typically this is on the local ssd/nvme. Nuke organizes it and it's easy to see what all is in this folder when
tidying up.
It takes a certain amount of time to cache the files down from a server - usually within the reach of a coffee for a typical shot - and from there you work essentially network free for the duration. Project files are saved to the network, where they are presumably
backed up a dozen different ways and/or snapshot regularly. Renders go to the server. If you render out a portion of your script as a precomp, you'll need to cache that or any other new renders when bringing it back into your comp.
Nuke retains the link to the network footage, so that if it reaches locally and doesn't find it, it reverts back to the network path. This is handy for renderfarm nodes and the like, which often don't have the footage cached locally.
I've done this (albeit totally manually and not near as elegantly) in AE using the proxy functionality, where it puts a little dot next to the footage/element in the project bin and allows you to switch between 2 locations of the footage. Which was originally
meant to be for working with large formats at lower rez, but I believe I was able to use the same size footage in 2 different locations. Don't quote me, it's been a while...
Does make me wonder out loud if someone has hot rodded this workflow and automated things better on aescripts or rendergarden or something. Most of the larger studios I've worked at have tweaked this in Nuke to be constantly doing it in the background without
the user even thinking about it.
From: After Effects Mail List <AE-List@media-motion.tv> on behalf of Stephen van Vuuren <AE-List@media-motion.tv>
Sent: Friday, November 16, 2018 11:47 AM
To: After Effects Mail List
Subject: Re: [AE] Constant hangs when switching between apps
> I thought we were supposed to be editing 8K off the cloud by now.
Yeah, the death of the desktop has been forecast about as often as the end of the world and it just doesn’t happen. Again, it’s just physics and economics. Whatever makes the cloud faster and cheaper makes local computing “more faster”
and “more cheaper” and will continue unless some revolutionary breakthrough in time, matter and space changes all that.
If you google “death of the desktop” you will get forecasts going back into the 80’s - then the dumb terminal was going to take over until mobile tech arrived.
Now we just have pocket desktops instead of pocket dumb terminals. In fact, I think foldout phones will kill off the tablet. Possibly the laptop in a much longer timeframe but iOS and Android will need to have much better cursor, mouse
and keyboard features to make that happen.
But computing needs in the foreseeable future will always demand beefier stuff for power users and at least in us old guys lifetimes, no evidence points to that changing.
From: After Effects Mail List <AE-List@media-motion.tv>
Sent: Friday, November 16, 2018 1:13 PM
To: After Effects Mail List <AE-List@media-motion.tv>
Subject: Re: [AE] Constant hangs when switching between apps
I thought we were supposed to be editing 8K off the cloud by now.
> A properly configured SAN works pretty darn well across multiple workstations, but can still bog down during peak use.
Which is a failure of engineering 101 – you design infrastructure to handle Max Q i.e. beyond peak use, not for normal traffic. Sure, this basic principle is violated all the time in the world and creates a lot of problems (DC Beltway),
but in the case of storage, there is no reason other than it’s expedient. It’s certainly not cheaper or fasters.
A film is – or should be – more like music than like fiction. It should be a progression of moods and feelings. The theme, what’s behind the emotion, the meaning, all that comes later.
A properly configured SAN works pretty darn well across multiple workstations, but can still bog down during peak use.
It’s when a NAS or SMB/AFP filesharing is used as if it’s a SAN that things tend to go sideways.
Working on one film with very unique specs for several years is very different than 90-95% of production studios and agencies who may be turning over 1-15 jobs per week and has multiple artists working on several projects simultaneously
(and possibly even hopping from machine to machine). Don’t even start trying to figure out how to go about archiving these jobs if the work files are all over the place - especially when using freelancers who have different organization and work habits and
may leave before the project is even completed.
Try integrating a render farm with multiple artists using the setup you’re recommending...
Once you add in live action plates, editorial, mgfx, and vfx tasks and even if you could possibly manage all of this you would cause nearly the same amount of network thrashing just trying to get artists the right source files, not to
mention always having to wait for the files to be synced to the right places.
The network would come to a standstill every time you started a new project while trying to sync everything to the right destinations.
A better option in AE’s case is to have a large SSD for your Disk Cache; you should then effectively be reducing network traffic but more intelligently and without the IT overhead.
> Yeah, I'm not sure how we'd effectively have a team of artists working on a bunch of different shots simultaneously on the same project if the files had to be synced locally before they could use them.
I did this many thousand of times on my IMAX film. It’s just getting the workflow right and because you’re only copying a file once or twice across a LAN, instead of hundreds or thousands of times, the speed improvement and cost reduction
is very large.
>>using Render Garden to distribute a render that's going right to an editor, etc. and all of that needs to happen immediately -- it doesn't work for something to sync overnight.
It’s just a question of what gets distributed where and when – and during my heavy years on the film, I had hundreds of syncs and mirrors happening a week – dozens on busy days.
>> We sometimes even sync entire projects from our network drive to Dropbox so remote freelancers can work on them as well.
You just argued for working local solution. A remote freelancer is functionally identical to a network workstation which is remote from central server. Just much greater latency and poorer bandwidth, so the issue is painfully easy to
see. But that does mean this issue vanishes when someone is inside the building. The identical structural issues are all there, just short enough duration that we put up with it.
But just because something only takes seconds or minutes longer working network workstation then working locally in the most conservative examples, that means over time, months, years, we are talking about massive effects on cost and
performance as well increased risks of problems.
>> I'm all for things going faster, but that seems impossible without working off of central storage on a network. I'd be curious what kind of infrastructure and workflow design could get around this.
Again, I’m in the final stages of my IMAX film. We had 100+ volunteers but the bulk of the work was done here, primarily by me but due to the slowness of working with the files, I normally had three workstations plus a laptop running
around me (while one machine was previewing frames, move to the other) so that’s 4 workstations plus 15 render boxes all rendering projects from the three workstation, 2 nearline servers and two archival servers and 2 cloud backup services.
We have three main teams of volunteers all working remotely and using both Dropbox and Gdrive to sync projects and assets but for some on slow connections, FedEx and hard drive was the way to go.
Never was a file worked over the network. 33 Million files at peak, 700 TB at peak.
The scope of how is both beyond email discussion and would vary widely based on the biz.
> Then every VFX studio I've worked at, between 4 and 400 employees, is doing it wrong apparently.
Yeah, I'm not sure how we'd effectively have a team of artists working on a bunch of different shots simultaneously on the same project if the files had to be synced locally before they could use them. We're constantly updating an asset,
bringing in a new version of the 3D render to comp while it's still rendering, using Render Garden to distribute a render that's going right to an editor, etc. and all of that needs to happen immediately -- it doesn't work for something to sync overnight.
We sometimes even sync entire projects from our network drive to Dropbox so remote freelancers can work on them as well.
I'm all for things going faster, but that seems impossible without working off of central storage on a network. I'd be curious what kind of infrastructure and workflow design could get around this.
Workstations working directly off network storage is slower, more expensive, more prone to failure and a huge waste of time and money.
Then every VFX studio I've worked at, between 4 and 400 employees, is doing it wrong apparently.
|
