|
I would also note that Nvidia makes NO attempt to fix the REAL problem with software slomo, which is deconvolution of the original motion blur to a shutter angle matching the target frame rate.
Tim Sassoon
Venice, CA
On Jun 20, 2018, 12:19 PM -0700, Peter Litwinowicz <AE-List@media-motion.tv>, wrote:
Maybe Peter with RE:Vision will chime in this discussion and correct me if I am wron
Well I will chime in.
Mostly you are correct about Twixtor. Also Twixtor will let you segment the images (with mattes) as well as hand-guide the tracking (with splines, points and the like).
Where I think these systems can improve is in the automatic recognition of the moving objects in your frame, i.e recognizing "the person walking" and "the picket fence" as two different objects. I
No, this is not really how it works. If there happens to be a sequence similar enough to your scene of a picket fence and person in the training sequences for the neural network, then the nVidia version MIGHT work okay, might not. But it doesn't recognize objects as such, more than it pattern matches what is happening over a large set of training sequences. I guess that's a form of recognizing objects, true, but it doesn't really "understand" the objects per se. If there is no such "similar" sequence in the training sequences, then you are SOL. What do you do then?
And, as people have said here, the artifacts in the nVidia examples are still there. BUT, what is different than with Twixtor, is that with Twixtor you can go in and fix the problems. I don't see how you go in and fix the nVidia version based on training sequences, other than with painting. I guess that can be true with Twixtor too, but with the optical flow tools we provide, you can fix, segment (via roto) and guide the tracking. I have no idea how you would do that with nVidia's neural network-trained version.
And perhaps someone can correct me if I'm wrong about nVidia's version.
Pete
My understanding is that in order to get good result with any of the optical flow solutions, the system needs to be able to define the contour of your moving āobjectsā (i.e a person, a ball, a car, etcā¦) in your frame. Better the system is capable of recognizing these objects, better results you will get. Twixtor Pro version will let you āhelpā the system to define these contour by providing a mask for your object. As we know it can be time consuming to do rotoscoping. Where I think these systems can improve is in the automatic recognition of the moving objects in your frame, i.e recognizing "the person walking" and "the picket fence" as two different objects. I am not familiar with the proposed technology by nVidia, but maybe they improved the analysis of a video and the system is capable of calculating automatically the displacement of all objects in a frame separately?
Maybe Peter with RE:Vision will chime in this discussion and correct me if I am wrong šā¦ and maybe gives us a better understanding of the optical flow technology in generalā¦ without revealing his secret sauce for Twixtor!
Optical Flow and Twixtor have limitations. Try a slo-mo of a person walking next to a picket fence, and see how wacky the pickets become with any method besides frame-blending. There have been occasions where Iāve stitched together the different methods with masking and editing, as there seems not to be a Silver Bullet so far. If this is it, Iām interested! Thanks for the headās up.
|
