From: "Peter Litwinowicz" Received: from outmail146037.authsmtp.co.uk ([62.13.146.37] verified) by media-motion.tv (CommuniGate Pro SMTP 6.1.0) with ESMTPS id 6467849 for AE-List@media-motion.tv; Wed, 20 Jun 2018 21:11:05 +0200 Received: from punt24.authsmtp.com (punt24.authsmtp.com [62.13.128.105]) by punt21.authsmtp.com. (8.15.2/8.15.2) with ESMTP id w5KJImvY022123 for ; Wed, 20 Jun 2018 20:18:48 +0100 (BST) (envelope-from pete@revisionfx.com) Received: from mail-c245.authsmtp.com (mail-c245.authsmtp.com [62.13.128.245]) by punt24.authsmtp.com. (8.15.2/8.15.2) with ESMTP id w5KJImrT008192 for ; Wed, 20 Jun 2018 20:18:48 +0100 (BST) (envelope-from pete@revisionfx.com) Received: from [10.0.0.100] (c-73-158-152-40.hsd1.ca.comcast.net [73.158.152.40]) (authenticated bits=0) by mail.authsmtp.com (8.15.2/8.15.2) with ESMTPA id w5KJIkxV071166 for ; Wed, 20 Jun 2018 20:18:47 +0100 (BST) (envelope-from pete@revisionfx.com) User-Agent: Microsoft-MacOutlook/14.7.7.170905 Date: Wed, 20 Jun 2018 12:18:44 -0700 Subject: Re: [AE] nVidias Slo mo demo To: After Effects Mail List Message-ID: Thread-Topic: [AE] nVidias Slo mo demo References: In-Reply-To: Mime-version: 1.0 Content-type: multipart/alternative; boundary="B_3612341928_4548995" X-Server-Quench: c236398f-74be-11e8-8e1a-9cb654bb2504 X-AuthReport-Spam: If SPAM / abuse - report it at: http://www.authsmtp.com/abuse X-AuthRoute: OCdyYwEUBFZNQRgX IjkJEi5bSAIjKA0K ChgGOyJ3ZX4IThh5 N0tWL1MWNVoARFZf HSJNFVZUTyc0B3x3 awtZfQddaEtGVQFi Hk9KXFNXHBtpBB4f Ah4BUBxwchpGNjl1 HjwbBTg9XEN7dEB7 DE9REG8PKzRia2Yc WUgOagRQIlZXfx4Q Yk13AXJcfGxUNHxl QwU5ZW1pYSNlJXYd cSggGBoUf0cHHzcn DysGGn0qGgUJSiA0 R38A X-Authentic-SMTP: 61633133393137.1039:706 X-AuthFastPath: 0 (Was 255) X-AuthVirus-Status: No virus detected - but ensure you scan with your own anti-virus system. > This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. --B_3612341928_4548995 Content-type: text/plain; charset="UTF-8" Content-transfer-encoding: quoted-printable > Maybe Peter with RE:Vision will chime in this discussion and correct me i= f I > am wron >=20 Well I will chime in. =20 Mostly you are correct about Twixtor. Also Twixtor will let you segment th= e images (with mattes) as well as hand-guide the tracking (with splines, points and the like). > Where I think these systems can improve is in the automatic recognition o= f the > moving objects in your frame, i.e recognizing "the person walking" and "t= he > picket fence" as two different objects. I No, this is not really how it works. If there happens to be a sequence similar enough to your scene of a picket fence and person in the training sequences for the neural network, then the nVidia version MIGHT work okay, might not. But it doesn't recognize objects as such, more than it pattern matches what is happening over a large set of training sequences. I guess that's a form of recognizing objects, true, but it doesn't really "understand" the objects per se. If there is no such "similar" sequence in the training sequences, then you are SOL. What do you do then? And, as people have said here, the artifacts in the nVidia examples are still there. BUT, what is different than with Twixtor, is that with Twixto= r you can go in and fix the problems. I don't see how you go in and fix the nVidia version based on training sequences, other than with painting. I guess that can be true with Twixtor too, but with the optical flow tools we provide, you can fix, segment (via roto) and guide the tracking. I have no idea how you would do that with nVidia's neural network-trained version. And perhaps someone can correct me if I'm wrong about nVidia's version. Pete From: After Effects Mail List on behalf of David Baud Reply-To: After Effects Mail List Date: Wednesday, June 20, 2018 at 11:31 AM To: After Effects Mail List Subject: Re: [AE] nVidias Slo mo demo My understanding is that in order to get good result with any of the optica= l flow solutions, the system needs to be able to define the contour of your moving =E2=80=9Cobjects=E2=80=9D (i.e a person, a ball, a car, etc=E2=80=A6) in your frame. B= etter the system is capable of recognizing these objects, better results you will get. Twixtor Pro version will let you =E2=80=9Chelp=E2=80=9D the system to define these contour by providing a mask for your object. As we know it can be time consuming to do rotoscoping. Where I think these systems can improve is in the automatic recognition of the moving objects in your frame, i.e recognizing "the person walking" and "the picket fence" as two different objects. I am not familiar with the proposed technology by nVidia, but mayb= e they improved the analysis of a video and the system is capable of calculating automatically the displacement of all objects in a frame separately? Maybe Peter with RE:Vision will chime in this discussion and correct me if = I am wrong =F0=9F=98=89=E2=80=A6 and maybe gives us a better understanding of the optical f= low technology in general=E2=80=A6 without revealing his secret sauce for Twixtor! David Baud Colorist & Finishing Editor david at kosmos-productions.com > On Jun 20, 2018, at 11:59 , Jim Curtis wrote: >=20 > Optical Flow and Twixtor have limitations. Try a slo-mo of a person walk= ing > next to a picket fence, and see how wacky the pickets become with any met= hod > besides frame-blending. There have been occasions where I=E2=80=99ve stitched > together the different methods with masking and editing, as there seems n= ot to > be a Silver Bullet so far. If this is it, I=E2=80=99m interested! Thanks for = the > head=E2=80=99s up. --B_3612341928_4548995 Content-type: text/html; charset="UTF-8" Content-transfer-encoding: quoted-printable
Maybe Peter with RE:Vision will chime in this disc= ussion and correct me if I am wron

Wel= l I will chime in.
 
Mostly you are correct about T= wixtor.  Also Twixtor will let you segment the images (with mattes) as = well as hand-guide the tracking (with splines, points and the like).
Where I = think these systems can improve is in the automatic recognition of the movin= g objects in your frame, i.e recognizing "the person walking" and "the picke= t fence" as two different objects. I

N= o, this is not really how it works.  If there happens to be a sequence = similar enough to your scene of a picket fence and person in the training se= quences for the neural network, then the nVidia version MIGHT work okay, mig= ht not.  But it doesn't recognize objects as such, more than it pattern= matches what is happening over a large set of training sequences.  I g= uess that's a form of recognizing objects, true, but it doesn't really "unde= rstand" the objects per se.  If there is no such "similar" sequence in = the training sequences, then you are SOL.  What do you do then? 

And, as people have said here, the artifacts in the = nVidia examples are still there.  BUT, what is different than with Twix= tor, is that with Twixtor you can go in and fix the problems.  I don't = see how you go in and fix the nVidia version based on training sequences, ot= her than with painting.  I guess that can be true with Twixtor too, but= with the optical flow tools we provide, you can fix, segment (via roto) and= guide the tracking.  I have no idea how you would do that with nVidia'= s neural network-trained version.

And perhaps someo= ne can correct me if I'm wrong about nVidia's version.
Pete
<= div>

From: After E= ffects Mail List <AE-List@media-= motion.tv> on behalf of David Baud <AE-List@media-motion.tv>
Reply-To: After Effects Mail List <AE-List@media-motion.tv>
Date: Wednesday, June 20, 2018 at 11:31 AM
To: After Effects Mail List <AE-List@media-motion.tv>
Subject: Re: [AE] nVidias Slo mo demo

=
My understanding is that in order to get good r= esult with any of the optical flow solutions, the system needs to be able to= define the contour of your moving “objects” (i.e a person, a ba= ll, a car, etc…) in your frame. Better the system is capable of recogn= izing these objects, better results you will get. Twixtor Pro version will l= et you “help” the system to define these contour by providing a = mask for your object. As we know it can be time consuming to do rotoscoping.= Where I think these systems can improve is in the automatic recognition of = the moving objects in your frame, i.e recognizing "the person walking" and "= the picket fence" as two different objects. I am not familiar with the propo= sed technology by nVidia, but maybe they improved the analysis of a video an= d the system is capable of calculating automatically the displacement of all= objects in a frame separately?

Maybe Peter with RE:Vision will chime in this discussion and correct me if= I am wrong =F0=9F=98=89… and maybe gives us a better understanding of the opt= ical flow technology in general… without revealing his secret sauce fo= r Twixtor!

David Baud
Colorist &a= mp; Finishing Editor
= david at kosmos-productions= .com

On Jun 20, 2018, at 11:59 , Jim Curtis <AE-List@media-motion.tv>= ; wrote:

Optical Flow and Twixtor have limitations.  Try = a slo-mo of a person walking next to a picket fence, and see how wacky the p= ickets become with any method besides frame-blending.  There have been = occasions where I’ve stitched together the different methods with mask= ing and editing, as there seems not to be a Silver Bullet so far.  If t= his is it, I’m interested!  Thanks for the head’s up.
=

--B_3612341928_4548995--