It sounds like the Nvidia tech is generating those mattes automatically and using them to blend optical flow retimed versions of the footage, but that it's helped by a trained neural net.Ā Here's the summary of the
white paper they released; I bolded some key parts:
Given two consecutive frames, video interpolation aims at generating intermediate frame(s) to form both spatially and temporally coherent video sequences. While most existing methods focus on single-frame interpolation, we propose an end-to-end convolutional neural network for variable-length multi-frame video interpolation, where the motion interpretation and occlusion reasoning are jointly modeled. We start by computing bi-directional optical flow between the input images using a U-Net architecture. These flows are then linearly combined at each time step to approximate the intermediate bi-directional optical flows. These approximate flows, however, only work well in locally smooth regions and produce artifacts around motion boundaries. To address this shortcoming, we employ another U-Net to refine the approximated flow and also predict soft visibility maps. Finally, the two input images are warped and linearly fused to form each intermediate frame. By applying the visibility maps to the warped images before fusion, we exclude the contribution of occluded pixels to the interpolated intermediate frame to avoid artifacts. Since none of our learned network parameters are time-dependent, our approach is able to produce as many intermediate frames as needed. We use 1,132 video clips with 240-fps, containing 300K individual video frames, to train our network. Experimental results on several datasets, predicting different numbers of interpolated frames, demonstrate that our approach performs consistently better than existing methods.
So, yeah, David - it sounds like they've improved the video analysis part to deal with objects moving separately with machine learning.Ā Which current optical-flow based tech doesn't do, to the best of my knowledge.
Looks quite cool!Ā It'd be nice to see some samples of how it does on footage that isn't already slow-motion.Ā There are some still frames at the end of the above-linked white paper, but no motion.
- Nathan
