FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks

FlowNet 2.0 advances the concept of end-to-end learning of optical flow, building on the original FlowNet, which demonstrated that optical flow estimation could be cast as a learning problem but still could not match traditional variational methods in quality, particularly on small displacements and real-world data. The paper attributes its large improvements in quality and speed to three major contributions: focusing on the schedule of presenting training data, which proves very important; developing a stacked architecture that includes warping of the second image with intermediate optical flow; and introducing a subnetwork that specializes in small motions to better handle small displacements.

The resulting system is only marginally slower than the original FlowNet but decreases the estimation error by more than 50%, performing on par with state-of-the-art methods while running at interactive frame rates. The authors also present faster variants that allow optical flow computation at up to 140fps while matching the accuracy of the original FlowNet. This showed that carefully engineered deep networks could finally compete with, and match, traditional optical flow methods in both accuracy and speed.