What Makes Good Synthetic Training Data for Learning Disparity and Optical Flow Estimation?

May combine two claims.

Multistage training on multiple separate datasets works better than either one of the datasets by itself and better than a full mix of both.

“multistage training on multiple separate datasets works better than not only either one of the datasets by itself, but also than a full mix of both”

May combine two claims.

The generated FlyingThings3D scenes are not realistic in the naturalistic sense but allow for a large and diverse dataset.

“While the generated scenes are by no means realistic in the naturalistic sense of e.g . Sintel, they allow for a large and diverse dataset.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

FlyingThings3D is a 3D dataset rendered from true 3D models, with ground truth for stereo disparity, optical flow, and full scene flow.

“we took the randomization approach further and created FlyingThings3D: a 3D dataset rendered from true 3D models, with ground truth for stereo disparity, optical flow, and for the full scene flow task (Mayer et al, 2016).”

• cites Mayer et al, 2016Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The resulting FlyingChairs images resemble those from McCane et al (2001), but with thousands of randomized scenes instead of a few manually created ones.

“The resulting images resemble those from McCane et al (2001) ( cf. Fig. 1), yet with the crucial difference that there are thousands of randomized scenes instead of a few manually created ones.”

• cites McCane et al (2001) sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The Monkaa dataset contains articulated non-rigid motion of animals and extremely complex fur.

“Contrary to the datasets mentioned above, Monkaa contains articulated non-rigid motion of animals and extremely complex fur.”

May combine two claims.

The Sintel training set consists of 1041 image pairs, which is sufficient to train a network but not a very powerful one.

“Still, the dataset is small by deep learning standards: The training set consists of 1041 image pairs - which is sufficient to train a network, but not a very powerful one, as we show in this paper.”

May combine two claims.

Flickr textures generalize best to new datasets and yield the best results on Sintel.

“Table 4 shows that the Flickr textures generalize best to new datasets. In particular, they yield the best results on the Sintel dataset.”

May combine two claims.

Nonrigid deformations help on Sintel and KITTI, but not on the rigid FlyingChairs.

“Nonrigid deformations help on Sintel and KITTI, but not on the rigid FlyingChairs.”

May combine two claims.

Adding rotation motion yields the biggest improvement on FlyingChairs, which is also the test set with the strongest object rotation.

“The observations can be explained by looking at the types of object motion present in each test dataset: adding rotation motion yields the biggest improvement on FlyingChairs which is also the testset with the strongest object rotation.”

Probably not a substantial claim: may not pass S3 (load-bearing).

More data is always better.

“while more data is indeed always better”

• AssumptionRests on an assumption: It is generally accepted that more training data leads to better results in supervised training.

May combine two claims.

Fine-tuning on the Driving dataset teaches a network useful priors and largely improves performance on KITTI compared to training on data without such priors.

“The experiments in this paper confirm the intuition that fine-tuning on this data should teach a network useful priors, e.g . a flat road surface at the bottom, and thus largely improve the performance on KITTI compared to training on data without such priors.”

May combine two claims.

FlyingThings3D performs worse than FlyingChairs in the 40-160 pixel displacement range despite displacements in that range being well represented.

“although displacements between 40 and 160 pixels are well represented in FlyingThings3D, it still performs worse than FlyingChairs in that displacement range.”

May combine two claims.

Best results are achieved when using both augmentation and as much raw data as possible.

“we achieved the best results when using both augmentation and as much raw data as possible.”

May combine two claims.

The effect of training with nonrigid 2D deformations is noticeable in Sintel and strongest in KITTI.

“the effect of training with this is noticeable in Sintel, and strongest in KITTI.”

May combine two claims.

Color changes between frames always introduce new data because the unaugmented dataset does not contain this effect, and applying the same color augmentation to both frames does not change this.

“Contrary to this, color changes between frames always introduce new data: the unaugmented dataset does not contain this effect, and applying the same color augmentation to both frames does not change this.”

May combine two claims.

The FlyingThings3D dataset was not tuned to match any specific histogram and contains relatively few small displacements.

“The FlyingThings3D dataset was not tuned to match any specific histogram and contains relatively few small displacements.”

May combine two claims.

Adding only rotations does not lead to much better scores and even decreases the score on KITTI, while adding scaling motions on top yields a large improvement.

“While adding only rotations does not lead to much better scores and even decreases the score on KITTI, adding scaling motions on top yields a large improvement.”

May combine two claims.

A network trained on static lighting cannot exploit dynamic lighting features, but performs best on Sintel whose realistic lighting setup generates both static and dynamic effects.

“A network trained on static lighting cannot exploit dynamic lighting features, but performs best on Sintel whose realistic lighting setup generates both 'static' and 'dynamic' effects.”

May combine two claims. The quote only partly states this claim.

Additional geometry augmentation between the frames of a sample is an exception; it does not improve performance.

“The notable exception is additional geometry augmentation between the frames of a sample.”

May combine two claims.

Using the same displacement statistics as Sintel leads to the best results, whereas large deviations from these statistics lead to a large drop in performance.

“Using the same displacement statistics as Sintel (denoted as 'Sintel-like') leads to the best results, whereas large deviations from these statistics lead to a large drop in performance.”

May combine two claims.

A difference image between the disparity estimates of our two synthetic-data networks reveals that most changes occur at object boundaries, and the highlighted crops are located in an image region for which there is no ground truth at all.

“A difference image between the disparity estimates of our two synthetic-data networks reveals that most changes actually occur at object boundaries, and the highlighted crops are located in an image region for which there is no ground truth at all.”

Classified as probably not a substantial claim.

To learn optical flow estimation, any image pair with computable pixelwise ground truth displacements can serve as training data.

“This means that to learn this task, any image pair with computable pixelwise ground truth displacements can serve as training data.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

Creating photo-realistic image data is not trivial, and it is not obvious which aspects of the real world are relevant and must be modeled.

“The difficulties are that creating photo-realistic image data is not trivial, and that it is not obvious which aspects of the real world are relevant and must be modeled (these problems arise because we are unable to perfectly simulate the real world).”

Classified as probably not a substantial claim.

Starting off with a simple general scene and combining it with randomly sampled dynamic elements can yield datasets that are abstract but powerful and potentially of unlimited size.

“Using our randomized 2D and 3D datasets as examples, we first discuss how starting off with a simple general scene and combining it with randomly sampled dynamic elements can yield datasets that are abstract but powerful and potentially of unlimited size.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The 3D setting with combined object and camera motions allows for complex object flows and scene configurations.

“This 3D setting with combined object and camera motions allows for complex object flows and scene configurations.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The FlyingThings3D dataset combines sophisticated 3D rendering with procedural generation of scenes, allowing for arbitrary amounts of data without manual effort.

“The FlyingThings3D dataset combines sophisticated 3D rendering with the procedural generation of scenes which allows for arbitrary amounts of data without manual effort.”

May combine two claims.

Sintel provides ground truth for optical flow, disparities and occlusion areas, and has been largely adopted as a serious benchmark dataset despite its synthetic nature.

“Sintel provides ground truth for optical flow, disparities and occlusion areas, and has been largely adopted as a serious benchmark dataset despite its synthetic nature.”

• cites Butler et al, 2012Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

The key idea of synthetic data is that we have perfect knowledge and control of the virtual world, allowing us to produce perfect ground truth annotations along with the images.

“The key idea is that we have perfect knowledge and control of the virtual 'world' from which we create our image data, and that we are thus able to produce perfect ground truth annotations along with the images.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The MPI-Sintel dataset demonstrated the feasibility of using existing data from an open source movie to render videos together with the desired ground truth.

“The MPI-Sintel dataset demonstrated the feasibility of using existing data from an open source movie to render videos together with the desired ground truth (Butler et al, 2012).”

• cites Butler et al, 2012Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

A possible reason is that the simpler FlyingChairs dataset helps the network learn the general concept of finding correspondences without developing possibly confusing priors for 3D motion too early.

“A reason might be that the simpler FlyingChairs dataset helps the network learn the general concept of finding correspondences without developing possibly confusing priors for 3D motion too early.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

The disparity results are presumably valid for optical flow as well, just as the optical flow results can likely be applied to disparity estimation.

“Considering how closely related the two tasks are, the disparity results are presumably valid for optical flow as well, just as the optical flow results can likely be applied to disparity estimation.”

• AssumptionRests on an assumption: Optical flow and disparity estimation are closely related tasks.

Probably not a substantial claim: may not pass S3 (load-bearing).

The use of data augmentation within the training process is more efficient than storing and reading additional data from disk because data can be generated on the fly.

“While our 2D and 3D datasets offer much more training data than had previously been available, the use of data augmentation within the training process is much more efficient: additional data need not be stored and read from disk during training but can be generated on the fly.”

• AssumptionThe authors state this as an assumption: The learning algorithm is unable to filter out the small changes introduced by data augmentation and instead perceives the modified data as actual new data with new information.

May combine two claims.

An augmented existing sample is not the same as a truly new sample; augmentation cannot create new textures or change relative positions within a group of objects.

“However, an augmented existing sample is not the same as a truly new sample: for example, augmentation as we use it cannot create new textures or change the relative positions within a group of objects.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Any described geometry augmentation still yields a valid optical flow training sample by composing the computed flow induced by the augmentation operation with the existing flow of the original sample.

“Note that any of the described geometry augmentations still yields a valid optical flow training sample by simply composing the computed flow induced by the augmentation operation with the existing flow of the original sample.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

Adding these effects helps, but from the qualitative results one might expect a larger difference.

“This shows that adding these effects helps, but from the qualitative results one might expect a larger difference.”

May combine two claims. The quote only partly states this claim.

The dynamic lighting network performs worse on Sintel because Sintel scenes lack a strong directional light source, so objects have 'static' features but not the hard shadows produced in the dynamic setup.

“if there is no strong directional light source in a Sintel scene, the objects have 'static' features but not the hard shadows produced in our 'dynamic' setup.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The OVVV dataset uses a comparable approach, simulating analog video effects to make evaluation data for video surveillance systems more realistic.

“The OVVV dataset by Taylor et al (2007) uses a comparable approach, simulating analog video effects to make their evaluation data for video surveillance systems more realistic.”

• cites Taylor et al (2007)Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

Augmentation has inherent limits.

“However, augmentation has inherent limits”

• AssumptionRests on an assumption: Data augmentation increases the effective size of the training dataset.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Classical optical flow estimation methods are sensitive to displacement magnitude; variational methods have advantages for small displacements, while large displacements require extra treatment and combinatorial techniques.

“Classical optical flow estimation methods are sensitive to the displacement magnitude. Variational methods (Horn and Schunck, 1981) have clear advantages in case of small displacements, whereas large displacements require extra treatment and the use of additional combinatorial techniques (Brox et al, 2004; Brox and Malik, 2011).”

• cites Horn and Schunck, 1981 sourceNot checked yet

  Not checked in the free run.

• cites Brox et al, 2004Not checked yet

  Not checked in the free run.

• cites Brox and Malik, 2011 sourceNot checked yet

  Not checked in the free run.

May combine two claims.

In the real world and realistically lit synthetic data, interaction between light and object surfaces generates reflections, highlights, and shadows, which can violate the photoconsistency assumption and pose a challenge to vision algorithms.

“In the real world (and in realistically lit synthetic data, such as Sintel), the interaction between light and object surfaces generates reflections, highlights, and shadows. These phenomena pose a challenge to vision algorithms because they can violate the photoconsistency assumption.”

Classified as probably not a substantial claim.

The fact that synthetic data was not recorded by a physical camera already makes it different.

“yet the fact that this synthetic data was not recorded by a physical camera already makes it different.”

May combine two claims.

Disparity estimation for stereo cameras is a more geometrically constrained setting than optical flow, with fixed camera transformation, so fewer augmentation options are available.

“However, disparity estimation for stereo cameras is a more geometrically constrained setting than optical flow: the transformation between the two cameras is fixed, and e.g . rotating both views of a sample pair or zooming one view relative to the other would disturb the epipolar geometry. This means that for disparity the setting is more restricted and fewer augmentation options are available.”

The quote only partly states this claim.

The inferiority of the FlyingThings3D-trained network is partially explained by the difficulty of learning to distinguish surface materials.

“The latter would also partially explain why the FlyingThings3D dataset, which comes with realistic lighting and many object categories, leads to a network that is inferior to one trained on FlyingChairs.”

• AssumptionRests on an assumption: Some surfaces in Sintel and in real scenes are Lambertian, others shiny.

May combine two claims.

Presenting a too sophisticated dataset too early is disadvantageous, but at a later stage it actually helps.

“From this observation, we conclude that presenting a too sophisticated dataset too early is disadvantageous, but at a later stage it actually helps.”

May combine two claims.

Data augmentation serves purpose (a): if it only replicates effects found in original data, it increases effective size without changing domain, helping if dataset is not big enough.

“if it only replicates the effects found in the original data, then augmentation increases the effective size of the training set without changing the domain of the data. This can help if the dataset is just not big enough.”

• AssumptionRests on an assumption: Data augmentation increases the effective size of the training dataset.

Probably not a substantial claim: may not pass S3 (load-bearing).

The FlowNet was the first network trained end-to-end for optical flow estimation.

“The FlowNet was introduced by Dosovitskiy et al (2015). This network was the first to be trained end-to-end for optical flow estimation.”

• cites Dosovitskiy et al (2015)Not checked yet

  Not checked in the free run.

May combine two claims.

Not only the content of the data matters, but also the point of time at which it is presented to the network during training.

“This reveals that not only the content of the data matters, but also the point of time at which it is presented to the network during the training phase.”

Classified as probably not a substantial claim.

Due to time and compute power constraints, the authors could not evaluate both parts of their experiments suite on both optical flow and disparity estimation.

“Due to time and compute power constraints, we could not evaluate both parts of our experiments suite on both optical flow and disparity estimation.”

Classified as probably not a substantial claim.

Applying camera artifacts to rendered objects makes augmented-reality objects blend well into real images.

“As a result, augmented-reality objects blended well into the real images; without this adjustment, the rendered objects stood out because they looked unnaturally clean and clear.”

• cites Klein and Murray (2010) sourceNot checked yet

  Not checked in the free run.

May combine two claims.

The dynamic lighting scenario presents the most realistic effects and allows the network to learn about lighting effects, but makes the learning task harder because the network must distinguish materials and ignore specular highlight motion.

“This scenario presents the most realistic effects. It allows the network to learn about lighting effects, but this also makes the learning task harder: the network must learn to distinguish between different materials and that in case of specular materials, the changing position of a highlight is not supposed to induce optical flow.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Evaluation on Sintel covers generalization of the network to other synthetic datasets, while KITTI experiments cover generalization to real-world data in the restricted setting of a driving scenario.

“While the evaluation on Sintel covers the generalization of the network to other synthetic datasets, the KITTI experiments cover the generalization to real-world data in the restricted setting of a driving scenario.”

May combine two claims.

Realistic effects, such as sophisticated lighting models, are not important to learn basic optical flow, but merely induce minor improvements.

“Realistic effects, such as sophisticated lighting models, are not important to learn basic optical flow, but merely induce minor improvements.”

May combine two claims.

These findings are valid for optical flow (1-3) and disparity estimation (4).

“According to our evaluation, these findings are valid for optical flow (1-3) and disparity estimation (4).”

May combine two claims.

Diversity is important; a network trained on specialized data generalizes worse to other datasets than a network trained on diverse data.

“(1) Diversity is important. A network trained on specialized data generalizes worse to other datasets than a network trained on diverse data.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

These findings may not hold for high-level tasks such as object recognition.

“They may not hold for high-level tasks such as object recognition.”