The Pascal Visual Object Classes Challenge: A Retrospective

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Image and instance counts increased until 2011; between 2011 and 2012, only segmentation and action classification datasets were augmented, while classification, detection, and person layout were not.

“The number of images and instances in all the tasks was increased up to 2011.From 2011 to 2012 the number of images in the classification, detection and person layout tasks was not increased, and only those for segmentation and action classification were augmented.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Person layout evaluation criteria were relaxed from 2010 onward, but the task remained less popular than others.

“These criteria were relaxed from VOC 2010 on, though this task never became as popular as the others.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The person layout challenge had 850 annotated object instances in 609 images.

“There were 850 annotated objects instances in 609 images for the person layout challenge.”

May combine two claims.

NUS_SCM won competition 1 in 2012, outperformed all other methods for every class, and improved on 2011 winners for all classes except pottedplant.

“The winning method for competition 1 is NUS_SCM.Its performance exceeded all other methods (including those in competition 2) for all classes in 2012, and also improved on the 2011 winning entries in all but one class ('pottedplant').”

• AssumptionRests on an assumption: Participants followed the best practice guidelines of tuning parameters on the validation set and running algorithms only once on the test set.

May combine two claims.

Interleaving annotation and checking allowed errors to be caught earlier and substantially reduced post-hoc checking.

“In this way, errors were picked up and corrected earlier and post-hoc checking was substantially reduced.”

May combine two claims.

From 2009 onward, datasets for all tasks were previous years' images plus new images; before that, classification and detection used entirely new datasets each year.

“From 2009 onwards the data for all tasks consists of the previous years' images augmented with new images.Before this, in 2008 and earlier, an entirely new dataset was released each year for the classification/detection tasks.”

May combine two claims.

Dog, sheep, and cow showed large rank shifts under normalisation: dog fell five places to sixteenth, sheep rose five places to ninth, and cow rose five places to tenth.

“Other classes with large changes in rank are: 'dog' (down 5 places to 16), 'sheep' (up 5 places to 9) and 'cow' (also up 5 places to 10).”

Probably not a substantial claim: may not pass S3 (load-bearing).

About 50% of annotation time was spent on segmentation annotation and checking.

“Over time, the annotation period was adapted until around 50% of the time was spent on segmentation annotation and checking.”

May combine two claims.

Teams led by Felzenszwalb were joint winners in 2008 and 2009, and their released code led to widespread use and development of the method.

“Teams led by Felzenszwalb were joint winners of the challenge in 2008 and 2009, and the released code meant that this method was widely used/developed by others.”

May combine two claims.

The boat class was among the better classes for classification but among the worst for detection.

“The most dramatic difference is for the 'boat' class which is one of the better performing classes for classification but one of the worst for detection.”

• AssumptionThe authors state this as an assumption: The number of possible object locations in an image is very large relative to differences in positive counts between classes.
• AssumptionThe authors state this as an assumption: The number of negative examples is roughly equal across classes even when positive counts differ.

May combine two claims.

Cat and dog each fell three places in the normalised detection ranking, which reflected their higher frequency in the test set.

“The other two classes whose ranking drops after normalisation are 'cat' and 'dog', in both cases by three places in the ranking, reflecting the higher rate of occurrence of cats and dogs in the test set.”

• AssumptionThe authors state this as an assumption: The number of possible object locations in an image is very large relative to differences in positive counts between classes.
• AssumptionThe authors state this as an assumption: The number of negative examples is roughly equal across classes even when positive counts differ.

May combine two claims.

In classification, the spread of mean AP scores and the highest AP achieved gradually increased across years.

“The diversity in performances has gradually increased, as has the highest AP achieved.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

By 2012, action classification had 4588 training and validation images, with about 400 examples per action class.

“Indeed, by 2012 there were 4588 training and validation images, with about 400 examples of people carrying out each class of action.”

May combine two claims.

The University of Amsterdam's 2012 winning entry used selective search to hypothesise bounding boxes bottom-up from multiple segmentations, allowing more expensive features and classifiers.

“This is the route, termed 'selective search', followed by the University of Amsterdam entry which won in 2012, and allowed them to use more computationally expensive features and classifiers.”

• cites Van de Sande et al, 2011Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Checking grew to about 50% of annotators' time.

“Over time, the checking part of the annotation effort grew to take around 50% of the annotators' time.”

May combine two claims.

For most classes, the best detection AP gradually increased over the years, though this trend was weaker than for classification.

“For most classes the AP of the best performing method has gradually increased over the years, although this trend is less strong than for the classification task (see Fig. 21).”

May combine two claims.

In detection, the spread of mean AP scores and the highest AP achieved gradually increased across years.

“Again the diversity in performances has gradually increased, as has the highest AP achieved.”

May combine two claims.

For all classes, normalised detection precision increased with object size, except for occasional drops for extra-large objects, which are often highly truncated.

“For all classes we observe trends similar to those noted by Hoiem et al (2012) in that the normalised precision (see Eq. 6) increases as function of size, except that there is sometimes a drop for extra-large objects, which are often highly truncated.”

May combine two claims.

In the two-method super-classifier experiment, combined performance was often significantly better than the best individual method for some classes, and for some classes better than the all-method super-classifier.

“In Table 14, we note that the performance of the super-classifier trained using only scores from two methods is often significantly higher than the best-performing individual method (e.g. for 'bottle', 'chair' and 'potted plant'), and for some classes higher than that of the super-classifier trained for all methods together (e.g.'bicycle').”

• Also stated in: § 6 Progress Through Time · plain

  “Note that the performance of the super-classifier trained using only scores from two methods as input features (second to last column) is often significantly higher than the best-performing individual method (second column), and for some classes higher than that of the super-classifier trained all methods together (last column).”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Also in 2010, MIT/UCLA extended DPM with a three-layer architecture and data terms using visual words and HOG features.

“Also in 2010, the MIT/UCLA team (Zhu et al, 2010) extended the DPM model by allowing a 3-layer architecture (corresponding to a root, parts and sub-parts), and by incorporating data terms involving a histogram of 'visual words' for region appearance as well as HOG features for edges.”

• cites Zhu et al, 2010Not checked yet

  Not checked in the free run.

May combine two claims.

At first participants treated classification, detection, and segmentation independently, but over time the tasks became more cross-fertilised, with detection and segmentation helping classification.

“originally these tasks were treated quite independently by participants, but as the challenge progressed there was progressively more cross-fertilisation, with detection and segmentation being used to enhance the classification performance for example.”

May combine two claims.

Normalisation affected detection rankings less than classification rankings, with most detection classes barely moving.

“In general the impact of the normalisation is less for detection than for classification, with most classes hardly changing in rank.”

• AssumptionThe authors state this as an assumption: The number of possible object locations in an image is very large relative to differences in positive counts between classes.
• AssumptionThe authors state this as an assumption: The number of negative examples is roughly equal across classes even when positive counts differ.

May combine two claims.

Between 2009 and 2012, background false-positive errors increased as a percentage, while confusions with similar and dissimilar objects generally decreased.

“There is a marked trend that the percentage of background errors has increased between 2009 and 2012, while in general the confusions with similar and dissimilar objects have decreased.”

May combine two claims.

BONN_JOINT placed second in segmentation competition 5 and first in competition 6.

“The second placed method in competition 5 and the winning entry in competition 6 was BONN_JOINT which created multiple segmentations of each image and then sampled from a distribution over tilings constructed from these segments (see Ion et al, 2011a).”

Probably not a substantial claim: may not pass S3 (load-bearing).

In VOC 2009, Vedaldi et al used a cascade of methods of increasing complexity so that expensive non-linear methods ran only on a subset of locations chosen by cheaper methods.

“In VOC 2009, Vedaldi et al (2009) used a cascade of methods of increasing complexity using linear, quasi-linear and non-linear kernels, with the idea that the more expensive non-linear methods need only be run on a subset of locations determined by the cheaper methods.”

• cites Vedaldi et al (2009)Not checked yet

  Not checked in the free run.

May combine two claims.

The image groups were ordered by increasing difficulty, characterised by smaller objects, more truncation or occlusion, more illumination or viewpoint variation, and more background clutter.

“In each case the groups of images are ordered to show a steady increase in difficulty.This increase can be characterised by reduction in object size, increased truncation or occlusion, increased variation in illumination or viewpoint, increased background clutter.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Apart from some participants making multiple similar submissions, the organisers were not aware of any problems.

“Other than some participants mak-ing multiple similar submissions (which contravenes the best practice of choosing the method submitted using the validation set), there were no problems that we were aware of.”

May combine two claims.

Images that 27% of methods on average identified correctly are still harder, with more cluttered backgrounds and smaller bicycles.

“Images identified correctly by 27% of methods on average.Still harder images with more cluttered backgrounds and smaller bicycles.”

May combine two claims.

Cross-fertilisation developed between the challenges, with detectors used in segmentation and classification methods and unsupervised segmentation used to propose windows in detection methods, whereas originally methods applied to only one challenge.

“Third, has been the development of cross fertilisations between the challenges -detectors used as part of the segmentation and classification methods, and (unsupervised) segmentation used to propose windows in detection methodswhere originally methods were myopic in only applying to one of the three challenges.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Multiple objects from multiple classes can be present in the same image.

“Note that multiple objects from multiple classes may be present in the same image.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Seventy-three classification methods were submitted since 2009.

“Each row corresponds to one of the 73 classification methods submitted since 2009.”

May combine two claims.

Images that 94% of methods on average identified correctly are slightly harder: the aeroplane is still large but may be truncated or seen from a more unusual viewpoint, and the backgrounds are still largely uncluttered.

“Images identified correctly by 94% of methods on average.Slightly harder images where the aeroplane is still large but may be truncated or seen from a more unusual viewpoint.The backgrounds are still largely uncluttered.”

May combine two claims.

The super-classifier outperforms all other methods on some object classes, while its performance drops slightly on three classes.

“Gold indicates the increase in performance on object classes where the superclassifier beats all other methods, averaged across the two folds.The three classes for which performance drops slightly have been highlighted in red.”

May combine two claims.

Detection performance was generally better for non-truncated objects than truncated ones, except that the top three methods in 2009 and 2012 preferred truncated over non-truncated cats.

“and it is better for non-truncated objects than truncated ones (except that the top three methods in 2009 and 2012 all prefer truncated over nontruncated cats).”

May combine two claims.

Many papers evaluate results using different averaging methods, such as micro-averaging and macro-averaging, making results not comparable.

“In cases where this is not provided we still see many examples in papers of authors evaluating the results in different ways -for example some using micro-averaging (giving equal weight to each image) and others using macroaveraging (giving equal weight to each class) -so that results are not comparable.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The 2009 segmentation competition had twelve entries, probably the broadest set, and its methods built on and extended those from the previous year.

“The 2009 competition had probably the broadest set of entries, twelve in all, whose methods built on and extended those used in the previous year.”

May combine two claims.

Both the number of images and the number of objects more than doubled between 2008 and 2012.

“Both the number of images and number of objects more than doubled between 2008 and 2012.”

May combine two claims.

Images that 88% of methods on average identified correctly are still harder, with more cluttered backgrounds, smaller or more truncated aeroplanes, and more varied lighting.

“Images identified correctly by 88% of methods on average.Still harder images with more cluttered backgrounds, smaller or more truncated aeroplanes and more varied lighting.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The UP method appears only for the person class, because it entered only that class.

“Method UP is only used in the 'person' class since it only participated in that class.”

May combine two claims.

For most classes, the best segmentation accuracy increased between 2009 and 2012, often non-monotonically; from 2011 to 2012 there was substantial improvement in all classes except bus.

“For most classes the accuracy of the best performing method has increased between 2009 and 2012, although often not monotonically with the year.In particular, in the final years from 2011 to 2012 there was substantial improvement in all classes except bus.”

May combine two claims.

Images that 52% of methods on average identified correctly are slightly harder: the bicycle is smaller and may be seen from the rear or front.

“Images identified correctly by 52% of methods on average.Slightly harder images where the bicycle is smaller and may be seen from the rear or front.”

May combine two claims.

From 2010 to 2012, the two dominant segmentation methods were hierarchical random fields with a range of potentials, and multiple bottom-up segmentations combined with a classifier predicting segment-object overlap.

“The two dominant methods were: hierarchical random fields with a range of potentials, and the use of multiple bottom-up segmentations, combined with a classifier to predict the degree of overlap of a segment with an object.”

May combine two claims.

The 2008 segmentation competition had six participants and was the first year in which segmentation methods beat the automatic entries from the detection challenge.

“The 2008 segmentation competition had six participants and was the first year where the segmentation methods beat the automatic entries from the detection challenge.”

Probably not a substantial claim: may not pass S3 (load-bearing).

All images in the classification and detection challenges are annotated with bounding boxes for every instance of the twenty classes.

“There is complete annotation for the twenty classes: i.e. all images are annotated with bounding boxes for every instance of the twenty classes for the classification and detection challenges.”

May combine two claims.

The aspect ratio of a bounding box may carry information about the action, which is largely removed when only a point is given.

“The motivation for this additional point annotation was that the aspect ratio of the bounding box might provide some information on the action being performed, and this was almost entirely removed if only a point was provided.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Segmentation is more challenging than detection and typically takes longer to solve.

“Segmentation clearly is more challenging than detection and its solution tends to be more time consuming.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The test set's number of instances per class and number of images are approximately balanced with the trainval set, though exact instance counts are not released.

“Note, we do not release the exact numbers of object instances in the test set, but both the number of instances per class and number of images are approximately balanced with those in the trainval set.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

In 2012, the person of interest was specified by both a bounding box and a torso point, with separate competitions for each.

“In 2012 the person of interest was specified by both a bounding box and a point on the torso, and a separate competition defined for each.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Providing confidence levels allows ranking results and evaluating the false positive/false negative trade-off without arbitrary error costs.

“The provision of a confidence level allows results to be ranked such that the trade-off between false positives and false negatives can be evaluated, without defining arbitrary costs on each type of classification error.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Detection is more demanding than classification because it requires locating instances in an image.

“It is clear that the additional requirement to locate the instances in an image makes detection a more demanding task than classification.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Each object's annotation includes attributes such as orientation, occlusion, truncation, and difficulty.

“In addition to a bounding box for each object, attributes such as: 'orientation', 'occluded', 'truncated', 'difficult'; are specified.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Classification can be used with regression methods for counting, which have performed well without detection.

“Classification could also be used to put regression methods for counting into action, which have been shown to perform well without any detection (Lempitsky and Zisserman, 2010).”

• cites Lempitsky and Zisserman, 2010Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Person layout prediction success depends on correctly predicting which parts are present and their bounding boxes.

“The success of the layout prediction depends both on: (i) a correct prediction of parts present/absent (e.g. are the hands visible or occluded); (ii) a correct prediction of bounding boxes for the visible parts.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Participants were encouraged to tune parameters on the validation set and run algorithms only once on the test set.

“We encouraged participants to observe the best practice guidelines, given at the Pascal VOC best practice guidelines web page (2012), that parameters should be tuned on the validation set and algorithms run only once on the test set, so that the organisers were not asked to choose the best result for them.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Classification can be a useful initial step even when detection is the final goal, by guiding resources to promising images.

“Even in cases where detection is the end goal, classification may be an appropriate initial step to guide resources towards images that hold good promise of containing the target class.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Participants were required to submit only one entry per method.

“In addition to withholding the test data annotation, it was also required that participants submit only a single entry per method.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The test set annotations are not publicly released.

“Note, the annotations on the test set are not publicly released.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Detection supports more applications than classification, such as obstacle avoidance and tracking.

“It is also true that detection can support more applications than mere classification, e.g.obstacle avoidance, tracking, etc.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Detection can provide a good initialization for segmentation.

“Even if segmentation is the goal, detection can provide a good initialization (e.g.Leibe et al, 2004).”

• cites Leibe et al, 2004Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Introducing an 'other' action class in 2011 increased the challenge's difficulty.

“In 2011 an 'other' class was introduced (for actions different from the ten already specified).This increased the difficulty of the challenge.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The evaluation server limited how many times participants could submit results for earlier years because earlier images are subsets of the current year's.

“To add to this encouragement, the evaluation server restricted the number of times a participant could submit results for earlier years (since performance could be partially gauged as earlier years' images are a subset of the current year's).”

Classified as probably not a substantial claim.

The normalised precision measure enables meaningful comparison of classification accuracy between classes.

“This normalised measure will allow us to compare classification accuracy across classes meaningfully.”

May combine two claims.

Standard reporting may understate how hard it is to detect people; state-of-the-art people detection was only slightly better than detection of plants and chairs.

“This again suggests that the normal way of reporting results may underestimate the difficulty of detecting people in images -the state-of-the-art accuracy for detecting people is in fact only slightly better than that of detecting plants and chairs, despite all the special research effort that has gone into this case.”

• AssumptionThe authors state this as an assumption: The number of possible object locations in an image is very large relative to differences in positive counts between classes.
• AssumptionThe authors state this as an assumption: The number of negative examples is roughly equal across classes even when positive counts differ.

May combine two claims.

All submissions from 2009 onward provided predictions for the VOC 2009 test images, so that dataset served as a common test set for all submissions.

“All submissions since 2009 have provided predictions for the VOC 2009 test images, since images were only added to the test set over the years; thus this dataset provides a common test set for all submissions.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

A method's rank interval depends on all competition entrants, whereas its individual confidence interval depends only on that method.

“Note that rank ranges depend on all entrants in a competition, while the individual confidence interval is a property of a single method.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The winning NEC/UIUC entry in 2009 used local coordinate coding to replace vector quantisation, allowing a feature vector to be represented by more than one template.

“For example the winning NEC/UIUC entry in 2009 used local coordinate coding (LCC, Yang et al 2009) to replace the vector quantisation step, so that a feature vector could be represented by more than one template.”

• cites Yang et al 2009Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Snapshots of a dynamic world convey substantial information about the dynamics.

“The motivation was that the world is dynamic and snapshots of it still convey substantial information about these dynamics.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Action classes are not mutually exclusive; a person can perform multiple actions simultaneously.

“Note, the action classes are not exclusive, for example a person can be both 'riding a bicycle' and 'phoning'.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The SVM's C parameter was chosen on a held-out 10% validation set before training on all training data.

“An optimal value for the parameter C is first found by optimising classification performance on a held-out validation set (10% of the training dataset), and then the SVM is trained on all of the training data.”

May combine two claims.

The cat and dog rank changes were small relative to the person change and do not warrant a major reassessment of their difficulty.

“However, this is a relatively small change in rankings compared to that of the 'person' class and should not cause us to substantially re-evaluate the difficulty of these two classes.”

• AssumptionThe authors state this as an assumption: The number of possible object locations in an image is very large relative to differences in positive counts between classes.
• AssumptionThe authors state this as an assumption: The number of negative examples is roughly equal across classes even when positive counts differ.

Probably not a substantial claim: may not pass S3 (load-bearing).

For weight inspection, the C parameter was selected by cross-validation on 10% of the data.

“As before, an optimal value for the C parameter is found via cross-validation on 10% of the data.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Co-locating annotators enabled efficient discussion and flexible changes in priorities and training.

“This enabled efficient discussion, e.g. of challenging or unusual cases, and was very flexible in allowing changes of priorities and training.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The 2011 University of Amsterdam entry used classification-by-detection with candidate windows based on multiple segmentations instead of sliding windows.

“The 2011 entry from the University of Amsterdam used classification-by-detection, but instead of using sliding windows they proposed candidate windows based on multiple segmentations (Van de Sande et al, 2011).”

• cites Van de Sande et al, 2011Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

The same two-fold evaluation was applied to individual methods to make the comparison fair.

“To ensure a fair comparison, the same two-fold evaluation is also computed for the individual methods.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Experienced annotators were always included at each remote site to train novices.

“In addition, some experienced annotators were always included at each remote site to train novices.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Spatial pyramid matching was proposed as a way beyond simple bag-of-words by using histograms over regions of an image.

“One idea to go beyond the simple bag-of-words (BOW) representation was to use histograms on regions of the image, e.g.spatial pyramid matching (SPM, Lazebnik et al 2006), where representations for a nested pyramid of regions are computed.”

• cites Lazebnik et al 2006Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The VOC 2012 results provide a snapshot of the state of the art in the final year of the challenge.

“In this section we review the results of the VOC 2012 challenge to give a snapshot of the state-of-the-art in the final year of the challenge.”

• AssumptionRests on an assumption: The submitted methods are representative of the state of the art in 2012.

May combine two claims.

During the period, a new class of object category detector, the DPM of Felzenszwalb et al (2010), was developed and its open source code was provided to the community.

“First, has been the development of a new class of object category detector, the DPM of Felzenszwalb et al (2010), with open source code provided to the community.”

• cites Felzenszwalb et al (2010) sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Annotation was carried out according to guidelines that were distributed to all annotators.

“Annotation was performed according to a set of guidelines distributed to all annotators.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Segmentations are annotated at both the object and object class level.

“Segmentations are annotated at the object and object class level.”

May combine two claims.

These metric changes may have reduced but did not eliminate overfitting to the selected evaluation metrics.

“These changes may have reduced but not eliminated another type of reduction in diversity: that methods may end up overfitting to the particular evaluation metrics selected for each competition.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

The Pascal VOC challenges and workshops contributed to the surge in interest in category recognition in computer vision over the last decade and were mentioned in thousands of papers.

“The Pascal VOC challenges and workshops have certainly contributed to the surge in interest in category recognition in the computer vision community over the last decade, and have been mentioned in thousands of papers.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Images for the action classification task have been partially annotated with people, bounding boxes, reference points, and their actions.

“They have been partially annotated with people, bounding boxes, reference points and their actions.”

May combine two claims.

Adding new images each year while maintaining earlier splits mitigated overfitting and allowed progress from 2008 to 2012 to be measured on the same test set.

“This had a number of useful consequences: first, it has mitigated overfitting to the data (which might have happened if the data had not changed from year to year); and second, since the earlier year's data is available as subsets, progress can be measured from 2008 to 2012 using the same test set.”

Probably not a substantial claim: may not pass S3 (load-bearing).

A subset of images is additionally annotated with pixel-wise segmentation of each object present, to support the segmentation competition.

“A subset of images are also annotated with pixel-wise segmentation of each object present, to support the segmentation competition.”

May combine two claims.

In 2012, only three organisations participated in segmentation, suggesting that the methods were performing so well that a new participant would find it difficult to enter with an original method and get competitive results.

“In 2012 the number of participating organisations was down to three, suggesting that the methods used were performing so well it was difficult for a new participant to enter the competition with an original method and get competitive results.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Annotation for the action classification task was performed according to guidelines distributed to all annotators.

“Annotation was performed according to a set of guidelines distributed to all annotators.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The images used for the action classification task are disjoint from those used for the classification, detection, and segmentation tasks.

“Images for the action classification task are disjoint from those of the classification, detection and segmentation tasks.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Images for the person layout taster, whose test set is disjoint from the main tasks, have been additionally annotated with parts of the people (head, hands, and feet).

“Images for the person layout taster, where the test set is disjoint from the main tasks, have been additionally annotated with parts of the people (head, hands and feet).”

Probably not a substantial claim: may not pass S3 (load-bearing).

Because test data are images, there remains a danger that parameter choices will be assessed by eye on the test data.

“Since the data consists of images, there is still some danger that performance of different parameter choices will be assessed by eye on the test data.”

May combine two claims.

The 2009 dataset is a subset of the 2010, 2011, and 2012 datasets, and because training, validation, and test assignments were maintained, all methods could be measured each year using the same 2009 test data.

“For example, the 2009 dataset is a subset of 2010, 2011 and 2012, and as the assignments to training, validation and test are maintained, performance of all methods can be measured each year using the same 2009 test data (albeit with different training and validation data each year).”

Probably not a substantial claim: may not pass S3 (load-bearing).

In the Pascal dataset, each image has an annotation file that provides a bounding box and object class label for every object belonging to one of the twenty classes present in the image.

“Each image has an annotation file giving a bounding box and object class label for each object in one of the twenty classes present in the image.”

Probably not a substantial claim: may not pass S3 (load-bearing).

When multiple submissions were made, participants had to select a single submission without knowing the test scores.

“Here it should be added that in case of multiple submissions, participants were forced to select a single submission without knowing the respective test scores.”