Language-Based Augmentation to Address Shortcut Learning in Object-Goal Navigation

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Large language models trained on massive text corpora also exhibit shortcut learning, mostly from artifacts in their training data.

“Recent works show that even Large-Language Models (LLMs), pre-trained on text amounting to billions of words, suffer from shortcut learning, largely due to collection artifacts in training data [7], [21].”

• cites [7] sourcePartly supported

  Section 5, “Shortcut learning across deep learning” (page unavailable), says that prominent language models such as BERT and GPT-2 remain susceptible to shortcut learning and identifies dataset biases, including annotation artefacts, as sources of NLP shortcuts. It does not establish that the models were pretrained on billions of words or that collection artefacts are the predominant cause. (LLM, conf=0.84)

  Experimental — accuracy not yet measured
• cites [21] sourceNot checked — no full text of the source

Classified as probably not a substantial claim. May combine two claims.

EmbCLIP comes in two variants, a closed-world one that assumes known target objects and an open-world one, and both encode RGB egocentric views with a frozen CLIP image encoder with a ResNet-50 backbone.

“which has two different variations: a closed-world architecture, which assumes known target objects, and an open-world variant.Both encode RGB egocentric views using a frozen CLIP image encoder with a ResNet-50 backbone.”

• cites [1] sourcePartly supported

  Section 3 (p. 3) describes an ObjectNav baseline with a known goal category and a frozen CLIP ResNet-50 encoder for RGB images. The Introduction (p. 2) and Discussion (p. 8) describe a zero-shot extension that navigates to object targets unseen during training. The paper presents EmbCLIP as a series of baselines across several tasks, however, rather than defining exactly two closed-world and open-world variants; the retrieved text also does not explicitly establish that the zero-shot extension freezes its image encoder. (LLM, conf=0.82)

  Experimental — accuracy not yet measured

May combine two claims. The quote only partly states this claim.

Changing wall colors reduces the SOTA ObjectNav method's success rate by 69%, whereas the proposed L-B augmentation drops by only 23%.

“Where the SOTA ObjectNav method's success rate drops 69%, our proposal has only a drop of 23%.”

• cites [1] sourceCould not verify

  Section 4.1 and Table 1 (pp. 3–4) report ObjectNav success rates for CLIP and other agents, but not results after changing wall colors. Searches for wall-color changes, L-B augmentation, and the 69% and 23% drops found no passage in our copy addressing that comparison. (LLM, conf=0.75)

  Experimental — accuracy not yet measured

May combine two claims.

EmbCLIP's closed-world architecture performs better than its open-world variant in a closed-world setting, which is why it is used for the out-of-distribution generalization test.

“For our o.o.d.generalization test, we adopt Emb-CLIP's closed-world architecture given its better performance in a closed-world setting.”

• cites [1] sourceSupported
  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

ProcTHOR supports randomization of wall, ceiling, floor, and object textures and colors.

“Similarly, ProcTHOR [10] allows for randomizing e.g., textures and colors of walls, ceilings, floors and objects.”

• cites [10] Deitke, 2022, ProcTHOR: Largescale embodied AI using procedural generation, Advances in Neural Information Processing Systems sourceSupported

May combine two claims.

Advanced techniques that edit individual training frames, such as text-to-image models, are slow, costly, and error-prone.

“While more sophisticated methods for partially editing individual frames during training exist (e.g., text-to-image models [13], [14]), they are slow, computationally expensive and error-prone.”

• cites [13] Ramesh, 2022, Hierarchical text-conditional image generation with clip latents, ArXiv sourcePartly supported

  The Introduction identifies unCLIP as a text-to-image system, and §3.3 (p. 7) describes language-guided manipulation of existing images. §7 (p. 17) reports difficulties with rendered text and complex-scene details, and says improving the latter would require more training and inference compute. The paper does not establish that editing individual training frames with text-to-image models is generally slow, costly, and error-prone. (LLM, conf=0.84)

  Experimental — accuracy not yet measured
• cites [14] Ramesh, Jul 2021, Zero-shot text-to-image generation sourcePartly supported

  The Method (p. 2) confirms that this is a text-to-image model, and Training (p. 14) describes substantial computational demands. The Introduction (p. 1) notes that generated images can have artifacts. But the paper does not establish that the model is slow or used to edit individual training frames; its discussion of image-to-image translation says that capability emerged without changes to training (Section G, p. 20). (LLM, conf=0.86)

  Experimental — accuracy not yet measured

May combine two claims.

With the L-B augmentation, changing the wall colors of the target room incurs only a 23% relative drop in success (39% to 30%) and a 29% drop in SPL (0.17 to 0.12).

“When changing wall colors of the target room (1 wall color change), our method incurs only a 23% relative drop in success (39% → 30%) and 29% drop in SPL (0.17 → 0.12).”

• Also stated in: § VII. CONCLUSION AND LIMITATIONS · plain

  “When changing the target object's room, our method incurs a 23% relative drop in success rate whilst the SOTA ObjectNav method's success rate drops 69%.”

• AssumptionRests on an assumption: Differences between CLIP text embeddings of wall-color descriptions correspond to the visual variation of wall colors in the image embedding space, so adding such differences to image embeddings yields meaningful augmented visual representations.

Probably not a substantial claim: may not pass S3 (load-bearing).

Adding the L-B augmentation increases training time only marginally, from 88 to 90 GPU-hours for 30M steps.

“Note that training time is only marginally longer: from 88 to 90 GPU-hours (both 30M steps).”

• AssumptionRests on an assumption: Differences between CLIP text embeddings of wall-color descriptions correspond to the visual variation of wall colors in the image embedding space, so adding such differences to image embeddings yields meaningful augmented visual representations.

May combine two claims.

Going from zero to one wall color change, EmbCLIP shows a 67% relative drop in SPL (0.39 to 0.13) and a 56% relative drop in success rate (68% to 30%).

“On average (blue mean bar), we observe a 67% relative drop in SPL (0.39 → 0.13) and 56% in success rate (68% → 30%) going from 0 wall color changes to 1 wall color change, with even lower mean performance for more wall color changes.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Feature standardization before the RNN is used to maintain training stability.

“We standardize features before feeding into the RNN to ensure stability during training (as some features might dominate the loss due to large norms).”

May combine two claims.

When training data are deliberately limited, changing wall colors only in the testing scenes decreases performance significantly, and the root cause is the deceptive wall color changes.

“We found that, when deliberately limiting training data, only changing wall colors in testing scenes decreases performance significantly, with the root cause being the deceptive wall color changes.”

• AssumptionRests on an assumption: Wall color is a representative instance of shortcut bias, so results obtained on it extend to shortcut learning in ObjectNav more generally.

May combine two claims.

The proposed L-B augmentation modifies visual representations at the feature level, avoiding frame edits and simulator changes.

“We augment directly at feature-level, without editing individual frames or any changes to the simulator.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Human observers may struggle to notice shortcut biases, such as superficial training-data statistics like specific texture frequencies in image classification.

“Moreover, shortcut bias might be difficult to observe for humans e.g., superficial statistics in training data such as textures of specific frequencies in image classification tasks [22].”

• cites [22] Wang, 2022, Frequency shortcut learning in neural networks sourceNot checked — no full text of the source

May combine two claims.

When wall colors are changed only nondeceptively, the agent explores the entire scene, which produces longer episodes.

“In contrast, an agent will explore the entire scene when wall colors have only been changed nondeceptively, leading to longer episodes.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Each test set comprises 1080 evaluation episodes evenly distributed over the 5 scene layouts, the possible wall color permutations and the target objects.

“We run 1080 evaluation episodes per test set to evenly distribute episodes over the 5 layouts, possible wall color permutations and target objects.”

May combine two claims.

Domain-randomization augmentations may be unavailable in some simulators and are frequently hard to add after training.

“While ProcTHOR shows incredibly powerful results, such augmentations might not be available for all simulators, and more often than not, difficult to apply post-hoc.”

May combine two claims.

Domain randomization approaches need simulator modifications, which can be inflexible or hard to alter.

“Domain randomization methods e.g., randomizing textures, colors and shapes of objects or environments are commonly used to transfer policies in DRL [8]- [10].However, these methods specifically require changes to the simulator, which might be inflexible or difficult to modify e.g., high-fidelity simulators with training data reconstructed from real-world 3D scans [11], [12].”

• cites [8]-[10]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [11] sourceNot checked — no full text of the source
• cites [12] Ramakrishnan, Habitat-matterport 3d dataset (HM3d): 1000 large-scale 3d environments for embodied AINot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

The proposed L-B augmentation increases domain generalization by augmenting the agent's visual representations at feature-level, making them more invariant to changing environments.

“We increase domain generalization by augmenting the agent's visual representations at feature-level, such that these are more invariant to changing environments.”

May combine two claims.

The proposed method can be added after training because it needs no modifications to data or simulator.

“Contrary, our method can readily be applied post-hoc as it requires no changes to training data or the simulator.”

Classified as probably not a substantial claim.

Conventional domain randomization is inconvenient for more intricate biases (e.g., a pen always on a desk).

“Although a simple wall color bias might be addressed using conventional domain randomization, these methods are inconvenient considering more intricate biases (e.g., a pen is always on a desk).”

May combine two claims.

The method uses vision-language representations to perform feature-level augmentation without simulator changes.

“We leverage the vision-language representations for feature-level augmentations, without the need to modify the simulator.”

Probably not a substantial claim: may not pass S1 (asserted). May combine two claims. The quote only partly states this claim.

Under deceptive changes the agent navigates directly towards the learned color of the target room, which now sits in a non-target room, without exploring other rooms, and terminates the episode after failing to find the target object there.

“We conjecture that due to deceptive changes, the agent directly navigates towards the learned color of the target room, which is now placed in a nontarget room, without exploring any other rooms.”

May combine two claims.

The proposed method uses free-form natural language, making it easy to adapt to various biases.

“In contrast, our method utilizes free-form natural language, which allows for easily adapting to different biases.”

May combine two claims.

The ProcTHOR-10k interventions restrict scene variation to wall colors alone, the sole factor affecting performance.

“These interventions limit the house variations to just wall colors, which is the only aspect influencing the performance.”

May combine two claims.

Shortcut learning is the root cause of the poor generalization: the agent's navigation to target objects is driven by searching for the wall color linked to the target object's room.

“We find that shortcut learning is the root cause: the agent learns to navigate to target objects, by simply searching for the associated wall color of the target object's room.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Validation was performed every 200,000 frames and the reported results come from the checkpoint with the highest SPL.

“We perform validation every 200,000 frames and report results of the checkpoint with highest SPL.”