Novelty Detection in Reinforcement Learning with World Models

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

RIQN is an ensemble-based method that detects novelty by predicting action values from state-action pairs and predicts feature distribution at each time step based on previous feature values.

“RIQN is an ensemble based method that detects novelty by predicting action values from state-action pairs as input, and predicts the feature distribution at each time step based on the previous values of the features.”

• cites Danesh & Fern, 2021 sourceDisputed

  Section 3.1 explicitly says RIQN predicts feature distributions from previous feature values *rather than* predicting action values from state-action pairs. Section 3.2 supports the ensemble-based anomaly-detection aspect, but the claim's description of RIQN's inputs and outputs contradicts the paper. (LLM, conf=0.90)

  Checked by a reasoning model — may be wrong.

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Reinforcement learning with world models has recently achieved notable success because it is sample-efficient and can incorporate Markov decision process techniques.

“Reinforcement learning (RL) using world models has found significant recent successes due to its strength in sampling efficiency and ability to incorporate well-studied Markov Decision Process techniques (Moerland et al., 2020;Robine et al., 2021).”

• cites Moerland et al., 2020 sourcePartly supported

  The Introduction (p. 2) says model-based RL has shown great success and defines it as combining planning and learning for MDP optimization. Section 7.1 (p. 35) describes using planning to reduce real-world sample complexity. The paper does not establish that these two strengths caused its recent successes, as the citing sentence claims. (LLM, conf=0.82)

  Experimental — accuracy not yet measured
• cites Robine et al., 2021Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

RIQN computes an anomaly score through the average L1 distance for each observed feature between ensemble samples and the actual observation at time t.

“Given these samples an anomaly score is computed through the average L1 distance for each observed feature between the samples and the actual observation at time t.”

• cites Danesh & Fern, 2021 sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

RIQN's high false positive rate was also observed by Danesh & Fern (2021).

“RIQN's high false positive rate was also observed by Danesh & Fern (2021).”

• cites Danesh & Fern, 2021 sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Learning an OOD model is difficult and may be impossible in certain situations.

“Recent research shows that learning an OOD model is a difficult problem to solve and may even prove to be impossible in certain situations (Zhang et al., 2021;Fang et al., 2022).”

• cites Zhang et al., 2021 sourceSupported
• cites Fang et al., 2022 sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The RIQN framework has shown success by focusing on aleatoric uncertainty and draws upon generated distributions of Implicit Quantile networks.

“The RIQN (Danesh & Fern, 2021) framework has shown to be successful by focusing on the aleoric uncertainty of the agent and draws upon the generated distributions of Implicit Quartille networks (Dabney et al., 2018).”

• cites Danesh & Fern, 2021 sourceSupported
• cites Dabney et al., 2018 sourceNot checked — no full text of the source

May combine two claims.

RIQN is sensitive to the selection of threshold λ and drift ∆, which can vary by environment and must be tuned, requiring a priori belief about anomaly shift sizes.

“However, RIQN is also sensitive to the selection of a threshold λ, and drift ∆ which can vary from environment to environment and must be tuned; which requires a priori belief of whether anomalies will result in large or small shifts in aleoric uncertainty.”

• cites Danesh & Fern, 2021 sourcePartly supported

  Section 3.2 (page unavailable) specifies a CUSUM threshold and drift and explains their roles; Section 4.3 says further CUSUM tuning may improve performance. The Impact of Horizon section shows that performance varies with Δ, but defines Δ as the detection horizon, not drift. The paper does not say these parameters must be tuned for each environment or that doing so requires a prior belief about shifts in aleatoric uncertainty. (LLM, conf=0.86)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Some methods using reward signal deterioration to detect novelty can be dangerous in high-stakes environments where it may be too late to recover, and may incorrectly map the novelty causing reward shift.

“Some methods attempt to use a measure of reward signal deterioration as a way to detect if a novelty has occurred, but that can prove to be dangerous in a high stakes environment where it may be too late to recover from a negative signal (Greenberg & Mannor, 2020).This method may also prove to have an incorrect mapping of the novelty that caused the reward shift.”

• cites Greenberg & Mannor, 2020 sourcePartly supported

  The Introduction (p. 1) describes detecting reward deterioration after environmental changes and stresses that critical failures must be detected quickly in risk-intolerant applications. It does not say that reward-based detection is itself dangerous because recovery may already be too late, or that it incorrectly identifies the novelty responsible for a reward shift. (LLM, conf=0.85)

  Experimental — accuracy not yet measured

May combine two claims.

For MiniGrid-DoorKey-6x6, the false positive rate is at most 10^-2 for the KL bound, .03 for PP-Mare, and .52 for RIQN.

“The false positive rates for Minigrid-DoorKey-6x6 is ≤ 10 -2 , .03, and .52 for the KL bound method, PP-Mare, and RIQN.”

May combine two claims.

The KL computation increases speed performance in Atari and DMC by factors of 10^2 and 10^3 respectively.

“We observe that the KL computation increases the speed performance in both the Atari and DMC domains, by a magnitude of 10 2 and 10 3 respectively.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The initial interpretation of KL for the IRIS world model had moderate success.

“Although our initial interpretation had moderate success, we expect that future work should be able to improve by adjusting the bound as well as addressing the exploration concerns given the exploration trick used in (Micheli et al., 2023), of which the”

May combine two claims.

The KL bound works with transformer-based world models and has very low false-positive rates.

“The KL bound works with transformer-based world models and has very low false-positive rates as consistent with the bound on other architectures.”

May combine two claims.

As training progresses, the model eventually discovers that incorporating history h_t further drives loss decrease, and the KL divergence with the history dropout rises.

“At some point the model discovers that incorporating history h t further drives loss decrease, and the KL divergence with the history dropout rises.”

May combine two claims.

Initially the difference of KL divergences is below the surprise line, and all new stimuli are flagged as novel, but the world model quickly learns the environment is predictable.

“Initially, the difference of KL divergences (blue) is below the orange line, and all new stimuli are flagged as novel, but the world model quickly learns that the environment is predictable.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In the nominal environment, the right-hand side of the bound is always higher than the left-hand side of the bound.

“The significant observation is that in the nominal environment, the right-hand side of the bound is always higher than the left-hand side of the bound.”

May combine two claims.

The transformer-based IRIS is slower to build evidence of the novelty, consistent with observations by Micheli et al. (2023) about under-sampled states on Atari-Freeway.

“The transformer-based IRIS (Micheli et al., 2023) is slower to build evidence of the novelty, which is consistent with observations by Micheli et al. (2023) (Appendix B) about under-sampled states on the Atari-Freeway environment; the effect on novelty detection is described further in our Appendix C.2.”

• cites Micheli et al., 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Anomaly detection drops rapidly or was never high to begin with.

“Anomaly detection drops rapidly or was never high to begin with.”

May combine two claims.

In contrast to RIQN, the anomaly score fluctuates and is higher for all environments except one.

“This is in contrast to RIQN, where the anomaly score fluctuates and is higher for all environments except one.”

May combine two claims.

Training the agent from both sides of the map before novelty (KLExplored) increases novelty detection accuracy because the transition is no longer flagged.

“To illustrate that it is an artifact of how the agent is learned, we provide an alternative training paradigm, KLExplored where we train the agent from both sides of the map prenovelty.Table 8 shows that this increases novelty detection accuracy because this transition is no longer flagged.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The most common transition initially flagged is from left to right.

“The most common transition initially flagged is from left to right.”

May combine two claims.

When a novelty occurs, the agent's converged policy becomes unreliable and the agent may make catastrophic mistakes.

“In cases where there is a novel change, the RL agent's converged policy is no longer reliable, and the agent can make catastrophic mistakes.”

May combine two claims.

The world model is initially reliant on ground truth x_t rather than noisy h_t, which stabilizes p(z|h,x) for proper training until the model understands the role of history h_t.

“A direct consequence of using common KL balancing techniques such as Equation ( 2) to bias the loss towards the prior error is that the world model is initially reliant on ground truth x t in contrast to a noisy h t .This stabilizes p ϕ (z t |h t , x t ) for proper training of p ϕ (z t |h t ) and p ϕ ( xt |h t , z t ) (Asperti & Trentin, 2020) until the model understands the role of history h t .”

• cites Asperti & Trentin, 2020Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Leveraging learned surrogates conditioned on the agent's actions appears to provide more informative signals of normality in higher-dimensional spaces, enabling the detection of novelties with minimal delay error.

“It appears that leveraging learned surrogates conditioned on the agent's actions appears to provide more informative signals of normality in higher-dimensional spaces, enabling the detection of novelties with minimal delay error.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The bound might be violated not only due to novelty but potentially due to the inherent non-metric behavior of KL divergence, particularly in high-dimensional or complex latent spaces.

“This means our bound might be violated not only due to novelty but potentially due to the inherent non-metric behavior of KL divergence, particularly in high-dimensional or complex latent spaces.”

May combine two claims.

The proposed method detects novelty without manually setting thresholds and without extra augmented data.

“We introduce a novel technique for novelty detection without the need to manually specify thresholds, and without the need for additional augmented data.”

May combine two claims.

Failure to make h_t and x_t useful may result in x_t being interpreted as a noisy signal or the transition prediction model with h_t not yet meeting sufficiently low KL divergence.

“Figure 1 illustrates that failure to do so may result in x t being interpreted as a noisy signal (blue differences collapse to orange) or the transition prediction model with h t has not yet met sufficiently low KL divergence (orange exceeds blue).”

May combine two claims.

The KL bound method is resilient to false positives while detecting novelty quickly and accurately.

“The KL bound method we introduce is demonstrably resilient to false positives while simultaneously detecting novelty quickly and accurately.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

There may not be sufficient representation of the agent's evaluation environment to detect novel OOD transitions.

“Furthermore, there may not be sufficient representation of the agent's evaluation environment to be able to detect novel Out-of-Distribution (OOD) transitions.”

Probably not a substantial claim: may not pass S2 (checkable).

The method appears robust enough to provide a basis for future research on inference-time novelties.

“Our method appears robust enough to provided the basis for future research on addressing inference-time novelties such as those that occur in the real world.”

May combine two claims.

This transition was possible during training of the world model, but it was illogical to consistently explore looking backwards and the policy quickly learned to discourage that behavior, depriving the world model of observations that should have been able to occur naturally.

“This transition was possible during the training of the world model, but it was illogical to consistently explore looking backwards and the policy quickly learned to discourage that behavior, depriving the world model of observations that should have been able to occur naturally.”

May combine two claims.

Novelty detection in RL needs to consider current context; assuming i.i.d. overlooks dependencies and contextual information crucial for accurate detection.

“In addition, novelty detection techniques in the RL domain need to consider the current context of the situation.Not considering context becomes problematic when assuming that data is independently and identically distributed (i.i.d.), as it overlooks dependencies and contextual information that are often crucial for accurate detection.”

May combine two claims.

The paper's primary result is built on the Dreamer world model, but the novelty bound technique can be applied to other types of world models with similar success.

“While our primary result is built on top of the Dreamer world model (Hafner et al., 2021), we also show that our novelty bound technique can be applied to other types of world models with similar success.”

• cites Hafner et al., 2021Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Not yet classified.

Simply relying on the policy may be powerless in determining if a novel transition has occurred.

“This suggests that simply relying on the policy may be powerless in determining if a novel transition has occurred.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The bound in Proposition 3.2 compares the posterior to three different priors and provides a useful approximation for detection, rather than a strict mathematical guarantee.

“The bound in Proposition 3.2 compares the posterior to three different priors and provides a useful approximation for detection, rather than a strict mathematical guarantee.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The KL bound cannot be used with diffusion-based world models that directly generate prior and posterior images instead of latent hidden states.

“Our KL bound method cannot be used with diffusion-based worldmodels that directly generate prior and posterior images instead of latent hidden states.”

Classified as probably not a substantial claim.

A properly trained model is one in which its learned representations h_t and x_t are useful in predicting the distribution p(z_t|h_t,x_t).

“A properly trained model is one in which its learned representations, h t and x t , are useful in the prediction of distribution p ϕ (z t |h t , x t ).”

Probably not a substantial claim: may not pass S3 (load-bearing).

The anomaly scores for each feature are individually used as values for the cusum algorithm to detect disturbances.

“The anomaly scores for each feature in the observation are then individually used as values for the cusum algorithm (Page, 1954) to detect any disturbances.”

• cites Page, 1954 sourceNot checked — no full text of the source

May combine two claims.

Choosing a threshold hyperparameter for distribution shift requires at least implicit prior knowledge of anomalies, which conflicts with the assumption that novelties are unanticipatable and unencountered during training.

“Setting such a parameter requires at least implicit foreknowledge of anomalies that can occur, which violates our assumptions that novelties be unanticipatable and unencountered during training.”

• AssumptionThe authors state this as an assumption: Novelties are unanticipatable and unencountered during training.

Classified as probably not a substantial claim.

Tuning thresholds and hyperparameters requires some prior intuition about the nature, scope, and scale of anomalies, which is disallowed in the paper's novelty research setting.

“This is essential because tuning thresholds and hyperparameters requires some a priori intuition about the nature, scope, and scale of anomalies, which is an assumption that is disallowed in our research setting on novelty.”

• AssumptionThe authors state this as an assumption: Novelties are unanticipatable and unencountered during training.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Experimentation in the real world is often infeasible, requiring virtual surrogate testbeds.

“While the research challenge of novelty detection is inspired by the real world, where sudden, unanticipatable, and permanent changes to how the world works occur relatively frequently, experimentation in the real world is often infeasible, requiring virtual surrogate testbeds.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

This novelty detection is technically correct because the agent is observing aspects it failed to learn about during training.

“This is technically correct because the agent is observing aspects of the environment that it failed to learn about during training.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Experimenter bias is a risk for novelty detection research because the PP-Mare and KL-Bound methods were developed before identifying Atari and DMC as sources of environments with alternatives.

“Experimenter bias is also a risk faced by those that research novelty detection-our PP-Mare and KL-Bound methods were developed before we identified Atari and DeepMind Control Suite as sources of environments with alternatives.”

May combine two claims.

The bound combats posterior collapse and measures possible over-fitting.

“Therefore, our bound is both combating posterior collapse, i.e., p ϕ (z t |x t ) = p ϕ (z t |h 0 ), as well as measuring possible over-fitting i.e KL[p ϕ (z t |h t , x t )||p ϕ (z t |h 0 , x t )] ≫ c for some large c.”