CoChat
Sign Up
PreprintCited by 1

Out-of-Distribution Dynamics Detection: RL-Relevant Benchmarks and Results

Danesh, Mohamad H, Fern, Alan·2021·arXiv
DOI 10.48550/arxiv.2107.04982arXiv 2107.04982
Open this paper in CoChat — free account

Abstract

We study the problem of out-of-distribution dynamics (OODD) detection, which involves detecting when the dynamics of a temporal process change compared to the training-distribution dynamics. This is relevant to applications in control, reinforcement learning (RL), and multi-variate time-series, where changes to test time dynamics can impact the performance of learning controllers/predictors in unknown ways. This problem is particularly important in the context of deep RL, where learned controllers often overfit to the training environment. Currently, however, there is a lack of established OODD benchmarks for the types of environments commonly used in RL research. Our first contribution is to design a set of OODD benchmarks derived from common RL environments with varying types and intensities of OODD. Our second contribution is to design a strong OODD baseline approach based on recurrent implicit quantile network (RIQN), which monitors autoregressive prediction errors for OODD detection. In addition to RIQN, we introduce and test three other simpler baselines. Our final contribution is to evaluate our baseline approaches on the benchmarks to provide results for future comparison.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2107.04982

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • RIQN is an ensemble-based method that detects novelty by predicting action values from state-action pairs and predicts feature distribution at each time step based on previous feature values.

    “RIQN is an ensemble based method that detects novelty by predicting action values from state-action pairs as input, and predicts the feature distribution at each time step based on the previous values of the features.”

    From Novelty Detection in Reinforcement Learning with World Models (2023) · cites this paper as Danesh & Fern, 2021

    Section 3.1 explicitly says RIQN predicts feature distributions from previous feature values *rather than* predicting action values from state-action pairs. Section 3.2 supports the ensemble-based anomaly-detection aspect, but the claim's description of RIQN's inputs and outputs contradicts the paper. (LLM, conf=0.90)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

  • RIQN is sensitive to the selection of threshold λ and drift ∆, which can vary by environment and must be tuned, requiring a priori belief about anomaly shift sizes.

    “However, RIQN is also sensitive to the selection of a threshold λ, and drift ∆ which can vary from environment to environment and must be tuned; which requires a priori belief of whether anomalies will result in large or small shifts in aleoric uncertainty.”

    From Novelty Detection in Reinforcement Learning with World Models (2023) · cites this paper as Danesh & Fern, 2021

    Section 3.2 (page unavailable) specifies a CUSUM threshold and drift and explains their roles; Section 4.3 says further CUSUM tuning may improve performance. The Impact of Horizon section shows that performance varies with Δ, but defines Δ as the detection horizon, not drift. The paper does not say these parameters must be tuned for each environment or that doing so requires a prior belief about shifts in aleatoric uncertainty. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • The RIQN framework has shown success by focusing on aleatoric uncertainty and draws upon generated distributions of Implicit Quantile networks.

    “The RIQN (Danesh & Fern, 2021) framework has shown to be successful by focusing on the aleoric uncertainty of the agent and draws upon the generated distributions of Implicit Quartille networks (Dabney et al., 2018).”

    From Novelty Detection in Reinforcement Learning with World Models (2023) · cites this paper as Danesh & Fern, 2021

  • RIQN's high false positive rate was also observed by Danesh & Fern (2021).

    “RIQN's high false positive rate was also observed by Danesh & Fern (2021).”

    From Novelty Detection in Reinforcement Learning with World Models (2023) · cites this paper as Danesh & Fern, 2021

  • RIQN computes an anomaly score through the average L1 distance for each observed feature between ensemble samples and the actual observation at time t.

    “Given these samples an anomaly score is computed through the average L1 distance for each observed feature between the samples and the actual observation at time t.”

    From Novelty Detection in Reinforcement Learning with World Models (2023) · cites this paper as Danesh & Fern, 2021

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 24

  1. Abdallah, C., Dawson, D. M., Dorato, P., and Jamshidi, M. Survey of robust control for rigid robots. IEEE Control Systems Magazine, 11(2):24-30, 1991.
  2. Aggarwal, C. C. High-dimensional outlier detection: the subspace method. In Outlier Analysis, pp. 149-184. Springer, 2017.
  3. An, J. and Cho, S. Variational autoencoder based anomaly detection using reconstruction probability. Special Lec- ture on IE, 2(1):1-18, 2015.
  4. Bengio, S., Vinyals, O., Jaitly, N., and Shazeer, N. Sched- uled sampling for sequence prediction with recurrent neu- ral networks. arXiv preprint arXiv:1506.03099, 2015. · arXiv
  5. Breiman, L. Random forests. Machine learning, 45(1): 5-32, 2001.
  6. Breunig, M. M., Kriegel, H.-P., Ng, R. T., and Sander, J. Lof: identifying density-based local outliers. In Proceedings of the 2000 ACM SIGMOD international conference on Management of data, pp. 93-104, 2000.
  7. Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. Openai gym. arXiv preprint arXiv:1606.01540, 2016. · arXiv
  8. Chalapathy, R., Menon, A. K., and Chawla, S. Anomaly detection using one-class neural networks. arXiv preprint arXiv:1802.06360, 2018. · arXiv
  9. Chauhan, S. and Vig, L. Anomaly detection in ecg time signals via deep long short-term memory networks. In 2015 IEEE International Conference on Data Science and Advanced Analytics (DSAA), pp. 1-7. IEEE, 2015.
  10. Cho, K., Van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., and Bengio, Y. Learn- ing phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078, 2014. · arXiv
  11. Dabney, W., Ostrovski, G., Silver, D., and Munos, R. Im- plicit quantile networks for distributional reinforcement learning. arXiv preprint arXiv:1806.06923, 2018. · arXiv
  12. Fujimoto, S., Hoof, H., and Meger, D. Addressing function approximation error in actor-critic methods. In Interna- tional Conference on Machine Learning, pp. 1587-1596.
  13. PMLR, 2018.
  14. Ham, J., Lee, D. D., Mika, S., and Schölkopf, B. A kernel view of the dimensionality reduction of manifolds. In Proceedings of the twenty-first international conference on Machine learning, pp. 47, 2004.
  15. Liu, F. T., Ting, K. M., and Zhou, Z.-H. Isolation forest. In 2008 Eighth IEEE International Conference on Data Mining, pp. 413-422. IEEE, 2008.
  16. Malhotra, P., Vig, L., Shroff, G., and Agarwal, P. Long short term memory networks for anomaly detection in time series. In Proceedings, volume 89, pp. 89-94. Presses universitaires de Louvain, 2015.
  17. Rotman, N. H., Schapira, M., and Tamar, A. Online safety assurance for deep reinforcement learning. arXiv preprint arXiv:2010.03625, 2020. · arXiv
  18. Sadik, S. and Gruenwald, L. Research issues in outlier detection for data streams. Acm Sigkdd Explorations Newsletter, 15(1):33-40, 2014.
  19. Sage, H., De Mathelin, M., and Ostertag, E. Robust control of robot manipulators: a survey. International Journal of control, 72(16):1498-1522, 1999.
  20. Schlegl, T., Seeböck, P., Waldstein, S. M., Schmidt-Erfurth, U., and Langs, G. Unsupervised anomaly detection with generative adversarial networks to guide marker discov- ery. In International conference on information process- ing in medical imaging, pp. 146-157. Springer, 2017.
  21. Schölkopf, B., Williamson, R. C., Smola, A. J., Shawe- Taylor, J., Platt, J. C., et al. Support vector method for novelty detection. In NIPS, volume 12, pp. 582-588.
  22. Thudumu, S., Branch, P., Jin, J., and Singh, J. J. A compre- hensive survey of anomaly detection techniques for high dimensional big data. Journal of Big Data, 7(1):1-30, 2020.
  23. PMLR, 2016.
  24. Zong, B., Song, Q., Min, M. R., Cheng, W., Lumezanu, C., Cho, D., and Chen, H. Deep autoencoding gaussian mixture model for unsupervised anomaly detection. In International Conference on Learning Representations, 2018.

Cited by

Sorted by citations

Authors
Danesh, M. H., & Fern, A.
Venue
arXiv
Publisher
arXiv
Published
2021
DOI
10.48550/arxiv.2107.04982
arXiv
2107.04982
References
1
Type
Preprint
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review