CoChat
Sign Up
PreprintCited by 9Reinforcement Learning in RoboticsMachine Learning and AlgorithmsAdaptive Dynamic Programming Control

The DOI matches its DataCite record.

Causal Imitation Learning under Temporally Correlated Noise

Open access
Swamy, Gokul, Choudhury, Sanjiban, Bagnell, J. Andrew, Wu, Zhiwei Steven·2022·arXiv
DOI 10.48550/arxiv.2202.01312arXiv 2202.01312
Open this paper in CoChat — free accountOpen PDF

Abstract

We develop algorithms for imitation learning from policy data that was corrupted by temporally correlated noise in expert actions. When noise affects multiple timesteps of recorded data, it can manifest as spurious correlations between states and actions that a learner might latch on to, leading to poor policy performance. To break up these spurious correlations, we apply modern variants of the instrumental variable regression (IVR) technique of econometrics, enabling us to recover the underlying policy without requiring access to an interactive expert. In particular, we present two techniques, one of a generative-modeling flavor (DoubIL) that can utilize access to a simulator, and one of a game-theoretic flavor (ResiduIL) that can be run entirely offline. We find both of our algorithms compare favorably to behavioral cloning on simulated control tasks.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2202.01312

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Predicting action sequences helps tackle temporally correlated confounders such as pauses in demonstrations.

    “Predicting action sequences also helps tackle temporally correlated confounders [65], such as pauses in demonstrations that are hard to model with Markovian single-step policies.”

    From Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware (2023) · cites this paper as [65]

    The Introduction (p. 2) says the paper develops algorithms to address temporally correlated confounding in imitation learning. But it describes instrumental-variable methods using past states, not predicting action sequences; the cited passages also do not identify pauses in demonstrations as an example (Introduction, p. 1; Section 4, p. 5). (LLM, conf=0.86)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 23

  1. Angrist, J. D., Imbens, G. W., and Rubin, D. B. Identification of causal effects using instrumental variables. Journal of the American statistical Association , 91(434): 444-455, 1996.
  2. Baird, L. Residual algorithms: Reinforcement learning with function approximation. In Machine Learning Proceedings 1995 , pp. 30-37. Elsevier, 1995.
  3. Bansal, M., Krizhevsky, A., and Ogale, A. S. Chauffeurnet: Learning to drive by imitating the best and synthesizing the worst. CoRR , abs/1812.03079, 2018. URL http: //arxiv.org/abs/1812.03079 . · arXiv
  4. Bassily, R., Nissim, K., Smith, A., Steinke, T., Stemmer, U., and Ullman, J. Algorithmic stability for adaptive data analysis. SIAM Journal on Computing , (0):STOC16-377, 2021.
  5. Bradtke, S. J. and Barto, A. G. Linear least-squares algorithms for temporal difference learning. Machine learning , 22(1):33-57, 1996.
  6. Chen, X. and Pouzo, D. Estimation of nonparametric conditional moment models with possibly nonsmooth generalized residuals. Econometrica , 80(1):277-321, 2012.
  7. Codevilla, F., Santana, E., L´ opez, A. M., and Gaidon, A. Exploring the limitations of behavior cloning for autonomous driving. CoRR , abs/1904.08980, 2019. URL http://arxiv.org/abs/1904.08980 . · arXiv
  8. de Haan, P., Jayaraman, D., and Levine, S. Causal confusion in imitation learning. Advances in Neural Information Processing Systems , 32:11698-11709, 2019.
  9. Desautels, T., Das, R., Calvert, J., Trivedi, M., Summers, C., Wales, D. J., and Ercole, A. Prediction of early unplanned intensive care unit readmission in a uk tertiary care hospital: a cross-sectional machine learning approach. BMJ open , 7(9):e017199, 2017.
  10. Freund, Y. and Schapire, R. E. A decision-theoretic generalization of on-line learning and an application to boosting. Journal of computer and system sciences , 55(1):119-139, 1997.
  11. Hartford, J., Lewis, G., Leyton-Brown, K., and Taddy, M. Deep iv: A flexible approach for counterfactual prediction. In International Conference on Machine Learning , pp. 1414-1423. PMLR, 2017.
  12. Hefny, A., Downey, C., and Gordon, G. J. Supervised learning for dynamical system learning. Advances in neural information processing systems , 28:1963-1971, 2015.
  13. Kakade, S. and Langford, J. Approximately optimal approximate reinforcement learning. In IN PROC. 19TH INTERNATIONAL CONFERENCE ON MACHINE LEARNING , pp. 267-274, 2002.
  14. Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014. · arXiv
  15. Kuefler, A., Morton, J., Wheeler, T., and Kochenderfer, M. Imitating driver behavior with generative adversarial networks. In 2017 IEEE Intelligent Vehicles Symposium (IV) , pp. 204-211. IEEE, 2017.
  16. Muller, U., Ben, J., Cosatto, E., Flepp, B., and Cun, Y. L. Off-road obstacle avoidance through end-to-end learning. In Advances in neural information processing systems , pp. 739-746. Citeseer, 2006.
  17. Ng, A. Y., Kim, H. J., Jordan, M. I., Sastry, S., and Ballianda, S. Autonomous helicopter flight via reinforcement learning. In NIPS , volume 16. Citeseer, 2003.
  18. Pearl, J. Causal diagrams for empirical research. Biometrika , 82(4):669-688, 1995.
  19. Pearl, J., Glymour, M., and Jewell, N. P. Causal inference in statistics: A primer . John Wiley & Sons, 2016.
  20. Soo, A., Zuege, D. J., Fick, G. H., Niven, D. J., Berthiaume, L. R., Stelfox, H. T., and Doig, C. J. Describing organ dysfunction in the intensive care unit: a cohort study of 20,000 patients. Critical Care , 23(1):1-15, 2019.
  21. Syrgkanis, V., Agarwal, A., Luo, H., and Schapire, R. E. Fast convergence of regularized learning in games. In Cortes, C., Lawrence, N., Lee, D., Sugiyama, M., and Garnett, R. (eds.), Advances in Neural Information Processing Systems , volume 28. Curran Associates, Inc., 2015. URL https://proceedings. neurips.cc/paper/2015/file/ 7fea637fd6d02b8f0adf6f7dc36aed93-Paper. pdf .
  22. Winship, C. and Morgan, S. L. The estimation of causal effects from observational data. Annual review of sociology , 25(1):659-706, 1999.
  23. Zhang, J., Kumor, D., and Bareinboim, E. Causal imitation learning with unobserved confounders. Advances in Neural Information Processing Systems , 33, 2020.

Cited by

Sorted by citations

Metadata

Authors
Swamy, G., Choudhury, S., Bagnell, J. A., & Wu, Z. S.
Affiliations
  • Carnegie Mellon University (US)
  • Cornell University (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2022
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2202.01312
arXiv
2202.01312
OpenAlex
W4225894963
Cited by
9
References
0
Citations by year
2025: 3 · 2024: 3 · 2023: 3
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by-sa/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
National Science Foundation (1939606)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
  • Physical Sciences › Engineering › Control and Systems Engineering › Adaptive Dynamic Programming Control
Keywords
imitation learning, temporally correlated noise, instrumental variable regression, behavior cloning, offline learning, generative models, game-theoretic learning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review