The DOI matches its Crossref record; 68 of the 68 references it cites were found in a registry or an index.
Robust motion in-betweening
Open accessAbstract
In this work we present a novel, robust transition generation technique that can serve as a new tool for 3D animators, based on adversarial recurrent neural networks. The system synthesises high-quality motions that use temporally-sparse keyframes as animation constraints. This is reminiscent of the job of in-betweening in traditional animation pipelines, in which an animator draws motion frames between provided keyframes. We first show that a state-of-the-art motion prediction model cannot be easily converted into a robust transition generator when only adding conditioning information about future keyframes. To solve this problem, we then propose two novel additive embedding modifiers that are applied at each timestep to latent representations encoded inside the network's architecture. One modifier is a time-to-arrival embedding that allows variations of the transition length with a single model. The other is a scheduled target noise vector that allows the system to be robust to target distortions and to sample different transitions given fixed keyframes. To qualitatively evaluate our method, we present a custom MotionBuilder plugin that uses our trained model to perform in-betweening in production scenarios. To quantitatively evaluate performance on transitions and generalizations to longer time horizons, we present well-defined in-betweening benchmarks on a subset of the widely used Human3.6M dataset and on LaFAN1, a novel high quality motion capture dataset that is more appropriate for transition generation. We are releasing this new dataset along with this work, with accompanying code for reproducing our baseline results.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
Labaca-Castro, R. (2023) · Machine Learning under Malware Attack · doi:10.1007/978-3-658-40442-0_9 · Cited by 19,272
GAN(Generative Adversarial Nets)
(2017) · Journal of Japan Society for Fuzzy Theory and Intelligent Informatics · doi:10.3156/jsoft.29.5_177_2 · Cited by 19,270
“Least squares generative adversarial networks,”
Mao, X., Li, Q., Xie, H., Lau, R. Y. K., Wang, Z., & Smolley, S. P. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.304 · Cited by 5,437
Film: Visual reasoning with a general conditioning layer
Perez, E., Strub, F., De Vries, H., Dumoulin, V., & Courville, A. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11671 · Cited by 2,490
International Journal of Computer Vision
(2013) · International Journal of Computer Vision · doi:10.1007/11263.1573-1405 · Cited by 2,340
Generative adversarial imitation learning
Ho, J., & Ermon, S. (2016) · arXiv · doi:10.48550/arxiv.1606.03476 · Cited by 1,942
On the Convergence of Adam and Beyond
Reddi, S. J., Kale, S., & Kumar, S. (2019) · arXiv · doi:10.48550/arxiv.1904.09237 · Cited by 1,771
(2021) · Computer Graphics Forum · doi:10.1111/(issn)1467-8659 · Cited by 1,230
Structural-RNN: Deep Learning on Spatio-Temporal Graphs
Jain, A., Zamir, A. R., Savarese, S., & Saxena, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.573 · Cited by 1,133
Kovar, L., Gleicher, M., & Pighin, F. (2008) · ACM SIGGRAPH 2008 classes · doi:10.1145/1401132.1401202 · Cited by 1,052
Gaussian Process Dynamical Models for Human Motion
Wang, J. M., Fleet, D. J., & Hertzmann, A. (2008) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2007.1167 · Cited by 1,020
On Human Motion Prediction Using Recurrent Neural Networks
Martinez, J., Black, M. J., & Romero, J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.497 · Cited by 984
DeepMimic: example-guided deep reinforcement learning of physics-based character skills
Peng, X. B., Abbeel, P., Levine, S., & van de Panne, M. (2018) · ACM Transactions on Graphics · doi:10.1145/3197517.3201311 · Cited by 880
Interactive control of avatars animated with human motion data
Lee, J., Chai, J., Reitsma, P. S. A., Hodgins, J. K., & Pollard, N. S. (2002) · Proceedings of the 29th annual conference on Computer graphics and interactive techniques · doi:10.1145/566570.566607 · Cited by 864
Recurrent Network Models for Human Dynamics
Fragkiadaki, K., Levine, S., Felsen, P., & Malik, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.494 · Cited by 860
A Learned Representation For Artistic Style
Dumoulin, V., Shlens, J., & Kudlur, M. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.07629 · Cited by 763
Witkin, A., & Kass, M. (1988) · ACM SIGGRAPH Computer Graphics · doi:10.1145/378456.378507 · Cited by 712
Modeling Human Motion Using Binary Latent Variables
Taylor, G. W., Hinton, G. E., & Roweis, S. T. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0173 · Cited by 689
A deep learning framework for character motion synthesis and editing
Holden, D., Saito, J., & Komura, T. (2016) · ACM Transactions on Graphics · doi:10.1145/2897824.2925975 · Cited by 655
Phase-functioned neural networks for character control
Holden, D., Komura, T., & Saito, J. (2017) · ACM Transactions on Graphics · doi:10.1145/3072959.3073663 · Cited by 546
Interactive motion generation from examples
Arikan, O., & Forsyth, D. A. (2002) · Proceedings of the 29th annual conference on Computer graphics and interactive techniques · doi:10.1145/566570.566606 · Cited by 539
peng (2017) · doi:10.1145/3072959.3073602 · Cited by 502
Interactive control of avatars animated with human motion data
Lee, J., Chai, J., Reitsma, P. S. A., Hodgins, J. K., & Pollard, N. S. (2002) · ACM Transactions on Graphics · doi:10.1145/566654.566607 · Cited by 467
Performance animation from low-dimensional control signals
Chai, J., & Hodgins, J. K. (2005) · ACM Transactions on Graphics · doi:10.1145/1073204.1073248 · Cited by 342
SIMBICON: simple biped locomotion control
Yin, K., Loken, K., & van de Panne, M. (2007) · ACM SIGGRAPH 2007 papers · doi:10.1145/1275808.1276509 · Cited by 328
HP-GAN: Probabilistic 3D Human Motion Prediction via GAN
Barsoum, E., Kender, J., & Liu, Z. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2018.00191 · Cited by 322
Efficient generation of motion transitions using spacetime constraints
Rose, C., Guenter, B., Bodenheimer, B., & Cohen, M. F. (1996) · Proceedings of the 23rd annual conference on Computer graphics and interactive techniques · doi:10.1145/237170.237229 · Cited by 321
Neural state machine for character-scene interactions
Starke, S., Zhang, H., Komura, T., & Saito, J. (2019) · ACM Transactions on Graphics · doi:10.1145/3355089.3356505 · Cited by 290
Mode-adaptive neural networks for quadruped motion control
Zhang, H., Starke, S., Komura, T., & Saito, J. (2018) · ACM Transactions on Graphics · doi:10.1145/3197517.3201366 · Cited by 277
Adversarial Geometry-Aware Human Motion Prediction
Gui, L.-Y., Wang, Y.-X., Liang, X., & Moura, J. M. F. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01225-0_48 · Cited by 276
Learning motion manifolds with convolutional autoencoders
Holden, D., Saito, J., Komura, T., & Joyce, T. (2015) · SIGGRAPH Asia 2015 Technical Briefs · doi:10.1145/2820903.2820918 · Cited by 263
Witkin, A., & Kass, M. (1988) · Proceedings of the 15th annual conference on Computer graphics and interactive techniques · doi:10.1145/54852.378507 · Cited by 254
Action-Agnostic Human Pose Forecasting
Chiu, H.-K., Adeli, E., Wang, B., Huang, D.-A., & Niebles, J. C. (2019) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2019.00156 · Cited by 212
Near-optimal character animation with continuous control
Treuille, A., Lee, Y., & Popović, Z. (2007) · ACM SIGGRAPH 2007 papers · doi:10.1145/1275808.1276386 · Cited by 207
Learning Human Motion Models for Long-Term Predictions
Ghosh, P., Song, J., Aksan, E., & Hilliges, O. (2017) · 2017 International Conference on 3D Vision (3DV) · doi:10.1109/3dv.2017.00059 · Cited by 207
DReCon: data-driven responsive control of physics-based characters
Bergamin, K., Clavet, S., Holden, D., & Forbes, J. R. (2019) · ACM Transactions on Graphics · doi:10.1145/3355089.3356536 · Cited by 204
Motion reconstruction using sparse accelerometer data
Tautges, J., Zinke, A., Krüger, B., Baumann, J., Weber, A., Helten, T., Müller, M., Seidel, H.-P., & Eberhardt, B. (2011) · ACM Transactions on Graphics · doi:10.1145/1966394.1966397 · Cited by 197
Motion graphs++: a compact generative model for semantic motion analysis and synthesis
Min, J., & Chai, J. (2012) · ACM Transactions on Graphics · doi:10.1145/2366145.2366172 · Cited by 197
Human Motion Prediction via Spatio-Temporal Inpainting
Hernandez, A., Gall, J., & Moreno, F. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00723 · Cited by 191
Efficient Nonlinear Markov Models for Human Motion
Lehrmann, A. M., Gehler, P. V., & Nowozin, S. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.171 · Cited by 187
A Neural Temporal Model for Human Motion Prediction
Gopalakrishnan, A., Mali, A., Kifer, D., Giles, L., & Ororbia, A. G. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01239 · Cited by 185 · cited 6×
“On another front, many auto-regressive deep learning methods that leverage high quality MOCAP for motion prediction have recently been proposed [Barsoum et al. 2018; Chiu et al. 2019; Fragkiadaki et al. 2015; Gopalakrishnan et al. 2019; Jain et al. 2016; Martinez et al. 2017; Pavllo et al. 2019].”
MethodResultInfluentialFiLM: Visual Reasoning with a General Conditioning Layer
Perez, E., Strub, F., de Vries, H., Dumoulin, V., & Courville, A. C. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1709.07871 · Cited by 184
Towards Natural and Accurate Future Motion Prediction of Humans and Animals
Liu, Z., Wu, S., Jin, S., Liu, Q., Lu, S., Zimmermann, R., & Li, C. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.01024 · Cited by 171
Continuous character control with low-dimensional embeddings
Levine, S., Wang, J. M., Haraux, A., Popović, Z., & Koltun, V. (2012) · ACM Transactions on Graphics · doi:10.1145/2185520.2185524 · Cited by 170
Robust solving of optical motion capture data by denoising
Holden, D. (2018) · ACM Transactions on Graphics · doi:10.1145/3197517.3201302 · Cited by 137
Correction to "Gaussian Process Dynamical Models for Human Motion" [Feb 08 283-298]
Wang, J. M., Fleet, D. J., & Hertzmann, A. (2008) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2008.91 · Cited by 122
Interactive character animation by learning multi-objective control
Lee, K., Lee, S., & Lee, J. (2018) · ACM Transactions on Graphics · doi:10.1145/3272127.3275071 · Cited by 117
Learning to Schedule Control Fragments for Physics-Based Characters Using Deep Q-Learning
Liu, L., & Hodgins, J. K. (2017) · ACM Transactions on Graphics · doi:10.1145/3072959.3083723 · Cited by 116
Constraint-based motion optimization using a statistical dynamic model
Chai, J., & Hodgins, J. K. (2007) · ACM SIGGRAPH Conference (SIGGRAPH) · doi:10.1145/1275808.1276387 · Cited by 114
Recurrent transition networks for character locomotion
Harvey, F. G., & Pal, C. (2018) · SIGGRAPH Asia Technical Briefs · doi:10.1145/3283254.3283277 · Cited by 104
Self-citationRobust task-based control policies for physics-based characters
Coros, S., Beaudoin, P., & van de Panne, M. (2009) · ACM Transactions on Graphics · doi:10.1145/1618452.1618516 · Cited by 96
Motion fields for interactive character locomotion
Lee, Y.-J., Wampler, K., Bernstein, G., Popović, J., & Popović, Z. (2010) · ACM SIGGRAPH Conference and Exhibition on Computer Graphics and Interactive Techniques in Asia (SIGGRAPH Asia) · doi:10.1145/1866158.1866160 · Cited by 85
Precomputing avatar behavior from human motion data
Lee, J., & Lee, K. H. (2005) · Graphical Models · doi:10.1016/j.gmod.2005.03.004 · Cited by 79
Constraint-based motion optimization using a statistical dynamic model
Chai, J., & Hodgins, J. K. (2007) · ACM Transactions on Graphics · doi:10.1145/1276377.1276387 · Cited by 77
Synthesis of Responsive Motion Using a Dynamic Model
Ye, Y., & Liu, C. K. (2010) · Computer Graphics Forum · doi:10.1111/j.1467-8659.2009.01625.x · Cited by 67
Learning to schedule control fragments for physics-based characters using deep q-learning
Liu, L., & Hodgins, J. (2017) · ACM Transactions on Graphics · doi:10.1145/3083723 · Cited by 49
Data-driven autocompletion for keyframe animation
Zhang, X., & van de Panne, M. (2018) · Motion in Games · doi:10.1145/3274247.3274502 · Cited by 48
Model-based Adversarial Imitation LearningPreprint
Baram, N., Anschel, O., & Mannor, S. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.02179 · Cited by 45
Robust task-based control policies for physics-based characters
Coros, S., Beaudoin, P. M., & van de Panne, M. (2009) · ACM SIGGRAPH Conference and Exhibition on Computer Graphics and Interactive Techniques in Asia (SIGGRAPH Asia) · doi:10.1145/1661412.1618516 · Cited by 40
A Deep Learning Framework for Character Motion Synthesis and Editing
Holden, D., Saito, J., & Komura, T. (2023) · ACM eBooks · doi:10.1145/3596711.3596789 · Cited by 39
Long-Term Human Motion Prediction by Modeling Motion Context and Enhancing Motion DynamicPreprint
Tang, Y., Ma, L., Liu, W., & Zheng, W. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.02513 · Cited by 33
Auto-Conditioned LSTM Network for Extended Complex Human Motion Synthesis
Li, Z., Zhou, Y., Xiao, S., He, C., & Li, H. (2017) · Cited by 32
Learning Human Motion Models for Long-term PredictionsPreprint
Ghosh, P. S., Song, J., Aksan, E., & Hilliges, O. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.02827 · Cited by 27
PLU: The Piecewise Linear Unit Activation FunctionPreprint
Nicolae, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.09534 · Cited by 22
Motion fields for interactive character locomotion
Lee, Y.-J., Wampler, K., Bernstein, G., Popović, J., & Popović, Z. (2010) · ACM SIGGRAPH Conference and Exhibition on Computer Graphics and Interactive Techniques in Asia (SIGGRAPH Asia) · doi:10.1145/1882262.1866160 · Cited by 21
Learning to schedule control fragments for physics-based characters using deep Q-learning
Liu, L., & Hodgins, J. K. (2017) · ACM Transactions on Graphics · doi:10.1145/3072959.3126784 · Cited by 8
Cited by
Action-Conditioned 3D Human Motion Synthesis with Transformer VAE
Petrovich, M., Black, M. J., & Varol, G. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01080 · Cited by 493
MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model
Zhang, M., Cai, Z., Pan, L., Hong, F., Guo, X., Yang, L., & Liu, Z. (2022) · arXiv · doi:10.48550/arxiv.2208.15001 · Cited by 437
MotionDiffuse: Text-Driven Human Motion Generation With Diffusion Model
Zhang, M., Cai, Z., Pan, L., Hong, F., Guo, X., Yang, L., & Liu, Z. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3355414 · Cited by 424
Executing your Commands via Motion Diffusion in Latent Space
Chen, X., Jiang, B., Liu, W., Huang, Z., Fu, B., Chen, T., & Yu, G. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01726 · Cited by 356
TEMOS: Generating Diverse Human Motions from Textual Descriptions
Petrovich, M., Black, M. J., & Varol, G. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-20047-2_28 · Cited by 353
Metadata
- Authors
- Harvey, F. G., Yurick, M., Nowrouzezahrai, D., & Pal, C.
- Venue
- ACM Transactions on Graphics
- Published
- 2020-08-12
- Volume
- 39
- Issue
- 4
- ISSN
- 0730-0301 (ISSN-L)
- 1557-7368
- OpenAlex
- W3048474702
- Cited by
- 266
- References
- 68
- FWCI
- 16.39
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.