CoChat
Sign Up
PreprintCited by 7Generative Adversarial Networks and Image SynthesisModel Reduction and Neural NetworksMarkov Chains and Monte Carlo Methods

The DOI matches its DataCite record.

Multisample Flow Matching: Straightening Flows with Minibatch Couplings

Open access
Pooladian, Aram-Alexandre, Ben-Hamu, Heli, Domingo-Enrich, Carles, Amos, Brandon, Lipman, Yaron, Chen, Ricky T. Q.·2023·arXiv
DOI 10.48550/arxiv.2304.14772arXiv 2304.14772
Open this paper in CoChat — free accountOpen PDF

Abstract

Simulation-free methods for training continuous-time generative models construct probability paths that go between noise distributions and individual data samples. Recent works, such as Flow Matching, derived paths that are optimal for each data sample. However, these algorithms rely on independent data and noise samples, and do not exploit underlying structure in the data distribution for constructing probability paths. We propose Multisample Flow Matching, a more general framework that uses non-trivial couplings between data and noise samples while satisfying the correct marginal constraints. At very small overhead costs, this generalization allows us to (i) reduce gradient variance during training, (ii) obtain straighter flows for the learned vector field, which allows us to generate high-quality samples using fewer function evaluations, and (iii) obtain transport maps with lower cost in high dimensions, which has applications beyond generative modeling. Importantly, we do so in a completely simulation-free manner with a simple minimization objective. We show that our proposed methods improve sample consistency on downsampled ImageNet data sets, and lead to better low-cost sample generation.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2304.14772

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 65

  1. Albergo, M. S. and Vanden-Eijnden, E. Building normal- izing flows with stochastic interpolants. International Conference on Learning Representations, 2023.
  2. Altschuler, J., Weed, J., and Rigollet, P. Near-linear time approximation algorithms for optimal transport via Sinkhorn iteration. In Advances in Neural Information Processing Systems 30, 2017.
  3. Amos, B. On amortizing convex conjugates for optimal transport. International Conference on Learning Repre- sentations, 2023.
  4. Arjovsky, M., Chintala, S., and Bottou, L. Wasserstein gen- erative adversarial networks. In International conference on machine learning, pp. 214-223. PMLR, 2017.
  5. Asadulaev, A., Korotin, A., Egiazarian, V., and Burnaev, E. Neural optimal transport with general cost functionals. arXiv preprint arXiv:2205.15403, 2022. · arXiv
  6. Benamou, J.-D. and Brenier, Y. A computational fluid me- chanics solution to the Monge-Kantorovich mass transfer problem. Numerische Mathematik, 84(3):375-393, 2000.
  7. Bunne, C., Stark, S. G., Gut, G., del Castillo, J. S., Lehmann, K.-V., Pelkmans, L., Krause, A., and Rätsch, G. Learning single-cell perturbation responses using neural optimal transport. bioRxiv, 2021.
  8. Bunne, C., Krause, A., and Cuturi, M. Supervised training of conditional Monge maps. arXiv preprint arXiv:2206.14262, 2022. · arXiv
  9. Caffarelli, L. The regularity of mappings with a convex potential. Journal of the American Mathematical Society, 5:99-104, 1992.
  10. Chen, R. T. Q. and Lipman, Y. Riemannian flow match- ing on general geometries. International Conference on Machine Learning, 2023.
  11. Chen, R. T. Q., Rubanova, Y., Bettencourt, J., and Duvenaud, D. K. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
  12. Chrabaszcz, P., Loshchilov, I., and Hutter, F. A downsam- pled variant of ImageNet as an alternative to the Cifar datasets. arXiv preprint arXiv:1707.08819, 2017. · arXiv
  13. Cuesta-Albertos, J., Matrán, C., and Tuero-Diaz, A. Opti- mal transportation plans and convergence in distribution. Journal of Multivariate Analysis, 60(1):72-83, 1997.
  14. Cuturi, M. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 26, 2013.
  15. Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Ieee, 2009.
  16. Dhariwal, P. and Nichol, A. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780-8794, 2021.
  17. Dinh, L., Sohl-Dickstein, J., and Bengio, S. Density esti- mation using real nvp. arXiv preprint arXiv:1605.08803, 2016. · arXiv
  18. Fan, J., Liu, S., Ma, S., Chen, Y., and Zhou, H. Scalable computation of monge maps with general costs. arXiv preprint arXiv:2106.03812, pp. 4, 2021. · arXiv
  19. Fatras, K., Zine, Y., Flamary, R., Gribonval, R., and Courty, N. Learning with minibatch Wasserstein: asymptotic and gradient properties. arXiv preprint arXiv:1910.04091, 2019. · arXiv
  20. Fatras, K., Zine, Y., Majewski, S., Flamary, R., Gribon- val, R., and Courty, N. Minibatch optimal transport distances; analysis and applications. arXiv preprint arXiv:2101.01792, 2021. · arXiv
  21. Ferradans, S., Papadakis, N., Peyré, G., and Aujol, J.-F. Regularized discrete optimal transport. SIAM Journal on Imaging Sciences, 7(3):1853-1882, 2014.
  22. Feydy, J., Charlier, B., Vialard, F.-X., and Peyré, G. Optimal transport for diffeomorphic registration. In International Conference on Medical Image Computing and Computer- Assisted Intervention, pp. 291-299. Springer, 2017.
  23. Finlay, C., Gerolin, A., Oberman, A. M., and Poola- dian, A.-A. Learning normalizing flows from Entropy- Kantorovich potentials. arXiv preprint arXiv:2006.06033, 2020a. · arXiv
  24. Finlay, C., Jacobsen, J.-H., Nurbekyan, L., and Oberman, A. How to train your neural ode: the world of jacobian and kinetic regularization. In International Conference on Machine Learning, pp. 3154-3164. PMLR, 2020b.
  25. Flamary, R., Courty, N., Gramfort, A., Alaya, M. Z., Bois- bunon, A., Chambon, S., Chapel, L., Corenflos, A., Fatras, K., Fournier, N., Gautheron, L., Gayraud, N. T., Janati, H., Rakotomamonjy, A., Redko, I., Rolet, A., Schutz, A., Seguy, V., Sutherland, D. J., Tavenard, R., Tong, A., and Vayer, T. Pot: Python optimal transport. Journal of Ma- chine Learning Research, 22(78):1-8, 2021. URL http: //jmlr.org/papers/v22/20-451.html.
  26. Gafni, O., Polyak, A., Ashual, O., Sheynin, S., Parikh, D., and Taigman, Y. Make-a-scene: Scene-based text- to-image generation with human priors. arXiv preprint arXiv:2203.13131, 2022. · arXiv
  27. Gale, D. and Shapley, L. S. College admissions and the sta- bility of marriage. The American Mathematical Monthly, 69(1):9-15, 1962.
  28. Gatys, L. A., Ecker, A. S., and Bethge, M. A neural algo- rithm of artistic style. arXiv preprint arXiv:1508.06576, 2015. · arXiv
  29. Genevay, A., Peyré, G., and Cuturi, M. Learning genera- tive models with Sinkhorn divergences. In International Conference on Artificial Intelligence and Statistics, pp. 1608-1617. PMLR, 2018.
  30. Ho, J., Jain, A., and Abbeel, P. Denoising diffusion proba- bilistic models. Advances in Neural Information Process- ing Systems, 33:6840-6851, 2020.
  31. Huang, C.-W., Chen, R. T., Tsirigotis, C., and Courville, A. Convex potential flows: Universal probability distribu- tions with optimal transport and convex optimization. In International Conference on Learning Representations, 2020.
  32. Hunter, J. D. Matplotlib: A 2d graphics environment. Com- puting in science & engineering, 9(3):90, 2007.
  33. Johnson, J., Alahi, A., and Fei-Fei, L. Perceptual losses for real-time style transfer and super-resolution. In Com- puter Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Pro- ceedings, Part II 14, pp. 694-711. Springer, 2016.
  34. Kluyver, T., Ragan-Kelley, B., Pérez, F., Granger, B. E., Bussonnier, M., Frederic, J., Kelley, K., Hamrick, J. B., Grout, J., Corlay, S., et al. Jupyter notebooks-a publishing format for reproducible computational workflows. In ELPUB, pp. 87-90, 2016.
  35. Lee, S., Kim, B., and Ye, J. C. Minimizing trajectory cur- vature of ode-based generative models. arXiv preprint arXiv:2301.12003, 2023. · arXiv
  36. Li, C.-L., Chang, W.-C., Cheng, Y., Yang, Y., and Póczos, B. MMD GAN: Towards deeper understanding of mo- ment matching network. Advances in neural information processing systems, 30, 2017.
  37. Lipman, Y., Chen, R. T., Ben-Hamu, H., Nickel, M., and Le, M. Flow matching for generative modeling. International Conference on Learning Representations, 2023.
  38. Liu, G.-H., Vahdat, A., Huang, D.-A., Theodorou, E. A., Nie, W., and Anandkumar, A. I 2 sb: Image-to-image Schr\" odinger bridge. International Conference on Ma- chine Learning, 2023.
  39. Liu, H., Gu, X., and Samaras, D. Wasserstein gan with quadratic transport cost. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 4832- 4841, 2019.
  40. Liu, X., Gong, C., and Liu, Q. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.03003, 2022. · arXiv
  41. Lübeck, F., Bunne, C., Gut, G., del Castillo, J. S., Pelk- mans, L., and Alvarez-Melis, D. Neural unbalanced opti- mal transport via cycle-consistent semi-couplings. arXiv preprint arXiv:2209.15621, 2022. · arXiv
  42. Makkuva, A., Taghvaei, A., Oh, S., and Lee, J. Optimal transport mapping via input convex neural networks. In International Conference on Machine Learning, pp. 6672- 6681. PMLR, 2020.
  43. McCann, R. J. A convexity principle for interacting gases. Advances in mathematics, 128(1):153-179, 1997.
  44. McKinney, W. Python for data analysis: Data wrangling with Pandas, NumPy, and IPython. " O'Reilly Media, Inc.", 2012.
  45. Nguyen, K., Nguyen, D., Pham, T., Ho, N., et al. Improving mini-batch optimal transport via partial transportation. In International Conference on Machine Learning, pp. 16656-16690. PMLR, 2022.
  46. Nurbekyan, L., Iannantuono, A., and Oberman, A. M. No- collision transportation maps. Journal of Scientific Com- puting, 82(2), 2020.
  47. Oliphant, T. E. Python for scientific computing. Computing in Science & Engineering, 9(3):10-20, 2007.
  48. Onken, D., Fung, S. W., Li, X., and Ruthotto, L. Ot-flow: Fast and accurate continuous normalizing flows via opti- mal transport. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 9223-9232, 2021.
  49. Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. Pytorch: An imperative style, high-performance deep learning library. In Advances in neural information processing systems, pp. 8026-8037, 2019.
  50. Peyré, G. and Cuturi, M. Computational optimal transport. Foundations and Trends® in Machine Learning, 11(5-6): 355-607, 2019.
  51. Pooladian, A.-A. and Niles-Weed, J. Entropic estimation of optimal transport maps. arXiv preprint arXiv:2109.12004, 2021. · arXiv
  52. Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022. · arXiv
  53. Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E., Ghasemipour, S. K. S., Ayan, B. K., Mahdavi, S. S., Lopes, R. G., et al. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint arXiv:2205.11487, 2022. · arXiv
  54. Schiebinger, G., Shu, J., Tabaka, M., Cleary, B., Subrama- nian, V., Solomon, A., Gould, J., Liu, S., Lin, S., Berube, P., et al. Optimal-transport analysis of single-cell gene expression identifies developmental trajectories in repro- gramming. Cell, 176(4):928-943, 2019.
  55. Seguy, V., Damodaran, B. B., Flamary, R., Courty, N., Rolet, A., and Blondel, M. Large-scale optimal transport and mapping estimation. arXiv preprint arXiv:1711.02283, 2017. · arXiv
  56. Solomon, J., De Goes, F., Peyré, G., Cuturi, M., Butscher, A., Nguyen, A., Du, T., and Guibas, L. Convolutional Wasserstein distances: Efficient optimal transportation on geometric domains. ACM Transactions on Graphics (TOG), 34(4):66, 2015.
  57. Solomon, J., Peyré, G., Kim, V. G., and Sra, S. Entropic metric alignment for correspondence problems. ACM Trans. Graph., 35(4):72:1-72:13, 2016.
  58. Song, Y., Durkan, C., Murray, I., and Ermon, S. Maximum likelihood training of score-based diffusion models. Ad- vances in Neural Information Processing Systems, 34: 1415-1428, 2021a.
  59. Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Er- mon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. International Conference on Learning Representations, 2021b.
  60. Tong, A., Huang, J., Wolf, G., Van Dijk, D., and Krish- naswamy, S. Trajectorynet: A dynamic optimal transport network for modeling cellular dynamics. In International conference on machine learning, pp. 9526-9536. PMLR, 2020.
  61. Tong, A., Malkin, N., Huguet, G., Zhang, Y., Rector-Brooks, J., Fatras, K., Wolf, G., and Bengio, Y. Conditional flow matching: Simulation-free dynamic optimal transport. arXiv preprint arXiv:2302.00482, 2023. · arXiv
  62. Van Der Walt, S., Colbert, S. C., and Varoquaux, G. The numpy array: a structure for efficient numerical compu- tation. Computing in Science & Engineering, 13(2):22, 2011.
  63. Varadarajan, V. S. On the convergence of sample probability distributions. Sankhyā: The Indian Journal of Statistics (1933-1960), 19(1/2):23-26, 1958.
  64. Villani, C. Optimal Transport: Old and New. Grundlehren der mathematischen Wissenschaften. Springer Berlin Hei- delberg, 2008.
  65. Wolansky, G. Semi-discrete optimal transport. arXiv preprint arXiv:1911.04348, Sep 2020. · arXiv

Cited by

Sorted by citations

Metadata

Authors
Pooladian, A.-A., Ben-Hamu, H., Domingo-Enrich, C., Amos, B., Lipman, Y., & Chen, R. T. Q.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2304.14772
arXiv
2304.14772
OpenAlex
W4367832869
Cited by
7
References
0
Citations by year
2025: 3 · 2024: 4
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
  • National Science Foundation (1922658)
  • Division of Chemistry
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
  • Physical Sciences › Physics and Astronomy › Statistical and Nonlinear Physics › Model Reduction and Neural Networks
  • Physical Sciences › Mathematics › Statistics and Probability › Markov Chains and Monte Carlo Methods
Keywords
flow matching, gradient variance reduction, transport maps, sample consistency, image generation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review