Detecting outliers when fitting data with nonlinear regression – a new method based on robust nonlinear regression and the false discovery rate
Open accessAbstract
Background: Nonlinear regression, like linear regression, assumes that the scatter of data around the ideal curve follows a Gaussian or normal distribution. This assumption leads to the familiar goal of regression: to minimize the sum of the squares of the vertical or Y-value distances between the points and the curve. Outliers can dominate the sum-of-the-squares calculation, and lead to misleading results. However, we know of no practical method for routinely identifying outliers when fitting curves with nonlinear regression. Results: We describe a new method for identifying outliers when fitting data with nonlinear regression. We first fit the data using a robust form of nonlinear regression, based on the assumption that scatter follows a Lorentzian distribution. We devised a new adaptive method that gradually becomes more robust as the method proceeds. To define outliers, we adapted the false discovery rate approach to handling multiple comparisons. We then remove the outliers, and analyze the data using ordinary least-squares regression. Because the method combines robust regression and outlier removal, we call it the ROUT method. When analyzing simulated data, where all scatter is Gaussian, our method detects (falsely) one or more outlier in only about 1–3% of experiments. When analyzing data contaminated with one or several outliers, the ROUT method performs well at outlier identification, with an average False Discovery Rate less than 1%. Conclusion: Our method, which combines a new method of robust nonlinear regression with a new method of outlier identification, identifies outliers from nonlinear curve fits with reasonable power and few false positives.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The ROUT method was applied at 1% aggressivity level to identify and eliminate outliers within all diagnostic groups to minimize skewedness.
“The ROUT method 43 was applied at 1% aggressivity level to identify and eliminate outliers within all diagnostic groups to minimize skewedness.”
From Prognostic utility of oral neutrophil counts in high-risk periodontitis: A retrospective study (2025) · cites this paper as [43]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482 · see citing works
An Algorithm for Least-Squares Estimation of Nonlinear Parameters
Marquardt, D. W. (1963) · Journal of the Society for Industrial and Applied Mathematics · doi:10.1137/0111030 · Cited by 30,728 · see citing works
Draper, N. R., & Smith, H. (2005) · Technometrics · doi:10.1198/tech.2005.s303 · Cited by 17,993 · see citing works
Press, W. H., Teukolsky, S. A., Vetterling, W. T., & Flannery, B. P. (1994) · Cited by 15,411 · see citing works
A method for the solution of certain non-linear problems in least squares
Levenberg, K. (1944) · Quarterly of Applied Mathematics · doi:10.1090/qam/10666 · Cited by 12,367 · see citing works
The control of the false discovery rate in multiple testing under dependency
Benjamini, Y., & Yekutieli, D. (2001) · The Annals of Statistics · doi:10.1214/aos/1013699998 · Cited by 10,490 · see citing works
Draper, N. R., & Smith, H. (1998) · Wiley series in probability and statistics · doi:10.1002/9781118625590 · Cited by 5,086 · see citing works
Numerical Recipes--The Art of Scientific Computing.
Fritsch, F. N., Press, W. H., Flannery, B. P., Teukolsky, S. A., & Vetterling, W. T. (1988) · Mathematics of Computation · doi:10.2307/2007938 · Cited by 4,846 · see citing works
Robust Statistics: The Approach Based on Influence Functions
Ruppert, D. (1987) · Technometrics · doi:10.2307/1269782 · Cited by 3,880 · see citing works
Gunst, R. F., Draper, N. R., & Smith, H. C. (1999) · Technometrics · doi:10.2307/1270576 · Cited by 3,795 · see citing works
BARNETT, V. D., Lewis, T., & Abeles, F. F. (1979) · Physics Today · doi:10.1063/1.2995737 · Cited by 2,834 · see citing works
More powerful procedures for multiple significance testing
Hochberg, Y., & Benjamini, Y. (1990) · Statistics in Medicine · doi:10.1002/sim.4780090710 · Cited by 2,715 · see citing works
The positive false discovery rate: a Bayesian interpretation and the q-value
Storey, J. D. (2003) · The Annals of Statistics · doi:10.1214/aos/1074290335 · Cited by 2,204 · see citing works
Alternatives to the Median Absolute Deviation
Rousseeuw, P. J., & Croux, C. (1993) · Journal of the American Statistical Association · doi:10.1080/01621459.1993.10476408 · Cited by 1,962 · see citing works
Understanding robust and exploratory data analysis
Hoaglin, D. C., Mosteller, F. C., & Tukey, J. W. (1983) · Wiley eBooks · Cited by 1,878 · see citing works
Robust Statistics: The Approach Based on Influence Functions.
Jennison, C., Hampel, F. R., Ronchetti, E. M., Rousseeuw, P. J., & Stahel, W. A. (1987) · Journal of the Royal Statistical Society Series A (General) · doi:10.2307/2981480 · Cited by 1,753 · see citing works
Atkinson, A. C., BARNETT, V. D., & Lewis, T. (1995) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.2307/2983451 · Cited by 1,688 · see citing works
Numerical Recipes in C-The Art of Scientific Computing.
G., W., Press, W. H., Flannery, B. P., Teukolsky, S. A., & Vetterling, W. T. (1989) · Mathematics of Computation · doi:10.2307/2008673 · Cited by 1,514 · see citing works
Alternatives to the Median Absolute Deviation
Rousseeuw, P. J., & Croux, C. (1993) · Journal of the American Statistical Association · doi:10.2307/2291267 · Cited by 1,489 · see citing works
Storey, J. D., Taylor, J., & Siegmund, D. (2003) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/j.1467-9868.2004.00439.x · Cited by 1,345 · see citing works
Balakrishnan, N., Barnett, V., & Lewis, T. (1995) · Biometrics · doi:10.2307/2533352 · Cited by 1,228 · see citing works
Understanding Robust and Exploratory Data Analysis.
Stuart, M., Hoaglin, D. C., Mosteller, F. C., & Tukey, J. W. (1984) · Journal of the Royal Statistical Society Series D (The Statistician) · doi:10.2307/2988240 · Cited by 398 · see citing works
Time-Efficient Algorithms for Two Highly Robust Estimators of Scale
Croux, C., & Rousseeuw, P. J. (1992) · Computational Statistics · doi:10.1007/978-3-662-26811-7_58 · Cited by 155 · see citing works
Outliers in Statistical Data (3rd edition)
Freeman, J. (1995) · Journal of the Operational Research Society · doi:10.1057/jors.1995.142 · Cited by 154 · see citing works
Robust estimation in very small samples
Rousseeuw, P. J., & Verboben, S. (2002) · Computational Statistics & Data Analysis · doi:10.1016/s0167-9473(02)00078-6 · Cited by 143 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- Seber GA Wild CJ Nonlinear Regression 2003 New York: John Wiley and Sons
Cited by
Sorted by citations
An ultrasensitive method for quantitating circulating tumor DNA with broad patient coverage
Newman, A. M., Bratman, S. V., To, J., Wynne, J. F., Eclov, N. C. W., Modlin, L. A., Liu, C. L., Neal, J. W., Wakelee, H. A., Merritt, R. E., et al. (2014) · Nature Medicine · doi:10.1038/nm.3519 · Cited by 2,220 · see citing works
Monteil, V., Kwon, H., Prado, P., Hagelkrüys, A., Wimmer, R. A., Stahl, M., Leopoldi, A., Garreta, E., Hurtado del Pozo, C., Prosper, F., et al. (2020) · Cell · doi:10.1016/j.cell.2020.04.004 · Cited by 2,122 · see citing works
Disruption of the beclin 1–BCL2 autophagy regulatory complex promotes longevity in mice
Fernández, Á. F., Sebti, S., Wei, Y., Zou, Z., Shi, M., McMillan, K. L., He, C., Ting, T., Liu, Y., Chiang, W.-C., et al. (2018) · Nature · doi:10.1038/s41586-018-0162-7 · Cited by 630 · see citing works
The Effects of Natural and Anthropogenic Microparticles on Individual Fitness in Daphnia magna
Ogonowski, M., Schür, C., Jarsén, Å., & Gorokhova, E. (2016) · PLOS ONE · doi:10.1371/journal.pone.0155063 · Cited by 480 · see citing works
Lee, G., You, H. J., Bajaj, J. S., Joo, S. K., Yu, J., Park, S., Kang, H., Park, J. H., Kim, J. H., Lee, D. H., et al. (2020) · Nature Communications · doi:10.1038/s41467-020-18754-5 · Cited by 395 · see citing works
Insular cortex neurons encode and retrieve specific immune responses
Koren, T., Yifa, R., Amer, M., Krot, M., Boshnak, N., Ben-Shaanan, T. L., Azulay-Debby, H., Zalayat, I., Avishai, E., Hajjo, H., et al. (2021) · Cell · doi:10.1016/j.cell.2021.10.013 · Cited by 334 · see citing works
Song, T., Song, X., Zhu, C., Patrick, R., Skurla, M., Santangelo, I., Green, M., Harper, D., Ren, B., Forester, B. P., et al. (2021) · Ageing Research Reviews · doi:10.1016/j.arr.2021.101503 · Cited by 346 · see citing works
Microbiota from young mice counteracts selective age-associated behavioral deficits
Boehme, M., Guzzetta, K. E., Bastiaanssen, T. F. S., van de Wouw, M., Moloney, G. M., Gual-Grau, A., Spichak, S., Olavarría-Ramírez, L., Fitzgerald, P., Morillas, E., et al. (2021) · Nature Aging · doi:10.1038/s43587-021-00093-9 · Cited by 313 · see citing works
An ocular glymphatic clearance system removes β-amyloid from the rodent eye
Wang, X., Lou, N., Eberhardt, A., Yang, Y., Kusk, P., Xu, Q., Förstera, B., Peng, S., Shi, M., Ladrón-de-Guevara, A., et al. (2020) · Science Translational Medicine · doi:10.1126/scitranslmed.aaw3210 · Cited by 239 · see citing works
Frese, S. A., Hutton, A. A., Contreras, L. N., Shaw, C. A., Palumbo, M. C., Casaburi, G., Xu, G., Davis, J. C. C., Lebrilla, C. B., Henrick, B. M., et al. (2017) · mSphere · doi:10.1128/msphere.00501-17 · Cited by 231 · see citing works
PDXNET Consortium, Woo, X. Y., EurOPDX Consortium, Giordano, J., Srivastava, A., Zhao, Z.-M., Lloyd, M. W., de Bruijn, R., Suh, Y.-S., Patidar, R., et al. (2021) · Nature Genetics · doi:10.1038/s41588-020-00750-6 · Cited by 190 · see citing works
Guidance Document on Good In Vitro Method Practices (GIVIMP)
OECD (2018) · OECD Series on Testing and Assessment · doi:10.1787/9789264304796-en · Cited by 161 · see citing works
Moisiadis, V. G., Constantinof, A., Kostaki, A., Szyf, M., & Matthews, S. G. (2017) · Scientific Reports · doi:10.1038/s41598-017-11635-w · Cited by 122 · see citing works
Zanos, P., Brown, K. A., Georgiou, P., Yuan, P., Zarate, C. A., Thompson, S. M., & Gould, T. D. (2023) · The Journal of Neuroscience · doi:10.1523/jneurosci.1316-22.2022 · Cited by 115 · see citing works
Nicotinamide Riboside Is a Major NAD+ Precursor Vitamin in Cow Milk
Trammell, S. A., Yu, L., Redpath, P., Migaud, M. E., & Brenner, C. (2016) · The Journal of Nutrition · doi:10.3945/jn.116.230078 · Cited by 112 · see citing works
Foguth, R. M., Flynn, R. W., de Perre, C., Iacchetta, M., Lee, L. S., Sepúlveda, M. S., & Cannon, J. R. (2019) · Toxicology and Applied Pharmacology · doi:10.1016/j.taap.2019.114623 · Cited by 103 · see citing works
Granata, C., Caruana, N. J., Botella, J., Jamnick, N. A., Huynh, K., Kuang, J., Janssen, H. A., Reljic, B., Mellett, N. A., Laskowski, A., et al. (2021) · Nature Communications · doi:10.1038/s41467-021-27153-3 · Cited by 93 · see citing works
Bush, R. A., Zeng, Y., Colosi, P., Kjellstrom, S., Hiriyanna, S., Vijayasarathy, C., Santos, M., Li, J., Wu, Z., & Sieving, P. A. (2016) · Human Gene Therapy · doi:10.1089/hum.2015.142 · Cited by 86 · see citing works
Nicotine aversion is mediated by GABAergic interpeduncular nucleus inputs to laterodorsal tegmentum
Wolfman, S. L., Gill, D. F., Bogdanic, F., Long, K., Al-Hasani, R., McCall, J. G., Bruchas, M. R., & McGehee, D. S. (2018) · Nature Communications · doi:10.1038/s41467-018-04654-2 · Cited by 81 · see citing works
Zanos, P., Highland, J. N., Liu, X., Troppoli, T. A., Georgiou, P., Lovett, J., Morris, P. J., Stewart, B. W., Thomas, C. J., Thompson, S. M., et al. (2019) · British Journal of Pharmacology · doi:10.1111/bph.14683 · Cited by 82 · see citing works
Oxytocin neurons mediate the effect of social isolation via the VTA circuits
Musardo, S., Contestabile, A., Knoop, M., Baud, O., & Bellone, C. (2022) · eLife · doi:10.7554/elife.73421 · Cited by 77 · see citing works
Gut-brain axis serotonergic responses to acute stress exposure are microbiome-dependent
Lyte, J. M., Gheorghe, C. E., Goodson, M. S., Kelley‐Loughnane, N., Dinan, T. G., Cryan, J. F., & Clarke, G. (2020) · Neurogastroenterology & Motility · doi:10.1111/nmo.13881 · Cited by 77 · see citing works
Dual midbrain and forebrain origins of thalamic inhibitory interneurons
Jager, P., Moore, G., Calpin, P., Durmishi, X., Salgarella, I., Menage, L., Kita, Y., Wang, Y., Kim, D. W., Blackshaw, S., et al. (2021) · eLife · doi:10.7554/elife.59272 · Cited by 61 · see citing works
Nicolini, C., Michalski, B., Toepp, S. L., Turco, C. V., D'Hoine, T., Harasym, D., Gibala, M. J., Fahnestock, M., & Nelson, A. J. (2020) · Neuroscience · doi:10.1016/j.neuroscience.2020.03.042 · Cited by 69 · see citing works
Cellular senescence is increased in airway smooth muscle cells of elderly persons with asthma
Aghali, A., Khalfaoui, L., Lagnado, A. B., Drake, L. Y., Teske, J. J., Pabelick, C. M., Passos, J. F., & Prakash, Y. S. (2022) · American Journal of Physiology-Lung Cellular and Molecular Physiology · doi:10.1152/ajplung.00146.2022 · Cited by 44 · see citing works
Chronic heart failure with diabetes mellitus is characterized by a severe skeletal muscle pathology
Garnham, J. O., Roberts, L. D., Espino‐Gonzalez, E., Whitehead, A., Swoboda, P. P., Koshy, A., Gierula, J., Paton, M. F., Cubbon, R. M., Kearney, M. T., et al. (2020) · Journal of Cachexia Sarcopenia and Muscle · doi:10.1002/jcsm.12515 · Cited by 32 · see citing works
Anderzhanova, E., Hafner, K., Genewsky, A. J., Soliman, A., Pöhlmann, M. L., Schmidt, M. V., Blum, R., Wotjak, C. T., & Gassen, N. C. (2020) · Neurobiology of Stress · doi:10.1016/j.ynstr.2020.100239 · Cited by 29 · see citing works
Illouz, T., Madar, R., Biragyn, A., & Okun, E. (2019) · Brain, Behavior, and Immunity · doi:10.1016/j.bbi.2018.10.004 · Cited by 22 · see citing works
Troup, M., Zalucki, O. H., Kottler, B. D., Karunanithi, S., Anggono, V., & van Swinderen, B. (2019) · Anesthesiology · doi:10.1097/aln.0000000000002850 · Cited by 21 · see citing works
Illouz, T., Nicola, R., Ben-Shushan, L., Madar, R., Biragyn, A., & Okun, E. (2021) · Communications Biology · doi:10.1038/s42003-021-01851-6 · Cited by 21 · see citing works
Location and temporal memory of objects declines in aged marmosets (Callithrix jacchus)
De Castro, V., & Girard, P. (2021) · Scientific Reports · doi:10.1038/s41598-021-88357-7 · Cited by 17 · see citing works
Controlled dietary phosphate loading in healthy young men elevates plasma phosphate and FGF23 levels
Gerber, J. S., Arroyo, E. M. P., Pastor, J., Correia, M., Rudloff, S., Moe, O. W., Egli-Spichtig, D., Mohebbi, N., & Wagner, C. A. (2025) · Pflügers Archiv - European Journal of Physiology · doi:10.1007/s00424-024-03046-4 · Cited by 15 · see citing works
Jaffar, J., McMillan, L., Wilson, N., Panousis, C., Hardy, C., Cho, H.-J., Symons, K., Glaspole, I., Westall, G., & Wong, M. (2022) · American Journal of Physiology-Lung Cellular and Molecular Physiology · doi:10.1152/ajplung.00165.2021 · Cited by 13 · see citing works
Decreased expression of the GLP-1 receptor after segmental artery injury in mice
Bjørnholm, K. D., Povlsen, G. K., Ougaard, M. E., Pyke, C., Rakipovski, G., Tveden-Nyborg, P., Lykkesfeldt, J., & Skovsted, G. F. (2021) · Journal of Endocrinology · doi:10.1530/joe-20-0608 · Cited by 10 · see citing works
Prognostic utility of oral neutrophil counts in high-risk periodontitis: A retrospective study
Prete, B. R. J., Barbour, A., Sun, C., Tenenbaum, H. C., Goldberg, M. B., & Glogauer, M. (2025) · Journal of Periodontology · doi:10.1002/jper.11360 · Cited by 5 · see citing works
- Authors
- Motulsky, H. J., & Brown, R. E.
- Corresponding author
- Motulsky, Harvey J
- ORCID
- Motulsky, Harvey J
- Venue
- BMC Bioinformatics
- Published
- 2006-03-09
- Volume
- 7
- Issue
- 1
- Pages
- 123-123
- ISSN
- 1471-2105 (ISSN-L)
- PMC
- PMC1472692
- OpenAlex
- W1833314573
- Cited by
- 1,616
- References
- 25
- Citations by year
- 2026: 126 · 2025: 173 · 2024: 206 · 2023: 184 · 2022: 174 · 2021: 192 · 2020: 128 · 2019: 104 · 2018: 94 · 2017: 60 · 2016: 48 · 2015: 41 · 2014: 22 · 2013: 22 · 2012: 9
- FWCI
- 4.43
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- BMC Bioinformatics (publishedVersion, cc-by)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- National Institutes of Health (5r43ca094450-02)
- National Cancer Institute (5 R43 CA094450-02, R43 CA094450)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Statistical Methods and Models
- Physical Sciences › Mathematics › Applied Mathematics › Statistical and numerical algorithms
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Applications
- Keywords
- outlier detection, nonlinear regression, false discovery rate, outlier removal, multiple comparisons, curve fitting, least squares regression
- MeSH terms
- Computer Simulation, Data Interpretation, Statistical, False Positive Reactions, Models, Biological, Numerical Analysis, Computer-Assisted, Regression Analysis, Models, Statistical, Nonlinear Dynamics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
