Digital taxonomist: Identifying plant species in community scientists’ photographs
Open accessAbstract
Automatic identification of plant specimens from amateur photographs could improve species range maps, thus supporting ecosystems research as well as conservation efforts. However, classifying plant specimens based on image data alone is challenging: some species exhibit large variations in visual appearance, while at the same time different species are often visually similar; additionally, species observations follow a highly imbalanced, long-tailed distribution due to differences in abundance as well as observer biases. On the other hand, most species observations are accompanied by side information about the spatial, temporal and ecological context. Moreover, biological species are not an unordered list of classes but embedded in a hierarchical taxonomic structure. We propose a multimodal deep learning model that takes into account these additional cues in a unified framework. Our Digital Taxonomist is able to identify plant species in photographs better than a classifier trained on the image content alone, the performance gained is over 6 percent points in terms of accuracy.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In another example, hierarchical structure of taxonomies and locational priors are leveraged to constrain plant species classification from iNaturalist in Switzerland, leading to improvements of state-of-the-art models of about 5%.
“In another example138, the hierarchical structure of taxonomies, as well as locational priors, are leveraged to constrain plant species classification from iNaturalist in Switzerland, leading to improvements of state-of-the-art models of about 5%.”
From Perspectives in machine learning for wildlife conservation (2022) · cites this paper as [138]
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 1× · see citing works
“In our implementation we use the popular ResNet architecture (He et al., 2016), although other networks could also be employed.”
MethodImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 3× · see citing works
“Our ResNet is pre-trained on ImageNet [8], a setting that has become common practice to speed up training and boost performance with limited data.”
BackgroundMethodInfluential(2014) · Cited by 33,419 · see citing works
A new dawn for citizen science
SILVERTOWN, J. W. (2009) · Trends in Ecology & Evolution · doi:10.1016/j.tree.2009.03.017 · Cited by 2,199 · cited 1× · see citing works
“The community science paradigm aims at involving the general public in scientific observations and investigations, and is particularly useful in cases where the experiment is characterized by a large spatial and/or temporal scale (Silvertown, 2009).”
BackgroundStochastic Gradient Descent Tricks
Bottou, L. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-35289-8_25 · Cited by 2,097 · see citing works
Citizen Science as an Ecological Research Tool: Challenges and Benefits
Dickinson, J. L., Zuckerberg, B., & Bonter, D. N. (2010) · Annual Review of Ecology Evolution and Systematics · doi:10.1146/annurev-ecolsys-102209-144636 · Cited by 1,919 · see citing works
Díaz, S. M., Settele, J., Brondízio, E. S., Ngo, H. T., Guèze, M., Agard, J. B. R., Arneth, A., Balvanera, P., Brauman, K. A., Butchart, S. H. M., et al. (2019) · Americanae (AECID Library) · Cited by 1,359 · see citing works
HD-CNN: Hierarchical Deep Convolutional Neural Networks for Large Scale Visual Recognition
Yan, Z., Zhang, H., Piramuthu, R., Jagadeesh, V., DeCoste, D., Di, W., & Yu, Y. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.314 · Cited by 936 · cited 1× · see citing works
“(Yan et al., 2015) was another early work that tackled hierarchical classification in the context of visual recognition.”
BackgroundSharpness-Aware Minimization for Efficiently Improving Generalization
Foret, P., Kleiner, A., Mobahi, H., & Neyshabur, B. (2020) · arXiv · doi:10.48550/arxiv.2010.01412 · Cited by 521 · cited 1× · see citing works
“Novel, deep learning-based methods can achieve higher accuracies on the same dataset without spatio-temporal priors (Foret et al., 2021).”
BackgroundPlant taxonomy and biosystematics
(1990) · Choice Reviews Online · doi:10.5860/choice.27-6336 · Cited by 507 · see citing works
Recommending plant taxa for supporting on-site species identification
Wittich, H. C., Seeland, M., Wäldchen, J., Rzanny, M., & Mäder, P. (2018) · BMC Bioinformatics · doi:10.1186/s12859-018-2201-7 · Cited by 471 · see citing works
Gaston, K. J., & Spicer, J. I. (1998) · Cited by 466 · see citing works
Meatyard, B. (2004) · Biological Conservation · doi:10.1016/j.biocon.2004.07.009 · Cited by 382 · see citing works
Birdsnap: Large-Scale Fine-Grained Visual Categorization of Birds
Berg, T., Liu, J., Lee, S. W., Alexander, M. L., Jacobs, D. W., & Belhumeur, P. N. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.259 · Cited by 313 · cited 1× · see citing works
“In (Berg et al., 2014) the location and time where a photo was taken are used to define a prior distribution over bird species occurrences.”
BackgroundTree-CNN: A hierarchical Deep Convolutional Neural Network for incremental learning
Roy, D., Panda, P., & Roy, K. (2020) · Neural Networks · doi:10.1016/j.neunet.2019.09.010 · Cited by 257 · cited 1× · see citing works
“, 2014), and more recently (Roy et al., 2018) analysed the use of hierarchical labels for visual recognition for the specific case of incremental learning.”
BackgroundXiao, T., Zhang, J., Yang, K., Peng, Y., & Zhang, Z. (2014) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/2647868.2654926 · Cited by 235 · see citing works
Country-wide high-resolution vegetation height mapping with Sentinel-2
Lang, N., Schindler, K., & Wegner, J. D. (2019) · Remote Sensing of Environment · doi:10.1016/j.rse.2019.111347 · Cited by 213 · see citing works
Self-citationCrop mapping from image time series: Deep learning with multi-scale label hierarchies
Türkoglu, M. Ö., D’Aronco, S., Perich, G., Liebisch, F., Streit, C., Schindler, K., & Wegner, J. D. (2021) · Remote Sensing of Environment · doi:10.1016/j.rse.2021.112603 · Cited by 179 · cited 2× · see citing works
“In (Turkoglu et al., 2021) the authors investigate the task of classifying agricultural crops from a sequence of satellite images, where the crop labels also exhibit a hierarchical structure (e.”
BackgroundSelf-citationDiscriminative Transfer Learning with Tree-based Priors
Srivastava, N., & Salakhutdinov, R. (2013) · Cited by 171 · see citing works
Presence-Only Geographical Priors for Fine-Grained Image Classification
Aodha, O. M., Cole, E., & Perona, P. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00969 · Cited by 149 · see citing works
Fine-Grained Representation Learning and Recognition by Exploiting Hierarchical Semantic Embedding
Chen, T., Wu, W., Gao, Y., Dong, L., Luo, X., & Lin, L. (2018) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3240508.3240523 · Cited by 111 · cited 1× · see citing works
“The authors of (Chen et al., 2018) propose a framework to predict the category scores at each hierarchy (tree) level in a top-down manner, with a multi-head network where each branch is responsible for a different level.”
BackgroundImproving Image Classification with Location Context
Tang, K., Paluri, M., Fei-Fei, L., Fergus, R., & Bourdev, L. D. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.121 · Cited by 108 · cited 1× · see citing works
“In (Tang et al., 2015) the authors investigate how to encode the image’s GPS coordinate to increase prediction accuracy.”
BackgroundButcher, G. S., Butcher, G. S., & Niven, D. K. (2007) · Cited by 76 · see citing works
Geo-Aware Networks for Fine-Grained Recognition
Chu, G., Potetz, B. R., Wang, W., Howard, A., Song, Y., Brücher, F., Leung, T., & Adam, H. (2019) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2019.00033 · Cited by 76 · cited 5× · see citing works
“The difference between the two approaches is that in the former (Chu et al., 2019) the geographical network is trained to improve the image-based prediction coming from the CNN, but cannot make a meaningful prediction on its own, i.”
BackgroundMethodInfluentialBaichoo, S., Souilmi, Y., Panji, S., Botha, G., Meintjes, A., Hazelhurst, S., Bendou, H., de Beste, E., Mpangase, P. T., Souiai, O., et al. (2018) · BMC Bioinformatics · doi:10.1186/s12859-018-2446-1 · Cited by 54 · see citing works
Hierarchical Image Classification using Entailment Cone Embeddings
Dhall, A., Makarova, A., Ganea, O.-E., Pavllo, D., Greeff, M., & Krause, A. (2020) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw50498.2020.00426 · Cited by 50 · cited 1× · see citing works
“Recently, (Dhall et al., 2020) have investigated and compared a number of strategies and loss functions to integrate hierarchical semantic structure into a CNN, including per-level classifiers, hierarchical softmax, and a marginalisation loss.”
BackgroundThe iWildCam 2021 Competition DatasetPreprint
Beery, S., Agarwal, A., Cole, E., & Birodkar, V. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.03494 · Cited by 42 · see citing works
Hierarchical Category Detector for Clothing Recognition from Visual Data
Kumar, S., & Zheng, R. (2017) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2017.272 · Cited by 13 · cited 3× · see citing works
“To integrate hierarchical labels, we adopt the marginalisation loss proposed in (Kumar and Zheng, 2017).”
BackgroundMethodInfluential
Flagged first, then by citations
2 of the 30 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Perspectives in machine learning for wildlife conservation
Tuia, D., Kellenberger, B., Beery, S., Costelloe, B. R., Zuffi, S., Risse, B., Mathis, A., Mathis, M. W., van Langevelde, F., Burghardt, T., et al. (2022) · Nature Communications · doi:10.1038/s41467-022-27980-y · Cited by 740 · see citing works
Deep learning as a tool for ecology and evolution
Borowiec, M. L., Dikow, R. B., Frandsen, P. B., McKeeken, A. J., Valentini, G., & White, A. E. (2022) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.13901 · Cited by 264 · see citing works
Brun, P., Karger, D. N., Zurell, D., Descombes, P., de Witte, L. C., de Lutio, R., Wegner, J. D., & Zimmermann, N. E. (2024) · Nature Communications · doi:10.1038/s41467-024-48559-9 · Cited by 47 · see citing works
“Lower susceptibility to sampling bias, in turn, reduces the need to sacri fi ce observations through thinning and allows harnessing information from the full dataset, for example, the explicit consideration of seasonal effects on observed biodiversity 19,20 .”
BackgroundSelf-citationFrom Coarse to Fine-Grained Open-Set Recognition
Lang, N., Snæbjarnarson, V., Cole, E., Aodha, O. M., Igel, C., & Belongie, S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01686 · Cited by 18 · see citing works
“While the use of hierarchical labels has been studied in closed-set recognition to reduce the severity of mistakes [6, 15, 17] in fashion [31], crop mapping [50], food [58] and species recognition [14], the role of hierarchical representations for fine-grained OSR is underexplored.”
Automatic fused multimodal deep learning for plant identification
Lapkovskis, A., Nefedova, N., & Beikmohammadi, A. (2025) · Frontiers in Plant Science · doi:10.3389/fpls.2025.1616020 · Cited by 16 · see citing works
Improved Wildlife Recognition through Fusing Camera Trap Images and Temporal Metadata
Liu, L., Mou, C., & Xu, F. (2024) · Diversity · doi:10.3390/d16030139 · Cited by 15 · see citing works
Harnessing deep learning for plant species classification: A comprehensive review
Zulfiqar, A., Ebroul Izquierdo, M., & Chandramouli, K. (2025) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2025.110663 · Cited by 11 · see citing works
No species left behind: borrowing strength to map data-deficient species
Sharma, S., Winner, K., Pollock, L. J., Thorson, J. T., Mäkinen, J., Merow, C., Pedersen, E. J., Chefira, K. F., Portmann, J. M., Iannarilli, F., et al. (2025) · Trends in Ecology & Evolution · doi:10.1016/j.tree.2025.04.010 · Cited by 11 · see citing works
Sütçüoğlu, G. G., & Önaç, A. K. (2025) · Scientific Reports · doi:10.1038/s41598-025-25178-y · Cited by 7 · see citing works
Elvekjaer, N., Martinez-Sanchez, L., Bonnet, P., Joly, A., Paracchini, M. L., & van der Velde, M. (2024) · Ecological Solutions and Evidence · doi:10.1002/2688-8319.12324 · Cited by 7 · see citing works
“Others include geo-location of the image, and from the position various environmental conditions such as climate and terrain can be inferred (de Lutio et al., 2021; Terry et al., 2020).”
InfluentialCLIP-Driven Few-Shot Species-Recognition Method for Integrating Geographic Information
Liu, L., Yang, L., Yang, F., Chen, F., & Xu, F. (2024) · Remote Sensing · doi:10.3390/rs16122238 · Cited by 7 · see citing works
Flowers, V., Frutos, C., MacKenzie, A. S., Fanning, R., & Fraser, E. E. (2023) · Citizen Science Theory and Practice · doi:10.5334/cstp.558 · Cited by 5 · see citing works
Pélissié, M., Nabholz, B., Labadessa, R., & Ancillotto, L. (2023) · Journal of Zoology · doi:10.1111/jzo.13086 · Cited by 4 · see citing works
Picón, A., Múgica, D., Egusquiza, I., Bereciartua, A., Romero, J., Jimenez, C. J., Klukas, C., Gómez-Zamanillo, L., Eggers, T., & Navarra-Mestre, R. (2025) · Smart Agricultural Technology · doi:10.1016/j.atech.2024.100761 · Cited by 3 · see citing works
Eijkelboom, I., Schulp, A. S., Amkreutz, L. W. S. W., Verheul, D., Verschoof, W. B., van der Vaart-Verschoof, S., Hogeweg, L., Brunink, D., Mol, D., Peeters, H., et al. (2025) · PeerJ · doi:10.7717/peerj.18927 · Cited by 3 · see citing works
“%) and a hierarchical structure ( e.g ., De Lutio et al., 2021 ) in the dataset.”
InfluentialPlant Species Classification Using Evolving Ensemble and Siamese Networks
Arno, J., Grace, O. M., Larridon, I., & Zhang, L. (2024) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc54092.2024.10831570 · Cited by 3 · see citing works
“Lutio et al. [16] proposed a model for classifying amateur photographs of plant specimens, whereby image meta-data were utilised to provide additional information.”
Lin, C., Chien, H.-Y., & Liu, K.-H. (2025) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2025.3573328 · Cited by 2 · see citing works
“The accuracy of these CNN-based classifications ranged from 89%–95% and appeared more optimistic and stable than ML-based classifications for individual plants [31].”
Advances in machine learning models for plant species identification: A scoping review
AlGemayel, C., Jaoude, D. A., Talhouk, S. N., Issa, I., & Ghassibe, C. (2025) · Ecological Informatics · doi:10.1016/j.ecoinf.2025.103464 · Cited by 2 · see citing works
FlorID – A nationwide identification service for plants from photos and habitat information
Brun, P., de Witte, L. C., Popp, M. R., Zurell, D., Karger, D. N., Descombes, P., de Lutio, R., Wegner, J. D., Bornand, C. N., Eggenberg, S., et al. (2025) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2025.106402 · Cited by 1 · see citing works
Self-citationRank-based deep learning from citizen-science data to model plant communitiesPreprint
Brun, P., Karger, D. N., Zurell, D., Descombes, P., de Witte, L. C., de Lutio, R., Wegner, J. D., & Zimmermann, N. E. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.05.30.542843 · Cited by 1 · see citing works
Self-citationFlorid – a Nationwide Identification Service for Plants from Photos and Habitat InformationPreprint
Brun, P., de Witte, L. C., Popp, M. R., Zurell, D., Karger, D. N., Descombes, P., de Lutio, R., Wegner, J. D., Bornand, C. N., Eggenberg, S., et al. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4830448 · Cited by 1 · see citing works
Self-citationTaxonomy-Aware Evaluation of Vision–Language Models
Snæbjarnarson, V., Du, K., Stoehr, N., Belongie, S., Cotterell, R., Lang, N., & Frank, S. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.00851
“To solve this gap, we have proposed a framework for using hierarchical evaluation measures to evaluate VLMs for FGVC, enabling a more fine-grained view of the specificity and accuracy of these models.”
Hierarchical Recognition of Tree Species
Świetlik, A., Wylazłowska, J., Kopeć, D., & Tomczyk, A. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-04197-5_20
- Authors
- de Lutio, R., She, Y., D’Aronco, S., Russo, S., Brun, P., Wegner, J. D., & Schindler, K.
- Affiliations
- ETH Zurich (CH)
- Swiss Federal Institute for Forest, Snow and Landscape Research (CH)
- University of Zurich (CH)
- Venue
- ISPRS Journal of Photogrammetry and Remote Sensing
- Publisher
- Elsevier BV
- Published
- 2021-12
- Volume
- 182
- Pages
- 112-121
- ISSN
- 0924-2716 (ISSN-L)
- OpenAlex
- W3208771837
- Cited by
- 23
- References
- 30
- Citations by year
- 2026: 1 · 2025: 10 · 2024: 7 · 2023: 3 · 2022: 2
- FWCI
- 1.08
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ISPRS Journal of Photogrammetry and Remote Sensing (publishedVersion, cc-by)
- Zurich Open Repository and Archive (University of Zurich) (acceptedVersion, cc-by)
- DORA WSL (Swiss Federal Institute for Forest, Snow and Landscape Research) (submittedVersion, other-oa)
- Repository for Publications and Research Data (ETH Zurich) (submittedVersion, cc-by)
- Zurich Open Repository and Archive (University of Zurich) (submittedVersion, cc-by)
- ETH Zürich Research Collection (cc-by)
- Zurich Open Repository and Archive (University of Zurich)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Life Sciences › Agricultural and Biological Sciences › Plant Science › Smart Agriculture and AI
- Physical Sciences › Engineering › Media Technology › Remote-Sensing Image Classification
- Keywords
- plant identification, citizen science, multimodal deep learning, species range maps, long-tailed distributions, observer bias, hierarchical taxonomy, species abundance
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
