The DOI matches its Crossref record; 72 of the 72 references it cites were found in a registry or an index.
Distributed temporal graph analytics with GRADOOP
Abstract
Temporal property graphs are graphs whose structure and properties change over time. Temporal graph datasets tend to be large due to stored historical information, asking for scalable analysis capabilities. We give a complete overview of Gradoop, a graph dataflow system for scalable, distributed analytics of temporal property graphs which has been continuously developed since 2005. Its graph model TPGM allows bitemporal modeling not only of vertices and edges but also of graph collections. A declarative analytical language called GrALa allows analysts to flexibly define analytical graph workflows by composing different operators that support temporal graph analysis. Built on a distributed dataflow system, large temporal graphs can be processed on a shared-nothing cluster. We present the system architecture of Gradoop, its data model TPGM with composable temporal graph operators, like snapshot, difference, pattern matching, graph grouping and several implementation details. We evaluate the performance and scalability of selected operators and a composed workflow for synthetic and real-world temporal graphs with up to 283 M vertices and 1.8 B edges, and a graph lifetime of about 8 years with up to 20 M new edges per year. We also reflect on lessons learned from the Gradoop effort.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
MapReduce: simplified data processing on large clusters
Dean, J., & Ghemawat, S. (2008) · Communications of the ACM · doi:10.1145/1327452.1327492 · Cited by 18,476
Maintaining knowledge about temporal intervals
Allen, J. F. (1983) · Communications of the ACM · doi:10.1145/182.358434 · Cited by 7,593
The Hadoop Distributed File System
Shvachko, K., Kuang, H., Radia, S., & Chansler, R. (2010) · 2010 IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST) · doi:10.1109/msst.2010.5496972 · Cited by 4,834
A Bridging Model for Parallel Computation
Valiant, L. G. (1990) · ACM eBooks · doi:10.1145/3749740.3749753 · Cited by 3,664
Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing
Zaharia, M., Chowdhury, M., Das, T., Dave, A., Ma, J. H., McCauley, M., Franklin, M. J. M., Shenker, S. J., & Stoica, I. G. (2012) · Cited by 3,622
Pregel: a system for large-scale graph processing
Malewicz, G., Austern, M. H., Bik, A. J. C., Dehnert, J. C., Horn, I., Leiser, N., & Czajkowski, G. (2010) · Proceedings of the 2010 ACM SIGMOD International Conference on Management of data · doi:10.1145/1807167.1807184 · Cited by 3,515
Bigtable: A Distributed Storage System for Structured Data
Chang, F., Dean, J., Ghemawat, S., Hsieh, W. C., Wallach, D. A., Burrows, M., Chandra, T., Fikes, A., & Gruber, R. E. (2008) · ACM Transactions on Computer Systems · doi:10.1145/1365815.1365816 · Cited by 3,460
Holme, P., & Saramäki, J. (2012) · Physics Reports · doi:10.1016/j.physrep.2012.03.001 · Cited by 2,736
Community detection algorithms: A comparative analysis
Lancichinetti, A., & Fortunato, S. (2009) · Physical Review E · doi:10.1103/physreve.80.056117 · Cited by 2,177
Resource Description Framework (RDF): Concepts and Abstract Syntax
Klyne, G., & Carroll, J. J. (2004) · Cited by 1,927
KNIME - the Konstanz information miner: version 2.0 and beyond
Berthold, M. R., Cebron, N., Dill, F., Gabriel, T. R., Kötter, T., Meinl, T., Ohl, P., Thiel, K., & Wiswedel, B. (2009) · ACM SIGKDD Explorations Newsletter · doi:10.1145/1656274.1656280 · Cited by 1,189
(2015) · KTH Publication Database DiVA (KTH Royal Institute of Technology) · Cited by 1,144
Time-varying graphs and dynamic networks
Casteigts, A., Flocchini, P., Quattrociocchi, W., & Santoro, N. (2012) · International Journal of Parallel, Emergent and Distributed Systems · doi:10.1080/17445760.2012.668546 · Cited by 592
Xin, R., Gonzalez, J. E., Franklin, M. J. M., & Stoica, I. G. (2013) · doi:10.1145/2484425.2484427 · Cited by 581
Cypher: An Evolving Query Language for Property Graphs
Francis, N., Green, A., Guagliardo, P., Libkin, L., Lindaaker, T., Marsault, V., Plantikow, S., Rydberg, M., Selmer, P., & Taylor, A. (2018) · Proceedings of the 2018 International Conference on Management of Data · doi:10.1145/3183713.3190657 · Cited by 545
Foundations of Modern Query Languages for Graph Databases
Angles, R., Arenas, M., Barceló, P., Hogan, A., Reutter, J. L., & Vrgoč, D. (2017) · ACM Computing Surveys · doi:10.1145/3104031 · Cited by 510
The Stratosphere platform for big data analytics
Alexandrov, A., Bergmann, R., Ewen, S., Freytag, J.-C., Hueske, F., Heise, A., Kao, O., Leich, M., Leser, U., Markl, V., et al. (2014) · The VLDB Journal · doi:10.1007/s00778-014-0357-y · Cited by 432
Jensen, C. S., & Snodgrass, R. (1999) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/69.755613 · Cited by 351
Gansner, E. R. (2006) · Cited by 309
The Gremlin graph traversal machine and language (invited talk)
Rodriguez, M. A. (2015) · doi:10.1145/2815072.2815073 · Cited by 299
Path problems in temporal graphs
Wu, H., Cheng, J. S.-C., Huang, S., Ke, Y., Lu, Y., & Xu, Y. (2014) · Proceedings of the VLDB Endowment · doi:10.14778/2732939.2732945 · Cited by 284
A Comparison of Current Graph Database Models
Angles, R. (2012) · 2012 IEEE 28th International Conference on Data Engineering Workshops · doi:10.1109/icdew.2012.31 · Cited by 260
Kostakos, V. (2009) · Physica A: Statistical Mechanics and its Applications · doi:10.1016/j.physa.2008.11.021 · Cited by 254
Kineograph: taking the pulse of a fast-changing and connected world
Cheng, R., Hong, J., Kyrola, A., Miao, Y., Weng, X., Wu, M., Yang, F., Zhou, L., Zhao, F., & Chen, E. (2012) · Proceedings of the 7th ACM european conference on Computer Systems · doi:10.1145/2168836.2168846 · Cited by 247
PGQL: a property graph query language
van Rest, O., Hong, S., Kim, J., Meng, X., & Chafi, H. (2016) · ACM SIGMOD Joint International Workshop on Graph Data Management Experiences & Systems (GRADES) and Network Data Analytics (NDA) (GRADES-NDA) · doi:10.1145/2960414.2960421 · Cited by 192
Kulkarni, K., & Michels, J.-E. (2012) · ACM SIGMOD Record · doi:10.1145/2380776.2380786 · Cited by 186
Iosup, A., Hegeman, T., Ngai, W. L., Heldens, S., Prat-Pèrez, A., Manhardto, T., Chafio, H., Capotă, M., Sundaram, N., Anderson, M. C., et al. (2016) · Proceedings of the VLDB Endowment · doi:10.14778/3007263.3007270 · Cited by 136
Large scale graph processing systems: survey and an experimental evaluation
Batarfi, O., Shawi, R. E., Fayoumi, A. G., Nouri, R., Beheshti, A., Barnawi, A., & Sakr, S. (2015) · Cluster Computing · doi:10.1007/s10586-015-0472-6 · Cited by 116
How Well Do Graph-Processing Platforms Perform? An Empirical Performance Evaluation and Analysis
Guo, Y., Biczak, M., Vărbănescu, A. L., Iosup, A., Martella, C., & Willke, T. L. (2014) · IEEE International Parallel and Distributed Processing Symposium (IPDPS) · doi:10.1109/ipdps.2014.49 · Cited by 106
A performance evaluation of open source graph databases
McColl, R., Ediger, D., Poovey, J. A., Campbell, D., & Bader, D. A. (2014) · Workshop on Parallel Programming for Analytics Applications (PPAA) · doi:10.1145/2567634.2567638 · Cited by 99
Time-Dependent Graphs: Definitions, Applications, and Algorithms
Wang, Y., Yuan, Y., Ma, Y., & Wang, G. (2019) · Data Science and Engineering · doi:10.1007/s41019-019-00105-0 · Cited by 94
Unicorn: a system for searching the social graph
Curtiss, M., Becker, I., Bosman, T., Doroshenko, S., Grijincu, L., Jackson, T., Kunnatur, S., Lassen, S. B., Pronin, P., Sankar, S., et al. (2013) · Proceedings of the VLDB Endowment · doi:10.14778/2536222.2536239 · Cited by 93
The ubiquity of large graphs and surprising challenges of graph processing: extended survey
Sahu, S., Mhedhbi, A., Salihoğlu, S., Lin, J., & ÖZSU, M. T. (2019) · The VLDB Journal · doi:10.1007/s00778-019-00548-x · Cited by 93
Beyond Analytics: The Evolution of Stream Processing Systems
Carbone, P., Fragkoulis, M., Kalavri, V., & Katsifodimos, A. (2020) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/3318464.3383131 · Cited by 85
Community Detection in Dynamic Social Networks Based on Multiobjective Immune Algorithm
Gong, M., Zhang, L., Ma, J., & Jiao, L. (2012) · Journal of Computer Science and Technology · doi:10.1007/s11390-012-1235-y · Cited by 80
ImmortalGraph: A System for Storage and Analysis of Temporal Graphs
Miao, Y., Han, W., Li, K., Wu, M., Yang, F., Zhou, L., Prabhakaran, V., Chen, E., & Chen, W. (2015) · ACM Transactions on Storage · doi:10.1145/2700302 · Cited by 72
Using Link Features for Entity Clustering in Knowledge Graphs
Saeedi, A., Peukert, E., & Rahm, E. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-93417-4_37 · Cited by 62
Self-citationManaging and mining large graphs
Shao, B., Wang, H., & Xiao, Y. (2012) · Proceedings - ACM-SIGMOD International Conference on Management of Data · doi:10.1145/2213836.2213907 · Cited by 57
Management and Analysis of Big Graph Data: Current Systems and Open Challenges
Junghanns, M., Petermann, A., Neumann, M., & Rahm, E. (2017) · Springer eBooks · doi:10.1007/978-3-319-49340-4_14 · Cited by 53
Self-citationPagrol: Parallel graph olap over large-scale attributed graphs
Wang, Z., Fan, Q., Wang, H., Tan, K., Agrawal, D., & Abbadi, A. E. (2014) · Proceedings - International Conference on Data Engineering · doi:10.1109/icde.2014.6816676 · Cited by 47
GEMINI: an integrative healthcare analytics system
Ling, Z. J., Tran, Q. T., Fan, J., Koh, G. C., Nguyen, T. C. L., Tan, C. S., Yip, J., & Zhang, M. (2014) · Proceedings of the VLDB Endowment · doi:10.14778/2733004.2733081 · Cited by 43
Benchmarking graph-processing platforms
Guo, Y., Vărbănescu, A. L., Iosup, A., Martella, C., & Willke, T. L. (2014) · ACM/SPEC International Conference on Performance Engineering (ICPE) · doi:10.1145/2568088.2576761 · Cited by 42
Bitemporal Data: Theory and Practice
Johnston, T. (2014) · Cited by 39
Raphtory: Streaming analysis of distributed temporal graphs
Steer, B. A., Cuadrado, F., & Clegg, R. G. (2019) · Future Generation Computer Systems · doi:10.1016/j.future.2019.08.022 · Cited by 38
Stateful functions as a service in action
Akhter, A., Fragkoulis, M., & Katsifodimos, A. (2019) · Proceedings of the VLDB Endowment · doi:10.14778/3352063.3352092 · Cited by 36
Analyzing extended property graphs with Apache Flink
Junghanns, M., Petermann, A., Teichmann, N., Gómez, K., & Rahm, E. (2016) · doi:10.1145/2980523.2980527 · Cited by 35
Self-citationScalable Matching and Clustering of Entities with FAMER
Saeedi, A., Nentwig, M., Peukert, E., & Rahm, E. (2018) · Complex Systems Informatics and Modeling Quarterly · doi:10.7250/csimq.2018-16.04 · Cited by 35
Self-citationAn Algebra and Equivalences to Transform Graph Patterns in Neo4j
Hölsch, J., & Grossniklaus, M. (2016) · KOPS (University of Konstanz) · Cited by 28
Moffitt, V. Z., & Stoyanovich, J. (2017) · doi:10.1145/3122831.3122838 · Cited by 28
Incremental Multi-source Entity Resolution for Knowledge Graph Completion
Saeedi, A., Peukert, E., & Rahm, E. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-49461-2_23 · Cited by 28
Self-citationHuang, H., Song, J., Lin, X., Ma, S., & Huai, J. (2016) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/2983323.2983335 · Cited by 26
Cypher-based Graph Pattern Matching in Gradoop
Junghanns, M., Kießling, M., Averbuch, A., Petermann, A., & Rahm, E. (2017) · doi:10.1145/3078447.3078450 · Cited by 21
Self-citationTink: A Temporal Graph Analytics Library for Apache Flink
Lightenberg, W., Pei, Y., Fletcher, G., & Pechenizkiy, M. (2018) · International World Wide Web Conference (WWW) · doi:10.1145/3184558.3186934 · Cited by 21
An Interval-centric Model for Distributed Computing over Temporal Graphs
Gandhi, S., & Simmhan, Y. L. (2020) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde48307.2020.00102 · Cited by 21
Practice of Streaming and Dynamic Graphs: Concepts, Models, Systems, and ParallelismPreprint
Besta, M., Fischer, M., Kalavri, V., Kapralov, M., & Hoefler, T. (2020) · arXiv (Cornell University) · doi:10.3929/ethz-b-000462546 · Cited by 20
GRADOOP: Scalable Graph Data Management and Analytics with HadoopPreprint
Junghanns, M., Petermann, A., Gómez, K., & Rahm, E. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.00548 · Cited by 19
Self-citationBIIIG: Enabling business intelligence with integrated instance graphs
Petermann, A., Junghanns, M., Müller, R., & Rahm, E. (2014) · IEEE International Conference on Data Engineering Workshops (ICDEW) · doi:10.1109/icdew.2014.6818294 · Cited by 19
Self-citationTime Traveling in Graphs using a Graph Database.
Semertzidis, K., & Pitoura, E. (2016) · Cited by 19
Constructions from Dots and LinesPreprint
Rodriguez, M. A., & Neubauer, P. B. (2010) · arXiv (Cornell University) · doi:10.48550/arxiv.1006.2361 · Cited by 18
Rudolf, M., Paradies, M., Bornhövd, C., & Lehner, W. (2013) · doi:10.1145/2484425.2484441 · Cited by 16
Erb, B., Meißner, E., Pietron, J., & Kargl, F. (2017) · ACM International Conference on Distributed and Event-Based Systems (DEBS) · doi:10.1145/3093742.3093913 · Cited by 15
Distributed Grouping of Property Graphs with GRADOOP
Junghanns, M., Petermann, A., & Rahm, E. (2017) · BTW · Cited by 14
Self-citationPetermann, A., Junghanns, M., & Rahm, E. (2017) · IEEE/ACM International Conference on Big Data Computing, Applications and Technologies (BDCAT) · doi:10.1145/3148055.3148064 · Cited by 12
Self-citationAnalyzing Temporal Graphs with Gradoop
Rost, C., Thor, A., & Rahm, E. (2019) · Datenbank-Spektrum · doi:10.1007/s13222-019-00325-8 · Cited by 8
Self-citationGraph Theory and Analysis of Biological Data in Computational Biology
Chao, S.-Y. (2009) · InTech eBooks · doi:10.5772/8205 · Cited by 7
Zooming Out on an Evolving Graph
Aghasadeghi, A., Moffitt, V. Z., Schelter, S., & Stoyanovich, J. (2020) · Movebank · doi:10.5441/002/edbt.2020.04 · Cited by 6
Knowledge Graph Completion with FAMER (DI2KG Challenge Winner).
Obraczka, D., Saeedi, A., & Rahm, E. (2019) · Knowledge Discovery and Data Mining · Cited by 4
Self-citationKricke, M., Peukert, E., & Rahm, E. (2019) · Gesellschaft für Informatik (GI) · doi:10.18420/btw2019-12 · Cited by 3
Self-citationBIGGR: Bringing Gradoop to Applications
Rostami, M. A., Kricke, M., Peukert, E., Kühne, S., Wilke, M., Dienst, S., & Rahm, E. (2019) · Datenbank-Spektrum · doi:10.1007/s13222-019-00306-x · Cited by 2
Self-citationEvolution Analysis of Large Graphs with Gradoop
Rost, C., Thor, A., Fritzsche, P., Gómez, K., & Rahm, E. (2020) · Communications in computer and information science · doi:10.1007/978-3-030-43823-4_33 · Cited by 2
Self-citationGraph Sampling with Distributed In-Memory Dataflow SystemsPreprint
Gómez, K., Täschner, M., Rostami, M. A., Rost, C., & Rahm, E. (2019) · arXiv (Cornell University) · doi:10.18420/btw2021-15 · Cited by 2
Self-citationRostami, M. A., Peukert, E., Wilke, M., & Rahm, E. (2019) · Gesellschaft für Informatik (GI) · doi:10.18420/btw2019-45 · Cited by 1
Self-citation
Research works printed but not linked: 4
- Angles, R.: The property graph database model. In: AMW, volume 2100 of CEUR Workshop Proceedings. CEUR-WS.org, (2018)
- Kricke, M., Peukert, E., Rahm, E.: Graph Data Transformations in Gradoop. In: BTW, volume P-289 of LNI, pp. 193–202, (2019)
- Obraczka, D., Saeedi, A., Rahm, E.: Knowledge graph completion with FAMER. In: Proc. KDD DI2KG workshop, (2019)
- Rostami, M.A., Peukert, E., Wilke, M., Rahm, E.: Big graph analysis by visually created workflows. In: Grust, T., Naumann, F., Böhm, A., Lehner, W., Härder, T., Rahm, E., Heuer, A., Klettke, M., Meyer, H. editors, BTW 2019, pp. 559–563, (2019)
Metadata
- Authors
- Rost, C., Gomez, K., Täschner, M., Fritzsche, P., Schons, L., Christ, L., Adameit, T., Junghanns, M., & Rahm, E.
- Venue
- The VLDB Journal
- Published
- 2021-05-19
- Volume
- 31
- Issue
- 2
- Pages
- 375-401
- ISSN
- 1066-8888
- 0949-877X
- Cited by
- 39
- References
- 72
- Language
- en
- Licence
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Funders
- Bundesministerium für Bildung und Forschung (BMBF 01IS18026B)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.