The DOI matches its Crossref record; 44 of the 44 references it cites were found in a registry or an index.
Compressed representations of sequences and full-text indexes
Abstract
Given a sequence S = s 1 s 2 … s n of integers smaller than r = O (polylog( n )), we show how S can be represented using nH 0 ( S ) + o ( n ) bits, so that we can know any s q , as well as answer rank and select queries on S , in constant time. H 0 ( S ) is the zero-order empirical entropy of S and nH 0 ( S ) provides an information-theoretic lower bound to the bit storage of any sequence S via a fixed encoding of its symbols. This extends previous results on binary sequences, and improves previous results on general sequences where those queries are answered in O (log r ) time. For larger r , we can still represent S in nH 0 ( S ) + o ( n log r ) bits and answer queries in O (log r /log log n ) time. Another contribution of this article is to show how to combine our compressed representation of integer sequences with a compression boosting technique to design compressed full-text indexes that scale well with the size of the input alphabet Σ. Specifically, we design a variant of the FM-index that indexes a string T [1, n ] within nH k ( T ) + o ( n ) bits of storage, where H k ( T ) is the k th-order empirical entropy of T . This space bound holds simultaneously for all k ≤ α log |Σ| n , constant 0 < α < 1, and |Σ| = O (polylog( n )). This index counts the occurrences of an arbitrary pattern P [1, p ] as a substring of T in O ( p ) time; it locates each pattern occurrence in O (log 1+ε n ) time for any constant 0 < ε < 1; and reports a text substring of length ℓ in O (ℓ + log 1+ε n ) time. Compared to all previous works, our index is the first that removes the alphabet-size dependance from all query times, in particular, counting time is linear in the pattern length. Still, our index uses essentially the same space of the k th-order entropy of the text T , which is the best space obtained in previous work. We can also handle larger alphabets of size |Σ| = O ( n β ), for any 0 < β < 1, by paying o ( n log|Σ|) extra space and multiplying all query times by O (log |Σ|/log log n ).
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Lecture Notes in Computer Science 1205
(1999) · Industrial Robot: An International Journal · doi:10.1108/ir.1999.04926fae.001 · Cited by 38,662
Information Retrieval: Data Structures and Algorithms
Frakes, W. B., & Baeza-Yates, R. A. (1992) · Cited by 2,757
A Block-sorting Lossless Data Compression Algorithm
Burrows, M. T., & Wheeler, D. J. (1994) · Cited by 2,333
Suffix Arrays: A New Method for On-Line String Searches
Manber, U., & Myers, G. (1993) · SIAM Journal on Computing · doi:10.1137/0222058 · Cited by 1,915
Opportunistic data structures with applications
Ferragina, P., & Manzini, G. (2002) · Proceedings 41st Annual Symposium on Foundations of Computer Science · doi:10.1109/sfcs.2000.892127 · Cited by 1,063
Self-citationFerragina, P., & Manzini, G. (2005) · Journal of the ACM · doi:10.1145/1082036.1082039 · Cited by 686
Self-citationHigh-order entropy-compressed text indexes
Grossi, R., Gupta, A. K., & Vitter, J. S. (2003) · doi:10.5555/644108.644250 · Cited by 648
Space-efficient static trees and graphs
Jacobson, G. (1989) · 30th Annual Symposium on Foundations of Computer Science · doi:10.1109/sfcs.1989.63533 · Cited by 634
(1995) · Choice Reviews Online · doi:10.5860/choice.32-5696 · Cited by 582
Compressed Suffix Arrays and Suffix Trees with Applications to Text Indexing and String Matching
Grossi, R., & Vitter, J. S. (2005) · SIAM Journal on Computing · doi:10.1137/s0097539702402354 · Cited by 431
Succinct indexable dictionaries with applications to encoding k-ary trees and multisets
Raman, R., Raman, V., & Satti, S. R. (2002) · doi:10.5555/545381.545411 · Cited by 387
An analysis of the Burrows—Wheeler transform
Manzini, G. (2001) · Journal of the ACM · doi:10.1145/382780.382782 · Cited by 375
Self-citationProceedings of the 2021 ACM-SIAM Symposium on Discrete Algorithms (SODA)
(2021) · Society for Industrial and Applied Mathematics eBooks · doi:10.1137/1.9781611976465 · Cited by 288
New indices for text: PAT Trees and PAT arrays
Gonnet, G. H., Baeza-Yates, R. A., & Snider, T. (1992) · Information Retrieval · Cited by 272
Clark, D. R. (1998) · Cited by 269
New text indexing functionalities of the compressed suffix arrays
Sadakane, K. (2003) · Journal of Algorithms · doi:10.1016/s0196-6774(03)00087-7 · Cited by 233
Rank/select operations on large alphabets: a tool for text indexing
Golynski, A., Munro, J. I., & Satti, S. R. (2006) · Symposium on Discrete Algorithms · doi:10.5555/1109557.1109599 · Cited by 182
Succinct Suffix Arrays Based on Run-Length Encoding
Mäkinen, V., & Navarro, G. (2005) · Lecture Notes in Computer Science · doi:10.1007/11496656_5 · Cited by 166
Self-citationAn experimental study of an opportunistic index
Ferragina, P., & Manzini, G. (2001) · doi:10.5555/365411.365458 · Cited by 165
Self-citationSuccinct representation of balanced parentheses, static trees and planar graphs
Munro, J. I., & Raman, V. (2002) · Annual Symposium on Foundations of Computer Science · doi:10.1109/sfcs.1997.646100 · Cited by 158
Succinct representations of lcp information and improvements in the compressed suffix arrays
Sadakane, K. (2002) · doi:10.5555/545381.545410 · Cited by 134
Boosting textual compression in optimal linear time
Ferragina, P., Giancarlo, R., Manzini, G., & Sciortino, M. (2005) · Journal of the ACM · doi:10.1145/1082036.1082043 · Cited by 128
Self-citationCompressed Text Databases with Efficient Query Algorithms Based on the Compressed Suffix Array
Sadakane, K. (2000) · Lecture notes in computer science · doi:10.1007/3-540-40996-3_35 · Cited by 124
Indexing text using the Ziv–Lempel trie
Navarro, G. (2004) · Journal of Discrete Algorithms · doi:10.1016/s1570-8667(03)00066-2 · Cited by 115
Self-citationFerragina, P., Manzini, G., Mäkinen, V., & Navarro, G. (2004) · Lecture notes in computer science · doi:10.1007/978-3-540-30213-1_23 · Cited by 110
Self-citationSqueezing succinct data structures into entropy bounds
Sadakane, K., & Grossi, R. (2006) · Symposium on Discrete Algorithms · doi:10.5555/1109557.1109693 · Cited by 109
When indexing equals compression: Experiments with compressing suffix arrays and applications
Foschini, L., Grossi, R., Gupta, A. K., & Vitter, J. S. (2006) · ACM Transactions on Algorithms · doi:10.1145/1198513.1198521 · Cited by 81
Annotating Large Genomes With Exact Word MatchesPreprint
Healy, J., Thomas, E. E., Schwartz, J. T., & Wigler, M. H. (2003) · Genome Research · doi:10.1101/gr.1350803 · Cited by 70
Low Redundancy in Static Dictionaries with O(1) Worst Case Lookup Time
Pagh, R. (1999) · Lecture notes in computer science · doi:10.1007/3-540-48523-6_56 · Cited by 58
Approximate string matching using compressed suffix arrays
Huynh, T., Hon, W.-K., Lam, T., & Sung, W.-K. K. (2005) · Theoretical Computer Science · doi:10.1016/j.tcs.2005.11.022 · Cited by 50
An experimental study of a compressed index
Ferragina, P., & Manzini, G. (2001) · Information Sciences · doi:10.1016/s0020-0255(01)00098-6 · Cited by 48
Self-citationCompressed Compact Suffix Arrays
Mäkinen, V., & Navarro, G. (2004) · Lecture notes in computer science · doi:10.1007/978-3-540-27801-6_32 · Cited by 42
Self-citationMäkinen, V., Navarro, G., & Sadakane, K. (2004) · Lecture notes in computer science · doi:10.1007/978-3-540-30551-4_59 · Cited by 34
Self-citationIndexing huge genome sequences for solving various problems.
Sadakane, K., & Shibuya, T. (2001) · PubMed · Cited by 34
A Simple Optimal Representation for Balanced Parentheses
Geary, R. F., Rahman, N., Raman, R., & Raman, V. (2004) · Lecture notes in computer science · doi:10.1007/978-3-540-27801-6_12 · Cited by 32
Compressed Index for a Dynamic Collection of Texts
Chan, H.-L., Hon, W.-K., & Lam, T. (2004) · Lecture notes in computer science · doi:10.1007/978-3-540-27801-6_34 · Cited by 31
Mäkinen, V. (2000) · Lecture notes in computer science · doi:10.1007/3-540-45123-4_26 · Cited by 26
Self-citationCompression boosting in optimal linear time using the Burrows-Wheeler Transform
Ferragina, P., & Manzini, G. (2004) · doi:10.5555/982792.982892 · Cited by 26
Self-citationOn compressing and indexing data
Ferragina, P., & Manzini, G. (2002) · UnipiEprints Open Archive (Università di Pisa) · Cited by 24
Self-citationLow Redundancy in Dictionaries with O(1) Worst Case Lookup Time
Pagh, R. (1998) · BRICS Report Series · doi:10.7146/brics.v5i28.19434 · Cited by 24
A SIMPLE ALPHABET-INDEPENDENT FM-INDEX
Grabowski, S., Navarro, G., Przywarski, R., Salinger, A., & Mäkinen, V. (2006) · International Journal of Foundations of Computer Science · doi:10.1142/s0129054106004467 · Cited by 20
Self-citationOptimal Partitions of Strings: A New Class of Burrows-Wheeler Compression Algorithms
Giancarlo, R., & Sciortino, M. (2003) · Lecture notes in computer science · doi:10.1007/3-540-44888-8_10 · Cited by 16
Compressed index for dynamic text
Hon, W.-K., Lam, T., Sadakane, K., Sung, W.-K. K., & Yiu, S. (2004) · DCC · doi:10.1109/dcc.2004.1281455 · Cited by 11
Ferragina, P., & Manzini, G. (2016) · Springer eBooks · doi:10.1007/978-1-4939-2864-4_54 · Cited by 5
Self-citation
Cited by
Li, D., Luo, R., Liu, C.-M., Leung, C.-M., Ting, H.-F., Sadakane, K., Yamashita, H., & Lam, T.-W. (2016) · Methods · doi:10.1016/j.ymeth.2016.02.020 · Cited by 2,079
Ligra: a lightweight graph processing framework for shared memory
Shun, J., & Blelloch, G. E. (2013) · Proceedings of the 18th ACM SIGPLAN symposium on Principles and practice of parallel programming · doi:10.1145/2442516.2442530 · Cited by 798
Navarro, G., & Mäkinen, V. (2007) · ACM Computing Surveys · doi:10.1145/1216370.1216372 · Cited by 735
Self-citationClaude, F., & Navarro, G. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-34109-0_18 · Cited by 47
Self-citationFarruggia, A., Gagie, T., Navarro, G., Puglisi, S. J., & Sirén, J. (2018) · The Computer Journal · doi:10.1093/comjnl/bxx108 · Cited by 14
Self-citation
Metadata
- Authors
- Ferragina, P., Manzini, G., Mäkinen, V., & Navarro, G.
- Venue
- ACM Transactions on Algorithms
- Published
- 2007-05
- Volume
- 3
- Issue
- 2
- Pages
- 20
- ISSN
- 1549-6325 (ISSN-L)
- 1549-6333
- OpenAlex
- W2107082304
- Cited by
- 373
- References
- 44
- FWCI
- 40.98
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.