Towards a holistic approach to auto-parallelization: integrating profile-driven parallelism detection and machine-learning based mapping
The DOI matches its Crossref record; 26 of the 26 references it cites were found in a registry or an index.
Abstract
Compiler-based auto-parallelization is a much studied area, yet has still not found wide-spread application. This is largely due to the poor exploitation of application parallelism, subsequently resulting in performance levels far below those which a skilled expert programmer could achieve. We have identified two weaknesses in traditional parallelizing compilers and propose a novel, integrated approach, resulting in significant performance improvements of the generated parallel code. Using profile-driven parallelism detection we overcome the limitations of static analysis, enabling us to identify more application parallelism and only rely on the user for final approval. In addition, we replace the traditional target-specific and inflexible mapping heuristics with a machine-learning based prediction mechanism, resulting in better mapping decisions while providing more scope for adaptation to different target architectures. We have evaluated our parallelization strategy against the NAS and SPEC OMP benchmarks and two different multi-core platforms (dual quad-core Intel Xeon SMP and dual-socket QS20 Cell blade). We demonstrate that our approach not only yields significant improvements when compared with state-of-the-art parallelizing compilers, but comes close to and sometimes exceeds the performance of manually parallelized codes. On average, our methodology achieves 96% of the performance of the hand-tuned OpenMP NAS and SPEC parallel benchmarks on the Intel Xeon platform and gains a significant speedup for the IBM Cell platform, demonstrating the potential of profile-guided and machine-learning based parallelization for complex multi-core platforms.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A training algorithm for optimal margin classifiers
Boser, B. E., Guyon, I. M., & Vapnik, V. N. (1992) · Proceedings of the fifth annual workshop on Computational learning theory · doi:10.1145/130385.130401 · Cited by 11,992
A Bridging Model for Parallel Computation
Valiant, L. G. (1990) · ACM eBooks · doi:10.1145/3749740.3749753 · Cited by 3,664
Bailey, D. H., Barszcz, E., Barton, J. T., Browning, D. S., Carter, R. L., Dagum, L., Fatoohi, R. A., Frederickson, P. O., Lasinski, T. A., Schreiber, R. S., et al. (1991) · The International Journal of Supercomputing Applications · doi:10.1177/109434209100500306 · Cited by 2,395 · cited 1×
“More specifically, we have used the NAS NPB (sequential v.2.3) and NPB (OpenMP v.2.3) codes ( 26 ) alongside the SPEC CPU2000 benchmarks and their corresponding SPEC OMP2001”
MethodInfluentialOptimizing Compilers for Modern Architectures: A Dependence-based Approach
Kennedy, K., & Allen, J. R. (2001) · Cited by 1,002
The parallel execution of DO loops
Lamport, L. (1974) · Communications of the ACM · doi:10.1145/360827.360844 · Cited by 611 · cited 3×
“sum0 += A[ Anext ] [ 0 ] [ 0 ] ∗ v [ c o l ] [ 0 ] + A[ Anext ] [ 0 ] [ 1 ] ∗ v [ c o l ] [ 1 ] + A[ Anext ] [ 0 ] [ 2 ] ∗ v [ c o l ] [ 2 ] ; sum1 += .”
Maximizing multiprocessor performance with the SUIF compiler
Hall, M. W., Anderson, J. M., Amarasinghe, S. P., Murphy, B. R., Liao, S.-W., Bugnion, E., & Lam, M. S. (1996) · Computer · doi:10.1109/2.546613 · Cited by 572 · cited 1×
“Polaris (6), SUIF-1 ( 7 ) and, more recently, Open64 (8).”
Optimistic parallelism requires abstractions
Kulkarni, M., Pingali, K., Walter, B., Ramanarayanan, G., Bala, K., & Chew, L. P. (2007) · Proceedings of the 28th ACM SIGPLAN Conference on Programming Language Design and Implementation · doi:10.1145/1250734.1250759 · Cited by 441 · cited 1×
“Complementary to the on-going work in auto-parallelization many high-level parallel programming languages ‐ such as Cilk-5 (9), OpenMP, StreamIt (10), UPC (11) and X10 (12) ‐ and programming models ‐ such as Galois ( 14 ), STAPL (15) and HTA (16) ‐ have been proposed.”
BackgroundA stream compiler for communication-exposed architectures
Gordon, M. I., Thies, W., Karczmarek, M., Lin, J., Meli, A. S., Lamb, A. A., Leger, C., Wong, J., Hoffmann, H., Maze, D., et al. (2002) · Proceedings of the 10th international conference on Architectural support for programming languages and operating systems · doi:10.1145/605397.605428 · Cited by 337 · cited 2×
“Complementary to the on-going work in auto-parallelization many high-level parallel programming languages ‐ such as Cilk-5 (9), OpenMP, StreamIt ( 10 ), UPC (11) and X10 (12) ‐ and programming models ‐ such as Galois (14), STAPL (15) and HTA (16) ‐ have been proposed.”
BackgroundOptimizing for reduced code space using genetic algorithms
Cooper, K. D., Schielke, P. J., & Subramanian, D. (1999) · ACM SIGPLAN/SIGBED Conference on Languages, Compilers, and Tools for Embedded Systems (LCTES) · doi:10.1145/314403.314414 · Cited by 321 · cited 1×
“For example, Cooper et al. ( 45 ) develop a technique to find “good” compiler optimization sequences for code size reduction.”
MethodSPEComp: A New Benchmark Suite for Measuring Parallel Computer Performance
Aslot, V., Domeika, M., Eigenmann, R., Gaertner, G., Jones, W. B., & Parady, B. (2001) · Lecture notes in computer science · doi:10.1007/3-540-44587-0_1 · Cited by 255 · cited 1×
“Investigations into the caus es of this behavior have revealed that the SPEC OMP codes are not equivalent to the sequential SPEC programs, but have been manually restructured ( 29 ).”
BackgroundInterprocedural dependence analysis and parallelization
Burke, M. G., & Cytron, R. K. (2004) · ACM SIGPLAN Notices · doi:10.1145/989393.989411 · Cited by 219 · cited 2×
“Automatic Parallelization. Static automatic parallelism extraction has been achieved on restricted DOALL and DOACROSS loops ( 3 ; 4; 5). Unfortunately, many parallelization oppor tunities could still not be discovered by a static analysis approach due to lack of information at the source code level.”
BackgroundMethodSoftware behavior oriented parallelization
Ding, C., Shen, X., Kelsey, K., Tice, C., Huang, R., & Zhang, C. (2007) · ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI) · doi:10.1145/1250734.1250760 · Cited by 190 · cited 1×
“Similarly, the Software Behavior-Oriented Parallelization ( 38 ) system allows the programmer to specify intended parallelism.”
BackgroundMaximizing parallelism and minimizing synchronization with affine transforms
Lim, A. W., & Lam, M. S. (1997) · Proceedings of the 24th ACM SIGPLAN-SIGACT symposium on Principles of programming languages - POPL '97 · doi:10.1145/263699.263719 · Cited by 179 · cited 2×
“Automatic Parallelization. Static automatic parallelism extraction has been achieved on restricted DOALL and DOACROSS loops (3; 4; 5 ). Unfortunately, many parallelization oppor tunities could still not be discovered by a static analysis approach due to lack of information at the source code level.”
BackgroundMethodSelf-calibrating Online Wearout Detection
Blome, J. A., Feng, S., Gupta, S., & Mahlke, S. A. (2007) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro.2007.35 · Cited by 171 · cited 1×
“Matthew et al. ( 33 ) have manually parallelized the SPECINT-2000 benchmarks with thread level speculation.”
MethodSUIF Explorer: an interactive and interprocedural parallelizer
Liao, S.-W., Ζ, A. D., Bosch, R. P., Ghuloum, A. M., & Lam, M. S. (1999) · ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming · doi:10.1145/301104.301108 · Cited by 160 · cited 2×
“For example, SUIF Explorer ( 13 ) helps the programmer to identify those loops that are likely to be parallelizable and assists the user in c hecking for correctness.”
BackgroundIterative optimization in the polyhedral model: part ii, multidimensional time
Pouchet, L.-N., Bastoul, C., Cohen, A., & Cavazos, J. (2008) · ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI) · doi:10.1145/1375581.1375594 · Cited by 147
Maximizing parallelism and minimizing synchronization with affine partitions
Lim, A. W., & Lam, M. S. (1998) · Parallel Computing · doi:10.1016/s0167-8191(98)00021-0 · Cited by 147
(2007) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro.2007.7 · Cited by 144 · cited 1×
“In ( 39 ), programmers mark potential parallel regions of the program, and the tool uses dynamic profiling information to find a good mapping of parallel candidates.”
MethodThe Jrpm system for dynamically parallelizing Java programs
Chen, M. K., & Olukotun, K. (2003) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1145/859618.859668 · Cited by 143
A performance analysis of the Berkeley UPC compiler
Chen, W.-Y., Bonachea, D., Duell, J., Husbands, P., Iancu, C., & Yelick, K. (2003) · International Conference on Supercomputing · doi:10.1145/782814.782825 · Cited by 142 · cited 1×
“Complementary to the on-going work in auto-parallelization many high-level parallel programming languages ‐ such as Cilk-5 (9), OpenMP, StreamIt (10), UPC ( 11 ) and X10 (12) ‐ and programming models ‐ such as Galois (14), STAPL (15) and HTA (16) ‐ have been proposed.”
BackgroundSemantical interprocedural parallelization: an overview of the PIPS project
Irigoin, F., Jouvelot, P., & Triolet, R. (2014) · International Conference on Supercomputing · doi:10.1145/2591635.2667163 · Cited by 137 · cited 2×
“Interactive parallelization tools ( 17 ; 18; 19; 20) provide a way to actively involve the programmer in the detection and mapping of application parallelism, but still demand great effort from the user.”
BackgroundThreaded multiple path execution
Wallace, S., Calder, B., & Tullsen, D. M. (1998) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1145/279358.279392 · Cited by 133
Interactive parallel programming using the ParaScope Editor
Kennedy, K., McKinley, K. S., & Tseng, C. (1991) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/71.86108 · Cited by 113 · cited 2×
“Interactive Parallelization. Interactive parallelization tools (13; 17; 18 ; 19; 20) provide a way to actively involve the programmer in the detection and mapping of application parallelism.”
BackgroundInterprocedural dependence analysis and parallelization
Burke, M., & Cytron, R. K. (1986) · ACM SIGPLAN Notices · doi:10.1145/13310.13328 · Cited by 69
Sensitivity analysis for automatic parallelization on multi-cores
Rus, S., Pennings, M., & Rauchwerger, L. (2007) · International Conference on Supercomputing · doi:10.1145/1274971.1275008 · Cited by 50 · cited 1×
“Dynamic Parallelization Rus et al. ( 34 ) applied sensitivity analysis to automatic parallelize programs whose behaviors may be sensitive to input data sets.”
BackgroundSUIF Explorer: an interactive and interprocedural parallelizer
Liao, S.-W., Ζ, A. D., Bosch, R. P., Ghuloum, A. M., & Lam, M. S. (1999) · ACM SIGPLAN Notices · doi:10.1145/329366.301108 · Cited by 41
Cited by
Milepost GCC: Machine Learning Enabled Self-tuning Compiler
Fursin, G., Kashnikov, Y., Memon, A. W., Chamski, Z., Temam, O., Namolaru, M., Yom-Tov, E., Mendelson, B., Zaks, A., Courtois, E., et al. (2011) · International Journal of Parallel Programming · doi:10.1007/s10766-010-0161-2 · Cited by 234
“We started combining Milepost technology with machine-learning based autoparallelization and predictive scheduling techniques [60,51,76].”
MethodSelf-citationAdaptive deep learning model selection on embedded systems
Taylor, B., Marco, V. S., Wolff, W., Elkhatib, Y., & Wang, Z. (2018) · Proceedings of the 19th ACM SIGPLAN/SIGBED International Conference on Languages, Compilers, and Tools for Embedded Systems · doi:10.1145/3211332.3211336 · Cited by 90
Self-citationLearning Performance-Improving Code Edits
Shypula, A., Madaan, A., Zeng, Y., Alon, U., Gardner, J., Hashemi, M., Neubig, G., Ranganathan, P., Bastani, O., & Yazdanbakhsh, A. (2023) · arXiv · doi:10.48550/arxiv.2302.07867 · Cited by 146
Metadata
- Authors
- Tournavitis, G., Wang, Z., Franke, B., & O'Boyle, M. F. P.
- Venue
- Proceedings of the 30th ACM SIGPLAN Conference on Programming Language Design and Implementation
- Publisher
- ACM
- Published
- 2009-06-15
- Pages
- 177-187
- ISSN
- 1531-7102 (ISSN-L)
- OpenAlex
- W2056579078
- Cited by
- 204
- References
- 26
- FWCI
- 13.72
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.