CoChat
Sign Up
Cited by 12Software Engineering Techniques and PracticesSoftware Testing and Debugging TechniquesSoftware System Performance and Reliability

Test Automation Process Improvement in a DevOps Team: Experience Report

Open access
Wang, Yuqing, Pyhajarvi, Maaret, Mantyla, Mika V.·2020·2020 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)
DOI 10.1109/icstw50294.2020.00057
Open this paper in CoChat — free accountOpen PDF

Abstract

How to successfully conduct test automation process improvement (TAPI) for continuous development, consisting of iterative software development, continuous testing, and delivery, is the challenge faced by many software organizations. In this paper, we present an experience report on TAPI in one DevOps team in F-Secure (a Finnish software company). The team builds Windows application software and exists in F-Secure's TAPI culture. The team self-reports high satisfaction and maturity in test automation for continuous development. To study their TAPI, we reviewed a collection of experience notes, team reflection reports and telemetry result reports. Then several meetings were held to discuss the details. We found that based on the understanding of the team, test automation maturity for continuous development is defined as a set of indicators, e.g., the increasing speed to release, improving the productivity of the team, high test efficiency. Second, the team indicated that a set of critical success factors have a major impact on successfully carrying out its TAPI, e.g., incremental approach, the whole team effort, test tool choice and architecture, telemetry. Third, we compare the TAPI practices in the observed team with the practices described in prior literature. The team believes that the existing test automation maturity approaches should include the identified practices like the whole team effort to build a more comprehensive test automation improvement model for the software industry.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/icstw50294.2020.00057

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The testing cycle can be shortened by utilising the most recent tools and services.

    “They discovered that the testing cycle can be shortened by utilising the most recent tools and services.”

    From Challenges and Solutions in Implementing Continuous Integration and Continuous Testing for Agile Quality Assurance (2023) · cites this paper as [112]

    The perceived success factors section (p. 5) reports that a test-lab tool provisioned environments in five seconds and that the team replaced test tools when better options appeared. This supports a narrower point about faster test setup and adaptable tool choice, but the paper does not establish that using the most recent tools and services shortened the overall testing cycle. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 22

  1. [1][1] J. Kroll, I. Richardson, R. Prikladnicki, and J. L. Audy, 'Empirical evidence in follow the sun software development: A systematic mapping study,' Information and Software Technology , vol. 93, pp. 30-44, 2018. -
  2. [2][2] D. M. Rafi, K. R. K. Moses, K. Petersen, and M. V. M¨ antyl¨ a, 'Benefits and limitations of automated software testing: Systematic literature review and practitioner survey,' in Proceedings of the 7th International Workshop on Automation of Software Test . IEEE Press, 2012, pp. 3642. -
  3. [3][3] K. Wiklund, S. Eldh, D. Sundmark, and K. Lundqvist, 'Technical debt in test automation,' in 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation . IEEE, 2012, pp. 887892. -
  4. [5][5] S. Eldh, K. Andersson, A. Ermedahl, and K. Wiklund, 'Towards a test automation improvement model (taim),' in 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation Workshops . IEEE, 2014, pp. 337-342. -
  5. [6][6] A. Furtado, S. Meira, and M. Gomes, 'Towards a maturity model in software testing automation,' in The Ninth International Conference on Software Engineering Advances , 2014, pp. 282-285. -
  6. [8][8] J. Kasurinen, O. Taipale, and K. Smolander, 'Software test automation in practice: empirical observations,' Advances in Software Engineering , vol. 2010, 2010. -
  7. [11][11] Y. Wang, M. M¨ antyl¨ a, S. Eldh, J. Markkula, K. Wiklund, T. Kairi, P. Raulamo-Jurvanen, and A. Haukinen, 'A self-assessment instrument for assessing test automation maturity,' in Proceedings of the Evaluation and Assessment on Software Engineering . ACM, 2019, pp. 145-154. -
  8. [12][12] Y. Wang, 'Test automation maturity assessment,' in 2018 IEEE 11th International Conference on Software Testing, Verification and Validation (ICST) . IEEE, 2018, pp. 424-425. -
  9. [13][13] V. Garousi, M. Felderer, and T. Hacalo˘ glu, 'Software test maturity assessment and test process improvement: A multivocal literature review,' Information and Software Technology , vol. 85, pp. 16-42, 2017. -
  10. [14][14] K. Hrabovsk´ a, B. Rossi, and T. Pitner, 'Software testing process models benefits & drawbacks: a systematic literature review,' arXiv preprint arXiv:1901.01450 , 2019. - · arXiv
  11. [15][15] P. Pocatilu, 'Automated software testing process,' Economy Informatics , vol. 1, pp. 97-99, 2002. -
  12. [16][16] V. Garousi and F. Elberzhager, 'Test automation: not just for test execution,' IEEE Software , vol. 34, no. 2, pp. 90-96, 2017. -
  13. [21][21] K. Karhu, T. Repo, O. Taipale, and K. Smolander, 'Empirical observations on software testing automation,' in 2009 International Conference on Software Testing Verification and Validation . IEEE, 2009, pp. 201209. -
  14. [23][23] D. Graham and M. Fewster, Experiences of test automation: case studies of software test automation . Addison-Wesley Professional, 2012. -
  15. [25][25] D. S. Cruzes and T. Dyba, 'Recommended steps for thematic synthesis in software engineering,' in 2011 International Symposium on Empirical Software Engineering and Measurement . IEEE, 2011, pp. 275-284. -
  16. [26][26] M. Kechagia, D. Mitropoulos, and D. Spinellis, 'Charting the api minefield using software telemetry data,' Empirical Software Engineering , vol. 20, no. 6, pp. 1785-1830, 2015. -
  17. [27][27] M. Fewster and D. Graham, Software test automation . Addison-Wesley Reading, 1999. -
  18. [28][28] C. Persson and N. Yilmazturk, 'Establishment of automated regression testing at abb: industrial experience report on'avoiding the pitfalls',' in Proceedings of the 19th IEEE international conference on Automated software engineering . IEEE Computer Society, 2004, pp. 112-121. -
  19. [29][29] M. Grindal, J. Offutt, and J. Mellin, 'On the testing maturity of software producing organizations,' in Testing: Academic & Industrial ConferencePractice And Research Techniques (TAIC PART'06) . IEEE, 2006, pp. 171-180. -
  20. [32][32] E. Collins, A. Dias-Neto, and V. F. de Lucena Jr, 'Strategies for agile software testing automation: An industrial experience,' in 2012 IEEE 36th Annual Computer Software and Applications Conference Workshops . IEEE, 2012, pp. 440-445. -
  21. [33][33] P. Raulamo-Jurvanen, M. M¨ antyl¨ a, and V. Garousi, 'Choosing the right test automation tool: a grey literature review of practitioner sources,' in Proceedings of the 21st International Conference on Evaluation and Assessment in Software Engineering . ACM, 2017, pp. 21-30. -
  22. [34][34] C. Wohlin, P. Runeson, M. H¨ ost, M. C. Ohlsson, B. Regnell, and A. Wessl´ en, Experimentation in software engineering . Springer Science & Business Media, 2012.

Cited by

Sorted by citations

Authors
Wang, Y., Pyhajarvi, M., & Mantyla, M. V.
Affiliations
University of Oulu (FI)
ORCID
Wang, Yuqing
Venue
2020 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)
Publisher
IEEE
Published
2020-10
Pages
314-321
DOI
10.1109/icstw50294.2020.00057
OpenAlex
W3016376596
Cited by
12
References
1
Citations by year
2025: 1 · 2024: 1 · 2023: 3 · 2022: 6 · 2021: 1
FWCI
1.90
Type
conference paper
Language
en
Open access
Licence
Locations
  • IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
  • Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
  • Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
Keywords
DevOps, continuous development, continuous testing, success factors, telemetry, test architecture, incremental method, release speed, test efficiency

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review