CoChat
Sign Up
Cited by 83Software Engineering Techniques and PracticesUsability and User Interface DesignRecommender Systems and Techniques

A/B testing: A systematic literature review

Open access
Quin, Federico, Weyns, Danny, Galster, Matthias, Silva, Camila Costa·2024·Journal of Systems and Software
DOI 10.1016/j.jss.2024.112011
Open this paper in CoChat — free account

Abstract

A/B testing, also referred to as online controlled experimentation or continuous experimentation, is a form of hypothesis testing where two variants of a piece of software are compared in the field from an end user’s point of view. A/B testing is widely used in practice to enable data-driven decision making for software development. While a few studies have explored different facets of research on A/B testing, no comprehensive study has been conducted on the state-of-the-art in A/B testing. Such a study is crucial to provide a systematic overview of the field of A/B testing driving future research forward. To address this gap and provide an overview of the state-of-the-art in A/B testing, this paper reports the results of a systematic literature review that analyzed primary studies. The research questions focused on the subject of A/B testing, how A/B tests are designed and executed, what roles stakeholders have in this process, and the open challenges in the area. Analysis of the extracted data shows that the main targets of A/B testing are algorithms, visual elements, and workflow and processes. Single classic A/B tests are the dominating type of tests, primarily based in hypothesis tests. Stakeholders have three main roles in the design of A/B tests: concept designer, experiment architect, and setup technician. The primary types of data collected during the execution of A/B tests are product/system data, user-centric data, and spatio-temporal data. The dominating use of the test results are feature selection, feature rollout, continued feature development, and subsequent A/B test design. Stakeholders have two main roles during A/B test execution: experiment coordinator and experiment assessor. The main reported open problems are related to the enhancement of proposed approaches and their usability. From our study we derived three interesting lines for future research: strengthen the adoption of statistical methods in A/B testing, improving the process of A/B testing, and enhancing the automation of A/B testing.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/j.jss.2024.112011

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • In many production-level consumer applications, end-to-end evaluations like A/B testing are preferred because they directly reflect user satisfaction.

    “In many production-level consumer applications, end-to-end evaluations like A/B testing (Quin et al., 2024) are preferred because they directly reflect user satisfaction.”

    From MemoryCD: Benchmarking Long-Context User Memory of LLM Agents for Lifelong Cross-Domain Personalization (2026) · cites this paper as Quin et al., 2024

    The Background (p. 2) says A/B tests evaluate software variants in live systems by exposing them to end users, and the Conclusion (p. 18) says they are widely used in web, search-engine, and e-commerce applications. The metrics discussion (p. 9) reports engagement and feedback measures, but does not establish that A/B testing is preferred over other evaluations or that its results directly reflect user satisfaction. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 190

  1. Abhishek, Vineet, Mannor, Shie, 2017. A nonparametric sequential test for online randomized experiments. In: Proceedings of the 26th International Conference on World Wide Web Companion. In: WWW '17 Companion, International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, CHE, pp. 610-616. http://dx.doi.org/10.1145/3041021.3054196. · DOI
  2. Agarwal, Deepak, Long, Bo, Traupman, Jonathan, Xin, Doris, Zhang, Liang, 2014. LASER: A scalable response prediction platform for online advertising. In: Proceedings of the 7th ACM International Conference on Web Search and Data Mining. WSDM '14, Association for Computing Machinery, New York, NY, USA, pp. 173-182. http://dx.doi.org/10.1145/2556195.2556252. · DOI
  3. Aharon, Michal, Kaplan, Yohay, Levy, Rina, Somekh, Oren, Blanc, Ayelet, Eshel, Neetai, Shahar, Avi, Singer, Assaf, Zlotnik, Alex, 2019a. Soft frequency capping for improved ad click prediction in yahoo gemini native. In: Proceedings of the 28th ACM International Conference on Information and Knowledge Management. CIKM '19, Association for Computing Machinery, New York, NY, USA, pp. 2793-2801. http://dx.doi.org/10.1145/3357384.3357801. · DOI
  4. Aharon, Michal, Somekh, Oren, Shahar, Avi, Singer, Assaf, Trayvas, Baruch, Vogel, Hadas, Dobrev, Dobri, 2019b. Carousel ads optimization in yahoo gemini native. In: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. KDD '19, Association for Computing Machinery, New York, NY, USA, pp. 1993-2001. http://dx.doi.org/10.1145/3292500. 3330740. · DOI
  5. Aiello, Luca, Arapakis, Ioannis, Baeza-Yates, Ricardo, Bai, Xiao, Barbieri, Nicola, Mantrach, Amin, Silvestri, Fabrizio, 2016. The role of relevance in sponsored search. In: Proceedings of the 25th ACM International on Conference on Information and Knowledge Management. CIKM '16, Association for Computing Machinery, New York, NY, USA, pp. 185-194. http://dx.doi.org/10.1145/2983323.2983840. · DOI
  6. Akase, Ryuya, Kawabata, Hiroto, Nishida, Akiomi, Tanaka, Yuki, Kaminaga, Tamaki, 2021. Related entity expansion and ranking using knowledge graph. In: Barolli, Leonard, Yim, Kangbin, Enokido, Tomoya (Eds.), Complex, Intelligent and Software Intensive Systems. Springer International Publishing, Cham, pp. 172-184. http://refhub.elsevier.com/S0164-1212(2400054-2/sb6)
  7. Alfaro-Flores, Rafael, Salas-Bonilla, José, Juillard, Loic, Esquivel-Rodríguez, Juan, 2021. Experiment-driven improvements in human-in-the-loop machine learning annotation via significance-based A/B testing. In: 2021 XLVII Latin American Computing Conference. CLEI, pp. 1-9. http://dx.doi.org/10.1109/CLEI53233.2021.9639977. · DOI
  8. Almeida, Joana, Casais, Beatriz, 2022. Subject line personalization techniques and their influence in the E-mail marketing open rate. In: Rocha, Alvaro, Adeli, Hojjat, Dzemyda, Gintautas, Moreira, Fernando (Eds.), Information Systems and Technologies. Springer International Publishing, Cham, pp. 532-540. http://refhub.elsevier.com/S0164-1212(2400054-2/sb8)
  9. Amatriain, Xavier, 2013. Beyond data: From user information to business value through personalized recommendations and consumer science. In: Proceedings of the 22nd ACM International Conference on Information & Knowledge Management. CIKM '13, Association for Computing Machinery, New York, NY, USA, pp. 2201-2208. http://dx.doi.org/10.1145/2505515.2514701. · DOI
  10. Ampatzoglou, Apostolos, Bibi, Stamatia, Avgeriou, Paris, Verbeek, Marijn, Chatzigeorgiou, Alexander, 2019. Identifying, categorizing and mitigating threats to validity in software engineering secondary studies. Inf. Softw. Technol. 106, 201-230. http://dx.doi.org/10.1016/j.infsof.2018.10.006. · DOI · Paper page
  11. Appiktala, Nirupama, Chen, Miao, Natkovich, Michael, Walters, Joshua, 2017. Demystifying dark matter for online experimentation. In: 2017 IEEE International Conference on Big Data (Big Data). pp. 1620-1626. http://dx.doi.org/10.1109/ BigData.2017.8258096.
  12. Auer, F., Felderer, M., 2018. Current state of research on continuous experimentation: A systematic mapping study. In: 2018 44th Euromicro Conference on Software Engineering and Advanced Applications. SEAA, IEEE Computer Society, Los Alamitos, CA, USA, pp. 335-344. http://dx.doi.org/10.1109/SEAA.2018.00062. · DOI
  13. Auer, Florian, Ros, Rasmus, Kaltenbrunner, Lukas, Runeson, Per, Felderer, Michael, 2021. Controlled experimentation in continuous experimentation: Knowledge and challenges. Inf. Softw. Technol. 134, 106551. http://dx.doi.org/10.1016/j.infsof. 2021.106551. · DOI
  14. Bakshy, Eytan, Frachtenberg, Eitan, 2015. Design and analysis of benchmarking experiments for distributed internet services. In: Proceedings of the 24th International Conference on World Wide Web. WWW '15, International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, CHE, pp. 108-118. http://dx.doi.org/10.1145/2736277.2741082. · DOI
  15. Barajas, Joel, Kwon, Jaimie, Akella, Ram, Flores, Aaron, Holtan, Marius, Andrei, Victor, 2012a. Marketing campaign evaluation in targeted display advertising. In: Proceedings of the Sixth International Workshop on Data Mining for Online Advertising and Internet Economy. ADKDD '12, Association for Computing Machinery, New York, NY, USA, http://dx.doi.org/10.1145/2351356.2351361. · DOI
  16. Barajas, Joel, Kwon, Jaimie, Akella, Ram, Flores, Aaron, Holtan, Marius, Andrei, Victor, 2012b. Measuring dynamic effects of display advertising in the absence of user tracking information. In: Proceedings of the Sixth International Workshop on Data Mining for Online Advertising and Internet Economy. ADKDD '12, Association for Computing Machinery, New York, NY, USA, http://dx.doi.org/10.1145/2351356. 2351364. · DOI
  17. Basili, Victor R., Caldiera, Gianluigi, Rombach, Dieter H., 1994. The goal question metric approach. I, John Wiley & Sons, http://refhub.elsevier.com/S0164-1212(2400054-2/sb17)
  18. Blask, Tobias, 2013. Applying Bayesian parameter estimation to A/B tests in e-business applications examining the impact of green marketing signals in sponsored search advertising. In: 2013 International Conference on E-Business (ICE-B). pp. 1-8. http://refhub.elsevier.com/S0164-1212(2400054-2/sb18)
  19. Blask, Tobias, Funk, Burkhardt, Schulte, Reinhard, 2011. Should companies bid on their own brand in sponsored search? In: Proceedings of the International Conference on E-Business. pp. 1-8. http://refhub.elsevier.com/S0164-1212(2400054-2/sb19)
  20. Borisyuk, Fedor, Malreddy, Siddarth, Mei, Jun, Liu, Yiqun, Liu, Xiaoyi, Maheshwari, Piyush, Bell, Anthony, Rangadurai, Kaushik, 2021. VisRel: Media search at scale. In: Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. KDD '21, Association for Computing Machinery, New York, NY, USA, pp. 2584-2592. http://dx.doi.org/10.1145/3447548.3467081. · DOI
  21. Borodovsky, Slava, Rosset, Saharon, 2011. A/b testing at sweetim: The importance of proper statistical analysis. In: 2011 IEEE 11th International Conference on Data Mining Workshops. pp. 733-740. http://dx.doi.org/10.1109/ICDMW.2011.19. · DOI
  22. Brown, Alex, Lush, Binky, Jansen, Bernard J., 2016. Pixel efficiency analysis: A quantitative web analytics approach. In: Proceedings of the Association for Information Science and Technology. Vol. 53, (1), pp. 1-10. http://dx.doi.org/10.1002/ pra2.2016.14505301040, arXiv:https://asistdl.onlinelibrary.wiley.com/doi/pdf/10. 1002/pra2.2016.14505301040.
  23. Budylin, Roman, Drutsa, Alexey, Katsev, Ilya, Tsoy, Valeriya, 2018. Consistent transformation of ratio metrics for efficient online controlled experiments. In: Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining. WSDM '18, Association for Computing Machinery, New York, NY, USA, pp. 55-63. http://dx.doi.org/10.1145/3159652.3159699. · DOI
  24. Cai, Tianchi, Cheng, Daxi, Liang, Chen, Liu, Ziqi, Gu, Lihong, Xie, Huizhi, Zhang, Zhiqiang, Zeng, Xiaodong, Gu, Jinjie, 2021. LinkLouvain: Link-aware A/B testing and its application on online marketing campaign. In: Jensen, Christian S., Lim, Ee-Peng, Yang, De-Nian, Lee, Wang-Chien, Tseng, Vincent S., Kalogeraki, Vana, Huang, Jen-Wei, Shen, Chih-Ya (Eds.), Database Systems for Advanced Applications. Springer International Publishing, Cham, pp. 499-510. http://refhub.elsevier.com/S0164-1212(2400054-2/sb24)
  25. Cámara, Javier, Kobsa, Alfred, 2009. Facilitating controlled tests of website design changes: A systematic approach. In: Gaedke, Martin, Grossniklaus, Michael, Díaz, Oscar (Eds.), Web Engineering. Springer Berlin Heidelberg, Berlin, Heidelberg, pp. 370-378. http://refhub.elsevier.com/S0164-1212(2400054-2/sb25)
  26. Chakraborty, Samit, Hoque, Md. Saiful, Rahman Jeem, Naimur, Biswas, Manik Chandra, Bardhan, Deepayan, Lobaton, Edgar, 2021. Fashion recommendation systems, models and methods: A review. Informatics 8 (3), http://dx.doi.org/10.3390/ informatics8030049, URL https://www.mdpi.com/2227-9709/8/3/49.
  27. Chen, Guangde, Chen, Bee-Chung, Agarwal, Deepak, 2017a. Social incentive optimization in online social networks. In: Proceedings of the Tenth ACM International Conference on Web Search and Data Mining. WSDM '17, Association for Computing Machinery, New York, NY, USA, pp. 547-556. http://dx.doi.org/10.1145/3018661. 3018700. · DOI
  28. Chen, Russell, Chen, Miao, Jadav, Mahendrasinh Ramsinh, Bae, Joonsuk, Matheson, Don, 2017b. Faster online experimentation by eliminating traditional A/A validation. In: 2017 IEEE International Conference on Big Data (Big Data). pp. 1635-1641. http://dx.doi.org/10.1109/BigData.2017.8258098. · DOI
  29. Chen, Nanyu, Liu, Min, Xu, Ya, 2019. How A/B tests could go wrong: Automatic diagnosis of invalid online experiments. In: Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining. WSDM '19, Association for Computing Machinery, New York, NY, USA, pp. 501-509. http://dx.doi.org/10. 1145/3289600.3291000.
  30. Claeys, Emmanuelle, Gançarski, Pierre, Maumy-Bertrand, Myriam, Wassner, Hubert, 2017. Regression tree for bandits models in A/B testing. In: Adams, Niall, Tucker, Allan, Weston, David (Eds.), Advances in Intelligent Data Analysis XVI. Springer International Publishing, Cham, pp. 52-62. http://refhub.elsevier.com/S0164-1212(2400054-2/sb30)
  31. Creswell, John, Guetterman, Timothy, 2018. Educational Research: Planning, Conducting, and Evaluating Quantitative and Qualitative Research, sixth ed. Pearson, New York, NY, USA. http://refhub.elsevier.com/S0164-1212(2400054-2/sb32)
  32. Dai, Xinyi, Xi, Yunjia, Zhang, Weinan, Liu, Qing, Tang, Ruiming, He, Xiuqiang, Hou, Jiawei, Wang, Jun, Yu, Yong, 2021. Beyond relevance ranking: A general graph matching framework for utility-oriented learning to rank. ACM Trans. Inf. Syst. 40 (2), http://dx.doi.org/10.1145/3464303. · DOI
  33. Daneva, Maya, Damian, Daniela, Marchetto, Alessandro, Pastor, Oscar, 2014. Empirical research methodologies and studies in requirements engineering: How far did we come? J. Syst. Softw. 95, 1-9. http://dx.doi.org/10.1016/j.jss.2014.06.035. · DOI
  34. de Feijter, Rico, Overbeek, Sietse, van Vliet, Rob, Jagroep, Erik, Brinkkemper, Sjaak, 2018. DevOps competences and maturity for software producing organizations. In: Gulden, Jens, Reinhartz-Berger, Iris, Schmidt, Rainer, Guerreiro, Sérgio, Guédria, Wided, Bera, Palash (Eds.), Enterprise, Business-Process and Information Systems Modeling. Springer International Publishing, Cham, pp. 244-259. http://refhub.elsevier.com/S0164-1212(2400054-2/sb35)
  35. De Souza, Wagner S., Pereira, Fernando O., Albuquerque, Vanessa G., Melegati, Jorge, Guerra, Eduardo, 2022. A framework model to support A/B tests at the class and component level. In: 2022 IEEE 46th Annual Computers, Software, and Applications Conference. COMPSAC, pp. 860-865. http://dx.doi.org/10.1109/COMPSAC54236. 2022.00136. · DOI
  36. Deng, Alex, 2015. Objective Bayesian two sample hypothesis testing for online controlled experiments. In: Proceedings of the 24th International Conference on World Wide Web. In: WWW '15 Companion, Association for Computing Machinery, New York, NY, USA, pp. 923-928. http://dx.doi.org/10.1145/2740908.2742563. · DOI
  37. Deng, Alex, Li, Tianxi, Guo, Yu, 2014. Statistical inference in two-stage online controlled experiments with treatment selection and validation. In: Proceedings of the 23rd International Conference on World Wide Web. WWW '14, Association for Computing Machinery, New York, NY, USA, pp. 609-618. http://dx.doi.org/10. 1145/2566486.2568028.
  38. Deng, Alex, Li, Yicheng, Lu, Jiannan, Ramamurthy, Vivek, 2021. On post-selection inference in A/B testing. In: Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. KDD '21, Association for Computing Machinery, New York, NY, USA, pp. 2743-2752. http://dx.doi.org/10.1145/3447548. 3467129. · DOI
  39. Deng, Alex, Lu, Jiannan, Litz, Jonthan, 2017. Trustworthy analysis of online A/B tests: Pitfalls, challenges and solutions. In: Proceedings of the Tenth ACM International Conference on Web Search and Data Mining. WSDM '17, Association for Computing Machinery, New York, NY, USA, pp. 641-649. http://dx.doi.org/10.1145/3018661. 3018677. · DOI
  40. Deng, Alex, Xu, Ya, Kohavi, Ron, Walker, Toby, 2013. Improving the sensitivity of online controlled experiments by utilizing pre-experiment data. In: Proceedings of the Sixth ACM International Conference on Web Search and Data Mining. WSDM '13, Association for Computing Machinery, New York, NY, USA, pp. 123-132. http://dx.doi.org/10.1145/2433396.2433413. · DOI
  41. Dimmery, Drew, Bakshy, Eytan, Sekhon, Jasjeet, 2019. Shrinkage estimators in online experiments. In: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. KDD '19, Association for Computing Machinery, New York, NY, USA, pp. 2914-2922. http://dx.doi.org/10.1145/3292500. 3330771. · DOI
  42. Dmitriev, Pavel, Frasca, Brian, Gupta, Somit, Kohavi, Ron, Vaz, Garnet, 2016. Pitfalls of long-term online controlled experiments. In: 2016 IEEE International Conference on Big Data (Big Data). pp. 1367-1376. http://dx.doi.org/10.1109/BigData.2016. 7840744. · DOI
  43. Dmitriev, Pavel, Gupta, Somit, Kim, Dong Woo, Vaz, Garnet, 2017. A dirty dozen: Twelve common metric interpretation pitfalls in online controlled experiments. In: Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD '17, Association for Computing Machinery, New York, NY, USA, pp. 1427-1436. http://dx.doi.org/10.1145/3097983.3098024. · DOI
  44. Dobaj, Jürgen, Riel, Andreas, Krug, Thomas, Seidl, Matthias, Macher, Georg, Egretzberger, Markus, 2022. Towards digital twin-enabled DevOps for CPS providing architecture-based service adaptation & verification at runtime. In: Proceedings of the 17th Symposium on Software Engineering for Adaptive and Self-Managing Systems. SEAMS '22, Association for Computing Machinery, New York, NY, USA, pp. 132-143. http://dx.doi.org/10.1145/3524844.3528057. · DOI
  45. Drutsa, Alexey, Gusev, Gleb, Serdyukov, Pavel, 2015a. Future user engagement prediction and its application to improve the sensitivity of online experiments. In: Proceedings of the 24th International Conference on World Wide Web. WWW '15, International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, CHE, pp. 256-266. http://dx.doi.org/10.1145/2736277. 2741116. · DOI
  46. Drutsa, Alexey, Gusev, Gleb, Serdyukov, Pavel, 2017a. Periodicity in user engagement with a search engine and its application to online controlled experiments. ACM Trans. Web 11 (2), http://dx.doi.org/10.1145/2856822. · DOI
  47. Drutsa, Alexey, Gusev, Gleb, Serdyukov, Pavel, 2017b. Using the delay in a treatment effect to improve sensitivity and preserve directionality of engagement metrics in A/B experiments. In: Proceedings of the 26th International Conference on World Wide Web. WWW '17, International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, CHE, pp. 1301-1310. http://dx.doi. org/10.1145/3038912.3052664. · DOI
  48. Drutsa, Alexey, Ufliand, Anna, Gusev, Gleb, 2015b. Practical aspects of sensitivity in online experimentation with user engagement metrics. In: Proceedings of the 24th ACM International on Conference on Information and Knowledge Management. CIKM '15, Association for Computing Machinery, New York, NY, USA, pp. 763-772. http://dx.doi.org/10.1145/2806416.2806496. · DOI
  49. Duan, Weitao, Ba, Shan, Zhang, Chunzhe, 2021. Online experimentation with surrogate metrics: Guidelines and a case study. In: Proceedings of the 14th ACM International Conference on Web Search and Data Mining. WSDM '21, Association for Computing Machinery, New York, NY, USA, pp. 193-201. http://dx.doi.org/10.1145/3437963. 3441737. · DOI
  50. Duivesteijn, Wouter, Farzami, Tara, Putman, Thijs, Peer, Evertjan, Weerts, Hilde J.P., Adegeest, Jasper N., Foks, Gerson, Pechenizkiy, Mykola, 2017. Have it both ways-From A/B testing to A&B testing with exceptional model mining. In: Altun, Yasemin, Das, Kamalika, Mielikäinen, Taneli, Malerba, Donato, Stefanowski, Jerzy, Read, Jesse, Žitnik, Marinka, Ceci, Michelangelo, Džeroski, Sašo (Eds.), Machine Learning and Knowledge Discovery in Databases. Springer International Publishing, Cham, pp. 114-126. http://refhub.elsevier.com/S0164-1212(2400054-2/sb51)
  51. Eckroth, Joshua, Schoen, Eric, 2019. A genetic algorithm for finding a small and diverse set of recent news stories on a given subject: How we generate aaai's ai-alert. In: 33rd AAAI Conference on Artificial Intelligence, AAAI 2019, 31st Innovative Applications of Artificial Intelligence Conference, IAAI 2019 and the 9th AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019. pp. 9357-9364, URL https://www.scopus.com/inward/record.uri?eid=2-s2. 0-85090801224&partnerID=40&md5=f3391d595e00df8a0cba7802c9043ebd, Cited by: 2.
  52. Efron, B., Tibshirani, R.J., 1994. An Introduction to the Bootstrap. CRC Press, URL https://books.google.be/books?id=MWC1DwAAQBAJ.
  53. Ermis, Beyza, Ernst, Patrick, Stein, Yannik, Zappella, Giovanni, 2020. Learning to rank in the position based model with bandit feedback. In: Proceedings of the 29th ACM International Conference on Information & Knowledge Management. CIKM '20, Association for Computing Machinery, New York, NY, USA, pp. 2405-2412. http://dx.doi.org/10.1145/3340531.3412723. · DOI
  54. Erthal, Vladimir M., de Souza, Bruno P., dos Santos, Paulo Sérgio M., Travassos, Guilherme H., 2022. A literature study to characterize continuous experimentation in software engineering. In: CIbSE 2022 - XXV Ibero-American Conference on Software Engineering. URL https://www.scopus.com/inward/record.uri?eid=2-s2.085137064966&partnerID=40&md5=04240b73ab90eb841083173be558b33f, Cited by: 0.
  55. Esteller-Cucala, Maria, Fernandez, Vicenc, Villuendas, Diego, 2019. Experimentation pitfalls to avoid in a/b testing for online personalization. In: Adjunct Publication of the 27th Conference on User Modeling, Adaptation and Personalization. In: UMAP'19 Adjunct, Association for Computing Machinery, New York, NY, USA, pp. 153-159. http://dx.doi.org/10.1145/3314183.3323853. · DOI
  56. Fabijan, Aleksander, Arai, Benjamin, Dmitriev, Pavel, Vermeer, Lukas, 2021. It takes a flywheel to fly: Kickstarting and growing the a/b testing momentum at scale. In: 2021 47th Euromicro Conference on Software Engineering and Advanced Applications. SEAA, pp. 109-118. http://dx.doi.org/10.1109/SEAA53835.2021. 00023. · DOI
  57. Fabijan, Aleksander, Dmitriev, Pavel, McFarland, Colin, Vermeer, Lukas, Holmström Olsson, Helena, Bosch, Jan, 2018. Experimentation growth: Evolving trustworthy A/B testing capabilities in online software companies. J. Softw.: Evol. Process 30 (12), e2113. http://dx.doi.org/10.1002/smr.2113, arXiv:https: //onlinelibrary.wiley.com/doi/pdf/10.1002/smr.2113, e2113 JSME-17-0210.R2. · DOI
  58. Fabijan, Aleksander, Dmitriev, Pavel, Olsson, Helena Holmström, Bosch, Jan, 2017a. The benefits of controlled experimentation at scale. In: 2017 43rd Euromicro Conference on Software Engineering and Advanced Applications. SEAA, pp. 18-26. http://dx.doi.org/10.1109/SEAA.2017.47. · DOI
  59. Fabijan, Aleksander, Dmitriev, Pavel, Olsson, Helena Holmström, Bosch, Jan, 2017b. The evolution of continuous experimentation in software product development: From data to a data-driven organization at scale. In: Proceedings of the 39th International Conference on Software Engineering. ICSE '17, IEEE Press, Los Alamitos, CA, USA, pp. 770-780. http://dx.doi.org/10.1109/ICSE.2017.76. · DOI
  60. Fabijan, A., Dmitriev, P., Olsson, H. Holmstrom, Bosch, J., 2020. The online controlled experiment lifecycle. IEEE Softw. 37 (02), 60-67. http://dx.doi.org/10.1109/MS. 2018.2875842. · DOI
  61. Fabijan, Aleksander, Gupchup, Jayant, Gupta, Somit, Omhover, Jeff, Qin, Wen, Vermeer, Lukas, Dmitriev, Pavel, 2019. Diagnosing sample ratio mismatch in online controlled experiments: A taxonomy and rules of thumb for practitioners. In: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. KDD '19, Association for Computing Machinery, New York, NY, USA, pp. 2156-2164. http://dx.doi.org/10.1145/3292500.3330722. · DOI
  62. Fabijan, Aleksander, Olsson, Helena Holmström, Bosch, Jan, 2015. Customer feedback and data collection techniques in software R&D: A literature review. In: Fernandes, João M., Machado, Ricardo J., Wnuk, Krzysztof (Eds.), Software Business. Springer International Publishing, Cham, pp. 139-153. http://dx.doi.org/10.1007/ 978-3-319-19593-3\_12.
  63. Fabijan, Aleksander, Olsson, Helena Holmström, Bosch, Jan, 2016. The lack of sharing of customer data in large software organizations: Challenges and implications. In: Sharp, Helen, Hall, Tracy (Eds.), Agile Processes, in Software Engineering, and Extreme Programming. Springer International Publishing, Cham, pp. 39-52. http://dx.doi.org/10.1007/978-3-319-33515-5\_4. · DOI
  64. Fairstein, Yaron, Haramaty, Elad, Lazerson, Arnon, Lewin-Eytan, Liane, 2022. External evaluation of ranking models under extreme position-bias. In: Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining. WSDM '22, Association for Computing Machinery, New York, NY, USA, pp. 252-261. http://dx.doi.org/10.1145/3488560.3498420. · DOI
  65. Feit, Elea McDonnell, Berman, Ron, 2019. Test & roll: Profit-maximizing A/B tests. Mark. Sci. 38 (6), 1038-1058. http://dx.doi.org/10.1287/mksc.2019.1194. · DOI
  66. Freno, Antonino, 2017. Practical lessons from developing a large-scale recommender system at zalando. In: Proceedings of the Eleventh ACM Conference on Recommender Systems. RecSys '17, Association for Computing Machinery, New York, NY, USA, pp. 251-259. http://dx.doi.org/10.1145/3109859.3109897. · DOI
  67. Fu, Kun, Meng, Fanlin, Ye, Jieping, Wang, Zheng, 2020. Compacteta: A fast inference system for travel time prediction. In: Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. KDD '20, Association for Computing Machinery, New York, NY, USA, pp. 3337-3345. http: //dx.doi.org/10.1145/3394486.3403386. · DOI
  68. Funk, Burkhardt, 2009. Optimizing price levels in e-commerce applications: An empirical study. In: ICETE 2009 -International Joint Conference on e-Business and Telecommunications. pp. 37-43, URL https://www. scopus.com/inward/record.uri?eid=2-s2.0-74549181430&partnerID=40&md5= 6dfdde67b807b3964c62fc8c1929dcf0, Cited by: 1.
  69. Galster, Matthias, Weyns, Danny, 2016. Empirical research in software architecture: How far have we come? In: 2016 13th Working IEEE/IFIP Conference on Software Architecture. WICSA, IEEE Press, Los Alamitos, CA, USA, pp. 11-20. http://dx.doi. org/10.1109/WICSA.2016.10. · DOI
  70. Giaimo, Federico, Andrade, Hugo, Berger, Christian, 2020. Continuous experimentation and the cyber-physical systems challenge: An overview of the literature and the industrial perspective. J. Syst. Softw. 170, 110781. http://dx.doi.org/10.1016/j.jss. 2020.110781. · DOI
  71. Gomez-Uribe, Carlos A., Hunt, Neil, 2016. The netflix recommender system: Algorithms, business value, and innovation. ACM Trans. Manage. Inf. Syst. 6 (4), http://dx.doi. org/10.1145/2843948. · DOI · Paper page
  72. Goswami, Anjan, Han, Wei, Wang, Zhenrui, Jiang, Angela, 2015. Controlled experiments for decision-making in e-commerce search. In: 2015 IEEE International Conference on Big Data (Big Data). IEEE Press, Los Alamitos, CA, USA, pp. 1094-1102. http://dx.doi.org/10.1109/BigData.2015.7363863. · DOI
  73. Gruson, Alois, Chandar, Praveen, Charbuillet, Christophe, McInerney, James, Hansen, Samantha, Tardieu, Damien, Carterette, Ben, 2019. Offline evaluation to make decisions about PlaylistRecommendation algorithms. In: Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining. WSDM '19, Association for Computing Machinery, New York, NY, USA, pp. 420-428. http://dx.doi.org/10.1145/3289600.3291027. · DOI
  74. Gui, Huan, Xu, Ya, Bhasin, Anmol, Han, Jiawei, 2015. Network A/B testing: From sampling to estimation. In: Proceedings of the 24th International Conference on World Wide Web. WWW '15, International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, CHE, pp. 399-409. http://dx.doi.org/ 10.1145/2736277.2741081. · DOI
  75. Gupchup, Jayant, Hosseinkashi, Yasaman, Dmitriev, Pavel, Schneider, Daniel, Cutler, Ross, Jefremov, Andrei, Ellis, Martin, 2018. Trustworthy experimentation under telemetry loss. In: Proceedings of the 27th ACM International Conference on Information and Knowledge Management. CIKM '18, Association for Computing Machinery, New York, NY, USA, pp. 387-396. http://dx.doi.org/10.1145/3269206. 3271747. · DOI
  76. Gupta, Shubham, Chokshi, Sneha, 2020. Digital marketing effectiveness using incrementality. In: Singh, Mayank, Gupta, P.K., Tyagi, Vipin, Flusser, Jan, Ören, Tuncer, Valentino, Gianluca (Eds.), Advances in Computing and Data Sciences. Springer Singapore, Singapore, pp. 66-75. http://refhub.elsevier.com/S0164-1212(2400054-2/sb77)
  77. Gupta, Somit, Kohavi, Ronny, Tang, Diane, Xu, Ya, Andersen, Reid, Bakshy, Eytan, Cardin, Niall, Chandran, Sumita, Chen, Nanyu, Coey, Dominic, et al., 2019. Top challenges from the first practical online controlled experiments summit. ACM SIGKDD Explor. Newsl. 21 (1), 20-35. http://refhub.elsevier.com/S0164-1212(2400054-2/sb78)
  78. Gupta, Somit, Ulanova, Lucy, Bhardwaj, Sumit, Dmitriev, Pavel, Raff, Paul, Fabijan, Aleksander, 2018. The anatomy of a large-scale experimentation platform. In: 2018 IEEE International Conference on Software Architecture. ICSA, pp. 1-109. http://dx.doi.org/10.1109/ICSA.2018.00009. · DOI
  79. Ha-Thuc, Viet, Dutta, Avishek, Mao, Ren, Wood, Matthew, Liu, Yunli, 2020. A counterfactual framework for seller-side A/B testing on marketplaces. In: Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. SIGIR '20, Association for Computing Machinery, New York, NY, USA, pp. 2288-2296. http://dx.doi.org/10.1145/3397271.3401434. · DOI
  80. He, Yan, Chen, Miao, 2017. A probabilistic, mechanism-indepedent outlier detection method for online experimentation. In: 2017 IEEE International Conference on Data Science and Advanced Analytics. DSAA, pp. 640-647. http://dx.doi.org/10.1109/ DSAA.2017.64.
  81. He, Yan, Yu, Lin, Chen, Miao, Choi, William, Matheson, Don, 2022. A clusterbased nearest neighbor matching algorithm for enhanced a/a validation in online experimentation. In: Companion Proceedings of the Web Conference 2022. WWW '22, Association for Computing Machinery, New York, NY, USA, pp. 136-140. http://dx.doi.org/10.1145/3487553.3524220. · DOI
  82. Humble, Jez, Farley, David, 2010. Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation, first ed. Addison-Wesley Professional, Illinois, IL, USA. http://refhub.elsevier.com/S0164-1212(2400054-2/sb83)
  83. Jiang, Hao, Yang, Fan, Wei, Wutao, 2020. Statistical reasoning of zero-inflated rightskewed user-generated big data A/B testing. In: 2020 IEEE International Conference on Big Data (Big Data). pp. 1533-1544. http://dx.doi.org/10.1109/BigData50022. 2020.9377996. · DOI
  84. Johari, Ramesh, Koomen, Pete, Pekelis, Leonid, Walsh, David, 2017. Peeking at A/B tests: Why it matters, and what to do about it. In: Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD '17, Association for Computing Machinery, New York, NY, USA, pp. 1517-1525. http://dx.doi.org/10.1145/3097983.3097992. · DOI
  85. Ju, Nianqiao, Hu, Diane, Henderson, Adam, Hong, Liangjie, 2019. A sequential test for selecting the better variant: Online A/B testing, adaptive allocation, and continuous monitoring. In: Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining. WSDM '19, Association for Computing Machinery, New York, NY, USA, pp. 492-500. http://dx.doi.org/10.1145/3289600.3291025. · DOI
  86. Kaplan, Jared, McCandlish, Sam, Henighan, Tom, Brown, Tom B., Chess, Benjamin, Child, Rewon, Gray, Scott, Radford, Alec, Wu, Jeffrey, Amodei, Dario, 2020. Scaling laws for neural language models. arXiv:2001.08361. · arXiv
  87. Keele, Staffs, et al., 2007. Guidelines for performing systematic literature reviews in software engineering. Technical report, Technical report, Ver. 2.3 EBSE Technical Report. EBSE. http://refhub.elsevier.com/S0164-1212(2400054-2/sb88)
  88. Kharitonov, Eugene, Drutsa, Alexey, Serdyukov, Pavel, 2017. Learning sensitive combinations of A/B test metrics. In: Proceedings of the Tenth ACM International Conference on Web Search and Data Mining. WSDM '17, Association for Computing Machinery, New York, NY, USA, pp. 651-659. http://dx.doi.org/10.1145/3018661. 3018708. · DOI
  89. Kharitonov, Eugene, Vorobev, Aleksandr, Macdonald, Craig, Serdyukov, Pavel, Ounis, Iadh, 2015. Sequential testing for early stopping of online experiments. In: Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval. SIGIR '15, Association for Computing Machinery, New York, NY, USA, pp. 473-482. http://dx.doi.org/10.1145/2766462. 2767729. · DOI
  90. King, Rochelle, Churchill, Elizabeth F., Tan, Caitlin, 2017. Designing with Data: Improving the User Experience with A/B Testing. O'Reilly Media, Inc., Sebastopol, CA, USA. http://refhub.elsevier.com/S0164-1212(2400054-2/sb91)
  91. Kohavi, Ron, Deng, Alex, Frasca, Brian, Longbotham, Roger, Walker, Toby, Xu, Ya, 2012. Trustworthy online controlled experiments: Five puzzling outcomes explained. In: Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD '12, Association for Computing Machinery, New York, NY, USA, pp. 786-794. http://dx.doi.org/10.1145/2339530. 2339653. · DOI
  92. Kohavi, Ron, Deng, Alex, Frasca, Brian, Walker, Toby, Xu, Ya, Pohlmann, Nils, 2013. Online controlled experiments at large scale. In: Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD '13, Association for Computing Machinery, New York, NY, USA, pp. 1168-1176. http://dx.doi.org/10.1145/2487575.2488217. · DOI
  93. Kohavi, Ron, Deng, Alex, Longbotham, Roger, Xu, Ya, 2014. Seven rules of thumb for web site experimenters. In: Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD '14, Association for Computing Machinery, New York, NY, USA, pp. 1857-1866. http://dx.doi.org/10. 1145/2623330.2623341.
  94. Kohavi, Ron, Longbotham, Roger, 2011. Unexpected results in online controlled experiments. SIGKDD Explor. Newsl. 12 (2), 31-35. http://dx.doi.org/10.1145/ 1964897.1964905.
  95. Kohavi, Ron, Longbotham, Roger, 2017. Online controlled experiments and A/B testing. Encycl. Mach. Learn. Data Min. 7 (8), 922-929. http://refhub.elsevier.com/S0164-1212(2400054-2/sb96)
  96. Kohavi, Ron, Longbotham, Roger, Sommerfield, Dan, Henne, Randal, 2009. Controlled experiments on the web: Survey and practical guide. Data Min. Knowl. Discov. 18, 140-181. http://dx.doi.org/10.1007/s10618-008-0114-1. · DOI
  97. Kohavi, Ron, Tang, Diane, Xu, Ya, 2020. Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press, Cambridge, United Kingdom, http://dx.doi.org/10.1017/9781108653985. · DOI · Paper page
  98. Kohavi, Ron, Thomke, Stefan, 2017. The surprising power of online experiments. Harvard Business Review 74-82. http://refhub.elsevier.com/S0164-1212(2400054-2/sb99)
  99. Kornilova, Anastasiia, Bernardi, Lucas, 2021. Mining the stars: Learning quality ratings with user-facing explanations for vacation rentals. In: Proceedings of the 14th ACM International Conference on Web Search and Data Mining. WSDM '21, Association for Computing Machinery, New York, NY, USA, pp. 976-983. http://dx.doi.org/10. 1145/3437963.3441812.
  100. Koukouvis, Kostantinos, Cubero, Roberto Alcañiz, Pelliccione, Patrizio, 2016. A/B testing in E-commerce sales processes. In: Crnkovic, Ivica, Troubitsyna, Elena (Eds.), Software Engineering for Resilient Systems. Springer International Publishing, Cham, pp. 133-148. http://refhub.elsevier.com/S0164-1212(2400054-2/sb101)
  101. Kumar, Ratnakar, Hasteer, Nitasha, 2017. Evaluating usability of a web application: A comparative analysis of open-source tools. In: 2017 2nd International Conference on Communication and Electronics Systems. ICCES, pp. 350-354. http://dx.doi.org/ 10.1109/CESYS.2017.8321296. · DOI
  102. Kumar, Anuj, Hosanagar, Kartik, 2017. Measuring the value of recommendation links on product demand. SSRN Electron. J. http://dx.doi.org/10.2139/ssrn.2909971. · DOI
  103. Lalmas, Mounia, Lehmann, Janette, Shaked, Guy, Silvestri, Fabrizio, Tolomei, Gabriele, 2015. Promoting positive post-click experience for in-stream yahoo gemini users. In: Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD '15, Association for Computing Machinery, New York, NY, USA, pp. 1929-1938. http://dx.doi.org/10.1145/2783258.2788581. · DOI
  104. Lee, Minyong R., Shen, Milan, 2018. Winner's curse: Bias estimation for total effects of features in online controlled experiments. In: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. KDD '18, Association for Computing Machinery, New York, NY, USA, pp. 491-499. http://dx.doi.org/10.1145/3219819.3219905. · DOI
  105. Lettner, Florian, Holzmann, Clemens, Hutflesz, Patrick, 2013. Enabling A/B testing of native mobile applications by remote user interface exchange. In: MorenoDíaz, Roberto, Pichler, Franz, Quesada-Arencibia, Alexis (Eds.), Computer Aided Systems Theory - EUROCAST 2013. Springer Berlin Heidelberg, Berlin, Heidelberg, pp. 458-466. http://refhub.elsevier.com/S0164-1212(2400054-2/sb106)
  106. Li, Paul Luo, Chai, Xiaoyu, Campbell, Frederick, Liao, Jilong, Abburu, Neeraja, Kang, Minsuk, Niculescu, Irina, Brake, Greg, Patil, Siddharth, Dooley, James, Paddock, Brandon, 2021a. Evolving software to be ML-driven utilizing real-world A/B testing: Experiences, insights, challenges. In: 2021 IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). pp. 170-179. http://dx.doi.org/10.1109/ICSE-SEIP52600.2021.00026. · DOI
  107. Li, Paul Luo, Dmitriev, Pavel, Hu, Huibin Mary, Chai, Xiaoyu, Dimov, Zoran, Paddock, Brandon, Li, Ying, Kirshenbaum, Alex, Niculescu, Irina, Thoresen, Taj, 2019. Experimentation in the operating system: The windows experimentation platform. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). pp. 21-30. http://dx.doi.org/10.1109/ ICSE-SEIP.2019.00011.
  108. Li, Lihong, Kim, Jin Young, Zitouni, Imed, 2015. Toward predicting the outcome of an A/B experiment for search relevance. In: Proceedings of the Eighth ACM International Conference on Web Search and Data Mining. WSDM '15, Association for Computing Machinery, New York, NY, USA, pp. 37-46. http://dx.doi.org/10. 1145/2684822.2685311.
  109. Li, Yiyang, Tao, Guanyu, Zhang, Weinan, Yu, Yong, Wang, Jun, 2017. Content recommendation by noise contrastive transfer learning of feature representation. In: Proceedings of the 2017 ACM on Conference on Information and Knowledge Management. CIKM '17, Association for Computing Machinery, New York, NY, USA, pp. 1657-1665. http://dx.doi.org/10.1145/3132847.3132855. · DOI
  110. Li, Ye, Xie, Hong, Lin, Yishi, Lui, John C.S., 2021b. Unifying offline causal inference and online bandit learning for data driven decision. In: Proceedings of the Web Conference 2021. WWW '21, Association for Computing Machinery, New York, NY, USA, pp. 2291-2303. http://dx.doi.org/10.1145/3442381.3449982. · DOI
  111. Li, Hannah, Zhao, Geng, Johari, Ramesh, Weintraub, Gabriel Y., 2022. Interference, bias, and variance in two-sided marketplace experimentation: Guidance for platforms. In: Proceedings of the ACM Web Conference 2022. WWW '22, Association for Computing Machinery, New York, NY, USA, pp. 182-192. http://dx.doi.org/10. 1145/3485447.3512063.
  112. Li, Chengbo, Zhu, Lin, Fu, Guangyuan, Du, Longzhi, Zhao, Canhua, Ma, Tianlun, Ye, Chang, Lee, Pei, 2021c. Learning to bundle proactively for on-demand meal delivery. In: Proceedings of the 30th ACM International Conference on Information & Knowledge Management. CIKM '21, Association for Computing Machinery, New York, NY, USA, pp. 3898-3905. http://dx.doi.org/10.1145/3459637.3481931. · DOI
  113. Liou, Kevin, Taylor, Sean J., 2020. Variance-weighted estimators to improve sensitivity in online experiments. In: Proceedings of the 21st ACM Conference on Economics and Computation. EC '20, Association for Computing Machinery, New York, NY, USA, pp. 837-850. http://dx.doi.org/10.1145/3391403.3399542. · DOI
  114. Liu, Sophia, Fabijan, Aleksander, Furchtgott, Michael, Gupta, Somit, Janowski, Pawel, Qin, Wen, Dmitriev, Pavel, 2019. Enterprise-level controlled experiments at scale: Challenges and solutions. In: 2019 45th Euromicro Conference on Software Engineering and Advanced Applications. SEAA, pp. 29-37. http://dx.doi.org/10.1109/ SEAA.2019.00013.
  115. Liu, Yuchu, Mattos, David Issa, Bosch, Jan, Olsson, Helena Holmström, Lantz, Jonn, 2021. Size matters? Or not: A/b testing with limited sample in automotive embedded software. In: 2021 47th Euromicro Conference on Software Engineering and Advanced Applications. SEAA, pp. 300-307. http://dx.doi.org/10.1109/ SEAA53835.2021.00046.
  116. Machmouchi, Widad, Awadallah, Ahmed Hassan, Zitouni, Imed, Buscher, Georg, 2017. Beyond success rate: Utility as a search quality metric for online experiments. In: Proceedings of the 2017 ACM on Conference on Information and Knowledge Management. CIKM '17, Association for Computing Machinery, New York, NY, USA, pp. 757-765. http://dx.doi.org/10.1145/3132847.3132850. · DOI
  117. Madeyski, Lech, Orzeszyna, Wojciech, Torkar, Richard, Józala, Mariusz, 2014. Overcoming the equivalent mutant problem: A systematic literature review and a comparative experiment of second order mutation. IEEE Trans. Softw. Eng. 40 (1), 23-42. http://dx.doi.org/10.1109/TSE.2013.44. · DOI
  118. Madlberger, Maria, Jizdny, Jiri, 2021. Impact of promotional social media content on click-through rate -evidence from a FMCG company. In: 20th International Conferences on WWW/Internet 2021 and Applied Computing 2021. pp. 3-10, URL https://www.scopus.com/inward/record.uri?eid=2-s2.0-85124068035& partnerID=40&md5=c0b8f49a3b48b3d561fd0ed305eb1895, Cited by: 0.
  119. Mahdavi-Hezavehi, Sara, Durelli, Vinicius H.S., Weyns, Danny, Avgeriou, Paris, 2017. A systematic literature review on methods that handle multiple quality attributes in architecture-based self-adaptive systems. Inf. Softw. Technol. 90, 1-26. http: //dx.doi.org/10.1016/j.infsof.2017.03.013. · DOI
  120. Masuda, Taisei, Murakami, Kyoko, Sugiura, Kenkichi, Sakui, Sho, Philip Schuring, Ron, Mori, Mitsuhiro, 2022. A phase 1/2 randomised placebo-controlled study of the COVID-19 vaccine mRNA-1273 in healthy Japanese adults: An interim report. Vaccine 40 (13), 2044-2052. http://dx.doi.org/10.1016/j.vaccine.2022.02.030. · DOI
  121. Mattos, David Issa, Bosch, Jan, Holmström Olsson, Helena, 2017a. More for less: Automated experimentation in software-intensive systems. In: Felderer, Michael, Méndez Fernández, Daniel, Turhan, Burak, Kalinowski, Marcos, Sarro, Federica, Winkler, Dietmar (Eds.), Product-Focused Software Process Improvement. Springer International Publishing, Cham, pp. 146-161. http://refhub.elsevier.com/S0164-1212(2400054-2/sb122)
  122. Mattos, David Issa, Bosch, Jan, Olsson, Helena Holmström, 2017b. Your system gets better every day you use it: Towards automated continuous experimentation. In: 2017 43rd Euromicro Conference on Software Engineering and Advanced Applications. SEAA, pp. 256-265. http://dx.doi.org/10.1109/SEAA.2017.15. · DOI
  123. Mattos, David Issa, Bosch, Jan, Olsson, Helena Holmström, 2018. Challenges and strategies for undertaking continuous experimentation to embedded systems: Industry and research perspectives. In: Garbajosa, Juan, Wang, Xiaofeng, Aguiar, Ademar (Eds.), Agile Processes in Software Engineering and Extreme Programming. Springer International Publishing, Cham, pp. 277-292. http://dx.doi.org/10.1007/978-3319-91602-6\_20. · DOI
  124. Mattos, David Issa, Bosch, Jan, Olsson, Helena Holmstrom, Maryam Korshani, Aita, Lantz, Jonn, 2020. Automotive A/B testing: Challenges and lessons learned from practice. In: 2020 46th Euromicro Conference on Software Engineering and Advanced Applications. SEAA, pp. 101-109. http://dx.doi.org/10.1109/SEAA51224. 2020.00026. · DOI
  125. Metrikov, Pavel, Diaz, Fernando, Lahaie, Sebastien, Rao, Justin, 2014. Whole page optimization: How page elements interact with the position auction. In: Proceedings of the Fifteenth ACM Conference on Economics and Computation. EC '14, Association for Computing Machinery, New York, NY, USA, pp. 583-600. http://dx.doi.org/10.1145/2600057.2602871. · DOI
  126. Miikulainen, Risto, Brundage, Myles, Epstein, Jonathan, Foster, Tyler, Hodjat, Babak, Iscoe, Neil, Jiang, Jingbo, Legrand, Diego, Nazari, Sam, Qiu, Xin, Scharff, Michael, Schoolland, Cory, Severn, Robert, Shagrin, Aaron, 2020. Ascend by evolv: AIbased massively multivariate conversion rate optimization. AI Mag. 41 (1), 44-60. http://dx.doi.org/10.1609/aimag.v41i1.5256. · DOI
  127. Okoshi, Tadashi, Tsubouchi, Kota, Tokuda, Hideyuki, 2019. Real-world product deployment of adaptive push notification scheduling on smartphones. In: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. KDD '19, Association for Computing Machinery, New York, NY, USA, pp. 2792-2800. http://dx.doi.org/10.1145/3292500.3330732. · DOI
  128. Ozawa, Takumi, Sekiguchi, Akiyuki, Tsuda, Kazuhiko, 2016. A method for the construction of user targeting knowledge for B2B industry website. Procedia Comput. Sci. 96, 1147-1155. http://dx.doi.org/10.1016/j.procs.2016.08.157, KnowledgeBased and Intelligent Information & Engineering Systems: Proceedings of the 20th International Conference KES-2016. · DOI
  129. Pelleg, Dan, Rokhlenko, Oleg, Szpektor, Idan, Agichtein, Eugene, Guy, Ido, 2016. When the crowd is not enough: Improving user experience with social media through automatic quality analysis. In: Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing. CSCW '16, Association for Computing Machinery, New York, NY, USA, pp. 1080-1090. http://dx.doi.org/ 10.1145/2818048.2820022. · DOI
  130. Peska, Ladislav, Vojtas, Peter, 2020. Off-line vs. On-line evaluation of recommender systems in small E-commerce. In: Proceedings of the 31st ACM Conference on Hypertext and Social Media. HT '20, Association for Computing Machinery, New York, NY, USA, pp. 291-300. http://dx.doi.org/10.1145/3372923.3404781. · DOI
  131. Poyarkov, Alexey, Drutsa, Alexey, Khalyavin, Andrey, Gusev, Gleb, Serdyukov, Pavel, 2016. Boosted decision tree regression adjustment for variance reduction in online controlled experiments. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD '16, Association for Computing Machinery, New York, NY, USA, pp. 235-244. http://dx.doi.org/10. 1145/2939672.2939688.
  132. Qu, Jia, Zhang, Jing, 2016. Validating mobile designs with agile testing in China: Based on baidu map for mobile. In: Marcus, Aaron (Ed.), Design, User Experience, and Usability: Design Thinking and Methods. Springer International Publishing, Cham, pp. 491-498. http://refhub.elsevier.com/S0164-1212(2400054-2/sb133)
  133. Quin, Federico, Weyns, Danny, Galster, Matthias, 2023. Study systematic literature review on A/B testing. URL https://people.cs.kuleuven.be/danny.weyns/material/ SLR\_AB/.
  134. Renz, Jan, Hoffmann, Daniel, Staubitz, Thomas, Meinel, Christoph, 2016. Using A/B testing in MOOC environments. In: Proceedings of the Sixth International Conference on Learning Analytics & Knowledge. LAK '16, Association for Computing Machinery, New York, NY, USA, pp. 304-313. http://dx.doi.org/10.1145/2883851. 2883876. · DOI
  135. Reza, Mohi, Kim, Juho, Bhattacharjee, Ananya, Rafferty, Anna N., Williams, Joseph Jay, 2021. The mooclet framework: Unifying experimentation, dynamic improvement, and personalization in online courses. In: Proceedings of the Eighth ACM Conference on Learning @ Scale. In: L@S '21, Association for Computing Machinery, New York, NY, USA, pp. 15-26. http://dx.doi.org/10.1145/3430895.3460128. · DOI
  136. Rodríguez, Pilar, Haghighatkhah, Alireza, Lwakatare, Lucy Ellen, Teppola, Susanna, Suomalainen, Tanja, Eskeli, Juho, Karvonen, Teemu, Kuvaja, Pasi, Verner, June M., Oivo, Markku, 2017. Continuous deployment of software intensive products and services: A systematic mapping study. J. Syst. Softw. 123, 263-291. http://dx.doi. org/10.1016/j.jss.2015.12.015. · DOI
  137. Ros, Rasmus, Runeson, Per, 2018. Continuous experimentation and A/B testing: A mapping study. In: Proceedings of the 4th International Workshop on Rapid Continuous Software Engineering. RCoSE '18, Association for Computing Machinery, New York, NY, USA, pp. 35-41. http://dx.doi.org/10.1145/3194760.3194766. · DOI
  138. Rosenfeld, Nir, Mansour, Yishay, Yom-Tov, Elad, 2017. Predicting counterfactuals from large historical data and small randomized trials. In: Proceedings of the 26th International Conference on World Wide Web Companion. In: WWW '17 Companion, International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, CHE, pp. 602-609. http://dx.doi.org/10.1145/3041021. 3054190. · DOI
  139. Sajeev, Sandra, Huang, Jade, Karampatziakis, Nikos, Hall, Matthew, Kochman, Sebastian, Chen, Weizhu, 2021. Contextual bandit applications in a customer support bot. In: Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. KDD '21, Association for Computing Machinery, New York, NY, USA, pp. 3522-3530. http://dx.doi.org/10.1145/3447548.3467165. · DOI
  140. Satyal, Suhrid, Weber, Ingo, Paik, Hye-young, Di Ciccio, Claudio, Mendling, Jan, 2017. AB-BPM: Performance-driven instance routing for business process improvement. In: Carmona, Josep, Engels, Gregor, Kumar, Akhil (Eds.), Business Process Management. Springer International Publishing, Cham, pp. 113-129. http://refhub.elsevier.com/S0164-1212(2400054-2/sb141)
  141. Satyal, Suhrid, Weber, Ingo, young Paik, Hye, Di Ciccio, Claudio, Mendling, Jan, 2019. Business process improvement with the AB-BPM methodology. Inf. Syst. 84, 283-298. http://dx.doi.org/10.1016/j.is.2018.06.007. · DOI
  142. Saveski, Martin, Pouget-Abadie, Jean, Saint-Jacques, Guillaume, Duan, Weitao, Ghosh, Souvik, Xu, Ya, Airoldi, Edoardo M., 2017. Detecting network effects: Randomizing over randomized experiments. In: Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD '17, Association for Computing Machinery, New York, NY, USA, pp. 1027-1035. http://dx.doi.org/10.1145/3097983.3098192. · DOI
  143. Schermann, Gerald, Schöni, Dominik, Leitner, Philipp, Gall, Harald C., 2016. Bifrost: Supporting continuous deployment with automated enactment of multi-phase live testing strategies. In: Proceedings of the 17th International Middleware Conference. Middleware '16, Association for Computing Machinery, New York, NY, USA, http: //dx.doi.org/10.1145/2988336.2988348. · DOI
  144. Shariat, Shahriar, Orten, Burkay, Dasdan, Ali, 2017. Online evaluation of bid prediction models in a large-scale computational advertising platform: Decision making and insights. Knowl. Inf. Syst. 51 (1), 37-60. http://dx.doi.org/10.1007/s10115-0160972-6. · DOI
  145. Shi, Fanjuan, Ghedira, Chirine, Marini, Jean-Luc, 2015. Context adaptation for smart recommender systems. IT Prof. 17 (6), 18-26. http://dx.doi.org/10.1109/MITP. 2015.96. · DOI
  146. Siegmund, Janet, Siegmund, Norbert, Apel, Sven, 2015. Views on internal and external validity in empirical software engineering. In: 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering. Vol. 1, IEEE Press, Los Alamitos, CA, USA, pp. 9-19. http://dx.doi.org/10.1109/ICSE.2015.24. · DOI
  147. Silberstein, Natalia, Somekh, Oren, Koren, Yair, Aharon, Michal, Porat, Dror, Shahar, Avi, Wu, Tingyi, 2020. Ad close mitigation for improved user experience in native advertisements. In: Proceedings of the 13th International Conference on Web Search and Data Mining. WSDM '20, Association for Computing Machinery, New York, NY, USA, pp. 546-554. http://dx.doi.org/10.1145/3336191.3371798. · DOI
  148. Siqueira, Jorge Gabriel, de Paula, Melise M.V., 2018. IPEAD A/B test execution framework. In: Proceedings of the XIV Brazilian Symposium on Information Systems. SBSI '18, Association for Computing Machinery, New York, NY, USA, http://dx.doi.org/10.1145/3229345.3229360. · DOI
  149. Siroker, Dan, Koomen, Pete, 2013. A/B Testing: The Most Powerful Way to Turn Clicks Into Customers, first ed. Wiley Publishing, Hoboken, NJ, USA. http://refhub.elsevier.com/S0164-1212(2400054-2/sb150)
  150. Spang, Bruce, Hannan, Veronica, Kunamalla, Shravya, Huang, Te-Yuan, McKeown, Nick, Johari, Ramesh, 2021. Unbiased experiments in congested networks. In: Proceedings of the 21st ACM Internet Measurement Conference. IMC '21, Association for Computing Machinery, New York, NY, USA, pp. 80-95. http://dx.doi.org/10.1145/ 3487552.3487851.
  151. Sriraman, Akshitha, Dhanotia, Abhishek, Wenisch, Thomas F., 2019. SoftSKU: Optimizing server architectures for microservice diversity @scale. In: 2019 ACM/IEEE 46th Annual International Symposium on Computer Architecture. ISCA, pp. 513-526. http://refhub.elsevier.com/S0164-1212(2400054-2/sb152)
  152. Stewart, Michael, Hodkiewicz, Melinda, Li, Sirui, 2023. Large language models for failure mode classification: An investigation. arXiv:2309.08181. · arXiv
  153. Sun, Fei, Jiang, Peng, Sun, Hanxiao, Pei, Changhua, Ou, Wenwu, Wang, Xiaobo, 2018. Multi-source pointer network for product title summarization. In: Proceedings of the 27th ACM International Conference on Information and Knowledge Management. CIKM '18, Association for Computing Machinery, New York, NY, USA, pp. 7-16. http://dx.doi.org/10.1145/3269206.3271722. · DOI
  154. Szpektor, Idan, Maarek, Yoelle, Pelleg, Dan, 2013. When relevance is not enough: Promoting diversity and freshness in personalized question recommendation. In: Proceedings of the 22nd International Conference on World Wide Web. WWW '13, Association for Computing Machinery, New York, NY, USA, pp. 1249-1260. http://dx.doi.org/10.1145/2488388.2488497. · DOI
  155. Tagami, Yukihiro, Hotta, Toru, Tanaka, Yusuke, Ono, Shingo, Tsukamoto, Koji, Tajima, Akira, 2014. Filling context-ad vocabulary gaps with click logs. In: Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD '14, Association for Computing Machinery, New York, NY, USA, pp. 1955-1964. http://dx.doi.org/10.1145/2623330.2623334. · DOI
  156. Tamburrelli, Giordano, Margara, Alessandro, 2014. Towards automated A/B testing. In: Le Goues, Claire, Yoo, Shin (Eds.), Search-Based Software Engineering. Springer International Publishing, Cham, pp. 184-198. http://refhub.elsevier.com/S0164-1212(2400054-2/sb157)
  157. Tang, Diane, Agarwal, Ashish, O'Brien, Deirdre, Meyer, Mike, 2010. Overlapping experiment infrastructure: More, better, faster experimentation. In: Proceedings of the 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD '10, Association for Computing Machinery, New York, NY, USA, pp. 17-26. http://dx.doi.org/10.1145/1835804.1835810. · DOI
  158. Toslali, Mert, Parthasarathy, Srinivasan, Oliveira, Fabio, Coskun, Ayse K., 2020. JACKPOT: Online experimentation of cloud microservices. In: HotCloud 2020 -12th USENIX Workshop on Hot Topics in Cloud Computing, co-located with USENIX ATC 2020. URL https://www.scopus.com/inward/record.uri?eid=2-s2.085091892156&partnerID=40&md5=cae12fe24f34f2bb0818e448f8c07fbf, Cited by: 1.
  159. Tsigkanos, Christos, Rani, Pooja, Müller, Sebastian, Kehrer, Timo, 2023. Variable discovery with large language models for metamorphic testing of scientific software. In: Computational Science - ICCS 2023. Springer Nature Switzerland, Cham, pp. 321-335. http://refhub.elsevier.com/S0164-1212(2400054-2/sb160)
  160. Tu, Ye, Basu, Kinjal, DiCiccio, Cyrus, Bansal, Romil, Nandy, Preetam, Jaikumar, Padmini, Chatterjee, Shaunak, 2021. Personalized treatment selection using causal heterogeneity. In: Proceedings of the Web Conference 2021. WWW '21, Association for Computing Machinery, New York, NY, USA, pp. 1574-1585. http://dx.doi.org/ 10.1145/3442381.3450075. · DOI
  161. Ueoka, Yutaro, Tsubouchi, Kota, Shimizu, Nobuyuki, 2020. Tackling cannibalization problems for online advertisement. In: Proceedings of the 28th ACM Conference on User Modeling, Adaptation and Personalization. UMAP '20, Association for Computing Machinery, New York, NY, USA, pp. 358-362. http://dx.doi.org/10. 1145/3340631.3394875.
  162. Vanderdonckt, Jean, Zen, Mathieu, Vatavu, Radu-Daniel, 2019. AB4web: An on-line A/B tester for comparing user interface design alternatives. Proc. ACM Hum.-Comput. Interact. 3 (EICS), http://dx.doi.org/10.1145/3331160. · DOI
  163. Vasthimal, Deepak Kumar, Srirama, Pavan Kumar, Akkinapalli, Arun Kumar, 2019. Scalable data reporting platform for A/B tests. In: 2019 IEEE 5th Intl Conference on Big Data Security on Cloud (BigDataSecurity), IEEE Intl Conference on High Performance and Smart Computing, (HPSC) and IEEE Intl Conference on Intelligent Data and Security. IDS, pp. 230-238. http://dx.doi.org/10.1109/BigDataSecurityHPSC-IDS.2019.00052. · DOI
  164. Vaswani, Ashish, Shazeer, Noam, Parmar, Niki, Uszkoreit, Jakob, Jones, Llion, Gomez, Aidan N., Kaiser, Lukasz, Polosukhin, Illia, 2023. Attention is all you need. arXiv:1706.03762. · arXiv
  165. Walper, Daniel, Kassau, Julia, Methfessel, Philipp, Pronold, Timo, Einhauser, Wolfgang, 2020. Optimizing user interfaces in food production: gaze tracking is more sensitive for A-B-testing than behavioral data alone. In: ACM Symposium on Eye Tracking Research and Applications. In: ETRA '20 Short Papers, Association for Computing Machinery, New York, NY, USA, pp. 1-4. http://dx.doi.org/10.1145/3379156. 3391351. · DOI
  166. Wang, Zenan, Carrion, Carlos, Lin, Xiliang, Ji, Fuhua, Bao, Yongjun, Yan, Weipeng, 2022. Adaptive experimentation with delayed binary feedback. In: Proceedings of the ACM Web Conference 2022. WWW '22, Association for Computing Machinery, New York, NY, USA, pp. 2247-2255. http://dx.doi.org/10.1145/3485447.3512097. · DOI
  167. Wang, Yu, Gupta, Somit, Lu, Jiannan, Mahmoudzadeh, Ali, Liu, Sophia, 2019. On heavy-user bias in A/B testing. In: Proceedings of the 28th ACM International Conference on Information and Knowledge Management. CIKM '19, Association for Computing Machinery, New York, NY, USA, pp. 2425-2428. http://dx.doi.org/ 10.1145/3357384.3358143. · DOI
  168. Wang, Jian, Hardtke, David, 2015. User latent preference model for better downside management in recommender systems. In: Proceedings of the 24th International Conference on World Wide Web. WWW '15, International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, CHE, pp. 1209-1219. http://dx.doi.org/10.1145/2736277.2741126. · DOI
  169. Wang, Weinan, Zhang, Xi, 2021. CONQ: Continuous quantile treatment effects for largescale online controlled experiments. In: Proceedings of the 14th ACM International Conference on Web Search and Data Mining. WSDM '21, Association for Computing Machinery, New York, NY, USA, pp. 202-210. http://dx.doi.org/10.1145/3437963. 3441779. · DOI
  170. Wei, Jason, Tay, Yi, Bommasani, Rishi, Raffel, Colin, Zoph, Barret, Borgeaud, Sebastian, Yogatama, Dani, Bosma, Maarten, Zhou, Denny, Metzler, Donald, Chi, Ed H., Hashimoto, Tatsunori, Vinyals, Oriol, Liang, Percy, Dean, Jeff, Fedus, William, 2022. Emergent abilities of large language models. arXiv:2206.07682. · arXiv
  171. Wu, Liang, Grbovic, Mihajlo, 2020. How airbnb tells you will enjoy sunset sailing in Barcelona? recommendation in a two-sided travel marketplace. In: Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. SIGIR '20, Association for Computing Machinery, New York, NY, USA, pp. 2387-2396. http://dx.doi.org/10.1145/3397271.3401444. · DOI
  172. Wu, Jie J.W., Mazzuchi, Thomas A., Sarkani, Shahram, 2023a. Comparison of multicriteria decision-making methods for online controlled experiments in a launch decision-making framework. Inf. Softw. Technol. 155, 107115. http://dx.doi.org/ 10.1016/j.infsof.2022.107115. · DOI
  173. Wu, Jie J.W., Mazzuchi, Thomas A., Sarkani, Shahram, 2023b. A multi-objective evolutionary approach towards automated online controlled experiments. J. Syst. Softw. 203, 111703. http://dx.doi.org/10.1016/j.jss.2023.111703. · DOI
  174. Wu, Yuhang, Zheng, Zeyu, Zhang, Guangyu, Zhang, Zuohua, Wang, Chu, 2022. Nonstationary A/B tests. In: Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. KDD '22, Association for Computing Machinery, New York, NY, USA, pp. 2079-2089. http://dx.doi.org/10.1145/3534678. 3539325. · DOI
  175. Xia, Tong, Bhardwaj, Sumit, Dmitriev, Pavel, Fabijan, Aleksander, 2019. Safe velocity: A practical guide to software deployment at scale using controlled rollout. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). pp. 11-20. http://dx.doi.org/10.1109/ICSESEIP.2019.00010. · DOI
  176. Xie, Yuxiang, Chen, Nanyu, Shi, Xiaolin, 2018. False discovery rate controlled heterogeneous treatment effect detection for online controlled experiments. In: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. KDD '18, Association for Computing Machinery, New York, NY, USA, pp. 876-885. http://dx.doi.org/10.1145/3219819.3219860. · DOI
  177. Xie, Yuxiang, Xu, Meng, Chow, Evan, Shi, Xiaolin, 2021. How to measure your app: A couple of pitfalls and remedies in measuring app performance in online controlled experiments. In: Proceedings of the 14th ACM International Conference on Web Search and Data Mining. WSDM '21, Association for Computing Machinery, New York, NY, USA, pp. 949-957. http://dx.doi.org/10.1145/3437963.3441742. · DOI
  178. Xu, Ya, Chen, Nanyu, 2016. Evaluating mobile apps with A/B and quasi A/B tests. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD '16, Association for Computing Machinery, New York, NY, USA, pp. 313-322. http://dx.doi.org/10.1145/2939672.2939703. · DOI
  179. Xu, Ya, Chen, Nanyu, Fernandez, Addrian, Sinno, Omar, Bhasin, Anmol, 2015. From infrastructure to culture: A/B testing challenges in large scale social networks. In: Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD '15, Association for Computing Machinery, New York, NY, USA, pp. 2227-2236. http://dx.doi.org/10.1145/2783258.2788602. · DOI
  180. Xu, Ye, Li, Zang, Gupta, Abhishek, Bugdayci, Ahmet, Bhasin, Anmol, 2014. Modeling professional similarity by mining professional career trajectories. In: Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD '14, Association for Computing Machinery, New York, NY, USA, pp. 1945-1954. http://dx.doi.org/10.1145/2623330.2623368. · DOI
  181. Xu, Yanbo, Mahajan, Divyat, Manrao, Liz, Sharma, Amit, Kıcıman, Emre, 2021. Split-treatment analysis to rank heterogeneous causal effects for prospective interventions. In: Proceedings of the 14th ACM International Conference on Web Search and Data Mining. WSDM '21, Association for Computing Machinery, New York, NY, USA, pp. 409-417. http://dx.doi.org/10.1145/3437963.3441821. · DOI
  182. Yaman, Sezin Gizem, Munezero, Myriam, Münch, Jürgen, Fagerholm, Fabian, Syd, Ossi, Aaltola, Mika, Palmu, Christina, Männistö, Tomi, 2017. Introducing continuous experimentation in large software-intensive product and service organisations. J. Syst. Softw. 133, 195-211. http://dx.doi.org/10.1016/j.jss.2017.07.009. · DOI
  183. Yang, Wanshan, Yang, Gemeng, Huang, Ting, Chen, Lijun, Liu, Youjian Eugene, 2018. Whales, dolphins, or minnows? Towards the player clustering in free online games based on purchasing behavior via data mining technique. In: 2018 IEEE International Conference on Big Data (Big Data). pp. 4101-4108. http://dx.doi. org/10.1109/BigData.2018.8622067. · DOI
  184. Ye, Runlong, Chen, Pan, Mao, Yini, Wang-Lin, Angela, Shaikh, Hammad, Zavaleta Bernuy, Angela, Williams, Joseph Jay, 2022. Behavioral consequences of reminder emails on students' academic performance: A real-world deployment. In: Proceedings of the 23rd Annual Conference on Information Technology Education. SIGITE '22, Association for Computing Machinery, New York, NY, USA, pp. 16-22. http://dx.doi.org/10.1145/3537674.3554740. · DOI
  185. Yoneda, Takeshi, Kozawa, Shunsuke, Osone, Keisuke, Koide, Yukinori, Abe, Yosuke, Seki, Yoshifumi, 2019. Algorithms and system architecture for immediate personalized news recommendations. In: IEEE/WIC/ACM International Conference on Web Intelligence. WI '19, Association for Computing Machinery, New York, NY, USA, pp. 124-131. http://dx.doi.org/10.1145/3350546.3352509. · DOI
  186. Young, Scott W.H., 2014. Improving library user experience with A/B testing: Principles and process. Weave: J. Library User Exp. 1, http://dx.doi.org/10.3998/weave. 12535642.0001.101. · DOI
  187. Yu, Miao, Lu, Wenbin, Song, Rui, 2020. A new framework for online testing of heterogeneous treatment effect. In: AAAI 2020 -34th AAAI Conference on Artificial Intelligence. pp. 10310-10317, URL https://www. scopus.com/inward/record.uri?eid=2-s2.0-85106588123&partnerID=40&md5= 53544f162212be7cd129e1f196debcd8, Cited by: 2.
  188. Zhang, He, Ali Babar, Muhammad, 2010. On searching relevant studies in software engineering. In: Proceedings of the 14th International Conference on Evaluation and Assessment in Software Engineering. EASE '10, BCS Learning & Development Ltd., Swindon, GBR, pp. 111-120. http://refhub.elsevier.com/S0164-1212(2400054-2/sb189)
  189. Zhao, Zhenyu, He, Yan, Chen, Miao, 2017. Inform product change through experimentation with data-driven behavioral segmentation. In: 2017 IEEE International Conference on Data Science and Advanced Analytics. DSAA, pp. 69-78. http: //dx.doi.org/10.1109/DSAA.2017.65. · DOI
  190. Zhu, Xizhou, Chen, Yuntao, Tian, Hao, Tao, Chenxin, Su, Weijie, Yang, Chenyu, Huang, Gao, Li, Bin, Lu, Lewei, Wang, Xiaogang, Qiao, Yu, Zhang, Zhaoxiang, Dai, Jifeng, 2023. Ghost in the minecraft: Generally capable agents for open-world environments via large language models with text-based knowledge and memory. arXiv:2305.17144. · arXiv

Cited by

Sorted by citations

Authors
Quin, F., Weyns, D., Galster, M., & Silva, C. C.
Affiliations
  • KU Leuven (BE)
  • Linnaeus University (SE)
  • University of Canterbury (NZ)
Corresponding author
Quin, Federico
ORCID
Venue
Journal of Systems and Software
Publisher
Elsevier BV
Published
2024-05
Volume
211
Pages
112011
ISSN
0164-1212 (ISSN-L)
DOI
10.1016/j.jss.2024.112011
OpenAlex
W4392059507
Cited by
83
References
1
Citations by year
2026: 30 · 2025: 39 · 2024: 14
FWCI
4.94
Type
journal article
Language
en
Open access
Licence
Locations
  • Journal of Systems and Software (publishedVersion, cc-by)
  • Lirias (KU Leuven) (submittedVersion, cc-by)
  • DiVA (Linnaeus University) (submittedVersion, other-oa)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
  • Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
  • Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
Keywords
A/B testing, continuous experimentation, hypothesis testing, data-driven decision making, software development, stakeholder roles, feature selection

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review