A family of experiments to quantify the benefits of adopting WebDriverManager and Selenium-Jupiter

Context: While test automation offers numerous benefits, it also introduces significant challenges. Two challenges that developers and testers face on a daily basis, particularly when using Selenium WebDriver to test web applications, are driver management (involving tasks such as version identification, download, installation, and maintenance) and management of test lifecycle phases (using specific test libraries, as for example JUnit, and inserting annotations into the code). These manual tasks make test suite development particularly tedious, error-prone, and expensive. Recently, to ease the burden on developers and testers, some Java libraries have been proposed, called WebDriverManager and Selenium-Jupiter, capable of automatically carrying out the driver management process for Selenium WebDriver and simplifying the development of test suites. These libraries appear to be very promising but until now no one has experimentally evaluated their effectiveness. Objective: To investigate the effectiveness of WebDriverManager and Selenium-Jupiter in reducing driver management times and boilerplate code. Method: We designed and conducted a family of experiments (three for WebDriverManager and two for Selenium-Jupiter) with 104 master student participants from the University of Genoa, Italy (across academic years 2021/2022 and 2022/2023) and nine professional participants. Results: Results indicate that the adoption of Selenium WebDriver with WebDriverManager significantly reduces setup time for multi-browser test suites from 33% to 50% (depending on the tester experience). Additionally, Selenium-Jupiter reduces test suite development time significantly (20% on average). Although it also decreases total code length, the reduction is relatively small compared to overall code length. Conclusion: WebDriverManager and Selenium-Jupiter can be seen as valuable solutions for enhancing testers’ productivity by shortening the time needed to develop test suites and minimizing the amount of code to write.