Reducing Flakiness in End-to-End Test Suites: An Experience Report

End-to-end (E2E) testing, a technique employed to assure quality of web applications, is cost-effective only if the test suite is not flaky. Flaky test scripts produce non-deterministic results that undermine testers’ trust and thus the usefulness of the entire test suite. Recently, we were involved in the refactoring of an existing automated flaky E2E test suite for a large Web application. In this paper, we report on our experience. During the refactoring, we have computed the effort made and formalized the procedure we followed in algorithmic way so that our experience can also be of help to other developers/testers. Our procedure allowed to reduce the flakiness to virtually zero w.r.t. the original flaky test suite. Moreover, as a positive side effect, the execution time of the test suite has been reduced by of the 57%.