Comparing the Effectiveness of Assertions with Differential Testing in the Context of Web Testing

Differential testing applied in the Web context compares the current web page under test with a snapshot considered correct taken from a previous version. This technique appears to be promising and an alternative to assertions in catching regressions due to the evolution of the web application under test. This paper empirically compares Selenium WebDriver test scripts equipped with (1) assertions and (2) differential testing implemented in the Recheck tool. The comparison included costs (both test scripts development time and execution time) and effectiveness (bugs detection capability) considering two different versions of differential testing implemented in Recheck, named implicit and explicit. Results show that, on average, Recheck (both explicit and implicit) is able to detect more bugs than classic assertions (up to +34% on complex apps). The development time is similar between the two approaches. The execution time is slightly higher than classic assertions for Recheck explicit (+33%), while it is by far higher when Recheck implicit is adopted (3.6 times). In conclusion, the best choice, considering both the effectiveness and the costs, appears to be Recheck explicit.