Two experiments for evaluating the impact of Hamcrest and AssertJ on assertion development

Context. Test automation enables continuous testing, a cornerstone of agile methods, and DevOps. Assertions play a fundamental role in test automation, and recently assertion libraries competing for Unit testing frameworks, such as, for example, JUnit or TestNG, emerged. Thus, it is imperative to gauge assertion libraries in terms of developer/tester productivity, allowing SQA Managers and Testers to select the best. Objective. The goal of this work is comparing two assertion libraries having a different approach (matchers vs. fluent assertions) w.r.t. two dependent variables: correctness of developed assertions and time to develop them. Method. We conducted two controlled experiments with Bachelor students in Computer Science and Master students in Computer Engineering. AssertJ is compared with Hamcrest, in a test development scenario with the Java language where 672 assertions were developed by 48 students overall. Results. The results show that: a) adopting AssertJ improves the tester’s productivity significantly during the development of assertions only for Bachelor students, and b) time of developing assertions is similar using AssertJ or Hamcrest in both the categories of participants. Conclusions. Testers and SQA managers selecting assertion libraries for their organizations should consider as first choice AssertJ in case of inexperienced developers/testers since our study shows that it increases the productivity of Bachelor students more than Hamcrest.