Tests, when not correctly implemented, can pass on incorrect system implementations rather than fail. In this case, they are named silent horrors or false-negative tests. They make releasing low-quality (buggy) versions of the software system more probable. Furthermore, faithfully implementing test specifications is crucial when they play the role of documentation, like when documenting components or services or driving legacy systems' re-engineering. This paper presents TestWizard, a novel approach and tool for automatically assessing individual tests' quality from the point of view of their coherence to specifications. TestWizard automatically assesses the quality of each individual test case w.r.t. its specification, providing detailed reports on why a single test is a false negative, hence helping testers fix them. Thus, TestWizard can help to automate the test code review process, which is still mainly manual today. The analysis of 1012 test implementations, developed by 123 students in three experiments, shows that TestWizard is (1) by far more accurate than code review performed by multiple students, (2) slightly better than code review performed by three senior experts, and (3) always able to detect a significant percentage of false-negative test methods (up to 21.22%).

Fight silent horror unit test methods by consulting a TestWizard

Cerioli M.;Lagorio G.;Leotta M.;Ricca F.
2021

Abstract

Tests, when not correctly implemented, can pass on incorrect system implementations rather than fail. In this case, they are named silent horrors or false-negative tests. They make releasing low-quality (buggy) versions of the software system more probable. Furthermore, faithfully implementing test specifications is crucial when they play the role of documentation, like when documenting components or services or driving legacy systems' re-engineering. This paper presents TestWizard, a novel approach and tool for automatically assessing individual tests' quality from the point of view of their coherence to specifications. TestWizard automatically assesses the quality of each individual test case w.r.t. its specification, providing detailed reports on why a single test is a false negative, hence helping testers fix them. Thus, TestWizard can help to automate the test code review process, which is still mainly manual today. The analysis of 1012 test implementations, developed by 123 students in three experiments, shows that TestWizard is (1) by far more accurate than code review performed by multiple students, (2) slightly better than code review performed by three senior experts, and (3) always able to detect a significant percentage of false-negative test methods (up to 21.22%).
File in questo prodotto:
Non ci sono file associati a questo prodotto.

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: http://hdl.handle.net/11567/1071087
 Attenzione

Attenzione! I dati visualizzati non sono stati sottoposti a validazione da parte dell'ateneo

Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus 2
  • ???jsp.display-item.citation.isi??? 1
social impact