Do Automatically Generated Unit Tests Find Real Faults? An Empirical Study of Effectiveness and Challenges

by Sina Shamshiri, Rene Just, José Miguel Rojas, Gordon Fraser, Phil McMinn, and Andrea Arcuri

International Conference on Automated Software Engineering (ASE 2015)



Rather than tediously writing unit tests manually, tools can be used to generate them automatically—sometimes even resulting in higher code coverage than manual testing. But how good are these tests at actually finding faults? To answer this question, we applied three state-of-the-art unit test generation tools for Java (Randoop, EvoSuite, and Agitar) to the 357 real faults in the Defects4J dataset and investigated how well the generated test suites perform at detecting these faults. Although the automatically generated test suites detected 55.7% of the faults overall, only 19.9% of all the individual test suites detected a fault. By studying the ... [more]


Reference

Sina Shamshiri, Rene Just, José Miguel Rojas, Gordon Fraser, Phil McMinn, and Andrea Arcuri. Do Automatically Generated Unit Tests Find Real Faults? An Empirical Study of Effectiveness and Challenges. International Conference on Automated Software Engineering (ASE 2015), pp. 201–211, 2015


Bibtex Entry
@inproceedings{Shamshiri2015,
  author    = "Shamshiri, Sina and Just, Rene and Rojas, Jos\'{e} Miguel and Fraser, Gordon and McMinn, Phil and Arcuri, Andrea",
  title     = "Do Automatically Generated Unit Tests Find Real Faults? An Empirical Study of Effectiveness and Challenges",
  booktitle = "International Conference on Automated Software Engineering (ASE 2015)",
  pages     = "201--211",
  year      = "2015",
  publisher = "ACM"
}