Automated grading is used to assess large numbers of students’ programs in software engineering courses, often utilising test suites to evaluate the correctness of these programs. However, test suites can vary in how they evaluate a program. In this paper, we investigate how much different suites influence generated grades, and how their properties contribute to this influence. We conduct a modified replication study of existing work, using students’ faulty solution programs and test suites that we constructed using a sampling approach. We find that differing test suites generate greatly varying grades, with the standard deviation of grades for each solution ... [more] typically representing 63% of the maximum change in grades. We also show that different properties of test suites influence the grades that they produce, including size, the ability to detect other students’ faults and artificial faults, and various coverage-based metrics. We use our findings to offer tutors with suggestions on how to build test suites that evaluate students’ software in a fair and consistent manner. Our suggestions include constructing test suites with high coverage, writing unique and diverse tests which evaluate solutions’ correctness in different ways, and to run the tests against artificial faults to determine their quality.

DOI

Bibtex Entry

@inproceedings{Clegg2021,
  author    = "Clegg, Ben and McMinn, Phil and Fraser, Gordon",
  title     = "An Empirical Study to Determine if Mutants Can Effectively Simulate Students' Programming Mistakes to Increase Tutors' Confidence in Autograding",
  booktitle = "ACM SIGCSE Technical Symposium on Computer Science Education (SIGCSE 2021)",
  pages     = "1055--1061",
  year      = "2021"
}