01676nas a2200217   4500000000100000000000100001008004100002260001200043653002500055653002100080653002100101653001400122100001300136700001600149245008900165856008300254300001200337490000600349520108900355022001401444       2021                            d  c03/202110aItem Response Theory10aAdaptive Testing10aModel Evaluation10aBenchmark1 aHao Song1 aPeter Flach00aEfficient and Robust Model Benchmarks with Item Response Theory and Adaptive Testing  uhttps://www.ijimai.org/journal/sites/default/files/2021-02/ijimai_6_5_11_0.pdf  a110-1180 v63 aProgress in predictive machine learning is typically measured on the basis of performance comparisons on benchmark datasets. Traditionally these kinds of empirical evaluation are carried out on large numbers of datasets, but this is becoming increasingly hard due to computational requirements and the often large number of alternative methods to compare against. In this paper we investigate adaptive approaches to achieve better efficiency on model benchmarking. For a large collection of datasets, rather than training and testing a given approach on every individual dataset, we seek methods that allow us to pick only a few representative datasets to quantify the model’s goodness, from which to extrapolate to performance on other datasets. To this end, we adapt existing approaches from psychometrics: specifically, Item Response Theory and Adaptive Testing. Both are well-founded frameworks designed for educational tests. We propose certain modifications following the requirements of machine learning experiments, and present experimental results to validate the approach.  a1989-1660