TY - JOUR
KW - Item Response Theory
KW - Adaptive Testing
KW - Model Evaluation
KW - Benchmark
AU - Hao Song
AU - Peter Flach
AB - Progress in predictive machine learning is typically measured on the basis of performance comparisons on benchmark datasets. Traditionally these kinds of empirical evaluation are carried out on large numbers of datasets, but this is becoming increasingly hard due to computational requirements and the often large number of alternative methods to compare against. In this paper we investigate adaptive approaches to achieve better efficiency on model benchmarking. For a large collection of datasets, rather than training and testing a given approach on every individual dataset, we seek methods that allow us to pick only a few representative datasets to quantify the model’s goodness, from which to extrapolate to performance on other datasets. To this end, we adapt existing approaches from psychometrics: specifically, Item Response Theory and Adaptive Testing. Both are well-founded frameworks designed for educational tests. We propose certain modifications following the requirements of machine learning experiments, and present experimental results to validate the approach.
IS - Special Issue on Artificial Intelligence, Paving the Way to the Future
M1 - 5
N2 - Progress in predictive machine learning is typically measured on the basis of performance comparisons on benchmark datasets. Traditionally these kinds of empirical evaluation are carried out on large numbers of datasets, but this is becoming increasingly hard due to computational requirements and the often large number of alternative methods to compare against. In this paper we investigate adaptive approaches to achieve better efficiency on model benchmarking. For a large collection of datasets, rather than training and testing a given approach on every individual dataset, we seek methods that allow us to pick only a few representative datasets to quantify the model’s goodness, from which to extrapolate to performance on other datasets. To this end, we adapt existing approaches from psychometrics: specifically, Item Response Theory and Adaptive Testing. Both are well-founded frameworks designed for educational tests. We propose certain modifications following the requirements of machine learning experiments, and present experimental results to validate the approach.
PY - 2021
SP - 110
EP - 118
T2 - International Journal of Interactive Multimedia and Artificial Intelligence
TI - Efficient and Robust Model Benchmarks with Item Response Theory and Adaptive Testing
UR - https://www.ijimai.org/journal/sites/default/files/2021-02/ijimai_6_5_11_0.pdf
VL - 6
SN - 1989-1660
ER -