Assessed by Machines: Development of a TAM-Based Tool to Measure AI-based Assessment Acceptance Among Students.

J.C. Sánchez Prieto; J. Cruz Benito; R. Therón; F. J. García Peñalvo

doi:10.9781/ijimai.2020.11.009

Authors

J.C. Sánchez Prieto Universidad de Salamanca
J. Cruz Benito IBM Research - Thomas J. Watson Research Center
R. Therón Universidad de Salamanca
F. J. García Peñalvo Universidad de Salamanca https://orcid.org/0000-0001-9987-5584

DOI:

https://doi.org/10.9781/ijimai.2020.11.009

Keywords:

Artificial Intelligence, e-assessment, Adoption

Supporting Agencies

We would like to thank to the GRIAL Research Group of the University of Salamanca the support received during this research. This work has been partially funded by the Spanish Government Ministry of Economy and Competitiveness throughout the DEFINES project (Ref. TIN2016-80172-R).

Abstract

In recent years, the use of more and more technology in education has been a trend. The shift of traditional learning procedures into more online and tech-ish approaches has contributed to a context that can favor integrating Artificial-Intelligence-based or algorithm-based assessment of learning. Even more, with the current acceleration because of the COVID-19 pandemic, more and more learning processes are becoming online and are incorporating technologies related to automatize assessment or help instructors in the process. While we are in an initial stage of that integration, it is the moment to reflect on the students' perceptions of being assessed by a non-conscious software entity like a machine learning model or any other artificial intelligence application. As a result of the paper, we present a TAM-based model and a ready-to-use instrument based on five aspects concerning understanding technology adoption like the AI-based assessment on education. These aspects are perceived usefulness, perceived ease of use, attitude towards use, behavioral intention, and actual use. The paper's outcomes can be relevant to the research community since there is a lack of this kind of proposal in the literature.

Downloads

Download data is not yet available.

References

[1] S. Makridakis, “The forthcoming artificial intelligence (AI) revolution: Its impact on society and firms,” Futures, vol. 90, pp. 46-60, 2017.

[2] I. Roll and R. Wylie, “Evolution and revolution in artificial intelligence in education,” International Journal of Artificial Intelligence in Education, vol. 26, no. 2, pp. 582-599, 2016. https://doi.org/10.1007/s40593-016-0110-3.

[3] A. Abdul, J. Vermeulen, D. Wang, B.Y. Lim and M. Kankanhalli, “Trends and Trajectories for Explainable, Accountable and Intelligible Systems: An HCI Research Agenda,” in Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, Montreal QC, Canada, 2018, pp. 1–18.

[4] European Commission, “Smart lie-detection system to tighten EU’s busy borders,” European Commission Website, 2018. https://ec.europa.eu/research/infocentre/article_en.cfm?id=/research/headlines/news/article_18_10_24-3_en.html%3Finfocentre&item=Infocentre&artid=49726&pk_campaign=rss_page

[5] J. Cook, “Amazon scraps “sexist AI” recruiting tool that showed bias against women,” The Telegraph 2018. https://www.telegraph.co.uk/technology/2018/10/10/amazon-scraps-sexist-ai-recruiting-tool-showedbias-against/.

[6] “China’s hi-tech war on its muslim minority,” The Guardian 2019. http://www.theguardian.com/news/2019/apr/11/china-hi-tech-war-onmuslim-minority-xinjiang-uighurs-surveillance-face-recognition.

[7] J. Kim, K. Merrill Jr., K. Xu and D.D. Sellnow, “My teacher is a machine: Understanding students’ perceptions of AI teaching assistants in online education”, International Journal of Human–Computer Interaction, pp. 1-10, 2020. https://doi.org/10.1080/10447318.2020.1801227.

[8] J.C. Sánchez-Prieto, A. Gamazo, J. Cruz-Benito, R. Therón and F.J. García-Peñalvo, “AI-Driven Assessment of Students: Current Uses and Research Trends” in Learning and Collaboration Technologies. Designing, Developing and Deploying Learning Experiences. HCII 2020. Lecture Notes in Computer Science, 2020, pp. 292-302.

[9] J.C. Sánchez-Prieto, J. Cruz-Benito, R. Therón and F.J. García-Peñalvo, “How to Measure Teachers’ Acceptance of AI-Driven Assessment in ELearning: A TAM-Based Proposal,” in Proceedings of the Seventh International Conference on Technological Ecosystems for Enhancing Multiculturality, León, Spain, 2019, pp. 181–186.

[10] J. Cruz-Benito, J.C. Sánchez-Prieto, R. Therón and F.J. García-Peñalvo, “Measuring Students’ Acceptance to AI-Driven Assessment in eLearning: Proposing a First TAM-Based Research Model,” in Learning and Collaboration Technologies. Designing, Developing and Deploying Learning Experiences. HCII 2019. Lecture Notes in Computer Science, 2019, pp. 15-25.

[11] P. García, S. Schiaffino and A. Amandi, “An enhanced bayesian model to detect students’ learning styles in web-based courses,” J.Comput.Assisted Learn., vol. 24, no. 4, pp. 305-315, 2008. https://doi.org/10.1111/j.1365-2729.2007.00262.x.

[12] Y. Wei, Q. Yang, J. Chen and J. Hu, “The exploration of a machine learning approach for the assessment of learning styles challenges,” Mechatronic Systems and Control, vol. 46, no. 3, pp. 121-126, 2018.

[13] C. Verma, A. S. Tarawneh, Z. Illes, V. Stoffova and S. Dahiya, “Gender Prediction of the European School’s Teachers Using Machine Learning: Preliminary Results,” in - 2018 IEEE 8th International Advance Computing Conference (IACC), 2018, pp. 213-220.

[14] D.B. Chin, I.M. Dohmen, B.H. Cheng, M.A. Oppezzo, C.C. Chase and D.L. Schwartz, “Preparing students for future learning with teachable agents,” Educational Technology Research and Development, vol. 58, no. 6, pp. 649- 669, 2010.

[15] D. Xu and H. Wang, “Intelligent agent supported personalization for virtual learning environments,” Decis.Support Syst., vol. 42, no. 2, pp. 825- 843, 2006.

[16] T.B. Sheridan, “Human–Robot interaction: Status and challenges,” Hum.Factors, vol. 58, no. 4, pp. 525-532, 2016. https://doi.org/10.1177/0018720816644364.

[17] R.J. Stiggins, N.F. Conklin and N.J. Bridgeford, “Classroom assessment: A key to effective education,” Educational Measurement: Issues and Practice, vol. 5, no. 2, pp. 5-17, 1986.

[18] J. Ma, J. Kang, E. Shaw and J. Kim, “Workflow-Based Assessment of Student Online Activities with Topic and Dialogue Role Classification” in Artificial Intelligence in Education. AIED 2011. Lecture Notes in Computer Science, 2011, pp. 187-195.

[19] S. Singh and S. P. Lal, “Educational courseware evaluation using Machine Learning techniques,” in - 2013 IEEE Conference on e-Learning, e-Management and e-Services, 2013, pp. 73-78.

[20] R. Zatarain-Cabada, M.L. Barrón-Estrada and J.M. Ríos-Félix, “Affective Learning System for Algorithmic Logic Applying Gamification” in Advances in Soft Computing. MICAI 2016. Lecture Notes in Computer Science, 2017, pp. 536-547.

[21] K. Kuk, I.Z. Milentijević, D. Ranđelović, B.M. Popović and P. Čisar, “The design of the personal enemy - MIMLebot as an intelligent agent in a game-based learning environment,” Acta Polytechnica Hungarica, vol. 14, pp. 121-139, 2017.

[22] D. Dudek, “Survey Analyser: Effective Processing of Academic Questionnaire Data,” in Information Systems Architecture and Technology: Proceedings of 39th International Conference on Information Systems Architecture and Technology – ISAT 20, 2019, pp. 245-257.

[23] A. Tzacheva, J. Ranganathan and R. Jadi, “Multi-Label Emotion Mining From Student Comments,” in Proceedings of the 2019 4th International Conference on Information and Education Innovations, Durham, United Kingdom, 2019, pp. 120-124.

[24] L. Wang, G. Hu and T. Zhou, “Semantic analysis of learners’ emotional tendencies on online MOOC education,” Sustainability, vol. 10, pp. 1-19, 2018.

[25] M. Wang, C. Wang, C. Lee, S. Lin and P. Hung, “Type-2 fuzzy set construction and application for adaptive student assessment system,” in - 2014 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE), 2014, pp. 888-894.

[26] T. Luchoomun, M. Chumroo and V. Ramnarain-Seetohul, “A Knowledge Based System for Automated Assessment of Short Structured Questions,” in - 2019 IEEE Global Engineering Education Conference (EDUCON), 2019, pp. 1349-1352.

[27] M. Floryan, T. Dragon, N. Basit, S. Dragon and B. Woolf, “Who Needs Help? Automating Student Assessment Within Exploratory Learning Environments,” in Artificial Intelligence in Education. AIED 2015. Lecture Notes in Computer Science, 2015, pp. 125-134.

[28] I.A. Hameed, “A Fuzzy System to Automatically Evaluate and Improve Fariness of Multiple-Choice Questions (MCQs) Based Exams,” in Proceedings of the 8th International Conference on Computer Supported Education, Rome, Italy, 2016, pp. 476–481.

[29] D. Gursoy, O.H. Chi, L. Lu and R. Nunkoo, “Consumers acceptance of artificially intelligent (AI) device use in service delivery,” Int.J.Inf. Manage., vol. 49, pp. 157-169, 2019.

[30] T. Zhang, D. Tao, X. Qu, X. Zhang, J. Zeng, H. Zhu and H. Zhu, “Automated vehicle acceptance in china: Social influence and initial trust are key determinants,” Transportation Research Part C: Emerging Technologies, vol. 112, pp. 220-233, 2020.

[31] M. Beynon, M.M.H. Goode, L.A. Moutinho and H.R. Snee, “Modelling satisfaction with automated banking channels,”, vol. 26, no. 4, pp. 77-94, 2005. https://doi.org/10.1300/J396v26n04_05.

[32] F.D. Davis, “Perceived usefulness, perceived ease of use, and user acceptance of information technology,” MIS Quarterly, vol. 13, no. 3, pp. 319-340, 1989. http://www.jstor.org/stable/249008.

[33] W.R. King and J. He, “A meta-analysis of the technology acceptance model,” Information & Management, vol. 43, no. 6, pp. 740-755, 2006.

[34] M. Al-Emran, V. Mezhuyev and A. Kamaludin, “Towards a conceptual model for examining the impact of knowledge management factors on mobile learning acceptance,” Technology in Society, vol. 61, pp. 101247, May 2020.

[35] S.S. Al-Gahtani, “Empirical investigation of e-learning acceptance and assimilation: A structural equation model,” Applied Computing and Informatics, vol. 12, no. 1, pp. 27-50, 2016.

[36] E. Ibili, D. Resnyansky and M. Billinghurst, “Applying the technology acceptance model to understand maths teachers’ perceptions towards an augmented reality tutoring system,” Education and Information Technologies, vol. 24, no. 5, pp. 2653-2675, 2019. https://doi.org/10.1007/s10639-019-09925-z.

[37] W. M. Al-Rahmi, N. Yahaya, A. A. Aldraiweesh, M. M. Alamri, N. A. Aljarboa, U. Alturki and A. A. Aljeraiwi, “Integrating technology acceptance model with innovation diffusion theory: An empirical investigation on students’ intention to use E-learning systems,” IEEE Access, vol. 7, pp. 26797-26809, 2019.

[38] T. Teo, F. Huang and C.K.W. Hoi, “Explicating the influences that explain intention to use technology among english teachers in china,” Interactive Learning Environments, vol. 26, no. 4, pp. 460-475, 2018. https://doi.org/10.1080/10494820.2017.1341940.

[39] J.C. Sánchez-Prieto, S. Olmos-Migueláñez and F.J. García-Peñalvo, “Informal tools in formal contexts: Development of a model to assess the acceptance of mobile technologies among teachers,” Comput.Hum.Behav., vol. 55, Part A, pp. 519-528, 2 2016.

[40] H.I. Jeong and Y. Kim, “The acceptance of computer technology by teachers in early childhood education,” Interactive Learning Environments, vol. 25, no. 4, pp. 496-512, 2017. https://doi.org/10.1080/10494820.2016.1143376.

[41] M. Fishbein and I. Ajzen, Belief, attitude, intention, and behavior: an introduction to theory and research. Reading, Massachusets, AddisonWesley Pub. Co., 1975.

[42] I. Ajzen, “From Intentions to Actions: A Theory of Planned Behavior,” in Action Control. SSSP Springer Series in Social Psychology, J. Kuhl and J. Beckmann, Eds: Springer Berlin Heidelberg, 1985, pp. 11-39.

[43] V. Venkatesh and F.D. Davis, “A theoretical extension of the technology acceptance model: Four longitudinal field studies,” Management Science, vol. 46, no. 2, pp. 186-204, 2000.

[44] V. Venkatesh and H. Bala, “Technology acceptance model 3 and a research agenda on interventions,” Decision Sciences, vol. 39, no. 2, pp. 273-315, 2008.

[45] K. Zarafshani, A. Solaymani, M. D’Itri, M.M. Helms and S. Sanjabi, “Evaluating technology acceptance in agricultural education in iran: A study of vocational agriculture teachers,” Social Sciences & Humanities Open, vol. 2, no. 1, pp. 100041, 2020.

[46] M. Ismail, E. Çelebi and H. Nadiri, “How student information system influence students’ trust and satisfaction towards the university?: An empirical study in a multicultural environment,” IEEE Access, vol. 7, pp. 111778-111789, 2019.

[47] O. Gillath, T. Ai, M. Branicky, S. Keshmiri, R. Davison and R. Spaulding, “Attachment and trust in artificial intelligence,” Comput.Hum.Behav., pp. 106607, 2020.

[48] E. Glikson and A.W. Woolley, “Human trust in artificial intelligence: Review of empirical research,” ANNALS, vol. 14, no. 2, pp. 627-660, 2020. https://doi.org/10.5465/annals.2018.0057.

[49] S.K. Sharma and M. Sharma, “Examining the role of trust and quality dimensions in the actual usage of mobile banking services: An empirical investigation,” International Journal of Information Management, vol. 44, pp. 65-75, 2019.

[50] Z. Shao, L. Zhang, X. Li and Y. Guo, “Antecedents of trust and continuance intention in mobile payment platforms: The moderating effect of gender,” Electronic Commerce Research and Applications, vol. 33, pp. 100823, 2019.

[51] M. Warkentin, S. Sharma, D. Gefen, G.M. Rose and P. Pavlou, “Social identity and trust in internet-based voting adoption,” Government Information Quarterly, vol. 35, no. 2, pp. 195-209, 2018.

[52] D. Gefen, E. Karahanna and D.W. Straub, “Trust and TAM in online shopping: An integrated model,” MIS Quarterly, vol. 27, no. 1, pp. 51-90, 2003. http://www.jstor.org/stable/30036519.

[53] A.A. Almazroi, H. Shen and F. Mohammed, “The Impact of Trust on the Adoption of Cloud Computing Services by University Students,” in Recent Trends in Data Science and Soft, 2019, pp. 902-911.

[54] F. Qin, K. Li and J. Yan, “Understanding user trust in artificial intelligencebased educational systems: Evidence from china,” Br J Educ Technol, vol. 51, no. 5, pp. 1693-1710, 2020. https://doi.org/10.1111/bjet.12994.

[55] L.M. Arciniega and A. Maldonado, “Assessing the impact of dispositional resistance to change on organizational attraction,” The Spanish journal of psychology, vol. 14, no. 02, pp. 798-807, 2011.

[56] I.B. Saksvik and H. Hetland, “Exploring dispositional resistance to change,” Journal of Leadership & Organizational Studies, vol. 16, no. 2, pp. 175-183, 2009.

[57] X. Guo, Y. Sun, N. Wang, Z. Peng and Z. Yan, “The dark side of elderly acceptance of preventive mobile health services in china,” Electronic Markets, vol. 23, no. 1, pp. 49-61, 2013. http://dx.doi.org/10.1007/s12525-012-0112-4.

[58] R.T. Cenfetelli, “Inhibitors and enablers as dual factor concepts in technology usage,” Journal of the Association for Information Systems, vol. 5, no. 11, pp. 473-492, 2004.

[59] J.C. Sánchez-Prieto, F. Huang, S. Olmos-Migueláñez, F.J. García-Peñalvo and T. Teo, “Exploring the unknown: The effect of resistance to change and attachment on mobile adoption among secondary pre-service teachers,” Br J Educ Technol, vol. 50, no. 5, pp. 2433-2449, 2019. https://doi.org/10.1111/bjet.12822.

[60] S. Taylor and P. Todd, “Assessing IT usage: The role of prior experience,” MIS Quarterly, vol. 19, no. 4, pp. 561-570, 1995. http://www.jstor.org/stable/249633.

[61] S. Taylor and P.A. Todd, “Understanding information technology usage: A test of competing models,” Information Systems Research, vol. 6, no. 2, pp. 144-176, 1995. http://dx.doi.org/10.1287/isre.6.2.144.

[62] J. Cruz-Benito, R. Therón, F.J. García-Peñalvo and E. Pizarro Lucas, “Discovering usage behaviors and engagement in an educational virtual world,” Comput.Hum.Behav., vol. 47, pp. 18-25, 2015.