Improving Asynchronous Interview Interaction with Follow-up Question Generation.

Pooja Rao S B; Manish Agnihotri; Dinesh Babu Jayagopi

doi:10.9781/ijimai.2021.02.010

Authors

Pooja Rao S B International Institute of Information Technology
Manish Agnihotri International Institute of Information Technology
Dinesh Babu Jayagopi International Institute of Information Technology

DOI:

https://doi.org/10.9781/ijimai.2021.02.010

Keywords:

Asynchronous Video Interview, Language Model, Question Generation, Conversational Agent, Follow-up Question Generation

Supporting Agencies

This work was partially funded by SERB Young Scientist grant (Grant no: YSS2015001074) of Dr. Jayagopi, Karnataka government’s MINRO grant and a grant from Accenture Technology Labs. We would like to thank all the participants who contributed for data collection.

Abstract

The user experience of an asynchronous video interview system, conventionally is not reciprocal or conversational. Interview applicants expect that, like a typical face-to-face interview, they are innate and coherent. We posit that the planned adoption of limited probing through follow-up questions is an important step towards improving the interaction. We propose a follow-up question generation model (followQG) capable of generating relevant and diverse follow-up questions based on the previously asked questions, and their answers. We implement a 3D virtual interviewing system, Maya, with capability of follow-up question generation. Existing asynchronous interviewing systems are not dynamic with scripted and repetitive questions. In comparison, Maya responds with relevant follow-up questions, a largely unexplored feature of irtual interview systems. We take advantage of the implicit knowledge from deep pre-trained language models to generate rich and varied natural language follow-up questions. Empirical results suggest that followQG generates questions that humans rate as high quality, achieving 77% relevance. A comparison with strong baselines of neural network and rule-based systems show that it produces better quality questions. The corpus used for fine-tuning is made publicly available.

Downloads

Download data is not yet available.

References

[1] talview.com, “Understanding recruitment troubles and trends,” 2016. [Online]. Available: https://info.talview.com/understanding-recruitmenttroubles-trends-research-2016.

[2] J. E. Salmons, Qualitative Online Interviews: Strategies, Design, and Skills. Thousand Oaks, CA, USA: Sage Publications, Inc., 2nd ed., 2014.

[3] S. Rasipuram, S. B. P. Rao, D. B. Jayagopi, “Asynchronous video interviews vs. face-to-face interviews for communication skill measurement: A systematic study,” in Proceedings of the 18th ACM International Conference on Multimodal Interaction, ICMI ’16, New York, NY, USA, 2016, pp. 370– 377, ACM.

[4] J. Levashina, C. J. Hartwell, F. P. Morgeson, M. A. Campion, “The structured employment interview: Narrative and quantitative review of the research literature,” Personnel Psychology, vol. 67, no. 1, pp. 241–293, 2014.

[5] J. Li, M. X. Zhou, H. Yang, G. Mark, “Confiding in and listening to virtual agents: The effect of personality,” in Proceedings of the 22nd International Conference on Intelligent User Interfaces, IUI ’17, New York, NY, USA, 2017, p. 275–286, Association for Computing Machinery.

[6] C. R. Rogers, R. E. Farson, University of Chicago., Industrial Relations Center., Active listening. Chicago: Industrial Relations Center, the University of Chicago, 1957.

[7] I. Sutskever, O. Vinyals, Q. V. Le, “Sequence to sequence learning with neural networks,” in Advances in neural information processing systems, 2014, pp. 3104–3112.

[8] S. B. P. Rao, M. Agnihotri, D. B. Jayagopi, “Automatic follow-up question generation for asynchronous interviews,” in Proceedings of the 1st Workshop on Intelligent Information Processing and Natural Language Generation, ECAI, 2020.

[9] D. Erhan, Y. Bengio, A. Courville, P.-A. Manzagol, P. Vincent, S. Bengio, “Why does unsupervised pre-training help deep learning?” J. Mach. Learn. Res., vol. 11, p. 625–660, Mar. 2010.

[10] H. Larochelle, Y. Bengio, J. Louradour, P. Lamblin, “Exploring strategies for training deep neural networks,” Journal of Machine Learning Research, vol. 10, no. 1, pp. 1–40, 2009.

[11] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” CoRR, 2018.

[12] A. Radford, K. Narasimhan, T. Salimans, I. Sutskever, “Improving language understanding by generative pre-training,” 2018.

[13] J. Howard, S. Ruder, “Universal language model fine-tuning for text classification,” in ACL, 2018.

[14] M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, L. S. Zettlemoyer, “Deep contextualized word representations,” ArXiv, 2018.

[15] Z. Huang, W. L. Xu, K. Yu, “Bidirectional lstm-crf models for sequence tagging,” ArXiv, 2015.

[16] A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, S. R. Bowman, “Glue: A multi-task benchmark and analysis platform for natural language understanding,” in BlackboxNLP@EMNLP, 2018.

[17] T. Wolf, V. Sanh, J. Chaumond, C. Delangue, “Transfer-transfo: A transfer learning approach for neural network based conversational agents,” ArXiv, 2019.

[18] A. Bosselut, H. Rashkin, M. Sap, C. Malaviya, A. Çelikyil-maz, Y. Choi, “Comet: Commonsense transformers for au-tomatic knowledge graph construction,” ArXiv, 2019.

[19] S. B. P. Rao, S. Rasipuram, R. Das, D. B. Jayagopi, “Automatic assessment of communication skill in non-conventional interview settings: a comparative study,” in Proceedings of the 19th ACM International Conference on Multimodal Interaction, 2017, pp. 221–229, ACM.

[20] J. F. Nunamaker, D. C. Derrick, A. C. Elkins, J. K. Burgoon, M. W. Patton, “Embodied conversational agent-based kiosk for automated interviewing,” Journal of Management Information Systems, vol. 28, no. 1, pp. 17–48, 2011.

[21] M. E. Hoque, M. Courgeon, J.-C. Martin, B. Mutlu, R. W. Picard, “Mach: My automated conversation coach,” in Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing, 2013, pp. 697–706, ACM.

[22] K. Anderson, E. André, T. Baur, S. Bernardini, M. Chollet, E. Chryssafidou, I. Damian, C. Ennis, A. Egges, P. Gebhard, et al., “The tardis framework: intelligent virtual agents for social coaching in job interviews,” in International Conference on Advances in Computer Entertainment Technology, 2013, pp. 476–491, Springer.

[23] T. Kawahara, “Spoken dialogue system for a human-like conversational robot erica,” in International Workshop Spoken Dialogue Systems, 2018.

[24] V. Rus, A. C. Graesser, “The question generation shared task and evaluation challenge,” in The University of Memphis. National Science Foundation, 2009.

[25] V. Rus, W. Brendan, P. Piwek, M. Lintean, S. Stoyanchev, C. Moldovan, “The question generation shared task and evaluation challenge,” in The University of Memphis. National Science Foundation, 2009.

[26] I. Serban, A. García-Durán, Çaglar Gülçehre, S. Ahn, A. P. S. Chandar, A. C. Courville, Y. Bengio, “Generating factoid questions with recurrent neural networks: The 30m factoid question-answer corpus,” CoRR, 2016.

[27] X. Du, J. Shao, C. Cardie, “Learning to ask: Neural question generation for reading comprehension,” in ACL, 2017.

[28] P. Rajpurkar, J. Zhang, K. Lopyrev, P. S. Liang, “Squad: 100, 000+ questions for machine comprehension of text,” in EMNLP, 2016.

[29] Z. Wang, A. S. Lan, W. Nie, A. E. Waters, P. J. Grimaldi, R. G. Baraniuk, “Qg-net: a data-driven question generation model for educational content,” in L@S, 2018.

[30] M.-H. Su, C.-H. Wu, K.-Y. Huang, Q.-B. Hong, H.-H. Huang, “Follow-up question generation using pattern-based seq2seq with a small corpus for interview coaching,” in Proc. Interspeech 2018, 2018, pp. 1006–1010.

[31] X. Qiu, X. Huang, “Convolutional neural tensor network architecture for community-based question answering,” in IJCAI, 2015.

[32] M.-H. Su, C.-H. Wu, Y. Chang, “Follow-up question generation using neural tensor network-based domain ontology population in an interview coaching system,” in INTER-SPEECH, 2019.

[33] W. Swartout, R. Artstein, E. Forbell, S. Foutz, H. Lane, B. Lange, J. Morie, D. Noren, S. Rizzo, D. Traum, “Virtual humans for learning,” AI Magazine, vol. 34, pp. 13–30, 1 2013, doi: 10.1609/aimag.v34i4.2487.

[34] R. López-Cózar, Z. Callejas, D. Griol, J. F. Quesada, “Review of spoken dialogue systems,” Loquens, vol. 1, no. 2, p. 012, 2014.

[35] L. S. Nguyen, D. Frauendorfer, M. S. Mast, D. Gatica-Perez, “Hire me: Computational inference of hirability in employment interviews based on nonverbal behavior,” IEEE transactions on multimedia.

[36] H. Tanaka, S. Sakti, G. Neubig, T. Toda, H. Negoro, H. Iwasaka, S. Nakamura, “Automated social skills trainer,” in Proceedings of the 20th International Conference on Intelligent User Interfaces, 2015, pp. 17–27, ACM.

[37] L. Hemamou, G. Felhi, V. Vandenbussche, J.-C. Martin, C. Clavel, “Hirenet: a hierarchical attention model for the automatic analysis of asynchronous video job interviews,” 2019.

[38] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, “Language models are unsupervised multitask learners,” 2019.

[39] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, I. Polosukhin, “Attention is all you need,” in NIPS, 2017.

[40] A. Fan, M. Lewis, Y. Dauphin, “Hierarchical neural story generation,” in ACL, 2018.

[41] Y. Bengio, R. Ducharme, P. Vincent, C. Jauvin, “A neural probabilistic language model,” Journal of machine learning research, vol. 3, no. Feb, pp. 1137–1155, 2003.

[42] J. Pennington, R. Socher, C. D. Manning, “Glove: Global vectors for word representation,” in EMNLP, 2014.

[43] A. See, P. J. Liu, C. D. Manning, “Get to the point: Summarization with pointer-generator networks,” in ACL, 2017.

[44] W. Hu, B. Liu, J. Ma, D. Zhao, R. Yan., “Aspect-based question generation,” in ICLR Workshop, 2018.

[45] G. Shires, “Web speech api: Draft community group report,” July 2019. [Online]. Available: https://w3c.github.io/speech-api/, [Online; posted 17-July-2019].

[46] T. Walker, “Presenting amazon sumerian: An easy way to create vr, ar, and 3d experiences,” November 2017. [Online]. Available: https://aws.amazon.com/blogs/aws/launch-presenting-amazon-sumerian/, [Online; posted 27-November-2017].