Speaker: Jaromir Savelka, ISP Alum, Research Associate, CMU School of Computer Science

TITLE: Navigating the New Era of Programming Education: Several Notes on LLMs' Capabilities to Handle Assignments, Support Students, and Help Educators Develop Learning Materials

ABSTRACT: The emergence of ChatGPT resulted in heated debates of its potential uses (e.g., exercise generation, code explanation) as well as misuses in programming classes (e.g., cheating). Perhaps, the most immediate question that likely occupies the minds of many instructors is how to assess learners' skills and knowledge in the presence of ubiquitous tools that could be easily utilized to pass the assessments. Equally important are the considerations around how to properly leverage LLM-powered technologies in educational settings and how to mitigate the risks they pose. In this talk, I will present results from several studies focused on the capabilities of LLMs (OpenAI's GPT family) with regard to answering the following questions: (i) To what degree can an LLM such as GPT-4 generate correct solutions to typical assessments in an introductory or intermediate course in Python in higher education? (ii) If provided with an LLM-powered assistance how do learners engage with a tool, what types of help they request, and how useful do they find the interaction. (iii) To what degree can an LLM support course design or authoring of learning content (e.g., assessments)? We provide evidence that programming instructors need to prepare for a world in which there is an easy-to-use widely accessible technology that can be utilized by learners to collect passing scores, with no effort whatsoever, on what today counts as viable programming knowledge and skills assessments. Lessons from this research can also be leveraged by programming educators and institutions who plan to augment their teaching with emerging LLM-powered tools as well as those who plan to make course design and authoring more efficient and effective.

BIO: I am a researcher associate at the School of Computer Science, Carnegie Mellon University (CMU) and a member of the Technology for Effective and Efficient Learning (TEEL) Lab at CMU. Before joining CMU in September 2020, I worked as a data scientist at international law firm Reed Smith (2017-2020). I obtained an undergraduate computer science and graduate law degrees from the Masaryk University in Brno, Czechia, as well as Ph.D. in Intelligent Systems mentored by prof. Kevin D. Ashley from the University of Pittsburgh. I have regularly published in and reviewed for Q1 journals, and presented at top-tier international conferences. I am a member of the editorial board of the Artificial Intelligence and Law journal. I have participated in NSF-funded research projects focused on using AI to increase fairness by improving access to justice (PI Kevin D. Ashley) and investigating the use of real-time data for augmenting teaching practice in project-based learning in STEM (PI Majd Sakr). My dissertation work was funded by the National Institute of Justice with its highly competitive Graduate Research Fellowship in Science, Technology, Engineering and Mathematics.