Lars Schöbitz September 29, 2023
- Proposalnumber:
- Project type: Focal Point Project
- Focus topic: AI in Teaching and Learning
- Start date: 2024-01-01
- End date: 2025-06-30
- Extended:
- Type of proposal
- English: Research Beyond the Lab & Into the Future: Dancing with Data & AI
- German: Forschung ausserhalb des Labors & in die Zukunft: Tanz mit Daten & KI
- Institute:
- Department:
- Postal address
- Main applicant: Lars Schöbitz
- Project manager: Lars Schöbitz
- Project partner:
1.1. Describe the project in 1-2 sentences (max. 400 characters; 57 to 90 words; will be published in the project database)
This Innovedum project targets the responsible incorporation of Large Language Models (LLMs) in data science education. It tackles key challenges: ethical AI usage, aiding novices in programming, optimising retrieval practices for enhanced learning, and utilising LLMs in coding examinations. This project seeks to cultivate a community of practice focused on AI’s role in data science education.
Das Innovedum-Projekt fokussiert auf die Integration von LLMs in der Data Science Ausbildung. Es adressiert die ethische KI-Nutzung, Hilfe für Programmieranfänger, verbesserte Lernmethoden und LLM-Einsatz in Prüfungen. Ein übergeordnetes Ziel ist auch, eine Community of Practice zu etablieren, die KI’s Rolle in der Datenwissenschaft betont.
The Global Health Engineering group offers a course titled “Research Beyond the Lab (rbtl)”, which imparts training on out-of-lab research projects, emphasising open science principles and effective research data management.
A critical component is teaching data science with the R statistical programming language. Given the advent of Large Language Models (LLMs) capable of auto-generating accurate programming solutions, there’s a need to integrate these tools responsibly into the curriculum. This Innovedum project delineates four main challenges:
- Ensuring responsible AI use
- Guiding novices to find answers to programming tasks
- Improving retrieval practice to enhance student learning
- Leveraging LLMs for coding exam settings
An overarching outcome of the project is to foster collaboration with other ETH initiatives and beyond, positioning the project not merely as an educational overhaul but also as a catalyst for building a robust community of practice centred on AI in data science education.
Die Gruppe Global Health Engineering bietet einen Kurs mit dem Titel “Research Beyond the Lab (rbtl)” an, der Schulungen zu Forschungsprojekten außerhalb des Labors vermittelt und dabei die Grundsätze von Open Science und des effektiven Forschungsdatenmanagements (Research Data Management) betont.
Ein wichtiger Bestandteil ist der Unterricht in Datenwissenschaft mit der statistischen Programmiersprache R. Angesichts des Aufkommens von Large Language Models (LLMs), die in der Lage sind, automatisch genaue Programmierlösungen zu generieren, besteht die Notwendigkeit, diese Werkzeuge verantwortungsvoll in den Lehrplan zu integrieren. In diesem Innovedum-Projekt werden vier wesentliche Herausforderungen beschrieben:
- Sicherstellung einer verantwortungsvollen Nutzung von KI
- Anleitung von Anfängern bei der Suche nach Antworten auf Programmieraufgaben
- Verbesserung der Abfragepraxis, um das Lernen der Studenten zu verbessern
- Nutzung von LLMs für Prüfungen
Ein übergreifendes Ergebnis des Projekts ist die Förderung der Zusammenarbeit mit anderen ETH-Initiativen und darüber hinaus, wobei das Projekt nicht nur als Überholung der Ausbildung, sondern auch als Katalysator für den Aufbau einer robusten Community of Practice, die sich auf KI in der Data-Science-Ausbildung konzentriert, positioniert wird.
2.1. Student involvement: Describe whether and how students were involved in the preparation and review of this project application. How will students be involved in project implementation?
The project implementation will be in coordination and collaboration with students that participate in our course (rbtl: Research beyond the lab) in spring semester 2024. They will be directly involved in the project implementation and will support the evaluation process. Results will then be used for the course in spring semester 2025.
2.2. Who advised you on your project and/or who gave you feedback on it (e.g. LET / Educational Developer(s) / department(s) / other faculty / other)?
An initial project idea was shared with Daniela Eawason as a response to the announcement of the new focal point topic. We shared our idea with Katrin Bentel (LET) who we have been continuously in touch with about open science ideas and computational reproducibility for about 1.5 years. We then shared our project idea with Kathrin Bentel and Julia Kehl (both LET) who we met personally to discuss. Their advice was integrated into a draft proposal, which was shared once more.
The draft proposal was also shared with employees of the Global Health Engineering group. Elizabeth Tilley (head of group) and Mian Zhong (Data Scientist and expert in Natural Language Processing) provided detailed feedback on the proposal.
We are aware of several other initiatives around the use of AI in teaching and learning at ETH. In addition to communicating the project ideas with the Innovedum and LET teams, we will actively reach out to other professionals at ETH to contribute to building a community of practice, including Manuel Sudau, Educational Developer with LET, Dennis Kochmann at D-MAVT, Christian Franck at D-ITET, Meike Akveld & Andreas Steiger at D-MATH and their product STACK, and lastly the applicants for the most recently funded Innovedum projects with a focus on exploring AI in academic writing (Lucio Isa & Melanie Paschke).
3.1. Starting position: Describe the current situation. What didactic and disciplinary challenges does your project address? (max 324/250 words). Info: Show the relevance of and the need leading to the project. List (if applicable) preliminary work to remedy the situation.
At the Global Health Engineering group, we teach a class (rbtl: Research Beyond the Lab) on conducting a research project out of the lab and applying open science principles and effective research data management. Amongst other data science tools, students learn to use the R statistical programming language, for which we assume no prior knowledge.
With the rise of Large Language Models (LLMs) and associated tools, such classes risk not providing added value to students if teachers miss out on including LLMs in their teaching. Students can now auto-generate solutions to programming assessments with high accuracy (Denny et al. 2023). A structured, defined, open approach is needed to encourage students to use such tools responsibly while teachers can enhance the learning experience and increase their efficiency and productivity.
This project addresses the following didactic and disciplinary challenges:
Challenge 1: Ensuring students use AI responsibly and effectively while developing their critical thinking, problem-solving and programming skills.
We will teach students how to use AI tools and discuss the ethical implications of using LLMs in programming, including the potential for over-reliance on AI-generated solutions and the importance of developing problem-solving skills.
Challenge 2: Helping students navigate the overwhelming amount of information available online to receive solutions to their coding problems.
LLMs have shown great potential to provide coding support from plain language descriptions. Once students know how to phrase an appropriate prompt, they will receive coherent functional examples.
Challenge 3: Enhancing retrieval practice, improving student learning, and enriching the learning experience in a time-constrained environment.
Preparing up-to-date formative assessments and authentic examples requires much time and thought. LLMs show great potential to automate parts of this process. Additionally, AI can serve as a creative engine to prepare analogies that support students in class, further enriching the learning experience (Mollick and Mollick 2023).
Challenge 4: Applying the knowledge and skills gained throughout the course to analyse data, draw conclusions, and demonstrate programming proficiency in a time-limited exam setting that allows for the use of LLMs.
3.2. Project goals: List the concrete project goals. (232/250 words). Info: 3 to 5 clear measurable objectives. Please fill the fields in German and English. The information will be published in Innovedum Public.
Each identified challenge is addressed by an associated work package, goal and measurable objective as presented Table 1 in the Appendix and in this public Google Sheet: https://docs.google.com/spreadsheets/d/1srD0UPBMVknayk0ni6iOma07yHvpzAOYg3jS2KEYVOI/edit#gid=0
We identified two to three specific activities per objectiveas presented Table 2 in the Appendix and in this public Google Sheet: https://docs.google.com/spreadsheets/d/1ATIyMmpVnVfX8Z17WUYYu1wjMx6rNA4snTOpVj4V5hM/edit#gid=0
Goal: Integrate Large Language Models (LLMs) into lectures and require it as a mandatory tool in class.
Objective 1: Within 12 months after the start of the project, a usage policy for LLMs and an associated teaching module is being re-used by other classes at ETH or beyond.
Goal: Use LLMs to provide code examples and support novices to find answers to programming tasks.
Objective 2: Ensure that 95% of AI-generated code examples and solutions are accurate, relevant, and helpful for novice programmers.
Goal: Use LLMs to create low-stakes tests for formative assessments (faded examples/scaffolding, Parsons problems, multiple choice quizzes).
Objective 3: Reduce the required time for generating formative assessments for R programming concepts by 90%.
Goal: Develop a technical programming exam that allows students to use LLMs.
Objective 4: Achieve a grade distribution in the final exam where at least 80% of the students score between 5 and 6, with no more than 10% scoring the maximum grade of 6.
Jede identifizierte Herausforderung wird durch ein zugehöriges Arbeitspaket, ein Ziel und ein messbares Objective angegangen, wie in Tabelle 1 im Anhang und in diesem öffentlichen Google Sheet dargestellt: https://docs.google.com/spreadsheets/d/1srD0UPBMVknayk0ni6iOma07yHvpzAOYg3jS2KEYVOI/edit#gid=0
Wir haben zwei bis drei spezifische Aktivitäten pro Objective festgelegt, wie in Tabelle 2 im Anhang und in diesem öffentlichen Google Sheet dargestellt: https://docs.google.com/spreadsheets/d/1ATIyMmpVnVfX8Z17WUYYu1wjMx6rNA4snTOpVj4V5hM/edit#gid=0
WP1 - Verantwortungsvolle Nutzung von KI
Ziel: Integration von Large Language Models (LLMs) in Vorlesungen und deren obligatorische Verwendung im Unterricht.
Objective 1: Innerhalb von 12 Monaten nach Projektstart wird eine Nutzungsrichtlinie für LLMs und ein dazugehöriges Lehrmodul von anderen Klassen an der ETH oder darüber hinaus wiederverwendet.
WP2 - KI als Tutor
Ziel: Verwendung von LLMs zur Bereitstellung von Codebeispielen und zur Unterstützung von Anfängern bei der Lösung von Programmieraufgaben.
Objective 2: Sicherstellen, dass 95% der KI-generierten Code-Beispiele und Lösungen korrekt, relevant und hilfreich für Programmieranfänger sind.
WP3 - Verbessern des Lernens
Ziel: Verwendung von LLMs zur Erstellung von Tests mit geringem Schwierigkeitsgrad für formative Beurteilungen (eingeblendete Beispiele/Scaffolding, Parsons-Probleme, Multiple-Choice-Tests).
Objective 3: Die benötigte Zeit für die Erstellung von formativen Beurteilungen für R-Programmierkonzepte um 90% reduzieren.
WP4 - Bewertung von Studenten
Ziel: Entwicklung einer technischen Programmierprüfung, die es den Studenten ermöglicht, LLMs zu verwenden.
Objective 4: Erreichen einer Notenverteilung in der Abschlussprüfung, bei der mindestens 80% der Studierenden eine Note zwischen 5 und 6 erreichen, wobei nicht mehr als 10% die Höchstnote 6 erreichen.
3.3. Project progression: Describe the progression of the project (288/300 words). Info box: Describe the different phases of the project.
We present a rough timeline for project activities in Table 2 of the Appendix. This timeline translates into five project phases.
We implement this project in a course we teach in Spring Semester 2024 (rbtl-fs24). Therefore, it is essential to include information about the usage of AI in the course catalogue now. Students need to be aware of the usage of AI.
The project’s first phase will identify a suitable candidate (Scientific Assistant) to support the project during the development and implementation phases. Further, we will make the exact choice of AI tools during this phase. Our current thinking is to ask students to use free versions of Perplexity (perplexity.ai), an AI tool that allows for free GPT-4 LLM usage.
While the syllabus and overall objectives for each module of the class will be defined at the before the start of the course in February 2024, we will prepare the class material flexibly, allowing for the inclusion of learnings from the use of AI.
After the course is completed, the project team will analyse and compile all materials that were generated as part of the work packages. We will analyse student evaluation surveys and present results to the scientific community.
As a final phase, the learnings from the project will be integrated into material for the next class in Spring Term 2025 (rbtl-fs25) and other courses that teach programming concepts.
TODO: Add in online form.
3.5. Teaching/Learning concept: Explain the didactic principles (e.g. experiences, theories and methods) which underpin your project. (279/300 words). Explain the didactic methodology that underpins the project. This can be your own experience (of that of other lectures) of didactic theories and methods from the literature.
The idea of using AI to help students with coding problems in R is based on the premise that novices in programming struggle to find help online, and AI tools can provide better code examples if students are taught how to use the relevant prompts. Several didactic theories and methods underpin this idea:
-
Co-creation and collaboration: Co-creation in AI-assisted learning involves students working with AI tools to solve problems and learn new concepts. This approach has been explored in studies such as the one on AI-generated programming code in higher education (Jonsson and Tholander 2022). The study found that AI tools can facilitate understanding and learning while improving the creative processes by promoting reflection and through the exploration of alternative solutions. These findings align with the didactic theory of constructivism, which emphasises that learners actively construct their understanding and knowledge through experiences and interactions.
-
Personalised learning: AI tools can offer personalised learning experiences by adapting to individual students’ needs, preferences, and learning styles. This approach is supported by the idea that learners have different ways of processing information and that tailored instruction can lead to better learning outcomes (Kaiss, Mansouri, and Poirier 2023). The approach is connected to self-determination theory, which defines three drivers of intrinsic motivation (competence, autonomy, relatedness), all positively impacted by personalised learning.
-
Connectivism: Within the classroom, we will engage and interact with the AI tools as a class, in small groups, and individually. The newest feature of Perplexity allows people to have shared Collections of prompt threads. It enables people to craft prompts together, share them with others, and make them public. This practice is supported by the theory of connectivism, which emphasises the importance of networks, social interactions, and the use of technology in the learning process.
- done in online form
3.7. Effects of the project: Describe the innovative aspects of your project and describe its expected effects and added value for teaching and learning at ETH for all of the following: (167/250 words). These aspects should be different from the normal advancement of teaching. Further benefits and effects can also be listed here.
Beyond the measurable objectives, the over-arching effect of this project will be to shift the culture around AI from shame and secrecy to one that promotes and embraces the practical and enjoyable aspects of this emerging technology. Through this project, we will demonstrate to educators (who may still be sceptical) that AI has a place in teaching and that it allows students and teachers to spend more time learning, discussing, and engaging and less time searching and running failed code. We will address important elements such as, ethics, privacy, liability, or social responsibility in AI.
- being aware of key ethical concerns related to AI (human bias, misinformation, privacy, security, etc.)
- being aware of the limitations and of using AI
- being aware of the responsibility for using AI and critically and evaluating all its output
- learning how to use AI for learning a new programming language
- learning how to use AI for debugging code
-
- being aware of key ethical concerns related to AI (human bias, misinformation, privacy, security, etc.)
-
learning how to use AI to deliver more engaging classes
-
demonstrating the time-saving benefits related to assignment and exam preparation
- being aware of key ethical concerns related to AI (human bias, misinformation, privacy, security, etc.)
- learning how to use AI
- embracing a culture of AI use
- belonging to a community of practice for use of AI in teaching and learning
Über die messbaren Ziele hinaus wird der übergreifende Effekt dieses Projekts darin bestehen, die Kultur rund um KI von Scham und Geheimniskrämerei zu einer Kultur zu verändern, die die praktischen und angenehmen Aspekte dieser aufstrebenden Technologie fördert und begrüßt. Durch dieses Projekt werden wir Pädagogen (die vielleicht immer noch skeptisch sind) zeigen, dass KI einen Platz im Unterricht hat und dass sie es Student:innen und Lehrpersonen ermöglicht, mehr Zeit mit dem Lernen, Diskutieren und Engagieren zu verbringen und weniger Zeit mit dem Suchen und Ausführen von fehlgeschlagenem Code. Wir werden wichtige Aspekte wie Ethik, Datenschutz, Haftung und soziale Verantwortung im Zusammenhang mit KI ansprechen.
- sind sich der wichtigsten ethischen Bedenken im Zusammenhang mit KI bewusst (menschliche Voreingenommenheit, Fehlinformationen, Datenschutz, Sicherheit usw.)
- sind sich der Grenzen und des Einsatzes von KI bewusst
- sind sich der Verantwortung für den Einsatz von KI bewusst und bewerten alle ihre Ergebnisse kritisch
- lernen, wie man KI zum Erlernen einer neuen Programmiersprache einsetzt
- lernen, wie man KI zum Debuggen von Code einsetzt
- Kenntnis der wichtigsten ethischen Bedenken im Zusammenhang mit KI (menschliche Voreingenommenheit, Fehlinformationen, Datenschutz, Sicherheit usw.)
- lernen, wie man KI einsetzt, um den Unterricht interessanter zu gestalten
- Demonstration der zeitsparenden Vorteile im Zusammenhang mit der Vorbereitung von Aufgaben und Prüfungen
- Kenntnis der wichtigsten ethischen Bedenken im Zusammenhang mit KI (menschliche Voreingenommenheit, Fehlinformationen, Datenschutz, Sicherheit usw.)
- lernen, wie man KI einsetzt
- eine Kultur der offenen und verantwortungsvollen KI-Nutzung zu schaffen
- Zugehörigkeit zu einer Praxisgemeinschaft für den Einsatz von KI in der Lehre und im Lernen
3.8. Evaluation strategy: Describe the evaluation strategy you will use to check achievement of project goals (see 3.2) and effects on teaching (see 3.7). (250 words)
- What approaches will you use?
- Are you planning measures for identifying interim results? If so, how will these results flow back into the project?
To verify the achievement of project goals and related objectives, we will work with Nore Dietmann from LET to support us in developing a student evaluation survey that explicitly targets ‘WP1 Responsible AI use’, and ‘WP2 AI as a Tutor’. We will implement one survey after teaching the 3-hour AI module, another survey at the end of the course, and a follow-up survey six months after course completion. In combination, these three surveys will help identify student satisfaction, improve course material, and ensure that the expected innovative effects are taking place.
We will receive interim results throughout the entire implementation phase, as students are explicitly asked to share their prompts and critically reflect on their usage. This process will allow results to flow back into the project as it is implemented. It is an interactive exercise and dialogue with participating students as they are using the AI tools.
As presented in Section 3.2, the defined objectives have a measurable outcome. We will use them to evaluate if the objectives were achieved. The evaluation results will be used to design activities for the continuation and iteration of the project.
Add in online form.
4.1.Sustainability: How will project results be maintained and utilised after the funding period? What project results can prospectively be utilised within ETH? Do you need additional funding for this? (265/300 words)
During the funding period, the project will be implemented twice within the same class (rbtl-fs24 and rbtl-fs25), and evaluated in collaboration with LET. For this, we will work with Nora Dietmann, who will support us in developing a student evaluation survey that explicitly targets the project’s objectives. The evaluation results will feed back into re-development course material, implemented and evaluated once again with our rbtl-fs-25 cohort of students.
We will actively communicate project progress and results to the ETH community and beyond using several media types and fora (see Section 4.2 Project communication). The project material is published as Open Educational Material with permissive licences and in an adaptable and reusable format (see Section 4.2 Project Communication & Section 5.2 Accessibility and e-Accessibility).
This project will last 18 months, and we aim to apply for another Innovedum focal point project on March 1st 2025, which will ensure the mid-term continuation of the project for another 18 months from July 2025 to December 2025. The objectives of this second round of funding will primarily focus on rolling out developed modules with identified partners at ETH (e.g., the Computational Competencies initiative, Data Stewardship Network).
Beyond these three years, the utilisation of project results and long-term maintenance will be ensured by the fact that the project lead, Lars Schöbitz, holds a permanent contract at the Global Health Engineering research group at D-MAVT. He is not a researcher but holds the position of an administrative technical assistant (Open Science Specialist & Data Steward). The research group is committed to the long-term continuation and scale-up of courses related to this project.
4.2. Project communication: How do you plan to publicise and document the progress of the project? What form will the final report for the Innovedum project database take? How will you disseminate project results? (308/300 words). Info: Interested ETH members should be able to gain insight into the project and the results. How do you ensure this?
This proposal, progress updates and project outputs will be published on GitHub with open and permissive licences as they are developed.
Each course we teach has a public website with access to all course materials. In addition, we will develop a project-dedicated website to highlight unique aspects of the project, and will share a final report in an accessible and interactive format. We will also share this report in the Innovedum public project database.
The project team is part of several networks and committees related to open science and data stewardship. We will use these established networks to take the opportunity to present the project at a minimum of two hosted events (ETH Library Data Stewardship Network, Swiss Reproducibility Network Working Groups).
We will share our first experiences at and contribute to the Innovation Learning & Teaching Fair 2024. “Refresh Teaching” is another platform and event for which we will prepare material to reach an even broader community.
As part of our openwashdata project that is supported by the Open Research Data Program of the ETH Board, we send a monthly newsletter to 110 (5% growth/week) people who are interested in applying open data practices in the greater water, sanitation, and hygiene (WASH) sector. Still, we will also highlight this project as it aligns with our overall mission to empower learners of all levels to engage with open data practices. This work will undoubtedly resonate with many members of the community.
We maintain an active LinkedIn account where we frequently highlight the work of our students and our teaching; we will dedicate a series of posts to this project, associated courses, and the links to the Innovedum programme.
Recently, we started to host a blog on our GHE website with monthly posts. We will dedicate several blog posts to this work, especially after course completion, when we have quantitative data and personal testimonials to report.
No
5.2 Accessibility and e-Accessibility: How do you ensure the flexible usability and accessibility of the learning media and materials created in the project? (max ca. 265/300 words)
As open science advocates, we will publish all learning media and materials as open educational resources. These resources play a vital role in ensuring flexible usability and accessibility, allowing for the customization and adaptation of content to suit various teaching styles and students’ preferences. For instance, our public course website is developed with Quarto open-source scientific and technical publishing system. The underlying code is hosted as a GitHub repository using permissive licences for all material (e.g. CC-BY-40, MIT License), offering educators a platform to access, modify, share and contribute to educational materials.
We will address all essential e-Accessibility features in our learning materials. (1) Flexible Output: our course website and hosted material will be available in HTML file format, allowing for the customization of materials in a variety of ways, responsive design, and custom styles. (2) Flexible input: ensured by our course website, all content will be reachable using the computer mouse and keyboard. (3) Information & Semantics: Our website will not contain scripted interactive components but will be provided in native HTML and hosted as a static website, ensuring the correct interpretation of content by screen readers. (4) Images and Multimedia: Every used image, prepared diagram, or visualisation will be prepared using the alternative text attribute in HTML. We will use decision-making support, such as the Alt-text-Decision-Tree of the Web Accessibility Initiative (WAI), to assess whether the provided text is meaningful and purposeful. We will contact experts at ETH (Anton Bolfing) to check the correctness of alternative texts. (5) Colours and Contrast: We will use colorblind-friendly colour palettes in our design and tools to ensure high contrast (e.g. colourcontrast.cc).
60’000.
| wp | challenge | goal | objective |
|---|---|---|---|
| WP1 - Responsible AI use | Ensuring students use AI responsibly and effectively while developing their critical thinking, problem-solving and programming skills. | Integrate Large Language Models (LLMs) into lectures and require it as a mandatory tool in class. | Within 12 months after the start of the project, a usage policy for LLMs and an associated teaching module is being re-used by other classes at ETH or beyond. |
| WP2 - AI as a Tutor | Helping students navigate the overwhelming amount of information available online to receive solutions to their coding problems. | Use LLMs to provide code examples and support novices to find answers to programming tasks. | Ensure that 95% of AI-generated code examples and solutions are accurate, relevant, and helpful for novice programmers. |
| WP3 - Enhance Learning | Enhancing retrieval practice, improving student learning, and enriching the learning experience in a time-constrained environment. | Use LLMs to create low-stakes tests for formative assessments (faded examples/scaffolding, Parsons problems, multiple choice quizzes). | Reduce the required time for generating formative assessments for R programming concepts by 90%. |
| WP4 - Student Assesment | Applying the knowledge and skills gained throughout the course to analyse data, draw conclusions, and demonstrate programming proficiency in a time-limited exam setting that allows for the use of LLMs. | Develop a technical programming exam that allows students to use LLMs. | Achieve a grade distribution in the final exam where at least 80% of the students score between 5 and 6, with no more than 10% scoring the maximum grade of 6. |
Table 1: Project work packages along four identified didactic and discplinarydisciplinary challenges with associated project goals and objectives.
| id | objective | act_id | activity | Q1 | Q2 | Q3 | Q4 | Q5 | Q6 |
|---|---|---|---|---|---|---|---|---|---|
| 1 | Within 12 months after the start of the project, a usage policy for LLMs and an associated teaching module is being re-used by other classes at ETH or beyond. | 1.1 | Develop a 3-hour module introducing LLMs for software development, ethical considerations and prompting AI for different use cases. | X | X | ||||
| 1 | Within 12 months after the start of the project, a usage policy for LLMs and an associated teaching module is being re-used by other classes at ETH or beyond. | 1.2 | Develop a usage policy that makes students aware of the limitations of LLMs and equires students to share links to their prompts and critically reflect on the usage of AI for each assessment. | X | X | ||||
| 1 | Within 12 months after the start of the project, a usage policy for LLMs and an associated teaching module is being re-used by other classes at ETH or beyond. | 1.3 | Establish a community of practice for using AI in teaching and learning using the chat server based on the matrix protocol offered by ETH. | X | X | X | X | X | X |
| 2 | Ensure that 95% of AI-generated code examples and solutions are accurate, relevant, and helpful for novice programmers. | 2.1 | Create a collection of 60 problem statements (20 for each level of complexity: low, medium, high) for R programming tasks, along with their respective AI prompts. | X | X | X | |||
| 2 | Ensure that 95% of AI-generated code examples and solutions are accurate, relevant, and helpful for novice programmers. | 2.2 | Identify an appropriate format to publish problem statements and prompts under an open and permissive licence as a prompt library. | X | X | ||||
| 2 | Ensure that 95% of AI-generated code examples and solutions are accurate, relevant, and helpful for novice programmers. | 2.3 | Conduct a student survey prior to the class to identify students’ prior knowledge in the R programming language and programming in general. | X | |||||
| 3 | Reduce the required time for generating formative assessments for R programming concepts by 90%. | 3.1 | For each class module that teaches R programming concepts, use LLMs to design a formative assessments with a minimum of 10 in total. | X | X | X | |||
| 3 | Reduce the required time for generating formative assessments for R programming concepts by 90%. | 3.2 | Write five formative assessments without the support of LLMs and compare the time requirements to those prepared with the support of AI. | X | X | ||||
| 3 | Reduce the required time for generating formative assessments for R programming concepts by 90%. | 3.3 | Write tutorials using the learnR R package for developed formative assessments (minimum 10 in total) and publish learnR tutorials as open-source code through GitHub using open and permissive licenses. | X | X | X | X | ||
| 4 | Achieve a grade distribution in the final exam where at least 80% of the students score between 5 and 6, with no more than 10% scoring the maximum grade of 6. | 4.1 | Create a set of 10-15 exam questions that cover a range of R programming concepts, techniques, and LLM applications, ensuring that at least 80% of the course material is represented in the questions. | X | X | ||||
| 4 | Achieve a grade distribution in the final exam where at least 80% of the students score between 5 and 6, with no more than 10% scoring the maximum grade of 6. | 4.2 | Develop a grading rubric with clear criteria for assessing students’ programming proficiency, problem-solving skills, and LLM usage. | X | X |
Table 2: Project objectives with defined activities and timeline for implementation.
Denny, Paul, James Prather, Brett A. Becker, James Finnie-Ansley, Arto Hellas, Juho Leinonen, Andrew Luxton-Reilly, Brent N. Reeves, Eddie Antonio Santos, and Sami Sarsa. 2023. “Computing Education in the Era of Generative AI.” arXiv. https://doi.org/10.48550/arXiv.2306.02608.
Jonsson, Martin, and Jakob Tholander. 2022. “Cracking the Code: Co-coding with AI in Creative Programming Education.” In Creativity and Cognition, 5–14. Venice Italy: ACM. https://doi.org/10.1145/3527927.3532801.
Kaiss, Wijdane, Khalifa Mansouri, and Franck Poirier. 2023. “Effectiveness of an Adaptive Learning Chatbot on Students’ Learning Outcomes Based on Learning Styles.” International Journal of Emerging Technologies in Learning (iJET) 18 (13): 250–61. https://doi.org/10.3991/ijet.v18i13.39329.
Mollick, Ethan R., and Lilach Mollick. 2023. “Assigning AI: Seven Approaches for Students, with Prompts.” {{SSRN Scholarly Paper}}. Rochester, NY. https://doi.org/10.2139/ssrn.4475995.