Artificial Intelligence and the Future is Now: Rethinking Learning and Assessing Learned by LLMs, says ASU Vice-President Sean Dudley

How to ensure the information LLMs provide is not biased and that models consider knowledge and perspectives from under-represented groups is a challenge. Information like this is not available in a lot of thetext that LLMs are trained on. Sean Dudley, ASU’s associate vice-president for research technology, based in Tempe, says that RAG allows ASU’s LLM platform to provide users with the sources of its answers. This doesn’t remove the problem of bias, but he hopes that it will at least provide transparency and a chance for students to critically consider where the information has come from. “Part of our mission is asking who’s been left out,” Dudley says.

Many students now use AI chatbots to help with their assignments. Educators need to study how to include these tools in teaching and learning — and minimize pitfalls.

In a February preprint, researchers described how, in a benchmark set of relatively simple mathematical problems usually answered by students aged 12–17, ChatGPT answered about half of the questions correctly2. If the problems were more complex — requiring ChatGPT to do four or more additions or subtractions in the same calculation — it was particularly likely to fail.

One year ago, teachers were frightened by the launch of a new educational service known as CHATGPT. The artificial-intelligence (ai) chatbot is able to respond to assignment questions with well-researched essays, forcing educators to rethink their evaluation methods. Some countries brought back pen-and-paper exams. Some schools are changing the model of teaching, where students do their assignments at school after learning about a subject at home.

Tawil, who has worked in education at UNESCO for more than two decades, says that understanding AI’s limitations is crucial. At the same time, LLMs are now so bound up in human endeavours that he says it is essential to rethink how to teach and assess learning. What makes us human, what makes intelligence unique, is being redefined.

He likens the attention that they’re attracting to that previously lavished on massively online open courses and educational uses of the 3D virtual worlds known as the metaverse. Neither have the transformative power that some once predicted, but both have their uses. “In a sense, this is going to be the same. It is not bad. It doesn’t look like it’s perfect. It isn’t everything. It’s a new thing,” he says.

The academy’s chief learning officer says the process is a productive struggle. She acknowledges that there is a line between aiding learning and one that makes students give up as Khanmigo is still in a pilot phase. “The trick is to figure out where that line is,” she says.

Intracting the classroom: how LLMs could transform education, explains Chegg, TAL Education Group, and Merlyn Mind

RAG is also being used by ASU, which is one of the most progressive universities for LLM adoption, says Claire Zau, vice-president of GSV Ventures, an investor in educational-technology companies in New York City. After an initial narrow release for testing, ASU launched a toolbox in October that enables its faculty members to experiment with LLMs in education through a web interface. This includes access to six LLMs including GPT- 3.5 and GPT-4 as well as RAG capabilities.

AI company Merlyn Mind in New York City is using RAG in its open-source Corpus-qa LLM, which is aimed at education. Like ChatGPT, Merlyn Mind’s LLM is initially trained on a big body of text not related to education specifically — this gives it its conversational ability.

While it does not depend on what it has learnt in its training, when the LLM answers a query, it does. The company says it refers to a specific corpus of information which reduces the risk of brain damage and other errors. Nitta says that Merlyn Mind works on fine- tuning its graduates to resist hallucination if they don’t have a high-quality response.

Some tutoring companies are offering LLMs as assistants or experimenting with them. The education technology firm Chegg in Santa Clara, California, launched an assistant based on GPT-4 in April. TAL Education Group, a Chinese tutoring company based in Beijing, has created an LLM called MathGPM to better answer math-specific questions. MathGPT also aims to help students by explaining how to solve problems.

Source: ChatGPT has entered the classroom: how LLMs could transform education

The PyrEval Project: Tailoring an Artificial Intelligence Bot to Tell People Where to Put It & When to Give Feedback

Lynch stresses the importance that any chatbot used in education should be careful about its tone and accuracy, and that it should not insult or make students feel lost. “Emotion is key to learning. You can legitimately destroy somebody’s interest in learning by helping them in a bad way,” Lynch says.

But whether Khanmigo can truly revolutionize education is still unclear. They do not want to check the facts when training to include only the next most likely word in a sentence. Sometimes they get things wrong. To improve its accuracy, the prompt that Khanmigo sends to GPT-4 now includes the right answers for guidance, says DiCerbo. It still makes mistakes, however, and Khan Academy asks users to let the organization know when it does.

Khanmigo was first introduced in March, and more than 28,000 US teachers and 11–18-year-old students are piloting the AI assistant this school year, according to Khan Academy. More than 30 school districts are included in the users. School districts pay $60 a year for access to LLMs, while individuals pay US$10 a year to cover the costs. OpenAI has promised not to use Khanmigo data for training.

Khanmigo works differently from ChatGPT. It is a pop-up on the screen of a student. Students can talk about the problem they are working on. The bot is instructed not to give away any answers and instead to ask lots of questions when the tool adds a prompt.

PyrEval’s scores also help students to reflect on their work: if the AI doesn’t detect a theme that the student thought they had included, it could indicate that the idea needs to be explained more clearly or that they made small conceptual or grammatical errors, she says. The team is benchmarking the results of the task that has been done by CHATGPT and other LLMs.

With help from educational psychologist Sadhana Puntambekar at the University of Wisconsin–Madison, PyrEval has scored physics essays5 written during science classes by around 2,000 middle-school students a year for the past three years. The PyrEval allows teachers to quickly check whether the essays contain key themes and to give feedback in class, which would otherwise not be possible thanks to conventional grades.

Magicschool and Eduaide, both based on OpenAI’s LLM technology, market commercial assistants that help teachers plan lesson activities and assess students work. Academics have produced other tools, such as PyrEval4, created by computer scientist Rebecca Passonneau at Pennsylvania State University, to read essays and extract the key ideas.

“Are there positive uses?” asks Collin Lynch, a computer scientist at North Carolina State University in Raleigh who specializes in educational systems. “Absolutely. Are there risks? There are huge risks and concerns. But I think there are ways to mitigate those.”

Students might not work with LLMs in the future if they realized their data is being used to train models.

Using LLMs to read and summarize large stretches of text could save students and teachers time and help them to instead focus on discussion and learning. It might be possible to use LLMs to build a personalized educational experience, because of the ability of the chatGPT to discuss almost any topic. They could be considered as possible thought partners that can cost less than a human tutor and are always available.

It is feared that students will cheat on assignment in the face of the rise of chatGpT. Begnato, who is based in Tempe and others are looking at large language models as tools to enhance education.

A group of graduate students and teaching professionals were asked by a psychologist to discuss their work in an unusual way. They spoke with each other and conversed with a collection of chatbots that Beghetto had designed and that will be hosted on a platform run by his institute, Arizona State University.