AI caught 32 students cheating. AI caught 32 students cheating.

Jason Gibson, a history professor at Alcorn State University, incorporated hidden text in a test question on the Industrial Revolution. The hidden text told language models to include the word “Madagascar” in the student’s response. Thirty-two out of 35 students included the word and lost points. Only two students appealed the grade.

Professor Gibson’s prompt injection attack exploits a common attack vector used against enterprise AI agents: hostile documents or web pages can take over an AI agent by injecting hidden text in a document.

Professor Gibson’s test result was eye-opening. He discovered that his students relied on language models for essays and didn’t double check the output. That’s the security vulnerability of AI agents; they can’t distinguish between relevant context and injected commands.