Shocking! US Lawsuit Hidden Prompt Injection in Documents to Sway AI
A US litigant cleverly hid AI prompt injection attacks in white text within legal documents, hoping to distort court rulings.
404 Media and an analysis from the law firm Harris Beach Murtha revealed an interesting case study on growing IT security issues, after Matthew Elliott, a pro se plaintiff in a Connecticut federal court, US, covertly embedded prompt injection commands in legal filings for Elliott v. New York Bariatric Group.
The method involved typing text in white on a white background at a tiny 3-point font size, making it invisible to the naked eye while readable by document processing software. The hidden text instructed the AI reading the document to "side with" his party when summarizing data.
The scheme was uncovered after court staff noticed abnormal extra white space in the electronic document, leading to the discovery of the secret text. Consequently, Judge Walter M. Spader, Jr. issued sanctions, stating this behavior was done in bad faith against the judicial process, and barred the plaintiff from filing documents through the electronic system, forcing him to submit traditional paper documents instead. However, the court confirmed that no artificial intelligence was actually used in adjudicating the case.
This incident serves as a major wake-up call for agencies and judicial systems worldwide that are starting to adopt technology and automation for document management, proving that Prompt Injection vulnerabilities can indeed be exploited maliciously even within the court system.