ทันเอไอ

Study Reveals: Minor Prompt Format Changes Can Swing AI Scores by Tens of Percent, Undermining Benchmark Reliability

Multiple studies expose a significant issue in AI: large language models exhibit drastically different performance just by varying prompt formatting, even when the core message is identical.

The reliability of AI model benchmarking is once again a subject of debate in the research community. Clear evidence has emerged showing that even with the same question, simply altering the prompt's presentation format (Prompt Formatting) – such as using headings, delimiters, or example order – can cause a massive shift in the accuracy scores of large language models (LLMs).

This phenomenon, termed "Format Sensitivity," has been confirmed by several recent studies. For instance, research by Sclar et al. (2024) found that the LLaMA-2-13B model showed a staggering 76 percentage point difference in accuracy just from minor prompt format adjustments in few-shot tasks. Another survey revealed that the GPT-3.5 model exhibited a score variance as high as 56 points when tested with 320 different prompt formats across 53 task types, with a median variance of 6.4 points – a figure significant enough to alter leaderboard rankings.

This issue isn't limited to the correctness of answers but also extends to schema compliance, such as instructing models to return data in valid JSON format. Studies have shown that success rates for this can fluctuate from 0% to 100%, depending on the task complexity and the model used.

While this topic is of significant interest with extensive supporting research, the 'Tun AI' news team found that a specific academic article, "Format Sensitivity Index," cited as a preliminary source, could not be confirmed on the arXiv database at this time. Nevertheless, the issue of prompt format sensitivity is widely acknowledged and being intensely studied within the field to develop more stable and reliable AI evaluation methods.