ทันเอไอ

Researchers: AI Coding Proficiency Measurement Still 'Volatile' with High Interference

Studies from top institutions reveal that AI coding capability assessments may be inaccurate due to unseen factors, leading to rankings that might not reflect models' true abilities.

AI coding capability rankings might not be as reliable as previously thought, as researchers from multiple institutions are questioning the accuracy of current standard benchmarks. While OpenAI has not directly confirmed this analysis, the issue of 'noise' in these measurements is now a widespread debate. Research from the Allen Institute for AI (Ai2) indicates that 'noise' (random variability) in some benchmarks, like MBPP, is significantly high, causing up to 15.7% error in model performance predictions. Anthropic's research, moreover, found that simply adjusting basic testing computer resources, such as RAM, can alter scores by up to 6 percentage points – sometimes more than the score difference between rival models on leaderboards. This highlights the challenge of establishing reliable AI measurement standards, which are crucial for the transparent and accurate advancement and comparison of large language models (LLMs).