Stroop Test Exposes Inherent LLM Flaw

Key Points:

• A recent study led by Suketu Patel revealed a fundamental flaw in large language models’ (LLMs) attention mechanisms, showing that these models suffer severe performance degradation on the Stroop task as input length increases, unlike human brains which maintain stable executive control.
• Testing premier models such as GPT-4o, Claude 3.5 Sonnet, GPT-5, Claude Opus 4.1, and Gemini 2.5, researchers found accuracy plummeted from over 90% on short lists to near zero on long or mixed-color word lists, indicating a catastrophic collapse in inhibitory control and task focus.
• Unlike humans who can suppress automatic word reading impulses to focus on color naming even over long sequences, transformer-based LLMs lack an explicit executive control mechanism, causing them to default to trained instincts and fail in conflict resolution under extended contexts.
• The study highlights a systemic architectural limitation in synthetic attention compared to biological attention, suggesting that incorporating executive control mechanisms similar to those in human cognition is essential for advancing toward true artificial general intelligence.
• These findings were published in PNAS Nexus under the paper “Deficient executive control in transformer attention,” emphasizing the need for new model designs that can adaptively regulate attention under rising interference for improved AI decision-making.