Alice in Wonderland: Simple Tasks Showing Complete Reasoning Breakdown in State-Of-the-Art Large Language Models

Alice in Wonderland: Simple Tasks Showing Complete Reasoning Breakdown in State-Of-the-Art Large Language Models
Data	2024
Autori	Marianna Nezhurina, Lucia Cipolina-Kun, Mehdi Cherti, J. Jitsev
URL	https://www.semanticscholar.org/paper/32e086dd56041ddb63d9e9e210c29a7fdeabdb6d
Topic	Ragionamento nei Large Language Models
Citazioni	5

Questo articolo evidenzia come i Large Language Models (LLM), nonostante le loro capacità avanzate, possano fallire in compiti di ragionamento semplici e di buon senso. Gli autori presentano un problema specifico, denominato "AIW problem", che i modelli LLM non riescono a risolvere in modo affidabile, mostrando incoerenza e scarsa capacità di ragionamento logico. L'articolo solleva dubbi sulla reale capacità di ragionamento dei LLM e incoraggia ulteriori ricerche per la creazione di benchmark più accurati per valutare tali capacità.

AIW Prompt

AIW: "Alice has N brothers and she also has M sisters. How many sisters does Alice’s brother have?"

AIW+: "Alice has 3 sisters. Her mother has 1 sister who does not have children - she has 7 nephews and nieces and also 2 brothers. Alice’s father has a brother who has 5 nephews and nieces in total, and who has also 1 son. How many cousins does Alice’s sister have? Solve this problem and provide the final answer in following form: "### Answer: "

AIW Ext: "Alice and Bob are sister and brother. Alice has 3 sisters and Bob has 1 brother. How many brothers does Alice have?"

Collegamenti

https://arxiv.org/pdf/2406.02061

Why "Grokking" AI Would Be A Key To AGI