Google study finds recall, not storage, limits model factuality
Google Research highlighted WikiProfile, a 2,150-fact benchmark spanning ten tasks and thirteen models, in an analysis of whether models encode facts they fail to answer directly. The authors report behavioral evidence that leading models encode 95–98% of tested facts but miss 26–34% under direct recall; extended reasoning reduces the failure rate to 11–12% and recovers 40–65% of encoded-but-not-directly-known facts.
Why it matters: If many factual failures are retrieval failures rather than absent knowledge, evaluation and system design should separately measure encoding, direct recall and recovery under reasoning or prompting.