LLM-Generated Investment Research Memos Need Validation

Abstract

LLMs are increasingly adopted to draft long-form investment research memos, but their reliability is judged claim by claim, while readers read each memo as a whole. In this paper, we show that a low claim-level error rate does not necessarily mean that a memo is reliable: even with a refutation rate of just 1-2%, roughly four in ten memos contain at least one claim contradicted by official filings. First, we generate 1,440 memos with four recent LLM-powered agents for 30 companies over 12 quarters, decompose them into 77,272 financial claims, and verify each against the same official source bundle by retrieving evidence, matching it to the claim, and verifying the outcome. Second, we find that only 1.17-1.77% of claims are refuted, yet 36.94-41.11% of memos contain at least one refuted claim, and agents differ in severity: Codex has the lowest memo-level rate; a qualitative review further groups errors into four patterns (time, unit, context, concept).