--- title: "How often is AI output wrong on real work tasks? (17–45%) | The AI at Work Fact Book" description: "17 to 45 percent of AI answers had a hallucination or significant error in independent tests on legal and news questions. Summaries, whole projects and how fast rates change." url: "https://evy.so/facts/ai-error-rates-at-work/" last_updated: "2026-10-16" --- F-23 · What Teams Trust # How often is AI output wrong on real work tasks? In independent tests on realistic professional questions, 17 to 45 percent of AI answers contained a hallucination or significant error (2024 to 2025). Rates fall sharply when the AI only summarizes a document it is given, and rise sharply for whole projects, and they change with every model release. 17–45% of AI answers had a hallucination or significant error, in independent tests (2024 to 2025) - Medium confidence - 3 primary sources - 2025 - Last checked October 14, 2026 ## Cite this EVY (2026). How often is AI output wrong on real work tasks? The AI at Work Fact Book, F-23. https://evy.so/facts/ai-error-rates-at-work/ (last checked October 14, 2026). CC BY 4.0. Copy citation [Data as JSON](https://evy.so/facts/ai-error-rates-at-work.json) [Markdown](https://evy.so/facts/ai-error-rates-at-work.md) ## How to read it Each test sets a different task and defines an error differently, and humans grade all three. Dates matter: the legal tools were tested in 2024 and have been updated since, and in the BBC's own comparison significant issues fell from 51% to 37% in six months. The citation test asks AI to find the source of a quote, a narrow task, so it sits above the range. Error rates depend heavily on whether the AI is given the source material. ## Every study on one scale Shaded: 17–45% One figure A range in one study Chart values as text - Scale: % of answers with a serious error. Shaded range: 17–45%. - Stanford, legal research tools (2025): 17–33% (GPT-4 43%) - EBU and BBC, news answers (2025): 45% significant issue, 20% accuracy - Tow Center, news citations (2025): over 60% (citations only) **Why medium confidence:** Two independent creators using expert human grading find serious-error rates in this band for question answering. The tasks, error definitions and models differ, the tests are one to two years old, and rates move quickly with each model release. ## Sources Sources for F-23 | Study | Creator | Year | Sample | Method | Figure | Link | | --- | --- | --- | --- | --- | --- | --- | | Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools (Journal of Empirical Legal Studies) | Varun Magesh, Faiz Surani, Matthew Dahl, Mirac Suzgun, Christopher D. Manning and Daniel E. Ho (Stanford RegLab and Yale) | 2025 | 202 preregistered legal research queries per system, tested March to May 2024 | Hand-graded by experts. Hallucinated means incorrect, or citing a source that does not support the claim. | Lexis+ AI 17%, Ask Practical Law AI 17%, Westlaw AI-Assisted Research 33% hallucinated; GPT-4 without retrieval 43% “each hallucinate between 17% and 33% of the time” | [doi.org ↗](https://doi.org/10.1111/jels.12413) Open-access article; the Wiley page blocks automated access. | | News Integrity in AI Assistants: An international PSM study | European Broadcasting Union and BBC, with 22 public service media organizations | 2025 | 2,709 answers from free ChatGPT, Copilot, Gemini and Perplexity, late May to early June 2025, 18 countries | Journalists rated each answer on accuracy, sourcing, context and more | 45% had at least one significant issue; 20% a significant accuracy issue; Gemini 76%, Copilot 37%, ChatGPT 36%, Perplexity 30% “Overall, 45% of all AI responses were found to have at least one ‘significant’ issue.” | [ebu.ch ↗](https://www.ebu.ch/files/live/sites/ebu/files/Publications/MIS/open/EBU-MIS-BBC_News_Integrity_in_AI_Assistants_Report_2025.pdf) | | AI Search Has a Citation Problem (Columbia Journalism Review) | Klaudia Jaźwińska and Aisvarya Chandrasekar, Tow Center for Digital Journalism | 2025 | 1,600 queries to eight AI search tools, February 2025 | Manual grading of whether each tool identified the source article of a quoted excerpt. A narrow citation task. | Incorrect answers to more than 60% of queries; Perplexity 37%, Grok 3 94% “Collectively, they provided incorrect answers to more than 60 percent of queries.” | [cjr.org ↗](https://www.cjr.org/tow_center/we-compared-eight-ai-search-engines-theyre-all-bad-at-citing-news.php) | Summarizing a document ## How often does AI invent content when summarizing a document? Much less often than when answering from its own knowledge. On Vectara's leaderboard of 22 September 2026, 108 models added unsupported content to 1.8% to 24.2% of summaries, with a median of about 9.6% and frontier models at about 6% to 12%. The judging is done by Vectara's own detection model, not by people. How often does AI invent content when summarizing a document? | Study | Creator | Year | Sample | Method | Figure | Link | | --- | --- | --- | --- | --- | --- | --- | | Hallucination Leaderboard (HHEM-2.3) | Vectara | 2026 | Over 7,700 articles summarized by each of 108 models; leaderboard dated 22 September 2026 | Automated benchmark judged by Vectara's own detection model. Vectara sells retrieval and hallucination-detection products. | Hallucination rates from 1.8% to 24.2% of summaries; median about 9.6% “This evaluates how often an LLM introduces hallucinations when summarizing a document.” | [github.com ↗](https://github.com/vectara/hallucination-leaderboard) | Whole projects ## How often can AI agents complete whole work projects? Rarely, but fast-rising. On the Remote Labor Index, which gives AI agents real paid freelance projects averaging 29 hours of human work, the best agent delivered work a client would accept on 21.25% of projects as of 14 October 2026, up from 2.5% at launch in October 2025. How often can AI agents complete whole work projects? | Study | Creator | Year | Sample | Method | Figure | Link | | --- | --- | --- | --- | --- | --- | --- | | Remote Labor Index: Measuring AI Automation of Remote Work | Center for AI Safety and Scale AI | 2026 | 230 real freelance projects in design, data analysis, game development, video and more | Human evaluators judged whether each AI deliverable matched the professional's and was acceptable as commissioned work. Scale AI sells AI evaluation. | Best agent 21.25% of projects (read 14 October 2026); 2.5% at launch in October 2025 “The best-performing model achieves an automation rate of only 21.25%.” | [labs.scale.com ↗](https://labs.scale.com/leaderboard/rli) | ## Numbers you may have seen These figures circulate widely. Here is what the original source says. - **AI assistants get the news wrong 45% of the time (EBU and BBC).** 45% of answers had at least one significant issue of any kind, mostly sourcing. Significant accuracy errors were 20%. - **AI search engines are wrong 60% of the time (Tow Center).** The test asked eight tools to find the source article of a quoted excerpt. It measures news citation, not general accuracy. - **Legal AI tools are hallucination-free.** Stanford's preregistered test found the tools hallucinated on 17% to 33% of queries. Vendor wording refers to whether linked citations exist, a narrower claim. - **Chatbots hallucinate up to 27% of the time (Vectara).** That comes from 2023 coverage of Vectara's first leaderboard. In September 2026 the range across 108 models was 1.8% to 24.2%, on a new test. - **AI can do only 2.5% of remote work.** 2.5% was the best agent at the Remote Labor Index launch in October 2025. In October 2026 it was 21.25%, counting whole projects done to client standard. ## More from What Teams Trust - [F-21**24–53%**How much do workers trust the output of AI tools at work?](https://evy.so/facts/trust-in-ai-output/) - [F-22**41–44%**How often do people check AI output before using it?](https://evy.so/facts/checking-ai-output/) - [F-24**2–12%**Which work tasks do people want AI to do on its own, and which do they want to keep?](https://evy.so/facts/tasks-workers-want-ai-to-do/) - [F-25**30–44%**How many workers use AI in ways that break company policy?](https://evy.so/facts/ai-use-against-company-policy/) - [F-26**42–48%**How often do employees put company data into public AI tools?](https://evy.so/facts/company-data-in-public-ai/) - [F-27**21–29%**How many organizations have had a security incident or data breach involving AI?](https://evy.so/facts/ai-security-breaches/) [The whole chapter](https://evy.so/facts/what-teams-trust/) · [Chapter poster](https://evy.so/facts/what-teams-trust/sheet/) · [All of The AI at Work Fact Book](https://evy.so/facts/)