AI AgentsExplainer

What Is an AI Agent, and How Often Is It Right

An agent is a language model in a loop with tools. I measured what that loop is worth on 1,534 questions, and the result explains why I still check every answer.

By Vladimir Beskorovainyi

4 min read

Illustration: BESK TECH

An AI agent is a language model placed in a loop with tools. It chooses an action, runs it, reads the result and chooses again until the task is done. A chatbot answers once. An agent acts, checks and retries, which makes it useful and is also where its errors accumulate.

A coding assistant that edits files and runs the tests is an agent. So is a support bot that looks up an order and issues a refund. The definition is short. The harder question is how often the result is right, and how you would know. I measured it on 1,534 questions.

A model in a loop

A chatbot receives a question and returns text. An agent receives a goal and works toward it in steps. At each step the model picks an action, a program carries it out, and the result goes back to the model. The loop ends when the model decides the task is done or a limit is reached.

ChatbotAgent
InputA questionA goal
OutputTextA completed action or result
ToolsNone, or a few it calls onceSearch, code, databases, files
StepsOneAs many as the task needs
Who checks the resultThe readerThe agent first, then the reader

The last row matters most. An agent can see that an action ran. It cannot see that it was the right action.

One agent, step by step

Here is a small agent I published with full code and results. It turns a question in plain English into a database query.

  1. It reads the database schema and the question.
  2. The model writes an SQL query.
  3. The program runs the query against the database.
  4. If the database returns an error, the error text goes back to the model with a request to repair the query.
  5. Steps 3 and 4 repeat until the query runs or two repair attempts are used up.
  6. The agent returns the final query, and the program runs it to get the answer.

Larger agents have more tools and longer loops, but the structure is the same.

What the loop is worth

I gave this agent 1,534 questions with known correct answers and counted how many it got right. The questions come from BIRD, a public benchmark for turning English into SQL. I hosted open-weight models myself on rented GPUs, and the best of them was Qwen2.5-Coder-32B.

ConfigurationCorrect answers
Model alone, one attempt842 of 1,534 (54.89%)
With the repair loop862 (56.19%)
With the loop and voting across five candidates861 (56.13%)

The loop repaired 27 answers and broke 7, a net gain of 20 questions out of 1,534. The gain is statistically significant (paired McNemar test, p = 0.0008), and it is small. Voting across five candidates added nothing measurable. It cost about five times the tokens, and the 95th-percentile model time per question went from 10.6 to 50.0 seconds.

The loop catches queries that fail to run. It does not catch queries that run and return the wrong answer. The database reports a syntax error. It does not report that you counted the wrong column. Of the 672 answers that were still wrong after the loop, 640 came from queries that ran without any error.

A loop is worth exactly as much as the feedback it receives. This one receives database errors. An agent that runs a test suite receives more, and gets more from its loop.

The agent does not know which answers are wrong

At 56.19%, more than four answers in ten are wrong, and 95% of the wrong ones arrive looking exactly like a correct one. The agent reports success in both cases.

Teams that deploy agents arrive at the same place. A study of agents in production, presented at ICML 2026, found that 68% of them execute at most ten steps before a human steps in, and 74% depend primarily on human evaluation.1 A separate study of 15 models concluded that recent gains in capability have produced only small gains in reliability.2

Why I check everything

So I check everything, literally everything. Skip the check and one of the answers will be wrong, and nothing will tell you which.

The check is part of the task, the same way a review is part of shipping code. An agent shortens the work. It does not remove the last step.

Where an agent pays off

An agent earns its place where a result is cheap to verify and expensive to produce. Code with a test suite fits, because the tests say pass or fail. Searching a large codebase fits, because you open the file and look. A database query fits only when you have a known answer to compare against.

It is a poor fit where an error is costly and nobody will look. In that setting the loop gives you speed and confidence, and neither of them is accuracy.

An AI agent is a capable junior colleague who never says "I am not sure". Give it work you can review, and review it.

Footnotes

  1. Measuring Agents in Production, arXiv:2512.04123, version 4, June 2026. Data collected in 2025. ↩

  2. Towards a Science of AI Agent Reliability, arXiv:2602.16666, ICML 2026. ↩

Sources

  1. How far do on-premises open LLMs get on Text-to-SQL? arXiv:2606.29733, version 2
  2. nl2sql-onprem-bench, code and per-item results
  3. Measuring Agents in Production, arXiv:2512.04123
  4. Towards a Science of AI Agent Reliability, arXiv:2602.16666

Vladimir Beskorovainyi

Vladimir Beskorovainyi, Enterprise AI Architect & CTO, author of the Amazon bestseller "Enterprise AI: The KATA Way", founder of Besk Tech (besk.tech)

Get new articles by email

One message when something new is published. No other mail, and you can stop at any time.

The address is used for this one letter and nothing else. Privacy

← All articles