Treating AI as a Teammate: How We Calibrate Trust and Inspect Autonomous AI Reasoning

Second article in a three-part series drawn from our internal study of how engineers work with agentic AI. Seven developers, interviewed over two weeks in August 2026.

Developers have found a new kind of teammate. This “agentic” teammate can inspect the code, propose a plan, write the code, revise its own work, and carry out multiple chains of tasks with minimal intervention. Yet this assistant is different from a human teammate: it does not share our organisational history, social context, or intuitive understanding of what matters. So, it can act independently without necessarily understanding the consequences of those actions in the same way we do.

That raises a basic question: how do we collaborate with a non-human teammate, and what does it mean to trust one?

Across our interviews, developers were willing to delegate a surprising amount of work to AI agents. This delegation was bounded by certain criteria that developed from their trust in the agents. Trust was not treated as a general belief that the AI was reliable. It was allocated according to the task, the environment, and the consequences of being wrong. Hence, boundaries developed around what the agent could do, where it could act, and when human judgment had to return.

One thing is worth noticing early, because everything else follows from it. Almost all of this control is applied before the agent runs. Developers set the boundaries, prepare the environment, configure the behaviour, and supply the context. Then they let it go. Very little control is exercised while the agent is working.

The trust boundaries

One of the clearest patterns in our interviews was that developers did not decide what to delegate using a single rule. At the lower end of the risk spectrum, the boundary was flexible. Routine, reversible, and low-stakes work could often be handed over completely. Participants delegated tasks such as unit tests, boilerplate CRUD, minor merge conflicts, and reading project history. But what counted as an easy or safe task differed between developers. A task one developer considered trivial might still require supervision for another. Here, delegation depended largely on task complexity. Developers adjusted autonomy according to their own confidence, experience, and familiarity with the work.

At the opposite end of the spectrum, the boundary behaved very differently. When an action became difficult or impossible to reverse, participants stopped treating autonomy as a matter of personal comfort. Production databases, migrations, deployment, infrastructure, and other actions with a large blast radius were consistently treated as human-controlled territory. These differences across the spectrum can be seen in Figure 1.

One participant reduced the boundary to a standing instruction: the agent could change local code, but it should never deploy, push, or migrate. This distinction matters because these hard limits were not based primarily on whether developers believed the agent was capable. An agent might generate a migration correctly and still not be allowed to run it. So, a strong track record did not automatically move the boundary.

The trust model that emerges is therefore not simply: the better the AI performs, the more we let it do. Developers appear to evaluate autonomy along two different dimensions: how difficult the task is and how recoverable a mistake would be. Task complexity creates a negotiable boundary, but irreversibility creates an inviolable one.

Safety comes from where the agent can reach

One participant stated that they allowed highly autonomous behaviour in an experimental environment while refusing the same setup for anything touching production. The agent was permitted to act with very little supervision, not because the work was safe, but because the environment was contained.

This is a different kind of control from the irreversibility boundary, and in practice it is the more scalable one. Judging reversibility requires a decision for every consequential action. Containment is arranged once, and then a large amount of autonomy becomes affordable inside it.

It is also the reason developers can be genuinely permissive without being reckless. Autonomy is not being granted to the agent in general. It is being granted inside a space where being wrong is cheap.

Seeing the rules is not the same as knowing they were followed

All of this depends on the agent being adequately informed, and developers invest heavily in making sure it is. As we described in the first article, that effort runs into a limit that no tooling removes: the agent knows the codebase and the written task, while the human still holds the business problem, the client's needs, and the reasons the work exists at all. Rules and specifications are the portion of context that could be written down. Everything that follows rests on that portion being both present and applied.

Generation is now cheap, and judgment is not. An agent can produce far more code than a developer can meaningfully review, and the constraint has moved downstream to inspection. That is part of why the pre-run controls matter so much: if judgment is the scarce resource, boundaries and environments have to absorb risks that inspection cannot.

But inspection has a harder problem than volume. Developers spend substantial effort supplying agents with project rules, specifications, documentation, and other context. These artefacts are supposed to constrain the agent's decisions and make its behaviour more consistent with the project.

Having a rule available to the model is not the same as knowing that the rule influenced a particular decision. From the developer's point of view, several different mechanisms can produce the same output. A rule might have been loaded and followed too literally. It might have been available but ignored. Or it might not have been retrieved at all.

These are very different failures internally, and they look almost identical from outside the system. What we learned from the interviews illustrates exactly that. Two developers worked on a shared feature, each with their own agent. A rule had been written for one situation. Later, it was applied in a different context where it did not fit. Neither developer noticed the mismatch before the agents acted on it, because from each developer's point of view, both outputs were correct.

It was entirely reversible, so the hard boundary never engaged. It touched nothing in production, so containment was irrelevant. Neither agent behaved incorrectly—each followed the context it was given. The code was correct. Tests would pass. A line-by-line review of either developer's diff would show nothing wrong, because nothing in either diff was wrong. The problem only existed in the relationship between two pieces of work that no single review covered.

This is the gap we think deserves more attention. The controls developers build are calibrated to catastrophe: severe, and visible. The harm we actually observed was mundane and cumulative.

Trust is something we design around

Treating AI as a teammate does not mean deciding that the AI is trustworthy and progressively removing supervision.

Our participants instead built systems around the possibility that the agent could be both highly capable and wrong. They delegated more when work was reversible. They retained hard boundaries around actions with irreversible consequences. They used local and experimental environments to reduce the cost of failure. They configured agents to challenge their thinking. And as generation accelerated, they redirected more of their own effort toward deciding whether the resulting work should be accepted.

We give people different levels of access, require review for consequential changes, separate development from production environments, and introduce approval gates where mistakes would be expensive. The difference is that agentic AI forces many of those assumptions to become explicit—and shows us where we drew them in the wrong place. As agents become more capable, asking “Do you trust the AI?” becomes a less useful question.

The more useful questions are: trusted to do what, under which conditions, how reversible is failure, and what evidence will a developer have that the work deserves to move forward?