Agent FAQ

Everything around the model: prompts, tools (including MCP servers), the filesystem, memory, skills, subagents, permissions, and human approval steps. The model reasons. The harness decides what it can see, do, and remember.

In LangChain's Deep Agents, the harness has four groups: execution environment, context management, delegation, and steering.

Its inputs change even when your code doesn't. Models get updated or retired, tools and data schemas change, and prompts get edited. A prompt edit can change behavior while data monitors stay silent, so the model, prompt, tools, and parameters need to be versioned and checked together.

Each agent needs a named owner who decides on releases and answers for the success measure. A shared harness engineer owns prompts, tools, and permissions. Data owners certify the inputs. Reviewers approve outputs during Pilot and Deploy.

On a small team one person can hold several roles, but write down who does. Unfilled owner placeholders on an agent page are the first thing to fix.

Make one targeted change at a time and write down why. Run the regression suite, the must-pass cases, and the holdout set. Accept it only if new cases pass without breaking old ones, and have a second person review the diff. Update the config manifest and change log.

A group of eval cases you don't tune against. Tuning against a fixed set tends to overfit: the fix works on those cases and nowhere else. If the holdout set doesn't improve, the change probably doesn't generalize.

Treat it as a release. Run the full eval suite on the new model before switching. Then remove harness components one at a time and compare results, because some scaffolding may no longer be needed. Anthropic's guidance is to “re-examine a harness, stripping away pieces that are no longer load-bearing.”

Roll out gradually and keep the old config to roll back to.

No. Only remove what your evals show is no longer needed. If you stay on a fixed older model, scaffolding that compensates for its weaknesses stays necessary.

A floating alias like “latest” can change the model under you with no change on your side. Pin an exact version and tag every trace with it, so you can tell whether a behavior change came from the model or from something you edited.

Three places: cases engineers write by hand, production failures, and user corrections or edits. Production failures and user edits are usually the best source. Tag each case by behavior, such as tool selection, so you can run subsets.

There's no sourced number. A reasonable starting point is 20 to 30 cases from real examples, split into an optimization set and a holdout set, with a small must-pass subset. Add a case every time something fails.

No. Remove cases that always pass and no longer tell you anything, and add cases from recent failures. LangChain's authors say the suite shouldn't only grow, because model capabilities and desired behaviors change.

Every step of a run: LLM calls, tool calls, retrieval, and subagent work, with cost and latency, tagged with the config version. Track tool-error rate, retries, cost per run, and how often reviewers edit or reject output.

Tools named in the sources include LangSmith, Langfuse, Arize Phoenix, Braintrust, MLflow, DeepEval, and promptfoo. They haven't been compared here.

Graders tend to approve flawed work, and a separate grader is easier to tune. Anthropic found that tuning a standalone evaluator to be skeptical is far more tractable than making a generator critique itself. Calibrate the grader against human judgments.

When a run gets long, the harness summarizes or clears earlier content to stay within the model's limit. Preprints report that this can silently drop standing rules, such as safety constraints, because summaries treat them as low priority.

Pin critical rules so compaction can't remove them, keep large tool output in files, and log failures that happen after compaction. One paper's figure that about half of constraints survive uniform compaction comes from a single study.

In Deep Agents, subagents start with a fresh context by default. They only know what you put in their instructions. Include the facts they need, or use fork mode, which inherits the parent's history.

It can be. If it records something false, the mistake comes back in later runs. Review changes to AGENTS.md the way you review code, keep it in version control, and add an eval for any wrong belief you remove.

Filesystem permissions= rules are evaluated first-match-wins, and operations that match no rule are allowed. A gap means access. End each rule list with an explicit deny, and review the rules monthly and whenever you add a tool or subagent.

Permissions also don't apply to sandbox backends.

For any tool that writes, sends, or changes something. Deep Agents supports this through interrupt_on. Keep it on, extend it when you add tools, and revisit it before moving an agent to a wider group.

Also check whether reviewers have started approving without reading, for example near-zero edits over several weeks.

Contain it by pausing the agent or tightening approval. Save the trace, inputs, and config version. Find the cause: model, prompt, tool, data, or permission. Fix it in the harness rather than patching one output. Add the failure as an eval case, then log the incident and tell the agent's users.

Add a heartbeat alert, a failure count, and a daily check that the batch ran and how many items it handled. Silent stops are easy to miss because nothing errors out.

Check failed runs and cost daily, read a sample of traces weekly, review permissions and data sources monthly, and review each agent against its success measure quarterly. Re-test the harness on every model release.

In week one, name an owner for each agent and fill in the placeholders on the agent pages. Then turn on tracing and pin the model for one agent, build 20 to 30 eval cases, and run the first weekly review.

Start with the agent that has real users and use it as the template for the others.