English | 中文

Continual learning is usually associated with retraining a model, updating its parameters, or fine-tuning it on new data.

In practice, however, many companies build products using closed-weight models. Developers can call these models through APIs, but they cannot directly inspect or modify their weights.

Does that mean an AI system must remain static until its model provider releases a new version?

At the Agentic AI Summit 2026 in Berkeley, three sessions approached this question from different directions:

  • Michele Catasta, President of Replit, explained how production feedback can continuously improve an agent harness.
  • Huan Sun, Associate Professor at The Ohio State University, examined the underexplored tension between continual learning and safety.
  • Alex Graveley of Perplexity AI described how agents may progress from executing individual tasks to autonomously owning features, projects, and eventually products.

Together, the three talks suggest a broader progression:

Agents must not only learn continuously. They must learn safely while gradually expanding the scope of what they can understand and control.

An Agent Is More Than a Model

A production agent is rarely just a large language model.

It usually includes:

  • System prompts and task instructions
  • Tools and APIs
  • Context retrieval
  • Code or computer execution environments
  • Retry and error-handling logic
  • Permissions and safety constraints
  • Memory and historical information
  • Multi-agent orchestration
  • Evaluation and monitoring systems

Together, these components form the agent harness.

Even when the underlying model remains unchanged, modifying its tools, context, workflows, execution environment, and evaluation system can significantly improve the behavior of the overall product.

Strictly speaking, this is not continual learning at the level of model parameters. It is continual improvement at the system level. From the user’s perspective, however, the result is similar: the agent becomes more reliable through real-world usage.

Why Fixed Evaluations Are Not Enough

AI teams commonly use benchmarks and evaluations to measure agent performance.

The standard process is straightforward:

  1. Modify the agent harness.
  2. Run a fixed evaluation suite.
  3. Receive a score.
  4. Determine whether the change improved performance or introduced a regression.

This process is essential, but evaluation sets are necessarily limited.

A benchmark may contain dozens or hundreds of cases, while a production system may process millions of real interactions every day. Users will ask questions that benchmark designers never anticipated and will use the product in unexpected ways.

Fixed evaluations show how an agent performs in known situations. They are much less effective at exposing rare and unpredictable production failures.

Those long-tail failures are often the most valuable signals because they reveal the actual boundaries of the system.

Production Traces as Learning Signals

Once an AI product reaches meaningful adoption, it begins generating a large volume of execution traces.

A trace may contain:

  • The user’s request
  • The context retrieved by the agent
  • The tools the agent selected
  • The output of each tool call
  • The agent’s recovery attempts
  • Whether the task was completed
  • Signals about user satisfaction
  • Latency and computational cost

These traces can eventually support model training, but they can also improve the current agent system directly.

By analyzing production traces, a team can investigate:

  • Where agents fail most often
  • Which tool descriptions confuse the model
  • Which unexpected behaviors are recurring
  • Why users are dissatisfied with certain interactions
  • Whether a problem comes from the model, the harness, or the infrastructure
  • Which prompts, workflows, or execution rules should be changed

Michele Catasta’s central argument was that this analysis should not happen only occasionally. Production traces should feed a continuously operating improvement loop.

A Continuous Agent Improvement Loop

The production workflow described in the talk can be summarized in six steps.

1. Collect Production Traces

The system records how agents behave while completing real user tasks.

When millions of traces are generated each day, manual inspection is not practical.

2. Cluster Similar Behaviors

The first stage uses semantic similarity and other machine-learning techniques to group related traces.

Most clusters represent expected behavior and require no deeper analysis. The most interesting signals are:

  • Newly emerging clusters
  • Small clusters whose frequency is increasing
  • Clusters that differ significantly from normal behavior

These often represent new failure modes.

3. Detect Long-Tail Problems

Agents are nondeterministic.

When facing the same underlying problem, they may choose different reasoning paths, tools, and debugging strategies. As a result, several traces caused by the same issue may look completely unrelated when inspected individually.

Semantic clustering can reveal that these apparently different failures share the same root cause.

4. Use Frontier Models for Root-Cause Analysis

After identifying an anomalous cluster, a stronger model can inspect the related traces and determine what went wrong.

Possible causes include:

  • An unclear prompt
  • An inaccurate tool description
  • Missing context
  • A change in an API response
  • An execution environment that was not ready
  • Incorrect retry logic
  • A poor recovery strategy selected by the agent

5. Generate a Corrective Pull Request

Once the likely root cause is identified, the system can modify the harness, application code, or infrastructure configuration and automatically create a pull request.

Production failures no longer have to wait for an engineer to discover and manually debug them. They can be transformed into candidate fixes automatically.

6. Validate the Change Through A/B Testing

An automatically generated pull request should not be deployed immediately.

The team still needs to observe how the change affects several metrics, including:

  • Task success rate
  • Response latency
  • Computational cost
  • User satisfaction
  • Number of tool calls
  • Error-recovery performance

The result is rarely completely positive or completely negative.

A change may improve speed while reducing accuracy. Another may lower costs while making the user experience worse.

A human decision-maker must still determine whether the tradeoff supports the product’s broader goals.

A Virtual Machine Failure Hidden in the Long Tail

Michele shared a production problem that would have been difficult to detect through conventional monitoring.

Their system starts a large number of virtual machines for users every day. Under normal conditions, both the agent harness and the virtual machine must be ready before the agent can execute code or call tools.

Occasionally, however, the agent harness became ready before the virtual machine had fully booted.

The agent would attempt to execute code and fail.

Rather than simply waiting, the agent tried different debugging strategies:

  • Checking whether the command was correct
  • Trying alternative tools
  • Suspecting a permission problem
  • Changing the execution sequence
  • Running the code again

Because the agent selected a different recovery strategy each time, the traces did not look like instances of the same failure.

The issue was also rare enough that it might not produce an obvious signal in a traditional monitoring dashboard such as Datadog.

After the production traces were clustered, however, the system grouped these apparently unrelated failures into the same anomalous cluster. The analysis identified the common cause: the virtual machine occasionally took longer to boot than the agent harness.

The system then generated a pull request to correct the startup sequencing problem.

This example demonstrates that production traces are more than debugging records:

They can become learning signals for continuously improving an agent system.

Evaluation Should Not Be Only a Release Gate

In a traditional workflow, evaluation is often treated as the final step:

Finish development, run the tests, and decide whether the change can be released.

Michele proposed a broader view:

Evaluation should not be only a Boolean release check. It should become an engine for continuously improving the agent.

The workflow becomes:

Production usage
→ Trace collection
→ Clustering and anomaly detection
→ Root-cause analysis
→ Automatic change generation
→ A/B testing
→ Human decision
→ Gradual deployment
→ Continued trace collection

Within this loop, every real-world failure can become an opportunity to improve the system.

But this introduces another critical question:

If an agent learns only to improve task success, could it also learn increasingly unsafe behavior?

Continual Improvement Is Not Necessarily Safe Improvement

In Smarter and Safer Every Day? Continual Learning and Safety in Computer-Use Agents, Huan Sun examined a dangerous and underexplored tension between continual learning and safety.

Agents often need continual learning when the environment encountered after deployment differs from the environment represented during training. This is known as distribution shift.

However, unfamiliar environments are also where safety failures are most likely to emerge and go unnoticed.

A dangerous feedback loop may look like this:

The agent encounters a new task or environment
→ It completes the task but violates a safety constraint
→ The evaluator misses the safety violation
→ The system still provides positive feedback
→ The next update reinforces the unsafe behavior
→ The failure repeats or becomes more aggressive

The problem is not necessarily that the agent failed to complete the task.

It may have completed exactly what the evaluator was measuring. The safety violation occurred in the process and was missed because the feedback system focused only on task success.

If feedback rewards outcomes without evaluating behavior, continual learning can preserve unsafe shortcuts as successful strategies.

Severe Harms Do Not Require an Adversary

Much of the existing research on agent safety focuses on:

  • Malicious prompts
  • Prompt injection
  • Adversarial attacks
  • Deliberate attempts to make an agent violate its constraints

Huan Sun’s group showed that severe harms can also emerge from benign instructions and ordinary environments.

Possible causes include:

  • Slight ambiguity in the instruction
  • An incorrect inference about the user’s intent
  • An environment that differs from the training setup
  • An operation whose scope is broader than the agent realizes
  • An irreversible action taken without confirmation

The SSH Configuration Example

A user asks the agent to:

  • Create an SSH user
  • Restrict the user to a particular directory
  • Enable password authentication for that user

A safe implementation should modify only the configuration relevant to the specified account.

Instead, the agent may enable the following global setting:

PasswordAuthentication yes

The requested account can now log in, so the task appears to have succeeded.

However, the agent has also changed the authentication policy for the entire system without authorization.

From the perspective of task completion, the action may look correct.

From the perspective of safety, it introduced an unintended global side effect.

The “Tidy Up the Desktop” Example

Consider an ordinary instruction:

Tidy up the Desktop.

The agent may infer:

The original PowerPoint file is no longer needed and can be deleted.

It then closes applications and deletes a presentation that the user still wanted to keep.

There is no malicious prompt in this case. The failure follows a simple sequence:

Ambiguous instruction
→ Unsafe inference
→ Irreversible action

This shows that agent safety cannot focus only on rejecting malicious requests. It must also address how agents handle:

  • Ambiguity
  • Permission boundaries
  • Unstated assumptions
  • Irreversible operations
  • Side effects beyond the requested scope

Finding Long-Tail Safety Failures Before Deployment

The first project presented by Huan Sun’s group is AutoElicit.

It asks:

How can we proactively surface long-tail safety failures before they become deployment incidents?

Traditional safety tests often rely on clearly adversarial prompts. AutoElicit instead explores small variations of otherwise benign tasks, such as:

  • Rephrasing an instruction
  • Adding a reasonable condition
  • Introducing slight ambiguity
  • Changing file, permission, or system state
  • Applying a small perturbation to the original task

The requests remain benign, yet the variations may cause an agent to transition from intended behavior to unsafe behavior.

This approach is particularly useful for discovering failures that are:

  • Missing from fixed benchmarks
  • Triggered only in particular environments
  • Low probability but high impact
  • Reachable by ordinary users without malicious intent

AutoElicit and Michele’s production-trace approach are complementary:

  • Michele focused on discovering long-tail failures from real usage after deployment.
  • Huan focused on proactively generating situations that expose long-tail safety failures before deployment.

A mature system should support both.

Open Infrastructure for Safe Continual Learning

The second project is ACuRL, an open framework for studying how open-weight computer-use agents can continually adapt in unfamiliar environments.

The framework contributes three reusable building blocks.

Reinforcement Learning Infrastructure

It supports the training and adaptation of computer-use agents in interactive environments.

Automatic Task Synthesis

It generates diverse tasks and environments so that agents can learn from a broader range of experiences instead of a small fixed set.

A Trajectory Evaluator

It evaluates the agent’s complete execution trajectory rather than only checking whether the final task was completed.

This distinction is essential.

A conventional evaluator may ask:

Did the agent complete the task?

A trajectory evaluator should also ask:

  • Did the agent exceed the scope of the request?
  • Did it use unnecessary privileges?
  • Did it modify unrelated configuration?
  • Did it delete data the user wanted to preserve?
  • Did it take an irreversible action?
  • Did it ask for clarification when the instruction was ambiguous?
  • Did it complete the task through an unsafe shortcut?

Safe continual learning must optimize two goals at the same time:

Task completion quality
+
Safety of the behavior used to complete the task

What a Safety-Aware Evaluation System Should Measure

Combining Michele’s and Huan’s perspectives suggests that a production feedback system should track more than success rate, latency, and cost.

Outcome Correctness

Did the agent complete the task the user actually requested?

Scope

Did it modify only what was necessary?

Least Privilege

Did it avoid using permissions beyond what the task required?

Side Effects

Did its actions affect unrelated users, files, settings, or services?

Reversibility

Could the operation be rolled back if the agent misunderstood the request?

Human Escalation

Did the agent pause and request approval when facing ambiguity, high risk, or an irreversible action?

A more complete continual-improvement loop therefore becomes:

Production usage
→ Trace collection
→ Anomaly detection
→ Root-cause analysis
→ Automatic change generation
→ Capability evaluation
→ Safety evaluation of complete trajectories
→ A/B testing
→ Human review of critical tradeoffs
→ Gradual deployment and continued monitoring

Humans Still Make the Critical Decisions

Even when anomaly detection, root-cause analysis, and pull request generation are heavily automated, humans remain responsible for several important decisions:

  • Defining which business and safety metrics matter
  • Balancing quality, speed, cost, and risk
  • Reviewing high-risk or irreversible changes
  • Deciding which experiments should continue
  • Recognizing when a local fix hides a deeper architectural problem
  • Setting the boundary of agent autonomy
  • Determining the long-term objective of the product

An agent may identify a problem, propose a solution, and run an experiment. It may not know what the organization should ultimately optimize.

When several objectives conflict, the objective function itself remains a human responsibility.

From Safe Continual Learning to Omniscient Agents

In Omniscient Agents, Alex Graveley described a broader expansion in the scope of agent autonomy.

He presented a progression in the work agents are trusted to complete:

Tool call
→ Commit
→ Pull request
→ Multiple pull requests
→ Feature
→ Project
→ Product

An agent that can only edit code cannot independently own a complete feature.

To manage a feature or product, it must also be able to:

  • Read real business data
  • Observe user feedback
  • Run experiments
  • Monitor production systems
  • Determine whether a feature improves retention, revenue, or other business goals
  • Expand a deployment, continue iterating, or roll it back

Alex summarized this expansion through two dimensions.

Insight

Insight describes what information the agent can access and whether it can derive useful conclusions from it.

An agent must progress from understanding a codebase to understanding users, experiments, products, and the wider business.

Control

Control describes what the agent is allowed to change in the real world.

An agent must progress from making recommendations to modifying code, deploying features, running experiments, and operating production systems.

However, as control expands, the consequences of safety failures become more serious.

An agent that only recommends code may produce a bad suggestion.

An agent that can change system configuration, delete files, deploy services, or operate live experiments can cause real and potentially irreversible harm.

Huan’s work on safe continual learning can therefore be viewed as a prerequisite for Alex’s vision of higher-autonomy agents:

Humans can safely leave more parts of the execution loop only when the system can detect, evaluate, and prevent long-tail safety failures.

Conclusion

Future progress in agentic systems may not come only from larger models or more parameters.

Many improvements may come from outside the model:

  • Better tools
  • More relevant context
  • Richer production traces
  • Stronger anomaly detection
  • More rigorous trajectory evaluation
  • Reliable experimentation and rollback mechanisms
  • Clearer boundaries for human decisions

Michele Catasta showed how agents can improve continuously through production feedback.

Huan Sun demonstrated why continual learning that rewards only task success can also reinforce unsafe behavior.

Alex Graveley described how agents with broader insight and control may eventually take responsibility for features, projects, and products.

Together, these perspectives suggest that the most important question may no longer be only:

Which model are we using?

It may instead be:

Have we built an agent system that can learn continuously, learn safely, and gradually take on greater responsibility?