Lloyaldocs
Understand

Adaptive compute through semantic pruning

Whole lines of work are pruned on judgement, not windows compacted on arithmetic

On this page

Compaction is positional: hit a token count, summarise the window. Pruning here is semantic: AgentPolicy decides a line of work is finished, drifted or no longer owned, and that whole branch returns its KV to its siblings mid-run. Headroom becomes something the pool recovers rather than something it runs out of — which is what makes effort a dial instead of a budget you spend once.

This guide is how you drive that: what the pool watches, where the limits sit, what an Agent should do next, and how work is saved before its memory is reclaimed.

It assumes the ownership model from Thinking in Lloyal — scopes, Agents, and what ends together:

  • Effection owns the lifetime of the harness and pool;
  • the AgentPool owns Agent execution and branch lifecycle;
  • an Agent is not an Effection Task;
  • stopping one Agent is not the same as halting its parent scope.

BranchStore accounts. ContextPressure observes. AgentPolicy decides. AgentPool enforces.

Policy interprets live execution at explicit lifecycle boundaries. The pool remains responsible for decode, settlement, recovery, and pruning.

The quickest way into the model is to separate ownership, policy, and mechanical safety.

Start here

The decision hierarchy

A policy decision sits inside a larger execution stack:

Structure
Effection scope
└── AgentPool                              owns the live execution
    ├── BranchStore                        accounts for inference state
    ├── ContextPressure                    freezes the current resource facts
    ├── AgentPolicy                        chooses the next behavioural action
    └── Pool mechanics                     safely apply that action
        ├── decode and commit
        ├── Tool dispatch and settlement
        ├── recovery
        └── branch pruning

Read it from the outside in:

  1. Effection decides whether the pool still has an owner.
  2. BranchStore records what live inference state exists.
  3. ContextPressure presents a stable snapshot of what remains.
  4. AgentPolicy interprets the situation.
  5. AgentPool performs the decision without violating runtime safety.

Policy never owns the Session, schedules native decode, or prunes branches directly.

The three questions

1. Does this work still have an owner?

That is a structured-concurrency question.

Structure
Session released
    ↓
harness scope halts
    ↓
pool and all child work unwind

AgentPolicy is not consulted to preserve work whose owner has disappeared.

2. Should this Agent continue?

That is a policy question.

Structure
pool remains alive
    ↓
Agent reaches a decision boundary
    ↓
policy chooses continue, nudge, retry, return, stop, or recover

The Agent may stop while its siblings and parent pool continue.

3. Can the requested action be performed safely?

That is an AgentPool question.

Policy can request recovery or provide a preferred report budget. The pool computes what fits, enforces grammar and token limits, and protects the native context.

The five rules

  1. Policy is a rulebook, not a scheduler.
  2. Pressure is a snapshot, not a mutable global gauge.
  3. The soft limit closes new work.
  4. The hard limit ends working turns; the floor bounds what a report may write.
  5. Policy requests recovery; the pool makes it mechanically safe.

The two axes

Pressure affects both strategy and lifecycle, but these are not the same decision.

Structure
strategy:
explore ───────────────────────────────→ exploit

lifecycle:
work ─────────────→ report ────────────→ recover or prune

An Agent can exploit while still healthy.

shouldExplore() === false does not mean the Agent should stop.

Configure it

This page is how the pool decides. To set its numbers and hooks — the budget row, a custom policy, wrap-up and cancel — see Agent policy; for the five moments of a tool call, Tool hooks and guards.

1. Know what you own

Scope termination and Agent termination

These events can look similar in a UI but have different meanings.

The owner disappears

Structure
browser disconnects
        ↓
Session is released
        ↓
harness scope halts
        ↓
pool, Tools, subscriptions, and child Operations unwind

Effection owns this behaviour.

There is no longer a live parent waiting for the result.

One Agent stops inside a live pool

Structure
pool remains alive
        ↓
policy decides this Agent should stop
        ↓
its branch remains temporarily available
        ↓
recovery may extract useful findings
        ↓
branch is pruned
        ↓
siblings continue

AgentPolicy and AgentPool own this behaviour.

The result may still be valuable to the active query.

Effection halt ends an ownership scope. Policy termination changes an Agent’s lifecycle inside a scope that still exists.

Do not use policy hooks to imitate scope cancellation. Do not treat Session cancellation as an ordinary policy stop.

2. See what the pool sees

The four-layer decision path

Structure
BranchStore
    owns branches and accounts for live KV cells
        ↓
ContextPressure
    freezes a consistent view of what remains
        ↓
AgentPolicy
    maps Agent state and pressure to an action
        ↓
AgentPool
    executes the action, batches native work, and preserves safety

BranchStore accounts

BranchStore tracks:

  • shared prefixes;
  • private branch suffixes;
  • branch positions;
  • cells consumed by generation and prefill;
  • cells reclaimed by pruning.

It does not decide whether the work is useful.

ContextPressure observes

ContextPressure derives stable facts for one decision boundary:

TypeScript
remaining = nCtx - cellsUsed;
headroom = remaining - softLimit;
critical = remaining < hardLimit;
percentAvailable = round(remaining / nCtx * 100);

It does not nudge, kill, retry, or recover an Agent.

AgentPolicy decides

Policy receives some combination of:

  • the Agent;
  • parsed model output;
  • a pending Tool result;
  • a pressure snapshot;
  • pool configuration;
  • policy state such as elapsed time.

It returns a declarative action.

AgentPool enforces

The pool:

  • advances branches;
  • commits sampled tokens;
  • dispatches Tools;
  • admits or defers prefill;
  • forces terminal grammar during recovery;
  • clamps report budgets to what fits;
  • salvages partial reports;
  • prunes branches;
  • emits lifecycle events.

Policy never bypasses those mechanics.

Pressure is a photograph, not a live gauge

The pool freezes ContextPressure at phase boundaries.

Every policy decision in that phase sees the same baseline.

Without a snapshot:

Structure
Agent A evaluated first
    sees more space

Agent B evaluated later
    sees less space

The same cohort could behave differently merely because an array was iterated in another order.

Instead:

Structure
phase begins
    ↓
freeze one pressure snapshot
    ↓
all policy decisions use that snapshot
    ↓
batch native mutations
    ↓
next phase receives a new snapshot

SETTLE maintains local admission accounting while building its batch because native cellsUsed updates only when the prefill is committed.

Reason from the supplied snapshot. Do not poll native pressure inside a hook and do not mutate it.

Every Agent spends live-attention budget

Each Agent spends from the Session’s finite live-attention budget through:

  • generated tokens;
  • Tool-result prefill;
  • recovery prompts and reports;
  • nested Agent work;
  • private suffixes across sibling branches.

Shared prefixes are amortised, but new private tokens still occupy live cells.

Pressure is not a static prompt-length limit. It changes as the cohort works.

3. Set the limits

The two limits are not symmetric

The most common mistake is treating softLimit and hardLimit as stronger and weaker forms of the same cutoff.

They have different jobs.

Soft limit: the new-work boundary

TypeScript
headroom = remaining - softLimit;

When headroom > 0, ordinary new work may be admitted.

When headroom <= 0, the system should contract:

  • do not spawn new research Agents;
  • defer oversized ordinary Tool results;
  • nudge non-terminal calls towards reporting;
  • narrow from exploration towards exploitation;
  • preserve capacity for downstream work and recovery.

The soft limit is advisory.

Crossing it does not automatically make decode unsafe.

Hard limit: where working turns end

TypeScript
critical = remaining < hardLimit;

The hard limit is mechanical:

  • an ordinary Agent must not continue into another working decode below it;
  • the pool stops the working turn and turns it into a report;
  • a report is budgeted from the room above the floor, the decode workspace the next batch needs, so the reserve between the hard line and the floor is the report’s to write into;
  • the configured value must sit at or above the floor, the context’s batch size.
Structure
used KV       ordinary headroom       soft-to-hard reserve      hard reserve     floor
█████████████████████████████████████░░░░░░░░░░░░░░░░░░░░░░░░░│▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒│
                                      ↑                         ↑                ↑
                                  soft limit                hard limit    batch workspace

Soft says “finish the work”. Hard says “you may not continue the work; write what you found into the space below”.

Raising softLimit does not reduce recovery to zero

A report uses:

Structure
room = remaining - floor

not:

Structure
headroom = remaining - softLimit

The floor is the context’s batch size, the one physical line. Raising softLimit nudges earlier, and raising hardLimit stops working turns earlier; neither shortens a report, which is sized, admitted and finished against the room. The scheduler walks the three lines in order.

4. Decide in the loop

The pool’s decision cycle

Structure
SPAWN + EXTEND
      ↓
PRODUCE
      ↓
COMMIT
      ↓
DRAIN
      ↓
SETTLE
      ↓
DISPATCH

SPAWN + EXTEND

  • admit queued Agents if they fit;
  • extend a shared spine for sequential workflows;
  • apply pending per-Agent cancellation.

PRODUCE

  • freeze a pressure snapshot;
  • ask policy whether an Agent must exit before generation;
  • sample one token for every active branch;
  • parse stop boundaries and Tool calls;
  • run the gates on the emitted call — the framework’s authorization, the tool’s own, the harness’s — before the policy is asked;
  • ask onProduced() what the output means.
  • for a call of the terminal tool, ask onReturn whether the turn may end, and what the result becomes.

COMMIT

  • commit the sampled cohort through BranchStore.

DRAIN

  • receive completed off-loop Tool executions;
  • return their post-processing to the single native loop;
  • ask afterExecute what each completion was — an attempt, a retry, a failure.

SETTLE

  • decide whether Tool results and recovery turns fit;
  • defer oversized ordinary work;
  • ask beforeAdmit when waiting cannot resolve the deferral — the tool’s hook, the harness’s, else drop;
  • batch prefill, book each admitted item and ask afterAdmit for a follow-up, then reactivate the admitted Agents.

DISPATCH

  • ask shouldExplore() how content-boundary Tools should score;
  • execute Tool calls;
  • ask afterExecute what an inline completion was; park a retry, settle a failure.

Policy is consulted at specific decision boundaries. It does not control the loop.

5. Save the work before you reclaim it

Recovery: save useful work before pruning

An Agent may stop without a voluntary terminal result because of:

  • context pressure;
  • elapsed time;
  • maximum turns;
  • graceful wind-down;
  • a policy hard exit.

Its branch may still contain useful evidence.

onRecovery() chooses:

TypeScript
type RecoveryAction =
  | { type: "extract"; prompt: PromptText }   // { systemPrompt, content } — your words, given the report's word budget
  | { type: "skip" };

Policy owns the meaning

Policy decides:

  • whether the work is worth extracting;
  • how to instruct the Agent to compress it;
  • whether recovery should be staggered or parallel;
  • an optional preferred report budget.

AgentPool owns the mechanics

The pool:

  • computes what safely fits;
  • forces the terminal Tool grammar;
  • prefills the recovery turn;
  • applies a token stop where bounded;
  • parses the terminal result;
  • salvages truncated output where possible;
  • emits recovery events;
  • prunes the branch.

Policy asks for the report. The pool guarantees that asking is mechanically safe.

Recovery is a real turn

Recovery is not string extraction from hidden state.

The pool injects a final instruction into the Agent’s live branch and constrains generation to the terminal result contract.

Why a report may write below the hard line

Near the soft boundary:

Structure
remaining ≈ softLimit

there may be almost no ordinary headroom:

Structure
headroom = remaining - softLimit ≈ 0

If recovery were budgeted only from headroom, the system would be unable to report exactly when reporting is required.

Instead:

Structure
recovery capacity = remaining - floor

Everything above the floor is available for converting unfinished work into a result: the soft-to-hard band and the hard reserve alike. The hard line is a working-turn line and plays no part in sizing a report.

The soft boundary means:

Stop opening ordinary work.

It does not mean:

No more recovery tokens may be decoded.

Choose what recovery costs

You set how much effort the pool spends saving work. recoveryShape picks between lossless and fast.

Staggered

Structure
recover A
    ↓
prune A
    ↓
recover B
    ↓
prune B

Properties:

  • one Agent at a time;
  • each report benefits from earlier pruning;
  • larger or uncapped reports;
  • maximum individual finding preservation;
  • blocks pool progress;
  • suited to high-effort, quality-first workflows.

Parallel

Structure
budget the cohort
        ↓
admit recovery turns
        ↓
decode them in the pool loop
        ↓
prune each as it finishes

Properties:

  • reports share recovery capacity;
  • report sizes are mechanically bounded;
  • better batching and latency;
  • suited to low/medium effort and graceful wind-down.

An explicit recoveryBudget caps a parallel recovery report and a voluntary report in flight; the pool clamps it to what safely fits. A staggered recovery has no cap: it ends at its stop token or when the room reaches the floor.

In-flight terminal-report salvage

If an Agent is force-stopped while already emitting its terminal Tool call, restarting recovery would:

  1. discard the partial report;
  2. inject another turn;
  3. consume additional KV;
  4. risk failing again in the same exhausted context.

The pool instead parses and salvages the existing partial terminal output without further decode.

This is a pool mechanic, not a policy hook.

Policy decides that the Agent should stop. The pool chooses the least-destructive safe route.

Three ways to end work

Wrap-up reaps every agent to recovery; cancelling one agent discards it with no recovery; a halt unwinds the scope with everything in it. How a harness sends each is on Agent policy.

6. Keep work in place: fast-forward

The problem it solves

Some work is worth keeping only if it succeeds: a user turn carrying a long video, a large tool result, a trial continuation. Writing it straight onto the branch that needs it is a gamble near a full context.

  • A failed prefill leaves a branch damaged. A long multimodal prefill is decoded in chunks. If one fails partway, the earlier chunks are already in the KV while the branch's position says they are not. On recurrent layers (Qwen3.5) the state cannot be rolled back by position at all. The only safe move is to prune the branch, and with it everything forked from it.
  • Keeping a child used to mean deleting everything else. prune refuses a parent with live children. promote and retainOnly keep one branch by releasing every other one. So a child's work could take its parent's place only when nothing else was running.

On a session's trunk, the first point means a failed media turn used to cost the whole conversation. Inside a pool, the second means one Agent could not keep a checkpoint without ending its siblings.

How it works

A fork is a checkpoint any branch can keep or discard while the rest of the tree runs on:

Structure
parent ──fork──→ child          the child shares the parent's cells; no copy
                   │
              work on the child
                   │
        ┌──────────┴──────────┐
        ↓                     ↓
parent.fastForward(child)   child.prune()
the parent takes the        the work is discarded;
child's state and keeps     the parent never saw it
its handle

parent.fastForward(child) is git's merge --ff-only. The parent takes everything that follows the child's tokens: its KV and position, its logits, its sampler and grammar, its metrics and logit bias. The parent keeps its handle, its place in the tree, its own children and its steer. The child is disposed.

Every reference to the parent stays valid: an SDK Branch, a session's trunk, a trace record. The parent's other children keep their cells.

TypeScript
const trial = parent.forkSync();
try {
  await trial.prefillMultimodal(prompt, inputs);
  parent.fastForwardSync(trial);   // keep it
} catch (e) {
  trial.pruneSync();               // discard it; the parent is as it was
  throw e;
}

Session.prefillUserMultimodal writes a warm turn this way whenever a sequence slot is free. A failed image or video turn then prunes the fork, and the conversation carries on. With no slot free, the turn is written in place, and a failure there still releases the trunk (see What it costs).

What it costs

  • No cells move. The child's new tokens were counted when they were decoded, and are not counted again when they are kept. The parent's own cells were shared with the child, so letting the parent's old sequence go frees nothing.
  • Recurrent state survives, because nothing is copied: the parent takes over a sequence whose state is already right.
  • One sequence slot per fork in flight. When no slot is free, Session.prefillUserMultimodal writes the turn in place instead, and a failure there still releases the trunk.

When it is refused

A fast-forward is refused, with nothing changed, unless all of these hold:

Condition Why
The child was forked from this parent Unrelated branches, siblings, grandchildren and the parent itself have no fast-forward
The parent has not decoded since the fork The child would overwrite tokens the parent wrote after it
The parent has not changed since the fork Accepting a token, setting logits, a grammar, sampler or bias, or reseeding changes state without moving position; each one bumps the branch's revision, and the child remembers the revision it was forked at
The child has no children of its own Its children would lose their parent
The child is not poisoned A prefill that failed partway marks its branch, and every fork of it, as poisoned; prune it instead

No native work may be in flight on either branch: await the child's work before keeping it, as before prune.

Fast-forward and pressure

A fast-forward changes no pressure fact: cellsUsed, remaining and headroom are the same before and after it. What it changes is what a failure near the limit costs. A prefill that overflows the context fails on the fork, and pruning the fork returns its cells; the parent, and every sibling, keeps working.

It does not make room. An Agent is spawned only when its task prompt fits above softLimit, so a turn that leaves the pool softLimit or less still starts a run that cannot spawn one. Refuse such a turn before it is written; for attachments, admitted does this with a reserve (see Attachments).

7. A worked pressure example

Assume:

Structure
nCtx      = 32,768
softLimit = 2,048
hardLimit =   512
cellsUsed = 29,500

Then:

Structure
remaining = 3,268
headroom  = 1,220
critical  = false

Ordinary work still has approximately 1,220 cells above the soft reserve.

Later:

Structure
cellsUsed = 30,900
remaining = 1,868
headroom  =  -180
critical  = false

The soft boundary is crossed:

  • stop opening research;
  • defer oversized ordinary results;
  • nudge towards reporting;
  • a report still has up to 1,868 - 512 = 1,356 cells of room above the floor (here the hard line sits on the floor, at the batch size).

Later:

Structure
cellsUsed = 32,300
remaining = 468
critical  = true

An ordinary Agent must not produce another token.

The pool stops it before decode and applies salvage or recovery.

headroom <= 0 and critical === true are not the same state.

Relationship to the programming model

Structured concurrency answers:

Who owns this work, and what happens when its owner ends?

AgentPolicy answers:

While the owner and AgentPool remain alive, what should this Agent do next?

Structure
Effection scope
    owns the pool
        ↓
AgentPool
    owns Agent execution and BranchStore mechanics
        ↓
ContextPressure
    presents stable resource facts
        ↓
AgentPolicy
    returns behavioural decisions
        ↓
AgentPool
    safely applies them

That separation allows harness developers to customise judgement under scarcity without rewriting lifecycle management or inference scheduling.

Return to Thinking in Lloyal.

↑
Search