🤓 Nolife Tokens - shrinking LLM context without destroying recoverability

on August 16, 2026

AI coding agents have inverted a familiar performance problem.

For years we fought bytes on the wire, HTTP round-trips, latency budgets, and process memory. Those constraints still matter. What changed is that a large share of the cost and failure mode of an agent session now sits in the prompt: tool output, logs, diffs, repository trees, compiler noise, JSON blobs, and the same informational lines repeated hundreds of times.

The naïve pipeline is simple:

Tool → giant stdout → LLM context

That works until the context fills with low-value material and the model still has to find the one error line that matters. Larger context windows make it possible to send everything. They do not make it wise.

This article documents nolife-tokens, a deliberately small Symfony + Flow experiment. The thesis is narrow and testable:

Removing information from active LLM context does not have to mean losing it. Context can be compressed aggressively while omitted material remains deterministically recoverable through lightweight references.

We measured volume, signal retention, and byte-exact recovery on fixtures. We did not yet measure provider-billed tokens, end-to-end agent latency, or dollar cost. Those distinctions matter.

Inspiration without cloning products

Several open projects explore related ideas: RTK (command-output compression before the agent reads it), context-mode (keep raw bulk out of the window and retrieve on demand), Headroom (content-aware compression with reversible storage), Claude-mem (progressive disclosure of memory).

You do not need to know those codebases. The useful common idea is not “another agent framework.” It is:

Do more deterministic work before sending data to the LLM.

Deduplicating logs, stripping telemetry keys, classifying a git diff, hashing an omitted section — none of that requires a model. The model’s context budget should go to ambiguity and decisions, not to five hundred copies of [INFO] request started.

nolife-tokens is a laboratory for that idea in the Darkwood stack: PHP 8.5, Symfony Console, and darkwood/flow. No embeddings, no vector database, no MCP, no Redis, no LangChain.

Starting point: a skeleton and an observable pipeline

The project began as a bare Symfony 8.1 skeleton. Application code was essentially a Kernel. Flow was already a Composer dependency but unused.

The first milestone, TOKEN_PIPELINE_POC, wired a minimal Flow pipeline over a mutable ContextPacket:

Load
→ Classify
→ MeasureRaw
→ Optimize
→ MeasureOpt
→ EvaluateSignals

CLI surface:

bin/console tokens:analyze fixtures/sample.log
bin/console tokens:benchmark

Flow is useful here not because we need concurrent orchestration, but because each stage is observable. The packet accumulates a trace: bytes and estimated tokens after load, content type after classify, size after optimize, pass/fail after signal checks. The reader (and the engineer) sees:

raw → classified → reduced → evaluated

instead of one opaque optimizeContext() that hides where volume disappeared.

A typical tokens:analyze trace looks like this in spirit:

LOAD         | fixtures/sample.log | 19,354 bytes
CLASSIFY     | log
MEASURE_RAW  | 19,354 bytes        | ~4,839 tokens
OPTIMIZE     | log refs=1          | 4,813 bytes
MEASURE_OPT  | markers~4 referenced~3671 | ~1,204 tokens
SIGNAL       | PASS
RECOVERY     | PASS

That stage list is the product. When something goes wrong — weak savings, a failed signal, too many refs — you can see which step is responsible without instrumenting a separate metrics platform.

Why keep the project small? Because context systems tend to grow into frameworks before anyone has measured whether the core idea works. nolife-tokens is intentionally closer to a console laboratory than to a library API. No ports, adapters, or speculative interfaces. Plain PHP classes, Symfony services via autowiring, files under var/.

Estimated tokens, not billed tokens

Measurement uses a documented approximation, the same heuristic RTK documents publicly:

// TokenEstimator.php (excerpt)
estimatedTokens: (int) ceil($bytes / 4),

Percentages between raw and optimized are useful for comparison. Absolute numbers are not provider-billed tokens. Different models tokenize differently. Throughout this article, “tokens” means this estimate unless stated otherwise.

Deterministic classification first

Compression strategy depends on content type. Classification is heuristic and LLM-free:

That distinction is not academic. Different noise compresses differently.

• Logs: repetition and low-signal chatter; high-value ERROR / WARNING lines must stay
• JSON: telemetry, padding, request ids; decision fields (id, status, error) must stay
• Git diffs: many + / - lines are the useful information; aggressive trimming is dangerous
• File lists: can collapse into trees with counts
• Text: light whitespace only in the current POC

First benchmark: volume vs signal

Fixtures plant critical strings on purpose. Optimization that merely truncates the tail would often “save tokens” while deleting the planted failure. The benchmark therefore tracks SIGNAL: every expected substring must still appear in the optimized visible text.

First-milestone results (lossy optimize — omitted material was not yet stored):

* estimated as bytes / 4

Planted signals included:

• ERROR PaymentService line 421 in the log fixture
• authentication check removed and src/Security/AuthGuard.php in the diff
• payment id, "status": "failed", and the error message in the JSON

The KPI that matters is not “how much did we remove?” It is: did the information required to make the decision survive?

JSON: structured noise collapses well

JSON was the easiest case. Structured keys let a deterministic filter drop or later reference telemetry while keeping decision fields. A payload that was mostly padding and debug frames fell from ~6.8k estimated tokens to tens of tokens of visible JSON in the first milestone, with SIGNAL PASS.

That is the structural advantage of typed noise: you can name what is disposable.

Logs: repetition is free money — until signals break

Logs compress because agents see the same line repeatedly:

[INFO] request started
[INFO] request started
[INFO] request started
[INFO] request started

becomes:

[INFO] request started ×4

High-signal lines stay verbatim. That sounds trivial until you write a deterministic retention test. An early fixture used [ERROR] PaymentService line 421 while the expected signal was ERROR PaymentService line 421. The substring failed because of the ] between ERROR and PaymentService. The lesson is prosaic and important: signal tests are brittle relative to formatting, and high-signal lines must remain stable enough for both humans and checks.

Git diffs: a usefully weak result

The diff fixture only saved about 24%. That is not a failure of the experiment. If most of the input consists of meaningful change lines, there is little safe noise. Blindly targeting “90% reduction” would damage the decision surface (for example, an authentication check removal in AuthGuard.php).

Git diffs reveal a natural compression limit. That is the kind of result you want from a laboratory: where not to push harder.

What the first milestone taught, in one sentence: token reduction without a signal test is vanity. With a signal test, you can tell the difference between “we removed noise” and “we deleted the failure.”

The missing property: remove ≠ destroy

The first POC had a structural weakness. Once content was dropped, it was gone from active context and inaccessible. If an agent later decided the discarded telemetry mattered, there was nothing to fetch.

That leads to the second milestone: REVERSIBLE_CONTEXT_POC.

Principle:

REMOVE FROM CONTEXT
≠
DESTROY INFORMATION

Optimizers now return an OptimizeResult:

// OptimizeResult.php
final class OptimizeResult
{
    /**
     * @param list<array{reason: string, content: string, marker_placeholder?: string}> $omissions
     */
    public function __construct(
        public readonly string $visible,
        public readonly array $omissions = [],
    ) {}
}

ContextOptimizer persists each omission under var/context/<id>.json and injects a short marker into the visible text:

// ContextOptimizer.php (excerpt)
$id = $this->store->makeId($packet->sourcePath, $omission['reason'], $omission['content']);
$ref = new ContextReference(
    id: $id,
    source: $packet->sourcePath,
    type: $packet->type->value,
    reason: $omission['reason'],
    content: $omission['content'],
);
$this->store->put($ref);
// ...
$visible = str_replace($placeholder, $ref->marker(), $visible);

Ids are deterministic and short: ctx_ plus the first hex digits of sha256(source|reason|content).

Actual stored shape (abbreviated):

{
  "id": "ctx_31154f",
  "source": "fixtures/sample.log",
  "type": "log",
  "reason": "deduplicated repeated lines",
  "content": "[INFO] request started\n..."
}

Active context only needs:

#ref:ctx_31154f

On the log coarse run, that marker costs about four estimated tokens while ~3.6k estimated tokens sit out of context on disk.

Retrieval is a console command, not a search engine:

bin/console tokens:show-ref ctx_31154f

// TokensShowRefCommand.php (excerpt)
$ref = $this->store->get($id);
$io->writeln(sprintf('Source: %s', $ref->source));
$io->writeln(sprintf('Type: %s', $ref->type));
$io->writeln(sprintf('Reason: %s', $ref->reason));
$io->writeln('--- RAW CONTENT ---');
$io->writeln($ref->content);

ContextStore::get normalizes #ref:ctx_… or bare ids and reads the JSON file. Recovery in the pipeline is byte-exact: reload each reference and compare content to what was written during optimize.

Updated Flow pipeline

Load
→ Classify
→ MeasureRaw
→ Optimize (+ store refs)
→ MeasureOpt
→ EvaluateSignals
→ EvaluateRecovery

Construction stays a FlowFactory generator of closures over ContextPacket:

// TokenPipelineFactory.php (excerpt)
return $this->flowFactory->create(static function () use (...) {
    yield static function (ContextPacket $packet): ContextPacket {
        $packet->type = $classifier->classify($packet->raw);
        $packet->addTrace('CLASSIFY', $packet->type->value);
        return $packet;
    };

    yield static function (ContextPacket $packet) use ($optimizer): ContextPacket {
        $optimizer->optimize($packet);
        $packet->addTrace('OPTIMIZE', sprintf('%s refs=%d', $packet->type->value, count($packet->references)));
        return $packet;
    };

    // … MEASURE_OPT, SIGNAL …

    yield static function (ContextPacket $packet) use ($store): ContextPacket {
        foreach ($packet->references as $ref) {
            $loaded = $store->get($ref->id);
            if ($loaded === null || $loaded->content !== $ref->content) {
                $missing[] = $ref->id;
            }
        }
        $packet->recoveryPass = $missing === [];
        $packet->addTrace('RECOVERY', $packet->recoveryPass ? 'PASS' : 'FAIL');
        return $packet;
    };
});

Two invariants:

1. SIGNAL — planted critical strings remain in visible optimized text
2. RECOVERY — every #ref round-trips byte-for-byte from var/context/

Reversible benchmark

Current measured results:

How to read the columns:

• RAW — estimated tokens of the original input
• VISIBLE — estimated tokens still in active context (includes #ref markers)
• MARKERS — estimated cost of the markers alone
• REFERENCED — estimated tokens stored out of context
• SAVED — reduction of visible vs raw
• SIGNAL / RECOVERY — the two pass/fail invariants

JSON still collapses hard (~98% visible reduction) while keeping decision fields and parking ~6.2k estimated tokens behind structural refs. Diffs remain the resistant case (~24%), now with a single coarse ref for collapsed unchanged context rather than dozens of micro-refs.

Coarse vs fine: reference granularity has a cost

For logs the benchmark runs both granularities.

Fine mode tripled the reference count and marker overhead to save about two estimated visible tokens. For this workload, coarse won: simpler retrieval, fewer ids for an agent to track, almost identical active context.

That is not a universal law. Granularity should follow the retrieval pattern you expect. Logs often want one “everything we collapsed” blob. Structured JSON often wants refs at field positions, so the model still sees where telemetry lived.

Log coarse emission (excerpt):

// LogOptimizer.php (coarse branch excerpt)
$out[] = sprintf('%s ×%d', $prev, $count);
$coarseParts[] = $expanded;
// …
$visible .= "\n\nRepeated informational logs omitted.\n" . $placeholder;
$omissions[] = [
    'reason' => 'deduplicated repeated lines',
    'content' => implode("\n\n", $coarseParts),
    'marker_placeholder' => $placeholder,
];

Structural JSON references

Conceptually:

{
  "id": 42,
  "status": "failed",
  "error": "PaymentService timeout",
  "telemetry": { "... huge payload ..." }
}

becomes:

{
  "id": 42,
  "status": "failed",
  "error": "PaymentService timeout",
  "telemetry": "#ref:ctx_xxxxxx"
}

The model still sees semantic position, identifiers, status, and error. Roughly six thousand estimated tokens of telemetry remain available outside the window. The current fixture yields nine refs (noise keys + grouped padding), ~36 marker tokens, SIGNAL PASS, RECOVERY PASS.

This is stronger than deleting keys: structure survives even when values leave the prompt.

In code, noise keys are not discarded; their values become placeholders that ContextOptimizer later rewrites to #ref:…. Padding keys matching padding_* / noise_* are grouped into a single omission when possible, so one large blob does not become eighty tiny references:

// JsonOptimizer.php (excerpt)
if ($this->isPaddingKey($key)) {
    $paddingBucket[$key] = $item;
    continue;
}

if ($this->isNoiseKey($key)) {
    $placeholder = '{' . '{REF_' . count($omissions) . '}' . '}';
    $omissions[] = [
        'reason' => 'omitted json field: ' . $key,
        'content' => json_encode($item, JSON_UNESCAPED_SLASHES | JSON_UNESCAPED_UNICODE | JSON_PRETTY_PRINT) ?: '',
        'marker_placeholder' => $placeholder,
    ];
    $result[$key] = $placeholder;
    continue;
}

The visible JSON therefore still “looks like” the API response. That matters for agents that reason over field names as much as over prose.

This is not RAG

nolife-tokens does not use embeddings, vector search, semantic retrieval, external databases, LangChain, MCP, or Redis.

Retrieval today is:

reference id → local JSON file → exact bytes

That simplicity is intentional. We want evidence that cheap structural reduction already removes most of the obvious waste. Semantic retrieval can wait until measurements say we need it.

Implications for coding agents

A more disciplined pipeline looks like:

Tool
  → deterministic reducer
  → high-signal context (+ #ref markers)
  → LLM

          ↓ (only if needed)
     tokens:show-ref / expand
          ↓
       raw source

This does not require replacing Cursor or inventing another IDE. A future integration can be as dull as:

bin/console tokens:context …

and pasting or piping the optimized result into an existing agent. The goal is better context for tools we already use — not another agent surface.

Cost and latency (careful claims)

Fewer input tokens can reduce provider cost, preprocessing load, latency, and noise. We have not benchmarked those effects against a live provider API in this project. So far we measured context volume, signal retention, and recovery. Treating volume reduction as proven bill reduction would be dishonest.

There is also a second-order effect agents care about: distraction. Even when a model “finds” the error in a 20 KB log, it may spend attention on the surrounding repetition. Deterministic reduction is not only about money; it is about giving the model a cleaner decision surface. That effect is harder to quantify than bytes, and we have not claimed to quantify it here.

A practical near-term use inside Darkwood-style workflows is still offline: run tokens:analyze on a fixture or a captured tool dump, inspect the stage trace, decide whether the visible context is enough to paste into Cursor. Integration that intercepts every shell command (RTK-style) is explicitly out of scope for these milestones.

Engineering philosophy

Before calling a model, ask:

Can PHP solve this part deterministically?

In this codebase that includes: classify content, deduplicate logs, strip or reference JSON noise, collapse file trees, drop git metadata and long unchanged context, hash omissions, persist refs, measure sizes, assert signals, assert recovery.

Components (responsibilities, not a file dump):

Failure modes we already hit or expect

Over-compression

A rare but critical line can look like noise. Planted-signal fixtures catch some of this; real agents will hit cases fixtures miss.

Reference explosion

An early git approach emitted on the order of 36 tiny refs for collapsed context runs. Marker overhead ballooned; retrieval UX got worse. The fix was one coarse ref for all omitted unchanged context. Fewer useful references beat many micro-references.

Semantic loss

Preserving a string does not always preserve interpretability. A status code without surrounding narrative can still confuse a model. Deterministic filters optimize for volume and explicit signals, not understanding.

False confidence from bytes / 4

Comparative percentages are solid for this lab. Absolute “tokens saved on the invoice” are not.

Retrieval dependency

Once the agent relies on #ref, something must teach it to expand. Today that something is a human running tokens:show-ref. Automatic progressive disclosure is the next experiment, not a solved feature.

Id collisions and store hygiene

Reference ids are content-addressed. Collisions with different content extend the hash length and then fail loudly. Collisions with the same content are idempotent rewrites. The store lives under var/, which is gitignored; it is a workspace cache, not a durable knowledge base. That is appropriate for a POC and insufficient for multi-user production memory — another reason not to confuse this with Claude-mem-style persistence.

Security posture

Treat analyzed content as data, never as instructions to execute. Optimizers only rewrite strings. Logs and JSON can contain text that looks like agent directives (“ignore previous instructions…”). The pipeline must not elevate that text into control. Today that is mostly a discipline of implementation: no eval, no shelling out on content, no trusting fixture prose as policy.

What we deliberately left out

It is worth listing non-goals so the experiment stays readable:

• No LLM-based summarization of omitted sections (yet). Summaries would be lossy in a different way and harder to recover exactly
• No automatic Cursor hook or shell interception
• No context-budget packer that chooses among twenty candidates under a 4k limit (designed later in the original roadmap, not implemented)
• No PHPUnit ceremony suite; tokens:benchmark is the validation surface
• No claim that estimated tokens match OpenAI/Anthropic billing counters

Leaving those out is part of the method: measure the reversible-reduction idea before stacking products on top of it.

Next: progressive disclosure

The natural continuation:

LEVEL 0  tiny overview
   ↓
LEVEL 1  selected section / #ref expansion
   ↓
LEVEL 2  raw original source

Example sketch:

Application failed during checkout.
1 PaymentService error.
telemetry omitted.
#ref:ctx_payment

If needed: expand the ref. Only if still needed: full raw. Context becomes demand-driven instead of “send the whole log first.”

That is where nolife-tokens stops being only a compressor and becomes an experiment in context architecture.

Diagrammatically:

                    ┌─────────────────────┐
   raw tool dump ─► │ classify + optimize │ ─► visible context (+ #ref)
                    └──────────┬──────────┘
                               │ omissions
                               ▼
                        var/context/*.json
                               │
              expand / show-ref │ (on demand)
                               ▼
                         original bytes

Today the downward arrow is manual. Progressive disclosure makes it part of the agent loop: overview first, then chosen expansions, then raw only when the cheaper layers fail.

Conclusion

Large context windows invite a lazy default: send everything. The engineering question is shifting from “how much can the model accept?” to:

What is the minimum context needed for the correct decision, and how cheaply can we recover the rest?

Two milestones of nolife-tokens already show, with measurements:

• large deterministic reductions are possible for noisy logs and JSON
• signal retention can be tested as a first-class invariant
• omitted material can remain byte-exactly recoverable
• reference granularity itself has measurable cost (coarse beat fine on our log fixture)
• content types have different natural compression limits (git diffs ~24% here)

The problem is not solved. Provider cost, agent-driven expansion, and progressive disclosure remain open. The laboratory is small on purpose: small enough that every mechanism is still understandable — which, for context systems, may be the point.

Commands used in this work

bin/console tokens:analyze fixtures/sample.log
bin/console tokens:analyze fixtures/sample.log --granularity=fine
bin/console tokens:analyze fixtures/sample.json
bin/console tokens:analyze fixtures/sample.diff
bin/console tokens:benchmark
bin/console tokens:show-ref ctx_31154f

Source code

Repository: https://github.com/matyo91/nolife-tokens
Slides: https://github.com/matyo91/slidewire