prism

PRISM stands for PageRank-Indexed, Symbol-aware Model. The problem it was built for is that a local model with a 4K or 16K context window cannot read your codebase, so something has to decide what it sees. Most tooling answers with embedding similarity. This answers with the call graph.

It runs against Ollama by default, with Anthropic available behind a flag, because the whole point was making small local models useful rather than making a large hosted one slightly cheaper.

The headline is that it did not work, in the specific sense that it lost to a much simpler baseline on the benchmark I built to test it. The architecture is below because I still think the parts are right, and the measurement is below that because it is the more useful half of the project.

Pass one: ranking a codebase with no model involved

Every run starts deterministically. Tree-sitter parses the source, a symbol relationship graph is built in NetworkX, and PageRank ranks it. No LLM is involved, the result is reproducible, and it is cached per file with hash invalidation.

Ranking by PageRank rather than by text similarity is the actual bet. A function that half the codebase calls is architecturally central whether or not its name resembles the task. Symbols land in three tiers by percentile: CORE is the top ten percent, SUPPORT the next thirty, PERIPHERAL the rest.

The task is not ignored. Keywords from the task description build a personalisation vector, so the same repository ranks differently depending on what you asked for:

personalization = self._build_personalization(graph, task_keywords or [])
pagerank_scores = nx.pagerank(
    graph, alpha=self.config.pagerank_alpha,
    personalization=personalization, max_iter=100, weight='weight',
)

Centrality decides the shape, the task tilts it. If PageRank fails to converge the ranker falls back to uniform scores and carries on, because pass one returning nothing is worse than pass one returning something unranked.

This part is fast and it works: about 50 to 200 files per second, roughly 100 ms to build and rank a thousand symbols, under five seconds for a 50,000-line codebase.

Pass two: keeping working memory bounded

The compact index goes to a model, which picks the symbols it wants to see in full. Those get fetched as token-budgeted source slices and fed into a synthesis loop.

The constraint that shaped this is that output can exceed the context window just as easily as input can. OutputLog keeps working memory proportional to the number of chunks rather than the size of everything produced so far: each synthesis call sees the compact index, the current slices, and prior log entries, never the full prior output. Documenting a large project therefore costs the same per call as documenting a small one.

Every budget decision uses tiktoken, with a four-characters-per-token approximation only as a fallback. Character counts are never used to decide whether something fits, because being wrong about that is how you discover a truncation at the far end of a long run.

There are two model roles rather than one. A lightweight reader model does context planning, section evaluation and fallback chunk reading; a stronger synthesis model does the actual output and the worker tool loops. On local presets both point at the same model, but the split means the cloud preset does not spend a frontier model on deciding which file to open.

The coding agent

On top of the two passes sits a session pipeline. A planner breaks a task into sections sized to roughly one context window of work. An evaluator scores them and splits any that are over budget, recursively, to a maximum depth of three. Worker agents then run a tool loop per section against ten tools, including apply_diff, grep_symbol, query_graph for callers and callees, and run_tests.

Shell access is allowlisted rather than open, and scoped to the project root.

The mechanism I like most is automatic test injection. After every file-mutating tool call the test suite runs against the changed files and the result is appended to the tool output, so the model gets a correctness signal on the next turn without having to think to ask for one. This later turned out to be both the most valuable idea in the project and the source of its worst bug.

Sessions are persisted to JSON after every completed section, so an interrupted run resumes rather than restarting, and interface changes in one section propagate as revision flags to sections downstream.

Then I measured it

Every claim above was architectural reasoning. The 365 unit tests mock the LLM, so they verify plumbing and say nothing about effectiveness. So I built an eval harness: a seed repository of 469 lines across seven modules, hidden-test scoring, and three arms. oneshot pastes the files into one prompt. toolloop is a plain agent loop of about forty lines with none of PRISM’s machinery. prism is the full pipeline.

At a 16K window, over eight tasks and 24 runs:

arm	solved	regressions	LLM calls	tokens
oneshot	4/8	0	1.0	1,742
toolloop	7/8	0	14.9	68,226
prism	4/8	1	10.8	46,293

The plain tool loop solved every task PRISM solved, plus three more. PRISM won zero tasks the tool loop lost, and produced the only regression in the whole run. Pasting the files into a single prompt matched PRISM’s score using 26 times fewer tokens.

The compression claim did not survive contact either. A 10 to 20 times reduction assumes long function bodies. On the benchmark repository the rendered index came to 7,380 tokens describing 3,346 tokens of source, which is a 2.2 times expansion, because per-symbol overhead exceeds short function bodies.

The caveat, and why I do not lean on it

The honest reading is that this benchmark is unfair to PRISM. The entire seed package is 3,346 tokens inside a 16,384-token window, about twenty percent of it. There was no context problem to solve, which is the only problem PRISM exists for, and the compression machinery fired zero times across eight runs. The table says do not reach for this on a small repository. It does not evaluate the design.

So I ran the regime it was built for, a 4K window, and the sign flipped: PRISM solved 2 of 3 where the tool loop solved 1 of 3, including both tasks PRISM had failed at 16K.

That is a hypothesis with weak support, not a finding, and it is worth being precise about why. It is one discordant pair, paired significance p=1.000. PRISM still spent 1.5 times the tokens and 2.1 times the calls. It burned thirty calls and 84,000 tokens failing a task it never solved. And run-to-run variance is large enough to swamp all of it: the tool loop solved one task and then failed the identical task twenty minutes later, same arm, same config, temperature 0.2.

What would actually settle it is written down rather than hand-waved: three repeats across eight or more tasks at the 4K window, a seed repository large enough that context is scarce at any window size, and an arm that keeps pass one but drops the planner, to separate the index’s contribution from its cost. Partial data suggests the index hurts at 16K, where it is larger than the source it describes.

Seven bugs a green suite could not see

Building the benchmark found seven production bugs on first contact, four of which had been silently disabling the auto-test feature entirely. The unit suite, all 365 tests of it, was green throughout.

The two worst are worth stating in full:

run_tests shelled out to bare pytest, so the project root never reached sys.path and any healthy repository without a src layout exited with an error. Worse, the automatic post-edit test ran pytest against the edited source file, which collects zero tests, and reported that to the worker as “Tests FAILED, fix the issue” after every single edit. The mechanism I described above as the best idea in the project had been actively lying to the model on every turn.

The rest are the same species. Neither tool loop passed temperature through, so worker loops ran at the provider default of 1.0 while plain completions ran at 0.2, making every arm comparison partly a comparison of sampling temperature. Unknown Ollama models were assumed unable to call tools and were silently routed into a weaker JSON fallback path. An incremental indexer compared OS-native paths against forward-slashed ones, so stale symbols accumulated on every re-index. One preset requested 4,096 completion tokens inside a 4,096-token window.

None of these are exotic. All of them are invisible to a test suite that mocks the model, because each one lives exactly at the boundary the mock replaces. The lesson I would carry to any agent project: a green unit suite that mocks the LLM tells you the plumbing is connected and nothing whatsoever about whether the agent works.

What I would keep

The eval harness, without hesitation. It is the part that generalises, it is the only reason I know any of the above, and it turned a project I was arguing for into a project I have numbers about. It also carries the ablation arms that matter more than the headline: one that drops auto-test, one that drops the index, one that gives the plain loop the index without the planner. Those are the questions worth answering, and they are the ones I would run first if I picked this up again.

The architecture I am less sure about, and that is the honest position rather than a modest one. Graph centrality is a better selection signal than embedding similarity for code, and I still believe that. Whether it is worth ten times the token cost of pasting the files in is a separate question, and on everything I have measured so far the answer is no.