An LLM-Wiki is a knowledge base where an LLM agent maintains a structured, interlinked collection of markdown pages between you and your raw sources. You curate, the agent writes, the wiki compounds. This page walks through what it is, how it compares to the alternatives, how it actually works, and how to start your own in under five minutes.
Most setups for asking an LLM about your documents work like this: index a pile of files, retrieve relevant chunks at query time, generate an answer, forget. Same question tomorrow → same retrieval, same answer, same forgetting. Nothing accumulates.
An LLM-Wiki flips that. Sources still arrive as raw files, but now an agent maintains a derived markdown layer in between — a structured, interlinked wiki it owns and updates. Each ingest leaves durable prose behind: a new concept page, an extended entity, a flagged contradiction. Future questions read what's already been thought through. The wiki compounds.
You curate sources. The agent reads them and writes the wiki. You read what the agent wrote.
Raw, immutable inputs — papers, transcripts, blog clippings, screenshots, your own notes.
You curate. The agent reads but never writes here.
Derived markdown — concepts, entities, syntheses, comparisons,
questions, plus index.md and log.md.
The agent owns and maintains it.
A single CLAUDE.md (or AGENTS.md) file —
page templates, ingest/query/lint workflows, style rules.
You + the agent co-evolve it.
An LLM-Wiki is a folder of files on your filesystem. Not an app, not a database, not a service. The folder sits wherever folders sit: a local directory, inside an Obsidian vault, in a git repo, synced via iCloud or Dropbox.
your-wiki/
├── CLAUDE.md ← schema (agents read this first)
├── README.md
├── sources/ ← Layer 1 — raw inputs (you curate)
├── wiki/ ← Layer 2 — agent-owned markdown
│ ├── index.md hand-curated entry points
│ ├── overview.md narrative tour
│ ├── log.md append-only change log
│ ├── concepts/ ideas, frameworks, terminology
│ ├── entities/ people, orgs, products, papers
│ ├── syntheses/ multi-source thematic essays
│ ├── comparisons/ side-by-side: A vs. B
│ └── questions/ open questions / FAQ
└── derived/ ← optional charts, decks, exports
[[wiki-links]] resolve natively; you get backlinks and a graph view of how pages connect.Three files do most of the structural work. The excerpts below are real — pulled from the wiki this page is part of.
The agent reads this on every open. Declares page templates, workflows, style rules. Co-evolves with use.
# CLAUDE.md — LLM Wiki Schema
> Pattern adapted from Karpathy's *llm-wiki* gist.
## TL;DR for the agent
You are the maintainer of a wiki about <your topic>. The user
curates sources in `sources/`. You compile them into pages under
`wiki/`. You never modify `sources/`. You always update `wiki/log.md`.
## Page templates
### Concept page (`wiki/concepts/<slug>.md`)
Opens with a blockquote tagline, then sections:
"What it is", "Why it matters", "Key ideas", "Related",
"Sources", "Continue reading". Cross-references use [[wiki-links]].
### Entity page
Same shape with: Summary, Key facts, Timeline, Related,
Sources, Continue reading.
(Plus templates for synthesis, comparison, question.)
## Workflows
### `ingest` — new file appeared in `sources/`
1. Read the source. Generate *.meta.md if missing.
2. Decide: extend an existing page, or warrant a new one?
3. Apply the change. Append to wiki/log.md.
4. Update wiki/index.md if a new top-level page was created.
### `lint` — periodic hygiene pass
Every page has blockquote tagline, ≥ 2 outgoing links,
## Sources, ## Continue reading. No orphans. No dead links.
## Style rules
- Short, plain sentences. One idea per page.
- Cite everything. No claim without a source link.
- Dates: ISO YYYY-MM-DD.
Hand-curated by the agent. Each entry has a one-line description so a reader can scan and dive in.
# Index
> A curated table of contents. Every link has a one-line
> description so you can scan and dive in.
If this is your first visit, start with [[overview]] for a
narrative tour.
## Concepts
- [[concepts/llm-as-judge]] — using an LLM to score outputs
- [[concepts/golden-dataset]] — the labelled benchmark you trust
- [[concepts/eval-design]] — what makes a benchmark useful
- [[concepts/evaluator-drift]] — when judges change their minds
- [[concepts/chain-of-thought-eval]] — scoring the reasoning, not just the answer
## Entities
- [[entities/hendrycks]] — author of MMLU
- [[entities/eleuther-ai]] — built lm-eval-harness
- [[entities/percy-liang]] — leads HELM at Stanford
## Syntheses
- [[syntheses/when-llm-judges-fail]] — patterns of judge failure
- [[syntheses/eval-vs-production-correlation]] — what really matters
## Open questions
- [[questions/how-to-eval-creativity]]
- [[questions/handle-evaluator-drift]]
Every ingest, lint, or schema change appends an entry. Audit trail for the wiki. Newest at the top.
# Change Log
Append-only. Newest at the top.
---
## 2026-05-18 — schema migration: human-friendly conventions
- updated: CLAUDE.md page templates
- one-liner is now a blockquote (> …) lead, not a labeled field
- **Status:** dropped from concept/entity/synthesis pages
- every page now ends with ## Continue reading footer
- migrated all 18 existing pages to the new format
## 2026-04-26 — ingest: hendrycks-mmlu-paper.pdf + helm-overview.md
- created concepts: golden-dataset, eval-design, llm-as-judge,
chain-of-thought-eval, evaluator-drift
- created entities: hendrycks, eleuther-ai, percy-liang
- created syntheses: when-llm-judges-fail
- queued questions: how-to-eval-creativity, handle-evaluator-drift
## 2026-04-26 — wiki initialized
- created: CLAUDE.md (schema, v1)
- created: scaffolding under wiki/ and sources/
- notes: ready to ingest sources.
Andrej Karpathy proposed the LLM-Wiki pattern as a public gist in April 2026. His framing:
"Obsidian is the IDE; the LLM is the programmer; the wiki is the codebase."
Karpathy positions the pattern as the resolution to a problem first posed in Vannevar Bush's "As We May Think" (1945) — the Memex idea — that has waited eighty years for a maintainer that doesn't get bored.
(Beer)—invented→(VSM).refutes isn't in it, you can't represent disagreement.[[wiki-links]] drawn in the prose.| RAG | Knowledge Graph | LLM-Wiki | |
|---|---|---|---|
| What gets stored | Vectors of raw chunks | Typed triples | Synthesized markdown pages |
| How retrieval works | Top-K nearest vectors | Structured query → graph traversal | Agent reads relevant pages directly |
| Reasoning at ingest? | No — just embed | Extract typed triples per schema | Yes — synthesis, cross-link, flag contradictions |
| Schema design upfront? | None | Heavy (formal ontology) | Light (CLAUDE.md; evolves) |
| Cross-references | None | Typed, structured | Untyped, prose-embedded |
| Auditability | Opaque — why this chunk? | Structured provenance | Cited markdown + dated log.md |
| Human-readable | No | With specialized tools | Yes — just markdown |
Variables: N = chunks (RAG) or sources/pages (wiki);
L = avg chunk/page length;
d = embedding dimension (~1536);
K = top-K retrieved or pages read;
T = triples; h = hops; LLM = forward-pass cost.
| RAG | Knowledge Graph | LLM-Wiki | |
|---|---|---|---|
| Ingest per source | O(L) embed + O(d · log N) insert | O(L · LLM) extract triples | O((L + K·L) · LLM) synthesis |
| One-time setup | None | Heavy: ontology (human-weeks) | Light: CLAUDE.md (human-hours) |
| Query | O(d · log N) + O(K · L · LLM) | O(h · log T · branching) — no LLM | O(K · L · LLM) |
| Storage | O(N · d) — typically GB | O(T) — typically MB–GB | O(N · L) — typically MB |
| Calibration | 1M chunks at d=1536 → ~6 GB; query ~1 s | 1M triples → ~50–500 MB; query 10–100 ms | 100 pages → ~1 MB; query 1–3 s |
| Scales gracefully to | N ≈ 10⁸+ | T ≈ 10⁹+ | N ≈ 10² – 10³ |
The wiki has a small, fixed vocabulary of things you ask the agent to do. Each is a verb that operates on the same folder.
A new source appeared in sources/.
The agent reads it, generates a *.meta.md sibling if
missing, decides whether to extend an existing page or create a new
one, makes the change, and appends to log.md.
You ask a question.
The agent searches wiki/ first, falls back to
sources/, and surfaces gaps as new
questions/ entries. Always cites.
Periodic hygiene pass.
Checks: every page has a tagline, ≥ 2 outgoing links, a
## Sources block, a ## Continue reading
footer; no orphans; no dead links; log.md is dated.
The escape hatch.
Delete wiki/, re-derive every page from
sources/ + CLAUDE.md. A reproducibility
check — proves your sources can fully regenerate the wiki.
CLAUDE.md (or AGENTS.md) is the file the agent
reads first. It defines what counts as a thing, what a page looks like,
and how the operations work. Here's the concept-page template from the
starter:
# <Concept Name>
> <single-sentence summary that doubles as a tagline>
## What it is
<2–6 sentence explanation, plain English first>
## Why it matters
<the practical or theoretical consequence>
## Key ideas
- bullet
- bullet
## Related
- [[concepts/<other>]]
- [[entities/<paper-or-person>]]
## Sources
- [[sources/<file>]] — <one-line note>
## Continue reading
- **<short reader-facing label>** → [[<target-page>]]
- **<short reader-facing label>** → [[<target-page>]]
There are five page templates total — concept, entity, synthesis, comparison, question — each with its own shape. The schema also declares style rules (kebab-case filenames, ISO dates, no claim without a source), forbidden behaviors (don't edit sources, don't invent citations), and the four operations.
After your first ingest, you'll notice things you'd have done differently
— a page split where you'd have merged, the tone is off, a template
doesn't fit your domain. Don't fix the wiki by hand.
Edit CLAUDE.md instead, ask the agent to re-ingest, and the
next pass produces what you wanted.
The schema is the contract; the wiki is its output. Iterating on the schema is how a generic template becomes your wiki.
This is what an agent renders, page by page — clean markdown with a blockquote tagline, structured sections, cross-links, citations, and a navigational footer.
Using a large language model to score the outputs of other models — fast, cheap, and shockingly correlated with human raters when designed well.
The technique of asking an LLM (often a stronger or more expensive one than the model being evaluated) to grade candidate outputs against criteria you specify. Replaces or supplements human rating in production eval pipelines.
Production LLMs need continuous eval, and human rating doesn't scale. A well-designed LLM-judge correlates with humans at roughly ~1% of the cost, enabling daily eval runs you'd never afford otherwise. The golden-dataset is the bedrock; the judge is the multiplier.
Every page follows this shape. The agent writes them; you read them; the cross-links form the graph you see in Obsidian.
LLM-Wikis win in a specific regime: a corpus you actually care about (not too big), questions that recur (not one-off), and an audience that includes humans. Below are the use cases I've seen pay off, grouped by who they serve.
A topic you're learning, mastering, or thinking through over weeks or months.
Papers in a domain, synthesized as you go.
Sources: papers, conference talks, blog posts
Sample pages: concepts for techniques, entities for papers and authors, syntheses across themes
A topic you're mastering across textbooks and papers.
Sources: textbooks, papers, blog series, YouTube talks
Sample pages: concept pages for ideas, entity pages for key figures, open questions
New job, new stack, new sub-field. The wiki becomes your ramp-up artifact.
Sources: company docs, runbooks, Slack threads
Sample pages: concepts for jargon, entities for systems and people, syntheses for "how we do X"
A hobby with a literature: climbing, fermentation, tea, Go.
Sources: books, blog posts, podcast notes, your own experiments
Sample pages: techniques, gear and equipment, people in the field
The pattern as a context layer for an AI-augmented data or engineering team.
The team's shared understanding of what this thing actually is.
Sources: PRDs, design docs, customer interviews
Pages: business concepts, system entities, decision syntheses
Runbooks, alert response, on-call procedures.
Sources: alert configs, postmortems, on-call notes
Pages: per-alert playbook, per-system runbook, cross-cutting "common failure modes"
Postmortems become sources; the wiki finds patterns across them.
Sources: postmortem docs
Pages: per-incident, per-system, recurring failure modes
Every experiment is a source; the wiki synthesizes "what we've learned."
Sources: experiment writeups, eval reports, weights / metrics
Pages: per-dataset, per-architecture, lessons-learned syntheses
Proposals are sources; the wiki tracks themes, reversals, contradictions.
Sources: ADR docs, RFCs
Pages: per-decision, themed syntheses, contradictions
Tickets, interviews, NPS → a living "why customers ask about X."
Sources: support tickets, interview notes, survey responses
Pages: per-feature, per-complaint, per-segment
The LLM-Wiki pattern doesn't care how many wikis you have. The choice is one of relationship: are your wikis independent, or are they sub-topics of a larger theme?
Different domains that don't reference each other. E.g., your work wiki, a hobby wiki, a side-project wiki.
Setup: each is its own Obsidian vault.
Linking: none across vaults — switch with Obsidian's vault picker.
Graph: one per vault — they don't see each other.
Multiple narrow wikis that share a broader theme. E.g., an LLM Wiki vault containing Ontology, RAG, and Distributed systems sub-wikis.
Setup: one Obsidian vault with multiple sub-wiki folders, each with its own CLAUDE.md.
Linking: [[wiki-links]] resolve across sub-wikis natively.
Graph: unified — color-coded by sub-wiki folder.
The most common case — one vault, multiple sub-wikis, with a hand-curated README.md at the top:
LLM Wiki/ ← Obsidian vault root
├── .obsidian/ ← vault config (colors, plugins)
├── README.md ← hand-curated hub: "what's here"
├── Ontology/ ← sub-wiki 1
│ ├── CLAUDE.md
│ ├── sources/
│ └── wiki/
├── RAG/ ← sub-wiki 2
│ ├── CLAUDE.md
│ ├── sources/
│ └── wiki/
└── Distributed-Systems/ ← sub-wiki 3
├── CLAUDE.md
├── sources/
└── wiki/
Three ways to cross-link inside this structure:
[[retrieval-augmented-generation]] — Obsidian finds it anywhere in the vault.[[../RAG/wiki/concepts/retrieval-augmented-generation]] — unambiguous, robust against renames.README.md lists each sub-wiki with a one-line description.cd into a sub-wiki folder, the agent reads that sub-wiki's CLAUDE.md. Each sub-wiki has its own independent ingest / query / lint loop. Sub-wikis don't talk to each other agentically — only through Obsidian's link resolution.
The wiki isn't always the right tool. Where it loses:
(> ~1000 sources). Vector RAG scales better — synthesis-at-ingest gets expensive at that scale.
"Find me the doc that says X." Wiki transformations get in the way of raw retrieval.
Real-time prices, logs, telemetry. Wikis are for slow-changing knowledge.
Content needing exact-quote provenance. The transformation step complicates audit.
Concretely: what does an ML team's "experiment log" wiki look like after a few months of use?
sources/
├── 2026-04-15-baseline-finetune.md
├── 2026-04-22-lr-sweep.md
├── 2026-05-03-data-cleaning.md
├── 2026-05-10-attention-ablation.md
├── 2026-05-18-eval-redesign.md
└── 2026-05-24-instruction-tuning.md
wiki/
├── concepts/
│ ├── learning-rate-tuning.md
│ ├── data-quality-effects.md
│ └── attention-head-pruning.md
├── entities/
│ └── datasets/training-set-v2.md
├── syntheses/
│ └── what-actually-moves-the-needle.md
├── comparisons/
│ └── full-ft-vs-lora.md
└── questions/
└── why-did-attention-ablation-help.md
Instead of six disconnected experiment writeups in Notion that nobody re-reads, the team gets a navigable wiki where "what we've learned about X" is a single page with citations back to the underlying experiments. New hires read the wiki; experienced team members keep adding sources; the synthesis is always current.
A working LLM-Wiki needs three things: a markdown editor (for reading), an LLM agent (for writing), and the template repo (for the schema). All three are free and take a couple of minutes each.
Any agent that reads a project-level instructions file:
The agent reads CLAUDE.md on open and behaves accordingly.
Obsidian (recommended): native [[wiki-links]], backlinks, graph view.
VS Code, vim, or any markdown reader works for individual pages.
Optional: git, if you want a version history of how the wiki evolved.
git clone https://github.com/hong-chu/llm-wiki-starter.git my-wiki
cd my-wiki
Narrow enough that ~10 sources can cover it meaningfully. "Retrieval-augmented generation" works; "machine learning" does not. Good directions: a research area you're learning, your team's runbooks, a hobby with a literature, a sub-domain at work.
CLAUDE.md
Open CLAUDE.md and replace the TL;DR for the
agent block with one sentence describing your topic. This is
the only edit required to make the template yours; everything else
in the schema you can iterate on after the first ingest.
sources/
PDFs, blog posts pasted as .md, transcripts, your own
notes — anything you've already engaged with and want to remember.
The agent will read each one on the first ingest.
ingest
The agent reads sources/, decides what pages to
create, and starts filling out wiki/. A first ingest
of 2–3 sources typically produces 5–10 wiki pages. Read what came
out — some will be exactly right; some will surprise you.
A folder you own, full of markdown pages an agent wrote for you, cross-linked into a navigable graph you can browse in Obsidian. You add sources over time; the wiki compounds. The schema co-evolves with your taste.
Specifically, after a few sessions:
CLAUDE.md tuned to your domainIf you'd rather build a wiki from scratch without cloning, open Claude Code in an empty folder and paste this prompt:
Scaffold an LLM-wiki for the topic <YOUR TOPIC>, following the
pattern in Andrej Karpathy's gist
(https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f).
Create:
- CLAUDE.md (the schema, with page templates for concepts/entities/
syntheses/comparisons/questions and ingest/query/lint workflows)
- An empty sources/ directory I'll fill in
- An empty wiki/ tree: concepts/, entities/, syntheses/, comparisons/,
questions/, plus index.md, overview.md, log.md
Use kebab-case.md filenames, [[wiki-links]] for cross-references, and
end every page with a ## Sources section.
The agent will create the structure from scratch. The result should look like the directory tree from Tab 01.
Disciplines that separate "neat trick" from "actual living wiki." These come from running the pattern repeatedly and watching what fails.
The single biggest predictor of wiki health is topic narrowness. Most "my wiki isn't working" stories trace back to a topic that's too broad.
Each good topic is a narrow slice of the corresponding bad one. Same row → same domain, different scope.
You can name 5–10 sources you'd actually want to remember.
The agent can't decide what belongs — wiki sprawls and decays.
The first ingest sets the tone for everything that follows. Treat it like a calibration step, not the finish line.
CLAUDE.md before adding more sourcesThis list is exactly what lint checks. Internalize it and your wiki passes lint by default.
[[wiki-links]] — the page connects to neighbors## Sources section with citations back to sources/## Continue reading footer with 2–3 curated next-page linksllm-as-judge.md, not LLM_as_Judge.mdThe single most-important discipline. Master this and the wiki maintains itself.
CLAUDE.md, not the page. The schema is the contract; the wiki is its output. Hand-editing the output is fighting your own agent.
Workflow when a page disappoints:
CLAUDE.md to fix the rule
The agent doesn't change CLAUDE.md on its own. That part stays human-initiated — the contract is yours.
*.meta.md sibling with title, author, date, why-it-matters.YYYY-MM-DD-short-slug.ext (e.g., 2026-04-15-mmlu-paper.pdf).sources/ but never writes there. Period.Run lint weekly or after each major ingest. Two flavors of things it should catch:
index.md in ≤ 2 hops)[[wiki-links]] (target doesn't exist)**TODO** banner)## Sourceslog.md drift (last entry > 2 weeks old)The six most common ways people break their own wiki:
Using the pattern for one-off questions or unbounded topics. Use it for specific, bounded, recurring domains only.
You fight your own agent. Edit CLAUDE.md and re-ingest instead.
Letting your wiki live inside Notion AI or ChatGPT memory. Keep it as portable markdown.
Every change should append to log.md. Otherwise the audit trail breaks and trust evaporates.
When sources disagree, capture both in a ## Contradictions section. Don't silently pick one or split the difference.
Trusting the first ingest blindly. You must read what the agent wrote and iterate the schema — otherwise mediocre output becomes "fact."
Optional — for the curious. Two frameworks from outside the LLM world (philosophy and cybernetics) that sharpen how you think about what an LLM-Wiki is and what it's missing.
Ontology is the branch of philosophy that asks "what exists?". Five canonical answers have been proposed over the last 2,500 years. They aren't steps in a hierarchy — they're rival frames, each catching what the others miss.
| Lens | What it sees | Wiki reading |
|---|---|---|
| Substance | Things with properties (Aristotle) | Files, the vault, the markdown itself |
| Process | Events, flow (Whitehead, Heraclitus) | Ingest, query, lint — the live activity |
| Relational | Connections, networks (Leibniz) | The link graph between pages |
| Informational | Patterns of distinction (Wheeler) | CLAUDE.md, schemas, conventions |
| Phenomenological | What shows up for a being (Heidegger) | You — your taste, your refusal, what you care about |
The river metaphor: water (substance), flow (process), the network of banks and rain (relational), the pattern of differences (informational), the river as it shows up to someone on the bank (phenomenological). All five descriptions are true; none is sufficient alone.
Stafford Beer (1926–2002) was a British cybernetician. He asked a different question: "what does any system — a cell, a team, a nation — need to stay alive in its environment?" His answer is the Viable System Model: six interacting functions. Drop any one and the system dies in a specific, recognizable way.
| Layer | Function | Kitchen analogy |
|---|---|---|
| S1 Operations | Doing the work | Line cooks cooking |
| S2 Coordination | Preventing collisions | The expediter |
| S3 Regulation | Optimizing current service | Head chef adjusting on the fly |
| S3* Audit | Independent inspection | Food-safety inspector walking through |
| S4 Scanning | Watching the world | GM tracking trends, weather, reservations |
| S5 Identity | What this system is for | The owner's vision for the restaurant |
Run LLM-Wiki through the framework. What does it cover, and what does it miss?
| Layer | What it would do in a wiki | LLM-Wiki coverage |
|---|---|---|
| S1 Operations | Ingest, query, lint | ✓ Present |
| S2 Coordination | index.md, schema conventions | ✓ Present |
| S3 Regulation | The lint operation, schema-as-the-loop | ✓ Present |
| S3* Audit | log.md, citations on every page | ✓ Present |
| S4 Scanning | Proactively watching the world for new sources | ✗ Absent by design |
| S5 Identity | What this wiki is for, what it refuses | ⚠️ Implicit only |
LLM-Wiki gives you S1–S3* out of the box. To build a fuller system, two layers need to be added.
LLM-Wiki doesn't watch the world. Sources are dropped in by hand. To add S4:
Tools like OpenClaw bring S4. Claude Code by itself is reactive — it waits for you.
LLM-Wiki's S5 is hidden inside CLAUDE.md and your curation choices. To make it explicit:
identity.md at the rootWithout explicit S5, the wiki drifts: capable but directionless.
Removing VSM layers from the top down produces a hierarchy of autonomy. A practical framing for evaluating any AI infra project, not just wikis:
| Level | What's lost | State |
|---|---|---|
| L4 | — | All layers present · full cybernetic autonomy |
| L3 | – S5 | Capable but drifts · no direction |
| L2 | – S5, S4, S3* | No outside sense, no self-inspection |
| L1 | – S5, S4, S3*, S3, S2 | Uncoordinated execution |
| L0 | Only S1 remnants | Not a system · just actions |
Most "AI + notes" stacks today sit at L2–L3. LLM-Wiki out of the box is roughly L3: S1–S3* present, but no S4 watchers and an implicit S5. Adding watchers and an explicit identity.md lifts you to L4.
identity.md and no scheduled watchers, your system is structurally L3 — capability without direction. Lots of energy, no compass.
Putting the lenses together: a complete Personal OS isn't LLM-Wiki alone. It's a composition of four pieces, each picked at its strongest.
identity.md on top — makes S5 first-classIf this lens resonates, the worked example is the Ontology vault this page was generated from. It contains real pages on:
concepts/ontology — the five lenses, deepenedconcepts/viable-system-model — Beer's framework with the kitchen analogy and the autonomy ladderconcepts/triad — the human-AI-fossil architectureconcepts/personal-os — the composite stacksyntheses/personal-os-composite-architecture — the full proposalquestions/what-belongs-in-identity-md — the open thread on S5Read those pages, then come back to your own wiki and ask: what's my S5? What's my S4? Which lens am I underusing? Those answers sharpen the pattern from "neat trick" to "real architecture."