# Klaus Knowledge — Wikipedia Vector + Graph Corpus

> **What this is.** All of Simple English Wikipedia (~250k topics) as semantic
> vectors **and** a wikilink graph for drill-down, deduped, in **one sovereign
> store** (Neon Postgres + `pgvector` + an `edges` table). Ingestion runs on
> congosky.cloud (Cloudflare edge), **never locally**, and is driven by the four
> arms: **CLI + API + SkyBadger + skill**. Cite-or-refuse is preserved: the query
> API only ever *quotes* retrieved text, so it cannot hallucinate.

## The tiers (honest about what runs where)

| Tier | Store | Search | Network | Status |
|---|---|---|---|---|
| **0 — offline floor** | `klaus/corpus/wiki/*.json` shards | lexical (`klaus/engine/corpus.js`) | none | shipped |
| **device mid-tier** | expanded shards (graph-central top-N, deduped) | lexical | none | `build_device_tier.py` |
| **1 — online semantic** | **Neon `pgvector` + `edges`** | semantic KNN + graph drill-down | required | this build |

> **Why not offline semantic?** A true offline semantic tier needs an embedding
> model *on the phone* to vectorise the query — 20–90 MB even quantised, which
> blows the BadgerNet metered-low-end-phone budget. So the on-device mid-tier is
> an **expanded lexical** slice (works today, no model); offline semantic is a
> **named limit** with a follow-up (ship a tiny on-device embedder). See LIMITS.

## Why Neon (not Vectorize)

Doctrine: **EAT the core** (compute, DB, storage → sovereign substrate), rent only
the edge; **one record, never duplicated**. Vectors + graph + dedup metadata live
in **one Postgres**, re-homeable onto our own substrate later — not three rented
services. Embeddings come from Workers AI (rented edge), kept behind
`functions/_lib/embed.js` so that one dependency is swappable.

## Minimum footprint, maximum performance

The embedding column is ~89% of each row's footprint, so vector precision — not a
text codec — is the real lever. The store uses:

- **`halfvec(768)` (float16) embeddings** — ~2× smaller than `vector(768)` with
  negligible recall loss and faster KNN (verified: clean citations at ~0.75 cosine).
  Halving the dominant column ~halves the whole store (measured 4352→2248 kB / 498
  rows; projects ~1.1 GB vs ~2.2 GB at 250k). HNSW on `halfvec_cosine_ops`.
- **`lz4` TOAST on the `text` column** — free, faster decompress than pglz.

*Note:* the estate's "the420code / Lucid" is a knowledge-model doctrine
(one-record / delta-store), **not** a byte-compressor, and its engine lives outside
this repo — so it is not an applicable codec here. `halfvec` is the correct
pgvector-native realization of "compressed vector storage."

**Further, if footprint must shrink harder:** binary-quantize to `bit(768)` (96
bytes/vec, ~16–32× smaller) with a Hamming pre-filter + `halfvec` rerank — biggest
footprint cut, at the cost of a two-stage query and recall tuning. `halfvec` is the
recommended default (smaller *and* faster, no recall cost); binary is a deliberate
push toward minimum bytes.

## Data flow (all server-side)

```
MediaWiki allpages walk (resumable, apcontinue in Neon)
  → dedup  (content_sha exact + simhash near-dup within an LSH band)
  → embed  (Workers AI @cf/baai/bge-base-en-v1.5, 768-d)   [phone never embeds]
  → upsert docs(vector) + edges(wikilinks) into Neon
  → advance ingest_cursor  (one bounded batch per tick; resumes forever)
```
Query: `POST /klaus/ask {q}` → embed q → `pgvector` KNN → cite-or-refuse (τ floor)
→ graph drill-down (related topics via `edges`).

## The four arms

- **CLI** — `congo corpus status | ingest | resume | ask | seed | build-device-tier`
  ([cli/congo/services/corpus_svc.py](../cli/congo/services/corpus_svc.py))
- **API** — [functions/klaus/ask.js](../functions/klaus/ask.js) (query) +
  `functions/admin/corpus/{ingest,status,seed}.js` (guarded by `CORPUS_ADMIN_KEY`)
- **SkyBadger** — `IngestSkyBadger` watches progress / stalls / dedup / embed-lag
  ([cli/congo/services/skybadger/ingest_agent.py](../cli/congo/services/skybadger/ingest_agent.py))
- **Skill** — [.claude/skills/corpus-ingest](../.claude/skills/corpus-ingest/SKILL.md)

## Runbook

```bash
# 0. one-time: apply the schema (needs NEON_DATABASE_URL)
psql "$NEON_DATABASE_URL" -f corpus/schema.sql

# 1. Pages project bindings (dashboard or wrangler): AI, NEON_DATABASE_URL, CORPUS_ADMIN_KEY
# 2. drive the ingestion (edge does the work; this loop just ticks it)
congo config set corpus_admin_key <key>
congo corpus resume --batch 30          # runs to completion; resumable
congo corpus status                     # docs / embedded / edges / dedup rate
congo corpus ask "what is gravity?"     # cite + graph drill-down
# 3. after complete: build the offline mid-tier
congo corpus build-device-tier --top 4000
```

## Proof & demo (the honesty contract, made visible)

The one guarantee the online tier makes is **cite-or-refuse: every answer is a grounded citation
(a real source + cosine ≥ τ) or an honest refusal — never generated prose, so it cannot
hallucinate.** Two artifacts make that checkable, not just claimed:

- **Proof** — [`corpus/proof.py`](proof.py): a zero-dep script that hits the live `/klaus/ask` with a
  battery of answerable topics, nonsense/adversarial inputs, and a graph query, then asserts four
  invariants: *grounding* (every answer cited-or-refused), *recall sanity* (it really does answer),
  *anti-hallucination* (nonsense is **never** cited), and *graph drill-down*. Run it:
  ```sh
  python3 corpus/proof.py                # → PASS: cites or refuses; does not hallucinate
  ```
  Verified live 2026-07-13: grounding ✓ · 3/6 answerable cited · 4/4 nonsense refused · 6 graph
  neighbours. (Honest texture the proof surfaces: "atom"/"Sun"/"water" currently *refuse* on the
  partial ingest, and "Einstein" retrieves the neighbour *Elsa Einstein* — a real grounded citation,
  not a fabrication. Partial knowledge, zero invention.)

- **Demo** — [`klaus/knowledge/index.html`](../klaus/knowledge/) (live at **`/klaus/knowledge/`**): a
  frugal, brand-skinned page (imports `brand/tokens.css`; no framework, no build) where you ask a
  question and watch it either quote a sentence with its source + a **cosine-match meter** and
  **graph drill-down chips** (walk the wikilink graph), or show a first-class **honest refusal**. The
  refusal is styled as a result, not an error — because refusing to guess *is* the feature.

## Cost (FinOps — crosses a spend line; needs AJ sign-off before a live run)

- **Neon storage**: ~250k × 768-d `vector` ≈ ~0.75 GB + HNSW overhead → a few GB.
- **Workers AI**: ~250k one-time embed calls (ingest) + one per query thereafter.
- Both are metered. Estimate through `guard/finops_gate.py` and get sign-off; the
  ingestion must not start on autopilot. Bounded first (`--ticks`) to sample cost.

## Named limits (honest-status doctrine)

1. **Determinism**: the semantic tier is **not** byte-reproducible (embeddings
   aren't). Cite-or-refuse survives (no hallucination); the "byte-identical citation"
   guarantee does **not**. The lexical Tier-0 keeps its exact-parity contract.
2. **Offline semantic**: not shipped — needs an on-device embedder (mobile-budget).
3. **Near-dup dedup** is LSH-banded simhash (block-local), not global — a dup whose
   band differs can slip through; acceptable, and measured via `dedup_rate`.
4. **`sw.js` precache** lists a fixed shard set; growing the device tier's shard
   count requires updating it (or moving sw.js to read the manifest).