Context Foundry — Architecture

Goals and boundaries

Local first

Documents, Qdrant storage, embeddings, and responses stay on the developer machine with the default settings. Hosted providers are explicit opt-ins.

Understandable

The RAG loop is plain Python: load, chunk, embed, retrieve, assemble context, generate. No hidden agent graph.

Replaceable edges

Small protocols isolate chat and embeddings. Each can use a different provider without changing ingestion or retrieval.

Portfolio ready

A web UI, OpenAPI endpoints, architecture notes, tests, CI, releases, and observable source scores demonstrate the whole system.

Not production SaaS

No accounts, tenancy, distributed workers, or cloud database. These only belong after usage creates a real need.

Zero required spend

Ollama, Qdrant local mode, Python libraries, GitHub Actions, and GitHub Pages are available without a paid runtime.

System context

Userbrowser or API client

FastAPI appworkspace, source lifecycle, retrieve, query, health

Local data and modelsOllama, embedded Qdrant, retained source files

Optional hosted checksTypeSafe Jev receives claims and selected excerpts only when enabled

The FastAPI process owns orchestration and the browser workspace. Ollama owns model inference. Qdrant runs in-process, while vectors and retained originals persist beneath RAG_DATA_DIR; no database server or Docker container is required. The Jev edge is inactive by default.

Browser workspace

The interface keeps the question, grounded answer, and exact source passage in one reading flow. It exposes system state early and moves implementation detail behind deliberate disclosure.

Context Foundry showing a cited answer, Jev support check, and source passage from the NIST AI Risk Management Framework

Evidence stays in view

The document library controls query scope, numbered citations activate the matching source, and the source header distinguishes cited passages from the complete retrieved set. PDF citations open the retained original on the cited page.

Reading comes first

The question uses an editorial serif face, answer copy has a restrained reading measure, and labels describe outcomes in plain language. Copy and source actions sit beside the content they affect.

Trust before detail

The header shows whether Jev is off, observing, or repairing. Inline claim states and the audit summary communicate the result first; probability bars, repair attempts, privacy disclosure, and the technical record remain available on demand.

Progress and recovery

Staged status text explains long local-model work, an active request can be stopped, and a failed or cancelled question leaves the previous answer intact. The three-column layout collapses into a usable mobile reading order.

Three explicit pipelines

Ingestion

Validate file size and supported extension.
Extract text from PDF, Markdown, or plain text.
Preserve Markdown headings and split on natural boundaries with overlap.
Embed chunks in batches with the configured embedding provider.
Create or validate the Qdrant collection dimension.
Retain the original under a content-addressed ID and record the complete page count.
Replace stale chunks and source files for the same filename, then store citation metadata.

Question answering

Split clear compound questions into at most three self-contained retrieval parts.
Embed each retrieval question with the same embedding provider.
Rank Qdrant cosine and BM25 results independently, optionally scoped to selected documents.
Fuse both ranked lists with reciprocal-rank fusion; the threshold applies only to dense results.
Select concise supporting evidence, searching deeper for exhaustive requests.
Return explicit source lists extractively; validate synthesized claims against a JSON schema and source terms.
Fall back to the most relevant cited paragraph when local-model drafts remain ungrounded.
Optionally audit cited support with Jev and replace a weak citation only after a qualifying fresh check.
Render Markdown and citation markers deterministically, then return the full retrieved proof.

Evaluation data

Generate optional direct-fact synthetic silver candidates from stable source evidence.
Validate source support, answer support, schema integrity, and duplicate questions.
Review the source first, then approve, edit, or reject each candidate.
Freeze only approved_gold cases into a versioned release dataset.
Run deterministic retrieval, grounding, citation, and abstention gates; use Ragas only as a secondary diagnostic.

Technology choices

Concern	Choice	Why it fits	Alternative / trigger
Runtime	Python 3.13 + uv	Typed, cross-platform, reproducible lockfile, fast project commands.	Python 3.12 remains compatible if a dependency requires it.
HTTP/UI	FastAPI + vanilla HTML/CSS/JS	OpenAPI for free and one small browser client without a Node toolchain.	Add a frontend framework only when client state becomes complex.
Chat	Ollama `/api/chat`	Local inference, no token bill, simple documented JSON API.	OpenAI-compatible adapter for another local or hosted endpoint.
Embeddings	Ollama `/api/embed` + `embeddinggemma`	Batch inputs, local execution, and independent model configuration.	Swap only the embedding provider and rebuild the collection.
Vector database	Qdrant client local mode	Persistent vector search with the same client API available for a later server migration.	Run Qdrant server when concurrent writers or remote access matter.
Retrieval	Cosine + BM25Plus + RRF	Combines semantic and exact-term ranks without mixing incompatible raw score scales.	Add a cross-encoder only after a frozen dataset proves the gain.
Document parsing	PyMuPDF4LLM + RapidOCR	Extracts layout-aware Markdown and automatically OCRs image-only pages locally.	Add language models only when non-English scans are required.
Claim audit	Optional TypeSafe Jev	Checks each generated claim against its cited excerpts and can verify a replacement citation.	Keep disabled for a fully local path; calibrate thresholds before enforcement.
Model transport	HTTPX	One async HTTP client covers Ollama and OpenAI-compatible endpoints.	Provider SDK only when it exposes required behavior missing from HTTP.
Evaluation	Deterministic gates + optional Ragas	Human-approved gold drives repeatable release metrics; local judges add semantic diagnostics.	Use a stronger local judge when hardware permits.
Quality	Ruff, mypy, pytest, nox, pre-commit	Fast local checks mirror CI; configuration lives in `pyproject.toml`.	None until project scale justifies more specialized tooling.

Provider seams

Swappability is two structural typing contracts, not a dependency-injection framework. Configuration selects one implementation for each contract at startup.

class EmbeddingProvider(Protocol):
    async def embed(self, texts: list[str]) -> list[list[float]]: ...

class ChatProvider(Protocol):
    async def answer(self, question: str, context: str,
                     citation_count: int) -> GroundedResponse: ...

Default

RAG_EMBEDDING_PROVIDER=ollama
RAG_CHAT_PROVIDER=ollama
Uses independently configured models against the local Ollama base URL.

Optional

openai-compatible targets any compatible endpoint. Base URL, model, and API key are environment settings; neither OpenAI nor a paid account is required.

Data and retrieval model

Vector

The embedding is stored with cosine distance. Collection dimensionality is discovered from the first embedding and checked thereafter.

Payload

document_id, source_sha256, source, page, page_count, chunk_index, and the actual chunk text support stable evidence anchors, citations, hybrid retrieval, document filtering, deletion, and replacement on re-ingestion.

Original files

Uploads are retained beneath RAG_DATA_DIR/sources/<collection-hash> by content-addressed document ID. The UI can download a source or render a PDF page without re-uploading it.

Hybrid rank

BM25Plus ranks stored chunk text while Qdrant ranks embeddings. Reciprocal-rank fusion combines positions, not incompatible raw score magnitudes.

Chunking

PyMuPDF4LLM preserves PDF layout as Markdown; RapidOCR handles scanned pages locally; LangChain's standalone splitter preserves headings and natural text boundaries while enforcing configurable character limits.

Consistency rule

Changing the embedding model or provider requires rebuilding the collection. Mixing vector spaces produces invalid similarity results.

Runtime, privacy, and failure behavior

Area	Design
Configuration	Environment variables with safe local defaults; `.env` is ignored and `.env.example` documents every setting.
Health	`/health` reports readiness, provider choices, Jev mode, upload limit, and whether both model endpoints are local without exposing keys or document contents.
Privacy	The default path sends no document data to a third party. A remote model endpoint or Jev mode is an explicit operator decision; Jev receives generated claims and selected excerpt text, never retained originals or filenames.
Prompt safety	Retrieved documents are untrusted data. The system prompt tells the model to ignore instructions inside context and answer only from evidence.
Input boundary	Extensions and upload size are validated; filenames are metadata only and never used as filesystem destinations. Original downloads and PDF previews use non-cacheable, same-origin endpoints.
Failure modes	Unavailable models, malformed documents, dimension changes, invalid structured output, and empty retrieval return actionable API errors.
Quality signals	The UI reports cited and retrieved counts, source scores, answer latency, and optional claim-support distributions; deterministic JSONL metrics gate releases while local Ragas judges remain diagnostic.
Backup	Stop the app and copy `RAG_DATA_DIR`, including its `sources` subtree. Keep external originals when a rebuild path independent of application storage is required.

Delivery model

feature/*branch from dev

→

devdefault integration branch

→

mainrelease-ready history

Pull requests

CI checks formatting, typing, tests, and commit messages. Feature pull requests target dev; release pull requests target main.

Versions

Conventional Commits drive semantic versions. Pushes to main can update the changelog, tag the release, and create a GitHub Release.

Documentation

GitHub Pages publishes this static document at no cost. Application deployment remains local to preserve the zero-spend promise.

Key decisions and upgrade triggers

Focused frameworksUse LangChain's standalone splitter, rank-bm25, native Ollama schemas, and optional Ragas evaluation. Keep orchestration in plain Python while the visible loop remains smaller.

Local QdrantSkip Docker and cloud storage. Move to Qdrant server when multiple processes, remote clients, or concurrent ingestion are required.

Structured chunksPreserve Markdown headings and natural text boundaries, then enforce configurable character limits with overlap.

Hybrid retrievalFuse cosine and BM25Plus rankings with RRF. Add a local cross-encoder only if the evaluation set proves top-k precision is still insufficient.

Hosted checksKeep Jev off by default. Observe mode annotates claim support without changing answers; repair mode changes only a citation that passes a fresh support check.

No cloud IaCPulumi, AWS, and paid hosting conflict with the local zero-cost scope. The provider and vector-store seams preserve a migration path.

Golden evaluation datasetA checked-in schema and deterministic runner score human-verified questions, expected evidence, required facts, abstention, and latency. Ragas is a secondary local diagnostic and never promotes silver cases.