Building a Local RAG: Ollama × sqlite-vec on One Machine (and the three silent accuracy killers)

Follow almost any "build a local RAG" tutorial and it works. That is the problem. It works, and then after a week of real use you notice that documents you definitely indexed never come back, and answers are plausible but subtly off. With no error to chase, the next move is usually to swap in a bigger model or fiddle with chunk size.

This guide is built to remove that phase up front. We build the minimum working setup, then fix the three things that quietly destroy retrieval quality in local RAG, each confirmed against primary documentation. All three run without raising an error, which is exactly why they never make it onto the list of suspects.

1. What we are building

No external services. This runs with networking off.

We deliberately avoid frameworks like LangChain, because all three failures in this article happen underneath that abstraction. Seeing what is actually sent over the wire once makes them diagnosable even after you go back to a framework.

2. The minimum that works

2.1 One embedding

ollama pull nomic-embed-text

curl http://localhost:11434/api/embed -d '{
  "model": "nomic-embed-text",
  "input": "search_document: Expense reports close at month end and are due by the 10th."
}'

{
  "model": "nomic-embed-text",
  "embeddings": [[0.0100, -0.0017, 0.0500, "…768 values…"]]
}

Note that embeddings is an array of arrays. Pass an array to input and you embed a whole batch in one call. That detail matters in §3.1.

2.2 Store and query

-- 768 dimensions. Change the model and you change this (a mismatch errors on insert)
create virtual table chunks using vec0(
  chunk_embedding float[768]
);

-- Retrieval: five nearest neighbours to the question vector
select rowid, distance
from chunks
where chunk_embedding match :query_vector
order by distance
limit 5;

A vec0 virtual table with a float[768] column and a match query. That is the entire retrieval layer of a local RAG.

An honest caveat: the sqlite-vec README states plainly that it is pre-v1 and to expect breaking changes. That is fine for a personal tool or an internal spike. If it goes into something you deliver to a client, pin the version and have a migration path.

It works now. And this is where the article actually starts.

3. Three silent accuracy killers

3.1 You are on the superseded endpoint

Ollama has two embedding endpoints, and the docs are explicit about the older one:

Note: this endpoint has been superseded by /api/embed

The difference is not cosmetic.

Copy from an older post and two things happen at once:

1. The response shape differs, so code expecting embeddings[0] gets undefined. You will catch this.
2. You cannot batch, so ten thousand chunks means ten thousand HTTP round trips. This looks like "indexing is slow" and you will not catch it.

The second one is what costs you. When reindexing is slow, you stop re-running it, which means you stop trying different chunk strategies, which means you keep whatever you guessed first. The accuracy problem shows up as experiments you never ran, so it is never attributed to the endpoint.

// Batch. Start with a small batch size and raise it — the ceiling is model and memory bound.
const res = await fetch("http://localhost:11434/api/embed", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({
    model: "nomic-embed-text",
    input: batch.map((c) => `search_document: ${c.text}`),
  }),
});
const { embeddings } = (await res.json()) as { embeddings: number[][] };

3.2 truncate defaults to true

Straight from the official parameter list for /api/embed:

truncate: truncates the end of each input to fit within context length. Returns error if false and context length is exceeded. Defaults to true

So by default, anything past the context length is cut off and processed anyway. No error.

In a RAG pipeline that means: you indexed a long chunk, but only its beginning was embedded. Anything written in the second half of that chunk is not represented in the index and therefore cannot be retrieved — ever, for any question. Nothing warns you at index time, so there is no signal that it happened.

Fix it in two steps:

// 1. While developing, turn truncation OFF so oversized input fails loudly
body: JSON.stringify({ model, input, truncate: false })
// 2. Then fix the splitter — divide the text, do not cut it

Rebuilding an index is a one-time cost. Running for months on silently truncated vectors is a permanent one.

3.3 You are not sending the task prefix (the most common one)

This is the single most frequent cause of "local RAG accuracy is just mediocre."

The official nomic model card puts it in bold:

Important: the text prompt must include a task instruction prefix, instructing the model which task is being performed.

For example, if you are implementing a RAG application, you embed your documents as search_document: <text here> and embed your user queries as search_query: <text here>.

Documents and questions take different prefixes.

The model uses the prefix to decide how to place the text in its embedding space. Questions and documents are shaped differently — one is interrogative, the other expository — so embedding them identically makes similarity a poor proxy for relevance. The prefix is how you tell the model about that asymmetry.

The trap is that it works without them. You get vectors, you get results, the ranking is just steadily a bit wrong. There is no error to attribute it to, so the time goes into bigger models instead.

// Route everything through purpose-specific helpers so raw text cannot be passed
const asDocument = (text: string) => `search_document: ${text}`;
const asQuery = (text: string) => `search_query: ${text}`;

If callers have to remember the prefix, they will forget it. Close the hole in the type system, or expose two embedding functions and no way to pass a bare string.

Changing models changes the convention. This is a nomic convention, not a universal one. Some bge models attach an instruction to the query side only; others use no prefix at all. Swap the model without reading its card and this section's bug comes straight back.

4. Shortening vectors, correctly

nomic-embed-text-v1.5 is trained with Matryoshka Representation Learning, so the leading slice of a vector still carries meaning. Going from 768 to 512 or 256 cuts storage and search time proportionally.

But the official procedure is not "slice it." The model card's own example reads:

embeddings = F.layer_norm(embeddings, normalized_shape=(embeddings.shape[1],))
embeddings = embeddings[:, :matryoshka_dim]       # slice
embeddings = F.normalize(embeddings, p=2, dim=1)  # L2 normalize AFTER slicing

The order is the point. Re-normalize after slicing. Truncation changes the vector's magnitude, so skipping the final normalize distorts cosine distance. A share of "accuracy dropped when I reduced dimensions" is not Matryoshka failing — it is the missing re-normalize.

Ollama's /api/embed exposes a dimensions parameter, which is the simpler route. Follow the order above only when slicing yourself.

And before you shorten anything, check that you need to. Ten thousand chunks at 768 dimensions and 4 bytes is about 30 MB. For a personal or small-team corpus, dimension reduction usually solves a problem you do not have.

5. Proving retrieval actually improved

Judge the fixes above with numbers, not impressions. Nothing elaborate is required.

1. Write 20–30 realistic questions. Best written by the people who will use it; write them yourself and you will unconsciously write questions your corpus happens to answer.
2. For each, decide by hand which chunk should be retrieved.
3. Measure how often it appears in the top 5 (Recall@5).

const recallAt5 = cases.filter((c) => search(c.question, 5).includes(c.expectedChunkId)).length / cases.length;

That is enough to make "before and after adding the prefix" a comparison rather than a feeling. Tuning without a comparison is where local RAG projects lose the most time.

Resist measuring answer quality first. If the right chunk is not in the top 5, no model can answer correctly. Retrieval first, generation second.

6. If you use LM Studio

LM Studio exposes an OpenAI-compatible server, so you call /v1/embeddings and load an embedding model first. That is the only real difference — all three pitfalls in §3 apply unchanged. The task prefix in particular is a property of the model, not the runtime, so LM Studio needs it exactly as much as Ollama does.

When you catch yourself thinking "maybe a different runtime would be more accurate," re-read §3 first. Runtimes do not change what the embedding means.

7. When to stop using local

Honestly, there are conditions where this setup is the wrong answer.

• Your corpus fits in the prompt. RAG exists to solve "it does not fit." If it fits, passing the full text is faster and more accurate. Measure that first.
• Updates arrive every minute. Index freshness becomes the actual system. At that point a dedicated search service costs less in total than maintaining your own.
• Several people use it at once. The one-file simplicity of SQLite becomes the constraint. This is the natural moment to move to pgvector on PostgreSQL.
• You have not identified why accuracy is low yet. Work through §3 before changing architecture. In practice, implementation problems wear the costume of architecture problems remarkably often.

Conversely, if you have documents that cannot leave the building, a corpus that does not fit in a prompt, and daily-ish updates, this setup is genuinely production-usable. No GPU required.

Local RAG accuracy is decided by whether retrieval is assembled correctly, far more than by model size. All three problems here run without errors, which is precisely why they never get suspected. The upside is that once you know them, they do not come back. Work through §3 before touching model selection or chunk size.