SiteAsk: Moving the Embedding Model Out of the Server Function and Into the Browser

The short version

The chat button in the corner of this site answers questions about my projects and background. It is retrieval-augmented generation, but the pieces run in unusual places:

• Chunking and embedding happen once, at build time, into a static JSON file.
• The query embedding is computed in the visitor's browser with Transformers.js.
• Only the LLM call goes through a server route, and that route is rate limited.

SiteAsk is the same pattern pulled out into a reusable widget (site-ask/ and site-ask-react/ in the repo). It goes one step further and does retrieval in the browser as well. The rest of this post walks through the real code, why it ended up like this, and where it stops working.

Build time: a static index

scripts/build-rag-index.ts collects every published blog post, every project, and the résumé data in lib/profile-data.ts. It strips the markdown, packs paragraphs into chunks of roughly 900 characters, embeds each one, and writes lib/rag/index.json. There's no overlap between chunks and no tokenizer. For short posts and project write-ups, splitting on paragraphs has been good enough.

One detail I'd keep: project records have a track enum (product, lab, work), and the indexer rewrites it as a full sentence before embedding. The comment explains why: "the enum value alone embeds poorly." A question like "what did he build at a company?" needs the word company in the chunk to find it.

The embedder is one isomorphic module, lib/rag/embed.ts, used both by the Node indexer and by the browser:

export const EMBED_MODEL = "Xenova/all-MiniLM-L6-v2"
export const EMBED_DIM = 384

function getExtractor(): Promise<FeatureExtractionPipeline> {
  if (!extractorPromise) {
    extractorPromise = pipeline("feature-extraction", EMBED_MODEL, { dtype: "q8" })
  }
  return extractorPromise
}

export async function embed(text: string): Promise<number[]> {
  const extractor = await getExtractor()
  const output = await extractor(text, { pooling: "mean", normalize: true })
  return Array.from(output.data as Float32Array)
}

Build and query use the same model and the same q8 quantization, so the vectors are comparable. Both are L2-normalized, so a plain dot product is the cosine similarity. That means no vector database and no similarity library.

Query time: the flow

sequenceDiagram
  participant B as Browser
  participant R as /api/chat
  participant L as OpenRouter
  B->>B: lazy-load embed.ts, embed question (384 dims)
  B->>R: messages + queryEmbedding
  R->>R: rate limit, validate vector, dot-product top 6
  R->>L: system prompt with numbered context
  L-->>R: token stream
  R-->>B: plain-text stream + X-Sources header

In components/ask-amit.tsx the embedding module is loaded with a dynamic import(), and model warm-up starts when the panel opens. That way the download is usually underway before the visitor finishes typing. If the model isn't ready yet, the pending bubble says "Loading assistant model (first time only)…".

const embedder = await loadEmbedder()
const queryEmbedding = (await embedder.embed(question)).map((n) => Math.round(n * 1e6) / 1e6)

const res = await fetch("/api/chat", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({
    messages: history.map((m) => ({ role: m.role, content: m.content })),
    queryEmbedding,
  }),
})

The server route never sees the model. It checks that it got exactly 384 finite numbers, scores them against the index, and builds a prompt:

const queryEmbedding = (body as { queryEmbedding?: unknown }).queryEmbedding
if (
  !Array.isArray(queryEmbedding) ||
  queryEmbedding.length !== embeddingDim ||
  !queryEmbedding.every((n) => typeof n === "number" && Number.isFinite(n))
) {
  return json({ error: "Invalid or missing query embedding." }, 400)
}

const hits = retrieve(queryEmbedding as number[], 6)

retrieve() in lib/rag/retrieve.ts is a map, a sort and a slice over every chunk. Sources are deduplicated by URL and returned in an X-Sources header, so the widget can show links next to the streamed answer. The system prompt tells the model to use only the numbered context, cite with [1], and say so when the context doesn't cover the question.

To be precise about the title: on this site, retrieval runs on the server, because the route imports index.json. The SiteAsk widget in site-ask/src/widget.ts fetches the index as a static file and runs the same dot-product retrieval in the browser. It then POSTs { query, contexts } to your backend. In that version, the only work left for the server is the LLM call.

Why not just embed on the server?

That was the first version, and the git history shows the two ways it broke.

First failure (commit f1e7767, May 2026). The original /api/chat imported embed and ran the model inside the serverless function. onnxruntime-node brought in about 200 MB of CUDA and TensorRT binaries the function never used, and the bundle went over Vercel's 250 MB function limit. The fix moved query embedding into the browser and sent the vector instead. That commit also added the .npmrc line onnxruntime-node-install-cuda=skip, so the build-time indexer installs the CPU-only runtime.

Second failure (commit 04c2e05, September 2026). Moving the embedder into the browser wasn't the end of it. The client still loads lib/rag/embed.ts through a lazy import(), and Next's output file tracer followed that import. It copied roughly 150 MB of onnxruntime-node into all 33 server functions, even though none of them ever run it. The fix is a tracing exclusion in next.config.mjs:

// The embedder only ever runs in the browser (lazy import() in click
// handlers), but the tracer follows that import and copies ~150 MB of
// onnxruntime-node into every server function.
outputFileTracingExcludes: {
  '*': [
    'node_modules/**/onnxruntime-node/**',
    'node_modules/**/@huggingface/transformers/**',
  ],
},

The commit message records the result: the largest function went from 178 MB to 9.9 MB. The lesson I took from it is that "this code only runs in the browser" is a claim about runtime, and the bundler's file tracer doesn't know about it. If the server must not ship a dependency, check what actually ends up in the function output.

Keeping the one server route cheap

The route that remains is the one that costs money, so it has a few simple guards:

• lib/rag/rate-limit.ts allows 12 requests per IP per 60 seconds, held in memory. Its own comment says it resets on cold starts and isn't shared across instances. The real spending limit is the credit cap on the OpenRouter key; the limiter only slows down casual abuse.
• Messages are capped at 1,000 characters, and only the last 12 messages of history are sent. The answer is capped at 600 tokens, and /api/siteask caps it at 500.
• The default model is google/gemini-2.5-flash-lite, and it can be changed with OPENROUTER_MODEL.

According to the SiteAsk README, the four reference backends in site-ask/examples/ use the same defaults, plus a 32 KB body cap.

Where this stops working

These are the limits I know about. Some of them are the reason I wouldn't use this beyond a small site.

• The whole index is sent every time. Every chunk carries 384 numbers written as JSON text. Before I unpublished most of the blog, the index had 529 chunks and was about 2.4 MB. It's now 97 chunks and about 450 KB. On this site that file is bundled into the server route. In SiteAsk, the browser downloads all of it. A brute-force scan is fine at hundreds of chunks, but at tens of thousands you'd want a real vector store.
• The first question is slow. The visitor's browser has to download the model before it can embed anything. The comment in embed.ts says q8 keeps that around 25 MB. I haven't measured it across devices, and on a slow phone connection it's a real wait. SiteAsk also loads Transformers.js from esm.sh at runtime, which adds a third-party dependency to the request path.
• Public content only. The index is a static file anyone can download, so it can't include private or logged-in content. Anything you index, you publish.
• The SiteAsk backend trusts the client's context. Because retrieval happens in the browser, /api/siteask accepts whatever contexts array it receives, trimmed to 6 items of 800 characters each. A caller can send any text they like as "context". The rate limit and the token cap limit the cost, but the answer is only grounded if the client is honest.
• A stale index is silently wrong. The index is rebuilt by hand (build:rag). The script header warns that a stale index "cites anchors the page no longer has". That happened: commit f0fa4c8 exists only to rebuild the index after the site redesign changed the section anchors.
• Retrieval quality is basic. A small general-purpose sentence model, no reranking, no hybrid keyword search. Questions phrased very differently from the source text can miss.

For a portfolio or a small docs site, I think those trade-offs are right. Retrieval costs nothing to run, there's no vector database to operate, and the only paid call is the one that has to be. The code is in site-ask/ if you want the widget version.