Rebuilding My Blog’s Search

The search box was lying to you

For a long time the search box on /blog did something embarrassingly shallow: it matched your query against post titles and nothing else. Search “capacitive sensor” and you’d find the post with those words in the headline. Search “detecting touch with electronics” — which is what that post is actually about — and you got nothing. The body text of every post was invisible to search.

There was a good reason it was hard to fix, and it’s the most important thing to understand about how this blog works.

Posts here are React components, not markdown

Every post on this site is a real Next.js page component at src/app/blog/<slug>/page.tsx. There is no markdown file, no CMS, no frontmatter parser at request time. That’s what lets a post embed live, interactive React — but it also means there is no tidy body of text sitting in a database waiting to be searched. The “content” of a post is JSX, tangled up with classNames, hrefs, and component props.

So the rebuild had two jobs. First, get the actual prose out of those components and into Postgres. Then, make that text findable in a way that survives the gap between the words a reader types and the words I happened to write.

The mental model: rendering and indexing are decoupled

The core design decision — the one everything else hangs off of — is that rendering and indexing are two completely independent pipelines reading the same file:

• Rendering is just Next.js serving the component at /blog/<slug>. It needs no database.
• Indexing is a separate step (pnpm blog:index) that reads the file’s source, extracts the prose and metadata, and writes them to Postgres so the post shows up in listings and search.

The practical consequence: a post can render perfectly and still be invisible to search until it’s been indexed. Keeping those two things uncoupled is what made the rest of the work safe to ship incrementally.

Shipped as four moves

Rather than one heroic branch, this went out as four small PRs, each one independently reviewable and each one leaving the site in a working state. That structure wasn’t incidental — it’s what kept a “rewrite the search engine” project from ever having a scary big-bang merge.

1. Unblock the build

Before any search work, there was a landmine: indexing was wired into the build. A prebuild hook ran the indexer on every pnpm build, which meant the build wrote to production Postgres and failed outright if the database was unreachable. You can’t safely iterate on an indexer that runs every time you compile.

So the first PR did nothing user-facing at all. It removed the prebuild hook and made the two database-backed routes (the sitemap and the projects page) render dynamically instead of trying to hit the DB at build time. The acceptance test was blunt: pnpm install, lint, typecheck, and build all succeed with a deliberately bogus, unreachable DATABASE_URL. Boring, and load-bearing: everything after this depended on being able to build without a database.

2. Teach it to read posts

Now the real work: get searchable text out of a .tsx file. The tempting approach is a pile of regexes. That way lies madness — you’d scrape className soup and URLs into your search index and jam words together across tags.

Instead, the extractor parses each post with the TypeScript compiler’s own AST and walks it, collecting only the text a reader actually sees: JSX text, string and template-literal children (which conveniently also captures the code inside <CodeBlock>), and a small allowlist of attributes like alt and title. Styling props, URLs, and the metadata block are deliberately left out.

The extracted prose lands in a Post.content column, and the actual searching is done by Postgres full-text search over a generated tsvector column with a GIN index. It’s weighted — title beats excerpt beats body — and it’s STORED, so Postgres recomputes it automatically on every write. No triggers, no application-side vector maintenance, no chance of the text and its search vector drifting out of sync:

ALTER TABLE "Post" ADD COLUMN "searchVector" tsvector
  GENERATED ALWAYS AS (
    setweight(to_tsvector('english', coalesce("title",   '')), 'A') ||
    setweight(to_tsvector('english', coalesce("excerpt", '')), 'B') ||
    setweight(to_tsvector('english', coalesce("content", '')), 'C')
  ) STORED;

CREATE INDEX "Post_searchVector_idx" ON "Post" USING GIN ("searchVector");

The query side uses websearch_to_tsquery (so quotes and or behave the way people expect from a search box) with a fallback to plainto_tsquery when a query is nothing but stopwords or stray operators. Crucially, the search helper was written to return a full ranked list of post IDs rather than a pre-sliced page — because I already knew the next PR needed to fuse that list with a second one.

3. Give it understanding

Full-text search is great until the reader and the writer choose different words. “Detecting touch with electronics” shares almost no keywords with a post titled around “capacitive sensors,” yet they mean the same thing. That gap is exactly what embeddings close.

At index time, each post’s content is chunked and embedded with OpenAI’s text-embedding-3-small (1536 dimensions), and the vectors are stored in a PostChunk table using pgvector with an HNSW index. At query time, the search query gets embedded too, and posts are ranked by their best-matching chunk’s cosine distance to it.

Now there are two ranked lists for a query — one from full-text, one from semantic — and they disagree. Fusing them is the job of Reciprocal Rank Fusion, a delightfully simple idea: each list contributes a score of 1 / (k + rank) for every item, the scores are summed, and everything re-sorts by the total. Items that rank well in either list bubble up; items both lists agree on rise fastest. A post that shares zero keywords with your query can still surface purely on the strength of the semantic list.

// Two independent ranked lists -> one fused ranking (k = 60)
const ftsIds = await searchPostIds(db, q);          // Postgres full-text
let rankedIds = ftsIds;

if (q.trim().length >= 3) {
  const emb = await embedQuery(q, apiKey);          // never throws; null on failure
  if (emb) {
    const semantic = await searchPostIdsSemantic(db, emb);  // pgvector cosine
    rankedIds = reciprocalRankFusion([ftsIds, semantic]);   // RRF, JS-side
  }
}

The two constraints I cared most about here were cost and resilience:

• Re-indexing is free when nothing changed. Each set of chunks stores a hash of the content it was embedded from. On a re-run, a post whose content hash matches makes zero OpenAI calls. Editing one post re-embeds one post; there’s a FORCE_REEMBED escape hatch for the rare full rebuild.
• Semantic is strictly additive. No API key, a two-character query, a timeout, a rate limit, a missing pgvector extension — every one of those paths quietly degrades to full-text-only. The embedding calls are wrapped to never throw, so search can lose its semantic half and the box still works. That’s true in production too: the key is optional everywhere.

4. Make it automatic

The last piece closed the loop opened by the first. Indexing was decoupled from the build — good — but that left it as a manual step, and a manual step is a step you forget. So the site got its first GitHub Action: on any merge to main that touches the blog, it runs the migrations and re-indexes against production automatically.

on:
  push:
    branches: [main]
    paths:
      - "src/app/blog/**"
      - "src/scripts/**"
      - "prisma/**"
  workflow_dispatch:
    inputs:
      force: { description: "Force re-embed all posts", type: boolean, default: false }

Because the indexer takes its API key as a plain argument rather than reaching into the app’s validated environment, the Action needs only a database URL and an optional OpenAI key — not the whole constellation of auth and OAuth secrets the app uses. And a couple of fast-follows hardened it: overlapping runs now queue instead of cancelling (interrupting a migration mid-apply is not a thing you want), and the checkout step drops its credentials since the workflow only ever reads.

How a query flows now

Put it all together and a single trip through the search box looks like this:

1. Postgres full-text ranks every published post by weighted tsvector match.
2. The query is embedded and pgvector ranks posts by nearest chunk — unless anything about that path is unavailable, in which case this step is simply skipped.
3. Reciprocal Rank Fusion merges the two rankings into one.
4. The fused ID list flows through the same tag / date / featured filters and pagination that already existed — the API’s output shape never changed, so the UI didn’t have to.

What I’d underline for anyone doing this

The engineering lesson isn’t really about search — it’s about seams. Decoupling indexing from the build first is what made it safe to swap the extraction, add a vector column, and bolt on CI as three independent, reversible steps. Every PR left the site shippable. The Unsupported Postgres columns (tsvector, vector) let Prisma own the schema while raw SQL owns the parts Prisma can’t model. And designing the full-text helper to return a list rather than a page — a full PR before there was any semantic search to fuse with — is what let the two ranking systems meet without a rewrite.

Fittingly, this post is the feature exercising itself: it was written as a component, extracted by that AST walk, embedded, and indexed by the very Action described above. If you found it by searching for something that isn’t in the title — that’s the whole point.