Building Semantic Search for a Laravel Markdown Blog with Embeddings

· 6 min read
#ai #laravel #semantic-search #embeddings #markdown

Introduction

Keyword search works well when readers type the exact terms used in your article. Technical blogs are messier than that. A reader may search for "query optimization" while your post is framed around query plans, indexes, or database strategy.

That is where semantic search becomes useful.

For a Markdown-first blog, this is a particularly good fit because the content is clean, structured, and easy to process through an indexing pipeline.

Table of Contents

  • Minimal architecture for semantic search
  • Why chunking and metadata matter
  • Ranking beyond cosine similarity
  • A more practical search and indexing flow
  • How to evaluate search quality
  • Common pitfalls to avoid

A Minimal Architecture

For a personal or small team blog, a good baseline flow looks like this:

  1. Read all Markdown files.
  2. Parse frontmatter and body content.
  3. Chunk the content by sections or headings.
  4. Generate embeddings for each chunk.
  5. Store vectors together with useful metadata.
  6. Embed the user query and rank the closest matches.
namespace App\Services\Search;

class ChunkPostContent
{
    public function handle(string $markdown): array
    {
        $sections = preg_split('/\n##\s+/', $markdown);

        return collect($sections)
            ->map(fn (string $section) => trim($section))
            ->filter(fn (string $section) => mb_strlen($section) > 120)
            ->values()
            ->all();
    }
}

Do not overcomplicate the first version. For a few hundred posts, heading-based chunking is often enough.

Metadata Matters

Every stored vector should include metadata such as:

  • post slug
  • title
  • locale
  • section heading
  • publish date
  • tags

Metadata is what makes results explainable, filterable, and easier to debug. A vector without metadata becomes awkward to work with very quickly.

Chunking Is More Important Than Many Tutorials Admit

Most semantic-search tutorials spend a lot of time on vector stores and very little on chunking. In practice, chunking often affects retrieval quality more than switching to a slightly better embedding model.

If chunks are too large:

  • each vector carries too many ideas
  • search results become broad and generic
  • ranking struggles to identify the most relevant section

If chunks are too small:

  • important context gets lost
  • results become fragmented and hard to read
  • vector count grows unnecessarily fast

For a Markdown technical blog, a practical default is:

  • split by ## or ### headings first
  • merge extremely short sections into the next chunk
  • keep each chunk long enough to stand on its own

Additional Metadata Worth Storing

Alongside the obvious fields, it is often useful to keep:

  • a short excerpt for UI display
  • heading_level to distinguish major and minor sections
  • word_count so you can exclude thin chunks
  • is_draft to ensure unpublished content never leaks into search

Ranking Needs More Than Similarity

Cosine similarity is a starting point, not the whole ranking strategy. For technical content, you usually want additional signals such as:

  • a small freshness boost for newer posts
  • higher weight when the title matches the intent
  • a tag boost for related topics
  • exclusion of draft or overly thin content

Good semantic search is guided retrieval, not blind trust in vector scores.

A More Practical Search Flow

Instead of simply taking the top 5 chunks by similarity and showing them directly, a more useful flow is often:

  1. embed the query
  2. retrieve the top 20 chunks
  3. group them by post_slug
  4. apply business signals such as title match, tag match, and freshness
  5. keep the top 5 posts
  6. show the best chunk from each post as the snippet

That prevents a single article from filling the result list with multiple near-duplicate chunks.

Evaluating Search Quality Without Overbuilding

You do not need a full ML evaluation system for a blog. A practical approach is to define 20-30 representative queries such as:

  • optimize mysql queries
  • semantic search laravel blog
  • queue retry ai jobs

Then check:

  • does the correct post appear in the top 3?
  • is the snippet actually useful?
  • are the results too repetitive?

That kind of manual benchmark is cheap and often enough to catch regressions.

A More Realistic Indexing Pipeline

In practice, indexing often needs more than the minimal example:

  1. load Markdown files
  2. parse frontmatter
  3. optionally strip sections that should not dominate search, such as very long code fences
  4. chunk by heading
  5. normalize whitespace
  6. batch embedding calls
  7. upsert vector records

That pipeline should be idempotent. Reindexing a post should update it cleanly, not create duplicates.

Snippet Generation Also Shapes Perceived Search Quality

Even if retrieval is correct, the search experience still feels weak if the snippet is poor. A few simple rules help a lot:

  • prefer the highest-scoring chunk as the snippet source
  • trim by sentence or short paragraph, not arbitrary character length
  • show the related heading so users know where the result came from

Users often judge search quality not only by ranking, but by whether the snippet makes the result worth clicking.

When to Reindex

Because the blog is Markdown-first, the most practical indexing trigger is when a post is published or updated. A batch command is often enough:

php artisan blog:reindex-search --locale=en

Batch reindexing is easier to control and usually cheaper than reprocessing everything on every deploy.

Common Pitfalls

  • indexing draft content accidentally
  • using the same chunking strategy for every type of content
  • storing too little metadata to debug ranking problems
  • trusting similarity scores without testing real user intent

Avoiding those four issues already makes the first version much more useful.

Key takeaways:

  1. Semantic search is a strong fit for technical blogs because readers use many different phrasings for the same topic.
  2. Chunking and metadata usually affect retrieval quality more than teams expect.
  3. Good ranking combines vector similarity with title, tag, and freshness signals.
  4. A small benchmark query set is often enough to catch search-quality regressions.
  5. Indexing should be idempotent and must avoid draft leakage or low-value chunks.

Conclusion

Semantic search is one of the most useful AI features you can add to a technical blog. The input is clean, the user benefit is obvious, and the implementation can stay simple if you begin with small chunks, solid metadata, and ranking logic that includes real product signals.

Comments