Mini Search Engine
Build the machinery behind every search box: an inverted index, boolean matching, and TF-IDF ranking that puts the most relevant document first — the core of Elasticsearch in about forty lines.
How it works in the real world
Searching by scanning every document is hopeless at scale. Real engines invert the problem:
- Index — map each term to the set of documents containing it. Lookup becomes a dictionary hit instead of a scan.
- Match — intersect those sets to find documents containing every query word.
- Weight — a word in every document tells you nothing; a rare word is a strong signal. That is inverse document frequency.
- Rank — score each match by term frequency times IDF and sort.
This is genuinely how Lucene, Elasticsearch and Postgres full-text search begin. You'll build all four stages against a tiny corpus.
Build the inverted index
Map every word to the set of document ids containing it. Print the number of distinct terms, then the sorted posting list for python, data and language.
index.setdefault(word, set()).add(doc_id) creates the set on first sight.
docs = {
1: "python is a great language for data work",
2: "the python language is easy to read",
3: "data pipelines move data between systems",
4: "reading code is a skill you build",
}
index = {}
for doc_id, text in docs.items():
for word in text.split():
index.setdefault(word, set()).add(doc_id)
print(f"terms: {len(index)}")
for term in ("python", "data", "language"):
print(f"{term} -> {sorted(index[term])}")
Run your code to check this step…
Boolean AND search
Write search_all(query) returning the sorted ids of documents containing every query word. Print the result for four queries, including one that matches nothing.
Start with the first word's set and intersect (&) the rest. index.get(w, set()) handles unknown words.
docs = {
1: "python is a great language for data work",
2: "the python language is easy to read",
3: "data pipelines move data between systems",
4: "reading code is a skill you build",
}
index = {}
for doc_id, text in docs.items():
for word in text.split():
index.setdefault(word, set()).add(doc_id)
def search_all(query):
words = query.split()
hits = index.get(words[0], set())
for w in words[1:]:
hits = hits & index.get(w, set())
return sorted(hits)
for q in ("python language", "data", "python data", "missing"):
print(f"{q!r} -> {search_all(q)}")
Run your code to check this step…
Score terms with IDF
Document frequency is how many docs contain a term; IDF is log(N / df). Print df and idf (3 decimals) for python, data and the — note the rare word scores highest.
df = len(index[term]); idf = math.log(N / df). Format with f"{idf:.3f}".
docs = {
1: "python is a great language for data work",
2: "the python language is easy to read",
3: "data pipelines move data between systems",
4: "reading code is a skill you build",
}
import math
index = {}
for doc_id, text in docs.items():
for word in text.split():
index.setdefault(word, set()).add(doc_id)
N = len(docs)
for term in ("python", "data", "the"):
df = len(index[term])
idf = math.log(N / df)
print(f"{term:<9} df={df} idf={idf:.3f}")
Run your code to check this step…
Rank with TF-IDF
Score each matching document as the sum over query words of tf * idf, where tf is the word's share of that document's words. Print results for data python, best first.
tf = docs[doc_id].split().count(word) / len(docs[doc_id].split()). Accumulate per doc, then sort by (-score, doc_id).
docs = {
1: "python is a great language for data work",
2: "the python language is easy to read",
3: "data pipelines move data between systems",
4: "reading code is a skill you build",
}
import math
index = {}
for doc_id, text in docs.items():
for word in text.split():
index.setdefault(word, set()).add(doc_id)
N = len(docs)
def score(query):
ranked = {}
for word in query.split():
if word not in index:
continue
idf = math.log(N / len(index[word]))
for doc_id in index[word]:
tf = docs[doc_id].split().count(word) / len(docs[doc_id].split())
ranked[doc_id] = ranked.get(doc_id, 0) + tf * idf
return sorted(ranked.items(), key=lambda kv: (-kv[1], kv[0]))
print("query: data python")
for doc_id, s in score("data python"):
print(f" {s:.4f} [{doc_id}] {docs[doc_id]}")
Run your code to check this step…
Project shipped!
You just built a complete, working mini search engine — the same architecture running in production software. Pick your next build →