Executive Summary

Developer teams working with Chinese, Japanese, and Korean (CJK) text routinely encounter failing substring search in SQLite FTS5 because tokenizers are word-oriented and do not emit trigrams, which breaks code search, log analysis, chat retrieval, and LLM retrieval pipelines. This problem affects both SMBs and enterprises—roughly 2.0M engineering orgs building local-first search stacks—leading to missed matches and degraded retriever quality for multilingual corpora. A practical product is a one-line trigram tokenizer fix for FTS5 packaged as a drop-in SQLite extension with bindings, test suites, installers, and clear migration docs. Ship the core as open source and monetize via commercial builds, performance-tuned binaries, integration adapters for popular retrievers, and enterprise support contracts. The timing is favorable: the addressable market is on the order of $12.0B (2.0M orgs × $6,000 ACV) for developer search and indexing tooling, and demand is accelerating because LLM-driven retrieval needs reliable exact matching alongside semantic recall while multilingual content grows. Teams increasingly prefer embeddable, privacy-sensitive solutions over cloud-only services, which benefits a lightweight SQLite-compatible approach. You can stand out by keeping the change minimal and auditable—one-line tokenizer logic reduces security surface and makes adoption low-friction—while demonstrating CJK correctness, competitive performance, and compatibility across platforms. The main challenges are upstreaming or maintaining trust in a SQLite extension, handling trigram index size/performance tradeoffs, and creating enterprise adoption pathways; it’s worth pursuing if you can execute robust QA, clear migrations, and targeted go-to-market efforts to capture early adopters.

Market Opportunity

CJK substring search fails in FTS5 — one-line trigram tokenizer fix targets a $12.0B = 2.0M engineering orgs x $6,000 ACV (developer search & indexing tooling across SMBs to enterprises) total addressable market with medium saturation and a year-over-year growth rate of 18%.

Key trends driving demand: LLM-driven retrieval demand -- LLMs require high-quality retrievers combining exact matches and semantic recall, increasing demand for hybrid search solutions.; Local-first & privacy-sensitive tooling -- teams prefer embeddable indexes (SQLite) and private retrievers over cloud-only solutions for sensitive code/data.; Multilingual content growth -- increasing non-Latin/CJK code comments and chat logs exposes tokenizer and substring search shortcomings in default stacks.; Shift to lightweight infra -- developers favor low-ops, embeddable components (SQLite, small binaries) to reduce cloud costs and latency..

Key competitors include Elastic (Elasticsearch / Elastic Cloud), Algolia, MeiliSearch, PostgreSQL (pg_trgm / pgroonga) – adjacent workaround, ripgrep / grep / GitHub Code Search (developer workarounds).

Sign in to access

CJK substring search fails in FTS5 — one-line trigram tokenizer fix

Executive Summary

Market Validation

Market Opportunity

More in Developer Tools

Manage dozens of websites with centralized automation and governance

Reduce latency & cost with AI-driven backend optimization for mobile games

Missed sales from phone leads fixed by an API phone system that captures and qualifies

AI coding tools lose context, provide persistent cross-tool memory

Open-ended scientific tasks lack rigorous, domain-expert benchmarks

Fix fragile delivery-app checkout flows with AI-driven test & observability