CJK substring search fails in FTS5 — one-line trigram tokenizer fix

Executive Summary

Developer teams working with Chinese, Japanese, and Korean (CJK) text routinely encounter failing substring search in SQLite FTS5 because tokenizers are word-oriented and do not emit trigrams, which breaks code search, log analysis, chat retrieval, and LLM retrieval pipelines. This problem affects both SMBs and enterprises—roughly 2.0M engineering orgs building local-first search stacks—leading to missed matches and degraded retriever quality for multilingual corpora. A practical product is a one-line trigram tokenizer fix for FTS5 packaged as a drop-in SQLite extension with bindings, test suites, installers, and clear migration docs. Ship the core as open source and monetize via commercial builds, performance-tuned binaries, integration adapters for popular retrievers, and enterprise support contracts. The timing is favorable: the addressable market is on the order of $12.0B (2.0M orgs × $6,000 ACV) for developer search and indexing tooling, and demand is accelerating because LLM-driven retrieval needs reliable exact matching alongside semantic recall while multilingual content grows. Teams increasingly prefer embeddable, privacy-sensitive solutions over cloud-only services, which benefits a lightweight SQLite-compatible approach. You can stand out by keeping the change minimal and auditable—one-line tokenizer logic reduces security surface and makes adoption low-friction—while demonstrating CJK correctness, competitive performance, and compatibility across platforms. The main challenges are upstreaming or maintaining trust in a SQLite extension, handling trigram index size/performance tradeoffs, and creating enterprise adoption pathways; it’s worth pursuing if you can execute robust QA, clear migrations, and targeted go-to-market efforts to capture early adopters.