[BREAKGLASS] A FTS5 extension for signal_tokenizer.
| .github/workflows | ||
| benches | ||
| src | ||
| .gitignore | ||
| Cargo.lock | ||
| Cargo.toml | ||
| cbindgen.toml | ||
| LICENSE | ||
| README.md | ||
Overview
Signal-FTS5-Extension is a C ABI library which exposes a
FTS5 tokenizer function named
signal_tokenizer that:
- Segments UTF-8 strings into words according to Unicode standard
- Normalizes and removes diacritics from words
- Converts words to lower case
When used as a custom FTS5 tokenizer this enables application to support CJK symbols in full-text search.
Extension Build/Usage Example
cargo rustc --features extension -- --crate-type=cdylib
Load extension from ./target/release/libsignal_tokenizer.dylib.
CREATE VIRTUAL TABLE
fts
USING fts5(content, tokenize='signal_tokenizer')
Generating headers
cbindgen --profile release . -o target/release/fts5-tokenizer.h
Legal things
License
Copyright 2023 Signal Messenger, LLC.
Licensed under the AGPLv3: http://www.gnu.org/licenses/agpl-3.0.html