feat(server): implement FTS5-based full-text search (#5079)

* build: add sqlite_fts5 build tag to enable FTS5 support * feat: add SearchBackend config option (default: fts) * feat: add buildFTS5Query for safe FTS5 query preprocessing * feat: add FTS5 search backend with config toggle, refactor legacy search - Add searchExprFunc type and getSearchExpr() for backend selection - Rename fullTextExpr to legacySearchExpr - Add ftsSearchExpr using FTS5 MATCH subquery - Update fullTextFilter in sql_restful.go to use configured backend * feat: add FTS5 migration with virtual tables, triggers, and search_participants Creates FTS5 virtual tables for media_file, album, and artist with unicode61 tokenizer and diacritic folding. Adds search_participants column, populates from JSON, and sets up INSERT/UPDATE/DELETE triggers. * feat: populate search_participants in PostMapArgs for FTS5 indexing * test: add FTS5 search integration tests * fix: exclude FTS5 virtual tables from e2e DB restore The restoreDB function iterates all tables in sqlite_master and runs DELETE + INSERT to reset state. FTS5 contentless virtual tables cannot be directly deleted from. Since triggers handle FTS5 sync automatically, simply skip tables matching *_fts and *_fts_* patterns. * build: add compile-time guard for sqlite_fts5 build tag Same pattern as netgo: compilation fails with a clear error if the sqlite_fts5 build tag is missing. * build: add sqlite_fts5 tag to reflex dev server config * build: extract GO_BUILD_TAGS variable in Makefile to avoid duplication * fix: strip leading * from FTS5 queries to prevent "unknown special query" error * feat: auto-append prefix wildcard to FTS5 search tokens for broader matching Every plain search token now gets a trailing * appended (e.g., "love" becomes "love*"), so searching for "love" also matches "lovelace", "lovely", etc. Quoted phrases are preserved as exact matches without wildcards. Results are ordered alphabetically by name/title, so shorter exact matches naturally appear first. * fix: clarify comments about FTS5 operator neutralization The comments said "strip" but the code lowercases operators to neutralize them (FTS5 operators are case-sensitive). Updated comments to accurately describe the behavior. * fix: use fmt.Sprintf for FTS5 phrase placeholders The previous encoding used rune('0'+index) which silently breaks with 10+ quoted phrases. Use fmt.Sprintf for arbitrary index support. * fix: validate and normalize SearchBackend config option Normalize the value to lowercase and fall back to "fts" with a log warning for unrecognized values. This prevents silent misconfiguration from typos like "FTS", "Legacy", or "fts5". * refactor: improve documentation for build tags and FTS5 requirements Signed-off-by: Deluan <deluan@navidrome.org> * refactor: convert FTS5 query and search backend normalization tests to DescribeTable format Signed-off-by: Deluan <deluan@navidrome.org> * fix: add sqlite_fts5 build tag to golangci configuration Signed-off-by: Deluan <deluan@navidrome.org> * feat: add UISearchDebounceMs configuration option and update related components Signed-off-by: Deluan <deluan@navidrome.org> * fix: fall back to legacy search when SearchFullString is enabled FTS5 is token-based and cannot match substrings within words, so getSearchExpr now returns legacySearchExpr when SearchFullString is true, regardless of SearchBackend setting. * fix: add sqlite_fts5 build tag to CI pipeline and Dockerfile * fix: add WHEN clauses to FTS5 AFTER UPDATE triggers Added WHEN clauses to the media_file_fts_au, album_fts_au, and artist_fts_au triggers so they only fire when FTS-indexed columns actually change. Previously, every row update (e.g., play count, rating, starred status) triggered an unnecessary delete+insert cycle in the FTS shadow tables. The WHEN clauses use IS NOT for NULL-safe comparison of each indexed column, avoiding FTS index churn for non-indexed updates. * feat: add SearchBackend configuration option to data and insights components Signed-off-by: Deluan <deluan@navidrome.org> * fix: enhance input sanitization for FTS5 by stripping additional punctuation and special characters Signed-off-by: Deluan <deluan@navidrome.org> * feat: add search_normalized column for punctuated name search (R.E.M., AC/DC) Add index-time normalization and query-time single-letter collapsing to fix FTS5 search for punctuated names. A new search_normalized column stores concatenated forms of punctuated words (e.g., "R.E.M." → "REM", "AC/DC" → "ACDC") and is indexed in FTS5 tables. At query time, runs of consecutive single letters (from dot-stripping) are collapsed into OR expressions like ("R E M" OR REM*) to match both the original tokens and the normalized form. This enables searching by "R.E.M.", "REM", "AC/DC", "ACDC", "A-ha", or "Aha" and finding the correct results. * refactor: simplify isSingleUnicodeLetter to avoid []rune allocation Use utf8.DecodeRuneInString to check for a single Unicode letter instead of converting the entire string to a []rune slice. * feat: define ftsSearchColumns for flexible FTS5 search column inclusion Signed-off-by: Deluan <deluan@navidrome.org> * feat: update collapseSingleLetterRuns to return quoted phrases for abbreviations Signed-off-by: Deluan <deluan@navidrome.org> * feat: implement extractPunctuatedWords to handle artist/album names with embedded punctuation Signed-off-by: Deluan <deluan@navidrome.org> * feat: implement extractPunctuatedWords to handle artist/album names with embedded punctuation Signed-off-by: Deluan <deluan@navidrome.org> * refactor: punctuated word handling to improve processing of artist/album names Signed-off-by: Deluan <deluan@navidrome.org> * feat: add CJK support for search queries with LIKE filters Signed-off-by: Deluan <deluan@navidrome.org> * feat: enhance FTS5 search by adding album version support and CJK handling Signed-off-by: Deluan <deluan@navidrome.org> * refactor: search configuration to use structured options Signed-off-by: Deluan <deluan@navidrome.org> * feat: enhance search functionality to support punctuation-only queries and update related tests Signed-off-by: Deluan <deluan@navidrome.org> --------- Signed-off-by: Deluan <deluan@navidrome.org>
2026-02-21 17:52:42 -05:00
parent 6f5f58ae9d
commit 54de0dbc52
35 changed files with 1283 additions and 56 deletions
@@ -1,4 +0,0 @@
-package buildtags
-
-// This file is left intentionally empty. It is used to make sure the package is not empty, in the case all
-// required build tags are disabled.
@@ -0,0 +1,6 @@
+// Package buildtags provides compile-time enforcement of required build tags.
+//
+// Each file in this package is guarded by a build constraint and exports a variable
+// that main.go references. If a required tag is missing during compilation, the build
+// fails with an "undefined" error, directing the developer to use `make build`.
+package buildtags
@@ -2,10 +2,6 @@

 package buildtags

-// NOTICE: This file was created to force the inclusion of the `netgo` tag when compiling the project.
-// If the tag is not included, the compilation will fail because this variable won't be defined, and the `main.go`
-// file requires it.
-
-// Why this tag is required? See https://github.com/navidrome/navidrome/issues/700
+// The `netgo` tag is required when compiling the project. See https://github.com/navidrome/navidrome/issues/700

 var NETGO = true
@@ -0,0 +1,8 @@
+//go:build sqlite_fts5
+
+package buildtags
+
+// FTS5 is required for full-text search. Without this tag, the SQLite driver
+// won't include FTS5 support, causing runtime failures on migrations and search queries.
+
+var SQLITE_FTS5 = true
@@ -58,7 +58,7 @@ type configOptions struct {
 	SmartPlaylistRefreshDelay       time.Duration
 	AutoTranscodeDownload           bool
 	DefaultDownsamplingFormat       string
-	SearchFullString                bool
+	Search                          searchOptions `json:",omitzero"`
 	SimilarSongsMatchThreshold      int
 	RecentlyAddedByModTime          bool
 	PreferSortTags                  bool
@@ -82,6 +82,7 @@ type configOptions struct {
 	DefaultTheme                    string
 	DefaultLanguage                 string
 	DefaultUIVolume                 int
+	UISearchDebounceMs              int
 	EnableReplayGain                bool
 	EnableCoverAnimation            bool
 	EnableNowPlaying                bool
@@ -251,6 +252,11 @@ type extAuthOptions struct {
 	UserHeader     string
 }

+type searchOptions struct {
+	Backend    string
+	FullString bool
+}
+
 var (
 	Server = &configOptions{}
 	hooks  []func()
@@ -344,6 +350,8 @@ func Load(noConfigDump bool) {
 		os.Exit(1)
 	}

+	Server.Search.Backend = normalizeSearchBackend(Server.Search.Backend)
+
 	if Server.BaseURL != "" {
 		u, err := url.Parse(Server.BaseURL)
 		if err != nil {
@@ -392,6 +400,7 @@ func Load(noConfigDump bool) {
 	logDeprecatedOptions("Scanner.GenreSeparators", "")
 	logDeprecatedOptions("Scanner.GroupAlbumReleases", "")
 	logDeprecatedOptions("DevEnableBufferedScrobble", "") // Deprecated: Buffered scrobbling is now always enabled and this option is ignored
+	logDeprecatedOptions("SearchFullString", "Search.FullString")
 	logDeprecatedOptions("ReverseProxyWhitelist", "ExtAuth.TrustedSources")
 	logDeprecatedOptions("ReverseProxyUserHeader", "ExtAuth.UserHeader")
 	logDeprecatedOptions("HTTPSecurityHeaders.CustomFrameOptionsValue", "HTTPHeaders.FrameOptions")
@@ -539,6 +548,17 @@ func validateSchedule(schedule, field string) (string, error) {
 	return schedule, err
 }

+func normalizeSearchBackend(value string) string {
+	v := strings.ToLower(strings.TrimSpace(value))
+	switch v {
+	case "fts", "legacy":
+		return v
+	default:
+		log.Error("Invalid Search.Backend value, falling back to 'fts'", "value", value)
+		return "fts"
+	}
+}
+
 // AddHook is used to register initialization code that should run as soon as the config is loaded
 func AddHook(hook func()) {
 	hooks = append(hooks, hook)
@@ -585,7 +605,8 @@ func setViperDefaults() {
 	viper.SetDefault("enablemediafilecoverart", true)
 	viper.SetDefault("autotranscodedownload", false)
 	viper.SetDefault("defaultdownsamplingformat", consts.DefaultDownsamplingFormat)
-	viper.SetDefault("searchfullstring", false)
+	viper.SetDefault("search.fullstring", false)
+	viper.SetDefault("search.backend", "fts")
 	viper.SetDefault("similarsongsmatchthreshold", 85)
 	viper.SetDefault("recentlyaddedbymodtime", false)
 	viper.SetDefault("prefersorttags", false)
@@ -604,6 +625,7 @@ func setViperDefaults() {
 	viper.SetDefault("defaulttheme", "Dark")
 	viper.SetDefault("defaultlanguage", "")
 	viper.SetDefault("defaultuivolume", consts.DefaultUIVolume)
+	viper.SetDefault("uisearchdebouncems", consts.DefaultUISearchDebounceMs)
 	viper.SetDefault("enablereplaygain", true)
 	viper.SetDefault("enablecoveranimation", true)
 	viper.SetDefault("enablenowplaying", true)
@@ -52,6 +52,20 @@ var _ = Describe("Configuration", func() {
 		})
 	})

+	DescribeTable("NormalizeSearchBackend",
+		func(input, expected string) {
+			Expect(conf.NormalizeSearchBackend(input)).To(Equal(expected))
+		},
+		Entry("accepts 'fts'", "fts", "fts"),
+		Entry("accepts 'legacy'", "legacy", "legacy"),
+		Entry("normalizes 'FTS' to lowercase", "FTS", "fts"),
+		Entry("normalizes 'Legacy' to lowercase", "Legacy", "legacy"),
+		Entry("trims whitespace", "  fts  ", "fts"),
+		Entry("falls back to 'fts' for 'fts5'", "fts5", "fts"),
+		Entry("falls back to 'fts' for unrecognized values", "invalid", "fts"),
+		Entry("falls back to 'fts' for empty string", "", "fts"),
+	)
+
 	DescribeTable("should load configuration from",
 		func(format string) {
 			filename := filepath.Join("testdata", "cfg."+format)
@@ -7,3 +7,5 @@ func ResetConf() {
 var SetViperDefaults = setViperDefaults

 var ParseLanguages = parseLanguages
+
+var NormalizeSearchBackend = normalizeSearchBackend