feat(server): implement FTS5-based full-text search (#5079)

* build: add sqlite_fts5 build tag to enable FTS5 support

* feat: add SearchBackend config option (default: fts)

* feat: add buildFTS5Query for safe FTS5 query preprocessing

* feat: add FTS5 search backend with config toggle, refactor legacy search

- Add searchExprFunc type and getSearchExpr() for backend selection
- Rename fullTextExpr to legacySearchExpr
- Add ftsSearchExpr using FTS5 MATCH subquery
- Update fullTextFilter in sql_restful.go to use configured backend

* feat: add FTS5 migration with virtual tables, triggers, and search_participants

Creates FTS5 virtual tables for media_file, album, and artist with
unicode61 tokenizer and diacritic folding. Adds search_participants
column, populates from JSON, and sets up INSERT/UPDATE/DELETE triggers.

* feat: populate search_participants in PostMapArgs for FTS5 indexing

* test: add FTS5 search integration tests

* fix: exclude FTS5 virtual tables from e2e DB restore

The restoreDB function iterates all tables in sqlite_master and
runs DELETE + INSERT to reset state. FTS5 contentless virtual tables
cannot be directly deleted from. Since triggers handle FTS5 sync
automatically, simply skip tables matching *_fts and *_fts_* patterns.

* build: add compile-time guard for sqlite_fts5 build tag

Same pattern as netgo: compilation fails with a clear error if
the sqlite_fts5 build tag is missing.

* build: add sqlite_fts5 tag to reflex dev server config

* build: extract GO_BUILD_TAGS variable in Makefile to avoid duplication

* fix: strip leading * from FTS5 queries to prevent "unknown special query" error

* feat: auto-append prefix wildcard to FTS5 search tokens for broader matching

Every plain search token now gets a trailing * appended (e.g., "love" becomes
"love*"), so searching for "love" also matches "lovelace", "lovely", etc.
Quoted phrases are preserved as exact matches without wildcards. Results are
ordered alphabetically by name/title, so shorter exact matches naturally
appear first.

* fix: clarify comments about FTS5 operator neutralization

The comments said "strip" but the code lowercases operators to
neutralize them (FTS5 operators are case-sensitive). Updated comments
to accurately describe the behavior.

* fix: use fmt.Sprintf for FTS5 phrase placeholders

The previous encoding used rune('0'+index) which silently breaks with
10+ quoted phrases. Use fmt.Sprintf for arbitrary index support.

* fix: validate and normalize SearchBackend config option

Normalize the value to lowercase and fall back to "fts" with a log
warning for unrecognized values. This prevents silent misconfiguration
from typos like "FTS", "Legacy", or "fts5".

* refactor: improve documentation for build tags and FTS5 requirements

Signed-off-by: Deluan <deluan@navidrome.org>

* refactor: convert FTS5 query and search backend normalization tests to DescribeTable format

Signed-off-by: Deluan <deluan@navidrome.org>

* fix: add sqlite_fts5 build tag to golangci configuration

Signed-off-by: Deluan <deluan@navidrome.org>

* feat: add UISearchDebounceMs configuration option and update related components

Signed-off-by: Deluan <deluan@navidrome.org>

* fix: fall back to legacy search when SearchFullString is enabled

FTS5 is token-based and cannot match substrings within words, so
getSearchExpr now returns legacySearchExpr when SearchFullString
is true, regardless of SearchBackend setting.

* fix: add sqlite_fts5 build tag to CI pipeline and Dockerfile

* fix: add WHEN clauses to FTS5 AFTER UPDATE triggers

Added WHEN clauses to the media_file_fts_au, album_fts_au, and
artist_fts_au triggers so they only fire when FTS-indexed columns
actually change. Previously, every row update (e.g., play count, rating,
starred status) triggered an unnecessary delete+insert cycle in the FTS
shadow tables. The WHEN clauses use IS NOT for NULL-safe comparison of
each indexed column, avoiding FTS index churn for non-indexed updates.

* feat: add SearchBackend configuration option to data and insights components

Signed-off-by: Deluan <deluan@navidrome.org>

* fix: enhance input sanitization for FTS5 by stripping additional punctuation and special characters

Signed-off-by: Deluan <deluan@navidrome.org>

* feat: add search_normalized column for punctuated name search (R.E.M., AC/DC)

Add index-time normalization and query-time single-letter collapsing to
fix FTS5 search for punctuated names. A new search_normalized column
stores concatenated forms of punctuated words (e.g., "R.E.M." → "REM",
"AC/DC" → "ACDC") and is indexed in FTS5 tables. At query time, runs of
consecutive single letters (from dot-stripping) are collapsed into OR
expressions like ("R E M" OR REM*) to match both the original tokens and
the normalized form. This enables searching by "R.E.M.", "REM", "AC/DC",
"ACDC", "A-ha", or "Aha" and finding the correct results.

* refactor: simplify isSingleUnicodeLetter to avoid []rune allocation

Use utf8.DecodeRuneInString to check for a single Unicode letter
instead of converting the entire string to a []rune slice.

* feat: define ftsSearchColumns for flexible FTS5 search column inclusion

Signed-off-by: Deluan <deluan@navidrome.org>

* feat: update collapseSingleLetterRuns to return quoted phrases for abbreviations

Signed-off-by: Deluan <deluan@navidrome.org>

* feat: implement extractPunctuatedWords to handle artist/album names with embedded punctuation

Signed-off-by: Deluan <deluan@navidrome.org>

* feat: implement extractPunctuatedWords to handle artist/album names with embedded punctuation

Signed-off-by: Deluan <deluan@navidrome.org>

* refactor: punctuated word handling to improve processing of artist/album names

Signed-off-by: Deluan <deluan@navidrome.org>

* feat: add CJK support for search queries with LIKE filters

Signed-off-by: Deluan <deluan@navidrome.org>

* feat: enhance FTS5 search by adding album version support and CJK handling

Signed-off-by: Deluan <deluan@navidrome.org>

* refactor: search configuration to use structured options

Signed-off-by: Deluan <deluan@navidrome.org>

* feat: enhance search functionality to support punctuation-only queries and update related tests

Signed-off-by: Deluan <deluan@navidrome.org>

---------

Signed-off-by: Deluan <deluan@navidrome.org>
This commit is contained in:
Deluan Quintão
2026-02-21 17:52:42 -05:00
committed by GitHub
parent 6f5f58ae9d
commit 54de0dbc52
35 changed files with 1283 additions and 56 deletions
+4 -1
View File
@@ -62,11 +62,14 @@ func (a *dbAlbum) PostScan() error {
func (a *dbAlbum) PostMapArgs(args map[string]any) error {
fullText := []string{a.Name, a.SortAlbumName, a.AlbumArtist}
fullText = append(fullText, a.Album.Participants.AllNames()...)
participantNames := a.Album.Participants.AllNames()
fullText = append(fullText, participantNames...)
fullText = append(fullText, slices.Collect(maps.Values(a.Album.Discs))...)
fullText = append(fullText, a.Album.Tags[model.TagAlbumVersion]...)
fullText = append(fullText, a.Album.Tags[model.TagCatalogNumber]...)
args["full_text"] = formatFullText(fullText...)
args["search_participants"] = strings.Join(participantNames, " ")
args["search_normalized"] = normalizeForFTS(a.Name, a.AlbumArtist)
args["tags"] = marshalTags(a.Album.Tags)
args["participants"] = marshalParticipants(a.Album.Participants)
+6
View File
@@ -56,17 +56,23 @@ var _ = Describe("AlbumRepository", func() {
It("returns all records sorted", func() {
Expect(GetAll(model.QueryOptions{Sort: "name"})).To(Equal(model.Albums{
albumAbbeyRoad,
albumWithVersion,
albumCJK,
albumMultiDisc,
albumRadioactivity,
albumSgtPeppers,
albumPunctuation,
}))
})
It("returns all records sorted desc", func() {
Expect(GetAll(model.QueryOptions{Sort: "name", Order: "desc"})).To(Equal(model.Albums{
albumPunctuation,
albumSgtPeppers,
albumRadioactivity,
albumMultiDisc,
albumCJK,
albumWithVersion,
albumAbbeyRoad,
}))
})
+1
View File
@@ -102,6 +102,7 @@ func (a *dbArtist) PostMapArgs(m map[string]any) error {
similarArtists, _ := json.Marshal(sa)
m["similar_artists"] = string(similarArtists)
m["full_text"] = formatFullText(a.Name, a.SortArtistName)
m["search_normalized"] = normalizeForFTS(a.Name)
// Do not override the sort_artist_name and mbz_artist_id fields if they are empty
// TODO: Better way to handle this?
+37 -13
View File
@@ -193,7 +193,7 @@ var _ = Describe("ArtistRepository", func() {
Describe("Basic Operations", func() {
Describe("Count", func() {
It("returns the number of artists in the DB", func() {
Expect(repo.CountAll()).To(Equal(int64(2)))
Expect(repo.CountAll()).To(Equal(int64(4)))
})
})
@@ -228,13 +228,19 @@ var _ = Describe("ArtistRepository", func() {
idx, err := repo.GetIndex(false, []int{1})
Expect(err).ToNot(HaveOccurred())
Expect(idx).To(HaveLen(2))
Expect(idx).To(HaveLen(4))
Expect(idx[0].ID).To(Equal("F"))
Expect(idx[0].Artists).To(HaveLen(1))
Expect(idx[0].Artists[0].Name).To(Equal(artistBeatles.Name))
Expect(idx[1].ID).To(Equal("K"))
Expect(idx[1].Artists).To(HaveLen(1))
Expect(idx[1].Artists[0].Name).To(Equal(artistKraftwerk.Name))
Expect(idx[2].ID).To(Equal("R"))
Expect(idx[2].Artists).To(HaveLen(1))
Expect(idx[2].Artists[0].Name).To(Equal(artistPunctuation.Name))
Expect(idx[3].ID).To(Equal("S"))
Expect(idx[3].Artists).To(HaveLen(1))
Expect(idx[3].Artists[0].Name).To(Equal(artistCJK.Name))
// Restore the original value
artistBeatles.SortArtistName = ""
@@ -246,13 +252,19 @@ var _ = Describe("ArtistRepository", func() {
XIt("returns the index when PreferSortTags is true and SortArtistName is empty", func() {
idx, err := repo.GetIndex(false, []int{1})
Expect(err).ToNot(HaveOccurred())
Expect(idx).To(HaveLen(2))
Expect(idx).To(HaveLen(4))
Expect(idx[0].ID).To(Equal("B"))
Expect(idx[0].Artists).To(HaveLen(1))
Expect(idx[0].Artists[0].Name).To(Equal(artistBeatles.Name))
Expect(idx[1].ID).To(Equal("K"))
Expect(idx[1].Artists).To(HaveLen(1))
Expect(idx[1].Artists[0].Name).To(Equal(artistKraftwerk.Name))
Expect(idx[2].ID).To(Equal("R"))
Expect(idx[2].Artists).To(HaveLen(1))
Expect(idx[2].Artists[0].Name).To(Equal(artistPunctuation.Name))
Expect(idx[3].ID).To(Equal("S"))
Expect(idx[3].Artists).To(HaveLen(1))
Expect(idx[3].Artists[0].Name).To(Equal(artistCJK.Name))
})
})
@@ -268,13 +280,19 @@ var _ = Describe("ArtistRepository", func() {
idx, err := repo.GetIndex(false, []int{1})
Expect(err).ToNot(HaveOccurred())
Expect(idx).To(HaveLen(2))
Expect(idx).To(HaveLen(4))
Expect(idx[0].ID).To(Equal("B"))
Expect(idx[0].Artists).To(HaveLen(1))
Expect(idx[0].Artists[0].Name).To(Equal(artistBeatles.Name))
Expect(idx[1].ID).To(Equal("K"))
Expect(idx[1].Artists).To(HaveLen(1))
Expect(idx[1].Artists[0].Name).To(Equal(artistKraftwerk.Name))
Expect(idx[2].ID).To(Equal("R"))
Expect(idx[2].Artists).To(HaveLen(1))
Expect(idx[2].Artists[0].Name).To(Equal(artistPunctuation.Name))
Expect(idx[3].ID).To(Equal("S"))
Expect(idx[3].Artists).To(HaveLen(1))
Expect(idx[3].Artists[0].Name).To(Equal(artistCJK.Name))
// Restore the original value
artistBeatles.SortArtistName = ""
@@ -285,13 +303,19 @@ var _ = Describe("ArtistRepository", func() {
It("returns the index when SortArtistName is empty", func() {
idx, err := repo.GetIndex(false, []int{1})
Expect(err).ToNot(HaveOccurred())
Expect(idx).To(HaveLen(2))
Expect(idx).To(HaveLen(4))
Expect(idx[0].ID).To(Equal("B"))
Expect(idx[0].Artists).To(HaveLen(1))
Expect(idx[0].Artists[0].Name).To(Equal(artistBeatles.Name))
Expect(idx[1].ID).To(Equal("K"))
Expect(idx[1].Artists).To(HaveLen(1))
Expect(idx[1].Artists[0].Name).To(Equal(artistKraftwerk.Name))
Expect(idx[2].ID).To(Equal("R"))
Expect(idx[2].Artists).To(HaveLen(1))
Expect(idx[2].Artists[0].Name).To(Equal(artistPunctuation.Name))
Expect(idx[3].ID).To(Equal("S"))
Expect(idx[3].Artists).To(HaveLen(1))
Expect(idx[3].Artists[0].Name).To(Equal(artistCJK.Name))
})
})
@@ -377,7 +401,7 @@ var _ = Describe("ArtistRepository", func() {
// Admin users can see all content when valid library IDs are provided
idx, err := repo.GetIndex(false, []int{1})
Expect(err).ToNot(HaveOccurred())
Expect(idx).To(HaveLen(2))
Expect(idx).To(HaveLen(4))
// With non-existent library ID, admin users see no content because no artists are associated with that library
idx, err = repo.GetIndex(false, []int{999})
@@ -625,11 +649,11 @@ var _ = Describe("ArtistRepository", func() {
It("sees all artists regardless of library permissions", func() {
count, err := repo.CountAll()
Expect(err).ToNot(HaveOccurred())
Expect(count).To(Equal(int64(2)))
Expect(count).To(Equal(int64(4)))
artists, err := repo.GetAll()
Expect(err).ToNot(HaveOccurred())
Expect(artists).To(HaveLen(2))
Expect(artists).To(HaveLen(4))
exists, err := repo.Exists(artistBeatles.ID)
Expect(err).ToNot(HaveOccurred())
@@ -661,7 +685,7 @@ var _ = Describe("ArtistRepository", func() {
// Should see missing artist in GetAll by default for admin users
artists, err := repo.GetAll()
Expect(err).ToNot(HaveOccurred())
Expect(artists).To(HaveLen(3)) // Including the missing artist
Expect(artists).To(HaveLen(5)) // Including the missing artist
// Search never returns missing artists (hardcoded behavior)
results, err := repo.Search("Missing Artist", 0, 10)
@@ -767,19 +791,19 @@ var _ = Describe("ArtistRepository", func() {
It("CountAll returns correct count after gaining access", func() {
count, err := restrictedRepo.CountAll()
Expect(err).ToNot(HaveOccurred())
Expect(count).To(Equal(int64(2))) // Beatles and Kraftwerk
Expect(count).To(Equal(int64(4))) // Beatles, Kraftwerk, Seatbelts, and The Roots
})
It("GetAll returns artists after gaining access", func() {
artists, err := restrictedRepo.GetAll()
Expect(err).ToNot(HaveOccurred())
Expect(artists).To(HaveLen(2))
Expect(artists).To(HaveLen(4))
var names []string
for _, artist := range artists {
names = append(names, artist.Name)
}
Expect(names).To(ContainElements("The Beatles", "Kraftwerk"))
Expect(names).To(ContainElements("The Beatles", "Kraftwerk", "シートベルツ", "The Roots"))
})
It("Exists returns true for accessible artists", func() {
@@ -796,7 +820,7 @@ var _ = Describe("ArtistRepository", func() {
// With valid library access, should see artists
idx, err := restrictedRepo.GetIndex(false, []int{1})
Expect(err).ToNot(HaveOccurred())
Expect(idx).To(HaveLen(2))
Expect(idx).To(HaveLen(4))
// With non-existent library ID, should see nothing (non-admin user)
idx, err = restrictedRepo.GetIndex(false, []int{999})
+4 -1
View File
@@ -58,8 +58,11 @@ func (m *dbMediaFile) PostScan() error {
func (m *dbMediaFile) PostMapArgs(args map[string]any) error {
fullText := []string{m.FullTitle(), m.Album, m.Artist, m.AlbumArtist,
m.SortTitle, m.SortAlbumName, m.SortArtistName, m.SortAlbumArtistName, m.DiscSubtitle}
fullText = append(fullText, m.MediaFile.Participants.AllNames()...)
participantNames := m.MediaFile.Participants.AllNames()
fullText = append(fullText, participantNames...)
args["full_text"] = formatFullText(fullText...)
args["search_participants"] = strings.Join(participantNames, " ")
args["search_normalized"] = normalizeForFTS(m.FullTitle(), m.Album, m.Artist, m.AlbumArtist)
args["tags"] = marshalTags(m.MediaFile.Tags)
args["participants"] = marshalParticipants(m.MediaFile.Participants)
return nil
+1 -1
View File
@@ -39,7 +39,7 @@ var _ = Describe("MediaRepository", func() {
})
It("counts the number of mediafiles in the DB", func() {
Expect(mr.CountAll()).To(Equal(int64(10)))
Expect(mr.CountAll()).To(Equal(int64(13)))
})
Describe("CountBySuffix", func() {
+27 -4
View File
@@ -56,12 +56,22 @@ func al(al model.Album) model.Album {
return al
}
func alWithTags(a model.Album, tags model.Tags) model.Album {
a = al(a)
a.Tags = tags
return a
}
var (
artistKraftwerk = model.Artist{ID: "2", Name: "Kraftwerk", OrderArtistName: "kraftwerk"}
artistBeatles = model.Artist{ID: "3", Name: "The Beatles", OrderArtistName: "beatles"}
testArtists = model.Artists{
artistKraftwerk = model.Artist{ID: "2", Name: "Kraftwerk", OrderArtistName: "kraftwerk"}
artistBeatles = model.Artist{ID: "3", Name: "The Beatles", OrderArtistName: "beatles"}
artistCJK = model.Artist{ID: "4", Name: "シートベルツ", SortArtistName: "Seatbelts", OrderArtistName: "seatbelts"}
artistPunctuation = model.Artist{ID: "5", Name: "The Roots", OrderArtistName: "roots"}
testArtists = model.Artists{
artistKraftwerk,
artistBeatles,
artistCJK,
artistPunctuation,
}
)
@@ -70,11 +80,18 @@ var (
albumAbbeyRoad = al(model.Album{ID: "102", Name: "Abbey Road", AlbumArtist: "The Beatles", OrderAlbumName: "abbey road", AlbumArtistID: "3", EmbedArtPath: p("/beatles/1/come together.mp3"), SongCount: 1, MaxYear: 1969})
albumRadioactivity = al(model.Album{ID: "103", Name: "Radioactivity", AlbumArtist: "Kraftwerk", OrderAlbumName: "radioactivity", AlbumArtistID: "2", EmbedArtPath: p("/kraft/radio/radio.mp3"), SongCount: 2})
albumMultiDisc = al(model.Album{ID: "104", Name: "Multi Disc Album", AlbumArtist: "Test Artist", OrderAlbumName: "multi disc album", AlbumArtistID: "1", EmbedArtPath: p("/test/multi/disc1/track1.mp3"), SongCount: 4})
testAlbums = model.Albums{
albumCJK = al(model.Album{ID: "105", Name: "COWBOY BEBOP", AlbumArtist: "シートベルツ", OrderAlbumName: "cowboy bebop", AlbumArtistID: "4", EmbedArtPath: p("/seatbelts/cowboy-bebop/track1.mp3"), SongCount: 1})
albumWithVersion = alWithTags(model.Album{ID: "106", Name: "Abbey Road", AlbumArtist: "The Beatles", OrderAlbumName: "abbey road", AlbumArtistID: "3", EmbedArtPath: p("/beatles/2/come together.mp3"), SongCount: 1, MaxYear: 2019},
model.Tags{model.TagAlbumVersion: {"Deluxe Edition"}})
albumPunctuation = al(model.Album{ID: "107", Name: "Things Fall Apart", AlbumArtist: "The Roots", OrderAlbumName: "things fall apart", AlbumArtistID: "5", EmbedArtPath: p("/roots/things/track1.mp3"), SongCount: 1})
testAlbums = model.Albums{
albumSgtPeppers,
albumAbbeyRoad,
albumRadioactivity,
albumMultiDisc,
albumCJK,
albumWithVersion,
albumPunctuation,
}
)
@@ -101,6 +118,9 @@ var (
songDisc1Track01 = mf(model.MediaFile{ID: "2002", Title: "Disc 1 Track 1", ArtistID: "1", Artist: "Test Artist", AlbumID: "104", Album: "Multi Disc Album", DiscNumber: 1, TrackNumber: 1, Path: p("/test/multi/disc1/track1.mp3"), OrderAlbumName: "multi disc album", OrderArtistName: "test artist"})
songDisc2Track01 = mf(model.MediaFile{ID: "2003", Title: "Disc 2 Track 1", ArtistID: "1", Artist: "Test Artist", AlbumID: "104", Album: "Multi Disc Album", DiscNumber: 2, TrackNumber: 1, Path: p("/test/multi/disc2/track1.mp3"), OrderAlbumName: "multi disc album", OrderArtistName: "test artist"})
songDisc1Track02 = mf(model.MediaFile{ID: "2004", Title: "Disc 1 Track 2", ArtistID: "1", Artist: "Test Artist", AlbumID: "104", Album: "Multi Disc Album", DiscNumber: 1, TrackNumber: 2, Path: p("/test/multi/disc1/track2.mp3"), OrderAlbumName: "multi disc album", OrderArtistName: "test artist"})
songCJK = mf(model.MediaFile{ID: "3001", Title: "プラチナ・ジェット", ArtistID: "4", Artist: "シートベルツ", AlbumID: "105", Album: "COWBOY BEBOP", Path: p("/seatbelts/cowboy-bebop/track1.mp3")})
songVersioned = mf(model.MediaFile{ID: "3002", Title: "Come Together", ArtistID: "3", Artist: "The Beatles", AlbumID: "106", Album: "Abbey Road", Path: p("/beatles/2/come together.mp3")})
songPunctuation = mf(model.MediaFile{ID: "3003", Title: "!!!!!!!", ArtistID: "5", Artist: "The Roots", AlbumID: "107", Album: "Things Fall Apart", Path: p("/roots/things/track1.mp3")})
testSongs = model.MediaFiles{
songDayInALife,
songComeTogether,
@@ -112,6 +132,9 @@ var (
songDisc1Track01,
songDisc2Track01,
songDisc1Track02,
songCJK,
songVersioned,
songPunctuation,
}
)
+2 -1
View File
@@ -109,9 +109,10 @@ func booleanFilter(field string, value any) Sqlizer {
func fullTextFilter(tableName string, mbidFields ...string) func(string, any) Sqlizer {
return func(field string, value any) Sqlizer {
v := strings.ToLower(value.(string))
searchExpr := getSearchExpr()
cond := cmp.Or(
mbidExpr(tableName, v, mbidFields...),
fullTextExpr(tableName, v),
searchExpr(tableName, v),
)
return cond
}
+6 -3
View File
@@ -26,7 +26,9 @@ var _ = Describe("sqlRestful", func() {
Expect(r.parseRestFilters(context.Background(), options)).To(BeNil())
})
It(`returns nil if tries a filter with fullTextExpr("'")`, func() {
It(`returns nil if tries a filter with legacySearchExpr("'")`, func() {
DeferCleanup(configtest.SetupConfig())
conf.Server.Search.Backend = "legacy"
r.filterMappings = map[string]filterFunc{
"name": fullTextFilter("table"),
}
@@ -77,6 +79,7 @@ var _ = Describe("sqlRestful", func() {
BeforeEach(func() {
DeferCleanup(configtest.SetupConfig())
conf.Server.Search.Backend = "legacy"
tableName = "test_table"
mbidFields = []string{"mbid", "artist_mbid"}
filter = fullTextFilter(tableName, mbidFields...)
@@ -136,7 +139,7 @@ var _ = Describe("sqlRestful", func() {
Context("when SearchFullString config changes behavior", func() {
It("uses different separator with SearchFullString=false", func() {
conf.Server.SearchFullString = false
conf.Server.Search.FullString = false
result := filter("search", "test query")
andCondition, ok := result.(squirrel.And)
@@ -149,7 +152,7 @@ var _ = Describe("sqlRestful", func() {
})
It("uses no separator with SearchFullString=true", func() {
conf.Server.SearchFullString = true
conf.Server.Search.FullString = true
result := filter("search", "test query")
andCondition, ok := result.(squirrel.And)
+29 -3
View File
@@ -6,6 +6,7 @@ import (
. "github.com/Masterminds/squirrel"
"github.com/google/uuid"
"github.com/navidrome/navidrome/conf"
"github.com/navidrome/navidrome/log"
"github.com/navidrome/navidrome/model"
"github.com/navidrome/navidrome/utils/str"
)
@@ -15,6 +16,26 @@ func formatFullText(text ...string) string {
return " " + fullText
}
// searchExprFunc is the function signature for search expression builders.
type searchExprFunc func(tableName string, query string) Sqlizer
// getSearchExpr returns the active search expression function based on config.
// It falls back to legacySearchExpr when Search.FullString is enabled, because
// FTS5 is token-based and cannot match substrings within words.
// CJK queries are routed to likeSearchExpr, since FTS5's unicode61 tokenizer
// cannot segment CJK text.
func getSearchExpr() searchExprFunc {
if conf.Server.Search.Backend == "legacy" || conf.Server.Search.FullString {
return legacySearchExpr
}
return func(tableName, query string) Sqlizer {
if containsCJK(query) {
return likeSearchExpr(tableName, query)
}
return ftsSearchExpr(tableName, query)
}
}
// doSearch performs a full-text search with the specified parameters.
// The naturalOrder is used to sort results when no full-text filter is applied. It is useful for cases like
// OpenSubsonic, where an empty search query should return all results in a natural order. Normally the parameter
@@ -26,7 +47,8 @@ func (r sqlRepository) doSearch(sq SelectBuilder, q string, offset, size int, re
return nil
}
filter := fullTextExpr(r.tableName, q)
searchExpr := getSearchExpr()
filter := searchExpr(r.tableName, q)
if filter != nil {
sq = sq.Where(filter)
sq = sq.OrderBy(orderBys...)
@@ -59,13 +81,16 @@ func mbidExpr(tableName, mbid string, mbidFields ...string) Sqlizer {
return Or(cond)
}
func fullTextExpr(tableName string, s string) Sqlizer {
// legacySearchExpr generates LIKE-based search filters against the full_text column.
// This is the original search implementation, used when Search.Backend="legacy".
func legacySearchExpr(tableName string, s string) Sqlizer {
q := str.SanitizeStrings(s)
if q == "" {
log.Trace("Search using legacy backend, query is empty", "table", tableName)
return nil
}
var sep string
if !conf.Server.SearchFullString {
if !conf.Server.Search.FullString {
sep = " "
}
parts := strings.Split(q, " ")
@@ -73,5 +98,6 @@ func fullTextExpr(tableName string, s string) Sqlizer {
for _, part := range parts {
filters = append(filters, Like{tableName + ".full_text": "%" + sep + part + "%"})
}
log.Trace("Search using legacy backend", "query", filters, "table", tableName)
return filters
}
+261
View File
@@ -0,0 +1,261 @@
package persistence
import (
"fmt"
"regexp"
"strings"
"unicode"
"unicode/utf8"
. "github.com/Masterminds/squirrel"
"github.com/navidrome/navidrome/log"
)
// containsCJK returns true if the string contains any CJK (Chinese/Japanese/Korean) characters.
// CJK text doesn't use spaces between words, so FTS5's unicode61 tokenizer treats entire
// CJK phrases as single tokens, making token-based search ineffective for CJK content.
func containsCJK(s string) bool {
for _, r := range s {
if unicode.Is(unicode.Han, r) ||
unicode.Is(unicode.Hiragana, r) ||
unicode.Is(unicode.Katakana, r) ||
unicode.Is(unicode.Hangul, r) {
return true
}
}
return false
}
// fts5SpecialChars matches characters that should be stripped from user input.
// We keep only Unicode letters, numbers, whitespace, * (prefix wildcard), " (phrase quotes),
// and \x00 (internal placeholder marker). All punctuation is removed because the unicode61
// tokenizer treats it as token separators, and characters like ' can cause FTS5 parse errors
// as unbalanced string delimiters.
var fts5SpecialChars = regexp.MustCompile(`[^\p{L}\p{N}\s*"\x00]`)
// fts5PunctStrip strips everything except letters and numbers (no whitespace, wildcards, or quotes).
// Used for normalizing words at index time to create concatenated forms (e.g., "R.E.M." → "REM").
var fts5PunctStrip = regexp.MustCompile(`[^\p{L}\p{N}]`)
// fts5Operators matches FTS5 boolean operators as whole words (case-insensitive).
var fts5Operators = regexp.MustCompile(`(?i)\b(AND|OR|NOT|NEAR)\b`)
// fts5LeadingStar matches a * at the start of a token. FTS5 only supports * at the end (prefix queries).
var fts5LeadingStar = regexp.MustCompile(`(^|[\s])\*+`)
// normalizeForFTS takes multiple strings, strips non-letter/non-number characters from each word,
// and returns a space-separated string of words that changed after stripping (deduplicated).
// This is used at index time to create concatenated forms: "R.E.M." → "REM", "AC/DC" → "ACDC".
func normalizeForFTS(values ...string) string {
seen := make(map[string]struct{})
var result []string
for _, v := range values {
for _, word := range strings.Fields(v) {
stripped := fts5PunctStrip.ReplaceAllString(word, "")
if stripped == "" || stripped == word {
continue
}
lower := strings.ToLower(stripped)
if _, ok := seen[lower]; ok {
continue
}
seen[lower] = struct{}{}
result = append(result, stripped)
}
}
return strings.Join(result, " ")
}
// isSingleUnicodeLetter returns true if token is exactly one Unicode letter.
func isSingleUnicodeLetter(token string) bool {
r, size := utf8.DecodeRuneInString(token)
return size == len(token) && size > 0 && unicode.IsLetter(r)
}
// namePunctuation is the set of characters commonly used as separators in artist/album
// names (hyphens, slashes, dots, apostrophes). Only words containing these are candidates
// for punctuated-word processing; other special characters (^, :, &) are just stripped.
const namePunctuation = `-/.''`
// processPunctuatedWords handles words with embedded name punctuation before the general
// special-character stripping. For each punctuated word it produces either:
// - A quoted phrase for dotted abbreviations: R.E.M. → "R E M"
// - A phrase+concat OR for other patterns: a-ha → ("a ha" OR aha*)
func processPunctuatedWords(input string, phrases []string) (string, []string) {
words := strings.Fields(input)
var result []string
for _, w := range words {
if strings.HasPrefix(w, "\x00") || strings.ContainsAny(w, `*"`) || !strings.ContainsAny(w, namePunctuation) {
result = append(result, w)
continue
}
concat := fts5PunctStrip.ReplaceAllString(w, "")
if concat == "" || concat == w {
result = append(result, w)
continue
}
subTokens := strings.Fields(fts5SpecialChars.ReplaceAllString(w, " "))
if len(subTokens) < 2 {
// Single sub-token after splitting (e.g., N' → N): just use the stripped form
result = append(result, concat)
continue
}
// Dotted abbreviations (R.E.M., U.K.) — all single letters separated by dots only
if isDottedAbbreviation(w, subTokens) {
phrases = append(phrases, fmt.Sprintf(`"%s"`, strings.Join(subTokens, " ")))
} else {
// Punctuated names (a-ha, AC/DC, Jay-Z) — phrase for adjacency + concat for search_normalized
phrases = append(phrases, fmt.Sprintf(`("%s" OR %s*)`, strings.Join(subTokens, " "), concat))
}
result = append(result, fmt.Sprintf("\x00PHRASE%d\x00", len(phrases)-1))
}
return strings.Join(result, " "), phrases
}
// isDottedAbbreviation returns true if w uses only dots as punctuation and all sub-tokens
// are single letters (e.g., "R.E.M.", "U.K." but not "a-ha" or "AC/DC").
func isDottedAbbreviation(w string, subTokens []string) bool {
for _, r := range w {
if !unicode.IsLetter(r) && !unicode.IsNumber(r) && r != '.' {
return false
}
}
for _, st := range subTokens {
if !isSingleUnicodeLetter(st) {
return false
}
}
return true
}
// buildFTS5Query preprocesses user input into a safe FTS5 MATCH expression.
// It preserves quoted phrases and * prefix wildcards, neutralizes FTS5 operators
// (by lowercasing them, since FTS5 operators are case-sensitive) and strips
// special characters to prevent query injection.
func buildFTS5Query(userInput string) string {
q := strings.TrimSpace(userInput)
if q == "" {
return ""
}
var phrases []string
result := q
for {
start := strings.Index(result, `"`)
if start == -1 {
break
}
end := strings.Index(result[start+1:], `"`)
if end == -1 {
// Unmatched quote — remove it
result = result[:start] + result[start+1:]
break
}
end += start + 1
phrase := result[start : end+1] // includes quotes
phrases = append(phrases, phrase)
result = result[:start] + fmt.Sprintf("\x00PHRASE%d\x00", len(phrases)-1) + result[end+1:]
}
// Neutralize FTS5 operators by lowercasing them (FTS5 operators are case-sensitive:
// AND, OR, NOT, NEAR are operators, but and, or, not, near are plain tokens)
result = fts5Operators.ReplaceAllStringFunc(result, strings.ToLower)
// Handle words with embedded punctuation (a-ha, AC/DC, R.E.M.) before stripping
result, phrases = processPunctuatedWords(result, phrases)
result = fts5SpecialChars.ReplaceAllString(result, " ")
result = fts5LeadingStar.ReplaceAllString(result, "$1")
tokens := strings.Fields(result)
// Append * to plain tokens for prefix matching (e.g., "love" → "love*").
// Skip tokens that are already wildcarded or are quoted phrase placeholders.
for i, t := range tokens {
if strings.HasPrefix(t, "\x00") || strings.HasSuffix(t, "*") {
continue
}
tokens[i] = t + "*"
}
result = strings.Join(tokens, " ")
for i, phrase := range phrases {
placeholder := fmt.Sprintf("\x00PHRASE%d\x00", i)
result = strings.ReplaceAll(result, placeholder, phrase)
}
return result
}
// likeSearchColumns defines the core columns to search with LIKE queries.
// These are the primary user-visible fields for each entity type.
// Used as a fallback when FTS5 cannot handle the query (e.g., CJK text, punctuation-only input).
var likeSearchColumns = map[string][]string{
"media_file": {"title", "album", "artist", "album_artist"},
"album": {"name", "album_artist"},
"artist": {"name"},
}
// likeSearchExpr generates LIKE-based search filters against core columns.
// Each word in the query must match at least one column (AND between words),
// and each word can match any column (OR within a word).
// Used as a fallback when FTS5 cannot handle the query (e.g., CJK text, punctuation-only input).
func likeSearchExpr(tableName string, s string) Sqlizer {
s = strings.TrimSpace(s)
if s == "" {
log.Trace("Search using LIKE backend, query is empty", "table", tableName)
return nil
}
columns, ok := likeSearchColumns[tableName]
if !ok {
log.Trace("Search using LIKE backend, couldn't find columns for this table", "table", tableName)
return nil
}
words := strings.Fields(s)
wordFilters := And{}
for _, word := range words {
colFilters := Or{}
for _, col := range columns {
colFilters = append(colFilters, Like{tableName + "." + col: "%" + word + "%"})
}
wordFilters = append(wordFilters, colFilters)
}
log.Trace("Search using LIKE backend", "query", wordFilters, "table", tableName)
return wordFilters
}
// ftsSearchColumns defines which FTS5 columns are included in general search.
// Columns not listed here are indexed but not searched by default,
// enabling future additions (comments, lyrics, bios) without affecting general search.
var ftsSearchColumns = map[string]string{
"media_file": "{title album artist album_artist sort_title sort_album_name sort_artist_name sort_album_artist_name disc_subtitle search_participants search_normalized}",
"album": "{name sort_album_name album_artist search_participants discs catalog_num album_version search_normalized}",
"artist": "{name sort_artist_name search_normalized}",
}
// ftsSearchExpr generates an FTS5 MATCH-based search filter.
// If the query produces no FTS tokens (e.g., punctuation-only like "!!!!!!!"),
// it falls back to LIKE-based search.
func ftsSearchExpr(tableName string, s string) Sqlizer {
q := buildFTS5Query(s)
if q == "" {
s = strings.TrimSpace(s)
if s != "" {
log.Trace("Search using LIKE fallback for non-tokenizable query", "table", tableName, "query", s)
return likeSearchExpr(tableName, s)
}
return nil
}
ftsTable := tableName + "_fts"
matchExpr := q
if cols, ok := ftsSearchColumns[tableName]; ok {
matchExpr = cols + " : (" + q + ")"
}
filter := Expr(
tableName+".rowid IN (SELECT rowid FROM "+ftsTable+" WHERE "+ftsTable+" MATCH ?)",
matchExpr,
)
log.Trace("Search using FTS5 backend", "table", tableName, "query", q, "filter", filter)
return filter
}
+333
View File
@@ -0,0 +1,333 @@
package persistence
import (
"context"
"github.com/navidrome/navidrome/conf"
"github.com/navidrome/navidrome/conf/configtest"
"github.com/navidrome/navidrome/log"
"github.com/navidrome/navidrome/model"
"github.com/navidrome/navidrome/model/request"
. "github.com/onsi/ginkgo/v2"
. "github.com/onsi/gomega"
)
var _ = DescribeTable("buildFTS5Query",
func(input, expected string) {
Expect(buildFTS5Query(input)).To(Equal(expected))
},
Entry("returns empty string for empty input", "", ""),
Entry("returns empty string for whitespace-only input", " ", ""),
Entry("appends * to a single word for prefix matching", "beatles", "beatles*"),
Entry("appends * to each word for prefix matching", "abbey road", "abbey* road*"),
Entry("preserves quoted phrases without appending *", `"the beatles"`, `"the beatles"`),
Entry("does not double-append * to existing prefix wildcard", "beat*", "beat*"),
Entry("strips FTS5 operators and appends * to lowercased words", "AND OR NOT NEAR", "and* or* not* near*"),
Entry("strips special FTS5 syntax characters and appends *", "test^col:val", "test* col* val*"),
Entry("handles mixed phrases and words", `"the beatles" abbey`, `"the beatles" abbey*`),
Entry("handles prefix with multiple words", "beat* abbey", "beat* abbey*"),
Entry("collapses multiple spaces", "abbey road", "abbey* road*"),
Entry("strips leading * from tokens and appends trailing *", "*livia", "livia*"),
Entry("strips leading * and preserves existing trailing *", "*livia oliv*", "livia* oliv*"),
Entry("strips standalone *", "*", ""),
Entry("strips apostrophe from input", "Guns N' Roses", "Guns* N* Roses*"),
Entry("converts slashed word to phrase+concat OR", "AC/DC", `("AC DC" OR ACDC*)`),
Entry("converts hyphenated word to phrase+concat OR", "a-ha", `("a ha" OR aha*)`),
Entry("converts partial hyphenated word to phrase+concat OR", "a-h", `("a h" OR ah*)`),
Entry("converts hyphenated name to phrase+concat OR", "Jay-Z", `("Jay Z" OR JayZ*)`),
Entry("converts contraction to phrase+concat OR", "it's", `("it s" OR its*)`),
Entry("handles punctuated word mixed with plain words", "best of a-ha", `best* of* ("a ha" OR aha*)`),
Entry("strips miscellaneous punctuation", "rock & roll, vol. 2", "rock* roll* vol* 2*"),
Entry("preserves unicode characters with diacritics", "Björk début", "Björk* début*"),
Entry("collapses dotted abbreviation into phrase", "R.E.M.", `"R E M"`),
Entry("collapses abbreviation without trailing dot", "R.E.M", `"R E M"`),
Entry("collapses abbreviation mixed with words", "best of R.E.M.", `best* of* "R E M"`),
Entry("collapses two-letter abbreviation", "U.K.", `"U K"`),
Entry("does not collapse single letter surrounded by words", "I am fine", "I* am* fine*"),
Entry("does not collapse single standalone letter", "A test", "A* test*"),
Entry("preserves quoted phrase with punctuation verbatim", `"ac/dc"`, `"ac/dc"`),
Entry("preserves quoted abbreviation verbatim", `"R.E.M."`, `"R.E.M."`),
Entry("returns empty string for punctuation-only input", "!!!!!!!", ""),
Entry("returns empty string for mixed punctuation", "!@#$%^&", ""),
)
var _ = DescribeTable("normalizeForFTS",
func(expected string, values ...string) {
Expect(normalizeForFTS(values...)).To(Equal(expected))
},
Entry("strips dots and concatenates", "REM", "R.E.M."),
Entry("strips slash", "ACDC", "AC/DC"),
Entry("strips hyphen", "Aha", "A-ha"),
Entry("skips unchanged words", "", "The Beatles"),
Entry("handles mixed input", "REM", "R.E.M.", "Automatic for the People"),
Entry("deduplicates", "REM", "R.E.M.", "R.E.M."),
Entry("strips apostrophe from word", "N", "Guns N' Roses"),
Entry("handles multiple values with punctuation", "REM ACDC", "R.E.M.", "AC/DC"),
)
var _ = DescribeTable("containsCJK",
func(input string, expected bool) {
Expect(containsCJK(input)).To(Equal(expected))
},
Entry("returns false for empty string", "", false),
Entry("returns false for ASCII text", "hello world", false),
Entry("returns false for Latin with diacritics", "Björk début", false),
Entry("detects Chinese characters (Han)", "周杰伦", true),
Entry("detects Japanese Hiragana", "こんにちは", true),
Entry("detects Japanese Katakana", "カタカナ", true),
Entry("detects Korean Hangul", "한국어", true),
Entry("detects CJK mixed with Latin", "best of 周杰伦", true),
Entry("detects single CJK character", "a曲b", true),
)
var _ = Describe("likeSearchExpr", func() {
It("returns nil for empty query", func() {
Expect(likeSearchExpr("media_file", "")).To(BeNil())
})
It("returns nil for whitespace-only query", func() {
Expect(likeSearchExpr("media_file", " ")).To(BeNil())
})
It("generates LIKE filters against core columns for single CJK word", func() {
expr := likeSearchExpr("media_file", "周杰伦")
sql, args, err := expr.ToSql()
Expect(err).ToNot(HaveOccurred())
// Should have OR between columns for the single word
Expect(sql).To(ContainSubstring("OR"))
Expect(sql).To(ContainSubstring("media_file.title LIKE"))
Expect(sql).To(ContainSubstring("media_file.album LIKE"))
Expect(sql).To(ContainSubstring("media_file.artist LIKE"))
Expect(sql).To(ContainSubstring("media_file.album_artist LIKE"))
Expect(args).To(HaveLen(4))
for _, arg := range args {
Expect(arg).To(Equal("%周杰伦%"))
}
})
It("generates AND of OR groups for multi-word query", func() {
expr := likeSearchExpr("media_file", "周杰伦 greatest")
sql, args, err := expr.ToSql()
Expect(err).ToNot(HaveOccurred())
// Two groups AND'd together, each with 4 columns OR'd
Expect(sql).To(ContainSubstring("AND"))
Expect(args).To(HaveLen(8))
})
It("uses correct columns for album table", func() {
expr := likeSearchExpr("album", "周杰伦")
sql, args, err := expr.ToSql()
Expect(err).ToNot(HaveOccurred())
Expect(sql).To(ContainSubstring("album.name LIKE"))
Expect(sql).To(ContainSubstring("album.album_artist LIKE"))
Expect(args).To(HaveLen(2))
})
It("uses correct columns for artist table", func() {
expr := likeSearchExpr("artist", "周杰伦")
sql, args, err := expr.ToSql()
Expect(err).ToNot(HaveOccurred())
Expect(sql).To(ContainSubstring("artist.name LIKE"))
Expect(args).To(HaveLen(1))
})
It("returns nil for unknown table", func() {
Expect(likeSearchExpr("unknown_table", "周杰伦")).To(BeNil())
})
})
var _ = Describe("ftsSearchExpr", func() {
It("returns nil for empty query", func() {
Expect(ftsSearchExpr("media_file", "")).To(BeNil())
})
It("generates rowid IN subquery with MATCH and column filter", func() {
expr := ftsSearchExpr("media_file", "beatles")
sql, args, err := expr.ToSql()
Expect(err).ToNot(HaveOccurred())
Expect(sql).To(ContainSubstring("media_file.rowid IN"))
Expect(sql).To(ContainSubstring("media_file_fts"))
Expect(sql).To(ContainSubstring("MATCH"))
Expect(args).To(HaveLen(1))
Expect(args[0]).To(HavePrefix("{title album artist album_artist"))
Expect(args[0]).To(ContainSubstring("beatles*"))
})
It("generates correct FTS table name per entity", func() {
for _, table := range []string{"media_file", "album", "artist"} {
expr := ftsSearchExpr(table, "test")
sql, _, err := expr.ToSql()
Expect(err).ToNot(HaveOccurred())
Expect(sql).To(ContainSubstring(table + ".rowid IN"))
Expect(sql).To(ContainSubstring(table + "_fts"))
}
})
It("wraps query with column filter for known tables", func() {
expr := ftsSearchExpr("artist", "Beatles")
_, args, err := expr.ToSql()
Expect(err).ToNot(HaveOccurred())
Expect(args[0]).To(Equal("{name sort_artist_name search_normalized} : (Beatles*)"))
})
It("passes query without column filter for unknown tables", func() {
expr := ftsSearchExpr("unknown_table", "test")
_, args, err := expr.ToSql()
Expect(err).ToNot(HaveOccurred())
Expect(args[0]).To(Equal("test*"))
})
It("preserves phrase queries inside column filter", func() {
expr := ftsSearchExpr("media_file", `"the beatles"`)
_, args, err := expr.ToSql()
Expect(err).ToNot(HaveOccurred())
Expect(args[0]).To(ContainSubstring(`"the beatles"`))
})
It("preserves prefix queries inside column filter", func() {
expr := ftsSearchExpr("media_file", "beat*")
_, args, err := expr.ToSql()
Expect(err).ToNot(HaveOccurred())
Expect(args[0]).To(ContainSubstring("beat*"))
})
It("falls back to LIKE search for punctuation-only query", func() {
expr := ftsSearchExpr("media_file", "!!!!!!!")
Expect(expr).ToNot(BeNil())
sql, args, err := expr.ToSql()
Expect(err).ToNot(HaveOccurred())
Expect(sql).To(ContainSubstring("LIKE"))
Expect(args).To(ContainElement("%!!!!!!!%"))
})
It("returns nil for empty string even with LIKE fallback", func() {
Expect(ftsSearchExpr("media_file", "")).To(BeNil())
Expect(ftsSearchExpr("media_file", " ")).To(BeNil())
})
})
var _ = Describe("FTS5 Integration Search", func() {
var (
mr model.MediaFileRepository
alr model.AlbumRepository
arr model.ArtistRepository
)
BeforeEach(func() {
ctx := log.NewContext(context.TODO())
ctx = request.WithUser(ctx, adminUser)
conn := GetDBXBuilder()
mr = NewMediaFileRepository(ctx, conn)
alr = NewAlbumRepository(ctx, conn)
arr = NewArtistRepository(ctx, conn)
})
Describe("MediaFile search", func() {
It("finds media files by title", func() {
results, err := mr.Search("Radioactivity", 0, 10)
Expect(err).ToNot(HaveOccurred())
Expect(results).To(HaveLen(1))
Expect(results[0].Title).To(Equal("Radioactivity"))
Expect(results[0].ID).To(Equal(songRadioactivity.ID))
})
It("finds media files by artist name", func() {
results, err := mr.Search("Beatles", 0, 10)
Expect(err).ToNot(HaveOccurred())
Expect(results).To(HaveLen(3))
for _, r := range results {
Expect(r.Artist).To(Equal("The Beatles"))
}
})
})
Describe("Album search", func() {
It("finds albums by name", func() {
results, err := alr.Search("Sgt Peppers", 0, 10)
Expect(err).ToNot(HaveOccurred())
Expect(results).To(HaveLen(1))
Expect(results[0].Name).To(Equal("Sgt Peppers"))
Expect(results[0].ID).To(Equal(albumSgtPeppers.ID))
})
It("finds albums with multi-word search", func() {
results, err := alr.Search("Abbey Road", 0, 10)
Expect(err).ToNot(HaveOccurred())
Expect(results).To(HaveLen(2))
})
})
Describe("Artist search", func() {
It("finds artists by name", func() {
results, err := arr.Search("Kraftwerk", 0, 10)
Expect(err).ToNot(HaveOccurred())
Expect(results).To(HaveLen(1))
Expect(results[0].Name).To(Equal("Kraftwerk"))
Expect(results[0].ID).To(Equal(artistKraftwerk.ID))
})
})
Describe("CJK search", func() {
It("finds media files by CJK title", func() {
results, err := mr.Search("プラチナ", 0, 10)
Expect(err).ToNot(HaveOccurred())
Expect(results).To(HaveLen(1))
Expect(results[0].Title).To(Equal("プラチナ・ジェット"))
Expect(results[0].ID).To(Equal(songCJK.ID))
})
It("finds media files by CJK artist name", func() {
results, err := mr.Search("シートベルツ", 0, 10)
Expect(err).ToNot(HaveOccurred())
Expect(results).To(HaveLen(1))
Expect(results[0].Artist).To(Equal("シートベルツ"))
})
It("finds albums by CJK artist name", func() {
results, err := alr.Search("シートベルツ", 0, 10)
Expect(err).ToNot(HaveOccurred())
Expect(results).To(HaveLen(1))
Expect(results[0].Name).To(Equal("COWBOY BEBOP"))
Expect(results[0].ID).To(Equal(albumCJK.ID))
})
It("finds artists by CJK name", func() {
results, err := arr.Search("シートベルツ", 0, 10)
Expect(err).ToNot(HaveOccurred())
Expect(results).To(HaveLen(1))
Expect(results[0].Name).To(Equal("シートベルツ"))
Expect(results[0].ID).To(Equal(artistCJK.ID))
})
})
Describe("Album version search", func() {
It("finds albums by version tag via FTS", func() {
results, err := alr.Search("Deluxe", 0, 10)
Expect(err).ToNot(HaveOccurred())
Expect(results).To(HaveLen(1))
Expect(results[0].ID).To(Equal(albumWithVersion.ID))
})
})
Describe("Punctuation-only search", func() {
It("finds media files with punctuation-only title", func() {
results, err := mr.Search("!!!!!!!", 0, 10)
Expect(err).ToNot(HaveOccurred())
Expect(results).To(HaveLen(1))
Expect(results[0].Title).To(Equal("!!!!!!!"))
Expect(results[0].ID).To(Equal(songPunctuation.ID))
})
})
Describe("Legacy backend fallback", func() {
It("returns results using legacy LIKE-based search when configured", func() {
DeferCleanup(configtest.SetupConfig())
conf.Server.Search.Backend = "legacy"
results, err := mr.Search("Radioactivity", 0, 10)
Expect(err).ToNot(HaveOccurred())
Expect(results).To(HaveLen(1))
Expect(results[0].Title).To(Equal("Radioactivity"))
})
})
})
+97
View File
@@ -1,6 +1,8 @@
package persistence
import (
"github.com/navidrome/navidrome/conf"
"github.com/navidrome/navidrome/conf/configtest"
. "github.com/onsi/ginkgo/v2"
. "github.com/onsi/gomega"
)
@@ -11,4 +13,99 @@ var _ = Describe("sqlRepository", func() {
Expect(formatFullText("legiao urbana")).To(Equal(" legiao urbana"))
})
})
Describe("legacySearchExpr", func() {
It("returns nil for empty query", func() {
Expect(legacySearchExpr("media_file", "")).To(BeNil())
})
It("generates LIKE filter for single word", func() {
expr := legacySearchExpr("media_file", "beatles")
sql, args, err := expr.ToSql()
Expect(err).ToNot(HaveOccurred())
Expect(sql).To(ContainSubstring("media_file.full_text LIKE"))
Expect(args).To(ContainElement("% beatles%"))
})
It("generates AND of LIKE filters for multiple words", func() {
expr := legacySearchExpr("media_file", "abbey road")
sql, args, err := expr.ToSql()
Expect(err).ToNot(HaveOccurred())
Expect(sql).To(ContainSubstring("AND"))
Expect(args).To(HaveLen(2))
})
})
Describe("getSearchExpr", func() {
It("returns ftsSearchExpr by default", func() {
DeferCleanup(configtest.SetupConfig())
conf.Server.Search.Backend = "fts"
conf.Server.Search.FullString = false
expr := getSearchExpr()("media_file", "test")
sql, _, err := expr.ToSql()
Expect(err).ToNot(HaveOccurred())
Expect(sql).To(ContainSubstring("MATCH"))
})
It("returns legacySearchExpr when SearchBackend is legacy", func() {
DeferCleanup(configtest.SetupConfig())
conf.Server.Search.Backend = "legacy"
conf.Server.Search.FullString = false
expr := getSearchExpr()("media_file", "test")
sql, _, err := expr.ToSql()
Expect(err).ToNot(HaveOccurred())
Expect(sql).To(ContainSubstring("LIKE"))
})
It("falls back to legacySearchExpr when SearchFullString is enabled", func() {
DeferCleanup(configtest.SetupConfig())
conf.Server.Search.Backend = "fts"
conf.Server.Search.FullString = true
expr := getSearchExpr()("media_file", "test")
sql, _, err := expr.ToSql()
Expect(err).ToNot(HaveOccurred())
Expect(sql).To(ContainSubstring("LIKE"))
})
It("routes CJK queries to likeSearchExpr instead of ftsSearchExpr", func() {
DeferCleanup(configtest.SetupConfig())
conf.Server.Search.Backend = "fts"
conf.Server.Search.FullString = false
expr := getSearchExpr()("media_file", "周杰伦")
sql, _, err := expr.ToSql()
Expect(err).ToNot(HaveOccurred())
// CJK should use LIKE, not MATCH
Expect(sql).To(ContainSubstring("LIKE"))
Expect(sql).NotTo(ContainSubstring("MATCH"))
})
It("routes non-CJK queries to ftsSearchExpr", func() {
DeferCleanup(configtest.SetupConfig())
conf.Server.Search.Backend = "fts"
conf.Server.Search.FullString = false
expr := getSearchExpr()("media_file", "beatles")
sql, _, err := expr.ToSql()
Expect(err).ToNot(HaveOccurred())
Expect(sql).To(ContainSubstring("MATCH"))
})
It("uses legacy for CJK when SearchBackend is legacy", func() {
DeferCleanup(configtest.SetupConfig())
conf.Server.Search.Backend = "legacy"
conf.Server.Search.FullString = false
expr := getSearchExpr()("media_file", "周杰伦")
sql, _, err := expr.ToSql()
Expect(err).ToNot(HaveOccurred())
// Legacy should still use full_text column LIKE
Expect(sql).To(ContainSubstring("LIKE"))
Expect(sql).To(ContainSubstring("full_text"))
})
})
})