Files
navidrome/server/serve_index_test.go
T
Deluan Quintão 54de0dbc52 feat(server): implement FTS5-based full-text search (#5079)
* build: add sqlite_fts5 build tag to enable FTS5 support

* feat: add SearchBackend config option (default: fts)

* feat: add buildFTS5Query for safe FTS5 query preprocessing

* feat: add FTS5 search backend with config toggle, refactor legacy search

- Add searchExprFunc type and getSearchExpr() for backend selection
- Rename fullTextExpr to legacySearchExpr
- Add ftsSearchExpr using FTS5 MATCH subquery
- Update fullTextFilter in sql_restful.go to use configured backend

* feat: add FTS5 migration with virtual tables, triggers, and search_participants

Creates FTS5 virtual tables for media_file, album, and artist with
unicode61 tokenizer and diacritic folding. Adds search_participants
column, populates from JSON, and sets up INSERT/UPDATE/DELETE triggers.

* feat: populate search_participants in PostMapArgs for FTS5 indexing

* test: add FTS5 search integration tests

* fix: exclude FTS5 virtual tables from e2e DB restore

The restoreDB function iterates all tables in sqlite_master and
runs DELETE + INSERT to reset state. FTS5 contentless virtual tables
cannot be directly deleted from. Since triggers handle FTS5 sync
automatically, simply skip tables matching *_fts and *_fts_* patterns.

* build: add compile-time guard for sqlite_fts5 build tag

Same pattern as netgo: compilation fails with a clear error if
the sqlite_fts5 build tag is missing.

* build: add sqlite_fts5 tag to reflex dev server config

* build: extract GO_BUILD_TAGS variable in Makefile to avoid duplication

* fix: strip leading * from FTS5 queries to prevent "unknown special query" error

* feat: auto-append prefix wildcard to FTS5 search tokens for broader matching

Every plain search token now gets a trailing * appended (e.g., "love" becomes
"love*"), so searching for "love" also matches "lovelace", "lovely", etc.
Quoted phrases are preserved as exact matches without wildcards. Results are
ordered alphabetically by name/title, so shorter exact matches naturally
appear first.

* fix: clarify comments about FTS5 operator neutralization

The comments said "strip" but the code lowercases operators to
neutralize them (FTS5 operators are case-sensitive). Updated comments
to accurately describe the behavior.

* fix: use fmt.Sprintf for FTS5 phrase placeholders

The previous encoding used rune('0'+index) which silently breaks with
10+ quoted phrases. Use fmt.Sprintf for arbitrary index support.

* fix: validate and normalize SearchBackend config option

Normalize the value to lowercase and fall back to "fts" with a log
warning for unrecognized values. This prevents silent misconfiguration
from typos like "FTS", "Legacy", or "fts5".

* refactor: improve documentation for build tags and FTS5 requirements

Signed-off-by: Deluan <deluan@navidrome.org>

* refactor: convert FTS5 query and search backend normalization tests to DescribeTable format

Signed-off-by: Deluan <deluan@navidrome.org>

* fix: add sqlite_fts5 build tag to golangci configuration

Signed-off-by: Deluan <deluan@navidrome.org>

* feat: add UISearchDebounceMs configuration option and update related components

Signed-off-by: Deluan <deluan@navidrome.org>

* fix: fall back to legacy search when SearchFullString is enabled

FTS5 is token-based and cannot match substrings within words, so
getSearchExpr now returns legacySearchExpr when SearchFullString
is true, regardless of SearchBackend setting.

* fix: add sqlite_fts5 build tag to CI pipeline and Dockerfile

* fix: add WHEN clauses to FTS5 AFTER UPDATE triggers

Added WHEN clauses to the media_file_fts_au, album_fts_au, and
artist_fts_au triggers so they only fire when FTS-indexed columns
actually change. Previously, every row update (e.g., play count, rating,
starred status) triggered an unnecessary delete+insert cycle in the FTS
shadow tables. The WHEN clauses use IS NOT for NULL-safe comparison of
each indexed column, avoiding FTS index churn for non-indexed updates.

* feat: add SearchBackend configuration option to data and insights components

Signed-off-by: Deluan <deluan@navidrome.org>

* fix: enhance input sanitization for FTS5 by stripping additional punctuation and special characters

Signed-off-by: Deluan <deluan@navidrome.org>

* feat: add search_normalized column for punctuated name search (R.E.M., AC/DC)

Add index-time normalization and query-time single-letter collapsing to
fix FTS5 search for punctuated names. A new search_normalized column
stores concatenated forms of punctuated words (e.g., "R.E.M." → "REM",
"AC/DC" → "ACDC") and is indexed in FTS5 tables. At query time, runs of
consecutive single letters (from dot-stripping) are collapsed into OR
expressions like ("R E M" OR REM*) to match both the original tokens and
the normalized form. This enables searching by "R.E.M.", "REM", "AC/DC",
"ACDC", "A-ha", or "Aha" and finding the correct results.

* refactor: simplify isSingleUnicodeLetter to avoid []rune allocation

Use utf8.DecodeRuneInString to check for a single Unicode letter
instead of converting the entire string to a []rune slice.

* feat: define ftsSearchColumns for flexible FTS5 search column inclusion

Signed-off-by: Deluan <deluan@navidrome.org>

* feat: update collapseSingleLetterRuns to return quoted phrases for abbreviations

Signed-off-by: Deluan <deluan@navidrome.org>

* feat: implement extractPunctuatedWords to handle artist/album names with embedded punctuation

Signed-off-by: Deluan <deluan@navidrome.org>

* feat: implement extractPunctuatedWords to handle artist/album names with embedded punctuation

Signed-off-by: Deluan <deluan@navidrome.org>

* refactor: punctuated word handling to improve processing of artist/album names

Signed-off-by: Deluan <deluan@navidrome.org>

* feat: add CJK support for search queries with LIKE filters

Signed-off-by: Deluan <deluan@navidrome.org>

* feat: enhance FTS5 search by adding album version support and CJK handling

Signed-off-by: Deluan <deluan@navidrome.org>

* refactor: search configuration to use structured options

Signed-off-by: Deluan <deluan@navidrome.org>

* feat: enhance search functionality to support punctuation-only queries and update related tests

Signed-off-by: Deluan <deluan@navidrome.org>

---------

Signed-off-by: Deluan <deluan@navidrome.org>
2026-02-21 17:52:42 -05:00

333 lines
12 KiB
Go

package server
import (
"encoding/json"
"fmt"
"net/http"
"net/http/httptest"
"os"
"regexp"
"strconv"
"strings"
"time"
"github.com/navidrome/navidrome/conf"
"github.com/navidrome/navidrome/conf/configtest"
"github.com/navidrome/navidrome/conf/mime"
"github.com/navidrome/navidrome/consts"
"github.com/navidrome/navidrome/model"
"github.com/navidrome/navidrome/tests"
. "github.com/onsi/ginkgo/v2"
. "github.com/onsi/gomega"
)
var _ = Describe("serveIndex", func() {
var ds model.DataStore
mockUser := &mockedUserRepo{}
fs := os.DirFS("tests/fixtures")
BeforeEach(func() {
ds = &tests.MockDataStore{MockedUser: mockUser}
DeferCleanup(configtest.SetupConfig())
})
It("adds app_config to index.html", func() {
r := httptest.NewRequest("GET", "/index.html", nil)
w := httptest.NewRecorder()
serveIndex(ds, fs, nil)(w, r)
Expect(w.Code).To(Equal(200))
config := extractAppConfig(w.Body.String())
Expect(config).To(BeAssignableToTypeOf(map[string]any{}))
})
It("sets firstTime = true when User table is empty", func() {
mockUser.empty = true
r := httptest.NewRequest("GET", "/index.html", nil)
w := httptest.NewRecorder()
serveIndex(ds, fs, nil)(w, r)
config := extractAppConfig(w.Body.String())
Expect(config).To(HaveKeyWithValue("firstTime", true))
})
It("sets firstTime = false when User table is not empty", func() {
mockUser.empty = false
r := httptest.NewRequest("GET", "/index.html", nil)
w := httptest.NewRecorder()
serveIndex(ds, fs, nil)(w, r)
config := extractAppConfig(w.Body.String())
Expect(config).To(HaveKeyWithValue("firstTime", false))
})
DescribeTable("sets configuration values",
func(configSetter func(), configKey string, expectedValue any) {
configSetter()
r := httptest.NewRequest("GET", "/index.html", nil)
w := httptest.NewRecorder()
serveIndex(ds, fs, nil)(w, r)
config := extractAppConfig(w.Body.String())
Expect(config).To(HaveKeyWithValue(configKey, expectedValue))
},
Entry("baseURL", func() { conf.Server.BasePath = "base_url_test" }, "baseURL", "base_url_test"),
Entry("welcomeMessage", func() { conf.Server.UIWelcomeMessage = "Hello" }, "welcomeMessage", "Hello"),
Entry("maxSidebarPlaylists", func() { conf.Server.MaxSidebarPlaylists = 42 }, "maxSidebarPlaylists", float64(42)),
Entry("enableTranscodingConfig", func() { conf.Server.EnableTranscodingConfig = true }, "enableTranscodingConfig", true),
Entry("enableDownloads", func() { conf.Server.EnableDownloads = true }, "enableDownloads", true),
Entry("enableFavourites", func() { conf.Server.EnableFavourites = true }, "enableFavourites", true),
Entry("enableStarRating", func() { conf.Server.EnableStarRating = true }, "enableStarRating", true),
Entry("defaultTheme", func() { conf.Server.DefaultTheme = "Light" }, "defaultTheme", "Light"),
Entry("defaultLanguage", func() { conf.Server.DefaultLanguage = "pt" }, "defaultLanguage", "pt"),
Entry("defaultUIVolume", func() { conf.Server.DefaultUIVolume = 45 }, "defaultUIVolume", float64(45)),
Entry("uiSearchDebounceMs", func() { conf.Server.UISearchDebounceMs = 500 }, "uiSearchDebounceMs", float64(500)),
Entry("enableCoverAnimation", func() { conf.Server.EnableCoverAnimation = true }, "enableCoverAnimation", true),
Entry("enableNowPlaying", func() { conf.Server.EnableNowPlaying = true }, "enableNowPlaying", true),
Entry("gaTrackingId", func() { conf.Server.GATrackingID = "UA-12345" }, "gaTrackingId", "UA-12345"),
Entry("defaultDownloadableShare", func() { conf.Server.DefaultDownloadableShare = true }, "defaultDownloadableShare", true),
Entry("devSidebarPlaylists", func() { conf.Server.DevSidebarPlaylists = true }, "devSidebarPlaylists", true),
Entry("lastFMEnabled", func() { conf.Server.LastFM.Enabled = true }, "lastFMEnabled", true),
Entry("devShowArtistPage", func() { conf.Server.DevShowArtistPage = true }, "devShowArtistPage", true),
Entry("devUIShowConfig", func() { conf.Server.DevUIShowConfig = true }, "devUIShowConfig", true),
Entry("listenBrainzEnabled", func() { conf.Server.ListenBrainz.Enabled = true }, "listenBrainzEnabled", true),
Entry("enableReplayGain", func() { conf.Server.EnableReplayGain = true }, "enableReplayGain", true),
Entry("enableExternalServices", func() { conf.Server.EnableExternalServices = true }, "enableExternalServices", true),
Entry("devActivityPanel", func() { conf.Server.DevActivityPanel = true }, "devActivityPanel", true),
Entry("shareURL", func() { conf.Server.ShareURL = "https://share.example.com" }, "shareURL", "https://share.example.com"),
Entry("enableInspect", func() { conf.Server.Inspect.Enabled = true }, "enableInspect", true),
Entry("defaultDownsamplingFormat", func() { conf.Server.DefaultDownsamplingFormat = "mp3" }, "defaultDownsamplingFormat", "mp3"),
Entry("enableUserEditing", func() { conf.Server.EnableUserEditing = false }, "enableUserEditing", false),
Entry("enableSharing", func() { conf.Server.EnableSharing = true }, "enableSharing", true),
Entry("devNewEventStream", func() { conf.Server.DevNewEventStream = true }, "devNewEventStream", true),
)
DescribeTable("sets other UI configuration values",
func(configKey string, expectedValueFunc func() any) {
r := httptest.NewRequest("GET", "/index.html", nil)
w := httptest.NewRecorder()
serveIndex(ds, fs, nil)(w, r)
config := extractAppConfig(w.Body.String())
Expect(config).To(HaveKeyWithValue(configKey, expectedValueFunc()))
},
Entry("version", "version", func() any { return consts.Version }),
Entry("variousArtistsId", "variousArtistsId", func() any { return consts.VariousArtistsID }),
Entry("losslessFormats", "losslessFormats", func() any {
return strings.ToUpper(strings.Join(mime.LosslessFormats, ","))
}),
Entry("separator", "separator", func() any { return string(os.PathSeparator) }),
)
Describe("loginBackgroundURL", func() {
Context("empty BaseURL", func() {
BeforeEach(func() {
conf.Server.BasePath = "/"
})
When("it is the default URL", func() {
It("points to the default URL", func() {
conf.Server.UILoginBackgroundURL = consts.DefaultUILoginBackgroundURL
r := httptest.NewRequest("GET", "/index.html", nil)
w := httptest.NewRecorder()
serveIndex(ds, fs, nil)(w, r)
config := extractAppConfig(w.Body.String())
Expect(config).To(HaveKeyWithValue("loginBackgroundURL", consts.DefaultUILoginBackgroundURL))
})
})
When("it is the default offline URL", func() {
It("points to the offline URL", func() {
conf.Server.UILoginBackgroundURL = consts.DefaultUILoginBackgroundURLOffline
r := httptest.NewRequest("GET", "/index.html", nil)
w := httptest.NewRecorder()
serveIndex(ds, fs, nil)(w, r)
config := extractAppConfig(w.Body.String())
Expect(config).To(HaveKeyWithValue("loginBackgroundURL", consts.DefaultUILoginBackgroundURLOffline))
})
})
When("it is a custom URL", func() {
It("points to the offline URL", func() {
conf.Server.UILoginBackgroundURL = "https://example.com/images/1.jpg"
r := httptest.NewRequest("GET", "/index.html", nil)
w := httptest.NewRecorder()
serveIndex(ds, fs, nil)(w, r)
config := extractAppConfig(w.Body.String())
Expect(config).To(HaveKeyWithValue("loginBackgroundURL", "https://example.com/images/1.jpg"))
})
})
})
Context("with a BaseURL", func() {
BeforeEach(func() {
conf.Server.BasePath = "/music"
})
When("it is the default URL", func() {
It("points to the default URL with BaseURL prefix", func() {
conf.Server.UILoginBackgroundURL = consts.DefaultUILoginBackgroundURL
r := httptest.NewRequest("GET", "/index.html", nil)
w := httptest.NewRecorder()
serveIndex(ds, fs, nil)(w, r)
config := extractAppConfig(w.Body.String())
Expect(config).To(HaveKeyWithValue("loginBackgroundURL", "/music"+consts.DefaultUILoginBackgroundURL))
})
})
When("it is the default offline URL", func() {
It("points to the offline URL", func() {
conf.Server.UILoginBackgroundURL = consts.DefaultUILoginBackgroundURLOffline
r := httptest.NewRequest("GET", "/index.html", nil)
w := httptest.NewRecorder()
serveIndex(ds, fs, nil)(w, r)
config := extractAppConfig(w.Body.String())
Expect(config).To(HaveKeyWithValue("loginBackgroundURL", consts.DefaultUILoginBackgroundURLOffline))
})
})
When("it is a custom URL", func() {
It("points to the offline URL", func() {
conf.Server.UILoginBackgroundURL = "https://example.com/images/1.jpg"
r := httptest.NewRequest("GET", "/index.html", nil)
w := httptest.NewRecorder()
serveIndex(ds, fs, nil)(w, r)
config := extractAppConfig(w.Body.String())
Expect(config).To(HaveKeyWithValue("loginBackgroundURL", "https://example.com/images/1.jpg"))
})
})
})
})
})
var _ = Describe("addShareData", func() {
var (
r *http.Request
data map[string]any
shareInfo *model.Share
)
BeforeEach(func() {
data = make(map[string]any)
r = httptest.NewRequest("GET", "/", nil)
})
Context("when shareInfo is nil or has an empty ID", func() {
It("should not modify data", func() {
addShareData(r, data, nil)
Expect(data).To(BeEmpty())
shareInfo = &model.Share{}
addShareData(r, data, shareInfo)
Expect(data).To(BeEmpty())
})
})
Context("when shareInfo is not nil and has a non-empty ID", func() {
BeforeEach(func() {
shareInfo = &model.Share{
ID: "testID",
Description: "Test description",
Downloadable: true,
Tracks: []model.MediaFile{
{
ID: "track1",
Title: "Track 1",
Artist: "Artist 1",
Album: "Album 1",
Duration: 100,
UpdatedAt: time.Date(2023, time.Month(3), 27, 0, 0, 0, 0, time.UTC),
},
{
ID: "track2",
Title: "Track 2",
Artist: "Artist 2",
Album: "Album 2",
Duration: 200,
UpdatedAt: time.Date(2023, time.Month(3), 26, 0, 0, 0, 0, time.UTC),
},
},
Contents: "Test contents",
URL: "https://example.com/share/testID",
ImageURL: "https://example.com/share/testID/image",
}
})
It("should populate data with shareInfo data", func() {
addShareData(r, data, shareInfo)
Expect(data["ShareDescription"]).To(Equal(shareInfo.Description))
Expect(data["ShareURL"]).To(Equal(shareInfo.URL))
Expect(data["ShareImageURL"]).To(Equal(shareInfo.ImageURL))
var shareData shareData
err := json.Unmarshal([]byte(data["ShareInfo"].(string)), &shareData)
Expect(err).NotTo(HaveOccurred())
Expect(shareData.ID).To(Equal(shareInfo.ID))
Expect(shareData.Description).To(Equal(shareInfo.Description))
Expect(shareData.Downloadable).To(Equal(shareInfo.Downloadable))
Expect(shareData.Tracks).To(HaveLen(len(shareInfo.Tracks)))
for i, track := range shareData.Tracks {
Expect(track.ID).To(Equal(shareInfo.Tracks[i].ID))
Expect(track.Title).To(Equal(shareInfo.Tracks[i].Title))
Expect(track.Artist).To(Equal(shareInfo.Tracks[i].Artist))
Expect(track.Album).To(Equal(shareInfo.Tracks[i].Album))
Expect(track.Duration).To(Equal(shareInfo.Tracks[i].Duration))
Expect(track.UpdatedAt).To(Equal(shareInfo.Tracks[i].UpdatedAt))
}
})
Context("when shareInfo has an empty description", func() {
BeforeEach(func() {
shareInfo.Description = ""
})
It("should use shareInfo.Contents as ShareDescription", func() {
addShareData(r, data, shareInfo)
Expect(data["ShareDescription"]).To(Equal(shareInfo.Contents))
})
})
})
})
var appConfigRegex = regexp.MustCompile(`(?m)window.__APP_CONFIG__=(.*);</script>`)
func extractAppConfig(body string) map[string]any {
config := make(map[string]any)
match := appConfigRegex.FindStringSubmatch(body)
if match == nil {
return config
}
str, err := strconv.Unquote(match[1])
if err != nil {
panic(fmt.Sprintf("%s: %s", match[1], err))
}
if err := json.Unmarshal([]byte(str), &config); err != nil {
panic(err)
}
return config
}
type mockedUserRepo struct {
model.UserRepository
empty bool
}
func (u *mockedUserRepo) CountAll(...model.QueryOptions) (int64, error) {
if u.empty {
return 0, nil
}
return 1, nil
}