* fix(persistence): add nil guards to cursor wrapping in folder and mediafile repos
Prevent SIGSEGV panic when queryWithStableResults yields a zero-value
struct on the rows.Err() path (e.g., "database is locked" during
concurrent scanning). Extract cursor wrapping into wrapFolderCursor and
wrapMediaFileCursor with nil checks matching the existing pattern in
album_repository.go.
Fixes#5138
* fix(persistence): wrap original cursor error in nil guard messages
Use %w to preserve the underlying error (e.g., "database is locked")
so callers can use errors.Is/As for root cause analysis. Tests now
verify the original error is accessible via errors.Is.
* fix(persistence): add nil guards and error wrapping in album, folder, and mediafile cursor functions
Signed-off-by: Deluan <deluan@navidrome.org>
---------
Signed-off-by: Deluan <deluan@navidrome.org>
* fix(subsonic): optimize search3 for high-cardinality FTS queries
Use a two-phase query strategy for FTS5 searches to avoid the
performance penalty of expensive LEFT JOINs (annotation, bookmark,
library) on high-cardinality results like "the".
Phase 1 runs a lightweight query (main table + FTS index only) to get
sorted, paginated rowids. Phase 2 hydrates only those few rowids with
the full JOINs, making them nearly free.
For queries with complex ORDER BY expressions that reference joined
tables (e.g. artist search sorted by play count), the optimization is
skipped and the original single-query approach is used.
* fix(search): update order by clauses to include 'rank' for FTS queries
Signed-off-by: Deluan <deluan@navidrome.org>
* fix(search): reintroduce 'rank' in Phase 2 ORDER BY for FTS queries
Signed-off-by: Deluan <deluan@navidrome.org>
* fix(search): remove 'rank' from ORDER BY in non-FTS queries and adjust two-phase query handling
Signed-off-by: Deluan <deluan@navidrome.org>
* fix(search): update FTS ranking to use bm25 weights and simplify ORDER BY qualification
Signed-off-by: Deluan <deluan@navidrome.org>
* fix(search): refine FTS query handling and improve comments for clarity
Signed-off-by: Deluan <deluan@navidrome.org>
* fix(search): refactor full-text search handling to streamline query strategy selection and improve LIKE fallback logic.
Increase e2e coverage for search3
Signed-off-by: Deluan <deluan@navidrome.org>
* refactor: enhance FTS column definitions and relevance weights
Signed-off-by: Deluan <deluan@navidrome.org>
* fix(search): refactor Search method signatures to remove offset and size parameters, streamline query handling
Signed-off-by: Deluan <deluan@navidrome.org>
* fix(search): allow single-character queries in search strategies and update related tests
Signed-off-by: Deluan <deluan@navidrome.org>
* fix(search): make FTS Phase 1 treat Max=0 as no limit, reorganize tests
FTS Phase 1 unconditionally called Limit(uint64(options.Max)), which
produced LIMIT 0 when Max was zero. This diverged from applyOptions
where Max=0 means no limit. Now Phase 1 mirrors applyOptions: only add
LIMIT/OFFSET when the value is positive. Also moved legacy backend
integration tests from sql_search_fts_test.go to sql_search_like_test.go
and added regression tests for the Max=0 behavior on both backends.
* refactor: simplify callSearch function by removing variadic options and directly using QueryOptions
Signed-off-by: Deluan <deluan@navidrome.org>
* fix(search): implement ftsQueryDegraded function to detect significant content loss in FTS queries
Signed-off-by: Deluan <deluan@navidrome.org>
---------
Signed-off-by: Deluan <deluan@navidrome.org>
* build: add sqlite_fts5 build tag to enable FTS5 support
* feat: add SearchBackend config option (default: fts)
* feat: add buildFTS5Query for safe FTS5 query preprocessing
* feat: add FTS5 search backend with config toggle, refactor legacy search
- Add searchExprFunc type and getSearchExpr() for backend selection
- Rename fullTextExpr to legacySearchExpr
- Add ftsSearchExpr using FTS5 MATCH subquery
- Update fullTextFilter in sql_restful.go to use configured backend
* feat: add FTS5 migration with virtual tables, triggers, and search_participants
Creates FTS5 virtual tables for media_file, album, and artist with
unicode61 tokenizer and diacritic folding. Adds search_participants
column, populates from JSON, and sets up INSERT/UPDATE/DELETE triggers.
* feat: populate search_participants in PostMapArgs for FTS5 indexing
* test: add FTS5 search integration tests
* fix: exclude FTS5 virtual tables from e2e DB restore
The restoreDB function iterates all tables in sqlite_master and
runs DELETE + INSERT to reset state. FTS5 contentless virtual tables
cannot be directly deleted from. Since triggers handle FTS5 sync
automatically, simply skip tables matching *_fts and *_fts_* patterns.
* build: add compile-time guard for sqlite_fts5 build tag
Same pattern as netgo: compilation fails with a clear error if
the sqlite_fts5 build tag is missing.
* build: add sqlite_fts5 tag to reflex dev server config
* build: extract GO_BUILD_TAGS variable in Makefile to avoid duplication
* fix: strip leading * from FTS5 queries to prevent "unknown special query" error
* feat: auto-append prefix wildcard to FTS5 search tokens for broader matching
Every plain search token now gets a trailing * appended (e.g., "love" becomes
"love*"), so searching for "love" also matches "lovelace", "lovely", etc.
Quoted phrases are preserved as exact matches without wildcards. Results are
ordered alphabetically by name/title, so shorter exact matches naturally
appear first.
* fix: clarify comments about FTS5 operator neutralization
The comments said "strip" but the code lowercases operators to
neutralize them (FTS5 operators are case-sensitive). Updated comments
to accurately describe the behavior.
* fix: use fmt.Sprintf for FTS5 phrase placeholders
The previous encoding used rune('0'+index) which silently breaks with
10+ quoted phrases. Use fmt.Sprintf for arbitrary index support.
* fix: validate and normalize SearchBackend config option
Normalize the value to lowercase and fall back to "fts" with a log
warning for unrecognized values. This prevents silent misconfiguration
from typos like "FTS", "Legacy", or "fts5".
* refactor: improve documentation for build tags and FTS5 requirements
Signed-off-by: Deluan <deluan@navidrome.org>
* refactor: convert FTS5 query and search backend normalization tests to DescribeTable format
Signed-off-by: Deluan <deluan@navidrome.org>
* fix: add sqlite_fts5 build tag to golangci configuration
Signed-off-by: Deluan <deluan@navidrome.org>
* feat: add UISearchDebounceMs configuration option and update related components
Signed-off-by: Deluan <deluan@navidrome.org>
* fix: fall back to legacy search when SearchFullString is enabled
FTS5 is token-based and cannot match substrings within words, so
getSearchExpr now returns legacySearchExpr when SearchFullString
is true, regardless of SearchBackend setting.
* fix: add sqlite_fts5 build tag to CI pipeline and Dockerfile
* fix: add WHEN clauses to FTS5 AFTER UPDATE triggers
Added WHEN clauses to the media_file_fts_au, album_fts_au, and
artist_fts_au triggers so they only fire when FTS-indexed columns
actually change. Previously, every row update (e.g., play count, rating,
starred status) triggered an unnecessary delete+insert cycle in the FTS
shadow tables. The WHEN clauses use IS NOT for NULL-safe comparison of
each indexed column, avoiding FTS index churn for non-indexed updates.
* feat: add SearchBackend configuration option to data and insights components
Signed-off-by: Deluan <deluan@navidrome.org>
* fix: enhance input sanitization for FTS5 by stripping additional punctuation and special characters
Signed-off-by: Deluan <deluan@navidrome.org>
* feat: add search_normalized column for punctuated name search (R.E.M., AC/DC)
Add index-time normalization and query-time single-letter collapsing to
fix FTS5 search for punctuated names. A new search_normalized column
stores concatenated forms of punctuated words (e.g., "R.E.M." → "REM",
"AC/DC" → "ACDC") and is indexed in FTS5 tables. At query time, runs of
consecutive single letters (from dot-stripping) are collapsed into OR
expressions like ("R E M" OR REM*) to match both the original tokens and
the normalized form. This enables searching by "R.E.M.", "REM", "AC/DC",
"ACDC", "A-ha", or "Aha" and finding the correct results.
* refactor: simplify isSingleUnicodeLetter to avoid []rune allocation
Use utf8.DecodeRuneInString to check for a single Unicode letter
instead of converting the entire string to a []rune slice.
* feat: define ftsSearchColumns for flexible FTS5 search column inclusion
Signed-off-by: Deluan <deluan@navidrome.org>
* feat: update collapseSingleLetterRuns to return quoted phrases for abbreviations
Signed-off-by: Deluan <deluan@navidrome.org>
* feat: implement extractPunctuatedWords to handle artist/album names with embedded punctuation
Signed-off-by: Deluan <deluan@navidrome.org>
* feat: implement extractPunctuatedWords to handle artist/album names with embedded punctuation
Signed-off-by: Deluan <deluan@navidrome.org>
* refactor: punctuated word handling to improve processing of artist/album names
Signed-off-by: Deluan <deluan@navidrome.org>
* feat: add CJK support for search queries with LIKE filters
Signed-off-by: Deluan <deluan@navidrome.org>
* feat: enhance FTS5 search by adding album version support and CJK handling
Signed-off-by: Deluan <deluan@navidrome.org>
* refactor: search configuration to use structured options
Signed-off-by: Deluan <deluan@navidrome.org>
* feat: enhance search functionality to support punctuation-only queries and update related tests
Signed-off-by: Deluan <deluan@navidrome.org>
---------
Signed-off-by: Deluan <deluan@navidrome.org>
* fix: preserve created_at when moving songs between libraries (#5050)
When songs are moved between libraries, their creation date was being
reset to the current time, causing them to incorrectly appear in
"Recently Added". Three changes fix this:
1. Add hash:"ignore" to AlbumID in MediaFile struct so that Equals()
works for cross-library moves (AlbumID includes library prefix,
making hashes always differ between libraries)
2. Preserve album created_at in moveMatched() via CopyAttributes,
matching the pattern already used in persistAlbum() for
within-library album ID changes
3. Only set CreatedAt in Put() when it's zero (new files), and
explicitly copy missing.CreatedAt to the target in moveMatched()
as defense-in-depth for the INSERT code path
* test: add regression tests for created_at preservation (#5050)
Add tests covering the three aspects of the fix:
- Scanner: moveMatched preserves missing track's created_at
- Scanner: CopyAttributes called for album created_at on album change
- Scanner: CopyAttributes not called when album ID stays the same
- Persistence: Put sets CreatedAt to now for new files with zero value
- Persistence: Put preserves non-zero CreatedAt on insert
- Persistence: Put does not reset CreatedAt on update
Also adds CopyAttributes to MockAlbumRepo for test support.
* test: verify album created_at is updated in cross-library move test (#5050)
Added end-to-end assertion in the cross-library move test to verify that
the new album's CreatedAt field is actually set to the original value after
CopyAttributes runs, not just that the method was called. This strengthens
the test by confirming the mock correctly propagates the timestamp.
* fix(db): Include items with no annotation for starred=false, handle has_rating=false
* hardcode starred instead
* test: ensure albums and artists without annotations are included in starred and has_rating filters
Signed-off-by: Deluan <deluan@navidrome.org>
* refactor: replace starred and has_rating filters with annotationBoolFilter for consistency
Signed-off-by: Deluan <deluan@navidrome.org>
* fix: update annotationBoolFilter to handle boolean values correctly in SQL expressions
Signed-off-by: Deluan <deluan@navidrome.org>
---------
Signed-off-by: Deluan <deluan@navidrome.org>
Co-authored-by: Deluan <deluan@navidrome.org>
* feat(subsonic): add averageRating to API responses
Add averageRating attribute to Subsonic API responses for artists,
albums, and songs. The average is calculated across all user ratings.
* perf(db): add index for average rating queries
Add composite index on (item_id, item_type, rating) to optimize
the correlated subquery used for calculating average ratings.
Signed-off-by: Terry Raimondo <terry.raimondo@gmail.com>
* test: add tests for averageRating feature
Add tests for:
- Album.AverageRating calculation in persistence layer
- MediaFile.AverageRating calculation in persistence layer
- AverageRating mapping in subsonic response helpers
Signed-off-by: Terry Raimondo <terry.raimondo@gmail.com>
* test: improve averageRating rounding test with 3 users
Add third test user to fixtures and update rounding test to use
3 ratings (5 + 4 + 4) / 3 = 4.33 for proper decimal rounding coverage.
Signed-off-by: Terry Raimondo <terry.raimondo@gmail.com>
* perf: store avg_rating on entity tables instead of using subquery
- Add avg_rating column to album, media_file, and artist tables
- Update SetRating() to recalculate and store average when ratings change
- Read avg_rating directly from entity table in withAnnotation()
- Remove old annotation index migration (no longer needed)
This trades write-time computation for read-time performance by
pre-computing the average rating instead of using a correlated
subquery on every read.
* feat: add Subsonic.EnableAverageRating config option (default true)
Allow administrators to disable exposing averageRating in Subsonic API
responses if they don't want to expose other users' rating data.
The avg_rating column is still updated internally when users rate items,
but the value is only included in API responses when this option is enabled.
* address PR comments
- Use structs:"avg_rating" with db:"avg_rating" tag instead of SQL alias
- Remove avg_rating indexes (not needed)
- Populate avg_rating columns from existing ratings in migration
* Woops
* rename avg_rating column to average_rating
---------
Signed-off-by: Terry Raimondo <terry.raimondo@gmail.com>
* fix(deps): update wazero dependencies to resolve issues
Signed-off-by: Deluan <deluan@navidrome.org>
* fix(deps): update wazero dependency to latest version
Signed-off-by: Deluan <deluan@navidrome.org>
* fix: correct track ordering when sorting playlists by album
Fixed issue #3177 where tracks within multi-disc albums were displayed out of order when sorting playlists by album. The playlist track repository was using an incomplete sort mapping that only sorted by album name and artist, missing the critical disc_number and track_number fields.
Changed the album sort mapping in playlist_track_repository from:
order_album_name, order_album_artist_name
to:
order_album_name, order_album_artist_name, disc_number, track_number, order_artist_name, title
This now matches the sorting used in the media file repository, ensuring tracks are sorted by:
1. Album name (groups by album)
2. Album artist (handles compilations)
3. Disc number (multi-disc album discs in order)
4. Track number (tracks within disc in order)
5. Artist name and title (edge cases with missing metadata)
Added comprehensive tests with a multi-disc test album to verify correct sorting behavior.
* chore: sync go.mod and go.sum with master
* chore: align playlist album sort order with mediafile_repository (use album_id)
* fix: clean up test playlist to prevent state leakage in randomized test runs
---------
Signed-off-by: Deluan <deluan@navidrome.org>
* fix(server): remove includeMissing from search (always false)
Signed-off-by: Deluan <deluan@navidrome.org>
* fix(search): optimize search order by using natural order for improved performance
Signed-off-by: Deluan <deluan@navidrome.org>
---------
Signed-off-by: Deluan <deluan@navidrome.org>
* feat(subsonic): search by MBID functionality
Updated the search methods in the mediaFileRepository, albumRepository, and artistRepository to support searching by MBID in addition to the existing query methods. This change improves the efficiency of media file, album, and artist searches, allowing for faster retrieval of records based on MBID.
Signed-off-by: Deluan <deluan@navidrome.org>
* feat(subsonic): enhance MBID search functionality for albums and artists
Updated the search functionality to support searching by MBID for both
albums and artists. The fullTextFilter function was modified to accept
additional MBID fields, allowing for more comprehensive searches. New
tests were added to ensure that the search functionality correctly
handles MBID queries, including cases for missing entries and the
includeMissing parameter. This enhancement improves the overall search
capabilities of the application, making it easier for users to find
specific media items by their unique identifiers.
Signed-off-by: Deluan <deluan@navidrome.org>
* fix(subsonic): normalize MBID to lowercase for consistent querying
Updated the MBID handling in the SQL search logic to convert the input
to lowercase before executing the query. This change ensures that
searches are case-insensitive, improving the accuracy and reliability
of the search results when querying by MBID.
Signed-off-by: Deluan <deluan@navidrome.org>
---------
Signed-off-by: Deluan <deluan@navidrome.org>
* fix: implement RecentlyAddedByModTime support for mediafiles
Fixes#4046 by adding recently_added sort mapping to MediaFileRepository that respects the RecentlyAddedByModTime configuration setting. Previously, this feature only worked for albums, causing inconsistent behavior when clients requested tracks sorted by 'recently added'.
Changes include:
- Add mediaFileRecentlyAddedSort() function that returns 'updated_at' when RecentlyAddedByModTime=true, 'created_at' otherwise
- Add 'recently_added' sort mapping to mediafile repository
- Add comprehensive tests to verify both configuration scenarios
This ensures consistent sorting behavior between albums and tracks when using the RecentlyAddedByModTime feature.
* fix: update createdAt field to sort by recently added
Modified the createdAt field in the SongList component to include a sortBy
attribute set to "recently_added". This change ensures that the media files
are displayed in the order they were added, improving the user experience
when browsing through recently added items.
Signed-off-by: Deluan <deluan@navidrome.org>
* better testing
Signed-off-by: Deluan <deluan@navidrome.org>
---------
Signed-off-by: Deluan <deluan@navidrome.org>
* fix(subsonic): Sort songs by presence of lyrics for `getLyrics`
The current implementation of `getLyrics` fetches any songs matching the artist and title.
However, this misses a case where there may be multiple matches for the same artist/song, and one has lyrics while the other doesn't.
Resolve this by adding a custom SQL dynamic column that checks for the presence of lyrics.
* add options to selectMediaFile, update test
* more robust testing of GetAllByLyrics
* fix(subsonic): refactor GetAllByLyrics to GetAll with lyrics sorting
Signed-off-by: Deluan <deluan@navidrome.org>
* use has_lyrics, and properly support multiple sort parts
* better handle complicated internal sorts
* just use a simpler filter
* add note to setSortMappings
* remove custom sort mapping, improve test with different updatedat
* refactor tests and mock
Signed-off-by: Deluan <deluan@navidrome.org>
* default order when not specified is `asc`
Signed-off-by: Deluan <deluan@navidrome.org>
---------
Signed-off-by: Deluan <deluan@navidrome.org>
Co-authored-by: Deluan <deluan@navidrome.org>
* Start migration to dbx package
* Fix annotations and bookmarks bindings
* Fix tests
* Fix more tests
* Remove remaining references to beego/orm
* Add PostScanner/PostMapper interfaces
* Fix importing SmartPlaylists
* Renaming
* More renaming
* Fix artist DB mapping
* Fix playlist updates
* Remove bookmarks at the end of the test
* Remove remaining `orm` struct tags
* Fix user timestamps DB access
* Fix smart playlist evaluated_at DB access
* Fix search3
* fix(persistence): Update play_date on scrobble only when newer - #2262
Signed-off-by: Xidorn Quan <me@upsuper.org>
* expand iff
---------
Signed-off-by: Xidorn Quan <me@upsuper.org>
If files cannot be sorted by disc and track id, try by artist then
title.
One use case is a loose compilation of files with same album, album
artist, and no track numbers. File order was then undetermined, in
practice depended on insertion order in the database.
- Create model.DataStore, with provision for transactions
- Change all layers dependencies on repositories to use DataStore
- Implemented persistence.SQLStore
- Removed iTunes Bridge/Importer support
Deluan Quintão