feat(scanner): implement selective folder scanning and file system watcher improvements (#4674)

* feat: Add selective folder scanning capability Implement targeted scanning of specific library/folder pairs without full recursion. This enables efficient rescanning of individual folders when changes are detected, significantly reducing scan time for large libraries. Key changes: - Add ScanTarget struct and ScanFolders API to Scanner interface - Implement CLI flag --targets for specifying libraryID:folderPath pairs - Add FolderRepository.GetByPaths() for batch folder info retrieval - Create loadSpecificFolders() for non-recursive directory loading - Scope GC operations to affected libraries only (with TODO for full impl) - Add comprehensive tests for selective scanning behavior The selective scan: - Only processes specified folders (no subdirectory recursion) - Maintains library isolation - Runs full maintenance pipeline scoped to affected libraries - Supports both full and quick scan modes Examples: navidrome scan --targets "1:Music/Rock,1:Music/Jazz" navidrome scan --full --targets "2:Classical" * feat(folder): replace GetByPaths with GetFolderUpdateInfo for improved folder updates retrieval Signed-off-by: Deluan <deluan@navidrome.org> * test: update parseTargets test to handle folder names with spaces Signed-off-by: Deluan <deluan@navidrome.org> * refactor(folder): remove unused LibraryPath struct and update GC logging message Signed-off-by: Deluan <deluan@navidrome.org> * refactor(folder): enhance external scanner to support target-specific scanning Signed-off-by: Deluan <deluan@navidrome.org> * refactor(scanner): simplify scanner methods Signed-off-by: Deluan <deluan@navidrome.org> * feat(watcher): implement folder scanning notifications with deduplication Signed-off-by: Deluan <deluan@navidrome.org> * refactor(watcher): add resolveFolderPath function for testability Signed-off-by: Deluan <deluan@navidrome.org> * feat(watcher): implement path ignoring based on .ndignore patterns Signed-off-by: Deluan <deluan@navidrome.org> * refactor(scanner): implement IgnoreChecker for managing .ndignore patterns Signed-off-by: Deluan <deluan@navidrome.org> * refactor(ignore_checker): rename scanner to lineScanner for clarity Signed-off-by: Deluan <deluan@navidrome.org> * refactor(scanner): enhance ScanTarget struct with String method for better target representation Signed-off-by: Deluan <deluan@navidrome.org> * fix(scanner): validate library ID to prevent negative values Signed-off-by: Deluan <deluan@navidrome.org> * refactor(scanner): simplify GC method by removing library ID parameter Signed-off-by: Deluan <deluan@navidrome.org> * feat(scanner): update folder scanning to include all descendants of specified folders Signed-off-by: Deluan <deluan@navidrome.org> * feat(subsonic): allow selective scan in the /startScan endpoint Signed-off-by: Deluan <deluan@navidrome.org> * refactor(scanner): update CallScan to handle specific library/folder pairs Signed-off-by: Deluan <deluan@navidrome.org> * refactor(scanner): streamline scanning logic by removing scanAll method Signed-off-by: Deluan <deluan@navidrome.org> * test: enhance mockScanner for thread safety and improve test reliability Signed-off-by: Deluan <deluan@navidrome.org> * refactor(scanner): move scanner.ScanTarget to model.ScanTarget Signed-off-by: Deluan <deluan@navidrome.org> * refactor: move scanner types to model,implement MockScanner Signed-off-by: Deluan <deluan@navidrome.org> * refactor(scanner): update scanner interface and implementations to use model.Scanner Signed-off-by: Deluan <deluan@navidrome.org> * refactor(folder_repository): normalize target path handling by using filepath.Clean Signed-off-by: Deluan <deluan@navidrome.org> * test(folder_repository): add comprehensive tests for folder retrieval and child exclusion Signed-off-by: Deluan <deluan@navidrome.org> * refactor(scanner): simplify selective scan logic using slice.Filter Signed-off-by: Deluan <deluan@navidrome.org> * refactor(scanner): streamline phase folder and album creation by removing unnecessary library parameter Signed-off-by: Deluan <deluan@navidrome.org> * refactor(scanner): move initialization logic from phase_1 to the scanner itself Signed-off-by: Deluan <deluan@navidrome.org> * refactor(tests): rename selective scan test file to scanner_selective_test.go Signed-off-by: Deluan <deluan@navidrome.org> * feat(configuration): add DevSelectiveWatcher configuration option Signed-off-by: Deluan <deluan@navidrome.org> * feat(watcher): enhance .ndignore handling for folder deletions and file changes Signed-off-by: Deluan <deluan@navidrome.org> * docs(scanner): comments Signed-off-by: Deluan <deluan@navidrome.org> * refactor(scanner): enhance walkDirTree to support target folder scanning Signed-off-by: Deluan <deluan@navidrome.org> * fix(scanner, watcher): handle errors when pushing ignore patterns for folders Signed-off-by: Deluan <deluan@navidrome.org> * Update scanner/phase_1_folders.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * refactor(scanner): replace parseTargets function with direct call to scanner.ParseTargets Signed-off-by: Deluan <deluan@navidrome.org> * test(scanner): add tests for ScanBegin and ScanEnd functionality Signed-off-by: Deluan <deluan@navidrome.org> * fix(library): update PRAGMA optimize to check table sizes without ANALYZE Signed-off-by: Deluan <deluan@navidrome.org> * test(scanner): refactor tests Signed-off-by: Deluan <deluan@navidrome.org> * feat(ui): add selective scan options and update translations Signed-off-by: Deluan <deluan@navidrome.org> * feat(ui): add quick and full scan options for individual libraries Signed-off-by: Deluan <deluan@navidrome.org> * feat(ui): add Scan buttonsto the LibraryList Signed-off-by: Deluan <deluan@navidrome.org> * feat(scan): update scanning parameters from 'path' to 'target' for selective scans. * refactor(scan): move ParseTargets function to model package * test(scan): suppress unused return value from SetUserLibraries in tests * feat(gc): enhance garbage collection to support selective library purging Signed-off-by: Deluan <deluan@navidrome.org> * fix(scanner): prevent race condition when scanning deleted folders When the watcher detects changes in a folder that gets deleted before the scanner runs (due to the 10-second delay), the scanner was prematurely removing these folders from the tracking map, preventing them from being marked as missing. The issue occurred because `newFolderEntry` was calling `popLastUpdate` before verifying the folder actually exists on the filesystem. Changes: - Move fs.Stat check before newFolderEntry creation in loadDir to ensure deleted folders remain in lastUpdates for finalize() to handle - Add early existence check in walkDirTree to skip non-existent target folders with a warning log - Add unit test verifying non-existent folders aren't removed from lastUpdates prematurely - Add integration test for deleted folder scenario with ScanFolders Fixes the issue where deleting entire folders (e.g., /music/AC_DC) wouldn't mark tracks as missing when using selective folder scanning. * refactor(scan): streamline folder entry creation and update handling Signed-off-by: Deluan <deluan@navidrome.org> * feat(scan): add '@Recycle' (QNAP) to ignored directories list Signed-off-by: Deluan <deluan@navidrome.org> * fix(log): improve thread safety in logging level management * test(scan): move unit tests for ParseTargets function Signed-off-by: Deluan <deluan@navidrome.org> * review Signed-off-by: Deluan <deluan@navidrome.org> --------- Signed-off-by: Deluan <deluan@navidrome.org> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: deluan <deluan.quintao@mechanical-orchard.com>
2025-11-14 22:15:43 -05:00
parent bca76069c3
commit 28d5299ffc
52 changed files with 3221 additions and 374 deletions
@@ -10,6 +10,8 @@ import {
 } from 'react-admin'
 import { useMediaQuery } from '@material-ui/core'
 import { List, DateField, useResourceRefresh, SizeField } from '../common'
+import LibraryListBulkActions from './LibraryListBulkActions'
+import LibraryListActions from './LibraryListActions'

 const LibraryFilter = (props) => (
  <Filter {...props} variant={'outlined'}>
@@ -26,8 +28,9 @@ const LibraryList = (props) => {
      {...props}
      sort={{ field: 'name', order: 'ASC' }}
      exporter={false}
-      bulkActionButtons={false}
+      bulkActionButtons={!isXsmall && <LibraryListBulkActions />}
      filters={<LibraryFilter />}
+      actions={<LibraryListActions />}
    >
      {isXsmall ? (
        <SimpleList
@@ -0,0 +1,30 @@
+import React, { cloneElement } from 'react'
+import { sanitizeListRestProps, TopToolbar } from 'react-admin'
+import LibraryScanButton from './LibraryScanButton'
+
+const LibraryListActions = ({
+  className,
+  filters,
+  resource,
+  showFilter,
+  displayedFilters,
+  filterValues,
+  ...rest
+}) => {
+  return (
+    <TopToolbar className={className} {...sanitizeListRestProps(rest)}>
+      {filters &&
+        cloneElement(filters, {
+          resource,
+          showFilter,
+          displayedFilters,
+          filterValues,
+          context: 'button',
+        })}
+      <LibraryScanButton fullScan={false} />
+      <LibraryScanButton fullScan={true} />
+    </TopToolbar>
+  )
+}
+
+export default LibraryListActions
@@ -0,0 +1,11 @@
+import React from 'react'
+import LibraryScanButton from './LibraryScanButton'
+
+const LibraryListBulkActions = (props) => (
+  <>
+    <LibraryScanButton fullScan={false} {...props} />
+    <LibraryScanButton fullScan={true} {...props} />
+  </>
+)
+
+export default LibraryListBulkActions
@@ -0,0 +1,77 @@
+import React, { useState } from 'react'
+import PropTypes from 'prop-types'
+import {
+  Button,
+  useNotify,
+  useRefresh,
+  useTranslate,
+  useUnselectAll,
+} from 'react-admin'
+import { useSelector } from 'react-redux'
+import SyncIcon from '@material-ui/icons/Sync'
+import CachedIcon from '@material-ui/icons/Cached'
+import subsonic from '../subsonic'
+
+const LibraryScanButton = ({ fullScan, selectedIds, className }) => {
+  const [loading, setLoading] = useState(false)
+  const notify = useNotify()
+  const refresh = useRefresh()
+  const translate = useTranslate()
+  const unselectAll = useUnselectAll()
+  const scanStatus = useSelector((state) => state.activity.scanStatus)
+
+  const handleClick = async () => {
+    setLoading(true)
+    try {
+      // Build scan options
+      const options = { fullScan }
+
+      // If specific libraries are selected, scan only those
+      // Format: "libraryID:" to scan entire library (no folder path specified)
+      if (selectedIds && selectedIds.length > 0) {
+        options.target = selectedIds.map((id) => `${id}:`)
+      }
+
+      await subsonic.startScan(options)
+      const notificationKey = fullScan
+        ? 'resources.library.notifications.fullScanStarted'
+        : 'resources.library.notifications.quickScanStarted'
+      notify(notificationKey, 'info')
+      refresh()
+
+      // Unselect all items after successful scan
+      unselectAll('library')
+    } catch (error) {
+      notify('resources.library.notifications.scanError', 'warning')
+    } finally {
+      setLoading(false)
+    }
+  }
+
+  const isDisabled = loading || scanStatus.scanning
+
+  const label = fullScan
+    ? translate('resources.library.actions.fullScan')
+    : translate('resources.library.actions.quickScan')
+
+  const icon = fullScan ? <CachedIcon /> : <SyncIcon />
+
+  return (
+    <Button
+      onClick={handleClick}
+      disabled={isDisabled}
+      label={label}
+      className={className}
+    >
+      {icon}
+    </Button>
+  )
+}
+
+LibraryScanButton.propTypes = {
+  fullScan: PropTypes.bool.isRequired,
+  selectedIds: PropTypes.array,
+  className: PropTypes.string,
+}
+
+export default LibraryScanButton