Giters

kgorlen / lightroom

Windows bat and documentation for storing lightroom catalogs on NAS

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Lightroom Classic Catalogs on Network Attached Storage

Background

I currently have over 37K photos (>1TB) stored on a QNAP TS-253D NAS (Network Attached Storage) device with a capacity of 6TB (mirrored). The NAS is connected by 10Gbps Ethernet (1Gbps Ethernet also works well) to a custom-built PC and a mini PC, both running Windows 10 Home. To make storage management as simple as possible, the PC and laptop are configured dataless: virtually all files that must be backed up are stored on the NAS[1], which runs Duplicacy[2] to back up files nightly to external RAID 5 DAS (Direct Attached Storage) and off-site to Backblaze B2[3]. In particular, each user’s Pictures folder is located in their home folder on the NAS. The capacity of the NAS can be easily expanded by replacing the mirrored drives one-at-a-time with larger drives. The PCs don’t need much local storage, so they have only two 256GB SSDs, one for system files and applications and one for temporary files as needed by Photoshop and other applications.

I’ve been accessing the same Lightroom Classic (LrC) catalogs on a NAS since 2011, except for a few months in 2016 after replacing a Netgear ReadyNAS with a QNAP TS-251+, which had a defective SMB implementation that caused LrC and other failures. During this period I modified the lightroom.ffs_batch and full.ffs_batch files described below to copy the catalog .lrcat files in addition to the settings to/from an internal SSD as a temporary, unreliable workaround.

NOTE: I do not recommend QNAP: poor software quality, inadequate software regression testing, security vulnerabilities, bloatware, poor support, poor documentation.

CAUTION!

Lightroom Classic (LrC) normally will not access a catalog that resides on a network drive. In 2007, Adobe engineer Dan Tull[4][5] tested LrC catalogs on a corporate network drive by disconnecting the cable and observed catalog corruption, so Adobe coded LrC to disallow them on network drives. Technically, however, network drives are designed to look, smell, feel, and act like your own disk space.[6] An LrC catalog is an SQLite[7] database, the integrity of which is maintained via standard Windows file operations[8], which are also supported by SMB[9], the protocol Windows uses to access files on network drives. Thus, catalog corruption is the result of hardware failure or software defects, not inherent technical limitations or incompatibilities.[10][11][12][13] But since more hardware and software is involved in accessing network-attached storage than internal or directly-attached external storage, the risk of catalog corruption is greater. However, there is also risk in other methods proposed[14] for using a catalog from multiple machines, such as inadvertently disconnecting an external hard drive while Lightroom is running or while synchronizing with Dropbox, Google Drive, One Drive, etc.

Also, LrC currently requires that the Previews.lrdata folder must reside in the same folder as the catalog. Thus, when the catalog is on a network drive, previews will also reside there, slowing performance.

Setting up the NAS:

  • with an uninterruptible power supply (UPS), to protect against power failures,

  • with RAID, to protect against single-drive failures, and

  • as an "always up" device automatically running nightly backups, when LrC catalogs are less likely to be in use,

provides protections not typically provided on desktop PCs that offset the added risk of storing and accessing catalogs on the NAS.

Basic Setup

LrC will create and access catalogs on a network folder (\\NAS0\home\Pictures\Lightroom in this example) by means of a Windows 10 junction (%HOMEPATH%\Lightroom in this example).

For each <user> on each machine running LrC:

  1. Run cmd as Administrator and execute the following commands:

    cd %HOMEPATH%\..\<user>
mklink /d Lightroom-symlink "\\NAS0\home\Pictures\Lightroom"
mklink /j Lightroom Lightroom-symlink

    Be sure to use the same junction name for a given user on all PCs if synchronizing LrC settings across PCs as described below.

  2. If necessary, move catalogs folders to the network folder, e.g. to %HOMEPATH%\Lightroom\Catalogs.

  3. Run LrC while holding the Alt key to open the Select Catalog dialog, and either Choose a Different Catalog or Create a New Catalog in a folder below the junction, e.g. to %HOMEPATH%\Lightroom\Catalogs\Master.

When LrC opens a catalog, it creates a .lrcat.lock file in the catalog’s folder, which prevents other instances of LrC from accessing the same catalog at the same time.[15] However, if LrC crashes or is running when "Sleep", or "Switch user" is done, the catalog is left locked and cannot be accessed by other users or machines. Also, while LrC has a catalog open, any copy made of the catalog by another application such as Windows File History, Backup and Restore, Acronis, Macrium, Carbonite, CrashPlan, Google Drive, One Drive, Dropbox, etc. is not guaranteed to capture the catalog in a consistent state; thus, such a backup is not reliably valid.

  • Periodically check the integrity of and back up catalogs with LrC, and back up these backups.

  • Install gracefulexit.bat (instructions are in that file) and use it to put a PC to sleep, switch user, and logoff.

  • OPTIONAL: To reduce occurrences of abandoned file locks[16], disable Windows oplocks on .lock files (and FreeFileFileSync sync.ffs_db databases, see below) by adding the following to /etc/smb.conf on the NAS:

    veto oplock files = /*.lock/sync.ffs_db/

OPTIONAL: Synchronize LrC settings across all PCs

LrC changes settings files[17] frequently when running, so synchronization is implemented by running LrC via a shortcut which, in turn, runs a file synchronization utility (FreeFileSync in this case) before and after running LrC to sync settings from/to a folder on the NAS:

C:\Windows\System32\cmd.exe /c ""%AppData%\FreeFileSync\Config\lightroom.ffs_batch" && "%ProgramFiles%\Adobe\Adobe Lightroom Classic\Lightroom.exe" & "%AppData%\FreeFileSync\Config\lightroom.ffs_batch""

  1. Install FreeFileSync (FFS).

  2. Create %AppData%\FreeFileSync\Config folders to store each user’s lightroom.ffs_batch file.

  3. Create %AppData%\FreeFileSync\Logs folders to store each user’s FFS logs.

  4. Create folders on the NAS for each user’s LrC settings, e.g.:

    \\<my NAS>\home\AppData\Roaming\Adobe\Lightroom

  5. Edit lightroom.ffs_batch with FFS to change NAS0 to the name of your NAS.

  6. For each user, run LrC on the machine on which the latest LrC settings reside to sync these to the NAS, then open lightroom.ffs_batch with FFS on all other machines and manually sync the LrC settings from the NAS to %AppData%\Adobe\Lightroom.

  7. Pin the lightroom shortcut to each user’s Start Menu and/or taskbar (https://www.digitalcitizen.life/how-pin-special-windows-shortcuts-taskbar).

OPTIONAL: Synchronize LrC plug-ins across all PCs

LrC plug-ins are synchronized when updated by a file synchronization utility (FreeFileSync in this case). For each machine:

  1. Install FreeFileSync (FFS).

  2. Create the %CommonProgramFiles%\FreeFileSync\Config folder to store the Realtime.ffs_batch and Realtime.ffs_real files.

  3. Create the %CommonProgramFiles%\FreeFileSync\Log folder to store FFS logs.

  4. Create a folder on the NAS to store the FFS Realtime.ffs_batch and Realtime.ffs_real files, e.g.:

    \\<my NAS>\<my path>\FreeFileSync\Common Files\Config

  5. Create a folder on the NAS to store LrC plug-ins, e.g.:

    \\<my NAS>\<my path>\Lightroom\LR Plugins

  6. Assure that a user account e.g. myaccount, has RW permission on:

    • %CommonProgramFiles%\LR Plugins and subfolders

    • %CommonProgramFiles%\FreeFileSync and subfolders

    • \\<my NAS>\<my path>\Lightroom\LR Plugins and subfolders

    • \\<my NAS>\<my path>\FreeFileSync\Common Files\Config and subfolders

  7. Edit %CommonProgramFiles%\FreeFileSync\Config\Realtime.ffs_batch with FFS to change the path names to match those of your NAS folders. A folder pair to synchronize %Public%\Pictures\Screen Saver Photos is included; remove if not wanted.

  8. Open Realtime.ffs_batch with FFS on the machine on which the latest LrC plug-ins reside, sync these to the NAS, then open Realtime.ffs_batch with FFS on all other machines and manually sync the LrC plug-ins from the NAS to %ProgramFiles%\Common Files\LR Plugins.

  9. Create the Realtime.ffs_real file (see https://freefilesync.org/manual.php?topic=realtimesync):

    • run "%PROGRAMFILES%\Program Files\FreeFileSync\RealTimeSync.exe"

    • File → Open %CommonProgramFiles%\FreeFileSync\Config\Realtime.ffs_batch

    • File → Save as Realtime.ffs_real

  10. Create a Task Scheduler task to run RealTimeSync at system startup:

    • Run under a user account (e.g. myaccount) that has read and write permission on all synced folders

    • Run whether user is logged on or not

    • Trigger: At startup

    • Action: Start a program "%ProgramFiles%\FreeFileSync\RealTimeSync.exe"

    • Add arguments: "%CommonProgramFiles%\FreeFileSync\Config\Realtime.ffs_real"

    • Settings: Do not start a new instance

      1. Restart, then check the %CommonProgramFiles%\FreeFileSync\Log folder to confirm that folders are synchronizing.

OPTIONAL: Synchronize Photoshop preferences across all machines

Photoshop, Bridge, CameraRaw, Color, LensCorrection, and Plugins preferences[18] are synchronized when updated by a file synchronization utility (FreeFileSync in this case), and also by gracefulexit.bat on sleep, switch user, and logoff.

  1. Create a folder on the NAS for each user’s Photoshop preferences, e.g.:

    \\<my NAS>\home\AppData\Roaming\Adobe

  2. Copy the following files to each user’s %AppData%\FreeFileSync\Config folder:

    full.ffs_batch
watched.ffs_batch

  3. Edit these files to change NAS0 to the name of your NAS. Folder pairs for the Affinity settings, Desktop, and Favorites are included; remove if not wanted.

  4. For each user, open full.ffs_batch with FFS on the machine on which the latest Photshop preferences reside, sync these to the NAS, then open full.ffs_batch with FFS on all other machines and manually sync the setting from the NAS to the local machine.

  5. Create the watched.ffs_real file (see https://freefilesync.org/manual.php?topic=realtimesync):

    • run "%PROGRAMFILES%\FreeFileSync\RealTimeSync.exe"

    • File → Open %AppData%\FreeFileSync\Config\watched.ffs_batch

    • File → Save as %AppData%\FreeFileSync\Config\watched.ffs_real

  6. Copy the RealTimeSync shortcut to each user’s Startup folder:

    %AppData%\Microsoft\Windows\Start Menu\Programs\Startup

  7. Login as each user and check the %AppData%\FreeFileSync\Logs folder to confirm that folders are synchronizing.

References

About

Windows bat and documentation for storing lightroom catalogs on NAS

Languages

Language:Batchfile 100.0%