AllenInstitute / npc_lims

Tools to fetch and update paths, metadata and state for Mindscope Neuropixels sessions, in the cloud.

npc_lims

neuropixels cloud lab information management system Tools to fetch and update paths, metadata and state for Mindscope Neuropixels sessions, in the cloud.

quickstart

• make a new Python >=3.9 virtual environment with conda or venv (lighter option, since this package does not require pandas, numpy etc.):

  python -m venv .venv

• activate the virtual environment:

  • Windows

  .venv\scripts\activate

  • Unix

  source .venv/bin/activate.sh

• install the package:

  python -m pip install npc_lims

• setup credentials

  • required environment variables:
    • AWS S3
      • AWS_DEFAULT_REGION
      • AWS_ACCESS_KEY_ID
      • AWS_SECRET_ACCESS_KEY
      • to find and read files on S3
      • must have read access on relevant aind buckets
      • can be in a standard ~/.aws location, as used by AWS CLI or boto3
    • CodeOcean API
      • CODE_OCEAN_API_TOKEN
      • CODE_OCEAN_DOMAIN
      • to find processed data in "data assets" via the Codeocean API
      • generated in CodeOcean:
        • right click on Account (bottom left, person icon)
        • click User Secrets - these are secrets than can be made available as environment variables in CodeOcean capsules
        • go to Access Tokens and click Generate new token - this is for programatically querying CodeOcean's databases
          • in Token Name enter Codeocean API (read) and check read on capsules and datasets
          • a token will be generated: click copy (storing it in a password manager, if you use one)
        • head back to User Secrets where we'll paste it into a new secret via Add secret > API credentials - in description enter Codeocean API (read) - in API key enter CODE_OCEAN_API_KEY - in API secret paste the copied secret from before (should start with cop_...) CODE_OCEAN_DOMAIN is the codeocean https address, up to and including .org
  • environment variables can also be specified in a file named .env in the current working directory
    • example: https://www.dotenv.org/docs/security/env.html
    • be very careful that this file does not get pushed to public locations, e.g. github
      • if using git, add it to a .gitignore file in your project's root directory:

      .env*

• now in Python we can find sessions that are available to work with:

      >>> import npc_lims;
  
  # get a sequence of `SessionInfo` dataclass instances, one per session:
      >>> tracked_sessions: tuple[npc_lims.SessionInfo, ...] = npc_lims.get_session_info()
  
  # each `SessionInfo` instance has minimal metadata about its session:
      >>> tracked_sessions[0]                 # doctest: +SKIP
      npc_lims.SessionInfo(id='626791_2022-08-15', subject=626791, date='2022-08-15', idx=0, project='DRPilotSession', is_ephys=True, is_sync=True, allen_path=PosixUPath('//allen/programs/mindscope/workgroups/dynamicrouting/PilotEphys/Task 2 pilot/DRpilot_626791_20220815'))
          >>> tracked_sessions[0].is_ephys        # doctest: +SKIP
      False
  
  # currently, we're only tracking behavior and ephys sessions that use variants of https://github.com/samgale/DynamicRoutingTask/blob/main/TaskControl.py:
      >>> all(s.date.year >= 2022 for s in tracked_sessions)
      True

• "tracked sessions" are discovered via 3 routes:

  • https://github.com/AllenInstitute/npc_lims/blob/main/tracked_sessions.yaml
  • \\allen\programs\mindscope\workgroups\dynamicrouting\DynamicRoutingTask\DynamicRoutingTraining.xlsx
  • \\allen\programs\mindscope\workgroups\dynamicrouting\DynamicRoutingTask\DynamicRoutingTrainingNSB.xlsx