rcjackson / arm-test-data

Test data for the atmospheric data community toolkit (ACT)

arm-test-data

A place to share atmospheric data with the community, shared throughout the Atmospheric Radiation Measurement user facility and beyond!

Sample data sets

These files are used as sample data in openradar examples/notebooks and are downloaded by arm-test-data package:

• 201509021500.bi
• AAFNAV_COR_20181104_R0.ict
• NEON.D18.BARR.DP1.00002.001.000.010.001.SAAT_1min.2022-10.expanded.20221107T205629Z.csv
• NEON.D18.BARR.DP1.00002.001.sensor_positions.20221107T205629Z.csv
• NEON.D18.BARR.DP1.00002.001.variables.20221201T110553Z.csv
• anltwr_mar19met.data
• ayp22199.21m
• ayp22200.00m
• brw21001.dat
• brw_12_2020_hour.dat
• brw_CCl4_Day.dat
• co2_brw_surface-insitu_1_ccgg_MonthlyData.txt
• ctd21125.15w
• ctd22187.00t.txt
• enametC1.b1.20221109.000000.cdf
• gucmetM1.b1.20230301.000000.cdf
• list_of_files.txt
• maraosmetM1.a1.20180201.000000.nc
• marirtsstM1.b1.20190320.000000.nc
• marnavM1.a1.20180201.000000.nc
• met_brw_insitu_1_obop_hour_2020.txt
• met_lcl.nc
• mosaossp2M1.00.20191216.000601.raw.20191216000000.ini
• mosaossp2M1.00.20191216.130601.raw.20191216x193.sp2b
• mosaossp2auxM1.00.20191217.010801.raw.20191216000000.hk
• nsacloudphaseC1.c1.20180601.000000.nc
• nsasurfspecalb1mlawerC1.c1.20160609.080000.nc
• sgp30ebbrE13.b1.20190601.000000.nc
• sgp30ebbrE32.b1.20191125.000000.nc
• sgp30ebbrE32.b1.20191130.000000.nc
• sgp30ecorE14.b1.20190601.000000.cdf
• sgpaerich1C1.b1.20190501.000342.nc
• sgpaosccn2colaE13.b1.20170903.000000.nc
• sgpbrsC1.b1.20190705.000000.cdf
• sgpceilC1.b1.20190101.000000.nc
• sgpco2flx4mC1.b1.20201007.001500.nc
• sgpdlppiC1.b1.20191015.120023.cdf
• sgpdlppiC1.b1.20191015.121506.cdf
• sgpirt25m20sC1.a0.20190601.000000.cdf
• sgpmetE13.b1.20190101.000000.cdf
• sgpmetE13.b1.20190102.000000.cdf
• sgpmetE13.b1.20190103.000000.cdf
• sgpmetE13.b1.20190104.000000.cdf
• sgpmetE13.b1.20190105.000000.cdf
• sgpmetE13.b1.20190106.000000.cdf
• sgpmetE13.b1.20190107.000000.cdf
• sgpmetE13.b1.20190508.000000.cdf
• sgpmetE13.b1.20210401.000000.csv
• sgpmetE13.b1.yaml
• sgpmetE15.b1.20190508.000000.cdf
• sgpmetE31.b1.20190508.000000.cdf
• sgpmetE32.b1.20190508.000000.cdf
• sgpmetE33.b1.20190508.000000.cdf
• sgpmetE34.b1.20190508.000000.cdf
• sgpmetE35.b1.20190508.000000.cdf
• sgpmetE36.b1.20190508.000000.cdf
• sgpmetE37.b1.20190508.000000.cdf
• sgpmetE38.b1.20190508.000000.cdf
• sgpmetE39.b1.20190508.000000.cdf
• sgpmetE40.b1.20190508.000000.cdf
• sgpmetE9.b1.20190508.000000.cdf
• sgpmet_no_time.nc
• sgpmet_test_time.nc
• sgpmfrsr7nchE11.b1.20210329.070000.nc
• sgpmmcrC1.b1.1.cdf
• sgpmmcrC1.b1.2.cdf
• sgpmplpolfsC1.b1.20190502.000000.cdf
• sgprlC1.a0.20160131.000000.nc
• sgpsebsE14.b1.20190601.000000.cdf
• sgpsirsE13.b1.20190101.000000.cdf
• sgpsondewnpnC1.b1.20190101.053200.cdf
• sgpstampE13.b1.20200101.000000.nc
• sgpstampE31.b1.20200101.000000.nc
• sgpstampE32.b1.20200101.000000.nc
• sgpstampE33.b1.20200101.000000.nc
• sgpstampE34.b1.20200101.000000.nc
• sgpstampE9.b1.20200101.000000.nc
• sodar.20230404.mnd
• twpsondewnpnC3.b1.20060119.050300.custom.cdf
• twpsondewnpnC3.b1.20060119.112000.custom.cdf
• twpsondewnpnC3.b1.20060119.163300.custom.cdf
• twpsondewnpnC3.b1.20060119.231600.custom.cdf
• twpsondewnpnC3.b1.20060120.043800.custom.cdf
• twpsondewnpnC3.b1.20060120.111900.custom.cdf
• twpsondewnpnC3.b1.20060120.170800.custom.cdf
• twpsondewnpnC3.b1.20060120.231500.custom.cdf
• twpsondewnpnC3.b1.20060121.051500.custom.cdf
• twpsondewnpnC3.b1.20060121.111600.custom.cdf
• twpsondewnpnC3.b1.20060121.171600.custom.cdf
• twpsondewnpnC3.b1.20060121.231600.custom.cdf
• twpsondewnpnC3.b1.20060122.052600.custom.cdf
• twpsondewnpnC3.b1.20060122.111500.custom.cdf
• twpsondewnpnC3.b1.20060122.171800.custom.cdf
• twpsondewnpnC3.b1.20060122.232600.custom.cdf
• twpsondewnpnC3.b1.20060123.052500.custom.cdf
• twpsondewnpnC3.b1.20060123.111700.custom.cdf
• twpsondewnpnC3.b1.20060123.171600.custom.cdf
• twpsondewnpnC3.b1.20060123.231500.custom.cdf
• twpsondewnpnC3.b1.20060124.051500.custom.cdf
• twpsondewnpnC3.b1.20060124.111800.custom.cdf
• twpsondewnpnC3.b1.20060124.171700.custom.cdf
• twpsondewnpnC3.b1.20060124.231500.custom.cdf
• twpvisstgridirtemp.c1.20050705.002500.nc
• vdis.b1

Adding new datasets

To add a new dataset file, please follow these steps:

1. Add the dataset file to the data/ directory
2. From the command line, run python make_registry.py script to update the registry file residing in arm-test-data/registry.txt
3. Commit and push your changes to GitHub

Using datasets in notebooks and/or scripts

• Ensure the arm-test-data package is installed in your environment

  python -m pip install arm-test-data
  
  # or
  
  python -m pip install git+https://github.com/ARM-DOE/arm-test-data
  
  # or
  
  conda install -c conda-forge arm-test-data

• Import DATASETS and inspect the registry to find out which datasets are available

  In [1]: from arm_test_data import DATASETS
  
  In [2]: DATASETS.registry_files
  Out[2]: ['sample_file.nc`]

• To fetch a data file of interest, use the .fetch method and provide the filename of the data file. This will

  • download and cache the file if it doesn't exist already.
  • retrieve and return the local path

  In [4]: filepath = DATASETS.fetch('sample_data.nc')
  
  In [5]: filepath
  Out[5]: '/Users/mgrover/Library/Caches/arm-test-data/sample_sgp_data.nc'

• Once you have access to the local filepath, you can then use it to load your dataset into pandas or xarray or your package of choice:

  In [6]: radar = pyart.io.read(filepath)

Changing the default data cache location

The default cache location (where the data are saved on your local system) is dependent on the operating system. You can use the locate() method to identify it:

from arm_test_data import locate
locate()

The location can be overwritten by the ACT_TEST_DATA_DIR environment variable to the desired destination.