Unidata / siphon

Siphon - A collection of Python utilities for retrieving atmospheric and oceanic data from remote sources, focusing on being able to retrieve data from Unidata data technologies, such as the THREDDS data server.

Home Page:https://unidata.github.io/siphon

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

issues getting CFS point data

Soudaaa opened this issue · comments

Hi all.

I've been using siphon ncss class to get CFS/CFSv2 data for quite a while. However, recently my code stopped working with no aparent reason.
Below is the code and the error message I get when trying to download the data:

cat = TDSCatalog(f'https://www.ncei.noaa.gov/thredds/catalog/model-cfs_v2_anl_6h_pgb/{time.year}/{time.year}0{time.month}' + f'/{time.year}0{time.month}{time.day}/'
                 f'catalog.xml?dataset=cfs_v2_anl_6h_pgb/{time.year}/{time.year}0{time.month}' + f'/{time.year}0{time.month}{time.day}' + '/cdas1.t' + f'0{time.hour}' + 'z.pgrbh01.grib2')
    
cat_dataset = cat.datasets[0]
ncss = cat_dataset.subset()
query = ncss.query()
query.lonlat_point(-52, -28)
query.variables('Geopotential_height_isobaric', 'Geopotential_height_surface', 'Potential_temperature_sigma', 'Specific_humidity_isobaric',
                'Pressure_msl', 'Pressure_surface', 'Relative_humidity_sigma', 'Temperature_sigma', 'Temperature_isobaric',
                'v-component_of_wind_sigma', 'u-component_of_wind_sigma', 'v-component_of_wind_isobaric', 'u-component_of_wind_isobaric')

data = ncss.get_data(query)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[5], line 12
      7 query.lonlat_point(-52, -28)
      8 query.variables('Geopotential_height_isobaric', 'Geopotential_height_surface', 'Potential_temperature_sigma', 'Specific_humidity_isobaric',
      9                 'Pressure_msl', 'Pressure_surface', 'Relative_humidity_sigma', 'Temperature_sigma', 'Temperature_isobaric',
     10                 'v-component_of_wind_sigma', 'u-component_of_wind_sigma', 'v-component_of_wind_isobaric', 'u-component_of_wind_isobaric')
---> 12 data = ncss.get_data(query)

File ~/miniconda3/envs/py39/lib/python3.9/site-packages/siphon/ncss.py:115, in NCSS.get_data(self, query)
     94 """Fetch parsed data from a THREDDS server using NCSS.
     95 
     96 Requests data from the NCSS endpoint given the parameters in `query` and
   (...)
    112 
    113 """
    114 resp = self.get_query(query)
--> 115 return response_handlers(resp, self.unit_handler)

File ~/miniconda3/envs/py39/lib/python3.9/site-packages/siphon/ncss.py:294, in ResponseRegistry.__call__(self, resp, unit_handler)
    292 """Process the HTTP response using the appropriate handler."""
    293 mimetype = resp.headers['content-type'].split(';')[0]
--> 294 return self._reg.get(mimetype, self.default)(resp.content, unit_handler)

File ~/miniconda3/envs/py39/lib/python3.9/site-packages/siphon/ncss.py:405, in parse_csv_response(data, unit_handler)
    402 @response_handlers.register('text/plain')
    403 def parse_csv_response(data, unit_handler):
    404     """Handle CSV-formatted HTTP responses."""
--> 405     return squish([parse_csv_dataset(d, unit_handler) for d in data.split(b'\n\n')])

File ~/miniconda3/envs/py39/lib/python3.9/site-packages/siphon/ncss.py:405, in <listcomp>(.0)
    402 @response_handlers.register('text/plain')
    403 def parse_csv_response(data, unit_handler):
    404     """Handle CSV-formatted HTTP responses."""
--> 405     return squish([parse_csv_dataset(d, unit_handler) for d in data.split(b'\n\n')])

File ~/miniconda3/envs/py39/lib/python3.9/site-packages/siphon/ncss.py:432, in parse_csv_dataset(data, handle_units)
    430 fobj = BytesIO(data)
    431 names, units = parse_csv_header(fobj.readline().decode('utf-8'))
--> 432 arrs = np.genfromtxt(fobj, dtype=None, names=names, delimiter=',',
    433                      converters={'date': lambda s: parse_iso_date(s.decode('utf-8'))})
    434 d = {}
    435 for f in arrs.dtype.fields:

File ~/miniconda3/envs/py39/lib/python3.9/site-packages/numpy/lib/npyio.py:2291, in genfromtxt(fname, dtype, comments, delimiter, skip_header, skip_footer, converters, missing_values, filling_values, usecols, names, excludelist, deletechars, replace_space, autostrip, case_sensitive, defaultfmt, unpack, usemask, loose, invalid_raise, max_rows, encoding, ndmin, like)
   2289 # Raise an exception ?
   2290 if invalid_raise:
-> 2291     raise ValueError(errmsg)
   2292 # Issue a warning ?
   2293 else:
   2294     warnings.warn(errmsg, ConversionWarning, stacklevel=2)

ValueError: Some errors were detected !
    Line #38 (got 7 columns instead of 10)
    Line #39 (got 7 columns instead of 10)
    Line #40 (got 9 columns instead of 10)
    Line #41 (got 9 columns instead of 10)

I may have run into this issue after updating some package. I tried creating a new environment but the error persists.