📜 中文文档
Methods and paths for obtaining, managing, and utilizing open data for hydrological scientific computations.
- Free software: MIT license
- Documentation: https://hydro-opendata.readthedocs.io/en/latest/
In the era of artificial intelligence, data-driven hydrological models have been extensively researched and applied. With the advancements in remote sensing technologies and the trend towards open data sharing, accessing data has become more straightforward with a plethora of options. For researchers, questions like what data is required, what data can be accessed, where to download it, how to read it, and how to process it, are crucial. This repository aims to address these concerns.
This repository primarily focuses on external open data, categorizing data types, and creating a list. It aims to build a data flow and its tech stack that can seamlessly "download-store-process-read-write-visualize" the data.
From our current understanding, the external data suitable for hydrological modeling includes but is not limited to:
| Primary Category | Secondary Category | Update Frequency | Data Structure | Example |
|---|---|---|---|---|
| Basic Geography | Hydrological Elements | Static | Vector | Watershed boundary, site |
| Terrain | Static | Raster | DEM, flow direction, land use | |
| Weather & Meteorology | Reanalysis | Dynamic | Raster | ERA5 |
| Near Real-Time | Dynamic | Raster | GPM | |
| Forecast | Rolling | Raster | GFS | |
| Imagery | Satellite Remote Sensing | Dynamic | Raster | Landsat, Sentinel, MODIS |
| Street View Images | Static | Multimedia | ||
| Surveillance Videos | Dynamic | Multimedia | ||
| Drone Footage | Dynamic | Multimedia | ||
| Crowdsourced Data | POI | Static | Vector | Baidu Map |
| Social Networks | Dynamic | Multimedia | ||
| Hydrological Data | River Flow Data | Dynamic | Tabular | GRDC |
Data can be categorized based on their update frequency into static and dynamic data.
From a structural perspective, data can be classified into vector, raster, and multimedia (unstructured data).
Data inventory and its metadata. Returns a data list based on AOI.
Downloads data from external sources. Depending on the data source, the download methods may vary, including:
- Integration with official APIs, e.g., bmi_era5
- Retrieving data download links, e.g., Herbie, MultiEarth, Satpy. Most cloud data platforms like Microsoft, AWS, etc., organize data mostly as stac.
Preprocesses the data, such as watershed averaging, feature extraction, etc.
Uses kerchunk to convert different format data to zarr format and stores it in MinIO server. This enables cross-file reading and enhances data reading efficiency.
After data processing in MinIO, it supports cross-file reading. Just provide data type, time range, and spatial range parameters to fetch the data.
For remote sensing imagery, due to the vast amount of data, it's not feasible to download and read each file. One can use stac+stackstac to directly read Sentinel or Landsat data into an xarray dataset.
Integrates commonly used GIS tools, such as Kriging interpolation, Thiessen polygons, etc.
- Kriging interpolation: PyKrige
- Thiessen polygon: WhiteboxTools.VoronoiDiagram, scipy.spatial.Voronoi
- Watershed delineation: Rapid Watershed Delineation using an Automatic Outlet Relocation Algorithm, High-performance watershed delineation algorithm for GPU using CUDA and OpenMP
- Watershed averaging: plotting and creation of masks of spatial regions
Use leafmap to display geospatial data within the Jupyter platform.