series.apply big tiff / bad block
mmann1123 opened this issue · comments
I am running into an issue with large raster stacks, when I am applying my interpolator. Looks like maybe we just need to add a kw args parameter for series.apply. I will take a stab at it.
with gw.series(
a_grid, transfer_lib="numpy"
) as src: # , window_size=[128, 128],
src.apply(
func=interpolate_nan(
interp_type="linear",
missing_value=missing_data,
count=len(src.filenames),
),
outfile=f"./{band_name}/interpolated/S2_SR_interpolated_{grid}_{dates[0].strftime('%Y_%m')}_{dates[-1].strftime('%Y_%m')}.tif",
num_workers=6, # src.nchunks,
bands=1,
)The directory already exists. Skipping.
working on grid 0000000000-0000046592
['./B12/S2_SR_B12_M_2023_01-0000000000-0000046592.tif', './B12/S2_SR_B12_M_2023_02-0000000000-0000046592.tif', './B12/S2_SR_B12_M_2023_03-0000000000-0000046592.tif', './B12/S2_SR_B12_M_2023_04-0000000000-0000046592.tif', './B12/S2_SR_B12_M_2023_05-0000000000-0000046592.tif', './B12/S2_SR_B12_M_2023_06-0000000000-0000046592.tif', './B12/S2_SR_B12_M_2023_07-0000000000-0000046592.tif', './B12/S2_SR_B12_M_2023_08-0000000000-0000046592.tif']
[datetime.datetime(2023, 1, 1, 0, 0), datetime.datetime(2023, 2, 1, 0, 0), datetime.datetime(2023, 3, 1, 0, 0), datetime.datetime(2023, 4, 1, 0, 0), datetime.datetime(2023, 5, 1, 0, 0), datetime.datetime(2023, 6, 1, 0, 0), datetime.datetime(2023, 7, 1, 0, 0), datetime.datetime(2023, 8, 1, 0, 0)]
working on B12 0000000000-0000046592
NOTE: Dates are unknown, assuming regular time interval
86%|████████▌ | 4338/5032 [59:27<10:02, 1.15it/s] ERROR 1: TIFFAppendToStrip:Maximum TIFF file size exceeded. Use BIGTIFF=YES creation option.
ERROR 1: An error occurred while writing a dirty block from GDALRasterBand::IRasterIO
86%|████████▌ | 4339/5032 [59:28<09:29, 1.22it/s]
ERROR 1: TIFFAppendToStrip:Maximum TIFF file size exceeded. Use BIGTIFF=YES creation option.
ERROR 1: TIFFAppendToStrip:Maximum TIFF file size exceeded. Use BIGTIFF=YES creation option.
ERROR 1: [./B12/interpolated/S2_SR_interpolated_0000000000-0000046592_2023_01_2023_08.tif](https://file+.vscode-resource.vscode-cdn.net/home/mmann1123/Documents/github/YM_TZ_crop_classifier/B12/interpolated/S2_SR_interpolated_0000000000-0000046592_2023_01_2023_08.tif): WriteEncodedTile/Strip() failed.
ERROR 1: TIFFAppendToStrip:Maximum TIFF file size exceeded. Use BIGTIFF=YES creation option.
---------------------------------------------------------------------------
CPLE_AppDefinedError Traceback (most recent call last)
File rasterio/_io.pyx:1741, in rasterio._io.DatasetWriterBase.write()
File rasterio/_io.pyx:174, in rasterio._io.io_multi_band()
File rasterio/_err.pyx:195, in rasterio._err.exc_wrap_int()
CPLE_AppDefinedError: An error occurred while writing a dirty block from GDALRasterBand::IRasterIO
During handling of the above exception, another exception occurred:
RasterioIOError Traceback (most recent call last)
[/home/mmann1123/Documents/github/YM_TZ_crop_classifier/2_xr_fresh_extraction.py](https://file+.vscode-resource.vscode-cdn.net/home/mmann1123/Documents/github/YM_TZ_crop_classifier/2_xr_fresh_extraction.py) in line 50
[126](file:///home/mmann1123/Documents/github/YM_TZ_crop_classifier/2_xr_fresh_extraction.py?line=125) print(f"working on {band_name} {grid}")
[127](file:///home/mmann1123/Documents/github/YM_TZ_crop_classifier/2_xr_fresh_extraction.py?line=126) with gw.series(
[128](file:///home/mmann1123/Documents/github/YM_TZ_crop_classifier/2_xr_fresh_extraction.py?line=127) a_grid, transfer_lib="numpy"
[129](file:///home/mmann1123/Documents/github/YM_TZ_crop_classifier/2_xr_fresh_extraction.py?line=128) ) as src: # , window_size=[128, 128],
---> [130](file:///home/mmann1123/Documents/github/YM_TZ_crop_classifier/2_xr_fresh_extraction.py?line=129) src.apply(
[131](file:///home/mmann1123/Documents/github/YM_TZ_crop_classifier/2_xr_fresh_extraction.py?line=130) func=interpolate_nan(
[132](file:///home/mmann1123/Documents/github/YM_TZ_crop_classifier/2_xr_fresh_extraction.py?line=131) interp_type="linear",
[133](file:///home/mmann1123/Documents/github/YM_TZ_crop_classifier/2_xr_fresh_extraction.py?line=132) missing_value=missing_data,
[134](file:///home/mmann1123/Documents/github/YM_TZ_crop_classifier/2_xr_fresh_extraction.py?line=133) count=len(src.filenames),
[135](file:///home/mmann1123/Documents/github/YM_TZ_crop_classifier/2_xr_fresh_extraction.py?line=134) ),
[136](file:///home/mmann1123/Documents/github/YM_TZ_crop_classifier/2_xr_fresh_extraction.py?line=135) outfile=f"[./](https://file+.vscode-resource.vscode-cdn.net/home/mmann1123/Documents/github/YM_TZ_crop_classifier/){band_name}[/interpolated/S2_SR_interpolated_](https://file+.vscode-resource.vscode-cdn.net/interpolated/S2_SR_interpolated_){grid}_{dates[0].strftime('%Y_%m')}_{dates[-1].strftime('%Y_%m')}.tif",
...
[1293](file:///home/mmann1123/miniconda3/envs/xr_fresh/lib/python3.9/site-packages/geowombat/core/api.py?line=1292) )
File rasterio/_io.pyx:1743, in rasterio._io.DatasetWriterBase.write()
RasterioIOError: Read or write failed. An error occurred while writing a dirty block from GDALRasterBand::IRasterIO