Response MediaType CSV failing
jonaraphael opened this issue · comments
Sometimes the CSV format download seems to fail. f=geojson
does NOT fail in these same circumstances.
Here it works (4 rows of data):
https://api.cerulean.skytruth.org/collections/public.aoi_type/items
https://api.cerulean.skytruth.org/collections/public.aoi_type/items?f=csv
Here it fails (expect 141 rows of data, but returns 2 rows of garbled data):
https://api.cerulean.skytruth.org/collections/public.slick_plus/items?sortby=slick_timestamp&datetime=2023-12-01T00:00:00Z/2023-12-08T00:00:00Z&filter=machine_confidence%20GTE%200.6%20AND%20area%20GTE%2020000000
https://api.cerulean.skytruth.org/collections/public.slick_plus/items?sortby=slick_timestamp&datetime=2023-12-01T00:00:00Z/2023-12-08T00:00:00Z&filter=machine_confidence%20GTE%200.6%20AND%20area%20GTE%2020000000&f=csv
@jonaraphael thanks for the report,
It's interesting because using other streaming
output format like ndjson
seems to work fine
$ curl https://api.cerulean.skytruth.org/collections/public.slick_plus/items?sortby=slick_timestamp&datetime=2023-12-01T00:00:00Z/2023-12-08T00:00:00Z&filter=machine_confidence%20GTE%200.6%20AND%20area%20GTE%2020000000&f=ndjson | wc -l
10
$ curl https://api.cerulean.skytruth.org/collections/public.slick_plus/items?sortby=slick_timestamp&datetime=2023-12-01T00:00:00Z/2023-12-08T00:00:00Z&filter=machine_confidence%20GTE%200.6%20AND%20area%20GTE%2020000000&f=csv | wc -l
2
Found the bug 🙏
Line 766 in 8483d54
curl https://api.cerulean.skytruth.org/collections/public.slick_plus/items\?sortby\=slick_timestamp\&datetime\=2023-12-01T00:00:00Z/2023-12-08T00:00:00Z\&filter\=machine_confidence%20GTE%200.6%20AND%20area%20GTE%2020000000\&f\=csv\&offset\=2 | wc -l
Note:
#160 will be merged an released into a 0.6.0
release but I'll also backport this to a 0.5.*
version so you can redeploy cerulean_cloud
Thank you!