agourlay / dlm

Minimal HTTP download manager

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

could not determine file extension for https://oeis.org/search?q=id:A000023&fmt=json

anandijain opened this issue · comments

[2024-01-29 18:29:20] Unrecoverable error while processing https://oeis.org/search?q=id:A000022&fmt=json: standard I/O error - The filename, directory name, or volume label syntax is incorrect. (os error 123)                                                                                      
[2024-01-29 18:29:20] Could not determine file extension for https://oeis.org/search?q=id:A000023&fmt=json

thank you very much for the package its so useful!

im trying to download the OEIS database which (by https://oeis.org/wiki/JSON_Format,_Compressed_Files) does not have an extension for the urls, is there a way to tell dlm that these are going to be json files?

i guess i probably also need to give a full filename as theres no obvious way for dlm to derive a filename from the url list, which looks like this

https://oeis.org/search?q=id:A000001&fmt=json
https://oeis.org/search?q=id:A000002&fmt=json
https://oeis.org/search?q=id:A000003&fmt=json
https://oeis.org/search?q=id:A000004&fmt=json
https://oeis.org/search?q=id:A000005&fmt=json
https://oeis.org/search?q=id:A000006&fmt=json
https://oeis.org/search?q=id:A000007&fmt=json

Thanks for the report 👍

I think it would make sense to use the query parameters as file name, in that case search?q=id:A000022 without extension.

This sounds reasonable.

I think I have understood the issue!

The problem is not that the file extension is missing, dlm handles this case already by generating a filename based on the URL.

The actual issue is that some operating systems, like I guess Windows for you, are not accepting ? and & within file system paths.

The fix is to sanitize the path for that case 03ab686.

Please check the new version https://github.com/agourlay/dlm/releases/tag/v0.3.6 :)

closing due to inactivity and because I think it is fixed ;)