icy / google-group-crawler

[Deprecated] Get (almost) original messages from google group archives. Your data is yours.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Error when handling very large google groups

dado3212 opened this issue · comments

While my previous problem was resolved, when running it correctly I hit another problem. At a certain point, cat can't handle the total number of lines, and it will throw the error -bash: /bin/cat: Argument list too long on line 155. The resolution for this is to use ls, and for each filename cat just file, and run the process on them. There may also be a more elegant solution.

Oh, I see. Your list is too big . I'm taking a look on this. Stay tuned.!

@dado3212 Would you mind pulling the latest code and try again? Thanks a lot.

Hi @dado3212, are you able to fix the problem with new commit? Thanks.

Yup, that did it. Thank you!