Unable to handle big profile files

Question

Unable to handle big profile files

smhmd opened this issue 4 years ago · comments

Describe the bug
I have a profile file with 23,000 lines that used to work just fine. With 1.0.0 (and now 1.0.2), hostctl no longer works.
hostctl status didn't work. I removed all profiles to start fresh and the big file would not be added --from.

To Reproduce
Steps to reproduce the behavior:

sudo hostctl add nsfw --from https://github.com/smhmd/dotfiles/blob/master/.config/hosts/nsfw -q
(there's an idea. Profile files --from URLs.)

Expected behavior
hostctl to load the profile

System
OS: Linux

Additional context
Loading that profile worked in 0.10.x.

Gustavo Marin · Answer 1 · Tue Apr 14 2020 15:54:11 GMT+0800 (China Standard Time)

It does work, but it take 4 or 5 minutes if you add -q or not. I'll look into it.

simohamed · Answer 2 · Tue Apr 14 2020 15:57:35 GMT+0800 (China Standard Time)

For a point of reference, it was as instantaneous as you can cat the file in 0.x.

Gustavo Marin · Answer 3 · Tue Apr 14 2020 16:37:04 GMT+0800 (China Standard Time)

fixed on v1.0.3. tested with your example profile.

Gustavo Marin · Answer 4 · Tue Apr 14 2020 16:47:57 GMT+0800 (China Standard Time)

as reference, the issue was related to the uniqueness check of domains on a profile. I removed the check for now.

simohamed · Answer 5 · Tue Apr 14 2020 16:50:15 GMT+0800 (China Standard Time)

You can add it as a flag -u,--uniq?

Gustavo Marin · Answer 6 · Tue Apr 14 2020 18:16:43 GMT+0800 (China Standard Time)

added --uniq to v1.0.4 👍