Add in-progress messaging / Verbose output

[Jacob Roeland]

Currently when I'm either downloading from sources or importing files, very little output is written to the screen. I'd love it if I can see some kind of progress being made or some kind of logging information being printed. Even if I have to add a flag to get it.

[Matt Holt]

Yeah... this would be nice. What do you have in mind, specifically? Where should we add log.Printf statements and what should they say?

[Jacob Roeland]

I don't have a good knowledge of what exactly could be written. But what I was thinking of was mostly of the sort Starting download of photo.jpg... done. Or Found 32 tweets (plus media) to grab.

Really not expecting anything too detailed.

[Matt Holt]

Okie. I'll look into adding some progress next time I iterate through Timeliner!

[Alexandre Morignot]

If the source is able to give the count of elements, get-all could even show a progress bar.

[Mike West]

Hey Matt! As a Timeliner newbie, I'm staring at an initial get-all that's been running for ~26 hours. I do have a lot of photos, so it's possibly (probably!) still doing something, but it's tough for me to reason through. In particular, I have two questions that such a "verbose mode" could answer:

1. How much work is going to be done, total? Something along the lines of the "Found XXX items" suggestion above could help here. I don't know what kind of metadata you have up front, but estimating size/time would be nice to have ("Found XXX items, totaling YYY Mb (guessing at ~3h27m @ 100 Mb/s).")
2. Is work still happening right now? I could imagine -v outputting a periodic count perhaps every 1k items ("2020/12/30 08:43:20: Processed 9,000 of 123,000: YYY Mb downloaded, ZZZ Mb left to go."), and -v=moar outputting every item ("2020/12/30 08:43:20: Processing img_name_0001.jpg: 8,769 of 123,456; YYY Mb downloaded, ZZZ Mb left to go.")

Most of the metadata above is merely nice to have, but some sort of "Hey, I'm still working, I swear." indicator seems like the core of a verbose mode, and might not be difficult to tack on.

Thanks!

[Matt Holt]

Hi Mike 👋

Yeah, it's definitely working if you're not seeing any errors.

1. Sometimes we don't know. Last I checked, Google Photos doesn't say. We just go page by page, and I think there's up to 100 items per page. I don't think time estimates are going to be easy/valuable either, since there is global and local rate limiting in place to avoid saturating the network link and also to avoid service rate limits. Any estimate will look very slow even though individual file downloads could be very fast, since most of the delay is just in sleeping.

2. Again, this is almost impossible for any service since they don't often count all the items, they paginate instead.

As with most CLI programs, no output is good output -- i.e. you can be assured the program is working if it is running and hasn't outputted any errors. For now, if you want to verify, just observe the repository folder where you can see data files filling it up, or even inspect the database with a SQL(ite) GUI.

Still, lemme see if I have a few minutes today to implement a simple verbosity option. I think I prefer to keep it simple and just make verbose boolean, if that's OK. I can't promise all the fancy stats you want to see, but I can probably at least output when new items are being processed.

I already can guarantee you that verbose mode will slow down anything that isn't bottlenecked by network or I/O. So for example, some data sources that import the data from local files, printing in verbose mode will be very slow. But for Google Photos API it probably won't make much of a difference.

[Prabir Shrestha]

+1 for progress messaging. I'm also importing google photos.

Few things that I would like to see if possible.

• total number of photos downloaded
• total size download
• file getting downloaded

It could look something like this.

google_photos/prabir: downloading a.jpg size: 1mb count: 1
google_photos/prabir: downloading b.jpg size: 1mb count: 2
google_photos/prabir: downloaded a.jpg size: 1mb count:1
google_photos/prabir: downloaded b.jpg size: 1mb count:2

Could also go a bit fancy with coloring or some emojis so it is easier to visualize by glancing.

[Matt Holt]

Ok, in 41cce90 I've added a simple -v flag that enables verbose (debug) logging so you can see more of what is going on.

Of the data sources, only Google Photos has any kind of debug logging right now. And the central processor will log some things regardless of data source.

In the future when I have more time to devote, maybe we can switch to a structured logger like zap with proper log levels, consistent output, etc.