Giters

vzhd1701 / evernote-backup

Backup & export all Evernote notes and notebooks

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Notebooks skipped

felciano opened this issue · comments

commented

Thanks for making this tool available -- looks very promising. I just tried it with my Evernote configuration and found that it seems to have skipped a number of notebooks. These are neither in the final ENEX export nor in the Notebooks table in the sqlite database after the sync completes. Is there a debug mode or additional logging that can be turned on to get a more detailed activity trace?

commented

Thank you for the feedback! Unfortunately, I don't have an option of generation debug info at the moment. Could you please describe those notebooks in more detail? How many notes? Their content and names?

commented

The names are pretty generic, no special characters etc. Here are several examples:

  • 1326 Housekeeping
  • be who you are course

These have 5 and 34 notes in them respectively.

commented

I am very sorry for the huge delay. I've added --verbose option that will help you see all notes & notebooks that are coming from the Evernote server on sync.

You can run this command if you are on Linux to save the output log
evernote-backup --verbose sync &> log.txt

Or this one if you are on windows
evernote-backup --verbose sync > log.txt 2>&1

commented

Hi @vzhd1701 thanks for pushing out this new version. I've re-run a full fresh sync with logging and have confirmed that those notebook names never show up in the logs. It seems like they either aren't being retrieved from EN in the first sync, or there is some logic that is skipping them before the log statements are reached. I'm having a bit of trouble tracing through the code -- where is the function that retrieves the initial set of notebook names to iterate through during the very first sync?

commented

It looks like the skipped notebooks might be ones that were initially created by someone else, and that I was invited to. Is it possible that the code that is reading the list of notebooks to sync is only returning notebooks created by the user in question, vs all the notebooks that s/he has access to?

commented

Yes, you are right. Evernote separates linked notebooks from the ones created by the user.

I am using SyncChunkFilter API to get updates from the Evernote server. I forgot to add includeLinkedNotebooks flag since I personally never used shared notes.

I'll get right on fixing this issue. Thanks again for your feedback! I really appreciate it :)

commented

It turned out to be a little more complicated than just adding a single flag 😅
Please try out the new version and see if your problem is fixed.

commented

Thanks! Getting closer, but looks like an access error?

Current login will expire at 2022-08-23 15:25:06.
Syncing user notebooks...
User notebooks are up to date, nothing to sync!
Syncing linked notebooks...
7436 notes to download...
Requesting access to 16 linked notebooks...
Traceback (most recent call last):
  File "evernote_backup/cli.py", line 276, in main
  File "click/core.py", line 1137, in __call__
  File "click/core.py", line 1062, in main
  File "click/core.py", line 1668, in invoke
  File "click/core.py", line 1404, in invoke
  File "click/core.py", line 763, in invoke
  File "evernote_backup/cli.py", line 204, in sync
  File "evernote_backup/cli_app.py", line 120, in sync
  File "evernote_backup/note_synchronizer.py", line 123, in sync
  File "evernote_backup/note_synchronizer.py", line 153, in _authorize_linked_notebooks_for_notes
  File "evernote_backup/note_synchronizer.py", line 154, in <dictcomp>
  File "evernote_backup/evernote_client_sync.py", line 97, in auth_linked_notebook
  File "evernote_backup/evernote_client.py", line 131, in wrapper
  File "evernote_backup/evernote_client_util.py", line 23, in wrapper
  File "evernote/edam/notestore/NoteStore.py", line 6633, in authenticateToSharedNotebook
  File "evernote/edam/notestore/NoteStore.py", line 6657, in recv_authenticateToSharedNotebook
evernote.edam.error.ttypes.EDAMUserException: EDAMUserException(errorCode=5, parameter='shareKeyOrGlobalId')
commented

It should be fixed now. Please try the new version.

commented

Thanks @vzhd1701 definitely making progress. I'm seeing entries for these linked notebooks in the logs from the sync command, but the linked notebooks don't show up later when I run export.

FWIW, I'm seeing an extra Requesting access log entry for linked notes that I don't see for standard ones. Here's an example for a linked note:

2021-08-24 11:41:58,928 | [DEBUG] | Adding/updating linked notebook be who you are course [337aae30-b59d-46ed-97ae-a32ecdf9064e] -> [f9770101-cc76-4281-9413-a4a47bd54eec]
2021-08-24 11:41:58,932 | [DEBUG] | Adding/updating notebook 'be who you are course' [f9770101-cc76-4281-9413-a4a47bd54eec], stack 'Carlcianos Stack'
2021-08-24 11:42:05,956 | [DEBUG] | Requesting access to linked notebook be who you are course [337aae30-b59d-46ed-97ae-a32ecdf9064e]

Here's an example for a regular (non-linked) note (in this case, a note named "Movies to watch on New Year's Eve"), which only shows 2 log entries:

2021-08-24 11:40:14,587 | [DEBUG] | Scheduling note for sync 'Movies to watch on New Year's Eve' [9233f2d8-32ca-47a7-a40e-366d64a95840], notebook_id 88e9bfed-4d4e-497b-b5b3-e32b806b9d55
2021-08-24 11:43:50,105 | [DEBUG] | Adding/updating note 'Movies to watch on New Year's Eve' [9233f2d8-32ca-47a7-a40e-366d64a95840], notebook_id 88e9bfed-4d4e-497b-b5b3-e32b806b9d55
commented

I am sorry that it takes so long to fix this.. Thank you for your patience.

Do you see any "Adding/updating note" or "Scheduling note for sync" entries for the notes from 'be who you are course' notebook on sync? How many notes does that notebook have?

commented

No problem, glad to help. Let me know if you want me to share a similar notebook with you to help testing.

There are 34 notes in the "be who you are course" notebook. These represent notes from a single 34-day course. The individual notes do appear to show up in the log as "Scheduling note for sync", but not the "Scheduling note for sync". Here's an example:

➜  evernote-backup-cli grep "day 14" log.txt
2021-08-24 11:41:58,935 | [DEBUG] | Scheduling note for sync 'day 14' [b1067fbe-8e9d-4af1-ab46-1275d91d075b], notebook_id f9770101-cc76-4281-9413-a4a47bd54eec
commented

Let me know if you want me to share a similar notebook with you to help testing.

Yes, please, that would be helpful! Could you share this or a similar notebook with anon_pad@hotmail.com through evernote? It's my test account.

commented

Done. I shared a single notebook called "I'm Sharing These Notes" with 2 notes in it, titled "This is a simple note #1" and "This is a simple note #2". Invite will come from a test account ka...imi@gmail.com

commented

I have tested sync with your notes and everything went OK. I've made a couple of fixes that should make debugging this issue a little easier. Could you please try again with the new version?

commented

Thanks--very helpful. Looks like the export is failing because it thinks the notebooks are empty. Here's an example, using 1326 Housekeeping as the shared notebook, and Troubleshooting Internet Instructions as the note inside that notebook that ends up without content.

From the sync operation:

2021-08-28 12:29:41,926 | [DEBUG] | 4686261760 | Adding/updating linked notebook '1326 Housekeeping' [adda1ef3-e995-438d-912e-186fbd1586af] -> [c8732250-cbaa-4446-bfcc-a0a458c58bec]
2021-08-28 12:29:41,929 | [DEBUG] | 4686261760 | Adding/updating notebook '1326 Housekeeping' [c8732250-cbaa-4446-bfcc-a0a458c58bec], stack '1326 Masonic Mgmt Stack'
2021-08-28 12:29:41,931 | [DEBUG] | 4686261760 | Scheduling note for sync 'Troubleshooting Internet Instructions' [034139e2-4e5f-479f-aa98-ebd42b022be0], notebook_id [c8732250-cbaa-4446-bfcc-a0a458c58bec]
--
2021-08-28 12:29:46,881 | [DEBUG] | 4686261760 | Requesting access to linked notebook 1326 Housekeeping [adda1ef3-e995-438d-912e-186fbd1586af]

I poked around the sqlite db a bit at it looks like "1326 Housekeeping" shows up in the notebooks table, and there are 8 records in the notes table with the corresponding id in the notebook_guid column, which look correct (correct title, correct number of notes). However both is_active and raw_note seem to be NULL. Upon cursory inspection it looks like all of the notes that are shared end up with NULL value in raw_note.

In addition, it seems like the link between the notebook and notes table isn't being made. From the export operation:

2021-08-28 13:20:09,199 | [DEBUG] | 4497182208 | Exporting notebook '1326 Housekeeping' [c8732250-cbaa-4446-bfcc-a0a458c58bec], stack '1326 Masonic Mgmt Stack'
2021-08-28 13:20:09,200 | [DEBUG] | 4497182208 | Notebook is empty, skip

So it looks like there are two issues:

  1. You are pulling the metadata correctly, but the for shared notes, the note content isn't being retrieved on initial sync?
  2. The export code incorrectly believes the 1326 Housekeeping notebook is empty, despite showing 8 records with the right id in the notes table
commented

All notes that are just scheduled for download have raw_note as NULL. Do you see any Downloading note [034139e2-4e5f-479f-aa98-ebd42b022be0] records in the log? My guess was that the server returns an error when the note from the shared notebook is requested and that's why they are still empty after the sync.

We need to isolate this issue a bit... Could you please do the following?

Start fresh sync with new database. When it gets to downloading and adding notes, stop the program. Then open SQLite database and execute DELETE FROM notes WHERE guid NOT IN (SELECT notes.guid FROM notes JOIN notebooks_linked USING (notebook_guid)). This will remove all notes from non-shared notebooks from the equation. Then run the sync again and tell me what happens.

commented

Ah I didn't understand that the sync was happening in 2 phases. I suspect my log search shouldn't have been so locally focused. I do see 034139e2-4e5f-479f-aa98-ebd42b022be0 being downloaded:

2021-08-28 12:29:41,931 | [DEBUG] | 4686261760 | Scheduling note for sync 'Troubleshooting Internet Instructions' [034139e2-4e5f-479f-aa98-ebd42b022be0], notebook_id [c8732250-cbaa-4446-bfcc-a0a458c58bec]
2021-08-28 13:08:21,315 | [DEBUG] | 123145395654656 | Downloading note [034139e2-4e5f-479f-aa98-ebd42b022be0]
2021-08-28 13:08:22,532 | [DEBUG] | 123145395654656 | Finished downloading note [034139e2-4e5f-479f-aa98-ebd42b022be0]

However I now also see a CRITICAL exception caught further down for a different shared note (Real ID requirements), which aborts the sync. Presumably this is the culprit, and the other notes being unsynched is just a side-effect of the app terminating?

021-08-28 13:43:18,188 | [DEBUG] | 4591791616 | Adding/updating note 'Birthday Plan' [cc3093fd-49ab-44db-9c30-a2fff25e5be2], notebook_id [1cf84df7-4582-4542-b3bf-c92f4bdaf076]
2021-08-28 13:43:18,201 | [DEBUG] | 4591791616 | Added note [cc3093fd-49ab-44db-9c30-a2fff25e5be2]
2021-08-28 13:43:18,329 | [DEBUG] | 123145511198720 | Downloading note [36d20d40-975c-4599-aad2-819563370029]
2021-08-28 13:43:18,330 | [CRITICAL] | 4591791616 | Exception caught while downloading note 'Real ID requirements', terminating!
2021-08-28 13:43:18,338 | [WARNING] | 4591791616 | Aborting, please wait...
2021-08-28 13:43:18,340 | [DEBUG] | 123145527988224 | Finished downloading note [988c78ec-e09c-47e6-a442-0cfa329d8f7a]
2021-08-28 13:43:18,340 | [DEBUG] | 123145527988224 | Memory consumed: 0.0% [LIMIT 256.0 MB]
2021-08-28 13:43:18,447 | [DEBUG] | 123145511198720 | Finished downloading note [36d20d40-975c-4599-aad2-819563370029]
2021-08-28 13:43:18,447 | [DEBUG] | 123145511198720 | Memory consumed: 0.0% [LIMIT 256.0 MB]
2021-08-28 13:43:19,377 | [DEBUG] | 123145494409216 | Finished downloading note [68326595-779e-4497-bd80-d77344a0eb62]
2021-08-28 13:43:19,377 | [DEBUG] | 123145494409216 | Memory consumed: 0.007% [LIMIT 256.0 MB]
2021-08-28 13:43:23,876 | [DEBUG] | 123145477619712 | Finished downloading note [11f3899d-9683-483b-8e53-891b945effde]
2021-08-28 13:43:23,876 | [DEBUG] | 123145477619712 | Memory consumed: 0.056% [LIMIT 256.0 MB]
2021-08-28 13:43:25,500 | [DEBUG] | 123145460830208 | Finished downloading note [36f7313f-c42c-4b94-82ee-54d4b2af1cf5]
2021-08-28 13:43:25,500 | [DEBUG] | 123145460830208 | Memory consumed: 0.119% [LIMIT 256.0 MB]
2021-08-28 13:43:25,501 | [CRITICAL] | 4591791616 | Traceback (most recent call last):
  File "evernote_backup/cli.py", line 272, in main
  File "click/core.py", line 1137, in __call__
  File "click/core.py", line 1062, in main
  File "click/core.py", line 1668, in invoke
  File "click/core.py", line 1404, in invoke
  File "click/core.py", line 763, in invoke
  File "evernote_backup/cli.py", line 199, in sync
  File "evernote_backup/cli_app.py", line 123, in sync
  File "evernote_backup/note_synchronizer.py", line 196, in sync
  File "evernote_backup/note_synchronizer.py", line 341, in _download_scheduled_notes
  File "evernote_backup/note_synchronizer.py", line 369, in _process_download_chunk
  File "concurrent/futures/thread.py", line 57, in run
  File "evernote_backup/note_synchronizer.py", line 133, in __call__
  File "evernote_backup/evernote_client_sync.py", line 41, in get_note
  File "evernote_backup/evernote_client_sync.py", line 41, in <listcomp>
  File "evernote_backup/evernote_client_sync.py", line 132, in tags
  File "evernote_backup/evernote_client.py", line 131, in wrapper
  File "evernote_backup/evernote_client_util.py", line 28, in wrapper
  File "evernote/edam/notestore/NoteStore.py", line 3130, in listTags
  File "evernote/edam/notestore/NoteStore.py", line 3153, in recv_listTags
evernote.edam.error.ttypes.EDAMUserException: EDAMUserException(errorCode=3, parameter='authenticationToken')

Does it do the downloading in 2 passes, local notes first and then shared notes next? Is there a way to tell whether Real ID requirements is the first shared note it is actually trying to retrieve?

commented

Yes, the sync goes in 2 stages. First, it indexes everything that was changed since the last sync, then it downloads and updates all changed notes.

I think I've found and fixed the problem. It was crashing on notes from shared notebooks with tags in them. Please check out the new version!

commented

Looks promising -- all the notebooks have now exported, including the shared ones!

It does seem like the exported enex files are different size than those exported by Evernote directly. Are they expected to be identical?

commented

I am happy to hear that everything works OK now :)

It does seem like the exported enex files are different size than those exported by Evernote directly. Are they expected to be identical?

No, they are not expected to be byte-to-byte identical. But I did my best to make the export as close as possible to what Evernote produces. I managed to dig out the source code of the Evernote client export function and used it as a guide. So apart from slight differences in internal XML structure, everything else should be identical.

Thank you very much for your patience and feedback in helping to solve this issue! I really appreciate it!

commented

hi @vzhd1701 more than happy to have been able to help out. Really nice that you've built this app and made it available--thank you for that. I think we can close this bug as fixed.