[Bug] Unsteadiness of generating notes from html files

Question

[Bug] Unsteadiness of generating notes from html files

Quasimurdock opened this issue a year ago · comments

In the end of #2 @serifold mentioned the unsteady issue about generating cards from html process:

So, the problem now is that I can't get the whole words transformed into cards, and the pnpm note process seems unsteady, I tried 2 times, it gave me different amount of cards, and there is no info which words were not included.

I'd like to take a look into this in my spare time next week maybe. But if anyone give a hand out that'd be great too.

Quasimurdock · Answer 1 · Mon Aug 07 2023 14:47:35 GMT+0800 (China Standard Time)

In the preceding calling way I didn't handle the result of addNotes() correctly. If a note wasn't added successfully, it will return a null in result array of it. The structure of it is as follows:

{
    "result": [1496198395707, null],
    "error": null
}

I have pushed the commit #f3a4d52 to add excetion handling codes for addNotes() method calling.

Missings of words shall be logged now.

@serifold

Quasimurdock · Answer 2 · Mon Aug 07 2023 14:59:41 GMT+0800 (China Standard Time)

Hey, @serifold . So far this method is still like a black box to me for the unsteadiness execution was replicated by me just now. (It seems to happen when there's duplicated word already exists in ur Anki deck)

I think the problem of AnkiConnect API is not actually providing enough error messages. The params of previous cases by us are all solid fine but when we invoke the method it just blows out.

Otherwise it may owe to my wrongly invoking. Do u have any ideas about it?

Serif_ · Answer 3 · Mon Aug 07 2023 17:39:39 GMT+0800 (China Standard Time)

Hey, @serifold . So far this method is still like a black box to me for the unsteadiness execution was replicated by me just now. (It seems to happen when there's duplicated word already exists in ur Anki deck)

I think the problem of AnkiConnect API is not actually providing enough error messages. The params of previous cases by us are all solid fine but when we invoke the method it just blows out.

Otherwise it may owe to my wrongly invoking. Do u have any ideas about it?

Hi, @Quasimurdock, thanks for your commit, I have tried several times again, and this time I got missing words successfully:

But it's wired that I still get the amount of cards between 3535 and 3536. The first 2 or 3 times I got 3535, and the rest times I got 3536. If the amount is 3536, then everything should be fine, cause 3536 + 2(missing words) + 1(css file) = 3539, but I still can't understand why I get 3535, the error info and log are identical to those in 3536, I can't find the '1' difference between them.

And I don't understand this sentence you said:

(It seems to happen when there's duplicated word already exists in ur Anki deck)

We already have html files duplication check:

// check if duplicated file exists
    if (checkDuplicateFiles("./output/html", word + ".html")) {
      console.log(`[SKIP] Duplicated [${word}] file exists.`);
      continue;
    }

If the html files are all unique, then the cards generated from html files should be unique, too. How could Anki have duplicated cards?

And I added some error log in processHtmlFiles() such as:

Found nothing suspicious.

It seems that we have caught all the theoretical errors, at least I think it's not our fault now. If it's Anki Connect's or cheerio's bug, as you said, it's as complicated as a black box... Maybe we should open an issue in their repo.

Quasimurdock · Answer 4 · Mon Aug 07 2023 17:55:46 GMT+0800 (China Standard Time)

We already have html files duplication check

Hi, @serifold . Here I don't mean html files duplication check.(in the phase of generating html) I'm talking about when u already has a word e.g. Apple added in ur deck, then u try to add it again.(in the phase of converting html to notes when you test multiple times, more specifically while invoking the addNotes method which is provided by AnkiConnect)
Normally it will just skip that word and everything else should be fine. But I tested it several times and it results in what I said before(there's words missing and these words haven't been added in deck. But later if I invoke again they will be added normally), which is really odd.
Idk what happened😂

Serif_ · Answer 5 · Mon Aug 07 2023 19:51:06 GMT+0800 (China Standard Time)

We already have html files duplication check

Hi, @serifold . Here I don't mean html files duplication check.(in the phase of generating html) I'm talking about when u already has a word e.g. Apple added in ur deck, then u try to add it again.(in the phase of converting html to notes when you test multiple times, more specifically while invoking the addNotes method which is provided by AnkiConnect) Normally it will just skip that word and everything else should be fine. But I tested it several times and it results in what I said before(there's words missing and these words haven't been added in deck. But later if I invoke again they will be added normally), which is really odd. Idk what happened😂

Hi @Quasimurdock, I manually delete the IELTS-CamDict deck every time before I execute a test, that means every test create a new deck and push cards into it from scratch. Therefore, I think in my case, there shouldn't exist duplicate cards. However, I still can't figure out why the output amount is unsteady, and the difference is always '1' by far...

Quasimurdock · Answer 6 · Mon Aug 07 2023 20:13:58 GMT+0800 (China Standard Time)

@serifold Hi,
I found this issue which already mentioned the error message returning problem of addNotes() method. OP is writing a plugin for Obsidian with AnkiConnect API, and he/she met the similar annoying things about this API not returning proper error messages as we expected. He talked about he was using a 'multi' of 'addNote' rather than 'addNotes'.

Here's his detailed explaination about this multi stuff. I haven't checked it but It looks fine, maybe we should give it a try. It may not solve this unsteadiness problem but at least error messages will be more precise instead of just returning some null to me.

Quasimurdock · Answer 7 · Mon Aug 07 2023 20:27:59 GMT+0800 (China Standard Time)

and this one is for method updateNoteFields but I somehow wonder if this has any association with what we are talking about.

Serif_ · Answer 8 · Mon Aug 07 2023 22:41:28 GMT+0800 (China Standard Time)

@serifold Hi, I found this issue which already mentioned the error message returning problem of addNotes() method. OP is writing a plugin for Obsidian with AnkiConnect API, and he/she met the similar annoying things about this API not returning proper error messages as we expected. He talked about he was using a 'multi' of 'addNote' rather than 'addNotes'.

Here's his detailed explaination about this multi stuff. I haven't checked it but It looks fine, maybe we should give it a try. It may not solve this unsteadiness problem but at least error messages will be more precise instead of just returning some null to me.

Hey @Quasimurdock, I tried the multi action, it seems works better than previous:

Maybe this multi action is a more precise and elegant solution to output error info indeed.

Later I'll submit a PR.

Quasimurdock · Answer 9 · Tue Aug 08 2023 08:34:19 GMT+0800 (China Standard Time)

Morning @serifold ! Appreciate ur patience and contribution.

Thx 4 ur PR commit, It's all good! I've already merged it into main branch.

Now error hints shall be better.

Quasimurdock · Answer 10 · Tue Aug 08 2023 08:45:03 GMT+0800 (China Standard Time)

Getting back to the topic, does the odd unsteadiness still happen?

Serif_ · Answer 11 · Tue Aug 08 2023 09:46:57 GMT+0800 (China Standard Time)

Morning @serifold ! Appreciate ur patience and contribution.

Thx 4 ur PR commit, It's all good! I've already merged it into main branch.

Now error hints shall be better.

Morning @Quasimurdock, you're welcome, I'm glad I can contribute to this project.

During the test of the PR, I found the unsteadiness still exists. Moreover, I think it almost follows a pattern: the first 1-2 tests will generate 3535 cards, then the following tests will back to normal, these tests keep generating 3536 cards. It's just weird, although theoretical we can make cards generation more times to get rid of the unsteadiness, but it's really annoying.

Quasimurdock · Answer 12 · Tue Aug 08 2023 09:52:52 GMT+0800 (China Standard Time)

then the following tests will back to normal

Yeah, I met this either. I tend to believe it's not what we can control for now. It's not a super big deal but it's annoying. Luckily it won't bother those who just wanna use generated word cards.

Thus I added a wontfix label on it. Let's just set this aside now.

Serif_ · Answer 13 · Tue Aug 08 2023 09:59:10 GMT+0800 (China Standard Time)

then the following tests will back to normal

Yeah, I met this either. I tend to believe it's not what we can control for now. It's not a super big deal but it's annoying. Luckily it won't bother those who just wanna use generated word cards.

Thus I added a wontfix label on it. Let's just set this aside now.

Indeed, it seems we can't handle it at least now. Let's focus on some more important features.

Serif_ · Answer 14 · Tue Aug 08 2023 10:08:16 GMT+0800 (China Standard Time)

and this one is for method updateNoteFields but I somehow wonder if this has any association with what we are talking about.

In my opinion, perhaps we don't need this updateNoteFields action now:

We can do what we want such as add audio info right in the addNote action. We just generate cards which not includes modifying cards.

Quasimurdock · Answer 15 · Tue Aug 08 2023 10:19:53 GMT+0800 (China Standard Time)

No, I quoted that issue before cuz its discription about their odd problem is actually similar to ours:

Ah, it appears it doesn't work when the note browser is open. If I close the browser and then update a note, I can open it again and view the updated note.

Could I suggest adding some clear warning in updateNoteFields's section of notes.md that having note browser would likely cause the note to not be updated even if the API returns a successful promise? I have a feeling first timer user and developer alike would be very tempted to keep browser gui open to handily see if the command works (which would backfire XD)

They basically met the problem that if they invoke that update action without closing note browser, something weird would happen.

Serif_ · Answer 16 · Tue Aug 08 2023 10:36:10 GMT+0800 (China Standard Time)

No, I quoted that issue before cuz its discription about their odd problem is actually similar to ours:

Ah, it appears it doesn't work when the note browser is open. If I close the browser and then update a note, I can open it again and view the updated note.

Could I suggest adding some clear warning in updateNoteFields's section of notes.md that having note browser would likely cause the note to not be updated even if the API returns a successful promise? I have a feeling first timer user and developer alike would be very tempted to keep browser gui open to handily see if the command works (which would backfire XD)

They basically met the problem that if they invoke that update action without closing note browser, something weird would happen.

Just not update cards:

And: we don't use the updateNoteFields at all; every time the steadiness happens I was not opening the Anki browser. Therefore, in my opinion, it should not be corresponding with the cards amount problem. They are not the same type.

Quasimurdock · Answer 17 · Tue Aug 08 2023 14:34:04 GMT+0800 (China Standard Time)

This issue has been labeled as wontfix and reasons have been discussed along the thread so I'm gonna close it.

If anyone has any effective solution to this, he/she's welcomed to commit a new issue or PR.