cdclient.sqllite UTF8 issues

[Marie]

I tried getting cdclient.sqllite to work with the dashboard but each time I convert it and try to use it I keep getting the same errors around it not being able to decode UTF-8

[Daniel Seiler]

Which tool did you use? https://fdb.lu-dev.net should handle UTF-8 correctly.

EDIT: Technically what you have there is a failure to decode to UTF-8 as the text in question ("Café") is not UTF-8, because it has not been correctly re-encoded from the FDB, which uses ISO/IEC 8859-1 aka Latin-1. SQlite doesn't pervent storing malformed UTF-8, hence the error happening on read, not on write. What is there is the byte 0xE9 instead of the codepoint 0xE9, which in UTF-8 should be 0xC3 0xA9 (c.f. https://www.compart.com/en/unicode/U+00E9).

[Marie]

Which tool did you use? https://fdb.lu-dev.net/ should handle UTF-8 correctly.

I used the LCDR util originally as noted in the README

Use fdb_to_sqlite.py in lcdr's utilities on res/cdclient.fdb in the unpacked client to convert the client database to cdclient.sqlite

then I also tried with the version that the server generates which also produced the same issue

I also now tried the site you provided but the same error still occurs

Though interestingly enough with the sqlite generated through the site I now also get these additional messages which I will assume are normal

[Daniel Seiler]

Are you sure it's using the one you converted at all and not an auto-converted CDServer.sqlite?

What do your settings and paths look like? What happens when you replace that name to have a standard "e" with an Sqlite editor?

[Marie]

Are you sure it's using the one you converted at all and not an auto-converted CDServer.sqlite?

This actually led me to figure out what was going on it was using the auto-converted CDServer.sqlite from the actual server as that is the path defined in the docker-compose.yml file on the server repo

After I changed

CD_SQLITE_LOCATION=/app/cdclient/

to

CD_SQLITE_LOCATION=/app/luclient/res/

It started working