docs: troubleshooting mismatching keys error

[camille]

Description

👋🏻 Hi! Thanks for the amazing libs you're providing with Charm/Skate & alts. I've been struggling for quite a long time after having an issue while testing and setting up a self hosted charm instance. I thought it would be great to add some docs/troubleshooting guide somewhere in the repo.

Disclaimer: I hope I don't miss any existing doc, troubleshooting guide and please accept my apologies if the answer of my issue was already documented or straightforward to fix (I'm still very new to Charm/Skate).

Issue / Reproducibility

I wanted to test locally the self host instance and then self hosting it for real, using a systemd service. However, no matter what I did, I was hitting the issue Encryption key mismatch. What I think it happens is skate/charm is storing the keys somewhere in the system and if you remove the current instance you are using, to switch let's say from a testing instance (from charm serve to a real service), the keys used by the client will be the first one generated for the first server you use. And thus every request will fail with: Encryption key mismatch.

Steps:

# Step 1: we test our server
mkdir temp && cd temp
charm serve
export CHARM_HOST=localhost
skate set kitty meow
skate get kitty
# > meow OK!
# ----------- 
# Step 2: we kill the first server, and test it again
mkdir temp2 && cd temp2
charm serve
export CHARM_HOST=localhost
skate set kitty meow
# > Error: Encryption key mismatch KO!

So I think something like that happens: skate CLI is trying to reach server-2 with the credentials generated for server-1.

In order to fix that, I had to find where the keys where located (this is in /System/Volumes/Data/Users/${USER}/Library/Application\ Support/charm/localhost in my system), but it's not very straightforward. I didn't find any documentation or troubleshooting help about this.

What to do?

There's obsviously no issue to fix, but I think this is something which is worth documenting (because that's a path which sounds pretty generic to me, testing with the default charm serve command and switching to a systemd service well configured).

• What do you think?
• I can propose something and open a PR for updating Skate (or Charm?) README if you'd like, because I think it's not an easy issue to begin with when you start playing with Charm/Skate.

Thanks! 😺

EDIT: In order to be able to understand how to troubleshoot this, I had to

• Find this Issue in the Charm Repo
• Search for the default go DataPaths for my system provided by the library used in Charm
• Search in my system and remove the folders containing the RSA keys

So it was not super complicated, but not straightforward either! I would be happy to help document it and drop a few lines if you think it's interesting 😃

[Toby Padilla]

Wow! Thank you for this. It's really great feedback and troubleshooting. You are indeed correct that the keys are namespaced to the host, so if you do a lot of local testing, you'll end up in the very scenario you describe with the wrong keys in the data path under "localhost". There's also a bug that causes encryption key mismatch errors but it's fixed in a soon to be released version of the charm lib. Are you in our Slack? It might be good to chat about this in there. I'll think on it a bit too. You can join with https://charm.sh/slack, but happy to talk here as well.

[camille]

Thanks for the answer @toby ! Yes I have joined your Slack to search for information, while I was trying to troubleshoot the issue, you can DM me anytime @epsxy 😄. Happy to help! 💯

[Robert Mills]

If you're here looking to fix the same problem on a *nix machine, delete the keys located in ~/.local/share/charm