Api error

[Modez MARTIN]

Hey, look like your API is down

2023/09/14 11:08:33 hishtory fatal error at github.com/ddworken/hishtory/hishtory.go:386: failed to POST /api/v1/submit?source_device_id=**masked**: Post "https://api.hishtory.dev/api/v1/submit?source_device_id=**masked**": net/http: TLS handshake timeout

Also running hishtory uninstall make this :

2023/09/14 11:08:12 hishtory fatal error at github.com/ddworken/hishtory/hishtory.go:264: unknown command: uninstall

[David Dworken]

Hmm, are you still experiencing this issue? The API looks fine to me from:

• Manually checking https://api.hishtory.dev/healthcheck
• Looking at logs on the backend, all looks fine and requests are being received
• A datadog continous healthcheck (that runs every 2 minutes) has shown the API to be up with no issues

[Satyanarayana Kolluri]

Hi,

I can confirm that this issue is still occurring. I had to disable hishtory since I keep seeing curl command failures every time I ran a command. hishtory update and hishtory export are failing with dial TCP errors too.
It looks like https://api.hishtory.dev/healthcheck is down too.

[Satyanarayana Kolluri]

Pasting the error message from my terminal -

2023/09/15 10:38:55 hishtory v0.206 fatal error at github.com/ddworken/hishtory/client/cmd/query.go:58: failed to GET https://api.hishtory.dev/api/v1/get-deletion-requests?user_id=xxxx=&device_id=xxxx: Get "https://api.hishtory.dev/api/v1/get-deletion-requests?user_id=xxxx&device_id=xxxx": dial tcp 167.99.24.52:443: connect: connection refused

[Aleksander Grzyb]

Same for me:

2023/09/15 07:10:43 hishtory v0.193 fatal error at github.com/ddworken/hishtory/client/cmd/saveHistoryEntry.go:98: failed to POST /api/v1/submit?source_device_id(...)

[isjarvi]

this is a local running software, why need to post to the external network?

[David Dworken]

My apologies everyone, the server has now been fixed and is operational again. It seems like the Digitalocean instance that was running the backend got killed by Digitalocean (it seems to currently be stuck in a weird state where it is neither online nor offline), so I ended up having to restore from backups and create a new server. The good news is that the restore went off without a hitch and everything is now back up and running without any data loss.

In addition, if you run hishtory update you will get the latest version of hishtory which will properly handle errors of this type without spamming your terminal.

I'll continue to investigate this and will update this issue with any findings. So far, my conclusion is that two things went wrong here:

1. The Digitalocean instance that was hosting the backend got killed
2. This failure caused the hishtory client to fail to connect to the backend in an unhandled way. Normally, the hishtory client attempts to detect connection failures and automatically pause syncing. This is done both to ensure that backend failures don't cause user-visible errors, but also to ensure that hishtory works when the computer does not have a live network connection. But, the error message that was returned by the Go HTTP library was one that the hishtory client did not handle in lib.IsOfflineError(err error) which meant that it was displayed to the user, causing the user-visible breakage here.

Both 1) and 2) are significant problems. 1) has been fixed by bringing the server back up. 2) has been fixed by d4e01ce which is now included in the latest version of hishtory.

[David Dworken]

this is a local running software, why need to post to the external network?

The network requests are used for syncing support, so that if you install hishtory on another one of your machines (and securely copy over your cryptographic key used for identification and encrypting all data), you can share your shell history across all your machines. If you aren't interested in syncing, it is also possible to install hishtory in a fully offline mode (see "Offline Install") in the readme.

[David Dworken]

For anyone curious, here is the Synthetics graph from Datadog showing that the outage lasted ~1.5 hours and has now been fully fixed.

[David Dworken]

Resolving since the API has been back up and stable for the last 5 days.

[elvis]

this is a local running software, why need to post to the external network?

The network requests are used for syncing support, so that if you install hishtory on another one of your machines (and securely copy over your cryptographic key used for identification and encrypting all data), you can share your shell history across all your machines. If you aren't interested in syncing, it is also possible to install hishtory in a fully offline mode (see "Offline Install") in the readme.

same problem here. After reading the code, i found it can be update through the config file ~/.hishtory/.hishtory.config with setting is_offline to true. Can help update the doc or add config-get commands ?