Cannot write to a room(or leave it) with MatrixError: [403] No create event in auth events
VPaulV opened this issue · comments
Description
Hi Guys,
I have a simple docker based setup with workers. Everything worked fine for a year, but recently got an issue with the Synapse Admin room.
When I try to write something there or leave the channel I get the following error:
matrix-nginx | 172.18.0.2 - - [22/Nov/2023:13:17:41 +0000] "OPTIONS /_matrix/client/v3/rooms/!ehXvUhWNASUkSLvAGP%3Amatrix.org/send/m.room.message/m1700659050405.95 HTTP/1.1" 204 0 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Element/1.11.49 Chrome/118.0.5993.120 Electron/27.0.4 Safari/537.36" "79.248.1.168"
matrix-worker-5 | 2023-11-22 13:17:41,896 - synapse.http.server - 124 - INFO - PUT-10309 - <XForwardedForRequest at 0x7faaf2f6d850 method='PUT' uri='/_matrix/client/v3/rooms/!ehXvUhWNASUkSLvAGP%3Amatrix.org/send/m.room.message/m1700659050405.95' clientproto='HTTP/1.0' site='unix'> SynapseError: 403 - No create event in auth events
matrix-worker-5 | 2023-11-22 13:17:41,897 - synapse.access.http.unix - 465 - INFO - PUT-10309 - 172.18.0.2 - unix - {@user:redacted} Processed request: 0.022sec/0.000sec (0.006sec, 0.001sec) (0.002sec/0.011sec/9) 66B 403 "PUT /_matrix/client/v3/rooms/!ehXvUhWNASUkSLvAGP%3Amatrix.org/send/m.room.message/m1700659050405.95 HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Element/1.11.49 Chrome/118.0.5993.120 Electron/27.0.4 Safari/537.36" [0 dbevts]
matrix-nginx | 172.18.0.2 - - [22/Nov/2023:13:17:41 +0000] "PUT /_matrix/client/v3/rooms/!ehXvUhWNASUkSLvAGP%3Amatrix.org/send/m.room.message/m1700659050405.95 HTTP/1.1" 403 77 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Element/1.11.49 Chrome/118.0.5993.120 Electron/27.0.4 Safari/537.36" "79.248.1.168"
I tried to delete the room with the admin API and join back, no luck:
/ # synadm room delete-status -d uTabWIzPGlVybpcZ
{
"delete_id": "uTabWIzPGlVybpcZ",
"status": "failed",
"shutdown_room": {
"kicked_users": [],
"failed_to_kick_users": [
"@user:redacted"
],
"local_aliases": [],
"new_room_id": null
}
}
Steps to reproduce
Unfortunately, I have no idea how to reproduce it. It just randomly happened today after I tried to write something. Probably the problem is somehow related to retention I enabled 1 week ago, or history purge I did with the admin API (also around 1 week ago).
Homeserver
parlor.chf.house
Synapse Version
1.96.1
Installation Method
Docker (matrixdotorg/synapse)
Database
Single PostgresSQL instance. Didn't port it from SQLite nor did any backup restoration
Workers
Multiple workers
Platform
My private server, everything is running in docker containers
Configuration
No response
Relevant log output
When I try to write something there or leave the channel I get the following error:
matrix-nginx | 172.18.0.2 - - [22/Nov/2023:13:17:41 +0000] "OPTIONS /_matrix/client/v3/rooms/!ehXvUhWNASUkSLvAGP%3Amatrix.org/send/m.room.message/m1700659050405.95 HTTP/1.1" 204 0 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Element/1.11.49 Chrome/118.0.5993.120 Electron/27.0.4 Safari/537.36" "79.248.1.168"
matrix-worker-5 | 2023-11-22 13:17:41,896 - synapse.http.server - 124 - INFO - PUT-10309 - <XForwardedForRequest at 0x7faaf2f6d850 method='PUT' uri='/_matrix/client/v3/rooms/!ehXvUhWNASUkSLvAGP%3Amatrix.org/send/m.room.message/m1700659050405.95' clientproto='HTTP/1.0' site='unix'> SynapseError: 403 - No create event in auth events
matrix-worker-5 | 2023-11-22 13:17:41,897 - synapse.access.http.unix - 465 - INFO - PUT-10309 - 172.18.0.2 - unix - {@user:redacted} Processed request: 0.022sec/0.000sec (0.006sec, 0.001sec) (0.002sec/0.011sec/9) 66B 403 "PUT /_matrix/client/v3/rooms/!ehXvUhWNASUkSLvAGP%3Amatrix.org/send/m.room.message/m1700659050405.95 HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Element/1.11.49 Chrome/118.0.5993.120 Electron/27.0.4 Safari/537.36" [0 dbevts]
matrix-nginx | 172.18.0.2 - - [22/Nov/2023:13:17:41 +0000] "PUT /_matrix/client/v3/rooms/!ehXvUhWNASUkSLvAGP%3Amatrix.org/send/m.room.message/m1700659050405.95 HTTP/1.1" 403 77 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Element/1.11.49 Chrome/118.0.5993.120 Electron/27.0.4 Safari/537.36" "79.248.1.168"
Please let me know if I can provide any additional logs
kind of resolved by using room delete API with force_purge, but I am not sure is it a real solution...
No create event in auth events
Odd. My guess is that your server's view of the current room state got corrupted such that it didn't see the create event. That is very odd and should never happen.
kind of resolved by using room delete API with force_purge, but I am not sure is it a real solution...
Unfortunately I don't think we'll be able to diagnose what happened now that the data has been deleted. Do you have any kind of backup of the database while you were stuck unable to send messages?
Probably the problem is somehow related to retention I enabled 1 week ago, or history purge I did with the admin API (also around 1 week ago).
When you enabled retention, what Synapse version were you running?
Per #7642 (comment) it sounds like force-purging and rejoining the room was the right thing to do, however.
When you enabled retention, what Synapse version were you running?
I think it was 1.96.1
Do you have any kind of backup of the database while you were stuck unable to send messages?
Yes, I have a backup, so let me know if I can provide any logs to diagnose what happened
Apologies for the delay in responding, and thanks for getting back to me. I think the team isn't going to have much time to track this down right now, so I suggest we consider this issue closed. But: if you see this again in a room that was created on Synapse 1.94 or newer, please re-open and we can take another look.
(and I say 1.94 because that's when #16382 landed)