Giters

matrix-org / synapse

Synapse: Matrix homeserver written in Python/Twisted.

Home Page:https://matrix-org.github.io/synapse

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Performance issue, 100% CPU for some time related to _process_new_pulled_events_with_failed_pull_attempts

n-peugnet opened this issue · comments

commented

Description

Since the upgrade to Synapse 1.95 and 1.95.1 we some time have around 20minutes of unresponsiveness, due to the CPU usage going up to 100%. We have a small instance no workers and have enabled retention policy.

In the logs we have a LOT of these messages when this happens (thousands):

2023-11-14 21:26:53,382 - synapse.handlers.federation_event - 1742 - WARNING - _process_new_pulled_events_with_failed_pull_attempts-203-$WQoGGgI59Lhx3g-8WmcGIp-F3dqUJXwW1dvQv6tSpKk-$EkW0qlv1_PXeA8NtuoxUPo4tUzy68FZj5gphAR3qFOo-$GeBn6_jqQH8qAhq_P9E3Jvzhox4Ie1kyNJUE4yit3Og - Dropping event <FrozenEventV3 event_id=$GeBn6_jqQH8qAhq_P9E3Jvzhox4Ie1kyNJUE4yit3Og, type=m.room.member, state_key=@mathfold:matrix.org, outlier=False>, which relies on auth_event $YbYGoEa3I6SZ7WZayCHNIWGp3uswaxiyQz3kx84Mt5g, which could not be found
2023-11-14 21:26:53,382 - synapse.handlers.federation_event - 1742 - WARNING - _process_new_pulled_events_with_failed_pull_attempts-203-$WQoGGgI59Lhx3g-8WmcGIp-F3dqUJXwW1dvQv6tSpKk-$EkW0qlv1_PXeA8NtuoxUPo4tUzy68FZj5gphAR3qFOo-$GrvuQ56mbhx09wUZCZleoZjztU2V0wfD8Aq_asOmjag - Dropping event <FrozenEventV3 event_id=$GrvuQ56mbhx09wUZCZleoZjztU2V0wfD8Aq_asOmjag, type=m.room.member, state_key=@j.668:matrix.org, outlier=False>, which relies on auth_event $GAJFBkufbqFwuExUGUFbhsnhMC0OKmpXfi94X4avjZY, which could not be found
2023-11-14 21:26:53,382 - synapse.handlers.federation_event - 1742 - WARNING - _process_new_pulled_events_with_failed_pull_attempts-203-$WQoGGgI59Lhx3g-8WmcGIp-F3dqUJXwW1dvQv6tSpKk-$EkW0qlv1_PXeA8NtuoxUPo4tUzy68FZj5gphAR3qFOo-$JqriNbJVo0uwcRYquBgafjwzX4T40NZCbJcgoCrgKn0 - Dropping event <FrozenEventV3 event_id=$JqriNbJVo0uwcRYquBgafjwzX4T40NZCbJcgoCrgKn0, type=m.room.member, state_key=@zxc4564:matrix.org, outlier=False>, which relies on auth_event $kAoHJeSnwt2un1Z252CHlEtuYsQmxAb9Wb2uVJq_bZM, which could not be found
2023-11-14 21:26:53,382 - synapse.handlers.federation_event - 1742 - WARNING - _process_new_pulled_events_with_failed_pull_attempts-203-$WQoGGgI59Lhx3g-8WmcGIp-F3dqUJXwW1dvQv6tSpKk-$EkW0qlv1_PXeA8NtuoxUPo4tUzy68FZj5gphAR3qFOo-$Wlm_UrSeJbZ4l6LuZbLqU2fNE9tVKSUqpcXgy7voSgw - Dropping event <FrozenEventV3 event_id=$Wlm_UrSeJbZ4l6LuZbLqU2fNE9tVKSUqpcXgy7voSgw, type=m.room.member, state_key=@barbatos360:matrix.org, outlier=False>, which relies on auth_event $GAJFBkufbqFwuExUGUFbhsnhMC0OKmpXfi94X4avjZY, which could not be found
2023-11-14 21:26:53,382 - synapse.handlers.federation_event - 1742 - WARNING - _process_new_pulled_events_with_failed_pull_attempts-203-$WQoGGgI59Lhx3g-8WmcGIp-F3dqUJXwW1dvQv6tSpKk-$EkW0qlv1_PXeA8NtuoxUPo4tUzy68FZj5gphAR3qFOo-$sAfUEQK_telhJjWJKnwmkyvIg0uBYNFZHHKXWGTGBcU - Dropping event <FrozenEventV3 event_id=$sAfUEQK_telhJjWJKnwmkyvIg0uBYNFZHHKXWGTGBcU, type=m.room.member, state_key=@lucifer.s:matrix.org, outlier=False>, which relies on auth_event $GAJFBkufbqFwuExUGUFbhsnhMC0OKmpXfi94X4avjZY, which could not be found
2023-11-14 21:26:53,382 - synapse.handlers.federation_event - 1742 - WARNING - _process_new_pulled_events_with_failed_pull_attempts-203-$WQoGGgI59Lhx3g-8WmcGIp-F3dqUJXwW1dvQv6tSpKk-$EkW0qlv1_PXeA8NtuoxUPo4tUzy68FZj5gphAR3qFOo-$t55AqmMtWWqFDwIGyxs31NHMJbvLt1zwKuc5Yz5sID8 - Dropping event <FrozenEventV3 event_id=$t55AqmMtWWqFDwIGyxs31NHMJbvLt1zwKuc5Yz5sID8, type=m.room.member, state_key=@tagg181:matrix.org, outlier=False>, which relies on auth_event $GAJFBkufbqFwuExUGUFbhsnhMC0OKmpXfi94X4avjZY, which could not be found
2023-11-14 21:26:53,383 - synapse.handlers.federation_event - 1742 - WARNING - _process_new_pulled_events_with_failed_pull_attempts-203-$WQoGGgI59Lhx3g-8WmcGIp-F3dqUJXwW1dvQv6tSpKk-$EkW0qlv1_PXeA8NtuoxUPo4tUzy68FZj5gphAR3qFOo-$ROPuyhKEdUGgvd4DmAaGZcO7Y5NmHYpapYUj6A1FP_o - Dropping event <FrozenEventV3 event_id=$ROPuyhKEdUGgvd4DmAaGZcO7Y5NmHYpapYUj6A1FP_o, type=m.room.member, state_key=@zadiant:matrix.org, outlier=False>, which relies on auth_event $GAJFBkufbqFwuExUGUFbhsnhMC0OKmpXfi94X4avjZY, which could not be found
2023-11-14 21:26:53,383 - synapse.handlers.federation_event - 1742 - WARNING - _process_new_pulled_events_with_failed_pull_attempts-203-$WQoGGgI59Lhx3g-8WmcGIp-F3dqUJXwW1dvQv6tSpKk-$EkW0qlv1_PXeA8NtuoxUPo4tUzy68FZj5gphAR3qFOo-$tISd0yIoHacQ7mOFao8nCvdW5-1NJb8W1Gx2RL5Aw2Y - Dropping event <FrozenEventV3 event_id=$tISd0yIoHacQ7mOFao8nCvdW5-1NJb8W1Gx2RL5Aw2Y, type=m.room.member, state_key=@eastanzaa:matrix.org, outlier=False>, which relies on auth_event $GAJFBkufbqFwuExUGUFbhsnhMC0OKmpXfi94X4avjZY, which could not be found
2023-11-14 21:26:53,383 - synapse.handlers.federation_event - 1742 - WARNING - _process_new_pulled_events_with_failed_pull_attempts-203-$WQoGGgI59Lhx3g-8WmcGIp-F3dqUJXwW1dvQv6tSpKk-$EkW0qlv1_PXeA8NtuoxUPo4tUzy68FZj5gphAR3qFOo-$I65o5eCsiBYuajF7FTZcwSLbZcXfKfaGldaf4aXo5P4 - Dropping event <FrozenEventV3 event_id=$I65o5eCsiBYuajF7FTZcwSLbZcXfKfaGldaf4aXo5P4, type=m.room.member, state_key=@1dyablo:matrix.org, outlier=False>, which relies on auth_event $bxhFWiYB_-w5cTnmZJbsw0vrfjt5YHHfY345RK8A_5E, which could not be found
2023-11-14 21:26:53,383 - synapse.handlers.federation_event - 1742 - WARNING - _process_new_pulled_events_with_failed_pull_attempts-203-$WQoGGgI59Lhx3g-8WmcGIp-F3dqUJXwW1dvQv6tSpKk-$EkW0qlv1_PXeA8NtuoxUPo4tUzy68FZj5gphAR3qFOo-$ZufRiBlJ8fwr1V4lHB3Sa8wqrQ9tYppWEBBR-WUvZ2U - Dropping event <FrozenEventV3 event_id=$ZufRiBlJ8fwr1V4lHB3Sa8wqrQ9tYppWEBBR-WUvZ2U, type=m.room.member, state_key=@kharma:matrix.org, outlier=False>, which relies on auth_event $GAJFBkufbqFwuExUGUFbhsnhMC0OKmpXfi94X4avjZY, which could not be found
2023-11-14 21:26:53,383 - synapse.handlers.federation_event - 1742 - WARNING - _process_new_pulled_events_with_failed_pull_attempts-203-$WQoGGgI59Lhx3g-8WmcGIp-F3dqUJXwW1dvQv6tSpKk-$EkW0qlv1_PXeA8NtuoxUPo4tUzy68FZj5gphAR3qFOo-$F7OFHNwhXDpflYjHPUfbOq6x99ZnX9AJ9cuIM_-WL5M - Dropping event <FrozenEventV3 event_id=$F7OFHNwhXDpflYjHPUfbOq6x99ZnX9AJ9cuIM_-WL5M, type=m.room.member, state_key=@omxian:matrix.org, outlier=False>, which relies on auth_event $GAJFBkufbqFwuExUGUFbhsnhMC0OKmpXfi94X4avjZY, which could not be found

Not only from matrix.org, but from a lot of different servers.

Steps to reproduce

  • run synapse for some time

Sorry I really don't know what to add in my case.

Homeserver

club1.fr

Synapse Version

1.95.1

Installation Method

Debian packages from packages.matrix.org

Database

PostgreSQL single server

Workers

Single process

Platform

  • Ubuntu 20.04 server
  • Single machine, low power Intel atom C3558 CPU
  • 16 Gio of RAM
  • no virtualisation whatsoever

Configuration

I just disabled presence but it was enabled until now.

Retention policy is ON:

retention:
  # The message retention policies feature is disabled by default. Uncomment the
  # following line to enable it.
  #
  enabled: true

  # Default retention policy. If set, Synapse will apply it to rooms that lack the
  # 'm.room.retention' state event. Currently, the value of 'min_lifetime' doesn't
  # matter much because Synapse doesn't take it into account yet.
  #
  default_policy:
    min_lifetime: 1d
    max_lifetime: 3y

media_retention:
    local_media_lifetime: 2y
    remote_media_lifetime: 90d

Relevant log output

2023-11-14 21:21:21,272 - synapse.handlers.federation_event - 1742 - WARNING - _process_new_pulled_events_with_failed_pull_attempts-206-$N-epuXfdgHVb20KLF7o0wwN_ahseiHq0ph-6tJIw9WA-$Mgzc4YZ-W3_z2fL3igG9hjci5Af_BTG0-i7deqJEdio-$N7u0LNHLSR0NRhL3OljVVQZBAS74PD5osEuc9e8t4jo - Dropping event <FrozenEventV3 event_id=$N7u0LNHLSR0NRhL3OljVVQZBAS74PD5osEuc9e8t4jo, type=m.room.member, state_key=@glittervappy:matrix.org, outlier=False>, which relies on auth_event $_GYON-YfJTgOFS8f4kvGqLbPGbr9tx_HYX2hXyhrCLw, which could not be found
2023-11-14 21:21:21,273 - synapse.handlers.federation_event - 1742 - WARNING - _process_new_pulled_events_with_failed_pull_attempts-206-$N-epuXfdgHVb20KLF7o0wwN_ahseiHq0ph-6tJIw9WA-$Mgzc4YZ-W3_z2fL3igG9hjci5Af_BTG0-i7deqJEdio-$hp0HuMZaTkHkc49PYVcu0lqy_NL1EJA5dSsDtd2xN_c - Dropping event <FrozenEventV3 event_id=$hp0HuMZaTkHkc49PYVcu0lqy_NL1EJA5dSsDtd2xN_c, type=m.room.member, state_key=@hodgoblin:matrix.org, outlier=False>, which relies on auth_event $_GYON-YfJTgOFS8f4kvGqLbPGbr9tx_HYX2hXyhrCLw, which could not be found
2023-11-14 21:21:21,273 - synapse.handlers.federation_event - 1742 - WARNING - _process_new_pulled_events_with_failed_pull_attempts-206-$N-epuXfdgHVb20KLF7o0wwN_ahseiHq0ph-6tJIw9WA-$Mgzc4YZ-W3_z2fL3igG9hjci5Af_BTG0-i7deqJEdio-$-NukKwrCyezB2_TcKVapghXY3mNgsil2yqKw-1TCPog - Dropping event <FrozenEventV3 event_id=$-NukKwrCyezB2_TcKVapghXY3mNgsil2yqKw-1TCPog, type=m.room.member, state_key=@kostyaekaterina:matrix.org, outlier=False>, which relies on auth_event $_GYON-YfJTgOFS8f4kvGqLbPGbr9tx_HYX2hXyhrCLw, which could not be found
2023-11-14 21:21:21,273 - synapse.handlers.federation_event - 1742 - WARNING - _process_new_pulled_events_with_failed_pull_attempts-206-$N-epuXfdgHVb20KLF7o0wwN_ahseiHq0ph-6tJIw9WA-$Mgzc4YZ-W3_z2fL3igG9hjci5Af_BTG0-i7deqJEdio-$qloNFYsuACHHn3K0dKTFsYTy6ZvWAj1vxQ7dJ9ypxrw - Dropping event <FrozenEventV3 event_id=$qloNFYsuACHHn3K0dKTFsYTy6ZvWAj1vxQ7dJ9ypxrw, type=m.room.member, state_key=@matrx0th:matrix.org, outlier=False>, which relies on auth_event $_GYON-YfJTgOFS8f4kvGqLbPGbr9tx_HYX2hXyhrCLw, which could not be found
2023-11-14 21:21:21,273 - synapse.handlers.federation_event - 1742 - WARNING - _process_new_pulled_events_with_failed_pull_attempts-206-$N-epuXfdgHVb20KLF7o0wwN_ahseiHq0ph-6tJIw9WA-$Mgzc4YZ-W3_z2fL3igG9hjci5Af_BTG0-i7deqJEdio-$3zjjLhGqKV7F-PFWKMphnc7FO3B-fzZB1vEDDuL48ys - Dropping event <FrozenEventV3 event_id=$3zjjLhGqKV7F-PFWKMphnc7FO3B-fzZB1vEDDuL48ys, type=m.room.member, state_key=@underronin:matrix.org, outlier=False>, which relies on auth_event $_GYON-YfJTgOFS8f4kvGqLbPGbr9tx_HYX2hXyhrCLw, which could not be found
2023-11-14 21:21:21,275 - synapse.handlers.federation_event - 1742 - WARNING - _process_new_pulled_events_with_failed_pull_attempts-206-$N-epuXfdgHVb20KLF7o0wwN_ahseiHq0ph-6tJIw9WA-$Mgzc4YZ-W3_z2fL3igG9hjci5Af_BTG0-i7deqJEdio-$8KYJRHrJcZRh4q1Y2TvGhU7Sbpa7w12arxe1UOu1OL4 - Dropping event <FrozenEventV3 event_id=$8KYJRHrJcZRh4q1Y2TvGhU7Sbpa7w12arxe1UOu1OL4, type=m.room.member, state_key=@tttttttt:waifuhunter.club, outlier=False>, which relies on auth_event $_GYON-YfJTgOFS8f4kvGqLbPGbr9tx_HYX2hXyhrCLw, which could not be found
2023-11-14 21:21:21,275 - synapse.handlers.federation_event - 1742 - WARNING - _process_new_pulled_events_with_failed_pull_attempts-206-$N-epuXfdgHVb20KLF7o0wwN_ahseiHq0ph-6tJIw9WA-$Mgzc4YZ-W3_z2fL3igG9hjci5Af_BTG0-i7deqJEdio-$a7yDjHoWNoREUpLmH3YEzfCSsXZHhrLlge_97fGykjg - Dropping event <FrozenEventV3 event_id=$a7yDjHoWNoREUpLmH3YEzfCSsXZHhrLlge_97fGykjg, type=m.room.member, state_key=@fun_adminx:matrix.juggler.jp, outlier=False>, which relies on auth_event $_GYON-YfJTgOFS8f4kvGqLbPGbr9tx_HYX2hXyhrCLw, which could not be found
2023-11-14 21:21:21,276 - synapse.handlers.federation_event - 1742 - WARNING - _process_new_pulled_events_with_failed_pull_attempts-206-$N-epuXfdgHVb20KLF7o0wwN_ahseiHq0ph-6tJIw9WA-$Mgzc4YZ-W3_z2fL3igG9hjci5Af_BTG0-i7deqJEdio-$rjhdB4i0PxUu4PvSUtyN8lhgmgHmV2i_idxc81CPw9g - Dropping event <FrozenEventV3 event_id=$rjhdB4i0PxUu4PvSUtyN8lhgmgHmV2i_idxc81CPw9g, type=m.room.member, state_key=@jagasaki:koneko.chat, outlier=False>, which relies on auth_event $_GYON-YfJTgOFS8f4kvGqLbPGbr9tx_HYX2hXyhrCLw, which could not be found
2023-11-14 21:21:21,276 - synapse.handlers.federation_event - 1742 - WARNING - _process_new_pulled_events_with_failed_pull_attempts-206-$N-epuXfdgHVb20KLF7o0wwN_ahseiHq0ph-6tJIw9WA-$Mgzc4YZ-W3_z2fL3igG9hjci5Af_BTG0-i7deqJEdio-$OQphZ7H2jMD5HIje9JXyRx9j7-hpcgN1M_tKLK_BV6A - Dropping event <FrozenEventV3 event_id=$OQphZ7H2jMD5HIje9JXyRx9j7-hpcgN1M_tKLK_BV6A, type=m.room.member, state_key=@kemuri:koneko.chat, outlier=False>, which relies on auth_event $_GYON-YfJTgOFS8f4kvGqLbPGbr9tx_HYX2hXyhrCLw, which could not be found
2023-11-14 21:21:21,276 - synapse.handlers.federation_event - 1742 - WARNING - _process_new_pulled_events_with_failed_pull_attempts-206-$N-epuXfdgHVb20KLF7o0wwN_ahseiHq0ph-6tJIw9WA-$Mgzc4YZ-W3_z2fL3igG9hjci5Af_BTG0-i7deqJEdio-$J2VDMGJYfFitzsIX6dBomyk3vSX20rJ04RERJEaicPw - Dropping event <FrozenEventV3 event_id=$J2VDMGJYfFitzsIX6dBomyk3vSX20rJ04RERJEaicPw, type=m.room.member, state_key=@presloli:koneko.chat, outlier=False>, which relies on auth_event $_GYON-YfJTgOFS8f4kvGqLbPGbr9tx_HYX2hXyhrCLw, which could not be found
2023-11-14 21:21:21,276 - synapse.handlers.federation_event - 1742 - WARNING - _process_new_pulled_events_with_failed_pull_attempts-206-$N-epuXfdgHVb20KLF7o0wwN_ahseiHq0ph-6tJIw9WA-$Mgzc4YZ-W3_z2fL3igG9hjci5Af_BTG0-i7deqJEdio-$fn10pv4DQPFlnhv9-sCdz4S3x8oln7_y09KJZkgb_c4 - Dropping event <FrozenEventV3 event_id=$fn10pv4DQPFlnhv9-sCdz4S3x8oln7_y09KJZkgb_c4, type=m.room.member, state_key=@greexxx:midov.pl, outlier=False>, which relies on auth_event $_GYON-YfJTgOFS8f4kvGqLbPGbr9tx_HYX2hXyhrCLw, which could not be found
2023-11-14 21:21:21,277 - synapse.handlers.federation_event - 1742 - WARNING - _process_new_pulled_events_with_failed_pull_attempts-206-$N-epuXfdgHVb20KLF7o0wwN_ahseiHq0ph-6tJIw9WA-$Mgzc4YZ-W3_z2fL3igG9hjci5Af_BTG0-i7deqJEdio-$9Fkw2-BdSO_fG_Ua9afnzYdFn7VJ3_9UUWOf6jq_DzI - Dropping event <FrozenEventV3 event_id=$9Fkw2-BdSO_fG_Ua9afnzYdFn7VJ3_9UUWOf6jq_DzI, type=m.room.member, state_key=@gunuk:midov.pl, outlier=False>, which relies on auth_event $_GYON-YfJTgOFS8f4kvGqLbPGbr9tx_HYX2hXyhrCLw, which could not be found
2023-11-14 21:21:21,277 - synapse.handlers.federation_event - 1742 - WARNING - _process_new_pulled_events_with_failed_pull_attempts-206-$N-epuXfdgHVb20KLF7o0wwN_ahseiHq0ph-6tJIw9WA-$Mgzc4YZ-W3_z2fL3igG9hjci5Af_BTG0-i7deqJEdio-$zoHjtSPUDAXPLA3x3JGplPwiZAgObX6JxHzy4TLVye0 - Dropping event <FrozenEventV3 event_id=$zoHjtSPUDAXPLA3x3JGplPwiZAgObX6JxHzy4TLVye0, type=m.room.member, state_key=@krestomi:midov.pl, outlier=False>, which relies on auth_event $_GYON-YfJTgOFS8f4kvGqLbPGbr9tx_HYX2hXyhrCLw, which could not be found
2023-11-14 21:21:21,277 - synapse.handlers.federation_event - 1742 - WARNING - _process_new_pulled_events_with_failed_pull_attempts-206-$N-epuXfdgHVb20KLF7o0wwN_ahseiHq0ph-6tJIw9WA-$Mgzc4YZ-W3_z2fL3igG9hjci5Af_BTG0-i7deqJEdio-$FRVNQUfvTSfPdUiWZ0s1DbQHlOUAkH4b5kOxj7z29fI - Dropping event <FrozenEventV3 event_id=$FRVNQUfvTSfPdUiWZ0s1DbQHlOUAkH4b5kOxj7z29fI, type=m.room.member, state_key=@septemberrain:midov.pl, outlier=False>, which relies on auth_event $_GYON-YfJTgOFS8f4kvGqLbPGbr9tx_HYX2hXyhrCLw, which could not be found

Anything else that would be useful to know?

2023-11-14-222341_663x623_scrot
2023-11-14-224649_723x387_scrot
(the purple one is synapse- _process_new_pulled_events_with_failed_pull_attempts)

commented

Since the upgrade to Synapse 1.95 and 1.95.1

What version of Synapse did you upgrade from?

That particular error comes while backfilling events and comes from:

synapse/synapse/handlers/federation_event.py

Lines 1736 to 1748 in 43d1aa7

for auth_event_id in event.auth_event_ids():
ae = persisted_events.get(auth_event_id)
if not ae:
# the fact we can't find the auth event doesn't mean it doesn't
# exist, which means it is premature to reject `event`. Instead we
# just ignore it for now.
logger.warning(
"Dropping event %s, which relies on auth_event %s, which could not be found",
event,
auth_event_id,
)
return
auth.append(ae)

It looks like all those events are from the room !VXimQkuRxeRUFhboju:anontier.nl. I'm slightly suspicious that you have the retention policies feature enabled as that has been known to be buggy in the past. Have you had it enabled for a long time?

commented

Since the upgrade to Synapse 1.95 and 1.95.1

What version of Synapse did you upgrade from?

I upgraded from 1.92.3 to 1.95.0 then later to 1.95.1

It looks like all those events are from the room !VXimQkuRxeRUFhboju:anontier.nl.

How did you know which room these events are related to? After a quick look, I won't be sad if I have to purge this room from the server.

I'm slightly suspicious that you have the retention policies feature enabled as that has been known to be buggy in the past. Have you had it enabled for a long time?

I have it enabled for a long time yes, since Tue Aug 10 18:00:55 2021 exactly. And I was also afraid that it might be related.

commented

It looks like all those events are from the room !VXimQkuRxeRUFhboju:anontier.nl.

How did you know which room these events are related to? After a quick look, I won't be sad if I have to purge this room from the server.

Event IDs are globally unique, so I did:

SELECT room_id FROM events WHERE event_id = '...';

I'm slightly suspicious that you have the retention policies feature enabled as that has been known to be buggy in the past. Have you had it enabled for a long time?

I have it enabled for a long time yes, since Tue Aug 10 18:00:55 2021 exactly. And I was also afraid that it might be related.

There were known bugs in the feature until Synapse ~1.94, so I think this might be the cause. You could purge the room entirely and then rejoin it. I know that's not an ideal solution though.

commented

How did you know which room these events are related to? After a quick look, I won't be sad if I have to purge this room from the server.

Event IDs are globally unique, so I did:

SELECT room_id FROM events WHERE event_id = '...';

But if the event has been dropped as the log suggests, can I still try this query on my own homeserver?

I have it enabled for a long time yes, since Tue Aug 10 18:00:55 2021 exactly. And I was also afraid that it might be related.

There were known bugs in the feature until Synapse ~1.94, so I think this might be the cause.

That's what I saw by following a little bit the project on GitHub.

You could purge the room entirely and then rejoin it. I know that's not an ideal solution though.

Thank you for your advice, I just purged the room. We only had a single user from club1.fr in this room so that's not a big loss.
I will keep monitoring the server and the logs to see if this problem arise again, but until then I will close this issue.

commented

Feel free to shout if you see it come back!