Server timeout when prefetching a large amount of refs
martiert opened this issue · comments
client: vdirsyncer-0.19.2, also reproduced with HEAD of main branch.
server: davmail-6.2.0
python: 3.11.9
OS: Linux (NixOS)
debug: <CIMultiDictProxy('Server': 'DavMail Gateway 6.2.0-3464', 'DAV': '1, calendar-access, calendar-auto-schedule, calendarserver-private-events, addressbook', 'Date': 'Fri, 26 Apr 2024 07:13:21 +0000
', 'Expires': 'Fri, 26 Apr 2024 07:13:21 +0000', 'Cache-Control': 'private, max-age=0', 'Transfer-Encoding': 'chunked', 'Content-Type': 'text/xml;charset=UTF-8', 'Connection': 'keep-alive')>
debug: <StreamReader>
error: Unknown error occurred for calendar_cisco/calendar:
error: Use `-vdebug` to see the full traceback.
debug: File "/nix/store/91zpapc0v9sy12f0im5kck4w7m3585q9-python3.11-vdirsyncer-0.19.2/lib/python3.11/site-packages/vdirsyncer/cli/tasks.py", line 72, in sync_collection
debug: await sync.sync(
debug: File "/nix/store/91zpapc0v9sy12f0im5kck4w7m3585q9-python3.11-vdirsyncer-0.19.2/lib/python3.11/site-packages/vdirsyncer/sync/__init__.py", line 146, in sync
debug: a_nonempty = await a_info.prepare_new_status()
debug: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
debug: File "/nix/store/91zpapc0v9sy12f0im5kck4w7m3585q9-python3.11-vdirsyncer-0.19.2/lib/python3.11/site-packages/vdirsyncer/sync/__init__.py", line 66, in prepare_new_status
debug: async for href, item, etag in self.storage.get_multi(prefetch):
debug: File "/nix/store/91zpapc0v9sy12f0im5kck4w7m3585q9-python3.11-vdirsyncer-0.19.2/lib/python3.11/site-packages/vdirsyncer/storage/dav.py", line 529, in get_multi
debug: await response.content.read()
debug: File "/nix/store/2y4s33nck0v5x3grszrcb7c1vmbgg3gm-python3.11-aiohttp-3.9.3/lib/python3.11/site-packages/aiohttp/streams.py", line 373, in read
debug: block = await self.readany()
debug: ^^^^^^^^^^^^^^^^^^^^
debug: File "/nix/store/2y4s33nck0v5x3grszrcb7c1vmbgg3gm-python3.11-aiohttp-3.9.3/lib/python3.11/site-packages/aiohttp/streams.py", line 395, in readany
debug: await self._wait("readany")
debug: File "/nix/store/2y4s33nck0v5x3grszrcb7c1vmbgg3gm-python3.11-aiohttp-3.9.3/lib/python3.11/site-packages/aiohttp/streams.py", line 301, in _wait
debug: with self._timer:
debug: File "/nix/store/2y4s33nck0v5x3grszrcb7c1vmbgg3gm-python3.11-aiohttp-3.9.3/lib/python3.11/site-packages/aiohttp/helpers.py", line 735, in __exit__
debug: raise asyncio.TimeoutError from None
A potential fix for the issue can be found in https://github.com/martiert/vdirsyncer/tree/batch_prefetch which batches the requests into chunks of 20