Reducing Time for _changes Request in CouchDB
homu-konamilk opened this issue Β· comments
Thank you for developing and maintaining the Obsidian LiveSync plugin; it's been incredibly useful for syncing notes across my devices. However, I've encountered a significant issue related to request timeouts that frequently disrupts the synchronization process, and I'm hoping for some guidance or suggestions on how to address it.
Abstract
The synchronization process frequently fails with a TypeError: Failed to fetch
error. This failure seems to be related to the timing out of CouchDB's _changes
requests when accessed through Cloudflare Tunnel, based on observations and logs.
Environment
- Obsidian Version: 1.5.11
- Plugin Version: 0.22.16
- Server: self-hosted and accessed via Cloudflare Tunnel
Issue Detail
The process frequently fails with a TypeError: Failed to fetch
error. While Developer Tools suggest a CORS policy issue blocking the _changes
request to CouchDB, my observations indicate a potentially different cause.
Requests other than _changes
to the same CouchDB instance do not face similar issues, and _changes
requests consistently fail at exactly 100 seconds. Additionally, research into Cloudflare Tunnel's behavior points out an unofficial but widely noted timeout limit of 100 seconds.
These elements lead me to speculate that the disruptions are not due to CORS configuration but are likely caused by a timeout issue related to Cloudflare Tunnel.
Error Message from Developer Tools
Access to fetch at 'https://<couchdb_url>/_changes?style=all_docs&feed=longpoll&heartbeat=30000&filter=_selector&since=<since>&limit=25' from origin 'app://obsidian.md' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.
Cloudflare Tunnel Log Snippet
2024-03-25T06:37:24Z debug http POST https://<couchdb_url>/_changes?style=all_docs&feed=longpoll&heartbeat=30000&filter=_selector&since=<since>&limit=25 ...
2024-03-25T06:39:04Z error cloudflared Request failed {"connIndex":0,"dest":"https://<couchdb_url>/_changes?style=all_docs&feed=longpoll&heartbeat=30000&filter=_selector&since=<since>&limit=25","error":"context canceled","type":"http"}
Observations and Cloudflare Tunnel logs consistently show that each failure occurs exactly at the 100-second mark. This pattern suggests a timeout issue is at the heart of these frequent disruptions, rather than any misconfiguration with CORS.
Questions and Assistance Request
Given Cloudflare's 100-second timeout limit (which cannot be altered without an Enterprise plan), I'm seeking advice on potentially reducing the duration of the _changes
requests made by the plugin. Are there configuration options or optimizations within the plugin or CouchDB settings that could help mitigate this frequent timeout issue?
I appreciate any guidance you can provide and am willing to assist with further debugging information if needed. Thank you for your time and effort in maintaining this valuable plugin.
Report materials
Report from the LiveSync
Report from hatch
---- Obsidian info ----
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) obsidian/1.5.8 Chrome/120.0.6099.283 Electron/28.2.3 Safari/537.36
---- remote config ----
uuids:
algorithm: sequential
max_count: "1000"
cluster:
n: "3"
q: "2"
cors:
credentials: "true"
headers: accept, authorization, content-type, origin, referer
max_age: "3600"
methods: GET, PUT, POST, HEAD, DELETE
origins: app://obsidian.md,capacitor://localhost,http://localhost
chttpd:
backlog: "512"
bind_address: any
max_db_number_for_dbs_info_req: "100"
max_http_request_size: "4294967296"
port: "5984"
prefer_minimal: Cache-Control, Content-Length, Content-Range, Content-Type, ETag, Server, Transfer-Encoding, Vary
require_valid_user: "true"
server_options: "[{recbuf, undefined}]"
socket_options: "[{sndbuf, 262144}, {nodelay, true}]"
attachments:
compressible_types: text/*, application/javascript, application/json, application/xml
compression_level: "8"
admins: π
πΈπ·π΄πΆππΈπ·
query_server_config:
os_process_limit: "100"
reduce_limit: "true"
vendor:
name: The Apache Software Foundation
feature_flags:
partitioned||*: "true"
chttpd_auth:
authentication_redirect: /xxxxx/_utils/session.html
require_valid_user: "true"
replicator:
connection_timeout: "30000"
http_connections: "20"
interval: "60000"
max_churn: "20"
max_jobs: "500"
retries_per_request: "5"
socket_options: "[{keepalive, true}, {nodelay, false}]"
ssl_certificate_max_depth: "3"
startup_jitter: "5000"
verify_ssl_certificates: "false"
worker_batch_size: "500"
worker_processes: "4"
ssl:
port: "6984"
log:
level: info
writer: stderr
indexers:
couch_mrview: "true"
couch_peruser:
database_prefix: userdb-
delete_dbs: "false"
enable: "false"
httpd:
WWW-Authenticate: Basic realm="couchdb"
allow_jsonp: "false"
authentication_handlers: "{couch_httpd_auth, cookie_authentication_handler}, {couch_httpd_auth, default_authentication_handler}"
bind_address: 127.0.0.1
enable_cors: "true"
enable_xframe_options: "false"
max_http_request_size: "4294967296"
port: "5986"
secure_rewrites: "true"
socket_options: "[{sndbuf, 262144}]"
ioq.bypass:
compaction: "false"
os_process: "true"
read: "true"
shard_sync: "false"
view_update: "true"
write: "true"
ioq:
concurrency: "10"
ratio: "0.01"
couch_httpd_auth:
allow_persistent_cookies: "true"
auth_cache_size: "50"
authentication_db: π
πΈπ·π΄πΆππΈπ·
authentication_redirect: π
πΈπ·π΄πΆππΈπ·
iterations: "10"
require_valid_user: "false"
timeout: "600"
secret: π
πΈπ·π΄πΆππΈπ·
couchdb_engines:
couch: couch_bt_engine
couchdb:
attachment_stream_buffer_size: "4096"
changes_doc_ids_optimization_threshold: "100"
database_dir: ./data
default_engine: couch
default_security: admin_only
file_compression: snappy
max_dbs_open: "500"
max_document_size: "50000000"
os_process_timeout: "5000"
single_node: "true"
users_db_security_editable: "false"
uuid: π
πΈπ·π΄πΆππΈπ·
view_index_dir: ./data
---- Plug-in config ---
version:0.22.16
couchDB_URI: self-hosted(HTTPS)
couchDB_USER: π
πΈπ·π΄πΆππΈπ·
couchDB_PASSWORD: π
πΈπ·π΄πΆππΈπ·
couchDB_DBNAME: π
πΈπ·π΄πΆππΈπ·
liveSync: true
syncOnSave: false
syncOnStart: false
savingDelay: 200
lessInformationInLog: false
gcDelay: 0
versionUpFlash: ""
minimumChunkSize: 20
longLineThreshold: 250
showVerboseLog: false
suspendFileWatching: false
trashInsteadDelete: true
periodicReplication: false
periodicReplicationInterval: 60
syncOnFileOpen: false
encrypt: true
passphrase: π
πΈπ·π΄πΆππΈπ·
usePathObfuscation: true
doNotDeleteFolder: false
resolveConflictsByNewerFile: false
batchSave: false
deviceAndVaultName: ""
usePluginSettings: false
showOwnPlugins: false
showStatusOnEditor: true
showStatusOnStatusbar: true
showOnlyIconsOnEditor: false
usePluginSync: false
autoSweepPlugins: false
autoSweepPluginsPeriodic: false
notifyPluginOrSettingUpdated: false
checkIntegrityOnSave: false
batch_size: 25
batches_limit: 25
useHistory: true
disableRequestURI: true
skipOlderFilesOnSync: true
checkConflictOnlyOnOpen: false
showMergeDialogOnlyOnActive: false
syncInternalFiles: false
syncInternalFilesBeforeReplication: false
syncInternalFilesIgnorePatterns: \/node_modules\/, \/\.git\/, \/obsidian-livesync\/
syncInternalFilesInterval: 60
additionalSuffixOfDatabaseName: edbb3b83325e880e
ignoreVersionCheck: false
lastReadUpdates: 22
deleteMetadataOfDeletedFiles: false
syncIgnoreRegEx: ""
syncOnlyRegEx: ""
customChunkSize: 50
readChunksOnline: true
watchInternalFileChanges: true
automaticallyDeleteMetadataOfDeletedFiles: 0
disableMarkdownAutoMerge: false
writeDocumentsIfConflicted: false
useDynamicIterationCount: false
syncAfterMerge: false
configPassphraseStore: ""
encryptedPassphrase: π
πΈπ·π΄πΆππΈπ·
encryptedCouchDBConnection: π
πΈπ·π΄πΆππΈπ·
permitEmptyPassphrase: false
useIndexedDBAdapter: true
useTimeouts: false
writeLogToTheFile: false
doNotPaceReplication: false
hashCacheMaxCount: 300
hashCacheMaxAmount: 50
concurrencyOfReadChunksOnline: 30
minimumIntervalOfReadChunksOnline: 25
hashAlg: xxhash64
suspendParseReplicationResult: false
doNotSuspendOnFetching: false
useIgnoreFiles: false
ignoreFiles: .gitignore
syncOnEditorSave: false
pluginSyncExtendedSetting: {}
syncMaxSizeInMB: 50
settingSyncFile: ""
writeCredentialsForSettingSync: false
notifyAllSettingSyncFile: false
isConfigured: true
settingVersion: 0
Obsidian debug info
Debug info
SYSTEM INFO:
Obsidian version: v1.5.11
Installer version: v1.5.8
Operating system: Darwin Kernel Version 23.3.0: Thu Dec 21 02:29:41 PST 2023; root:xnu-10002.81.5~11/RELEASE_ARM64_T8122 23.3.0
Login status: not logged in
Insider build toggle: off
Live preview: on
Base theme: light
Community theme: none
Snippets enabled: 0
Restricted mode: off
Plugins installed: 3
Plugins enabled: 2
1: Code Editor Shortcuts v1.14.0
2: Self-hosted LiveSync v0.22.16
RECOMMENDATIONS:
Community plugins: for bugs, please first try updating all your plugins to latest. If still not fixed, please try to make the issue happen in the Sandbox Vault or disable community plugins.
Plug-in log
Plug-in log
2024/3/25 16:40:13->LiveSync begin...
2024/3/25 16:40:17->Replication paused
2024/3/25 16:40:17->Replication paused
2024/3/25 16:41:58->HTTP:POST (36) to:/_changes?style=all_docs&feed=longpoll&heartbeat=30000&filter=_selector&since=98460-g1AAAACReJzLYWBgYMpgTmHgzcvPy09JdcjLz8gvLskBCScyJNX___8_K4M5iYFh_5xcoBi7salhqoFpCrp6HCbksQBJhgYg9R9u0EEGsEGGZqYmRomJ6NqyAGsmLGM&limit=25 -> failed
2024/3/25 16:41:58->TypeError:Failed to fetch
2024/3/25 16:41:58->Replication paused
2024/3/25 16:42:00->Replication paused
Network log
Sorry, I realized I misunderstood CouchDB's behavior for _changes
requests. I see that when the feed=longpoll
parameter is given, it waits until there is a change, instead of sending a response immediately.
I discovered a solution within the plugin's settings. By enabling the Use timeouts instead of heartbeats
option, I was able to resolve the issue effectively.
My apologies for any confusion caused, and I hope this resolution might be of help to others who might face similar issues.