MongoError: connection 0 to localhost:27017 timed out —> Topology was destroyed
gordongli opened this issue · comments
Many errors of this kind appear after node makedash.js runs for a while, and the script fails to generate many pages/assets.
err { MongoError: Topology was destroyed
at nextFunction (/home/gordonl/Measure/node_modules/mongodb/node_modules/mongodb-core/lib/cursor.js:547:27)
at Cursor.next [as _next] (/home/gordonl/Measure/node_modules/mongodb/node_modules/mongodb-core/lib/cursor.js:701:3)
at fetchDocs (/home/gordonl/Measure/node_modules/mongodb/lib/cursor.js:857:10)
at toArray (/home/gordonl/Measure/node_modules/mongodb/lib/cursor.js:884:3)
at /home/gordonl/Measure/node_modules/mongodb/lib/cursor.js:841:5
at Cursor.toArray (/home/gordonl/Measure/node_modules/mongodb/lib/cursor.js:840:10)
at Object.module.exports.dateDiffsByTimePeriods (/home/gordonl/Measure/widgets/widgetUtils.js:84:28)
at Object.module.exports [as module] (/home/gordonl/Measure/widgets/repo/34_prTimeStatsChart.js:5:17)
at async.mapSeries.results.filter.h (/home/gordonl/Measure/lib/dashboards.js:96:28)
at /home/gordonl/Measure/node_modules/async/dist/async.js:1121:9 name: 'MongoError', message: 'Topology was destroyed' }
One of the dashboard widgets ("prTimeStatsChart", running against
the "sanjayy" contributor had a problem, so I've skipped over it.
This is really an internal error, and should be reported.
(The error is described like this, which will help in the report:
Topology was destroyed,
node_modules/mongodb/node_modules/mongodb-core/lib/cursor.js:547)
@gordongli Out of curiosity, how much memory does the instance you're running mongo on have?
GHCrawler with Docker is running on a Linux box with 24GB memory. docker ps -s shows that the docker_mongo_1's size is 0B (virtual 366MB). Connecting to mongo from remote, show dbs shows ghcrawler 0.845GB.
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES SIZE
0a9356050aa6 docker_dashboard "npm start" 3 days ago Up 3 days 0.0.0.0:4000->4000/tcp, 5858/tcp docker_dashboard_1 0B (virtual 840MB)
078fe325891b docker_crawler "npm start" 3 days ago Up 3 days 0.0.0.0:3000->3000/tcp, 5858/tcp docker_crawler_1 0B (virtual 812MB)
35cfed5c7969 metabase/metabase:latest "/app/run_metabase.sh" 3 days ago Up 3 days 0.0.0.0:5000->3000/tcp docker_metabase_1 554kB (virtual 245MB)
5ade5270fc42 rabbitmq:management "docker-entrypoint.s…" 3 days ago Up 3 days 4369/tcp, 5671/tcp, 0.0.0.0:5672->5672/tcp, 15671/tcp, 25672/tcp, 0.0.0.0:15672->15672/tcp docker_rabbitmq_1 144B (virtual 151MB)
5279e526c7d8 mongo:latest "docker-entrypoint.s…" 3 days ago Up 3 days 0.0.0.0:27017->27017/tcp, 0.0.0.0:28017->28017/tcp docker_mongo_1 0B (virtual 366MB)
21b4d6dfe9c0 redis:latest "docker-entrypoint.s…" 3 days ago Up 3 days 0.0.0.0:6379->6379/tcp docker_redis_1 0B (virtual 107MB)
While we were seeing this with very small instances during early build, I've not been able to reproduce it recently. We're looking into it further now.
@gordongli I've made some changes to the error reporting. "Topology was destroyed" is a cascade error; it happens when something else goes wrong, and then our control flow has a problem and causes all the widgets to throw an error. If you're still experiencing the problem, then before all the "topology" errors there should be another error reported which is the thing that's the actual problem; if you could follow up with that error, we can fix it.
I tried it again, and it was mongo timing out:
connection 0 to localhost:27017 timed out
MongoError: connection 0 to localhost:27017 timed out
at Function.MongoError.create (/home/gordonl/Measure/node_modules/mongodb/node_modules/mongodb-core/lib/error.js:29:11)
at Socket.<anonymous> (/home/gordonl/Measure/node_modules/mongodb/node_modules/mongodb-core/lib/connection/connection.js:200:20)
at Socket.g (events.js:291:16)
at emitNone (events.js:86:13)
at Socket.emit (events.js:185:7)
at Socket._onTimeout (net.js:339:8)
at ontimeout (timers.js:365:14)
at tryOnTimeout (timers.js:237:5)
at Timer.listOnTimeout (timers.js:207:5)
I see you've made a fix in the most recent commit to execute the calls in batches, which seems to prevent that. I've run into another issue, however, which is the node running out of heap memory (I believe this has occurred before as well), and it still relates to mongo. I've made a new issue here: #76
<--- Last few GCs --->
5382881 ms: Mark-sweep 1303.3 (1404.6) -> 1303.2 (1406.6) MB, 1540.7 / 0.0 ms [allocation failure] [GC in old space requested].
5384277 ms: Mark-sweep 1303.2 (1406.6) -> 1303.2 (1406.6) MB, 1395.6 / 0.0 ms [allocation failure] [GC in old space requested].
5385622 ms: Mark-sweep 1303.2 (1406.6) -> 1305.1 (1404.6) MB, 1345.0 / 0.0 ms [last resort gc].
5386890 ms: Mark-sweep 1305.1 (1404.6) -> 1307.1 (1404.6) MB, 1267.4 / 0.0 ms [last resort gc].
<--- JS stacktrace --->
==== JS stack trace =========================================
Security context: 0x3ff7c8cfb51 <JS Object>
2: _next [/home/gordonl/Measure/node_modules/mongodb/node_modules/mongodb-core/lib/cursor.js:~700] [pc=0x51e82082034] (this=0xe9c5a8d3859 <a Cursor with map 0x3a49d2873cd1>,callback=0x3b7e7ca24ec9 <JS Function (SharedFunctionInfo 0x265f061c5641)>)
3: fetchDocs(aka fetchDocs) [/home/gordonl/Measure/node_modules/mongodb/lib/cursor.js:857] [pc=0x51e822cbcc3] (this=0x3ff7c804381 <undefined>)...
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
1: node::Abort() [node]
2: 0x101167c [node]
3: v8::Utils::ReportApiFailure(char const*, char const*) [node]
4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [node]
5: v8::internal::Factory::NewFixedArray(int, v8::internal::PretenureFlag) [node]
6: v8::internal::TypeFeedbackVector::New(v8::internal::Isolate*, v8::internal::Handle<v8::internal::TypeFeedbackMetadata>) [node]
7: 0xafa2e0 [node]
8: 0xafa493 [node]
9: 0xafac08 [node]
10: 0xafb86e [node]
11: 0xb03dbb [node]
12: v8::internal::Compiler::Compile(v8::internal::Handle<v8::internal::JSFunction>, v8::internal::Compiler::ClearExceptionFlag) [node]
13: v8::internal::Runtime_CompileLazy(int, v8::internal::Object**, v8::internal::Isolate*) [node]
14: 0x51e80e092a7
Aborted
Hi,
I'm also getting the same connection timeout error in my server when i deployed my application in my local machine everything is working as expected but only for few bigger database queries i'm getting this error :
{ result: { result: { MongoNetworkError: connection 4 to localhost:27017 timed out at Socket.<anonymous> (/var/www/.../../node_modules/mongodb-core/lib/connection/connection.js:259:7) at Object.onceWrapper (events.js:277:13) at Socket.emit (events.js:189:13) at Socket._onTimeout (net.js:443:8) at ontimeout (timers.js:436:11) at tryOnTimeout (timers.js:300:5) at listOnTimeout (timers.js:263:5) at Timer.processTimers (timers.js:223:10) name: 'MongoNetworkError', errorLabels: [Array], [Symbol(mongoErrorContextSymbol)]: {} } } }
Please guide me how to fix this issue. thanks in advance.
@jogendra-yadav just to confirm: are you using Measure when you get this error? Or a different application?
@stuartlangridge : No I'm not, But here is my db query which is giving this error :
dbConnection.production.collection(collectionName).aggregate([{ $match: { isMapped: false }}, { $group: { _id: { supplier: "$supplier", hc: "$hc" }, mp_inv: { "$first": "$mp_inv" }, lng: { "$first": "$lng" }, addr: { "$first": "$addr" }, ccr: { "$first": "$ccr" }, hn: { "$first": "$hn" }, isclubbedfare: { "$first": "$isclubbedfare" }, disc: { "$first": "$disc" }, instanceKey: { "$first": "$instanceKey" }, city: { "$first": "$city" }, th: { "$first": "$th" }, oc: { "$first": "$oc" }, area: { "$first": "$area" }, cKey: { "$first": "$cKey" }, dk: { "$first": "$dk" }, ta: { "$first": "$ta" }, isMapped: { "$first": "$isMapped" }, fs: { "$first": "$fs" }, amenitiesList: { "$first": "$amenitiesList" }, baseId: { "$first": "$baseId" }, supplier: { "$first": "$supplier" }, key: { "$first": "$key" }, lat: { "$first": "$lat" }, riArr: { "$first": "$riArr" }, mp: { "$first": "$mp" }, sr: { "$first": "$sr" }, ispreprred: { "$first": "$ispreprred" }, isPostPaid: { "$first": "$isPostPaid" }, currencyType: { "$first": "$currencyType" }, hc: { "$first": "$hc" }, trac: { "$first": "$trac" }, am: { "$first": "$am" }, pt: { "$first": "$pt" }, maxp: { "$first": "$maxp" }, dist: { "$first": "$dist" } }}], { allowDiskUse: true })
This collection has many number of documents. And it will increase more.
@jogendra-yadav I'm sorry, but this is the repository for the Measure software, not a general MongoDB help account; we won't be able to help you here. The software you're using may have its own help forum -- the documentation for it may tell you that -- or you may have more luck posting this question on Stack Overflow (https://stackoverflow.com) which is designed as a help forum for more general programming questions. Good luck finding an answer!