P0 - Investigate balance aggregator communication error on production
aminlatifi opened this issue · comments
In impact-graph logs there are footprints of communication issues with balance aggregator
@aminlatifi Can you share the log?? And the time of the error when the error happend?
@aminlatifi Can you share the log?? And the time of the error when the error happend?
You can see them here
There are plenty like this
{
"name": "giveth-impact-graph",
"hostname": "aafc3d6891f2",
"pid": 70,
"level": 50,
"msg": "getLeastIndexedBlockTimeStamp >> error Error: Request failed with status code 403\n at createError (/usr/src/app/node_modules/axios/lib/core/createError.js:16:15)\n at settle (/usr/src/app/node_modules/axios/lib/core/settle.js:17:12)\n at IncomingMessage.handleStreamEnd (/usr/src/app/node_modules/axios/lib/adapters/http.js:269:11)\n at IncomingMessage.emit (node:events:538:35)\n at IncomingMessage.emit (node:domain:475:12)\n at endReadableNT (node:internal/streams/readable:1345:12)\n at processTicksAndRejections (node:internal/process/task_queues:83:21) {\n config: {\n url: 'https://aggregator.giveth.io/fetch-state/least-indexed-block-timestamp',\n method: 'get',\n headers: {\n Accept: 'application/json, text/plain, */*',\n 'User-Agent': 'axios/0.21.4'\n },\n params: {},\n transformRequest: [ [Function: transformRequest] ],\n transformResponse: [ [Function: transformResponse] ],\n timeout: 0,\n adapter: [Function: httpAdapter],\n xsrfCookieName: 'XSRF-TOKEN',\n xsrfHeaderName: 'X-XSRF-TOKEN',\n maxContentLength: -1,\n maxBodyLength: -1,\n validateStatus: [Function: validateStatus],\n transitional: {\n silentJSONParsing: true,\n forcedJSONParsing: true,\n clarifyTimeoutError: false\n },\n 'axios-retry': { retryCount: 0, lastRequestTime: 1704279020446 },\n data: undefined\n },\n request: <ref *1> ClientRequest {\n _events: [Object: null prototype] {\n error: [Array],\n abort: [Function (anonymous)],\n aborted: [Function (anonymous)],\n connect: [Function (anonymous)],\n socket: [Function (anonymous)],\n timeout: [Function (anonymous)],\n prefinish: [Function: requestOnPrefinish]\n },\n _eventsCount: 7,\n _maxListeners: undefined,\n outputData: [],\n outputSize: 0,\n writable: true,\n destroyed: false,\n _last: true,\n chunkedEncoding: false,\n shouldKeepAlive: false,\n maxRequestsOnConnectionReached: false,\n _defaultKeepAlive: true,\n useChunkedEncodingByDefault: false,\n sendDate: false,\n _removedConnection: false,\n _removedContLen: false,\n _removedTE: false,\n _contentLength: 0,\n _hasBody: true,\n _trailer: '',\n finished: true,\n _headerSent: true,\n _closed: false,\n socket: TLSSocket {\n _tlsOptions: [Object],\n _secureEstablished: true,\n _securePending: false,\n _newSessionPending: false,\n _controlReleased: true,\n secureConnecting: false,\n _SNICallback: null,\n servername: 'aggregator.giveth.io',\n alpnProtocol: false,\n authorized: true,\n authorizationError: null,\n encrypted: true,\n _events: [Object: null prototype],\n _eventsCount: 10,\n connecting: false,\n _hadError: false,\n _parent: null,\n _host: 'aggregator.giveth.io',\n _readableState: [ReadableState],\n _maxListeners: undefined,\n _writableState: [WritableState],\n allowHalfOpen: false,\n _sockname: null,\n _pendingData: null,\n _pendingEncoding: '',\n server: undefined,\n _server: null,\n ssl: [TLSWrap],\n _requestCert: true,\n _rejectUnauthorized: true,\n parser: null,\n _httpMessage: [Circular *1],\n [Symbol(res)]: [TLSWrap],\n [Symbol(verified)]: true,\n [Symbol(pendingSession)]: null,\n [Symbol(async_id_symbol)]: 32726462,\n [Symbol(kHandle)]: [TLSWrap],\n [Symbol(kSetNoDelay)]: false,\n [Symbol(lastWriteQueueSize)]: 0,\n [Symbol(timeout)]: null,\n [Symbol(kBuffer)]: null,\n [Symbol(kBufferCb)]: null,\n [Symbol(kBufferGen)]: null,\n [Symbol(kCapture)]: false,\n [Symbol(kBytesRead)]: 0,\n [Symbol(kBytesWritten)]: 0,\n [Symbol(connect-options)]: [Object],\n [Symbol(RequestTimeout)]: undefined\n },\n _header: 'GET /fetch-state/least-indexed-block-timestamp HTTP/1.1\\r\\n' +\n 'Accept: application/json, text/plain, */*\\r\\n' +\n 'User-Agent: axios/0.21.4\\r\\n' +\n 'Host: aggregator.giveth.io\\r\\n' +\n 'Connection: close\\r\\n' +\n '\\r\\n',\n _keepAliveTimeout: 0,\n _onPendingData: [Function: nop],\n agent: Agent {\n _events: [Object: null prototype],\n _eventsCount: 2,\n _maxListeners: undefined,\n defaultPort: 443,\n protocol: 'https:',\n options: [Object: null prototype],\n requests: [Object: null prototype] {},\n sockets: [Object: null prototype],\n freeSockets: [Object: null prototype] {},\n keepAliveMsecs: 1000,\n keepAlive: false,\n maxSockets: Infinity,\n maxFreeSockets: 256,\n scheduling: 'lifo',\n maxTotalSockets: Infinity,\n totalSocketCount: 1,\n maxCachedSessions: 100,\n _sessionCache: [Object],\n [Symbol(kCapture)]: false\n },\n socketPath: undefined,\n method: 'GET',\n maxHeaderSize: undefined,\n insecureHTTPParser: undefined,\n path: '/fetch-state/least-indexed-block-timestamp',\n _ended: true,\n res: IncomingMessage {\n _readableState: [ReadableState],\n _events: [Object: null prototype],\n _eventsCount: 3,\n _maxListeners: undefined,\n socket: [TLSSocket],\n httpVersionMajor: 1,\n httpVersionMinor: 1,\n httpVersion: '1.1',\n complete: true,\n rawHeaders: [Array],\n rawTrailers: [],\n aborted: false,\n upgrade: false,\n url: '',\n method: null,\n statusCode: 403,\n statusMessage: 'Forbidden',\n client: [TLSSocket],\n _consuming: false,\n _dumped: false,\n req: [Circular *1],\n responseUrl: 'https://aggregator.giveth.io/fetch-state/least-indexed-block-timestamp',\n redirects: [],\n [Symbol(kCapture)]: false,\n [Symbol(kHeaders)]: [Object],\n [Symbol(kHeadersCount)]: 12,\n [Symbol(kTrailers)]: null,\n [Symbol(kTrailersCount)]: 0,\n [Symbol(RequestTimeout)]: undefined\n },\n aborted: false,\n timeoutCb: null,\n upgradeOrConnect: false,\n parser: null,\n maxHeadersCount: null,\n reusedSocket: false,\n host: 'aggregator.giveth.io',\n protocol: 'https:',\n _redirectable: Writable {\n _writableState: [WritableState],\n _events: [Object: null prototype],\n _eventsCount: 2,\n _maxListeners: undefined,\n _options: [Object],\n _ended: true,\n _ending: true,\n _redirectCount: 0,\n _redirects: [],\n _requestBodyLength: 0,\n _requestBodyBuffers: [],\n _onNativeResponse: [Function (anonymous)],\n _currentRequest: [Circular *1],\n _currentUrl: 'https://aggregator.giveth.io/fetch-state/least-indexed-block-timestamp',\n [Symbol(kCapture)]: false\n },\n [Symbol(kCapture)]: false,\n [Symbol(kNeedDrain)]: false,\n [Symbol(corked)]: 0,\n [Symbol(kOutHeaders)]: [Object: null prototype] {\n accept: [Array],\n 'user-agent': [Array],\n host: [Array]\n }\n },\n response: {\n status: 403,\n statusText: 'Forbidden',\n headers: {\n server: 'nginx/1.24.0',\n date: 'Wed, 03 Jan 2024 10:50:20 GMT',\n 'content-type': 'text/html',\n 'content-length': '153',\n connection: 'close',\n 'strict-transport-security': 'max-age=31536000; includeSubDomains'\n },\n config: {\n url: 'https://aggregator.giveth.io/fetch-state/least-indexed-block-timestamp',\n method: 'get',\n headers: [Object],\n params: {},\n transformRequest: [Array],\n transformResponse: [Array],\n timeout: 0,\n adapter: [Function: httpAdapter],\n xsrfCookieName: 'XSRF-TOKEN',\n xsrfHeaderName: 'X-XSRF-TOKEN',\n maxContentLength: -1,\n maxBodyLength: -1,\n validateStatus: [Function: validateStatus],\n transitional: [Object],\n 'axios-retry': [Object],\n data: undefined\n },\n request: <ref *1> ClientRequest {\n _events: [Object: null prototype],\n _eventsCount: 7,\n _maxListeners: undefined,\n outputData: [],\n outputSize: 0,\n writable: true,\n destroyed: false,\n _last: true,\n chunkedEncoding: false,\n shouldKeepAlive: false,\n maxRequestsOnConnectionReached: false,\n _defaultKeepAlive: true,\n useChunkedEncodingByDefault: false,\n sendDate: false,\n _removedConnection: false,\n _removedContLen: false,\n _removedTE: false,\n _contentLength: 0,\n _hasBody: true,\n _trailer: '',\n finished: true,\n _headerSent: true,\n _closed: false,\n socket: [TLSSocket],\n _header: 'GET /fetch-state/least-indexed-block-timestamp HTTP/1.1\\r\\n' +\n 'Accept: application/json, text/plain, */*\\r\\n' +\n 'User-Agent: axios/0.21.4\\r\\n' +\n 'Host: aggregator.giveth.io\\r\\n' +\n 'Connection: close\\r\\n' +\n '\\r\\n',\n _keepAliveTimeout: 0,\n _onPendingData: [Function: nop],\n agent: [Agent],\n socketPath: undefined,\n method: 'GET',\n maxHeaderSize: undefined,\n insecureHTTPParser: undefined,\n path: '/fetch-state/least-indexed-block-timestamp',\n _ended: true,\n res: [IncomingMessage],\n aborted: false,\n timeoutCb: null,\n upgradeOrConnect: false,\n parser: null,\n maxHeadersCount: null,\n reusedSocket: false,\n host: 'aggregator.giveth.io',\n protocol: 'https:',\n _redirectable: [Writable],\n [Symbol(kCapture)]: false,\n [Symbol(kNeedDrain)]: false,\n [Symbol(corked)]: 0,\n [Symbol(kOutHeaders)]: [Object: null prototype]\n },\n data: '<html>\\r\\n' +\n '<head><title>403 Forbidden</title></head>\\r\\n' +\n '<body>\\r\\n' +\n '<center><h1>403 Forbidden</h1></center>\\r\\n' +\n '<hr><center>nginx/1.24.0</center>\\r\\n' +\n '</body>\\r\\n' +\n '</html>\\r\\n'\n },\n isAxiosError: true,\n toJSON: [Function: toJSON]\n}",
"time": "2024-01-03T10:50:20.459Z",
"v": 0
}
Still exists
{
"name": "giveth-impact-graph",
"hostname": "1dd8c1009a36",
"pid": 88,
"level": 50,
"msg": "getLeastIndexedBlockTimeStamp >> error Error: connect ECONNREFUSED 127.0.1.1:443\n at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1157:16) {\n errno: -111,\n code: 'ECONNREFUSED',\n syscall: 'connect',\n address: '127.0.1.1',\n port: 443,\n config: {\n url: 'https://dev.giveth.io/givpower-balance-aggregator/fetch-state/least-indexed-block-timestamp',\n method: 'get',\n headers: {\n Accept: 'application/json, text/plain, */*',\n 'User-Agent': 'axios/0.21.4'\n },\n params: {},\n transformRequest: [ [Function (anonymous)] ],\n transformResponse: [ [Function: transformResponse] ],\n timeout: 0,\n adapter: [Function: httpAdapter],\n xsrfCookieName: 'XSRF-TOKEN',\n xsrfHeaderName: 'X-XSRF-TOKEN',\n maxContentLength: -1,\n maxBodyLength: -1,\n validateStatus: [Function: validateStatus],\n transitional: {\n silentJSONParsing: true,\n forcedJSONParsing: true,\n clarifyTimeoutError: false\n },\n 'axios-retry': { retryCount: 3, lastRequestTime: 1704625246565 },\n data: undefined\n },\n request: <ref *1> Writable {\n _writableState: WritableState {\n objectMode: false,\n highWaterMark: 16384,\n finalCalled: false,\n needDrain: false,\n ending: false,\n ended: false,\n finished: false,\n destroyed: false,\n decodeStrings: true,\n defaultEncoding: 'utf8',\n length: 0,\n writing: false,\n corked: 0,\n sync: true,\n bufferProcessing: false,\n onwrite: [Function: bound onwrite],\n writecb: null,\n writelen: 0,\n afterWriteTickInfo: null,\n buffered: [],\n bufferedIndex: 0,\n allBuffers: true,\n allNoop: true,\n pendingcb: 0,\n constructed: true,\n prefinished: false,\n errorEmitted: false,\n emitClose: true,\n autoDestroy: true,\n errored: null,\n closed: false,\n closeEmitted: false,\n [Symbol(kOnFinished)]: []\n },\n _events: [Object: null prototype] {\n response: [Function: handleResponse],\n error: [Function: handleRequestError]\n },\n _eventsCount: 2,\n _maxListeners: undefined,\n _options: {\n maxRedirects: 21,\n maxBodyLength: 10485760,\n protocol: 'https:',\n path: '/givpower-balance-aggregator/fetch-state/least-indexed-block-timestamp',\n method: 'GET',\n headers: [Object],\n agent: undefined,\n agents: [Object],\n auth: undefined,\n hostname: 'dev.giveth.io',\n port: null,\n nativeProtocols: [Object],\n pathname: '/givpower-balance-aggregator/fetch-state/least-indexed-block-timestamp'\n },\n _ended: true,\n _ending: true,\n _redirectCount: 0,\n _redirects: [],\n _requestBodyLength: 0,\n _requestBodyBuffers: [],\n _onNativeResponse: [Function (anonymous)],\n _currentRequest: ClientRequest {\n _events: [Object: null prototype],\n _eventsCount: 7,\n _maxListeners: undefined,\n outputData: [],\n outputSize: 0,\n writable: true,\n destroyed: false,\n _last: true,\n chunkedEncoding: false,\n shouldKeepAlive: false,\n maxRequestsOnConnectionReached: false,\n _defaultKeepAlive: true,\n useChunkedEncodingByDefault: false,\n sendDate: false,\n _removedConnection: false,\n _removedContLen: false,\n _removedTE: false,\n _contentLength: 0,\n _hasBody: true,\n _trailer: '',\n finished: true,\n _headerSent: true,\n _closed: false,\n socket: [TLSSocket],\n _header: 'GET /givpower-balance-aggregator/fetch-state/least-indexed-block-timestamp HTTP/1.1\\r\\n' +\n 'Accept: application/json, text/plain, */*\\r\\n' +\n 'User-Agent: axios/0.21.4\\r\\n' +\n 'Host: dev.giveth.io\\r\\n' +\n 'Connection: close\\r\\n' +\n '\\r\\n',\n _keepAliveTimeout: 0,\n _onPendingData: [Function: nop],\n agent: [Agent],\n socketPath: undefined,\n method: 'GET',\n maxHeaderSize: undefined,\n insecureHTTPParser: undefined,\n path: '/givpower-balance-aggregator/fetch-state/least-indexed-block-timestamp',\n _ended: false,\n res: null,\n aborted: false,\n timeoutCb: null,\n upgradeOrConnect: false,\n parser: null,\n maxHeadersCount: null,\n reusedSocket: false,\n host: 'dev.giveth.io',\n protocol: 'https:',\n _redirectable: [Circular *1],\n [Symbol(kCapture)]: false,\n [Symbol(kNeedDrain)]: false,\n [Symbol(corked)]: 0,\n [Symbol(kOutHeaders)]: [Object: null prototype]\n },\n _currentUrl: 'https://dev.giveth.io/givpower-balance-aggregator/fetch-state/least-indexed-block-timestamp',\n [Symbol(kCapture)]: false\n },\n response: undefined,\n isAxiosError: true,\n toJSON: [Function: toJSON]\n}",
"time": "2024-01-07T11:00:46.569Z",
"v": 0
}
Impact-graph had not been able to fetch power balances from balance aggregator for last two rounds, at least 2 weeks!
I'm jumping on that and taking a look
When moving production impact graph to the new giveth-all deployment, we left the balance-aggregator service on the original server. However - the reverse proxy used in that instance (nginx) had very restricted allowlist settings. I added the new backend under the allowed IPs and restarted nginx. I hope this alleviated the issue - can you check again @aminlatifi ?
I will open another issue to integrate this service also to giveth-all, or the upcoming giveth-dapp-services instance
@geleeroyale we still have issue
{
"name": "giveth-impact-graph",
"hostname": "5c5829428a54",
"pid": 69,
"level": 50,
"msg": "getLeastIndexedBlockTimeStamp >> error Error: connect ECONNREFUSED 68.183.74.154:443\n at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1157:16) {\n errno: -111,\n code: 'ECONNREFUSED',\n syscall: 'connect',\n address: '68.183.74.154',\n port: 443,\n config: {\n url: 'https://aggregator.giveth.io/fetch-state/least-indexed-block-timestamp',\n method: 'get',\n headers: {\n Accept: 'application/json, text/plain, */*',\n 'User-Agent': 'axios/0.21.4'\n },\n params: {},\n transformRequest: [ [Function (anonymous)] ],\n transformResponse: [ [Function: transformResponse] ],\n timeout: 0,\n adapter: [Function: httpAdapter],\n xsrfCookieName: 'XSRF-TOKEN',\n xsrfHeaderName: 'X-XSRF-TOKEN',\n maxContentLength: -1,\n maxBodyLength: -1,\n validateStatus: [Function: validateStatus],\n transitional: {\n silentJSONParsing: true,\n forcedJSONParsing: true,\n clarifyTimeoutError: false\n },\n 'axios-retry': { retryCount: 3, lastRequestTime: 1704893726858 },\n data: undefined\n },\n request: <ref *1> Writable {\n _writableState: WritableState {\n objectMode: false,\n highWaterMark: 16384,\n finalCalled: false,\n needDrain: false,\n ending: false,\n ended: false,\n finished: false,\n destroyed: false,\n decodeStrings: true,\n defaultEncoding: 'utf8',\n length: 0,\n writing: false,\n corked: 0,\n sync: true,\n bufferProcessing: false,\n onwrite: [Function: bound onwrite],\n writecb: null,\n writelen: 0,\n afterWriteTickInfo: null,\n buffered: [],\n bufferedIndex: 0,\n allBuffers: true,\n allNoop: true,\n pendingcb: 0,\n constructed: true,\n prefinished: false,\n errorEmitted: false,\n emitClose: true,\n autoDestroy: true,\n errored: null,\n closed: false,\n closeEmitted: false,\n [Symbol(kOnFinished)]: []\n },\n _events: [Object: null prototype] {\n response: [Function: handleResponse],\n error: [Function: handleRequestError]\n },\n _eventsCount: 2,\n _maxListeners: undefined,\n _options: {\n maxRedirects: 21,\n maxBodyLength: 10485760,\n protocol: 'https:',\n path: '/fetch-state/least-indexed-block-timestamp',\n method: 'GET',\n headers: [Object],\n agent: undefined,\n agents: [Object],\n auth: undefined,\n hostname: 'aggregator.giveth.io',\n port: null,\n nativeProtocols: [Object],\n pathname: '/fetch-state/least-indexed-block-timestamp'\n },\n _ended: true,\n _ending: true,\n _redirectCount: 0,\n _redirects: [],\n _requestBodyLength: 0,\n _requestBodyBuffers: [],\n _onNativeResponse: [Function (anonymous)],\n _currentRequest: ClientRequest {\n _events: [Object: null prototype],\n _eventsCount: 7,\n _maxListeners: undefined,\n outputData: [],\n outputSize: 0,\n writable: true,\n destroyed: false,\n _last: true,\n chunkedEncoding: false,\n shouldKeepAlive: false,\n maxRequestsOnConnectionReached: false,\n _defaultKeepAlive: true,\n useChunkedEncodingByDefault: false,\n sendDate: false,\n _removedConnection: false,\n _removedContLen: false,\n _removedTE: false,\n _contentLength: 0,\n _hasBody: true,\n _trailer: '',\n finished: true,\n _headerSent: true,\n _closed: false,\n socket: [TLSSocket],\n _header: 'GET /fetch-state/least-indexed-block-timestamp HTTP/1.1\\r\\n' +\n 'Accept: application/json, text/plain, */*\\r\\n' +\n 'User-Agent: axios/0.21.4\\r\\n' +\n 'Host: aggregator.giveth.io\\r\\n' +\n 'Connection: close\\r\\n' +\n '\\r\\n',\n _keepAliveTimeout: 0,\n _onPendingData: [Function: nop],\n agent: [Agent],\n socketPath: undefined,\n method: 'GET',\n maxHeaderSize: undefined,\n insecureHTTPParser: undefined,\n path: '/fetch-state/least-indexed-block-timestamp',\n _ended: false,\n res: null,\n aborted: false,\n timeoutCb: null,\n upgradeOrConnect: false,\n parser: null,\n maxHeadersCount: null,\n reusedSocket: false,\n host: 'aggregator.giveth.io',\n protocol: 'https:',\n _redirectable: [Circular *1],\n [Symbol(kCapture)]: false,\n [Symbol(kNeedDrain)]: false,\n [Symbol(corked)]: 0,\n [Symbol(kOutHeaders)]: [Object: null prototype]\n },\n _currentUrl: 'https://aggregator.giveth.io/fetch-state/least-indexed-block-timestamp',\n [Symbol(kCapture)]: false\n },\n response: undefined,\n isAxiosError: true,\n toJSON: [Function: toJSON]\n}",
"time": "2024-01-10T13:35:26.862Z",
"v": 0
}Can you investigate?
@aminlatifi I noticed that upon testing from Impact-Graph Production. The requests to the balance aggregator will be rate limited after 120 requests
@aminlatifi UPDATE: The requests were caught as a lot off access logs that were considered as a false positive by our DDOS protection service (Fail2ban)
Ok I've added the impact-graph as a whitelisted IP so that it can now do the requests without limitations (It won't be identified as a false positive anymore)
@aminlatifi Please confirm if it's working now or not. Awaiting your feedback
@geleeroyale @Rolazo FYI
Ok I've added the impact-graph as a whitelisted IP so that it can now do the requests without limitations (It won't be identified as a false positive anymore)
@aminlatifi Please confirm if it's working now or not. Awaiting your feedback
@geleeroyale @Rolazo FYI
It's working well. Thanks.