Giters

apple / foundationdb

FoundationDB - the open source, distributed, transactional key-value store

Home Page:https://apple.github.io/foundationdb/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

rare/ReadSkewReadWrite.toml hits QuietDatabaseConsistencyCheckStartFail due to StorageServersRecruiting

sfc-gh-jshim opened this issue · comments

commented

Code location and commit hash:

foundationdb/fdbserver/QuietDatabase.actor.cpp

Line 805 in 3f2404c

ASSERT(!ddGotStuck || !g_network->isSimulated());

Ensemble ID: 20230117-050643-nightly_correctness_main_x86_64-23954e7360d5fe3e

Seems like DD is still running past test timeout.
Please consider bumping up the timeout or modify the test to run shorter even in corner cases.

Trace snippet:

  <QuietDatabaseConsistencyCheckStartFail Severity="40" ErrorKind="Unset" Time="4456.760772" DateTime="2023-01-17T06:11:04Z" Type="QuietDatabaseConsistencyCheckStartFail" Machine="3.4.3.3:1" ID="0000000000000000" Reasons="StorageServersRecruiting" FailedAfter="4000.11" Timeout="4000" ThreadID="11847168932737437394" Backtrace="addr2line -e fdbserver.debug -p -C -f -i 0x4511601 0x295c1d8 0x2958166 0x1b6ea78 0x1b6e99d 0x1b6ea78 0x1b6e99d 0x1b6ea78 0x1b6e99d 0x1b6ea78 0x1b6e99d 0x1babfc8 0x1babe33 0x2942868 0x2941ba3 0x1b6ea78 0x293b14d 0x2940d24 0x294012b 0x2933ad4 0x293314c 0x1e17cc1 0x1e1773a 0x4330f88 0x43304c3 0x1aaf858 0x43f4745 0x2f048e7 0x7fa374061555" LogGroup="default" Roles="TS"/>
  <InternalError Severity="40" ErrorKind="BugDetected" Time="4456.760772" DateTime="2023-01-17T06:11:04Z" Type="InternalError" Machine="3.4.3.3:1" ID="0000000000000000" Error="internal_error" ErrorDescription="An internal error occurred" ErrorCode="4100" FailedAssertion="!ddGotStuck || !g_network-&gt;isSimulated()" File="/home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/fdbserver/QuietDatabase.actor.cpp" Line="805" ThreadID="11847168932737437394" Backtrace="addr2line -e fdbserver.debug -p -C -f -i 0x4511601 0x445881a 0x295c80f 0x2958166 0x1b6ea78 0x1b6e99d 0x1b6ea78 0x1b6e99d 0x1b6ea78 0x1b6e99d 0x1b6ea78 0x1b6e99d 0x1babfc8 0x1babe33 0x2942868 0x2941ba3 0x1b6ea78 0x293b14d 0x2940d24 0x294012b 0x2933ad4 0x293314c 0x1e17cc1 0x1e1773a 0x4330f88 0x43304c3 0x1aaf858 0x43f4745 0x2f048e7 0x7fa374061555" LogGroup="default" Roles="TS"/>
  <SystemError Severity="40" ErrorKind="Unset" Time="4456.760772" DateTime="2023-01-17T06:11:04Z" Type="SystemError" Machine="3.4.3.3:1" ID="0000000000000000" Error="internal_error" ErrorDescription="An internal error occurred" ErrorCode="4100" ThreadID="11847168932737437394" Backtrace="addr2line -e fdbserver.debug -p -C -f -i 0x4511601 0x4458e4f 0x445882d 0x295c80f 0x2958166 0x1b6ea78 0x1b6e99d 0x1b6ea78 0x1b6e99d 0x1b6ea78 0x1b6e99d 0x1b6ea78 0x1b6e99d 0x1babfc8 0x1babe33 0x2942868 0x2941ba3 0x1b6ea78 0x293b14d 0x2940d24 0x294012b 0x2933ad4 0x293314c 0x1e17cc1 0x1e1773a 0x4330f88 0x43304c3 0x1aaf858 0x43f4745 0x2f048e7 0x7fa374061555" LogGroup="default" Roles="TS"/>
  <Crash Severity="40" ErrorKind="BugDetected" Time="4456.760772" DateTime="2023-01-17T06:11:04Z" Type="Crash" Machine="3.4.3.3:1" ID="0000000000000000" Signal="6" Name="Aborted" Trace="addr2line -e fdbserver.debug -p -C -f -i 0x445882d 0x295c80f 0x2958166 0x1b6ea78 0x1b6e99d 0x1b6ea78 0x1b6e99d 0x1b6ea78 0x1b6e99d 0x1b6ea78 0x1b6e99d 0x1babfc8 0x1babe33 0x2942868 0x2941ba3 0x1b6ea78 0x293b14d 0x2940d24 0x294012b 0x2933ad4 0x293314c 0x1e17cc1 0x1e1773a 0x4330f88 0x43304c3 0x1aaf858 0x43f4745 0x2f048e7 0x7fa374061555" ThreadID="11847168932737437394" Backtrace="addr2line -e fdbserver.debug -p -C -f -i 0x4511601 0x44e71e2 0x7fa37441c630 0x445882d 0x295c80f 0x2958166 0x1b6ea78 0x1b6e99d 0x1b6ea78 0x1b6e99d 0x1b6ea78 0x1b6e99d 0x1b6ea78 0x1b6e99d 0x1babfc8 0x1babe33 0x2942868 0x2941ba3 0x1b6ea78 0x293b14d 0x2940d24 0x294012b 0x2933ad4 0x293314c 0x1e17cc1 0x1e1773a 0x4330f88 0x43304c3 0x1aaf858 0x43f4745 0x2f048e7 0x7fa374061555" LogGroup="default" Roles="TS"/>
  <WarningLimitExceeded Severity="30" WarningCount="4512"/>
  <TestUnexpectedlyNotFinished Severity="40"/>
  <StdErrOutput Severity="40" Output="Assertion !ddGotStuck || !g_network-&gt;isSimulated() failed @ /home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/fdbserver/QuietDatabase.actor.cpp 805:"/>
  <StdErrOutput Severity="40" Output=" addr2line -e fdbserver.debug -p -C -f -i 0x2958166 0x1b6ea78 0x1b6e99d 0x1b6ea78 0x1b6e99d 0x1b6ea78 0x1b6e99d 0x1b6ea78 0x1b6e99d 0x1babfc8 0x1babe33 0x2942868 0x2941ba3 0x1b6ea78 0x293b14d 0x2940d24 0x294012b 0x2933ad4 0x293314c 0x1e17cc1 0x1e1773a 0x4330f88 0x43304c3 0x1aaf858 0x43f4745 0x2f048e7 0x7fa374061555"/>
  <StdErrOutput Severity="40" Output="SIGNAL: Aborted (6)"/>
  <StdErrOutput Severity="40" Output="Trace: addr2line -e fdbserver.debug -p -C -f -i 0x445882d 0x295c80f 0x2958166 0x1b6ea78 0x1b6e99d 0x1b6ea78 0x1b6e99d 0x1b6ea78 0x1b6e99d 0x1b6ea78 0x1b6e99d 0x1babfc8 0x1babe33 0x2942868 0x2941ba3 0x1b6ea78 0x293b14d 0x2940d24 0x294012b 0x2933ad4 0x293314c 0x1e17cc1 0x1e1773a 0x4330f88 0x43304c3 0x1aaf858 0x43f4745 0x2f048e7 0x7fa374061555"/>

Assigned to @sfc-gh-xwang as the test workload owner.