Debug build is broken (node_mksnapshot)

Question

Debug build is broken (node_mksnapshot)

targos opened this issue a year ago · comments

Diff to remove C++20 feature:

diff --git a/deps/v8/src/compiler/graph-visualizer.cc b/deps/v8/src/compiler/graph-visualizer.cc
index 0eb8b2cd42..c20557a70e 100644
--- a/deps/v8/src/compiler/graph-visualizer.cc
+++ b/deps/v8/src/compiler/graph-visualizer.cc
@@ -294,7 +294,6 @@ void JsonPrintAllSourceWithPositionsWasm(
   os << "\"inlinings\": {";
   for (size_t i = 0; i < positions.size(); ++i) {
     if (i != 0) os << ", ";
-    DCHECK(source_map.contains(positions[i].inlinee_func_index));
     size_t source_id = source_map.find(positions[i].inlinee_func_index)->second;
     SourcePosition inlining_pos = positions[i].caller_pos;
     os << '"' << i << "\": {\"inliningId\": " << i

./configure --ninja --debug
make

Error:

[1/5] ACTION node: node_mksnapshot_9b7a2d2290b02e76d66661df74749f56
FAILED: gen/node_snapshot.cc
cd ../../; export BUILT_FRAMEWORKS_DIR=/Users/mzasso/git/nodejs/canary/out/Debug; export BUILT_PRODUCTS_DIR=/Users/mzasso/git/nodejs/canary/out/Debug; export CONFIGURATION=Debug; export EXECUTABLE_NAME=node; export EXECUTABLE_PATH=node; export FULL_PRODUCT_NAME=node; export PRODUCT_NAME=node; export PRODUCT_TYPE=com.apple.product-type.tool; export SDKROOT=/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk; export SRCROOT=/Users/mzasso/git/nodejs/canary/out/Debug/../../; export SOURCE_ROOT="${SRCROOT}"; export TARGET_BUILD_DIR=/Users/mzasso/git/nodejs/canary/out/Debug; export TEMP_DIR="${TMPDIR}"; export XCODE_VERSION_ACTUAL=1431;/Users/mzasso/git/nodejs/canary/out/Debug/node_mksnapshot /Users/mzasso/git/nodejs/canary/out/Debug/gen/node_snapshot.cc


#
# Fatal error in ../../deps/v8/src/snapshot/serializer.cc, line 1272
# Debug check failed: !read_only_space->writable().
#
#
#
#FailureMessage Object: 0x16daf43b8
 1: 0x102a095e0 node::DumpBacktrace(__sFILE*) [/Users/mzasso/git/nodejs/canary/out/Debug/node_mksnapshot]
 2: 0x102c2f1fc node::NodePlatform::GetStackTracePrinter()::$_3::operator()() const [/Users/mzasso/git/nodejs/canary/out/Debug/node_mksnapshot]
 3: 0x102c2f1bc node::NodePlatform::GetStackTracePrinter()::$_3::__invoke() [/Users/mzasso/git/nodejs/canary/out/Debug/node_mksnapshot]
 4: 0x1042d9a9c V8_Fatal(char const*, int, char const*, ...) [/Users/mzasso/git/nodejs/canary/out/Debug/node_mksnapshot]
 5: 0x1042d9638 std::__1::enable_if<!std::is_function<std::__1::remove_pointer<char>::type>::value && !std::is_enum<char>::value && has_output_operator<char, v8::base::CheckMessageStream>::value, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>::type v8::base::PrintCheckOperand<char>(char) [/Users/mzasso/git/nodejs/canary/out/Debug/node_mksnapshot]
 6: 0x103e8e72c v8::internal::Serializer::SerializeReadOnlyObjectReference(v8::internal::HeapObject, v8::internal::SnapshotByteSink*) [/Users/mzasso/git/nodejs/canary/out/Debug/node_mksnapshot]
 7: 0x103e61a24 v8::internal::CodeSerializer::SerializeObjectImpl(v8::internal::Handle<v8::internal::HeapObject>, v8::internal::SerializerDeserializer::SlotType) [/Users/mzasso/git/nodejs/canary/out/Debug/node_mksnapshot]
 8: 0x103e8d730 v8::internal::Serializer::ObjectSerializer::VisitPointers(v8::internal::HeapObject, v8::internal::FullMaybeObjectSlot, v8::internal::FullMaybeObjectSlot) [/Users/mzasso/git/nodejs/canary/out/Debug/node_mksnapshot]
 9: 0x103c1d1a4 v8::internal::HeapObject::IterateBody(v8::internal::Map, int, v8::internal::ObjectVisitor*) [/Users/mzasso/git/nodejs/canary/out/Debug/node_mksnapshot]
10: 0x103e8ce04 v8::internal::Serializer::ObjectSerializer::SerializeContent(v8::internal::Map, int) [/Users/mzasso/git/nodejs/canary/out/Debug/node_mksnapshot]
11: 0x103e8b7a4 v8::internal::Serializer::ObjectSerializer::SerializeObject() [/Users/mzasso/git/nodejs/canary/out/Debug/node_mksnapshot]
12: 0x103e8cc14 v8::internal::Serializer::ObjectSerializer::Serialize(v8::internal::SerializerDeserializer::SlotType) [/Users/mzasso/git/nodejs/canary/out/Debug/node_mksnapshot]
13: 0x103e62024 v8::internal::CodeSerializer::SerializeObjectImpl(v8::internal::Handle<v8::internal::HeapObject>, v8::internal::SerializerDeserializer::SlotType) [/Users/mzasso/git/nodejs/canary/out/Debug/node_mksnapshot]
14: 0x103e8d730 v8::internal::Serializer::ObjectSerializer::VisitPointers(v8::internal::HeapObject, v8::internal::FullMaybeObjectSlot, v8::internal::FullMaybeObjectSlot) [/Users/mzasso/git/nodejs/canary/out/Debug/node_mksnapshot]
15: 0x103c1d1a4 v8::internal::HeapObject::IterateBody(v8::internal::Map, int, v8::internal::ObjectVisitor*) [/Users/mzasso/git/nodejs/canary/out/Debug/node_mksnapshot]
16: 0x103e8ce04 v8::internal::Serializer::ObjectSerializer::SerializeContent(v8::internal::Map, int) [/Users/mzasso/git/nodejs/canary/out/Debug/node_mksnapshot]
17: 0x103e8b7a4 v8::internal::Serializer::ObjectSerializer::SerializeObject() [/Users/mzasso/git/nodejs/canary/out/Debug/node_mksnapshot]
18: 0x103e8cc14 v8::internal::Serializer::ObjectSerializer::Serialize(v8::internal::SerializerDeserializer::SlotType) [/Users/mzasso/git/nodejs/canary/out/Debug/node_mksnapshot]
19: 0x103e62024 v8::internal::CodeSerializer::SerializeObjectImpl(v8::internal::Handle<v8::internal::HeapObject>, v8::internal::SerializerDeserializer::SlotType) [/Users/mzasso/git/nodejs/canary/out/Debug/node_mksnapshot]
20: 0x103e8d730 v8::internal::Serializer::ObjectSerializer::VisitPointers(v8::internal::HeapObject, v8::internal::FullMaybeObjectSlot, v8::internal::FullMaybeObjectSlot) [/Users/mzasso/git/nodejs/canary/out/Debug/node_mksnapshot]
21: 0x103c43158 void v8::internal::CallIterateBody::apply<v8::internal::BytecodeArray::BodyDescriptor, v8::internal::ObjectVisitor>(v8::internal::Map, v8::internal::HeapObject, int, v8::internal::ObjectVisitor*) [/Users/mzasso/git/nodejs/canary/out/Debug/node_mksnapshot]
22: 0x103c1d1a4 v8::internal::HeapObject::IterateBody(v8::internal::Map, int, v8::internal::ObjectVisitor*) [/Users/mzasso/git/nodejs/canary/out/Debug/node_mksnapshot]
23: 0x103e8ce04 v8::internal::Serializer::ObjectSerializer::SerializeContent(v8::internal::Map, int) [/Users/mzasso/git/nodejs/canary/out/Debug/node_mksnapshot]
24: 0x103e8b7a4 v8::internal::Serializer::ObjectSerializer::SerializeObject() [/Users/mzasso/git/nodejs/canary/out/Debug/node_mksnapshot]
25: 0x103e8cc14 v8::internal::Serializer::ObjectSerializer::Serialize(v8::internal::SerializerDeserializer::SlotType) [/Users/mzasso/git/nodejs/canary/out/Debug/node_mksnapshot]
26: 0x103e62024 v8::internal::CodeSerializer::SerializeObjectImpl(v8::internal::Handle<v8::internal::HeapObject>, v8::internal::SerializerDeserializer::SlotType) [/Users/mzasso/git/nodejs/canary/out/Debug/node_mksnapshot]
27: 0x103e8d730 v8::internal::Serializer::ObjectSerializer::VisitPointers(v8::internal::HeapObject, v8::internal::FullMaybeObjectSlot, v8::internal::FullMaybeObjectSlot) [/Users/mzasso/git/nodejs/canary/out/Debug/node_mksnapshot]
28: 0x103c43664 void v8::internal::CallIterateBody::apply<v8::internal::SharedFunctionInfo::BodyDescriptor, v8::internal::ObjectVisitor>(v8::internal::Map, v8::internal::HeapObject, int, v8::internal::ObjectVisitor*) [/Users/mzasso/git/nodejs/canary/out/Debug/node_mksnapshot]
29: 0x103c1d1a4 v8::internal::HeapObject::IterateBody(v8::internal::Map, int, v8::internal::ObjectVisitor*) [/Users/mzasso/git/nodejs/canary/out/Debug/node_mksnapshot]
30: 0x103e8ce04 v8::internal::Serializer::ObjectSerializer::SerializeContent(v8::internal::Map, int) [/Users/mzasso/git/nodejs/canary/out/Debug/node_mksnapshot]
31: 0x103e8b7a4 v8::internal::Serializer::ObjectSerializer::SerializeObject() [/Users/mzasso/git/nodejs/canary/out/Debug/node_mksnapshot]
32: 0x103e8cc14 v8::internal::Serializer::ObjectSerializer::Serialize(v8::internal::SerializerDeserializer::SlotType) [/Users/mzasso/git/nodejs/canary/out/Debug/node_mksnapshot]
33: 0x103e62024 v8::internal::CodeSerializer::SerializeObjectImpl(v8::internal::Handle<v8::internal::HeapObject>, v8::internal::SerializerDeserializer::SlotType) [/Users/mzasso/git/nodejs/canary/out/Debug/node_mksnapshot]
34: 0x103e887bc v8::internal::Serializer::VisitRootPointers(v8::internal::Root, char const*, v8::internal::FullObjectSlot, v8::internal::FullObjectSlot) [/Users/mzasso/git/nodejs/canary/out/Debug/node_mksnapshot]
35: 0x103e6172c v8::internal::CodeSerializer::SerializeSharedFunctionInfo(v8::internal::Handle<v8::internal::SharedFunctionInfo>) [/Users/mzasso/git/nodejs/canary/out/Debug/node_mksnapshot]
36: 0x103e61348 v8::internal::CodeSerializer::Serialize(v8::internal::Isolate*, v8::internal::Handle<v8::internal::SharedFunctionInfo>) [/Users/mzasso/git/nodejs/canary/out/Debug/node_mksnapshot]
37: 0x1033618b0 v8::ScriptCompiler::CreateCodeCacheForFunction(v8::Local<v8::Function>) [/Users/mzasso/git/nodejs/canary/out/Debug/node_mksnapshot]
38: 0x102aee1f0 node::builtins::BuiltinLoader::LookupAndCompileInternal(v8::Local<v8::Context>, char const*, std::__1::vector<v8::Local<v8::String>, std::__1::allocator<v8::Local<v8::String>>>*, node::builtins::BuiltinLoader::Result*) [/Users/mzasso/git/nodejs/canary/out/Debug/node_mksnapshot]
39: 0x102aee9f8 node::builtins::BuiltinLoader::LookupAndCompile(v8::Local<v8::Context>, char const*, node::Realm*) [/Users/mzasso/git/nodejs/canary/out/Debug/node_mksnapshot]
40: 0x102aef050 node::builtins::BuiltinLoader::CompileAndCall(v8::Local<v8::Context>, char const*, int, v8::Local<v8::Value>*, node::Realm*) [/Users/mzasso/git/nodejs/canary/out/Debug/node_mksnapshot]
41: 0x102979b14 node::InitializePrimordials(v8::Local<v8::Context>) [/Users/mzasso/git/nodejs/canary/out/Debug/node_mksnapshot]
42: 0x102979734 node::GetPerContextExports(v8::Local<v8::Context>) [/Users/mzasso/git/nodejs/canary/out/Debug/node_mksnapshot]
43: 0x102979974 node::InitializePrimordials(v8::Local<v8::Context>) [/Users/mzasso/git/nodejs/canary/out/Debug/node_mksnapshot]
44: 0x10297a1b8 node::InitializeMainContextForSnapshot(v8::Local<v8::Context>) [/Users/mzasso/git/nodejs/canary/out/Debug/node_mksnapshot]
45: 0x102979ccc node::InitializeContext(v8::Local<v8::Context>) [/Users/mzasso/git/nodejs/canary/out/Debug/node_mksnapshot]
46: 0x102979c68 node::NewContext(v8::Isolate*, v8::Local<v8::ObjectTemplate>) [/Users/mzasso/git/nodejs/canary/out/Debug/node_mksnapshot]
47: 0x10297194c node::CommonEnvironmentSetup::CommonEnvironmentSetup(node::MultiIsolatePlatform*, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>>*, node::EmbedderSnapshotData const*, unsigned int, std::__1::function<node::Environment* (node::CommonEnvironmentSetup const*)>) [/Users/mzasso/git/nodejs/canary/out/Debug/node_mksnapshot]
48: 0x102971cf0 node::CommonEnvironmentSetup::CommonEnvironmentSetup(node::MultiIsolatePlatform*, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>>*, node::EmbedderSnapshotData const*, unsigned int, std::__1::function<node::Environment* (node::CommonEnvironmentSetup const*)>) [/Users/mzasso/git/nodejs/canary/out/Debug/node_mksnapshot]
49: 0x102971f74 node::CommonEnvironmentSetup::CreateForSnapshotting(node::MultiIsolatePlatform*, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>>*, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&) [/Users/mzasso/git/nodejs/canary/out/Debug/node_mksnapshot]
50: 0x102c7bb54 node::SnapshotBuilder::Generate(node::SnapshotData*, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, std::__1::optional<std::__1::basic_string_view<char, std::__1::char_traits<char>>>) [/Users/mzasso/git/nodejs/canary/out/Debug/node_mksnapshot]
51: 0x102c7d294 node::SnapshotBuilder::Generate(std::__1::basic_ostream<char, std::__1::char_traits<char>>&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, std::__1::optional<std::__1::basic_string_view<char, std::__1::char_traits<char>>>) [/Users/mzasso/git/nodejs/canary/out/Debug/node_mksnapshot]
52: 0x10291b4e0 BuildSnapshot(int, char**) [/Users/mzasso/git/nodejs/canary/out/Debug/node_mksnapshot]
53: 0x10291b238 main [/Users/mzasso/git/nodejs/canary/out/Debug/node_mksnapshot]
54: 0x194bf3f28 start [/usr/lib/dyld]
/bin/sh: line 1: 17853 Trace/BPT trap: 5       /Users/mzasso/git/nodejs/canary/out/Debug/node_mksnapshot /Users/mzasso/git/nodejs/canary/out/Debug/gen/node_snapshot.cc
ninja: build stopped: subcommand failed.
make: *** [node_g] Error 1

/cc @nodejs/v8 @joyeecheung does it ring a bell?

Michaël Zasso · Answer 1 · Wed Aug 09 2023 23:47:39 GMT+0800 (China Standard Time)

Note that I'm trying to create a debug build because the release build, while it can be created, is also deeply broken:

$ ./node "-p" "process.versions.openssl != undefined"
[1]    22522 segmentation fault  ./node "-p" "process.versions.openssl != undefined"

Michaël Zasso · Answer 2 · Wed Aug 09 2023 23:54:07 GMT+0800 (China Standard Time)

By looking at the GH actions results, it seems the problems have started between Aug 3 and Aug 4.

Michaël Zasso · Answer 3 · Wed Aug 09 2023 23:58:17 GMT+0800 (China Standard Time)

On the release build, I get the following stack trace. Note that I reverted 6a4952f before building locally.

$ lldb ./node
(lldb) target create "./node"
Current executable set to '/Users/mzasso/git/nodejs/canary/node' (arm64).
(lldb) run -p "process.versions.openssl != undefined"
Process 22575 launched: '/Users/mzasso/git/nodejs/canary/node' (arm64)
Process 22575 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x157c)
    frame #0: 0x00000001007602d8 node`v8::internal::ScopeInfo::scope_type() const [inlined] v8::internal::TaggedField<v8::internal::Smi, 0, v8::internal::V8HeapCompressionScheme>::load(host=HeapObject @ x8, offset=8) at tagged-field-inl.h:64:20 [opt]
   61  	template <typename T, int kFieldOffset, typename CompressionScheme>
   62  	T TaggedField<T, kFieldOffset, CompressionScheme>::load(HeapObject host,
   63  	                                                        int offset) {
-> 64  	  Tagged_t value = *location(host, offset);
   65  	  DCHECK_NE(kFieldOffset + offset, HeapObject::kMapOffset);
   66  	  return T(tagged_to_full(host.ptr(), value));
   67  	}
Target 0: (node) stopped.
warning: node was compiled with optimization - stepping may behave oddly; variables may not be available.
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x157c)
  * frame #0: 0x00000001007602d8 node`v8::internal::ScopeInfo::scope_type() const [inlined] v8::internal::TaggedField<v8::internal::Smi, 0, v8::internal::V8HeapCompressionScheme>::load(host=HeapObject @ x8, offset=8) at tagged-field-inl.h:64:20 [opt]
    frame #1: 0x00000001007602d8 node`v8::internal::ScopeInfo::scope_type() const [inlined] v8::internal::TorqueGeneratedScopeInfo<v8::internal::ScopeInfo, v8::internal::HeapObject>::flags(this=0x000000016fdfd778) const at scope-info-tq-inl.inc:1547:11 [opt]
    frame #2: 0x00000001007602d4 node`v8::internal::ScopeInfo::scope_type() const [inlined] v8::internal::ScopeInfo::Flags(this=0x000000016fdfd778) const at scope-info-inl.h:32:39 [opt]
    frame #3: 0x00000001007602d4 node`v8::internal::ScopeInfo::scope_type(this=0x000000016fdfd778) const at scope-info.cc:688:32 [opt]
    frame #4: 0x00000001002ffb0c node`v8::internal::Scope* v8::internal::Scope::DeserializeScopeChain<v8::internal::Isolate>(isolate=0x0000000108008000, zone=<unavailable>, scope_info=ScopeInfo @ 0x000000016fdfd778, script_scope=0x000000010782c430, ast_value_factory=0x0000000107839c00, deserialization_mode=kIncludingVariables) at scopes.cc:425:21 [opt]
    frame #5: 0x0000000100790d98 node`void v8::internal::Parser::DeserializeScopeChain<v8::internal::Isolate>(this=0x000000016fdfd920, isolate=0x0000000108008000, info=0x000000016fdfdf40, maybe_outer_scope_info=MaybeHandle<v8::internal::ScopeInfo> @ x23, mode=kIncludingVariables) at parser.cc:507:23 [opt]
    frame #6: 0x00000001007969d0 node`v8::internal::Parser::ParseFunction(this=0x000000016fdfd920, isolate=0x0000000108008000, info=0x000000016fdfdf40, shared_info=Handle<v8::internal::SharedFunctionInfo> @ x22) at parser.cc:881:3 [opt]
    frame #7: 0x00000001007b8e90 node`v8::internal::parsing::ParseFunction(info=0x000000016fdfdf40, shared_info=Handle<v8::internal::SharedFunctionInfo> @ x22, isolate=0x0000000108008000, mode=kYes) at parsing.cc:88:10 [opt]
    frame #8: 0x0000000100375bc0 node`v8::internal::Compiler::Compile(isolate=0x0000000108008000, shared_info=Handle<v8::internal::SharedFunctionInfo> @ x21, flag=KEEP_EXCEPTION, is_compiled_scope=0x000000016fdfe258, create_source_positions_flag=<unavailable>) at compiler.cc:2575:8 [opt]
    frame #9: 0x00000001003764ac node`v8::internal::Compiler::Compile(isolate=0x0000000108008000, function=Handle<v8::internal::JSFunction> @ x20, flag=KEEP_EXCEPTION, is_compiled_scope=0x000000016fdfe258) at compiler.cc:2639:8 [opt]
    frame #10: 0x000000010084d7c8 node`v8::internal::Runtime_CompileLazy(int, unsigned long*, v8::internal::Isolate*) [inlined] v8::internal::__RT_impl_Runtime_CompileLazy(args=<unavailable>, isolate=0x0000000108008000)0>, v8::internal::Isolate*) at runtime-compiler.cc:64:8 [opt]
    frame #11: 0x000000010084d760 node`v8::internal::Runtime_CompileLazy(args_length=<unavailable>, args_object=0x000000016fdfe2e0, isolate=0x0000000108008000) at runtime-compiler.cc:45:1 [opt]

Michaël Zasso · Answer 4 · Wed Aug 09 2023 23:59:49 GMT+0800 (China Standard Time)

/cc @LeszekSwirski

Leszek Swirski · Answer 5 · Thu Aug 10 2023 21:23:16 GMT+0800 (China Standard Time)

Do you have a V8 commit range between those dates? My suspicion would be crbug.com/v8/14247

Michaël Zasso · Answer 6 · Thu Aug 10 2023 22:36:40 GMT+0800 (China Standard Time)

I can retry locally with V8 11.8.14, which includes https://chromium-review.googlesource.com/c/v8/v8/+/4762666

I'm just not sure whether I should keep the default value of v8_enable_extensible_ro_snapshot to true or set it to false for Node.js

Leszek Swirski · Answer 7 · Fri Aug 11 2023 00:00:04 GMT+0800 (China Standard Time)

You should set it to false.

Michaël Zasso · Answer 8 · Fri Aug 11 2023 01:08:47 GMT+0800 (China Standard Time)

Thanks, the build is fixed!

There's only one test that fails: test/parallel/test-code-cache.js.
I'll need your help @joyeecheung.

Michaël Zasso · Answer 9 · Fri Aug 11 2023 01:12:08 GMT+0800 (China Standard Time)

I pushed the current state to canary-base (main repo) and canary (this repo)

Joyee Cheung · Answer 10 · Fri Aug 11 2023 01:17:15 GMT+0800 (China Standard Time)

Does nodejs/node#49099 help with the crash (when v8_enable_extensible_ro_snapshot is set to true)?

Michaël Zasso · Answer 11 · Fri Aug 11 2023 01:35:27 GMT+0800 (China Standard Time)

It seems to help a bit with the release build (which doesn't crash anymore), but:

Some snapshot tests fail (test-snapshot-argv1 and test-snapshot-net)

test failures

=== release test-snapshot-argv1 ===                                           
Path: parallel/test-snapshot-argv1
undefined:1


SyntaxError: Unexpected end of JSON input
    at JSON.parse (<anonymous>)
    at Object.<anonymous> (/Users/mzasso/git/nodejs/canary/test/parallel/test-snapshot-argv1.js:51:23)
    at Module._compile (node:internal/modules/cjs/loader:1241:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1295:10)
    at Module.load (node:internal/modules/cjs/loader:1091:32)
    at Module._load (node:internal/modules/cjs/loader:938:12)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:83:12)
    at node:internal/main/run_main_module:23:47

Node.js v21.0.0-pre
Command: out/Release/node /Users/mzasso/git/nodejs/canary/test/parallel/test-snapshot-argv1.js


=== release test-snapshot-net ===                                             
Path: parallel/test-snapshot-net
[stdout]:


[stderr]:
node:assert:125
  throw new AssertionError(obj);
  ^

AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:

null !== 0

    at Object.<anonymous> (/Users/mzasso/git/nodejs/canary/test/parallel/test-snapshot-net.js:50:10)
    at Module._compile (node:internal/modules/cjs/loader:1241:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1295:10)
    at Module.load (node:internal/modules/cjs/loader:1091:32)
    at Module._load (node:internal/modules/cjs/loader:938:12)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:83:12)
    at node:internal/main/run_main_module:23:47 {
  generatedMessage: true,
  code: 'ERR_ASSERTION',
  actual: null,
  expected: 0,
  operator: 'strictEqual'
}

Node.js v21.0.0-pre
Command: out/Release/node /Users/mzasso/git/nodejs/canary/test/parallel/test-snapshot-net.js

the DCHECK is still failing node_mksnapshot for the debug build:

#
# Fatal error in ../../deps/v8/src/snapshot/serializer.cc, line 1272
# Debug check failed: !read_only_space->writable().
#
#
#
#FailureMessage Object: 0x16d827fe8

I applied nodejs/node#49099 and commented the #ifdef to avoid recompiling V8 from scratch

Michaël Zasso · Answer 12 · Fri Aug 11 2023 14:19:35 GMT+0800 (China Standard Time)

Actually, test-snapshot-argv1 is flaky. When it fails, the string passed to JSON.parse is empty.

Joyee Cheung · Answer 13 · Fri Aug 11 2023 22:49:34 GMT+0800 (China Standard Time)

Some notes of what I found so far:

About the !read_only_space->writable() DCHECK in node_mksnapshot: it was crashing because whenever we compile a builtin, we try to save the code cache of it, even when we are building a snapshot (and with extensible RO snapshot the assumption is that embedders should not serialize code cache for a snapshot-building isolate, which has an unfinalized RO-space). We actually don't need the built-in loader to serialize the code cache when creating the snapshot (it's going to be serialized separate), so that path should've been removed anyway, I opened nodejs/node#49108 for this.

And after applying the patch above, the node binary could be built with v8_enable_extensible_ro_snapshot=true but when launched it crashed with another encoded.page_index < static_cast<int>(ro_space()->pages().size()) DCHECK:

See stack trace

#
# Fatal error in ../../deps/v8/src/snapshot/read-only-deserializer.cc, line 92
# Debug check failed: encoded.page_index < static_cast<int>(ro_space()->pages().size()) (6 vs. 6).
#
#
#
#FailureMessage Object: 0x16d021fe8
 1: 0x102ef4d20 node::DumpBacktrace(__sFILE*) [/Users/joyee/projects/node-v8/out/Debug/node]
 2: 0x10310bee4 node::NodePlatform::GetStackTracePrinter()::$_3::operator()() const [/Users/joyee/projects/node-v8/out/Debug/node]
 3: 0x10310bea4 node::NodePlatform::GetStackTracePrinter()::$_3::__invoke() [/Users/joyee/projects/node-v8/out/Debug/node]
 4: 0x104e679a8 V8_Fatal(char const*, int, char const*, ...) [/Users/joyee/projects/node-v8/out/Debug/node]
 5: 0x104e67544 std::__1::enable_if<!std::is_function<std::__1::remove_pointer<char>::type>::value && !std::is_enum<char>::value && has_output_operator<char, v8::base::CheckMessageStream>::value, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>::type v8::base::PrintCheckOperand<char>(char) [/Users/joyee/projects/node-v8/out/Debug/node]
 6: 0x103ebe948 v8::internal::ReadOnlyHeapImageDeserializer::Decode(v8::internal::ro::EncodedTagged) const [/Users/joyee/projects/node-v8/out/Debug/node]
 7: 0x103ebe328 v8::internal::ReadOnlyHeapImageDeserializer::DeserializeReadOnlySegment() [/Users/joyee/projects/node-v8/out/Debug/node]
 8: 0x103ebe0d4 v8::internal::ReadOnlyHeapImageDeserializer::DeserializeImpl() [/Users/joyee/projects/node-v8/out/Debug/node]
 9: 0x103ebdad8 v8::internal::ReadOnlyDeserializer::DeserializeIntoIsolate() [/Users/joyee/projects/node-v8/out/Debug/node]
10: 0x103889a38 v8::internal::ReadOnlyHeap::DeserializeIntoIsolate(v8::internal::Isolate*, v8::internal::SnapshotData*, bool) [/Users/joyee/projects/node-v8/out/Debug/node]
11: 0x1038894bc v8::internal::ReadOnlyHeap::SetUp(v8::internal::Isolate*, v8::internal::SnapshotData*, bool) [/Users/joyee/projects/node-v8/out/Debug/node]
12: 0x1036c5e8c v8::internal::Isolate::Init(v8::internal::SnapshotData*, v8::internal::SnapshotData*, v8::internal::SnapshotData*, bool) [/Users/joyee/projects/node-v8/out/Debug/node]
13: 0x1036c6fa4 v8::internal::Isolate::InitWithSnapshot(v8::internal::SnapshotData*, v8::internal::SnapshotData*, v8::internal::SnapshotData*, bool) [/Users/joyee/projects/node-v8/out/Debug/node]
14: 0x103ed38f8 v8::internal::Snapshot::Initialize(v8::internal::Isolate*) [/Users/joyee/projects/node-v8/out/Debug/node]
15: 0x1033e1928 v8::Isolate::Initialize(v8::Isolate*, v8::Isolate::CreateParams const&) [/Users/joyee/projects/node-v8/out/Debug/node]
16: 0x102e4fcd8 node::NewIsolate(v8::Isolate::CreateParams*, uv_loop_s*, node::MultiIsolatePlatform*, node::SnapshotData const*, node::IsolateSettings const&) [/Users/joyee/projects/node-v8/out/Debug/node]
17: 0x10309f740 node::NodeMainInstance::NodeMainInstance(node::SnapshotData const*, uv_loop_s*, node::MultiIsolatePlatform*, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&) [/Users/joyee/projects/node-v8/out/Debug/node]
18: 0x10309fa40 node::NodeMainInstance::NodeMainInstance(node::SnapshotData const*, uv_loop_s*, node::MultiIsolatePlatform*, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&) [/Users/joyee/projects/node-v8/out/Debug/node]
19: 0x102fa2994 node::StartInternal(int, char**) [/Users/joyee/projects/node-v8/out/Debug/node]
20: 0x102fa25e4 node::Start(int, char**) [/Users/joyee/projects/node-v8/out/Debug/node]
21: 0x104920c98 main [/Users/joyee/projects/node-v8/out/Debug/node]
22: 0x18ae37f28 start [/usr/lib/dyld]

This was because we were compiling the code cache in an unfinalized isolate in the snapshot builder. nodejs/node#49099 updated the code cache compilation routine to create the code cache from a deserialized isolate in the second pass. With that patch the node binary can be built and a handful of snapshot tests passed. However some of the snapshot test still fail with the encoded.page_index < static_cast<int>(ro_space()->pages().size()) DCHECK. I am not yet sure what's happening with those tests (we use almost the same code for the built-in snapshot and the customized snapshot, it seems with the two patches the built-in snapshot works but the customized one doesn't)

Actually, test-snapshot-argv1 is flaky. When it fails, the string passed to JSON.parse is empty.

It was actually crashing with a stack trace similar to the one folded above, just that the test doesn't log stderr when it fails (and instead go ahead parsing the stdout which is empty if the process crashes)

Joyee Cheung · Answer 14 · Fri Aug 11 2023 23:18:41 GMT+0800 (China Standard Time)

I have a repro at this branch (with v8_enable_extensible_ro_snapshot enabled, and the two patches mentioned above applied): https://github.com/joyeecheung/node/tree/ro-code-cache

This crashes when trying to deserialize the freshly built snapshot in order to build the code cache in BuildCodeCacheFromSnapshot():

./configure --ninja --debug
out/Debug/node --build-snapshot test/fixtures/snapshot/console.js # This deserializes in the same process

And this crashes when just trying to deserialize the snapshot (without the code cache):

./configure --ninja --debug
out/Debug/node --build-snapshot test/fixtures/snapshot/console.js
out/Debug/node --snapshot-blob snapshot.blob # This deserialize in another process

Curiously enough it doesn't always fail, for example this still works locally for me:

out/Debug/node --build-snapshot test/fixtures/snapshot/mutate-fs.js
out/Debug/node --snapshot-blob snapshot.blob test/fixtures/snapshot/check-mutate-fs.js

(this is basically what test/parallel/test-snapshot-cjs-main.js does, and similarly test/parallel/test-snapshot-api.js etc. doesn't fail for me either). So I wonder if this has anything to do with the serialization of read-only space itself? cc @schuay

Jakob Gruber · Answer 15 · Wed Aug 16 2023 17:58:44 GMT+0800 (China Standard Time)

I already see the crash from #252 (comment) at step:

./configure --ninja --debug
out/Debug/node --build-snapshot test/fixtures/snapshot/console.js

(Also, node takes forever to build locally. It seems to be stuck on obj/gen/torque-generated/src/builtins/wasm-to-js-tq-csa.cc...)

I suspect this is related to ro-promotion creating forward-refs (from one RO page to the next). Deserialization can't handle these yet since RO pages are created in order as we deserialize. Looking..

Jakob Gruber · Answer 16 · Wed Aug 16 2023 18:18:54 GMT+0800 (China Standard Time)

Yes, confirmed:

index 603010ed09..91f6ecfc81 100644
--- a/deps/v8/src/snapshot/read-only-serializer.cc
+++ b/deps/v8/src/snapshot/read-only-serializer.cc
@@ -231,6 +231,23 @@ class EncodeRelocationsVisitor final : public ObjectVisitor {
 
     // Encode:
     ro::EncodedTagged encoded = Encode(isolate_, o.GetHeapObject());
+
+    {
+      BasicMemoryChunk* host_chunk = BasicMemoryChunk::FromAddress(slot.address());
+      int host_page_index = 0;
+      for (ReadOnlyPage* page : isolate_->heap()->read_only_space()->pages()) {
+        if (host_chunk == page) break;
+        ++host_page_index;
+      }
+      BasicMemoryChunk* target_chunk = BasicMemoryChunk::FromAddress(o.GetHeapObject().address());
+      int target_page_index = 0;
+      for (ReadOnlyPage* page : isolate_->heap()->read_only_space()->pages()) {
+        if (target_chunk == page) break;
+        ++target_page_index;
+      }
+      CHECK_LE(target_page_index, host_page_index);
+    }
+
     memcpy(segment_->contents.get() + slot_offset, &encoded,
            ro::EncodedTagged::kSize);

...

# Check failed: target_page_index <= host_page_index (4 vs. 3).

Jakob Gruber · Answer 17 · Thu Aug 17 2023 18:39:38 GMT+0800 (China Standard Time)

@joyeecheung, please test the fix at crrev.com/c/4788992. What's your workflow to patch in some V8 version on top of node patches?

Michaël Zasso · Answer 18 · Fri Aug 18 2023 18:55:04 GMT+0800 (China Standard Time)

I just tried locally with the V8 lkgr and the other patches from https://github.com/joyeecheung/node/tree/ro-code-cache

It's almost all good. Only one test failure with the debug build:

=== debug test-single-executable-application-use-code-cache ===
Path: sequential/test-single-executable-application-use-code-cache
Wrote single executable preparation blob to sea-prep.blob
(node:92408) ExperimentalWarning: Single executable application is an experimental feature and might change at any time
(Use `sea --trace-warnings ...` to show where the warning was created)
(node:92408) Warning: Code cache data rejected.
/Users/mzasso/git/nodejs/canary/test/common/index.js:697
        throw new TypeError(
        ^

TypeError: "Warning" was triggered without being expected.
Warning: Code cache data rejected.
    at wrapSafe (node:internal/modules/cjs/loader:1173:15)
    at embedderRunCjs (node:internal/util/embedding:19:27)
    at node:internal/main/embedding:18:8
    at process.<anonymous> (/Users/mzasso/git/nodejs/canary/test/common/index.js:697:15)
    at process.emit (node:events:526:35)
    at doEmitWarning (node:internal/process/warning:89:11)
    at process.processTicksAndRejections (node:internal/process/task_queues:81:21)

Node.js v21.0.0-pre
node:child_process:929
    throw err;
    ^

Error: Command failed: /Users/mzasso/git/nodejs/canary/test/.tmp.3860/sea -a --b=c d
(node:92408) ExperimentalWarning: Single executable application is an experimental feature and might change at any time
(Use `sea --trace-warnings ...` to show where the warning was created)
(node:92408) Warning: Code cache data rejected.
/Users/mzasso/git/nodejs/canary/test/common/index.js:697
        throw new TypeError(
        ^

TypeError: "Warning" was triggered without being expected.
Warning: Code cache data rejected.
    at wrapSafe (node:internal/modules/cjs/loader:1173:15)
    at embedderRunCjs (node:internal/util/embedding:19:27)
    at node:internal/main/embedding:18:8
    at process.<anonymous> (/Users/mzasso/git/nodejs/canary/test/common/index.js:697:15)
    at process.emit (node:events:526:35)
    at doEmitWarning (node:internal/process/warning:89:11)
    at process.processTicksAndRejections (node:internal/process/task_queues:81:21)

Node.js v21.0.0-pre

    at checkExecSyncError (node:child_process:890:11)
    at execFileSync (node:child_process:926:15)
    at Object.<anonymous> (/Users/mzasso/git/nodejs/canary/test/sequential/test-single-executable-application-use-code-cache.js:56:43)
    at Module._compile (node:internal/modules/cjs/loader:1241:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1295:10)
    at Module.load (node:internal/modules/cjs/loader:1091:32)
    at Module._load (node:internal/modules/cjs/loader:938:12)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:83:12)
    at node:internal/main/run_main_module:23:47 {
  status: 1,
  signal: null,
  output: [
    null,
    Buffer(19) [Uint8Array] [
      72, 101, 108, 108, 111,  44,
      32, 119, 111, 114, 108, 100,
      33,  32, 240, 159, 152, 138,
      10
    ],
    Buffer(882) [Uint8Array] [
       40, 110, 111, 100, 101,  58,  57,  50,  52,  48,  56,  41,
       32,  69, 120, 112, 101, 114, 105, 109, 101, 110, 116,  97,
      108,  87,  97, 114, 110, 105, 110, 103,  58,  32,  83, 105,
      110, 103, 108, 101,  32, 101, 120, 101,  99, 117, 116,  97,
       98, 108, 101,  32,  97, 112, 112, 108, 105,  99,  97, 116,
      105, 111, 110,  32, 105, 115,  32,  97, 110,  32, 101, 120,
      112, 101, 114, 105, 109, 101, 110, 116,  97, 108,  32, 102,
      101,  97, 116, 117, 114, 101,  32,  97, 110, 100,  32, 109,
      105, 103, 104, 116,
      ... 782 more items
    ]
  ],
  pid: 92408,
  stdout: Buffer(19) [Uint8Array] [
    72, 101, 108, 108, 111,  44,
    32, 119, 111, 114, 108, 100,
    33,  32, 240, 159, 152, 138,
    10
  ],
  stderr: Buffer(882) [Uint8Array] [
     40, 110, 111, 100, 101,  58,  57,  50,  52,  48,  56,  41,
     32,  69, 120, 112, 101, 114, 105, 109, 101, 110, 116,  97,
    108,  87,  97, 114, 110, 105, 110, 103,  58,  32,  83, 105,
    110, 103, 108, 101,  32, 101, 120, 101,  99, 117, 116,  97,
     98, 108, 101,  32,  97, 112, 112, 108, 105,  99,  97, 116,
    105, 111, 110,  32, 105, 115,  32,  97, 110,  32, 101, 120,
    112, 101, 114, 105, 109, 101, 110, 116,  97, 108,  32, 102,
    101,  97, 116, 117, 114, 101,  32,  97, 110, 100,  32, 109,
    105, 103, 104, 116,
    ... 782 more items
  ]
}

Node.js v21.0.0-pre
Command: out/Debug/node /Users/mzasso/git/nodejs/canary/test/sequential/test-single-executable-application-use-code-cache.js

Joyee Cheung · Answer 19 · Fri Aug 18 2023 19:45:23 GMT+0800 (China Standard Time)

Ah, right, SEA needs to build the code cache from an deserialized isolate too (it does things separately)

What's your workflow to patch in some V8 version on top of node patches?

@schuay We have a utility to backport a V8 commit to a Node.js checkout: https://github.com/nodejs/node-core-utils/blob/main/docs/git-node.md#git-node-v8 Normally if I want to backport it to Node.js with a proper backport commit to open PR with, I do something like this since I already have a local V8 check out

git node v8 backport --v8-dir ~/path/to/v8 $V8_COMMIT_SHA

Or, if I am just testing a V8 change locally without the intention of doing a backport PR, I do not use git node and just do this (which is easier for me to address any merge conflicts):

cd /path/to/v8
git format-patch $V8_COMMIT_SHA~1...$V8_COMMIT_SHA
cd /path/to/node
git am -3 --directory=deps/v8 /path/to/v8/*.patch

Joyee Cheung · Answer 20 · Fri Aug 18 2023 20:28:37 GMT+0800 (China Standard Time)

Locally nodejs/node#49226 fixes test-single-executable-application-use-code-cache.js for me, pushed it with #252 (comment) to https://github.com/joyeecheung/node/tree/ro-code-cache

Michaël Zasso · Answer 21 · Mon Sep 04 2023 19:16:45 GMT+0800 (China Standard Time)

This is fixed. Thanks everyone!