Test failures on AppleClang 14.0.3 (XCode 14.3.1)

Question

Test failures on AppleClang 14.0.3 (XCode 14.3.1)

aarlt opened this issue a year ago · comments

I just noticed that evmone seem not to work correctly on M1 machines. I ran into this after I updated my local installation to v0.10.0 and try to run the solidity test suite.

However, after I enabled the evmone test suite via -DEVMC_TESTING=true I noticed that some of the unit tests are failing:

[----------] Global test environment tear-down
[==========] 845 tests from 19 test suites ran. (410 ms total)
[  PASSED  ] 808 tests.
[  FAILED  ] 37 tests, listed below:
[  FAILED  ] state_transition.block_apply_withdrawal
[  FAILED  ] state_transition.create2_factory
[  FAILED  ] state_transition.create_tx
[  FAILED  ] state_transition.eof_invalid_initcode
[  FAILED  ] evmone/evm.sub_and_swap/advanced, where GetParam() = 0x104a207a0
[  FAILED  ] evmone/evm.sub_and_swap/baseline, where GetParam() = 0x104a207a8
[  FAILED  ] evmone/evm.sub_and_swap/bnocgoto, where GetParam() = 0x104a207b0
[  FAILED  ] evmone/evm.arith/advanced, where GetParam() = 0x104a207a0
[  FAILED  ] evmone/evm.arith/baseline, where GetParam() = 0x104a207a8
[  FAILED  ] evmone/evm.arith/bnocgoto, where GetParam() = 0x104a207b0
[  FAILED  ] evmone/evm.comparison/advanced, where GetParam() = 0x104a207a0
[  FAILED  ] evmone/evm.comparison/baseline, where GetParam() = 0x104a207a8
[  FAILED  ] evmone/evm.comparison/bnocgoto, where GetParam() = 0x104a207b0
[  FAILED  ] evmone/evm.divmod/advanced, where GetParam() = 0x104a207a0
[  FAILED  ] evmone/evm.divmod/baseline, where GetParam() = 0x104a207a8
[  FAILED  ] evmone/evm.divmod/bnocgoto, where GetParam() = 0x104a207b0
[  FAILED  ] evmone/evm.signextend_31/advanced, where GetParam() = 0x104a207a0
[  FAILED  ] evmone/evm.signextend_31/baseline, where GetParam() = 0x104a207a8
[  FAILED  ] evmone/evm.signextend_31/bnocgoto, where GetParam() = 0x104a207b0
[  FAILED  ] evmone/evm.exp_oog/advanced, where GetParam() = 0x104a207a0
[  FAILED  ] evmone/evm.exp_oog/baseline, where GetParam() = 0x104a207a8
[  FAILED  ] evmone/evm.exp_oog/bnocgoto, where GetParam() = 0x104a207b0
[  FAILED  ] evmone/evm.sar/advanced, where GetParam() = 0x104a207a0
[  FAILED  ] evmone/evm.sar/baseline, where GetParam() = 0x104a207a8
[  FAILED  ] evmone/evm.sar/bnocgoto, where GetParam() = 0x104a207b0
[  FAILED  ] evmone/evm.delegatecall/advanced, where GetParam() = 0x104a207a0
[  FAILED  ] evmone/evm.delegatecall/baseline, where GetParam() = 0x104a207a8
[  FAILED  ] evmone/evm.delegatecall/bnocgoto, where GetParam() = 0x104a207b0
[  FAILED  ] evmone/evm.eof_function_example1/baseline, where GetParam() = 0x104a207a8
[  FAILED  ] evmone/evm.eof_function_example1/bnocgoto, where GetParam() = 0x104a207b0
[  FAILED  ] evmone/evm.eof_function_example2/baseline, where GetParam() = 0x104a207a8
[  FAILED  ] evmone/evm.eof_function_example2/bnocgoto, where GetParam() = 0x104a207b0
[  FAILED  ] evmone/evm.callf_call_stack_size_1024/baseline, where GetParam() = 0x104a207a8
[  FAILED  ] evmone/evm.callf_call_stack_size_1024/bnocgoto, where GetParam() = 0x104a207b0
[  FAILED  ] evmone/evm.extcode/advanced, where GetParam() = 0x104a207a0
[  FAILED  ] evmone/evm.extcode/baseline, where GetParam() = 0x104a207a8
[  FAILED  ] evmone/evm.extcode/bnocgoto, where GetParam() = 0x104a207b0

37 FAILED TESTS

Alexander Arlt · Answer 1 · Wed May 24 2023 23:14:19 GMT+0800 (China Standard Time)

I also thought that this might be a general arm related problem - but everything seem to work as expected for linux running on aarch64.

Paweł Bylica · Answer 2 · Fri May 26 2023 18:27:15 GMT+0800 (China Standard Time)

I think I reproduced some failures on CircleCI M1 build: https://app.circleci.com/pipelines/github/ethereum/evmone/4570/workflows/a0cb3067-5e21-40d9-bee2-fb738647cd87/jobs/51838.

Paweł Bylica · Answer 3 · Mon May 29 2023 22:19:14 GMT+0800 (China Standard Time)

Hey @aarlt,

It looks the problem is in the newest AppleClang 14.0.3 (XCode 14.3.1) only.
If works fine in AppleClang 14.0.0 (XCode 14.2.0).

Unfortunately, sanitizers do not report any problems.

Alexander Arlt · Answer 4 · Tue May 30 2023 17:31:55 GMT+0800 (China Standard Time)

Ah ok! That's interesting.. so you think it's a compiler bug?

Alexander Arlt · Answer 5 · Tue May 30 2023 17:47:20 GMT+0800 (China Standard Time)

Ok, I just used gcc 13.1.0 now and everything seem to work as expected.

However, I also tried to compile the unit tests, but that was failing due to some linking errors. I may create another issue for that one later.

Thx @chfast :)

Alexander Arlt · Answer 6 · Tue May 30 2023 17:50:02 GMT+0800 (China Standard Time)

I just closed this issue for now because it seem to be only a problem with the latest apple clang. Let's see how everything will behave with future releases of apple clang.

Paweł Bylica · Answer 7 · Tue May 30 2023 17:50:48 GMT+0800 (China Standard Time)

It can be a compiler bug or a bug in our code revealed by the newer compiler. Anyway, we will take a look at this later because we don't have direct access to this compiler yet.

Paweł Bylica · Answer 8 · Thu Jun 01 2023 04:37:12 GMT+0800 (China Standard Time)

@yperbasis said intx is failing in XCode 14.3 due to __builtin_subcll.