Halmos Keccak producing incorrect values

[plotchy]

Describe the bug
Halmos keccak256 seems to mismatch the underlying bytecode representation

To Reproduce

function check_keccak_matching() public {
    bytes32 a = keccak256(hex"01");
    bytes32 b = keccak256(abi.encodePacked(hex"01"));
    assert(a == b);
}

function check_keccak_matching_2() public {
    bytes32 a = keccak256(hex"01");
    bytes32 b = keccak256(abi.encodePacked(uint8(1)));
    assert(a == b);
}

halmos --contract HERC721Test --print-full-model --print-failed-states

Running 2 tests for test/H_ERC721.t.sol:HERC721Test
[FAIL] check_keccak_matching() (paths: 1/3, time: 0.01s, bounds: [])
Counterexample: 
    msg_value = 0x0
    sha3_8 = [else -> 0xdfe7f977e71dba2ea1a68e21057beebb9be2ac30c6410aa38d4f3fbe41dcffd2]
# 2 / 3
PC: 0xaaaa0001 0x34d REVERT
Stack: [0x181, sha3_8(0x1), 0x5fe7f977e71dba2ea1a68e21057beebb9be2ac30c6410aa38d4f3fbe41dcffd2, 0xda, 0x50cb53c7]
Balance: balance_0
Storage:
- 0xaaaa0001: {0x0: {0x0: 0x10001}}
Path:
- msg_value == 0x0
- Not(sha3_8(0x1) == 0x5fe7f977e71dba2ea1a68e21057beebb9be2ac30c6410aa38d4f3fbe41dcffd2)
Output: 4e487b710000000000000000000000000000000000000000000000000000000000000001
Log: []
Balance updates:
Storage updates:
SHA3 hashes:
- 0x0: sha3_8(0x1)
External calls:

[FAIL] check_keccak_matching_2() (paths: 1/3, time: 0.01s, bounds: [])
Counterexample: 
    msg_value = 0x0
    sha3_8 = [else -> 0xdfe7f977e71dba2ea1a68e21057beebb9be2ac30c6410aa38d4f3fbe41dcffd2]
# 2 / 3
PC: 0xaaaa0001 0x34d REVERT
Stack: [0x181, sha3_8(0x1), 0x5fe7f977e71dba2ea1a68e21057beebb9be2ac30c6410aa38d4f3fbe41dcffd2, 0xda, 0x72430ece]
Balance: balance_0
Storage:
- 0xaaaa0001: {0x0: {0x0: 0x10001}}
Path:
- msg_value == 0x0
- Not(sha3_8(0x1) == 0x5fe7f977e71dba2ea1a68e21057beebb9be2ac30c6410aa38d4f3fbe41dcffd2)
Output: 4e487b710000000000000000000000000000000000000000000000000000000000000001
Log: []
Balance updates:
Storage updates:
SHA3 hashes:
- 0x0: sha3_8(0x1)
External calls:

Symbolic test result: 0 passed; 2 failed; time: 0.03s

seems that halmos is producing 0xdfe7f977e71dba2ea1a68e21057beebb9be2ac30c6410aa38d4f3fbe41dcffd2 whereas evm produces 0x5fe7f977e71dba2ea1a68e21057beebb9be2ac30c6410aa38d4f3fbe41dcffd2

Environment:

• OS: macOS
• Python version: Python 3.11.3
• Halmos and other dependency versions:
  Package Version

---

halmos 0.1.2
pip 23.1.2
setuptools 67.7.2
wheel 0.40.0
z3-solver 4.12.2.0

[plotchy]

Noticing now that its off by 1 bit only
0xdfe7f977e71dba2ea1a68e21057beebb9be2ac30c6410aa38d4f3fbe41dcffd2 ~~ 0x5fe7f977e71dba2ea1a68e21057beebb9be2ac30c6410aa38d4f3fbe41dcffd2

the leading bit is set vs not.

[Daejun Park]

This is a known limitation, and let me explain what happened.

Halmos doesn't actually execute the Keccak algorithm because it is too complicated to symbolically run. Instead, the Keccak function is encoded as a black box (referred to as an uninterpreted function abstraction) with certain desired properties, specifically the injectivity assumption to prevent hash collisions. This approach works well, unless the black box output is compared with a real hash output value.

In your example, although no hash output is directly provided in the source code, the expression keccak256(hex"01") is evaluated to its hash value at compile time (thanks to the smart compiler!), and the hash output is included in the compiled bytecode. Note that keccak256(abi.encodePacked(hex"01")) is not evaluated at compile time in the current compiler version. This leads to a comparison between a black-boxed hash output and a real hash output during symbolic execution. At the end, the assertion becomes merely an equality between a symbol and a constant, and the solver produces a counterexample, suggesting that the symbol may differ from the constant.

We were aware of this limitation but believed it would not frequently occur in real-world examples. We might be wrong. Could you please provide more context about where you encountered this issue?

[plotchy]

Ah, got it!

This did not come up in a real world example. I noticed the SHA3 hashes list printed out when debugging a trace and wanted to make a basic test to study how that list is used by halmos. This was an example that I thought would write to the list on the first keccak, and then iterate on the list upon the second keccak to retrieve the already known hash. I see how this went wrong with the compile time eval now.