Wasm vs EVM Benchmarking Tool

This is a tool to compile smart contracts for benchmarking purposes.

How to Use

This repository contains a selection of benchmarking projects. Each project contains a particular function implemented in different kind of smart contracts languages.

This tool is to be used as follows:

compile all smart contracts to their respective targets
after that use the compiled smart contracts to run benchmarks

Required Tools

nodejs and npm
rust and cargo
cargo contract CLI
Solang

How to Compile

first npm install

create a file config.json in the root folder with the following content

{
  "pathToSolang": "relative/or/absolute/path/to/solang/binary"
}

then select one of the projects and run either one of the commands
- npm run build-odd-product
- npm run build-odd-product-write
- npm run build-triangle
- npm run build-sha512

Be aware that this will take a couple of minutes. The script will also show the size of the compiled smart contracts. All files will be compiled with highest optimization settings.

Project

This repository contains three benchmarking projects

triangleNumbers: compute the triangle number of n, i.e., the sum of all numbers 1 <= i <= n
oddProduct: compute the product of the first n odd numbers
sha512: compute the iterated SHA-512 for a given input; the source files avoid the use of standard library functions and use direct implementations of SHA-512 instead

Every project contains the following 6 files:

baseline.js: a pure JavaScript implementation of the function
baseline.rs: a pure Rust implementation of the function
contract.rs: the ink! version of the smart contract
contract.sol: the Solidity version of the smart contract
contract.wat: the direct implementation of the smart contract in Wasm text format
pallet.rs: a Substrate pallet version of the function

The project files will be compiled as follows (these files will be generated in the build folder):

baseline: the compiled executable of baseline.js
baseline.js: just a copy of the verbatim baseline.js code
ink.contract: the compiled ink! smart contract with metadata, usable for pallet-contracts
solidity.contract: the Solidity smart contract compiled to Wasm via Solang, usable for pallet-contracts
wat.contract: the compiled Wasm text format smart contract from contract.wat, usable for pallet-contracts
pallet-chain: a compiled standalone Substrate node that contains the pallet from pallet.rs
solidity.evm: the Solidity smart contract compiled to EVM byte code, usable for Ethereum nodes or pallet-evm

The purpose of the baseline files is to compare the execution time of the contracts to natively executed Rust or JavaScript code

How to Benchmark

This section describes how to run the benchmarks using the compiled contracts and chains.

Wat Contract in Contracts Pallet

start a local Substrate node with pallet-contracts in --dev mode
- the node needs to implement pallet-contracts and its RPC API, e.g. the node template with pallet-contracts
launch Polkadot UI and connect to local node
go to Developer -> Contracts and click on "Upload & deploy code"
select the compiled file wat.contract
- use "ALICE" as deployment account
- then click "Next", then "Deploy", then "Sign and Submit"
select the contract and send a message
- it needs to be an exec message, i.e., the contract needs be executed on chain
- wait for the estimated gas to be shown in the field "max gas allowed"
once the message is processed, check the log output of the node
- the line starting with Prepared block for proposing of the according block will show the processing time in ms

Ink Contract in Contracts Pallet

same as for the wat contract but use the contract file ink.contract

Solidity Contract in Contracts Pallet

same as for the wat contract but use the contract file solidity.contract

Solidity Contract in EVM Pallet

run a local Substrate chain with pallets-evm
- fork all of frontier https://github.com/paritytech/frontier.git
- go to template folder
- inside run cargo build --features=runtime-benchmarks --release
- on the main frontier main folder run ./target/release/frontier-template-node --dev
launch Polkadot UI and connect to local node
- add developer configuration for EVM
go to Developer → Extrinsics and select the extrinisic evm → create and enter
- source: 0xd43593c715fdd31c61141abd04a99fd6822c8558 (Alice's Ethereum address)
- init: copy the compiled contract found inside the file solidity.evm (the complete line starting with 0x)
- value: 0
- gasLimit and maxFeePerGas: 4294967295
- remove items from the accessList at the end
- submit the transaction
immediately go to Network -> Chain info and locate the block that contained the contract creation
- it will be a block containing an evm.Created event
- open the event and note down the H160 address (this is the Ethereum address of the contract)
go to Developer → Extrinsics and select evm → call and specify
- source: 0xd43593c715fdd31c61141abd04a99fd6822c8558 (Alice's Ethereum address)
- target: the address of the contract (see above)
- input: a hex value encoding the input to the call using the Solidity ABI
  - prefix with 0x
  - concat the function selector
    - find this function selector in the file solidity.evm
  - add arguments
    - if they are fixed size, then one argument after another
    - addresses are uint160
    - any uint<M> or int<M> is left padded so that it has 32 bytes (with 00 when nonnegative or with ff when negative)
- value: 0
- gasLimit and maxFeePerGas: 4294967295
- remove items from the accessList at the end
- submit the transaction
once the transaction is processed, check the log output of the node
- the line starting with Prepared block for proposing of the according block will show the processing time in ms

Solidity Contract in Geth

first compile a local Geth configured with much higher gas limits
- install Go
- clone https://github.com/ethereum/go-ethereum.git
- make some changes to the standard configuration
  - edit eth/ethconfig/config.go and change Defaults config
    - change RPCGasCap to 5000000000
    - change RPCTxFeeCap to 1000
- run make geth

start geth in local development mode

go to the root folder of the cloned repository
if there is a folder data, remove it

run

./build/bin/geth --datadir data --http --dev --dev.gaslimit 1000000000 --http.corsdomain "https://remix.ethereum.org,http://remix.ethereum.org"

take note of the log line “HTTP server started” and note the endpoint
- it is probably 127.0.0.1:8545
in browser go to Remix IDE
in the side bar go to “File explorers” icon
- in file tree open folder “contract” and add a new file contract.sol
- paste the solidity contract contract.sol from the project source folder
in the side bar go to “Solidity compiler” icon
- select the compiler version matching the version of solc npm package in package.json (no nightly version)
- select EVM version "london"
- enable optimization with 200 runs
- then click button “Compile contract.sol”
- if required, click on the button “Compilation Details” and inspect details like BYTECODE → object for the generated bytecode or FUNCTIONHASHES for the selectors of the functions
in the side bar go to “Deploy & run transactions” icon
- in ENVIRONMENT select “Web 3 Provider”
  - in the popup ensure that the “Web3 Provider Endpoint” is the http server of the local geth node (see above)
  - confirm the popup
- in the CONTRACT pulldown ensure that the compiled contract is selected
- click “Deploy”
  - the local Geth node should mine a new block
- the deployed contract will now appear under "Deployed Contracts"
  - open it
  - select a function, enter the input values and click the button with the name of the function
    - the local node will mine a new block
  - in the console line starting Commit new sealing work there is a field elapsed that describes the time passed and the field gas describes the gas payed

Substrate Pallet

the compiled file pallet-chain is a complete standalone substrate node containing the pallet
the substrate pallet can be executed through the wasm (i.e., it will run through a Wasm engine) or through the native executor (i.e., the native compiled code will be executed)

execute either one of the following commands to start the node:

./pallet-chain --execution Wasm --dev
./pallet-chain --execution Native --dev

launch Polkadot UI and connect to local node
go to Developer -> Extrinsics and and select templateModule and select the function to call
select the compiled file wat.contract
- enter the required input values
- submit the transaction
once the transaction is processed, check the log output of the node
- the line starting with Prepared block for proposing of the according block will show the processing time in ms

JavaScript Baseline

go to the build folder and to the project subfolder
run node baseline.js and add any required command line arguments

Rust Baseline

go to the build folder and to the project subfolder
run ./baseline and add any required command line arguments

TorstenStueber / wasm-vs-evm