Giters

redis / redis-benchmarks-specification

The Redis benchmarks specification describes the cross-language/tools requirements and expectations to foster performance and observability standards around redis related technologies. Members from both industry and academia, including organizations and individuals are encouraged to contribute.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

[BUG]: "redis-benchmark-spec" test suite failures

odidev opened this issue · comments

commented

Describe the bug

I am executing benchmark tests following the documents here: < https://github.com/redis/redis-benchmarks-specification#testing-out-the-redis-benchmarks-spec-client-runner > on the Linux/AMD64 Ubuntu AWS instance.

However, tests failed with the below error:

2023-05-24 09:08:06 INFO Benchmark used memory at start of benchmark: 1g 
2023-05-24 09:08:06 INFO Checking if there is a keyspace check being enforced 
2023-05-24 09:08:06 INFO Ensuring keyspace length requirement = 1000000 is met. 
2023-05-24 09:08:06 ERROR The total numbers of keys in setup does not match the expected spec: 1000000!=1000002. Aborting... 
2023-05-24 09:08:06 CRITICAL Some unexpected exception was caught during local work. Failing test.... 
2023-05-24 09:08:06 CRITICAL <class 'Exception'>

Referring to the already raised ticket here: < #206 >, I already installed the latest version of redis-benchmarks-specification (0.1.70) available with pip. But I am getting the same error.

Can you please provide some pointers for the same?

Environment (please complete the following information):

  • OS: Ubuntu 22.04
  • PyPi version package: pip 23.1.2 from /usr/local/lib/python3.10/dist-packages/pip (python 3.10)
  • Platfrom: x86
commented

@odidev do you start with an empty DB before the test?
Can you retry with --flushall_on_every_test_start option enabled?

commented

@filipecosta90 , thank you for the response. --flushall_on_every_test_start helped running tests as expected.

I ran some random tests on both the x86_64 and ARM64 AWS instances. Those tests are passing.
However, there is a build variant used in all YML files, as below:

build-variants:
- gcc:8.5.0-amd64-debian-buster-default

The build variant is hardcoded for AMD64 platform. Since tests are passing even on the ARM64 platform, may I know, what is the significance of this build variant here?