[SPEC]: Test failures on the Linux/ARM64 platform
odidev opened this issue · comments
Improvement/Addition type
What changes to the current SPEC does this issue focuses upon?
- [] Add a benchmark variant
- [] Improve a benchmark variant
- [] Raise awareness for a new Redis benchmark tool
- [] Suggest a different Redis Build Variant
- [ * ] Other (fill details in section bellow)
Detail the requested use-case
I am working with redis-benchmark-specification on the Linux/ARM64 Ubuntu AWS instance.
I am following this documentation < https://github.com/redis/redis-benchmarks-specification#development > to execute tests with tox.
Out of 26 tests, 2 tests are failing, as can be seen from the logs below:
utils/tests/test_self_contained_coordinator.py ....F [ 80%]
utils/tests/test_self_contained_coordinator_memtier.py F.. [ 92%]
These 2 test cases are failing for the same assertion issue (while comparing the value of “result”) after calling the function “test_self_contained_coordinator_blocking_read()” in files test_self_contained_coordinator_memtier.py and test_self_contained_coordinator.py.
=================================================================================== short test summary info ====================================================================================
FAILED utils/tests/test_self_contained_coordinator.py::test_self_contained_coordinator_blocking_read - assert False == True
FAILED utils/tests/test_self_contained_coordinator_memtier.py::test_self_contained_coordinator_blocking_read - assert False == True
==================================================================== 2 failed, 24 passed, 249 warnings in 567.41s (0:09:27) ====================================================================
ERROR: InvocationError for command /home/ubuntu/redis-benchmarks-specification/.tox/integration-tests/bin/coverage run '--include=redis_benchmarks_specification/*' -m pytest -ra (exited with code 1)
On further debugging, it looks like the tests are failing to run docker container using image redislabs/memtier_benchmark:edge here < https://github.com/redis/redis-benchmarks-specification/blob/main/redis_benchmarks_specification/__self_contained_coordinator__/self_contained_coordinator.py#L1025 >, as this docker image is not available for the Linux/ARM64 platform at DockerHub.
I have added a comment in this already raised ticker < RedisLabs/memtier_benchmark#213 > in the memtier-benchmark GitHub repo, requesting the same.
Can you please provide some pointers whether my analysis seems correct?
Also, may I know, are you interested in adding Linux/ARM64 platform support to redis-benchmark-specification? Kindly provide your suggestions on the same.
Screenshots/CLI snippets of what's expected to happen
If applicable, add screenshots to help explain your problem. For example a chart.
Detail the used benchmark tool
What benchamrk tool to use:
- [] memtier_benchmark
- [] redis-benchmark
- [] redis-benchmark-go
- [] Other (name it):
Sample benchmark tool command to implement the requested use-case:
$ git clone https://github.com/redis/redis-benchmarks-specification.git && cd redis-benchmarks-specification
$ pip install poetry
$ poetry install
$ poetry run black .
$ poetry run flake8
$ pip3 install -r ./dev_requirements.txt
$ tox
Additional context
Add any other context about the problem here.
I built multi-arch memtier-benchmark docker image using Dockerfile here < https://github.com/RedisLabs/memtier_benchmark/blob/master/Dockerfile > and replaced “redislabs/memtier_benchmark:edge” with my image in the source code. Also, installed and started redis-server on port 6379. Now, all 26 tests are passing via tox on the Linux/ARM64 platform.
========================================================================= 26 passed, 255 warnings in 437.64s (0:07:17) =========================================================================
integration-tests run-test: commands[4] | coverage report -m
Name Stmts Miss Cover Missing
---------------------------------------------------------------------------------------------------------------------------
redis_benchmarks_specification/__api__/__init__.py 0 0 100%
redis_benchmarks_specification/__api__/app.py 118 17 86% 46-49, 54, 59-67, 76, 78, 115
redis_benchmarks_specification/__builder__/__init__.py 0 0 100%
redis_benchmarks_specification/__builder__/builder.py 191 58 70% 54-144, 167-168, 195, 222, 231-236, 241-258, 270, 296, 342, 373, 381-387, 424-430, 439
redis_benchmarks_specification/__builder__/schema.py 13 0 100%
redis_benchmarks_specification/__cli__/__init__.py 0 0 100%
redis_benchmarks_specification/__cli__/args.py 38 0 100%
redis_benchmarks_specification/__cli__/cli.py 122 33 73% 44-55, 90, 108, 129-135, 158, 188, 194, 207, 215-230, 243-277
redis_benchmarks_specification/__cli__/stats.py 317 308 3% 20-502
redis_benchmarks_specification/__common__/__init__.py 0 0 100%
redis_benchmarks_specification/__common__/builder_schema.py 86 11 87% 53, 81-86, 103-106, 114, 151
redis_benchmarks_specification/__common__/env.py 45 1 98% 79
redis_benchmarks_specification/__common__/package.py 19 11 42% 10-21
redis_benchmarks_specification/__common__/runner.py 84 13 85% 22, 27, 29-40, 45, 50-51, 119-120
redis_benchmarks_specification/__common__/spec.py 22 0 100%
redis_benchmarks_specification/__init__.py 6 2 67% 14-15
redis_benchmarks_specification/__runner__/__init__.py 0 0 100%
redis_benchmarks_specification/__runner__/args.py 47 0 100%
redis_benchmarks_specification/__runner__/runner.py 430 156 64% 92-100, 105-106, 142-150, 166-174, 177, 235, 251-254, 266-314, 380, 389-390, 420, 436-460, 473-480, 500, 505-510, 522-532, 537-554, 564-570, 573-574, 577-601, 620-621, 627, 651, 692-695, 719-736, 795-797, 864-873, 877-886, 889-899, 903, 908-909, 929-938, 941, 953, 982-987, 991-997, 1066-1172
redis_benchmarks_specification/__self_contained_coordinator__/__init__.py 0 0 100%
redis_benchmarks_specification/__self_contained_coordinator__/args.py 34 31 9% 25-153
redis_benchmarks_specification/__self_contained_coordinator__/artifacts.py 12 0 100%
redis_benchmarks_specification/__self_contained_coordinator__/build_info.py 41 3 93% 21, 37, 42
redis_benchmarks_specification/__self_contained_coordinator__/clients.py 8 0 100%
redis_benchmarks_specification/__self_contained_coordinator__/cpuset.py 11 0 100%
redis_benchmarks_specification/__self_contained_coordinator__/docker.py 27 14 48% 37-46, 59-90
redis_benchmarks_specification/__self_contained_coordinator__/prepopulation.py 22 14 36% 28-86
redis_benchmarks_specification/__self_contained_coordinator__/runners.py 195 158 19% 88-89, 120-621
redis_benchmarks_specification/__self_contained_coordinator__/self_contained_coordinator.py 333 137 59% 98-252, 315, 351, 359, 385, 440-448, 452, 463-468, 529-538, 620, 751-831, 838, 902-920, 935-945, 957-973, 1055-1060
redis_benchmarks_specification/__setups__/__init__.py 0 0 100%
redis_benchmarks_specification/__setups__/topologies.py 13 0 100%
---------------------------------------------------------------------------------------------------------------------------
TOTAL 2234 967 57%
integration-tests docker: remove 'd6f48015f9' (from 'datasink')
integration-tests docker: remove '4ec0386d4c' (from 'db_server')
___________________________________________________________________________________________ summary ____________________________________________________________________________________________
integration-tests: commands succeeded
congratulations :)
Closing this ticket since the redislabs/memtier-benchmark:edge docker image is now available for multi-arch platform at docker hub, and the above issue got resolved.