Too many failed reported in sudo-compliance-tests
rtczza opened this issue · comments
in sudo-rs/test-framework/sudo-compliance-tests run SUDO_UNDER_TEST=theirs cargo test :
test helpers::rsyslogd_works has been running for over 60 seconds
test su::cli::arguments_are_passed_to_shell has been running for over 60 seconds
test su::cli::dash_user_shell_arguments has been running for over 60 seconds
test su::cli::flag_after_positional_argument has been running for over 60 seconds
test su::default_target_is_root has been running for over 60 seconds
test su::does_not_panic_on_io_errors has been running for over 60 seconds
test su::env::vars_in_invoking_users_env_are_preserved has been running for over 60 seconds
test su::env::vars_set_by_su_override_existing_ones has been running for over 60 seconds
test su::env::vars_set_by_su_when_target_is_not_root has been running for over 60 seconds
test su::env::vars_set_by_su_when_target_is_root has been running for over 60 seconds
test su::explicit_target_user has been running for over 60 seconds
test su::flag_command::it_works has been running for over 60 seconds
test su::flag_command::pass_to_shell_via_c_flag has been running for over 60 seconds
test su::flag_command::positional_arguments_are_not_passed_to_command has been running for over 60 seconds
test su::flag_command::when_specified_more_than_once_only_last_value_is_used has been running for over 60 seconds
test su::flag_group::invoking_user_must_be_root has been running for over 60 seconds
test helpers::rsyslogd_works ... FAILED
test su::cli::dash_user_shell_arguments ... FAILED
test su::env::vars_in_invoking_users_env_are_preserved ... FAILED
test su::flag_command::pass_to_shell_via_c_flag ... FAILED
test su::env::vars_set_by_su_override_existing_ones ... FAILED
test su::default_target_is_root ... FAILED
test su::flag_group::invoking_user_must_be_root ... FAILED
test su::flag_command::it_works ... FAILED
test su::flag_command::when_specified_more_than_once_only_last_value_is_used ... FAILED
test su::env::vars_set_by_su_when_target_is_not_root ... FAILED
test su::cli::arguments_are_passed_to_shell ... FAILED
test su::does_not_panic_on_io_errors ... FAILED
...
final results:
test result: FAILED. 3 passed; 645 failed; 60 ignored; 0 measured; 0 filtered out; finished in 164.37s
error: test failed, to rerun pass `--lib`
Is it the wrong way to test? Or missed some steps?
rust version:
rustc 1.72.1 (d5c2e9c34 2023-09-13)
I ran the same command on the same folder and got this:
test result: ok. 648 passed; 0 failed; 60 ignored; 0 measured; 0 filtered out; finished in 164.67s
I have rust 1.75 but I doubt that's the issue. It seems there's something wrong with the containers as almost every single test is failing.
have run the following two commands, but still get the above error.
docker run --rm -it sudo-test-og
docker exec -u 1000:100 -it 70dc3ea36f56 bash
I have always had some doubts about the execution of docker.
Can you provide the commands for the entire testing process? Including how to run the docker you need to use
most likely the shared step of building the docker image is failing and that's causing all tests that rely on the docker image to fail. you can set the SUDO_TEST_VERBOSE_DOCKER_BUILD env var to see the build process of the docker image and figure out what's going wrong there. I suggest you also pass a test filter to only run one test, or as few tests as possible, to reduce the amount of unhelpful text in the output.
$ cd test-framework
$ SUDO_TEST_VERBOSE_DOCKER_BUILD=1 SUDO_UNDER_TEST=theirs cargo t -p sudo-compliance-tests -- insufficient_permissions_to_execute_shell
[+] Building 25.9s (9/9) FINISHED docker-container:mybuilder
=> [internal] booting buildkit 2.4s
=> => starting container buildx_buildkit_mybuilder0 2.4s
=> [internal] load build definition from Dockerfile 0.1s
=> => transferring dockerfile: 236B 0.0s
=> [internal] load metadata for docker.io/library/debian:bookworm-slim 2.2s
=> [internal] load .dockerignore 0.1s
=> => transferring context: 2B 0.0s
=> [1/3] FROM docker.io/library/debian:bookworm-slim@sha256:f4a83aa865a2b4a064ff142aa91c713180df9fcb86ce676b5de2981029379c37 0.1s
=> => resolve docker.io/library/debian:bookworm-slim@sha256:f4a83aa865a2b4a064ff142aa91c713180df9fcb86ce676b5de2981029379c37 0.0s
=> [2/3] RUN apt-get update && apt-get install -y --no-install-recommends sudo procps sshpass rsyslog && rm /etc/sudoers 13.4s
=> [3/3] WORKDIR /tmp 0.2s
=> exporting to docker image format 6.9s
=> => exporting layers 3.5s
=> => exporting manifest sha256:79c8abfb0d71d4b95dcf44d5edd32d71f340f8d5f0a12d1af1cda32aeddf8533 0.0s
=> => exporting config sha256:d46dfe4cae602bd5d7ee9cbc42ad71f71b83e266cea62b7885d858da19438350 0.0s
=> => sending tarball 3.3s
=> importing to docker 1.8s
=> => loading layer 19602c60b3cf 196.61kB / 17.25MB 1.8s
=> => loading layer 5f70bf18a086 32B / 32B 0.3s
test sudo::flag_login::insufficient_permissions_to_execute_shell ... ok
(SUDO_TEST_VERBOSE_DOCKER_BUILD works with both values of SUDO_UNDER_TEST)