test_fixture with valgrind integrated with exit code is not making ceedling fail

[math3os]

I have $VALGRIND variable set like that:
export VALGRIND="valgrind --leak-check=yes --error-exitcode=1 --quiet --error-markers=@VALGRIND-START,@VALGRIND-STOP"

and project.yaml setup like that

:tools:
  :test_fixture:
    :optional: TRUE
    :executable: $VALGRIND
    :arguments:
      - ${1}

when I run ceedling with verbosity[4] i see return code is 1 but ceedling continue its run without any fail

I there a way I could make ceedling handling that fail ?

I would expect ceedling to return with an exit code set to 1

[Michael Karlesky]

Sorry for this trouble. The reason Ceedling is not exiting is because test fixture handling intentionally does not treat a non-zero exit code as a build error. A Unity-based test executable sets its exit code equivalent to the number of failed test cases. That is, a test executable could encounter multiple failed test cases and still be considered successful (with a non-zero exit code).

There is an open PR for a Valgrind plugin (I've not looked at it). We are focusing on a major release that is Ceedling 0.32 and are not incorporating new plugins at the moment. But, I think good Valgrind support is an obvious opportunity for Ceedling. I'll make sure we deal with this properly in some time.

Can you explain what you are trying to accomplish and how you would expect Ceedling to behave? This will be useful towards eventual Valgrind support.

[math3os]

Thank you for this answer, it help me to understand better why it was not working.

Since ceeding is integrated to our CI/CD by shell script, i succeeded to return failure by adding --error-markers=@VALGRIND-START,@VALGRIND-STOP" to valgrind command, grep output of ceedling and return error if @Valgrind is found.

i'd like to have ceedling reporting any memory issue raised by valgrind. i guess with a command like memtest:all.

[Michael Karlesky]

Your solution is actually quite similar to how Ceedling handles Unity test executables. It scans the console output of the test executable for expected test result messages. If it finds those, it considers it a success. If it doesn't, then it's a build error. In the error case, it also does some specific checking for messages like “segmentation fault.”

The contributed but not yet integrated Valgrind plugin creates a separate set of tasks valgrind: for test builds. This seems similar to what you're thinking. I gather you'd want these tasks to generate reports and optionally cause a build error upon configure Valgrind conditions?