More verbose output from failed comparisons

[Wojciech Śmigaj]

In many unit testing frameworks, such as Boost.Test and Catch2, failed comparison assertions (such as EXPECT(value1 == value2)) produce a diagnostic message containing the values of the two expressions being compared. For example, a test such as

int a = 1;
int b = 2;
EXPECT(a == b);

might produce a message such as

error: check a == b has failed [1 != 2]

This can be quite helpful when debugging unit test failures, and @srherbener and I are wondering whether this feature could be added to eckit. We can think of two ways of doing that:

1. Define new macros such as EXPECT_EQUAL(actual, expected) that, on failure, will throw an exception with an error message including the values of the macro parameters. In general, a separate macro would be needed for each comparison operator (EXPECT_EQUAL, EXPECT_GREATER, EXPECT_GREATER_OR_EQUAL etc.) and for the special floating-point comparisons (EXPECT_APPROXIMATELY_EQUAL etc.).

2. Continue to use a single macro (EXPECT), but make the functions is_equal, is_greater_or_equal, is_approximately_equal etc. defined in eckit/types/FloatCompare.h print their arguments e.g. to the Log::info() stream if they're about to return false. For backward compatibility, this behaviour could be disabled by default and activated by passing true to a new optional parameter of these functions, say bool verbose=false. Alternatively, a new family of verbose wrappers to these function (verbose_is_equal, verbose_is_greater_or_equal etc.) could be defined; each wrapper would call the original function and print the arguments if that function returned false.

We'd be happy to submit a pull request implementing this feature in one of these ways if you think it could be a useful addition to eckit.

See https://github.com/JCSDA/eckit/pull/9 for the original discussion about this feature and an example implementation of the EXPECT_EQUAL macro.

[Tiago Quintino]

The rationale for not providing EXPECT_EQUAL and similar macros was:

1. it places additional requirements to the types passed into the expression, such as conversion to string or output to ostream
2. imposes a decision on which operand to use for the comparison (typically the operator==)

So, for generic comparisons EXPECT() continues to be the recommended macro.

However I do recognise that it is convenient to output enriched information in that case of failure.
One way to do this would be to provide a EXPECT_MSG(expr, msgcall) where the client would pass a callable (eg lambda) to report whatever the client code thinks is best.

EXPECT_MSG( a == b, [=](){ std::cout << a << "!=" << b << std::endl; }

The EXPECT_EQUAL(a, b) could then be implemented based on the above macro (as could EXPECT itself, by passing a null callable).

I am working on a development in this line. Stay tuned ;-)

[Wojciech Śmigaj]

Thanks -- it's great to hear such "verbose assertions" are likely to be added to eckit.

[Stephen Herbener]

@tlmquintino, the EXPECT_MSG idea looks good. Thanks for working on this!

[Willem Deconinck]

Commit cb6961d implements EXPECT_MSG and a few extra like EXPECT_EQUAL, EXPECT_NOT

[Wojciech Śmigaj]

Great -- thanks for adding it so quickly!