Giters

Shinmera / parachute

An extensible and cross-compatible testing framework.

Home Page:https://shinmera.github.io/parachute

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Failing and Skipped tests not working as expected

eduardoacye opened this issue · comments

commented

Thank you for writing this library!

Here are a couple of issues present in the latest version from quicklisp regarding failing and skipped tests.

Failing tests

Trying out all default test forms using the following definition:

(define-test failing-tests
  :description "Expecting all tests to fail"
  (true nil)
  (false 1)
  (fail 1)
  (is = 1 2)
  (isnt = 1 1)
  (is-values (values 1 1)
    (= 0)
    (= 2))
  (isnt-values (values 1 1)
    (= 1)
    (= 1))
  (of-type 'string 1)
  (finish (error "generic issue")))

Actual result

(ql:quickload :parachute)
To load "parachute":
  Load 3 ASDF systems:
    asdf documentation-utils form-fiddle
  Install 1 Quicklisp release:
    parachute
; Fetching #<URL "http://beta.quicklisp.org/archive/parachute/2019-03-07/parachute-20190307-git.tgz">
; 45.61KB
==================================================
46,702 bytes in 0.09 seconds (495.73KB/sec)
; Loading "parachute"
[package parachute]...............
(:PARACHUTE)
CL-USER> (in-package :parachute)
#<PACKAGE "PARACHUTE">
PARACHUTE> (define-test failing-tests
             :description "Expecting all tests to fail"
             (true nil)
             (false 1)
             (fail 1)
             (is = 1 2)
             (isnt = 1 1)
             (is-values (values 1 1)
               (= 0)
               (= 2))
             (isnt-values (values 1 1)
               (= 1)
               (= 1))
             (of-type 'string 1)
             (finish (error "generic issue")))
FAILING-TESTS
PARACHUTE> (test 'failing-tests)
        ？ PARACHUTE::FAILING-TESTS
  0.000 ✘   (true ())
  0.000 ✘   (false 1)
  0.000 ✘   (fail 1 error)
  0.000 ✘   (is = 1 2)
  0.000 ✘   (is = 1 1)
  0.002 ✔   (is-values (values 1 1) (= 0) (= 2))
  0.000 ✔   (isnt-values (values 1 1) (= 1) (= 1))
  0.000 ✘   (of-type 'string 1)
  0.003 ✘   (finish (error "generic issue"))
  0.042 ✘ PARACHUTE::FAILING-TESTS

;; Summary:
Passed:     2
Failed:     7
Skipped:    0

;; Failures:
   7/   9 tests failed in PARACHUTE::FAILING-TESTS
Expecting all tests to fail
The test form   ()
evaluated to    ()
when            t
was expected to be equal under GEQ.

The test form   1
evaluated to    1
when            ()
was expected to be equal under GEQ.

The test form   (capture-error 1)
evaluated to    ()
when            error
was expected to be equal under TYPEP.

The test form   2
evaluated to    2
when            1
was expected to be equal under =.

The test form   1
evaluated to    1
when            1
was expected to be unequal under =.

The test form   1
evaluated to    1
when            string
was expected to be equal under TYPEP.

The test form   (finish (error "generic issue"))
exited by       generic issue

Expected result

I was expecting (isnt = 1 1) to get reported as

  0.000 ✘   (isnt = 1 1)

instead of

  0.000 ✘   (is = 1 1)

The is-values and isnt-values tests should be failing.

Skipped tests

Skipping all default test forms using the following definition:

(define-test skipping-tests
  :description "Expecting all tests to be skipped"
  (skip "Failing tests"
    (true nil)
    (false 1)
    (fail 1)
    (is = 1 2)
    (isnt = 1 1)
    (is-values (values 1 1)
      (= 0)
      (= 2))
    (isnt-values (values 1 1)
      (= 1)
      (= 1))
    (of-type 'string 1)
    (finish (error "generic issue"))))

Actual result

CL-USER> (ql:quickload :parachute)
To load "parachute":
  Load 1 ASDF system:
    parachute
; Loading "parachute"

(:PARACHUTE)
CL-USER> (in-package :parachute)
#<PACKAGE "PARACHUTE">
PARACHUTE> (define-test skipping-tests
             :description "Expecting all tests to be skipped"
             (skip "Failing tests"
               (true nil)
               (false 1)
               (fail 1)
               (is = 1 2)
               (isnt = 1 1)
               (is-values (values 1 1)
                 (= 0)
                 (= 2))
               (isnt-values (values 1 1)
                 (= 1)
                 (= 1))
               (of-type 'string 1)
               (finish (error "generic issue"))))
SKIPPING-TESTS
PARACHUTE> (test 'skipping-tests)
        ？ PARACHUTE::SKIPPING-TESTS
        ？   SKIP: Failing tests
  0.000 ー     (true ())
  0.000 ー     (false 1)
  0.000 ー     (fail 1 error)
  0.000 ー     (is = 1 2)
  0.000 ー     (is = 1 1)
  0.000 ー     (is-values (values 1 1) (= 0) (= 2))
  0.000 ー     (isnt-values (values 1 1) (= 1) (= 1))
  0.000 ー     (of-type 'string 1)
  0.000 ✔     (finish (error "generic issue"))
  0.035 ✔   SKIP: Failing tests
  0.041 ✔ PARACHUTE::SKIPPING-TESTS

;; Summary:
Passed:     2
Failed:     0
Skipped:    8

Expected result

I was expecting the (finish (error "generic issue")) to get reported as:

  0.000 ー     (finish (error "generic issue"))

instead of

  0.000 ✔     (finish (error "generic issue"))

and have the Summary reflect that the test is being skipped.

Potential Fixes

It looks like in eval-in-context for finishing-result, when the test is skipped the (status result) is getting overridden with :passed.

Regarding the reporting of (isnt ...) as (is ...), I think it's sufficient to update the :expression value of the make-instance form in the isnt macro definition.

Regarding the is-values passing when expecting a failure, I think it's getting the :passed status, as the default status assigned at the :around method on eval-in-context for the result class. Heres a trace for that test:

PARACHUTE> (is-values (values 1 1) (= 0) (= 2))
  0: (PARACHUTE:EVAL-IN-CONTEXT NIL #<PARACHUTE:MULTIPLE-VALUE-COMPARISON-RESULT :UNKNOWN (is-values (values 1 1) (= 0) (= 2))>)
    1: ((METHOD PARACHUTE:EVAL-IN-CONTEXT :AROUND (T PARACHUTE:RESULT)) NIL #<PARACHUTE:MULTIPLE-VALUE-COMPARISON-RESULT :UNKNOWN (is-values (values 1 1) (= 0) (= 2))>)
      2: (PARACHUTE:STATUS #<PARACHUTE:MULTIPLE-VALUE-COMPARISON-RESULT :UNKNOWN (is-values (values 1 1) (= 0) (= 2))>)
      2: STATUS returned :UNKNOWN
      2: (PARACHUTE:CHECK-EVALUATABLE NIL #<PARACHUTE:MULTIPLE-VALUE-COMPARISON-RESULT :UNKNOWN (is-values (values 1 1) (= 0) (= 2))>)
        3: ((METHOD PARACHUTE:CHECK-EVALUATABLE (T PARACHUTE:MULTIPLE-VALUE-COMPARISON-RESULT)) NIL #<PARACHUTE:MULTIPLE-VALUE-COMPARISON-RESULT :UNKNOWN (is-values (values 1 1) (= 0) (= 2))>)
        3: (METHOD CHECK-EVALUATABLE
            (T MULTIPLE-VALUE-COMPARISON-RESULT)) returned
             NIL
      2: CHECK-EVALUATABLE returned NIL
      2: ((METHOD PARACHUTE:EVAL-IN-CONTEXT :AROUND (T T)) NIL #<PARACHUTE:MULTIPLE-VALUE-COMPARISON-RESULT :UNKNOWN (is-values (values 1 1) (= 0) (= 2))>)
        3: ((METHOD PARACHUTE:EVAL-IN-CONTEXT (T PARACHUTE:MULTIPLE-VALUE-RESULT)) NIL #<PARACHUTE:MULTIPLE-VALUE-COMPARISON-RESULT :UNKNOWN (is-values (values 1 1) (= 0) (= 2))>)
        3: (METHOD EVAL-IN-CONTEXT (T MULTIPLE-VALUE-RESULT)) returned (1 1)
      2: (METHOD EVAL-IN-CONTEXT :AROUND (T T)) returned (1 1)
      2: (PARACHUTE:STATUS #<PARACHUTE:MULTIPLE-VALUE-COMPARISON-RESULT :UNKNOWN (is-values (values 1 1) (= 0) (= 2))>)
      2: STATUS returned :UNKNOWN
    1: (METHOD EVAL-IN-CONTEXT :AROUND (T RESULT)) returned :PASSED
  0: EVAL-IN-CONTEXT returned :PASSED

I see that multiple-value-comparison-result inherits from both multiple-value-result and comparison-result, however, it appears that the eval-in-context implementation for comparison-result is not getting called on multiple-value-comparison-result instances, contrary to what one might think by seeing a specialization on value-expected-p for multiple-value-comparison-result.

commented

Thank you very much for the detailed report, your issues should be fixed now.