Allow tests to be marked ignored or pending

[Richard Gomes]

This issue is about providing simple functionalities for ignored and pending test cases.

I would like to contribute to uTest but I'm not sure if my suggestion would qualify to the skimmed down nature of uTest. I'm copying/pasting the overall idea here before submitting a PR, but only if I get approval from the project administrator. Otherwise, I will create a separate project. So, please let me know if my suggestion qualifies.

This is the overall idea:

1. You mark test cases the same way as you would do in ScalaTest:

"This is an ignored test case" - ignored {
   // this is the test code
   assert(false)
}

"This test case is ignored when running by Jenkins" - ignoredif(jenkins) {
   // this test has requirements which Jenkins is not going to fulfill, for whatever reason
   assert(false)
}

2. This is the barebones of how to provide the feature:

trait uTestActions {
  def ignored(action: ⇒ Unit): Unit = ignored
  def ignoredif(condition: Boolean)(action: ⇒ Unit): Unit = if(condition) ignored else action
  def pending(action: ⇒ Unit): Unit = {
    try {
      action
      pending("PENDING", new TestPendingException)
    } catch {
      case t: Throwable => pending("FAILED", new TestFailedException(t))
    }
  }

  private def ignored: Unit = {
    val t = new TestIgnoredException
    val it = t.getStackTrace.iterator
    val e = (1 to 5).map(_ => it.next).last
    report("IGNORED", e)
  }

  private def pending(state: String, t: Throwable): Unit = {
    val pos = t.getStackTrace.zipWithIndex.find(p => p._1.getMethodName == "pending").get._2
    val it = t.getStackTrace.iterator
    val e = (1 to pos+4).map(_ => it.next).last
    report(state, e)
  }

  private def report(state: String, e: StackTraceElement): Unit = {
    val _class  = e.getClassName
    val _method = e.getMethodName
    val _file   = e.getFileName
    val _line   = e.getLineNumber
    System.err.println(s"@@@@@@@@ ${state}:: ${_class}.${_method} at ${_file}:${_line}")
  }
}
class TestPendingException extends Exception
class TestIgnoredException extends Exception
class TestFailedException(cause: Throwable) extends Exception(cause)

[Dave Smith]

+1
I was about to write a request for this myself, utest is brilliant. The ability to mark a test as pending is the one-and-only thing I miss from ScalaTest.

[David Barri]

It would be good to add "sole" or "only" too which is equivalent to adding ignore to all other tests without the sole/only declaration. Good so you can easily just mark a few tests that you want to focus on temporarily.

[Richard Gomes]

My implementation is here: https://github.com/frgomes/utest-extra

Yeah... I still need to publish onto Sonatype.

[Li Haoyi]

yeah i'd be happy for a way to tag tests with metadata that can be queried or used by the testrunner. This would be easy with the test("foo"){} syntax since we can stuff extra arguments into the test call