specs2


the remix, now complete with user Q & A


What's new in the Scala BDD world?

http://specs2.org






Rose Toomey
ny-scala @ 24 May 2011
Revised 6 June with assistance from Eric Torreborre

Been here before? Your key to the remix


The existing presentation has been marked up for your convenience:

This is new content.

This content was wrong. So please delete it from your memory banks.

This is a correction.

This is a clarification.

This is what Eric Torreborre said.

specs2: State of the art executable software specifications

  • The evolution of specs2

    • Availability
  • Field guide to specs2

    • acceptance specs
    • unit specs
    • Explanation of thrown expectations
  • Migrating to specs2

    • Test case: migrating Salat
    • Configuring specs2
  • Cool new features of specs2

    • JSON matchers
    • Specs2 matchers in the wild
    • Contexts
    • Scalacheck
  • Online Resources
  • User Q & A

The evolution of specs2

specs is a DSL in Scala for doing BDD (Behaviour-Driven Development).

specs2 is a complete rewrite of specs 1.x.

The design principles of specs2

  1. Do not use mutable variables
  2. Use a simple structure
  3. Control the dependencies (no cycles)
  4. Control the scope of implicits


Eric Torreborre, the creator of specs and specs2, will be giving a presentation on the design philosophy of specs2 at Functional Programming Sydney in July.

Follow @specs2.org on Twitter for details!

Availability

specs2

Eric:

specs2 will not be available for Scala 2.7.7 because it depends on named parameters.

I have to maintain a separate branch for 2.8.x and 2.9.x so SNAPSHOTs will be more frequent on 2.9.0-1.


specs2 is available for Scala 2.8.0, 2.8.1, 2.9.0 and 2.9.0-1.

specs

specs 1.6.x is available for Scala 2.7.7, 2.8.1 and 2.9.0 at scala-tools.

Field Guide to specs2

Unit specifications

  • Extend the org.specs2.mutable.Specification trait
  • are mutable
  • Use should / in format

    • in creates an Example object containing a Result
    • should creates a group of Example objects

Acceptance specs

  • Extend the org.specs2.Specification trait
  • are functional when extending the default org.specs2.Specification trait
  • Must define a method called is that takes a Fragments object, which is composed of:

    • an optional SpecStart
    • a list of Fragment objects
    • an options SpecEnd

Both types of specifications contain a list of specification fragments provided by the is method in the SpecificationStructure trait.

What is a specification fragment?

  • Simple text

    • to describe the test case
  • Examples

    • a description and executable code that returns a Result, such as

      • a standard result (success, failure)
      • a matcher result
      • a boolean value
  • Steps and actions that return success or failure

    • reported only if an exception occurs
  • SpecStart and SpecEnd delimiters in acceptance specs
  • tagging fragments to define which fragments should be included or excluded
  • formatting fragments

    • such as line breaks and tabs to make the test output pleasing to the eye

Execution

specs2 executes examples concurrently by default. You have to explicitly specify when you need sequential execution.

Fragments are sorted in groups so that all the elements of the group can be executed concurrently.

As each group of Example fragments runs concurrently, each Result is collected in a sequence of ExecutedFragments, which are then reduced for reporting.

Step can be used to break up the sequences in order to do some intitialisation or cleanup.

What is a result?

An instance of org.specs2.executable.Result contains:

  • a message describing the outcome
  • a message describing the expectation(s)

StandardResults indicate the result of executing an example:

  • success
  • failure
  • anError
  • pending
  • skipped

A MatcherResult contains the outcome of an expectation, such as

"hello" must contain("lo")    // success
"hello" must contain("lz")    // failure

Creating a unit specification

package prasinous.unit

import org.specs2.mutable._
import prasinous._

class TrivialUnitSpec extends Specification {

  "String reverser" should {
    "reverse String" in {
      StringReverser("hello") must_== "olleh"
      StringReverser("") must beEmpty
      StringReverser(null) must beNull
    }
  }

  "Option string reverser" should  {
    "reverse Option[String]" in {
      OptionStringReverser(Some("hello")) must beSome("olleh")
      // it's in the spec, that means i expected it, right?
      OptionStringReverser(Some(null)) must beSome(null)
      OptionStringReverser(None) must beNone
    }
  }

}

Running a unit specification

Fire up sbt and run

> test-only prasinous.unit.TrivialUnitSpec

[info] == prasinous.unit.TrivialUnitSpec ==
[info] String reverser should
[info] + reverse String
[info]
[info] Option string reverser should
[info] + reverse Option[String]
[info]
[info]
[info] Total for specification TrivialUnitSpec
[info] Finished in 72 ms
[info] 2 examples, 0 failure, 0 error
[info]
[info] == prasinous.unit.TrivialUnitSpec ==

Creating an acceptance specification

package prasinous.acceptance

import org.specs2._
import prasinous._

class TrivialAcceptanceSpec extends Specification { def is =

  "This is a specification to check reversing Strings"      ^
                                                            p^
  "StringReverser should"                                   ^
    "reverse a String"                                      ! e1 ^
    "leave an empty String unaffected"                      ! e2 ^
    "not fall down snivelling when someone feeds it null"   ! e3 ^
                                                            p^
  "OptionStringReverser should"                             ^
    "reverse Option[String]"                                ! e4 ^
    "reverse None"                                          ! e5 ^
    "Damn it, will someone please fix the universe?"        ! e6

  def e1 = StringReverser("hello") must_== "olleh"
  def e2 = StringReverser("") must beEmpty
  def e3 = StringReverser(null) must beNull
  def e4 = OptionStringReverser(Some("hello")) must beSome("olleh")
  def e5 = OptionStringReverser(None) must beNone
  def e6 = OptionStringReverser(Some(null)) must beSome(null)
}

Acceptance specification syntax

def is kicks it off -

class TrivialAcceptanceSpec extends Specification { def is =

^ glues things together. p is a FormattingFragment.

 "This is a specification to check reversing Strings"      ^
                                                           p^
 "StringReverser should"                                   ^

"description" ! body creates an Example where body is a method that returns a Result.

"reverse a String"                                         ! e1

where

def e1 = StringReverser("hello") must_== "olleh"

returns MatchResult[Any]

Formatting

The specs2 user guide contains a detailed Layout section explaining how acceptance specifications are formatted.

Q: Will anything aid you with keeping acceptance specification syntax neatly lined up?

A: Alas, not yet.

Keep an eye on the mailing list and Scalariform - someone will surely do something soon.

Running an acceptance spec

Fire up sbt and run

> test-only prasinous.acceptance.TrivialAcceptanceSpec

[info] == prasinous.acceptance.TrivialAcceptanceSpec ==
[info] This is a specification to check reversing Strings
[info]
[info] StringReverser should
[info] + reverse a String
[info] + leave an empty String unaffected
[info] + not fall down snivelling when someone feeds it null
[info]
[info] OptionStringReverser should
[info] + reverse Option[String]
[info] + reverse None
[info] + Damn it, will someone please fix the universe?
[info]
[info] Total for specification TrivialAcceptanceSpec
[info] Finished in 228 ms
[info] 6 examples, 0 failure, 0 error
[info]

Acceptance specs are functional by default

package prasinous.acceptance

import org.specs2._

class FunctionalDemoSpec extends Specification { def is =

  "My functional spec demo should"                          ^
    "show that only the last Result is returned"            ! e1 ^
    "fail as expected when results are chained together"    ! e2

  def e1 = {
    1 must beGreaterThan(9999)  // this MatchResult is discarded
    1 must beLessThanOrEqualTo(1)
  }

  // this fails as expected
  def e2 = 1 must beGreaterThan(9999) and beLessThanOrEqualTo(1)

}

Yields:

[info] == prasinous.acceptance.FunctionalDemoSpec ==
[info] My functional spec demo should
[info] + show that only the last Result is returned
[error] x fail as expected when results are chained together
[error]     1 is less than 9999 (FunctionalDemoSpec.scala:10)
[info]

Mutable specs are not

class NotFunctionalDemoSpec extends org.specs2.mutable.Specification {
  "Mutable specs" should {
    "fail on any bad expectation" in {
      e1   // fails here
      e2   // this is bad too, but we never get here until we fix e1
    }
    "fail on chained bad expectations too" in {
      e3
    }
  }

  def e1 = {
    1 must beGreaterThan(9999) // this MatchResult is NOT discarded
    1 must beLessThanOrEqualTo(1)
  }
  def e2 = {
    1 must beGreaterThan(8888)  // this would fail but we never get here
  }
  def e3 = 1 must beGreaterThan(9999) and beLessThanOrEqualTo(1)
}

Yields:

[info] == prasinous.unit.NotFunctionalDemoSpec ==
[info] Mutable specs should
[error] x fail on any bad expectation
[error]     1 is less than 9999 (NotFunctionalDemoSpec.scala:8)
[error] x fail on chained bad expectations too
[error]     1 is less than 9999 (NotFunctionalDemoSpec.scala:12)

Thrown expectations: a big difference

Eric: If you mix ThrownExpectations to an Acceptance Spec it will change the behavior so that any matcher failing will stop the execution of an example.

Acceptance specs matcher behaviour is to return an Expectable which handles applying the matcher and returning a Result.

Unit specs throw expectations as soon as they fail.

If you want acceptance specs to throw expectations like unit specs, mix in ThrownExpectations.

It doesn't change the functional behaviour of acceptance specs (i.e. only the returned ``Result`` will affect the final outcome), but it's useful for interfacing with other test frameworks.

My spec exposed a bug which was promptly fixed in specs2 1.4, which has just been released.

Acceptance spec with thrown expectations mixed in

class ThrownExceptionsDemo extends Specification with ThrownExpectations { def is =

  "My functional spec demo should"                          ^
    "show that only the last Result is returned"            ! e1 ^
    "fail as expected when results are chained together"    ! e2

  def e1 = {
    1 must beGreaterThan(9999)  // fails because ThrownExpectations was mixed in
    1 must beLessThanOrEqualTo(1)
  }

  // fails because first assumption in chain is bad
  def e2 = 1 must beGreaterThan(9999) and beLessThanOrEqualTo(1)
}

Fails just like a mutable spec because ThrownExpectations is mixed in:

[info] == prasinous.acceptance.ThrownExceptionsDemo ==
[info] My functional spec demo should
[error] x show that only the last Result is returned
[error]     1 is less than 9999 (ThrownExceptionsDemo.scala:9)
[error] x fail as expected when results are chained together
[error]     1 is less than 9999 (ThrownExceptionsDemo.scala:10)

Migrating to specs2

Eric: The best way to introduce concurrency into examples is to isolate the mutable variables.

NEW Eric posted about specs2 migration on his blog.

  • unit specs - the path of least resistance
  • acceptance specs - requires complete restructuring, but in exchange for substantial benefits

Migrating to specs2

Use case: migrating my own project

On 9 March, I migrated Salat from specs 1.6.7 to specs2 1.0.1. It took about two hours, and it was easy.

My strategy was as follows:

  • update the specs2 dependencies
  • switch my base testing trait, SalatSpec from using org.specs.Specification to using org.specs2.mutable.Specification
  • address minor syntax changes to the matcher syntax
  • handle cases where I needed to do something before and/or after a unit test

Updating the dependencies

Drop in place and go:

val specs2 = "org.specs2" %% "specs2" % "1.4" % "test"
val scalaz = "org.specs2" %% "specs2-scalaz-core" % "6.0.RC2" % "test"

def specs2Framework = new TestFramework("org.specs2.runner.SpecsFramework")
override def testFrameworks = super.testFrameworks ++ Seq(specs2Framework)

val snapshots = "snapshots" at "http://scala-tools.org/repo-snapshots"
val releases  = "releases" at "http://scala-tools.org/repo-releases"

Migrating to specs2: restructuring my test trait

Before

trait SalatSpec extends Specification with PendingUntilFixed with Logging {
  val SalatSpecDb = "test_salat"
  detailedDiffs()
  doBeforeSpec {
    com.mongodb.casbah.commons.conversions.scala.RegisterConversionHelpers()
    com.mongodb.casbah.commons.conversions.scala.RegisterJodaTimeConversionHelpers()
  }
  doAfterSpec {
    MongoConnection().dropDatabase(SalatSpecDb)
  }

}

Migrating to specs2: restructuring my test trait

The preferred way to do this should be using map as shown in Generic specification with setup and teardown steps.

After (the way I originally did it)

trait SalatSpec extends Specification with Logging {
  val SalatSpecDb = "test_salat"
  override def is =
    Step {
      com.mongodb.casbah.commons.conversions.scala.RegisterConversionHelpers()
      com.mongodb.casbah.commons.conversions.scala.RegisterJodaTimeConversionHelpers()
    } ^
      super.is ^
      Step {
        MongoConnection().dropDatabase(SalatSpecDb)
      }

}

After (now using map)

trait SalatSpec extends Specification with Logging {
   override def map(fs: =>Fragments) = Step {
     com.mongodb.casbah.commons.conversions.scala.RegisterConversionHelpers()
     com.mongodb.casbah.commons.conversions.scala.RegisterJodaTimeConversionHelpers()
  } ^ fs ^ Step {
    MongoConnection().dropDatabase(SalatSpecDb)
  }
}

Migrating to specs2: restructuring my test trait

  • No more detailedDiffs() - the default settings were good enough:
  • No more PendingUntilFixed - in specs2 this is now part of the common specification features

Setup and teardown (before using map)

  • doBeforeSpec has been replaced by overriding is with a Step to register Casbah's conversion helpers
  • doAfterSpec has been replaced by using ^ to glue a final step onto the supertrait's is method

Setup and teardown (using map)

doBeforeSpec and doAfterSpec have been replaced by using map to clearly define that a Step occurs before my Fragments and after.

Migrating to specs2: changes to matchers

The following matcher forms are now preferred:

a must matcher(b)
a must not matcher(c)

Before

"My test string" must notContain("bingo")
dbo must notHaveKey("aa")

After

"My test string" must not contain("bingo")
dbo must not have key("aa")

Migrating to specs2: making things run sequentially

For the most part, the specs in Salat can run concurrently.

However, some examples for SalatDAO required sequential access to shared mutable state in a single MongoDB collection.

class SalatDAOSpec extends SalatSpec {

  // which most specs can execute concurrently, this particular spec needs to execute sequentially
  // to avoid mutating shared state: namely, the MongoDB collection referenced by the AlphaDAO

  override def is = args(sequential = true) ^ super.is
You can use just sequential as a shortcut for args(sequential = true) because it's frequently used.

Migrating to specs2: using scopes to set up data

Unit specs have Scope, a simple way of creating a new scope with variables that can be re-used in any example.

I used it to isolate the tedium of data setup so that my examples could focus on what I was really trying to achieve.

trait xiScope extends Scope {
  log.debug("before: dropping %s", XiDAO.collection.getFullName())
  XiDAO.collection.drop()
  XiDAO.collection.count must_== 0L

  val xi1 = Xi(x = "x1", y = Some("y1"))
  val xi2 = Xi(x = "x2", y = Some("y2"))
  val xi3 = Xi(x = "x3", y = Some("y3"))
  val xi4 = Xi(x = "x4", y = Some("y4"))
  val xi5 = Xi(x = "x5", y = None)
  val _ids = XiDAO.insert(xi1, xi2, xi3, xi4, xi5)
  _ids must contain(Option(xi1.id), Option(xi2.id), Option(xi3.id), Option(xi4.id), Option(xi5.id))
  XiDAO.collection.count must_== 5L
}

Migrating to specs2: using a scope

My new xiScope can be used as easily as:

"support using a projection on an Option field to filter out Nones" in new xiScope {
  // a projection on a findOne that matches xi1
  XiDAO.primitiveProjection[String](MongoDBObject("x" -> "x1"), "y") must beSome("y1")
  // a projection on a findOne that brings nothing back
  XiDAO.primitiveProjection[String](MongoDBObject("x" -> "x99"), "y") must beNone

  val projList = XiDAO.primitiveProjections[String](MongoDBObject(), "y")
  projList must haveSize(4)
  projList must contain("y1", "y2", "y3", "y4") // xi5 has a null value for y, not in the list
}

Control execution and reporting

Use arguments. It's that easy.

Inside a spec, pass them in to is. For instance, let's say you are working inside a web framework and you want to filter stacktraces to show only your own code.

def is = args(traceFilter = includeTrace("com.foo.confabulator"))

In sbt, you can pass in arguments: the example shown below will output to both console and html.

> test-only com.foo.confabulator.test.TryHarderSpec -- html console

JUnit Integration

class WithJUnitSpec extends SpecificationWithJUnit {
  "My spec" should {
    "run in JUnit too" in {
      success
    }
  }
}

For IDE support, you can still use @Runner:

import org.junit.runner._
import runner._

@RunWith(classOf[JUnitRunner])
class WithJUnitSpec extends Specification {
  "My spec" should {
    "run in JUnit too" in {
      success
    }
  }
}

Matchers: making expectations easy

In specs2, you can define expectations on anything that returns a Result.

  • Boolean
  • Standard Results

    • success, failure, anError, pending, etc.
  • Matcher result - specs2 has built in support for all these and more:

    • Any
    • Option / Either
    • Strings and Numbers
    • Exceptions
    • Iterable and Maps
    • XML and JSON
    • Scalaz
    • Parser Combinator matchers
  • ScalaCheck property
  • Mock expectation
  • DataTable
  • Forms

Where to find information about matchers

Iterable matchers

specs 1.x:

val list = List(1, 2, 3)
list must have size(3)
list must containInOrder(1, 2, 3)

specs2

Using only and inOrder we can state this in one shot:

List(1, 2, 3) must contain(1, 2, 3).only.inOrder

JSON matchers

The JSON matchers rely on the standard Scala libs.

/(value) looks for a value at the root of an Array

"""["name", "Joe" ]""" must /("name")

/(key -> value) looks for a pair at the root of a Map

"""{ "name": "Joe" }""" must /("name" -> "Joe")
"""{ "name": "Joe" }""" must not /("name2" -> "Joe")

*/(value) looks for a value present anywhere in a document, either as an entry in an Array or as the value for a key in a Map

*/(key -> value) looks for a pair anywhere in the document

JSON matchers can be chained:

"""{ "person": { "name": "Joe" } }""" must /("person") /("name" -> "Joe")


See JsonMatchersSpec in the specs2 project specs for more details.

specs2 matchers in the wild

Contexts: making things happen when you need them

The specs2 user guide has been updated to provide increased coverage on how to use Contexts - go see that immediately.
Also, see this helpful gist showing how to use implicit contexts to reduce duplication.

When you want to make sure that something happens for every example, use any combination of these traits:

  • Before
  • After
  • Around
  • Outside

Contexts provide an apply method which can be applied to the body of an example so that your code is executed when you need it relative to the example code.

Contexts of the same type can be composed and/or sequenced. Contexts can also extend each other to provide more specific setups.

Context now extends Scope. Whereas Scope brings what's inside into context with a mutable spec, a Context is about having the appropriate method being executed when and where you need it (http connection, database operations).

Demonstration of Around (unit spec)

Courtesy of a gist from Eric Torreborre. >> appends an Example to the unit spec's list of fragments.

class UnitSpec extends org.specs2.mutable.Specification {

  "This specification has examples which must be executed inside an http session" >> {
    "Example 1 is executed inside the session" >> http {
      success
    }
    "Example 2 is also executed inside the session" >> http {
      success
    }
  }

  object http extends Around {
    def around[T <% Result](t: =>T) = openHttpSession("test") {
      t // execute t inside a http session
    }
  }
}

Eric also has a gist sketching out how to use Outside and AroundOutside.

This old spec: specs 1.x

There's always something special in a dusty corner... Now it sees the light of day.

class BadIdea extends Specification {
  var badIdea = MSet.empty[Int]
  "My badly set up spec" should {
    shareVariables()  // now we're in for it...
    doFirst {
      println("setUp(): hey, who got rid of JUnit?")
      badIdea = MHashSet.empty[Int] ++= (1 to 100).toSet
      badIdea must notBeEmpty
    }
    "make use of shared mutable state" in {
      badIdea ++= (101 to 200).toSet
      badIdea must have size (200)
    }
    "have expectations based on previous sequential manipulation of shared mutable state" in {
      badIdea = badIdea.filter(_ % 2 == 0)
      badIdea must have size (100)
    }
    // MOAR...
    doLast {
      println("tearDown(): success!")
      badIdea.clear()
    }
  }
}

This old spec: hauled into specs2 with complete violation of intent

import scala.collection.mutable.{Set => MSet, HashSet => MHashSet}

class BadIdea extends org.specs2.mutable.Specification {
  var badIdea = MSet.empty[Int]

  override def is = args(sequential=true)^  Step {
      println("Doing something beforehand!")
      badIdea = MHashSet.empty[Int] ++= (1 to 100).toSet
      badIdea must not be empty
    } ^
      super.is ^ Step {
          println("Doing something afterwards!")
          badIdea.clear()
        }

  "My badly set up spec" should {
    "make use of shared mutable state" in {
      badIdea ++= (101 to 200).toSet
      badIdea must have size(200)
    }
    "have expectations based on previous sequential manipulation of shared mutable state" in {
      badIdea = badIdea.filter(_ % 2 == 0)
      badIdea must have size(100)
    }
    // Send help...
  }
}

This old spec: renovated

The simplest way to equip your examples with the state they expect: use Scope to create a reusable state sandbox and rebase the existing expectations to work with the new setup.

class BetterIdea extends Specification {

  // happens BEFORE each use case
  trait testData extends Scope {
    println("Setting up data JUST for you and your little dog, my pretty.")
    val betterIdea = (1 to 100).toSet
    betterIdea must have size(100)
  }

  "My better spec using contexts" should {
    "force each use case to have its own immutable data" in new testData {
      val newSet = betterIdea ++ (101 to 200).toSet
      newSet must have size(200)
    }
    "get rid of expectations that depend on shared state" in new testData {
      val anotherNewSet = betterIdea.filter(_ % 2 == 0)
      anotherNewSet must have size(50)  // changed expectation no longer depends on shared state!
    }
  }
}

This old spec: what about context instead of scope?

Now, imagine instead of a set, we were using shared mutable state like a database table.

Since I didn't want to introduce deps into my test code, pretend Before is actually populating a database table with known test data. Now using a Context instead of a Spec makes sense.

object setupData extends Before {
  var betterIdea = Set.empty[Int]
  def before {
    betterIdea ++= (1 to 100).toSet
    betterIdea must have size(100)
  }
}

"or do it with a context instead" >> {
  "now my example will be executed inside the setupData context" >> setupData {
    // sub in database access for "setupData.betterIdea"
    val newSet = setupData.betterIdea ++ (101 to 200).toSet
    newSet must have size (200)
  }
}

Given - When - Then

In specs 1.x, given, when and then were methods with no added value but displaying these words with a description.

In specs2, Given, When and Then are now RegexSteps that can extract typed values from parameterized text:

class GivenWhenThenSpec extends org.specs2.Specification { def is =
  "A given-when-then example for GCD"                   ^
    "Given the following number: ${4}"                  ^ number1 ^
    "And the following number: ${2}"                    ^ number2 ^
    "Then the greatest common denominator is ${2}"      ^ result ^
                                                        end
  object number1 extends Given[Long] {
    def extract(text: String): Long = extract1(text).toLong
  }
  object number2 extends When[Long, BinaryGCD] {
    def extract(number1: Long, text: String) =
      BinaryGCD(number1, extract1(text).toLong)
  }
  object result extends Then[BinaryGCD] {
    def extract(b: BinaryGCD, text: String): Result =
      b.gcd must_== extract1(text).toLong
  }
  case class BinaryGCD(u: Long, v: Long) {
    def gcd: Long = { /* Some impl */ }
  }
}

Using ScalaCheck to reduce drudgery

For all the ceremony in the previous GivenWhenThenSpec, it's dreadful:

  • it tests only a single case
  • we have to supply all the inputs and expectations by hand. Yawn.
  • how long would it take us to cover a halfway decent set of data?

Generating test case data by hand is a drag. ScalaCheck to the rescue!

Key concepts

  • a property is a testable unit that specifies the behaviour of a method

    • properties created from functions require an implciit Arbitrary[T] instance
  • generators are responsible for generating data in ScalaCheck

Consult the ScalaCheck User Guide for a detailed explanation of property and generator types.

specs2 support for ScalaCheck

specs2 comes with built-in support for ScalaCheck.

  • extend org.specs2.Specification with org.specs2.ScalaCheck
  • the ScalaCheck trait provides a check function that transforms a function to a org.scalacheck.Prop and then to a Result
  • import step classes (Given, When, etc) from org.specs2.specification.gen instead of org.specs2.specification
  • Given and When steps now return ScalaCheck generators
  • the extract method on Then takes an implicit Arbitrary[T]

More examples

See JsonSpec for an example of using ScalaCheck to generate JSON.

Using ScalaCheck

class ScalaCheckGwtSpec extends Specification with ScalaCheck { def is =
    "Testing Binary GCD calculator"                     ^
      "Given the following number n1"                   ^ number1 ^
      "And the following number n2"                     ^ number2 ^
      "When we take the greatest common denominator"    ^ gcd ^
      "Then the binary GCD matches the Euclidian GCS"   ^ result ^
                                                        end
  object number1 extends Given[Long] {
    def extract(text: String) = choose(-10L, 10L)
  }
  object number2 extends When[Long, (Long, Long)] {
    def extract(number1: Long, text: String) =
      for { n2 <- choose(-10L, 10L) } yield (number1, n2)
  }
  object gcd extends When[(Long, Long), BinaryGCD] {
    def extract(numbers: (Long, Long), text: String) =
      BinaryGCD(numbers._1, numbers._2)
  }
  object result extends Then[BinaryGCD] {
    def extract(text: String)(implicit op: Arbitrary[BinaryGCD]) = {
      check { (op: BinaryGCD) => op.gcd must_== EuclidianGCD(op.u, op.v) }
    }
  }
}

Running ScalaCheck

Using all the permutations of data generated by choose, we make the anemic GivenWhenThenSpec far more robust by running our expectation 100 times and checking our Binary GCD algorithm against the results of an indepdendent Euclidian GCD algorithm.

[info] == prasinous.acceptance.ScalaCheckGwtSpec ==
[info] Testing Binary GCD calculator
[info] Given the following number n1
[info] And the following number n2
[info] When we take the greatest common denominator
[info] + Then the binary GCD matches the Euclidian GCS
[info]
[info] Total for specification ScalaCheckGwtSpec
[info] Finished in 59 ms
[info] 1 example, 100 expectations, 0 failure, 0 error
[info]
[info] == prasinous.acceptance.ScalaCheckGwtSpec ==

And if we decide to run the expectation 10,000 times, all we have to do is change some inputs to choose.

Could you use Given - When - Then with unit specs?

Eric: I'm not actually sure how feasible this is actually because the GivenWhenThen steps in an Acceptance spec are very typechecked. In a mutable spec I would need to do runtime checks and have additional variables.

If there were suport for it, Eric suggests it might look like this:

class ScalaCheckGwtUnitSpec extends Specification with ScalaCheck {
  "Testing Binary GCD calculator" {
    "Given the following number n1" ! given { choose(-10L, 10L) }
    "And the following number n2"   ! when { (number1: Long, text: String) =>
       for {n2 <- choose(-10L, 10L)} yield (number1, n2)
    }
    "When we take the greatest common denominator" ! when { (numbers: (Long, Long), text: String) =>
      BinaryGCD(numbers._1, numbers._2)
    }
    "Then the binary GCD matches the Euclidian GCS " ! then check { (op: BinaryGCD) =>
      op.gcd must_== EuclidianGCD(op.u, op.v)
    }
  }
}





User Q & A

Question

Can specs2 be used with ScalaTest? and for what purpose?

Answer

Eric:

The specs2 matchers should be usable in ScalaTest (and reciprocally). You could use specs2 Json matchers for example while keeping the ScalaTest reporters and infrastructure.

Possibly the ScalaCheck API or the DataTables as well but I haven't tried it.

Those objects traits can also be reused in JUnit and TestNG modulo some adapation of the ThrownExpectations trait. See for example what is done here: JUnitMatchers.scala

Question

Can use cases written by a business user be parsed into a sample acceptance spec with default methods e1, e2, etc. populated?

Answer

Eric:

Possibly, yes. However my view on the subject is that business users can not conveniently do this. Note that the big difficulty is not so much in writing the first spec but more in maintaining the whole lot.

The long-term answer is that I would like to write a GUI client to support this case: with version control integration, iterative development,...

Actually ThoughtWorks is selling a tool that makes most of this:
http://www.thoughtworks-studios.com/agile-test-automation

Question

Did Given - When - Then come over from Ruby/Cuke?

Answer

Eric:

Absolutely. My take on it is:
* you don't have to use the given-when-then keywords but you can use whatever text you feel is natural
* the G-W-T sequence is statically typed: well as much as possible, it is still possible to fail when extracting values from the text

Question

What else can Given - When - Then be used for?

Answer

Eric:

I just see it as a guideline to write clear specifications along the "Arrange-act-assert" paradigm:
http://c2.com/cgi/wiki?ArrangeActAssert

Question

Is it possible to define the body inline when using "description" ! body format?

Answer

Eric: Yes.

class InlineFunctionalDemoSpec extends Specification { def is =

  "My functional spec demo should"                          ^
    "show that only the last Result is returned"            ! {
      1 must beGreaterThan(9999)  // this MatchResult is discarded
      1 must beLessThanOrEqualTo(1) } ^
    "fail as expected when results are chained together"    ! { 1 must beGreaterThan(9999) and beLessThanOrEqualTo(1) }

}

Question

What is AutoExample for?

Answer

Eric:

To avoid repetition between the example description and the example code when the code says it all. For example I used that a lot to specify matchers:

https://github.com/etorreborre/specs2/blob/1.3/src/test/scala/org/specs2/matcher/AnyMatchersSpec.scala

It's also used to enable the "backtick" notation: http://etorreborre.github.com/specs2/guide/org.specs2.guide.SpecStructure.html#Acceptance+specification (see "you can even push this idea further by writing:").

Question

What's the rationale behind acceptance specs? Isn't the way unit specs mingle text and code a benefit?

Answer

Eric:

My own rationale is the following. By being able to read the whole spec text with one glance I can better think about what I expect from my system. So when I implement something new, I usually spend some time adding the examples which make sense before implementing them. I also found my specs easier to understand when revisiting them after a few weeks.

Discussion

What is the actual process of putting together an acceptance spec?

Writing process

You might begin by just stubbing out expectations with inline results:

class AcceptanceProcessDraft extends Specification { def is =
  "first example" ! pending ^
  "second example" ! pending
}

Once the examples are clear, extract the results out into methods and begin populating them:

class AcceptanceProcessSecondDraft extends Specification { def is =
  "first example"    ! e1 ^
  "second example"   ! e2 ^
  "third example"    ! e3

  def e1 = pending
  def e2 = pending
  def e3 = pending
}

Discussion

What is the actual process of putting together an acceptance spec?

Notes from Eric

There have been numerous discussion of acceptance spec style and purpose on the specs2 mailing list.

Eric:

One thing I noticed is that I often split examples in two:
"this should do that" ! e1^
becomes


  "this should do that"  ^
     "in this case"         ! e1^
     "in that case"         ! e2^

Discussion

Compare and contrast specs/specs2 with ScalaTest.

My personal experience

Speaking personally, I started out with ScalaTest at work but quickly found specs to be a better tool for me.

Although ScalaTest appeared more direct to me at first, coming from a JUnit/TestNG background, over a period of two months I became dissatisfied:

  • the errors when tests failed were difficult to work with. When I was just starting out in Scala, trying to figure out where in an N-deep level nest of anonymous inner classes some expectation failed was very daunting to me.
  • ScalaTest dynamic matchers seemed initially appealing when I wanted to check values within an object graph but turned into a maintenance nightmare when I refactored model objects.

    I saw that some open source projects I liked were using specs, so I gave it a try. I appreciated the clean syntax of specs, and I liked the matcher syntax more than ScalaTest's.

    In simple testing setups ScalaTest and specs appear quite similar, so my transition from ScalaTest to specs was quite rapid. When my test cases failed, the errors made it extremely easy to target the failing lines.

Discussion

Compare and contrast specs/specs2 with ScalaTest.

My personal experience, continued

Although I came to specs for its simplicity, in the long run what kept me using specs was its power. So it was a natural for me to migrate Salat to specs2 when it was released. specs2, although a complete rewrite, had everything I liked about specs 1.x plus many new powerful features I could use to isolate mutability in my specs.

That said, Akka uses ScalaTest, and I quite admire the style and thoroughness of their tests as a model for anyone who wants to use ScalaTest.

There have been numerous discussions online. Here's a good StackOverflow thread with a comparison from Bill Venners:

Online resources



Sample code

Useful things to think about

Thank you

  • @etorreborre for writing and beautifully documenting specs2, and so much more:

    • for assisting at every stage in the preparation and remix of this presentation
    • for clarifying the user guide in response to some of the questions from this presentation
    • for being so responsive at more hours of the day and night than I could possibly expect (especially given the time difference between New York and Sydney - it's clear he's someone who cares a lot about helping people to understand and use specs2)
  • @softprops for picture-show
  • Novus Partners for hosting this ny-scala meetup
  • @n8han for filming this talk
  • all of the attendees at the presentation for displaying such interest and asking so many questions!
  • everyone who has re-tweeted this presentation, for helping to create a vital and interesting discussion around BDD testing frameworks in Scala