Synopsis

pyVows is a behavior driven development framework for Python.

pyVows is inspired by Vows, a BDD framework for node.js.

Much of what’s written here is based or the same as in Vows docs.

As its node.js counterpart does, pyVows executes your tests in parallel when it makes sense, and sequentially when there are dependencies.

Imagine we’re testing a method that sums two integers, like this:

    def test_sum_returns_42():
        result = add_two_numbers(41, 1)

        assert result
        assert int(result)
        assert result == 42
            

Even in a very simple scenario like this, we have three assertions in this test. Not too good if we want a single assertion per test. So, we could do it like this:

    def test_sum_returns_result():
        result = add_two_numbers(41, 1)
        assert result

    def test_sum_returns_a_number():
        result = add_two_numbers(41, 1)
        assert int(result)

    def test_sum_returns_42():
        result = add_two_numbers(41, 1)
        assert result == 42
            

Which is all fine and dandy, except that we are executing the add_two_numbers function three times. In this simple scenario, it might not matter if a function is executed many times. But with real code, we want to minimize calls, so our tests are always as fast as possible.

This is how the above test would look like in pyVows:

    class SumContext(Vows.Context):

        def topic(self):
            return add_two_numbers(41, 1)

        def we_get_a_result(self, topic):
            expect(topic).Not.to_be_null()

        def we_get_a_number(self, topic):
            expect(topic).to_be_numeric()

        def we_get_42(self, topic):
            expect(topic).to_equal(42)
            

Don’t worry if you don’t understand all of it. We’ll see it more thoroughly in the next sections.

Here’s another example, this time describing ‘division by zero’:

    # division_by_zero_vows.py

    from pyvows import Vows, expect

    # Create a Test Batch
    @Vows.batch
    class Divisions(Vows.Context):
        class WhenDividingANumberByZero(Vows.Context):
            def topic(self):
                return 42 / 0

            def we_get_division_by_zero_error(self, topic):
                expect(topic).to_be_an_error_like(ZeroDivisionError)

        class WhenDividingByOne(Vows.Context):
            def topic(self):
                return 42 / 1

            def we_get_the_same_number(self, topic):
                expect(topic).to_equal(42)

            

And run it:

$ pyvows division_by_zero_vows.py

---

Now let’s look at a more involved example. Suppose we have a module called the_good_things, with some fruit in it:

    class Strawberry(object):
        def __init__(self):
            self.color = '#ff0000';

        def isTasty(self):
            return True

    class PeeledBanana(object): pass

    class Banana(object):
        def __init__(self):
            self.color = '#fff333';

        def peel(self):
            return PeeledBanana()
            

Now write some tests in the_good_things_vows.py:

    from pyvows import Vows, expect

    from the_good_things import Strawberry, Banana, PeeledBanana

    @Vows.batch
    class TheGoodThings(Vows.Context):
        class AStrawberry(Vows.Context):
            def topic(self):
                return Strawberry()

            def is_red(self, topic):
                expect(topic.color).to_equal('#ff0000')

            def and_tasty(self, topic):
                expect(topic.isTasty()).to_be_true()

        class ABanana(Vows.Context):
            def topic(self):
                return Banana()

            class WhenPeeled(Vows.Context):
                def topic(self, banana):
                    return banana.peel()

                def returns_a_peeled_banana(self, topic):
                    expect(topic).to_be_instance_of(PeeledBanana)
            

And run the tests:

$ pyvows the_good_things_vows.py

Installing

The easiest way to install pyVows, is via pip, the Python package manager, like so:

$ pip install pyvows

Or to upgrade:

$ pip install --upgrade pyvows

Note: If you’re on a Debian-like system (Ubuntu) and pyvows fails to install, you may need to install these packages (as root):

# apt-get install libxslt-dev libxml2-dev libevent-dev

After those packages are installed, try installing pyvows again.

Guide

To understand pyVows, we’re going to start with a general overview of the different components involved in writing tests, and then go through some of them in more detail.

Structure of a test batch

Test batches in pyVows are the largest unit of tests. The convention is to have one test batch per file, and have the batch’s class match the file name. Test batches are created with @Vows.batch decorator.

    @Vows.batch
    class MyTestVows(Vows.Context):
        pass
            

Tests are added to suites in batches. This is done with the @Vows.batch decorator.

You can have as many batches in a suite as you want.

Batches are contexts, that can in itself contain contexts, which describe different components and states you want to test.

    @Vows.batch
    class AContext(Vows.Context):
        pass

    @Vows.batch
    class AnotherContext(Vows.Context):
        pass
            

Contexts are executed in parallel, and they are fully asynchronous. The order in which they finish is therefore undefined.

Contexts usually contain topics and vows, which in combination define your tests.

    @Vows.batch
    class AContext(Vows.Context):
        def topic(self):
            return "something"

        def i_am_a_vow(self, topic):
            # Test the results of the topic
            

Contexts can contain sub-contexts which get executed as soon as the parent context finishes:

    @Vows.batch
    class AContext(Vows.Context):
        def topic(self):
            return "something"

        def i_am_a_vow(self, topic):
            # Test the results of the topic

        class SubContext(Vows.Context):
            # Executed when AContext is done
            pass

    @Vows.batch
    class AnotherContext(Vows.Context):
        # Executed in Parallel to AContext
        pass
            

Summary

» A Suite is a set of one or more batches that pyVows will execute.

» A batch is a context, representing a structure of nested contexts.

» A context is a class with an optional topic, zero or more vows and zero or more sub-contexts.

» A topic is a function that returns a value.

» A vow is a function which receives the topic as an argument, and runs an assertion on it.

With that in mind, we can imagine the following grammar:

Suite   → Batch*
            Batch   → Context*
            Context → Topic? Vow* Context*
            

Here’s an annotated example:

    @Vows.batch                                             # Batch
    class Array(Vows.Context):                              # Context
        class AnArray(Vows.Context):                        # Sub-Context
            class WithThreeElements(Vows.Context):
                def topic(self):                            # Topic
                    return [1, 2, 3]

                def has_length_of_3(self, topic):           # Vow
                    expect(topic).to_length(3)              # Assertion

            class WithZeroElements(Vows.Context):           # Sub-Context
                def topic(self):                            # Topic
                    return []

                def has_a_length_of_0(self, topic):         # Vow
                    expect(topic).to_length(0)              # Assertion

                class WhenPopped(Vows.Context):
                    def topic(self, previous_topic):
                        return previous_topic.pop()

                    def raises_when_popped(self, topic):
                        expect(topic).to_be_an_error_like(IndexError)
            

How topics work

Vows introduces an incredibly powerful, yet very simple way of writing your tests. pyVows leverages the same approach towards the same goals.

Understanding topics is one of the keys to understanding pyVows. Unlike other testing frameworks, pyVows forces a clear separation between the element which is tested—the topic—and the tests themselves—the vows.

Let’s start with a simple example of a context:

    class Test42(Vows.Context):
        def topic(self):
            return 42

        def should_be_equal_to_42(self, topic):
            expect(topic).to_equal(42)
            

This demonstrates how the value of topic is passed down to our test function (referred to as a vow from now on) as an argument. Simple enough. Now what if we have multiple vows?

    @Vows.batch
    class Test42(Vows.Context):
        def topic(self):
            return 42

        def should_be_a_number(self, topic):
            expect(topic).to_be_numeric()

        def should_be_equal_to_42(self, topic):
            expect(topic).to_equal(42)
            

It works as expected; the value is passed down to each vow. Note that the topic function is only run once.

Scope

Sometimes, you’ll need a parent topic’s value, from inside a child topic. This is easy because of the notion of topic scope. Let’s look at an example:

    @Vows.batch
    class DataStore(Vows.Context):
        def topic(self):
            return DataStore()

        def should_respond_to_get(self, store):
            expect(store.get).to_be_a_function()

        def should_respond_to_put(self, store):
            expect(store.put).to_be_a_function()

        class CallingGet(Vows.Context):
            def topic(self, store):
                return store.get(42)

            def should_return_the_object_with_id_42(self, topic):
                expect(topic.id).toEqual(42)
            

In this example, the value of the top-level topic is passed as an argument to the nested topic, in the same manner in which it’s passed to vows. For clarity, I named both arguments which refer to the outer topic as store.

Note that the scoping isn’t limited to a single level. Consider:

            def topic(self, a, b, c):
                # a being the Parent topic
                # b being the Parent of parent topic
                # c being the Parent of parent of parent topic

                # return something
            

So the parent topics are passed along to each topic function in the certain order: the immediate parent is always the first argument (a), and the outer topics follow—b, then c, and so on—like the layers of an onion.

Running a suite

The simplest way to run a test suite is with the pyvows command:

pyvows vows/my_vows.py

The results will be printed to the console with the default reporter (which is very similar to Vows ‘dot-matrix’ reporter).

Running larger suites

When your tests become more complex, spanning multiple files, you’re going to need a way to run them as a single entity.

pyVows’ test runner can run multiple test suites at once. To use it, just pass the directory as the argument instead of the file, like this:

pyvows vows/

You can also pass options to pyvows. For example, to get pyvows to run only files that end in ‘_test’, just add the --pattern='*_test.py' option. The reference section has more information on the different options you can pass to it.

Order of execution and parallelism

We’ve already briefly covered how batches and contexts are executed. Now it’s now time to delve into more detail:

    @Vows.batch
    class ReadFile(Vows.Context):
        def topic(self):
            return exists('some_file.txt')

        class AfterSuccessfullyReading(Vows.Context):
            def topic(self, exists):
                if exists:
                    return open('some_file.txt').read()
                return None

            def if_exists_we_can_read_the_contents(self, topic):
                expect(topic).to_be_like('some string')
            

This example makes use of nested contexts. As you can see, the result of the parent topic is passed down to its children, as arguments.

This example is therefore, as a whole, mostly sequential—while remaining asynchronous.

---

Now let’s look at an example which uses parallel tests to check for some devices:

    @Vows.batch
    class Exists(Vows.Context):
        class StdOut(Vows.Context):
            def topic(self):
                return exists('/dev/stdout')

            def exists(self, topic):
                expect(topic).to_be_true()

        class Tty(Vows.Context):
            def topic(self):
                return exists('/dev/tty')

            def exists(self, topic):
                expect(topic).to_be_true()

        class DevNull(Vows.Context):
            def topic(self):
                return exists('/dev/null')

            def exists(self, topic):
                expect(topic).to_be_true()
            

In this case, the tests can finish in any order, and must not rely on each other. The test suite will exit when the last I/O call completes, and the assertions for it are run.

In other words: sibling contexts are executed in parallel, while nested contexts are executed sequentially. Remember—this all happens asynchronously, so while some contexts might be waiting for their parent context to finish, sibling contexts are executing in the meantime.

Context inheritance

pyVows context model allows for some interesting inheritance scenarios. Imagine we have the following method to test:

    def add(a, b):
        return a + b
            

You could write some vows like this:

    class Add(Vows.Context):
        def topic(self):
            return add(1, 2)

        def should_be_numeric(self, topic):
            expect(topic).to_be_numeric()

        def should_equal_to_three(self, topic):
            expect(topic).to_equal(3)
            

These might be redundant tests, but again, this is just an example.

Now, imagine we want to test for MANY different combinations. It would be very painful to write the exact same contexts and vows over and over, just to test different values. Let’s try writing the above a little differently:

    def addctx(a, b):
        class Context(Vows.Context):
            def topic(self):
                return add(a, b)

            def should_be_numeric(self, topic):
                expect(topic).to_be_numeric()

            def should_equal_to_three(self, topic):
                expect(topic).to_equal(a + b)

    class AddEquals3(addctx(1,2)):
        pass

    class AddEquals8(addctx(5,3)):
        pass

    class AddEquals10(addctx(6,4)):
        pass

    # and so on, and so forth
            

The above tests are taking advantage of the functional nature of Python, and returning a dynamic Context class as the result of addctx. This means every time addctx is called, it’s returning a different Context with awareness of the values a and b.

Context inheritance can also be a powerful tool to initialize database connections, web servers, and any other resources your vows might rely on. A pretty good example of this is the tornado pyvows project by Rafael Carício.

Using generative testing

Generative testing can be a powerful ally for having clean, lean tests. Let’s check back with our old friend, the add two numbers example:

    def add(a, b):
        return a + b

    class Add(Vows.Context):
        def topic(self):
            return add(1, 2)

        def should_be_numeric(self, topic):
            expect(topic).to_be_numeric()

        def should_equal_to_three(self, topic):
            expect(topic).to_equal(3)
            

Even though we could simplify this a lot with context inheritance (as demonstrated in the last section), we’re going to try something different this time. We’ll use generative testing.

Generative testing means having a test (or in our case, vows and contexts) executed one or more times, but with different arguments (topics) each pass.

Let’s rewrite the above using generative testing:

    def add(a, b):
        return a + b

    test_data = [
        (1, 2, 3),
        (2, 5, 7),
        (3, 4, 7),
        (5, 6, 11)
    ]

    class Add(Vows.Context):
        def topic(self):
            for item in test_data:
                a, b, c = item
                yield (add(a, b), c)

        def should_be_numeric(self, topic):
            sum, expected = topic
            expect(sum).to_be_numeric()

        def should_equal_to_expected(self, topic):
            sum, expected = topic
            expect(sum).to_equal(expected)
            

This means that should_be_numeric and should_equal_to_expected will be executed four times each, with a tuple of two items: the result of adding a and b, and the expected result.

Let’s go a step further, and check against even more scenarios:

     def add(a, b):
        return a + b

    a_samples = range(10)
    b_samples = range(10)

    class Add(Vows.Context):
        class ATopic(Vows.Context):
            def topic(self):
                for a in a_samples:
                    yield a

            class BTopic(Vows.Context):
                def topic(self, a):
                    for b in b_samples:
                        yield b

                class Sum(Vows.Context):
                    def topic(self, b, a):
                        yield (add(a, b), a + b)

                    def should_be_numeric(self, topic):
                        sum, expected = topic
                        expect(sum).to_be_numeric()

                    def should_equal_to_expected(self, topic):
                        sum, expected = topic
                        expect(sum).to_equal(expected)
            

This way, we’re testing the sum of all combinations of 0 to 9 plus 0 to 9, yet it is still very simple.

Now we can add as many scenarios as we can think of. And we won’t have to write a single new vow!

Asynchronous Topics

Imagine you need to test an asynchronous method called async_square. It takes a number and returns that number’s square—but it does so asynchronously.

Normally, this would be a problem. Your tests wouldn’t wait for the callback—or even pass a callback, for that matter. This is where pyvows comes to your rescue.

Testing asynchronous topics is as trivial as decorating the topic method with the Vows.async_topic decorator.

    def async_square(a, callback):
        callback(a * a, kwarg1=True) # This is async code, and could take a while

    class Square(Vows.Context):
        @Vows.async_topic
        def topic(self, callback):
            async_square(10, callback)

        def should_be_numeric(self, topic):
            expect(topic[0]).to_be_numeric()

        def should_equal_to_expected(self, topic):
            expect(topic[0]).to_equal(100)

        def should_pass_the_true_flag(self, topic):
            expect(topic.kwarg1).to_equal(True)
            expect(topic['kwarg1']).to_equal(True) # does the same as the previous line
            

Notice the callback argument in the topic. When you decorate the topic with async_topic, pyvows adds this argument.

Simply pass this argument as the callback to the method you wish to test, and let pyvows take care of the rest.

Assertions

pyVows features an extensible assertion model with many useful functions, as well as error reporting.

It’s always best to use the most specific assertion functions when testing a value. You’ll get much better error reporting, because your intention is clearer.

Let’s say we have the following array:

    ary = [1, 2, 3]
            

…and try to assert that it has 5 elements. With the built-in assert, we would do something like this:

    assert len(ary) == 5
            

And get the following error:

AssertionError:

Now let’s try that with one of our more specific assertion functions, to_length:

    expect(ary).to_length(5);
            

This reports the following error:

Expected topic([1, 2, 3]) to have 5 of length, but it had 3.

Other useful assertion functions bundled with pyVows include to_match, to_be_instance_of, to_include, and to_be_empty. Check out the reference for the full list.

Custom Assertions

Creating new assertions for use with expect is as simple as using the Vows.create_assertions decorator on a function. The function expects topic as the first parameter, and expectation second:

    @Vows.create_assertions
    def to_be_greater_than(topic, expected):
        return topic > expected
            

Now, the following expectation…

    expect(2).to_be_greater_than(3);
            

…will report:

Expected topic(2) to be greater than 3.

It will also create the corresponding not_ assertion:

    expect(4).not_to_be_greater_than(3);
            

Will report:

Expected topic(4) not to be greater than 3.

If you need more control over your error message, or your assertion doesn’t have a corresponding not_, you can use the lower-level Vows.assertion decorator and raise a VowsAssertionError. Here are lots of examples.

By raising a VowsAssertionError you get the benefit of highlighting the important values when your vows are broken.

If you still just wanna raise an AssertionError like old times, that’s supported:

    # NOTE: This is not a recommendation--just an example of what can be done
    @Vows.assertion
    def to_be_a_positive_integer(topic):
        assert type(topic) == int, "Expected %s to be a positive integer, but it's not even an integer" % (topic,)
        assert topic != 0, "Expected %s to be a positive integer, but it's 0" % (topic,)
        assert topic > 0, "Expected %s to be a positive integer, but it is a negative integer" % (topic,)

    @Vows.assertion
    def not_to_be_a_positive_integer(topic):
        assert topic <= 0, "Expected %s not to be a positive integer, but it was" % (topic,)
            

It’s recommended to always declare both the assertion and the not_ assertion (if applicable), so they can be used like this:

              expect(5).to_be_a_positive_integer()
              expect(-3).Not.to_be_a_positive_integer()
            

Reference

The runner and assertion modules are documented here.

Test runner

pyvows [FILE, ...] [options]
    

Run specific tests

$ pyvows test_1.py
    $ pyvows tests/
    

Run all tests in the current (or child) folders

$ pyvows
    

---

Options

-p, --pattern	Pattern of files to run as vows. Defaults to *_vows.py.
-c, --cover	Indicates that coverage of code should be shown. Defaults to True.
-l, --cover_package	Package to verify coverage. May be specified many times. Defaults to all packages.
-o, --cover_omit	Path of file to exclude from coverage. May be specified many times. Defaults to no files.
-t, --cover_threshold	Coverage number below which coverage is considered failing. Defaults to 80.0.
-r, --cover_report	Store the coverage report as the specified file.
-x, --xunit_output	Enable XUnit output.
-f, --xunit_file	Filename of the XUnit output. Defaults to pyvows.xml.
-v	Verbosity. Can be supplied multiple times to increase verbosity. Defaults to -vv.
--no-color	Does not colorize the output.
--help	Show help
--version	Show pyvows’ current version

Assertion functions

equality

    expect(4).to_equal(4)

    expect(5).Not.to_equal(4)
    

similarity

    expect("sOmE RandOm     CAse StRiNG").to_be_like('some random case string')

    expect(1).to_be_like(1)
    expect(1).to_be_like(1.0)
    expect(1).to_be_like(long(1))

    expect([1, 2, 3]).to_be_like([3, 2, 1])
    expect([1, 2, 3]).to_be_like((3, 2, 1))
    expect([[1, 2], [3,4]]).to_be_like([4, 3], [2, 1]])

    expect({ 'some': 1, 'key': 2 }).to_be_like({ 'key': 2, 'some': 1 })

    expect("sOmE RandOm     CAse StRiNG").Not.to_be_like('other string')
    expect(1).Not_to_be_like(2)
    expect([[1, 2], [3,4]]).Not.to_be_like([4, 4], [2, 1]])
    expect({ 'some': 1, 'key': 2 }).Not.to_be_like({ 'key': 3, 'some': 4 })
    

type

expect(os.path).to_be_a_function()
    expect(1).to_be_numeric()

    expect("some").Not.to_be_a_function()
    expect("some").Not.to_be_numeric()
    

truth

expect(True).to_be_true()
    expect("some").to_be_true()
    expect([1, 2, 3]).to_be_true()
    expect({ "a": "b" }).to_be_true()
    expect(1).to_be_true()

    expect(False).to_be_false()
    expect(None).to_be_false()
    expect("").to_be_false()
    expect(0).to_be_false()
    expect([]).to_be_false()
    expect({}).to_be_false()
    

None

expect(None).to_be_null()
    expect("some").Not.to_be_null()
    

inclusion

expect([1, 2, 3]).to_include(2)
    expect((1, 2, 3)).to_include(2)
    expect("123").to_include("2")
    expect({ "a": 1, "b": 2, "c": 3}).to_include("b")

    expect([1, 3]).Not.to_include(2)
    

regexp matching

expect('some').to_match(r'^[a-z]+')

    expect("Some").Not.to_match(r'^[a-z]+')
    

length

expect([1, 2, 3]).to_length(3)
    expect((1, 2, 3)).to_length(3)
    expect("abc").to_length(3)
    expect({ "a": 1, "b": 2, "c": 3}).to_length(3)

    expect([1]).Not.to_length(3)
    

emptiness

expect([]).to_be_empty()
    expect(tuple()).to_be_empty()
    expect({}).to_be_empty()
    expect("").to_be_empty()
    

exceptions

expect(RuntimeError()).to_be_an_error()

    expect(RuntimeError()).to_be_an_error_like(RuntimeError)

    expect(ValueError("error")).to_have_an_error_message_of("error")

    expect("I'm not an error").Not.to_be_an_error()

    expect(ValueError()).Not.to_be_an_error_like(RuntimeError)

    expect(ValueError("some")).Not.to_have_an_error_message_of("error")
    

About

Both pyVows and this website are HEAVILY inspired by the work done by Alexis Sellier, more commonly known as cloudhead. He’s responsible for toto, LESS, hijs, and a lot more.

More than that, he’s responsible for a shift in the way we write tests. We have cleaner, leaner, and meaner tests now. And they are fast.

pyVows was developed by Bernardo Heynemann, and features contribution by Rafael Carício, Fábio Costa, Daniel Truemper and Zearin.

The design for this website is the work of Alexis Sellier, and this website is intended as a compliment and as recognition of his great work, not as a copy.