There are several Python unit testing frameworks, but pytest is both the most popular and the easiest to use. It requires no boilerplate, no imports, no API. Just name your files, classes, or methods starting with “test”, type py.test into the command line, and observe the results.
Simple, right? And we want it that way. Pytest allows our tests to be uncluttered and easy to read. But as far as the output of those tests goes… we can do better! This article will show you how to master the pytest CLI.
To start off, make sure you have pytest installed:
pip install pytest
For those of you new to pytest, note that the package is named “pytest”, while the command to run it is “py.test”. The dot remains for historical reasons: pytest used to be part of the “py” library.
If you already have pytest installed, double-check your version number:
py.test --version
This guide is written with pytest version 2.7.0 or greater in mind. If you have an older version, you can upgrade using pip:
pip install --upgrade pytest
All good? Let’s get started!
Useful Switches
-x : Stop After First Failure
When you’re trying to track down a bug, it can help to run all of your tests, so that you can draw conclusions based on what groups of tests are failing.
That said, sometimes you just want to work through failing tests one by one, or there are so many tests in your project that running them all every single time (not to mention sifting through all the output) would be too time-consuming. There’s a nice simple switch to take care of this:
py.test -x
This will stop pytest after the first failure is encountered, saving you oodles of time. Is one just not enough? How about after two failures? No problem!
py.test --maxfail=2
-k : Run Tests Matching Keywords
What if you know some group of tests is going to fail, but you just don’t want to think about them right now? You could go through your test files and comment them out… but there’s a much better way! Let’s say you only want to run tests with the word “log” in the names:
py.test -k “log”
Easy… but wait! This also ran the tests with “logout” in the names. Let’s say we don’t want those:
py.test -k “log and not logout”
That’s right! Instead of just a simple string match, you can feed -k a full expression to use.
Controlling Verbosity
Of course, if you want to run all your tests, but just make them run more quietly, you can tone down their output like this:
py.test -q
The opposite is increasing the output, which will put all the names of your tests in a list, along with their pass and fail states:
py.test -v
This can be useful if you’re logging your tests for someone else to read. This will only control the state output, however. For the rest, see the next section!
Changing Defaults
Changing Traceback Printing
When a test fails, pytest will attempt to do a traceback to give you some information on what went wrong. And when I say “some”, I mean a lot… often too much. What if you could shorten that output, so that you didn’t have to scroll for pages to see the results of all your tests? Well, try this:
py.test --tb=short
Using –tb will change pytest’s traceback formatting to the format you specify. If “short” is still not short enough, you can do it one better:
py.test --tb=line
Or if you’re looking for something a little more familiar, you can set pytest to use the Python standard traceback formatting, like so:
py.test --tb=native
Invoke pdb On Failure
Oftentimes, when debugging a problem, you’ll stick the following statement on the line before the test failed to set a pdb breakpoint:
# code here import pdb; pdb.set_trace() # more code here
But why go through that hassle, when you can just do this when you run pytest?
py.test -x --pdb
This will invoke pdb immediately upon failing a test. Notice that I combined it with -x to stop pytest after the first failure. This combination is quite useful, as you don’t want to have to go through pdb for every single failing test.
Watching Files
All of these tricks will help you save time and energy while running tests. But all of them have one thing in common: you still have to head back into the command line and type them out. Well, we can take care of that, too. Enter pytest-xdist. This handy little package (in addition to other things) can keep an eye on your project files and re-run your tests every time it detects something has changed. Just install and run with the -f option:
pip install pytest-xdist py.test -f
This will immediately run all of your tests, then sit and wait for any file to be changed. Even better, if any tests failed in the previous run, it will re-run ONLY those tests until they pass.
Alternatively, you can use a package called pytest-watch:
pip install pytest-watch ptw
The downside to pytest-watch is that it’s a separate executable, and therefore you can’t use any of the tips in the rest of this article. On the other hand, it has some neat options, like configuring actions to occur on test failure. If you’re on OSX, try this out to have your computer actually vocalize how your tests are doing:
ptw --onpass=”say passed” --onfail=”say failed”
Final Thoughts
These are some of my favorite pytest options, but it’s really only the beginning. pytest has many options for reporting, collection, and debugging, and that’s not even getting into the rich assortment of plugins and packages to extend its capabilities. You’re going to be spending a lot of time running tests, so take the time to poke around and find the configuration that works best for you. It’ll be worth the investment!