To start with unit tests are supposed to be independent. So must be python-unittest. Tests executed through python-unittest
should be designed in such a way that they should be able to be run independently. Pure unit tests offer a benefit that when they fail, they often depicts what exactly went wrong. Still we tend to write functional tests, integration tests, and system tests with the unittest
framework and these tests won't be feasible to run without ordering them since Selenium automates the Browsing Context. To achieve the ordering, you at-least need to use a better naming convention for the testnames, as an example: test_1
, test_2
, test_3
, etc and this works because the tests are sorted respect to the built-in ordering for strings.
However, as per your observation the problem appears with test_10
and so on where sorting order seems to break. As an example, among 3 tests with name as test_1
, test_2
and test_10
, it seems unittest executes test_10
before test_2
:
Code:
import unittest
class Test(unittest.TestCase):
@classmethod
def setUp(self):
print("I'm in setUp")
def test_1(self):
print("I'm in test 1")
def test_2(self):
print("I'm in test 2")
def test_10(self):
print("I'm in test 10")
@classmethod
def tearDown(self):
print("I'm in tearDown")
if __name__ == "__main__":
unittest.main()
Console Output:
Finding files... done.
Importing test modules ... done.
I'm in setUp
I'm in test 1
I'm in tearDown
I'm in setUp
I'm in test 10
I'm in tearDown
I'm in setUp
I'm in test 2
I'm in tearDown
----------------------------------------------------------------------
Ran 3 tests in 0.001s
OK
Solution
Different solutions were offered in different discussions and some of them are as follows:
@max in the discussion Unittest tests order suggested to set the sortTestMethodsUsing
to None
as follows:
import unittest
unittest.TestLoader.sortTestMethodsUsing = None
@atomocopter in the discussion changing order of unit tests in Python suggested to set the sortTestMethodsUsing
to some value as follows:
import unittest
unittest.TestLoader.sortTestMethodsUsing = lambda _, x, y: cmp(y, x)
@ElmarZander in the discussion Unittest tests order suggested to use nose
and write your testcases as functions (and not as methods of some TestCase derived class) nose
doesn't fiddle with the order, but uses the order of the functions as defined in the file.
@Keiji in the discussion Controlling the order of unittest.TestCases mentions:
sortTestMethodsUsing expects a function like Python 2's cmp
, which
has no equivalent in Python 3 (I went to check if Python 3 had a <=>
spaceship operator yet, but apparently not; they expect you to rely on
separate comparisons for <
and ==
, which seems much a backwards
step...). The function takes two arguments to compare, and must return
a negative number if the first is smaller. Notably in this particular
case, the function may assume that the arguments are never equal, as
unittest
will not put duplicates in its list of test names.
With this in mind, here's the simplest way I found to do it, assuming
you only use one TestCase class:
def make_orderer():
order = {}
def ordered(f):
order[f.__name__] = len(order)
return f
def compare(a, b):
return [1, -1][order[a] < order[b]]
return ordered, compare
ordered, compare = make_orderer()
unittest.defaultTestLoader.sortTestMethodsUsing = compare
Then, annotate each test method with @ordered
:
class TestMyClass(unittest.TestCase):
@ordered
def test_run_me_first(self):
pass
@ordered
def test_do_this_second(self):
pass
@ordered
def test_the_final_bits(self):
pass
if __name__ == '__main__':
unittest.main()
This relies on Python calling annotations in the order the annotated
functions appear in the file. As far as I know, this is intended, and
I'd be surprised if it changed, but I don't actually know if it's
guaranteed behavior. I think this solution will even work in Python 2
as well, for those who are unfortunately stuck with it, though I
haven't had a chance to test this.
If you have multiple TestCase classes, you'll need to run ordered,
compare = make_orderer()
once per class before the class
definition, though how this can be used with sortTestMethodsUsing
will be more tricky and I haven't yet been able to test this either.
For the record, the code I am testing does not rely on the test
order being fixed - and I fully understand that you shouldn't rely on
test order, and this is the reason people use to avoid answering this
question. The order of my tests could be randomised and it'd work just
as well. However, there is one very good reason I'd like the order to
be fixed to the order they're defined in the file: it makes it so much
easier to see at a glance which tests failed.