Martin Pool's blog

Mail me RSS Bazaar distcc Canonical is hiring! flickr

py.test

Ian Bicking writes of py.test, a Python testing framework.

It is, to be sure, a big improvement on the standard unittest module, which has far strong of a Java flavor to be comfortable in Python. It is similar to my comfychair in some ways, and probably superior at the moment (though there are some comfychair features I am quite fond of.)

I think the problem with unittest (and to a much lesser extent the current version of comfychair) can be seen by considering a definition of a class: a class is a set of objects with different data sharing common behavior.

That implies one should create a class when there is particular behavior, and when that behavior is common to more than one object.

It is a mismatch if one needs to create a class to hold behaviour that isn't coupled into any particular object. Classes with only a single method are a bad smell but are common in unittest suites. It's also somewhat conceptually unclear what the object created by such a test fixture is meant to represent.

[more later?]

posted Wed 15 Dec 2004 in /software/testing | link

Python rocks for testing

I'm testing a custom XML-object mapping library. Writing test cases in Python is incredibly pleasant:

class BaseParseTest(comfychair.TestCase):
    def runtest(self):
        tt = pywbem.tupletree.xml_to_tupletree(self.input_xml)
        result = pywbem.tupleparse.parse_any(tt)
        self.assert_equal(result, self.expected_obj)

class KeyBinding_Numeric_Test(BaseParseTest):
    input_xml = """<KEYBINDING NAME="answer">
        <KEYVALUE VALUETYPE="numeric">42</KEYVALUE>
    </KEYBINDING>"""
    expected_obj = {"answer": 42}

and that's it! Rinse, repeat. Add more as problems are discovered.

I'm using my comfychair test library here, which is yet another rewrite of the JUnit idea, specialized for the kind of tests I often write.

In particular:

• It's easy to statically declare complex nested object structures. It would be in Perl or Ruby or Lisp too; it's hard in Java or C or C++. It's even hard to have an embedded multi-line XML document in those languages.
• It's easy to compare those nested objects.
• If something goes wrong, you're likely to get a nice traceback showing where things broke, and allowing use of the postmortem debugger.

It's only a matter of degree, but being 50% easier to create test cases has a compounding effect on the amount of test coverage.

Python has much weaker compile-time tests than say Java or C. The ease of writing tests tends to make up for this though, and to get you to a higher state than Java that merely compiles but is not yet tested.

posted Wed 6 Oct 2004 in /software/testing | link

Writing tests will make you happy

Mark Dominus has wise advice on testing:

Writing tests will make you happy

If you make it into a big chore, you won't do it

So don't make it into a big chore

Effective tests can be extremely simple

posted Tue 5 Oct 2004 in /software/testing | link

Rapid Testing

Jason pointed me to James Bach's writing on rapid testing. It looks interesting:

How is Rapid Testing different from normal software testing?

Testing practice differs from industry to industry, company to company, and tester to tester. But there are some elements that most test projects have in common. Let's call those common elements "normal testing". In our experience, normal testing involves writing test cases against some kind of specification. These test cases are fragmentary plans or procedures that loosely specify what a tester will do to test the product. The tester is then expected to perform these test cases on the product, repeatedly, throughout the course of the project.

Rapid testing differs from traditional testing in several major ways:

• Mission. In Rapid Testing we don't start with a task ("write test cases"), we start with a mission. Our mission may be "find important problems fast". If so, then writing test cases may not be the best approach to the test process. If, on the other hand, our mission is "please the FDA auditors", then we not only will have to write test cases, we'll have to write certain kinds of test cases and present them in a specifically approved format. Proceeding from an understanding of our mission, we take stock of our situation and look for the most efficient and effective actions we can take right now to move towards fulfilling that mission.
• Skills. To do any testing well requires skill. Normal testing downplays the importance of skill by focusing on the format of test documentation rather than the robustness of tests. Rapid Testing, as we describe it, highlights skill. It isn't a mechanical technique like making microwave popcorn, or filling out forms at the DMV. Robust tests are very important, so we practice critical thinking and experimental design skills. A novice tester will not do RT very well unless supervised and coached by a senior tester who is trained (or self-trained) in the art. We hope the articles and presentations on this site will help you work on those skills.
• Risk. Normal testing is focused on functional and structural product coverage. In other words, if the product can do X, then try X. Rapid Testing focuses on important problems. We gain an understanding of the product we're testing to the point where we can imagine what kinds of problems are more likely to happen and what problems would have more impact if they happened.Then we put most of our effort into testing for those problems. Rapid Testing is concerned with uncovering the most important information as soon as possible. [....]

posted Wed 19 Nov 2003 in /software/testing | link

Copyright (C) 1999-2007 Martin Pool.