sil2100//vx developer log

Welcome to my personal web page

Autopilot for Unity

Unity currently uses a very interesting tool for functional testing: Autopilot. It's a custom solution created with Unity in mind, but can also be used for any other system as well. We have recently decided to start getting rid of all manual testing in Unity and related components, meaning no more running away from automated testing. This is a quick look on how we can use Autopilot to perform automatic testing for our convenience with the Unity stack. All related to Ubuntu 12.10 and later of course.

2012-10-29 00:11

Empty

Current policy in Unity-related development is that every fix that is to be included is required to have automatic testing, if possible by technical means. This means that every merge request needs to have either unity testing or automated functional testing included - no manual tests are accepted any more. Some code that is obviously not-testable, like visual changes or some difficult crashers or others can be included as exceptions. But the overall rule is: all code needs to be tested.

Unity uses Google Test and Google Mock for unit and mock testing. For functional testing Autopilot is being used. There's a lot about gtest around, but it's harder to find anything regarding autopilot - especially that it's undergoing constant development even now.

The following post relates to Ubuntu 12.10 Quantal Quetzal and Autopilot from the project's PPA (ppa:autopilot/ppa) - version from the 1.2 series (for instance, 1.2+bzr88+pkg56~quantal1). The usage of the latest autopilot is always recommended for the latest Unity stack.
So, things that we will need for learning and using Autopilot with Unity:

I have provided an example test suite for Unity, packed up nicely in a stand-alone archive. The relevant sources are commented for better readability. All that is needed to run it is extracting the archive and running the ./start_demonstration.sh script. I will try to overview some of the specifics demonstrated below.

Get the example source archive for this post HERE.

How to run autopilot tests?

This depends on the version of Autopilot we want to use, but it's usually a similar set of steps. For 1.2, running AP tests is really very easy - installing the tests is not necessary, as they can be run from any path. First, we change to the autopilot test directory in the Unity source (./tests/autopilot/). Afterwards we can either check the number of tests available or run all of them or just the ones we're really interested in. We can be as specific as we want.


 cd ./tests/autopilot
 autopilot list unity # List all available unity autopilot tests
 autopilot run unity  # Run all available unity autopilot tests (takes A LOT of time!)
 
 autopilot list unity.tests.test_switcher # List all tests from the test_switcher suite
 # And we can run either one specific test (like below) or run just one test suite
 # We can be as specific as we want!
 autopilot run unity.tests.test_switcher.SwitcherTests.test_switcher_move_next

A small note to remember: when running autopilot tests, some tests tend to 'hang-up' and wait idle for some time. It seems to happen due to the current multi-threaded design of Autopilot. Just know that the test will resume its normal behavior in time, so patience sometimes might be needed.

How to write autopilot tests?

Writing autopilot tests for Unity is a rather easy task, as Unity provides really useful tools for specific actions one would want to perform during functional testing. The demo I have included shows most of the basic use-cases, but I'll also try to cover some of them here.

AP tests are grouped into test files, each having one or more test-case - while every test-case can have one or more test. In case of Unity, every new unity test-case needs to inherit from the UnityTestCase class (as defined in unity.tests), which includes all helpers necessary for testing the shell. Now, every such test-case can define tests that can be performed being methods of that class, each starting with a test_ prefix, e.g. test_if_foo_is_bar(self).

If a given test case needs some specific setup routines performed before running the selected test suite, we have the setUp() method available for override (just remember to run the parent's version at the beginning). We can put that inside as we want.


 from unity.tests import UnityTestCase

 class SomeTestCase(UnityTestCase):
   def setUp(self):
     super(SomeTestCase, self).setUp()

     # We can include any setup here - for instance cleanups, which I will mention about
     # in a moment
     self.addCleanup(self.dash.ensure_hidden)
     self.some_other_setup_routine()

   def test_check_if_oven_is_turned_off(self):
     """This check is supposed to check if the oven is turned off
     """
     # Do the check here
     # (...)

Autopilot uses so called emulators for doing 'specific' interaction with Unity. Some of the Unity-specific emulators can be found in ./unity/emulators/ path in the Unity autopilot directory. These emulators provide methods for doing more complex interaction with the system, e.g. revealing lenses, switching applications, providing states of components (visibility, results) etc. So, for instance, to force the dash to be visible, we can simply call self.dash.ensure_visible() in the body of our test method. There are, of course, some other methods of performing the very same task - but more manually.

Autopilot allows us to move the mouse cursor at will, as well as provide keyboard input as needed. So therefore we can also open the dash by, for instance, moving the mouse to the position of the dash icon with self.mouse.move(10, 30) and then performing self.mouse.click() or simply using self.keyboard.press_and_release("Super") to force a Super-key tap. Of course, there are more methods of doing the same, as for instance using self.keybinding_tap("dash/reveal") to force a keybinding tap. All these actions will ultimately lead to the dash being opened. Besides that, AP also alows us to start new applications, using self.start_app_window() - and many many more.

The best way of getting to know all the available functions is looking up emulator code or other existing tests. But here's a short list of those most obvious tools:

But simply being able to perform actions is not enough for functional testing. We actually need to somehow assert that the result of our actions is correct and the behavior is correct. Autopilot provides some interesting constructs for this as well.

Let's take our earlier case - we're opening the dash in Unity. So let's say we actually want to make sure the dash has been opened after we performed the mouse action or the keyboard press. For this, we can make use of the self.assertThat() method of the Autopilot test-case. Consider the following quick example:


 from autopilot.matchers import Eventuall
 from testtools.matchers import Equals
 from unity.tests import UnityTestCase

 class SomeTestCase(UnityTestCase):
   # (...)

   def test_if_dash_opens_on_super(self):
     """Check if pressing Super opens the dash when the dash is not visible
     """
     # First, let's assert that the dash is not visible
     self.assertThat(self.dash.visible, Equals(False))

     # Press the Super key...
     self.keyboard.press_and_release("Super")
     # ...and make sure that after some time, dash finally gets visible
     self.assertThat(self.dash.visible, Eventually(Equals(True)))

   # (...)

The Eventually() syntax gives the given assertion a short time period for it to actually happen (I think it's around 10 seconds now). If the assertion fails even after this time, the test fails. There are many constructs that we can use for assertions, these are mostly: Equals(), NotEquals(), GreaterThan(), LessThan() etc. All for our disposal. And at the end of all the tests we get a summary of how many tests have actually succeeded.

One more thing worth noting - remember the self.addCleanup() I mentioned earlier? This way we can add things we want the test to perform in case when the test ends (normally or due to a failure) - like, for instance, hide the dash. Whatever we set as an argument will be actually called at the end of the given test or test-case. Unity of course also checks by itself at the end of every test to make sure if the test was well-behaved, closing the dash and applications when necessary, but it's always better to deal with it by ourselves.

And that's more or less it. This is by no means a complete tutorial, but more of an overview of how Autopilot can be used for Unity. A real tutorial will probably come pretty soon, but it's not really needed to start writing useful tests already. Much of the magic can also be easily read from existing test-cases. Be sure to include testing (unit tests or AP tests) to every merge-request proposed for Unity - otherwise your merge will probably be rejected or bounced back for fixing.

For the demo code included in the archive (it's here if you missed it), I have also used Python bindings for Notify OSD in Ubuntu. No use of explaining on how it works, as there is a really good tutorial on developer.ubuntu.com - so just check it out.