Testing Unity3D with Autopilot

09 Mar

Previously, Gerry Boland wrote eloquently about the testing solution in place for the Unity2D project. When writing something as complicated as a desktop shell, there is a need for functional testing as well as low-level unit testing. For the 2D team, this role is filled by testability.

In the 3D team, we have autopilot: Autopilot aims to make writing functional tests for Unity as easy as possible. Previously, features that were too hard to test with unit tests were written as a “manual test”. Manual tests are text documents stored in the Unity source tree. When we prepare a point release of Unity we iterate through all the manual tests and ensure they all pass. This is obviously less than ideal – running the entire suite of manual tests takes a lot of time, and stops a developer from writing more code. Most of these tests can be automated.
Here’s an example that was written after someone noticed a bug in the dash command lens:

Dash search
This test makes sure that the right command is run when you search
using the dash. (see lp:856205)

#. Press Alt+F2
#. Press 'g'. Make sure you see some command name (like gcc)
#. Quickly type 'edit<Enter>' - so you'd run 'gedit'.

  The dash disappears, and gedit is run. If nothing happens or the first
  command is run (in this case gcc), this test failed.

This is a reasonable test specification, but does not need to be a manual test, we can automate this, and have autopilot perform the same actions as a user would. If we break down the test specification into smaller steps, we end up with the following:

  1. Reveal the command lens.
  2. Type “g”, and wait for the command lens to refresh and show us some results.
  3. Quickly type “edit” and hit “Enter”.
  4. Wait (with a timeout) for the gedit application to launch.
  5. If gedit was launched, the test passed. If the timeout was reached, the test failed.

Autopilot tests are standard python unit tests, so the resulting test will look familiar to people used to reading python unit tests:

def test_run_before_refresh(self):
    Hitting enter before view has updated results must
    run the correct command.

    kb = Keyboard()
    kb.type("edit", 0.1)
    kb.press_and_release("Enter", 0.1)
    self.addCleanup(call, "killall gedit".split())
    app_found = self.wait_until_application_is_running("Text Editor", 5)

This test is part of a larger class that contains several similar tests – I have omitted the rest of the class for clarity. There’s a few things happening here:

  • On lines 9, 11, and 12 we’re generating X11 keyboard events (we can handle mouse events as well). This allows us to manipulate the various parts of the desktop shell just like a user would.
  • On lines 8 and 14 we’re querying various parts of the unity shell for information. When we reveal the command lens we check that it’s actually showing by asking Unity internal state information. When we check for running applications we’re talking with the Bamf daemon.

We generate the X events using python-xlib and the xtest extension. Of course, we don’t want programmers to have to deal with Xlib themselves, so we wrap all Xlib calls into two classes: “Keyboard” and “Mouse”. They work exactly as you’d expect.

Getting information is a bit more interesting: many classes within unity contain information we would like in autopilot. In order to expose this information, a class must:

  1. Derive from the Introspectable class (defined in “Introspectable.h”)
  2. Implement the “GetName” and “AddProperties” methods.

Browsing the unity C++ source files will reveal many instances of introspectable objects. Introspectable objects form a tree within unity, with the main shell instance at the top. To build the tree, an introspectable instance can call the “AddChild” and “RemoveChild” methods. In this manner, a tree of objects is built. The full tree is huge, but you can see a small part of it below:

Even from this small part of the tree you can tell a lot about the state of my desktop shell: I have only one launcher (on monitor 0). Launcher keyboard navigation (as initiated by pressing Alt+F1 or Super+Tab) is not active, I have at least three launcher icons (actually many more, but they’re cut out of the image), including D-feet, gnome-terminal and the Quassel IRC client.

All this information (and more) is available to autopilot tests. The combination of a vast amount of information, and the fact that Python is incredibly fast to work in makes it trivially easy to write autopilot tests. This is one of the key ways in which we’re testing unity for the precise release cycle.

Running the Tests

Running the tests is simple. From the root of the unity source tree, you can list all the autopilot tests like this:

$ ./tools/autopilot list

You can run the entire autopilot suite like this:

$ ./tools/autopilot run

…or you can specify one or more tests you want to run (the entire suite takes a long time) like this:

$ ./tools/autopilot run autopilot.tests.test_command_lens.CommandLensSearchTests.test_run_before_refresh

Currently we have around 100 tests, and many of these are parameterized to expand in certain conditions. For example, if the test machine has multiple monitors configured, almost all the launcher tests are configured to run once on each monitor. We use a similar technique to ensure that we support multiple international languages in the dash.

On Jenkins

The entire autopilot test suite runs on jenkins as well. We’ve recently started recording failing autopilot tests, which helps us diagnose failing tests. This is especially important, as the test machine may have different hardware or software configuration from a developer’s machine: being able to see the test failing makes diagnosing errors a lot easier.

You can help!

We’re always happy to have people writing autopilot tests. If you know a bit of Python, and are willing to learn about testing Unity, please find me (“thomi”) in the #ubuntu-unity channel on – I’m happy to guide anyone who would like to get involved through writing, testing, and submitting your first autopilot test.

Leave a comment

Posted by on March 9, 2012 in Uncategorized


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: