RSS

Xpresser: Easy visual tests with Python

Automating tests is an essential goal of any QA Engineer. It saves precious human resources and allows to maintain a healthy continuous integration. Unit and integration tests are closest to the developer’s work: libraries, classes and functions. However, when a project is GUI-intensive or use software with poor introspection/accessibility capabilities, visual tests are very helpful. In this regard, we at Canonical are developing some good QA tools and one of them is Xpresser.

So, what is Xpresser? It’s a Python framework to do visual tests. It was originally developed by Gustavo Niemeyer. Xpresser’s main feature is to find saved images in the desktop. For example, if we want to check whether the Messaging Menu is active, we first save the blue icon and then command Xpresser to find it: Xpresser checks the whole visible desktop, as a regular person would do with its eyes. Xpresser is not only able to find images, but also to click on them. So in the previous case, will could open the Messaging Menu when active to then check whether Thunderbird is open. Xpresser even allows to simulate the keyboard.

To illustrate this, watch this useful screencast done by Chris Wayne, who works with us at Canonical as QA Engineer in OEM.

Plain easy, isn’t it?

Xpresser is currently available in this PPA ppa:xpresser-team/ppa for Ubuntu Precise and Oneiric.

 
4 Comments

Posted by on July 16, 2012 in Uncategorized

 

Continuous Integration with Jenkins and Launchpad

This post will describe how we are using Jenkins and Launchpad together to achieve Continuous integration.

Right now we are running our tests right after the code is (manually) reviewed and just before pushing them to trunk. We refer to this part of the process as autolanding and there are tools (tarmac) which can automate the process. However the manual process of code review can take quite some and our jenkins jobs are not triggered until the review is ready.

We would like to run tests much sooner, though. Ideally the workflow would be as follow:

  1. A new Merge Proposal in launchpad is created
  2. Jenkins job is triggered and starts the defined testing on defined configurations (e.g. tests on precise-i386, precise-amd64, quantal-i386, quantal-amd64, etc)
  3. Once all the tests finished the merge proposal is updated with the test result and a link to the jenkins job

In our case we are using libvirt and VMs to achieve a clean and isolated test environment. Therefore we are doing some post-test clean-up actions (reverting the VM to a predefined snapshot).

Our jenkins setup looks roughly like this.

In this case there are jenkins jobs:

  1. unity-ci – job which gets triggered as soon as a new merge proposal exists for lp:unity and does the testing
  2. generic_update-mp – parametrized job which will take test result, merge proposal and few other parameters and updates the merge proposal in launchpad with the result.

Flowchart for the whole process:

The setup for autolanding is very similar:

The only difference is that

  1. unity-autolanding job is triggered when a merge proposal is approved
  2. generic-land job will also merge the branch from merge proposal to trunk and does the actual push to trunk

The flowchart is slightly more complicated as there are more possible failure points (e.g. the testing might pass but the autolanding will fail because we require non-empty commit message and the commit message was not specified):

Right now we have a prototype in Python using a ScriptTrigger plugin for jenkins which is implementing all of this. I will describe in the next post how use it in practice together with some examples.

 
Leave a comment

Posted by on June 14, 2012 in Uncategorized

 

Testing Unity3D with Autopilot

Previously, Gerry Boland wrote eloquently about the testing solution in place for the Unity2D project. When writing something as complicated as a desktop shell, there is a need for functional testing as well as low-level unit testing. For the 2D team, this role is filled by testability.

In the 3D team, we have autopilot: Autopilot aims to make writing functional tests for Unity as easy as possible. Previously, features that were too hard to test with unit tests were written as a “manual test”. Manual tests are text documents stored in the Unity source tree. When we prepare a point release of Unity we iterate through all the manual tests and ensure they all pass. This is obviously less than ideal – running the entire suite of manual tests takes a lot of time, and stops a developer from writing more code. Most of these tests can be automated.
Here’s an example that was written after someone noticed a bug in the dash command lens:

Dash search
-----------
This test makes sure that the right command is run when you search
using the dash. (see lp:856205)

#. Press Alt+F2
#. Press 'g'. Make sure you see some command name (like gcc)
#. Quickly type 'edit<Enter>' - so you'd run 'gedit'.

Outcome
  The dash disappears, and gedit is run. If nothing happens or the first
  command is run (in this case gcc), this test failed.

This is a reasonable test specification, but does not need to be a manual test, we can automate this, and have autopilot perform the same actions as a user would. If we break down the test specification into smaller steps, we end up with the following:

  1. Reveal the command lens.
  2. Type “g”, and wait for the command lens to refresh and show us some results.
  3. Quickly type “edit” and hit “Enter”.
  4. Wait (with a timeout) for the gedit application to launch.
  5. If gedit was launched, the test passed. If the timeout was reached, the test failed.

Autopilot tests are standard python unit tests, so the resulting test will look familiar to people used to reading python unit tests:

def test_run_before_refresh(self):
    """
    Hitting enter before view has updated results must
    run the correct command.

    """
    kb = Keyboard()
    self.dash.reveal_command_lens()
    kb.type("g")
    sleep(1)
    kb.type("edit", 0.1)
    kb.press_and_release("Enter", 0.1)
    self.addCleanup(call, "killall gedit".split())
    app_found = self.wait_until_application_is_running("Text Editor", 5)
    self.assertTrue(app_found)

This test is part of a larger class that contains several similar tests – I have omitted the rest of the class for clarity. There’s a few things happening here:

  • On lines 9, 11, and 12 we’re generating X11 keyboard events (we can handle mouse events as well). This allows us to manipulate the various parts of the desktop shell just like a user would.
  • On lines 8 and 14 we’re querying various parts of the unity shell for information. When we reveal the command lens we check that it’s actually showing by asking Unity internal state information. When we check for running applications we’re talking with the Bamf daemon.

We generate the X events using python-xlib and the xtest extension. Of course, we don’t want programmers to have to deal with Xlib themselves, so we wrap all Xlib calls into two classes: “Keyboard” and “Mouse”. They work exactly as you’d expect.

Getting information is a bit more interesting: many classes within unity contain information we would like in autopilot. In order to expose this information, a class must:

  1. Derive from the Introspectable class (defined in “Introspectable.h”)
  2. Implement the “GetName” and “AddProperties” methods.

Browsing the unity C++ source files will reveal many instances of introspectable objects. Introspectable objects form a tree within unity, with the main shell instance at the top. To build the tree, an introspectable instance can call the “AddChild” and “RemoveChild” methods. In this manner, a tree of objects is built. The full tree is huge, but you can see a small part of it below:

Even from this small part of the tree you can tell a lot about the state of my desktop shell: I have only one launcher (on monitor 0). Launcher keyboard navigation (as initiated by pressing Alt+F1 or Super+Tab) is not active, I have at least three launcher icons (actually many more, but they’re cut out of the image), including D-feet, gnome-terminal and the Quassel IRC client.

All this information (and more) is available to autopilot tests. The combination of a vast amount of information, and the fact that Python is incredibly fast to work in makes it trivially easy to write autopilot tests. This is one of the key ways in which we’re testing unity for the precise release cycle.

Running the Tests

Running the tests is simple. From the root of the unity source tree, you can list all the autopilot tests like this:

$ ./tools/autopilot list

You can run the entire autopilot suite like this:

$ ./tools/autopilot run

…or you can specify one or more tests you want to run (the entire suite takes a long time) like this:

$ ./tools/autopilot run autopilot.tests.test_command_lens.CommandLensSearchTests.test_run_before_refresh

Currently we have around 100 tests, and many of these are parameterized to expand in certain conditions. For example, if the test machine has multiple monitors configured, almost all the launcher tests are configured to run once on each monitor. We use a similar technique to ensure that we support multiple international languages in the dash.

On Jenkins

The entire autopilot test suite runs on jenkins as well. We’ve recently started recording failing autopilot tests, which helps us diagnose failing tests. This is especially important, as the test machine may have different hardware or software configuration from a developer’s machine: being able to see the test failing makes diagnosing errors a lot easier.

You can help!

We’re always happy to have people writing autopilot tests. If you know a bit of Python, and are willing to learn about testing Unity, please find me (“thomi”) in the #ubuntu-unity channel on irc.freenode.net – I’m happy to guide anyone who would like to get involved through writing, testing, and submitting your first autopilot test.

 
Leave a comment

Posted by on March 9, 2012 in Uncategorized

 

Automated Testing of Qt apps with Testability Driver

Testing is hugely important to us at Canonical. We all strive to have Ubuntu reliable, consistent and fast. But we’re human, and we make mistakes. Sometimes a bugfix will break something else, and for something as complex as a desktop shell, it’s easy to miss these breakages. While manual tests can help reduce these regressions, realistically we need an automated system to emulate the users inputs and verify our software works as it should – and scream bloody murder if it doesn’t!

In Unity 2D (my project!), we have just introduced an automated User Experience testing system, based on a test framework called Testability Driver (I’ll just call it ‘Testability’ from now on). First off, a clarification:

Testability is for Qt-based applications only!

A limitation: yes, but this requirement comes with a great reward: Testability allows inspection of the tree of QObjects in a Qt application while it is running!

It can read and write object properties, call methods and slots, verify signals are emitted, as well as fake mouse/keyboard/gesture inputs, grab visual outputs, and measure performance.

And best of all, Testability is open source and maintained by Nokia! That means everyone can run and contribute tests! :)

To show it off, here is a screengrab of the Testability Visualizer application which allows you to connect to a running application, dig into the QObject tree and investigate what’s going on (here, the Unity 2D Launcher – it will be good to zoom in):

Image

On the left is an interactive snapshot of the UI, in the centre is the QObject tree which you can navigate, and on the right is the list of the selected QObject’s properties, methods and signals you can interact with.

On display in that screengrab are some of the attributes of the QDeclarativeImage object which draws the Terminal icon in the launcher (this is defined in QML!) – you can read off the source of the image, the x,y coordinates, width & height, and a whole lot more.

All these properties, methods, signals, etc, are scriptable via Ruby. Thus we can emulate every interaction that a user can make with the application, and determine the reaction matches our expectation.

And all this power comes with almost no changes* to the source code! (* = just need to add object names, see later)

This forms the foundation for the Unity 2D User Experience testing suite.

How Testability Works

First some definitions:
“target” = the machine with the software being tested
“host” = the machine running the test suite, and controlling the target

Testability works as follows

  • any Qt application using Qt4.6+ can be tested by executing it with the “-testability” switch.
  • a standalone server “qttasserver” runs in the background.
  • With the -testability switch, the Qt library tries to load a “libtestability.so” plugin which establishes a connection between the application and qttasserver, giving qttasserver access to the root node of the QObject tree.
  • qttasserver then climbs the QObject tree, reads all the info it can and converts it to an XML format. It also can receive commands and cause the application to react upon them (click here, type, etc..).
  • A series of Ruby scripts connect to qttasserver, receive and parse this XML and allow us to script tests and interactions with the application.

Image

Note that there is a clear divide between the machine of the target and of the host.

This means Testability is great for testing software on embedded devices too. (Indeed Meego has been using it for that exact task!). It also reduces the risk that packages used to run the test suite could interfere with the software being tested.

However there’s nothing stopping you having the target and host the same machine.

[The Unity 2D test framework also includes an extra helper library to control the X server, to control the mouse and keyboard, and to manipulate windows on the desktop. As far as X is concerned, a human is controlling it.]

With the ability to fake any form of user input, and directly read the output from the shell applications
themselves, we can test almost every behaviour of the desktop shell. And as a result, the quality of the user’s experience will only go up.

How to get Testability and start writing tests

Unfortunately Testability is not available in Ubuntu’s repositories just yet, but I have it packaged in a PPA. Installation takes a few steps, so I suggest that interested people consult this wiki page.

Tests are just Ruby scripts, so running tests is just a matter of running a Ruby script!

I think it easiest to show how to use Testability with an example:

# Example test for Unity 2D with Testability Driver.
#
# Note: this probably won't succeed on your installation. Only works
# on unity-2d trunk *right now*, something that will change soon.# This is only a taster!require 'tdriver'
include TDriverVerify

# Establish connection to 'qttasserver' on target
@sut = TDriver.connect_sut(:Id => 'sut_qt')

# Execute the application to test, supplying '-testability' flag
@app = @sut.run( :name => 'unity-2d-launcher',
                 :arguments => '-testability' )
# ------------- Start of Testing Code ----------------

# Check Launcher is 65 pixels wide
verify_true(1, 'Launcher is not 65 pixels wide') {
  @app.LauncherView()['width'] == '65'
}

# Check that Terminal tile is using correct icon source
verify_true(1, 'Terminal icon in Launcher is wrong') {
  @app.LauncherList( :name => 'main' ) \
      .QDeclarativeItem( :name => 'Terminal' ) \
      .QDeclarativeImage( :name => 'icon' )['source']     == 'image://icons/utilities-terminal'
}
# this is a bad test, as the name "Terminal" can be # localised, meaning a fail if you're using a non-English # installation. Choosing good objectNames is important!

# ------------------- Clean up -----------------------
# This closes (kills actually) the launcher when we're done.
@app.kill

There is some boiler-plate code there, but the bits I want to point out are:

@app.LauncherView()['width']

This causes Testability to search for an object of type “LauncherView()” in the QObject tree, and if found reads its “width” property and returns it. Then in Ruby, we can check this value matches what we expect (65).

The second test does a similar operation, but needs a little more help. As there are many tiles, we need to help Testability to find the exact tile we want. We do this by adding some object names (“main”, “Terminal”) to the C++/QML.

By consulting with the Visualizer image above, maybe you can see how Testability navigates the tree to find the icon source for the Terminal tile!

Once you can track down objects uniquely, you can then start interacting with them, send mouse clicks, set properties, call methods, etc. This power gives you great flexibility in testing your application!

Demo

As a demo, here is a video of a part of the Unity 2D test suite in action. On the right is Ubuntu Oneiric running inside VirtualBox, where the Launcher will be tested. On the left is my host machine terminal running the test suite.

Various hide/show tests are being performed, with windows in the way, mouse being controlled, keyboard shortcuts being pressed etc.

Our policy is that every new feature and bug fix in Unity 2D will now be tested like this. You can see our growing test suite here. This will ensure Unity 2D remains reliable, consistent and fast. Thanks to Testability!

Tests will improve your project’s quality too. Will you give it a try?

- Gerry Boland

 
5 Comments

Posted by on February 1, 2012 in Uncategorized

 

Tags: , , ,

Test Coverage Tutorial for C/C++ Autotools Projects

This tutorial describes how to produce test-coverage metrics for C/C++ projects. We start by instrumenting an autotools build to produce gcov output with gcc; we then generate test coverage artifacts by running our test suite, and finally we explore our results using lcov.

Introduction

We’d like to assess the quality of the existing test suite for each Product Strategy project. A measurement of test coverage will tell us what part of the project’s code is “covered” or exercised by its tests. 50% is a start; 80% would be a good goal for a neglected project; one rarely encounters 100% test coverage, but we’d like to get as close as we can. Initially we’ll use these findings to gain an overview of the test-quality of each project; ultimately these metrics will guide the improvement of our codebase, and enable us to monitor our progress using Jenkins and associated open-source tools.

A Three-Part Process in Several Steps

We’ll enable test-coverage of a particular C/C++ project in a three-part process:

  • enabling a special build
  • running the tests
  • studying the output

Our first step will be to enable a special build. Ultimately this just means adding a few flags to our gcc invocation, but it never seems so straightforward with autotools projects ;) . The Product Strategy Quality team has made available a set of files to facilitate this build–please check these out now if you’d like to follow along.

bzr co lp:coverage-tutorial

Inspecting this archive you’ll find

  • gcov.m4: an autoconf macro which will check for relevant tools
  • Makefile.am.coverage, which includes our coverage-enabled automake targets
  • an old revision of dbus-test-runner
  • a copy of this tutorial

Before we start let’s make sure we have lcov installed. lcov incoporates the GNU tool gcov and produces pretty HTML reports of our coverage results.

sudo apt-get lcov

(Note that the Debian lcov package includes genhtml.) We’re ready to begin.

1. Branch a project of interest

I heard you like testing, so for this example we’ll test dbus-test-runner which is a runner which runs tests against the dbus messaging system. An old revision is included in our coverage-tutorial archive; if you want to start fresh (or on a different project), you would

bzr branch lp:dbus-test-runner
2. Install the gcov.m4 autoconf macro and invoke it

Open gcov.m4: here’s where we’re defining the compiler flags and checking for necessary tools. Let’s put this file where autoconf can find it.

Your project may already have a directory for m4 macros:

$ grep AC_CONFIG_MACRO_DIR *
configure.ac:AC_CONFIG_MACRO_DIR([m4])

If you find such a directive, simply copy the gcov.m4 file into the named directory. If not, create an m4 directory and copy the gcov.m4 file there–and don’t forget to include a reference to the directory in the body of configure.ac (it’ll look like the grep result above).

3. Massage configure.ac to include our necessary coverage flags.

This is the essential move of our special build. gcc supports coverage reporting with the addition of a few compiler flags. Here they are in the gcov.m4 file:

# Remove all optimization flags from CFLAGS
changequote({,})
CFLAGS=`echo "$CFLAGS" | $SED -e 's/-O[0-9]*//g'`
changequote([,])
# Add the special gcc flags
COVERAGE_CFLAGS="-O0 -fprofile-arcs -ftest-coverage"
COVERAGE_CXXFLAGS="-O0 -fprofile-arcs -ftest-coverage"
COVERAGE_LDFLAGS="-lgcov"

When the tests are run, --fprofile-arcs produces a tally of the execution of each arc of the code–one .gcda file for each source file. The --ftest-coverage flag produces .gcno files, which link an arc to a source line so that we can see which lines were touched. We’ll watch these files being produced in a little bit; you can read about the flags at length in the GNU documentation. Note that we also remove optimization with the -O0 flag to get more precise results.

Now here’s the step which requires knowledge both of your code and little bit of autoconf. We’ve included the flags above (in step 3), but we’ll now need to edit our configure.ac to make the flags available to our build. For dbus-test-runner the diff looks like this:

=== modified file configure.ac
--- configure.ac 2010-12-08 02:35:12 +0000
+++ configure.ac 2011-12-06 21:42:04 +0000
@@ -45,6 +45,16 @@
 AM_GLIB_GNU_GETTEXT
 ###########################
+# gcov coverage reporting
+###########################
+
+m4_include([m4/gcov.m4])
+AC_TDD_GCOV
+AC_SUBST(COVERAGE_CFLAGS)
+AC_SUBST(COVERAGE_CXXFLAGS)
+AC_SUBST(COVERAGE_LDFLAGS)
+
+###########################
 # Files #
 ##########################

And then having added these flags to the build process, we need to actually actually add them to the build proper:

=== modified file src/Makefile.am
--- src/Makefile.am 2009-12-07 21:00:43 +0000
+++ src/Makefile.am 2011-12-06 21:42:04 +0000
@@ -3,6 +3,8 @@
 dbus_test_runner_SOURCES = dbus-test-runner.c
 dbus_test_runner_CFLAGS = $(DBUS_TEST_RUNNER_CFLAGS) \
+ $(COVERAGE_CFLAGS) \
-DDEFAULT_SESSION_CONF="\"$(datadir)/dbus-test-runner/session.conf\"" \
 -Wall -Werror
 dbus_test_runner_LDADD = $(DBUS_TEST_RUNNER_LIBS)
+dbus_test_runner_LDFLAGS = $(COVERAGE_LDFLAGS)

Here we risk running afoul of the autotools “ancient ones”. This is an uncomplicated project–if you’re not getting the results you want in the next step, I can recommend The Goat Book as a decent tutorial on how these macros work.

4. Verify that autoconf.sh --enable-gcov generates .gcno files

Having enabled our special build with the gcc compiler flags, let’s verify that autoconf is generating the .gcno files we’ve asked for:

$ ./autogen.sh --enable-gcov
...
$ find -name *.gcno
./src/dbus_test_runner-dbus-test-runner.gcno

If all goes well we’ll find a .gcno for each .o file compiled. If not, then our flags aren’t being respected–maybe something’s wrong with our configure.ac addition in step 3. It’s important to have these checkpoints along the way to divide the autotools process into testable pieces.

5. Install Makefile.am.coverage

So now that we have the instrumentation we need, we’ll use a special build to run the tests. Copy Makefile.am.coverage into the top-level directory.

cp ../Makefile.am.coverage .

Inspection shows that this file defines some extra make targets to generate coverage results using lcov. We’ll alert automake to this by simply including the file–here’s the diff for dbus-test-runner:

=== modified file Makefile.am
--- Makefile.am 2009-04-23 21:19:56 +0000
+++ Makefile.am 2011-12-19 18:00:38 +0000
@@ -1,1 +1,3 @@
 SUBDIRS = data src tests po
+
+include $(top_srcdir)/Makefile.am.coverage
6. Verify that make check generates .gcda files

Now we’ll actually run the tests.

$ make coverage-html
...
$ find -name *.gcda
./src/dbus_test_runner-dbus-test-runner.gcda

Again these .gcda files represent a tally of ‘arcs’ through the code. Note that we can generate these files not just while running tests, but also during normal execution–you can see the kinship with profiling. lcov (via gcov) has used these tallies with our .gcno files to produce line-by-line results which we’ll study in a moment. Here’s what the test coverage looks like for dbus-test-runner:

Overall coverage rate:
lines......: 78.8% (215 of 273 lines)
functions..: 86.4% (19 of 22 functions)
branches...: 61.2% (60 of 98 branches)

For a small project these figures show that we have a good test-suite to build on. We’re curious about what we’re missing, though, so we’ll investigate below. Before you register a comment about “no coverage results”, recognize that you may actually have zero code coverage. But at least you’re able to measure it, riiight?

7. Have a look at the lcov pages

lcov has produced a coverage-results directory at the top-level of our project; open index.html with a web browser and explore.

lcov index

At a glance our line coverage is 78.8%, and our “test-coverage progress bar” is yellow. Be aware of the difference between the offered metrics:

  • line coverage: how many lines have our tests touched
  • function coverage: how many functions have our tests touched
  • branch coverage: for the graph which describes all possible paths of control through this file–especially through conditionals, e.g.–what percentage has our tests touched

In our opinion the killer feature of the lcov output is the source-file display, which shows which lines weren’t touched by tests. Drill into the source directory to see the results for a particular file.

lcov detail

Here in dbus-test-runner.c line 93 we’ve hit the line that tests the status of G_IO_STATUS_NORMAL 61 times, but never taken the path in the line below, for which G_IO_STATUS_NORMAL is false.

Conclusion

We’ve presented an introduction to test coverage, featuring the gcc toolchain and autotools–as demonstrated with a set of utilities we’ve found useful here in the Quality group.

I hope it’s obvious that this tutorial is just the beginning–not just of applying these methods to our code (we and you), but also of understanding what test coverage means to the improvement of Quality. Our team has witnessed that having an observable measure like the lcov green bars encourages developers to increase test coverage. However while chasing into a particularly narrow corner of their code, our developers have produced some confounding results which have caused us to doubt the accuracy of the available tools–I hope that Thomas Voss will follow with a post on some of his findings in the future.

Regardless, test coverage will be an important metric for our group for the coming cycles, and the bars will continue to grow greener as Spring approaches. . . . Meanwhile I’ll be interested to learn about your autotools/gcc/coverage experiences:

  • What lcov alternatives have you tried?
  • Which hallowed autotools rituals have I profaned in preparing this tutorial?
  • What’s the highest coverage percentage you’ve witnessed in a GPL project?
 
1 Comment

Posted by on January 29, 2012 in Coverage, Quality

 

Quality (not only) in Budapest

Last week was my first one at Canonical and I spent it in Budapest where Ubuntu Engineering and Product Strategy teams had their rally. This is the first company where I was asked to travel on my first day and it did feel strange at the beginning :) After being a Canonical employee for 1+ week I think it is the best way how to start in a new job.

When talking about quality these are probably the key observations I have made from my new perspective of a Product Strategy Quality Lead:

  • Quality really is a focus. Every single engineer I’ve met in Budapest was thinking in terms like unit tests or code coverage and how to make Ubuntu (even) better.
  • More and more people inside Canonical are now focused solely on quality. We have several (great) Quality Engineers, a new QA Community Coordinator. I started last week and Canonical is still hiring some more.
  • A lot is actually already done and you can see and touch the results.  Just give 12.04 with the latest Unity 5.0 a try.
  • Even more is on the way and we will try to cover most of that on this blog as times go.

Being in Budapest was certainly overwhelming but it was worth it. I’m not a person who gets easily excited but now I really am (both from the people and the new products such as Ubuntu TV). I and other quality engineers plan to write more about tools, approaches or just things we consider useful, so please stay tuned.

Do you have any good experience with specific methodology or tool? How did it help with your software projects? Is there anything specific you would like to see on this blog?

 
Leave a comment

Posted by on January 19, 2012 in Quality

 

Google Test & Jenkins CI

With Canonical’s growing engagement with test driven development (TDD), we recently started introducing Google Test to a number of our upstream projects. Google Test is a unit testing framework targeted towards C & C++. It is lightweight and offers a load of features that ease unit testing tremendously. Most importantly, it supports automatic test case registration and the notions of test suite environments and fixtures. Here, a developer’s efficiency is our main focus. We acknowledge that providing a decent test coverage requires time and effort and we would like to free a developer from writing boiler plate code for test case setup, teardown and for acutally assembling his test cases.

Recently, we started working on an extension to Google test that introduces a dummy (read:headless) X server environment by means of a custom Google test environment and custom Google test fixtures. In summary, the extension starts and stops a dummy xserver and makes sure that a connection to the server is established correctly as part of the test setup procedure. On tear down, the connection to the X server is terminated before the server instance is stopped. Jump over to launchpad for the actual project and take a look at utouch-frame for a real-world application of the extension.

Test Case Result Integration with Jenkins CI

Continuous integration is a hot topic at Canonical and automatic execution of unit test suites is an integral part of our development model. For this reason, we store all unit test results in Google test’s XML dialect and export it to Jenkins to provide user, developers and QA with a summary of a project’s build status.

Conclusions & Outlook

With the introduction of Google test & Jenkins CI to Canonical’s testing landscape we want to provide the plumbing layer for a TDD environment that is loved by developers. It is the first step towards supporting engineers with meaningful metrics to further understand our software ecosystem and identify blank spots on our software map. Thus, stay tuned for Allan Lesage‘s work on code coverage calculation going to land in the not too distant future :-)

 
1 Comment

Posted by on December 12, 2011 in Quality

 
 
Follow

Get every new post delivered to your Inbox.