RSS

Automated Testing of Qt apps with Testability Driver

01 Feb

Testing is hugely important to us at Canonical. We all strive to have Ubuntu reliable, consistent and fast. But we’re human, and we make mistakes. Sometimes a bugfix will break something else, and for something as complex as a desktop shell, it’s easy to miss these breakages. While manual tests can help reduce these regressions, realistically we need an automated system to emulate the users inputs and verify our software works as it should – and scream bloody murder if it doesn’t!

In Unity 2D (my project!), we have just introduced an automated User Experience testing system, based on a test framework called Testability Driver (I’ll just call it ‘Testability’ from now on). First off, a clarification:

Testability is for Qt-based applications only!

A limitation: yes, but this requirement comes with a great reward: Testability allows inspection of the tree of QObjects in a Qt application while it is running!

It can read and write object properties, call methods and slots, verify signals are emitted, as well as fake mouse/keyboard/gesture inputs, grab visual outputs, and measure performance.

And best of all, Testability is open source and maintained by Nokia! That means everyone can run and contribute tests! :)

To show it off, here is a screengrab of the Testability Visualizer application which allows you to connect to a running application, dig into the QObject tree and investigate what’s going on (here, the Unity 2D Launcher – it will be good to zoom in):

Image

On the left is an interactive snapshot of the UI, in the centre is the QObject tree which you can navigate, and on the right is the list of the selected QObject’s properties, methods and signals you can interact with.

On display in that screengrab are some of the attributes of the QDeclarativeImage object which draws the Terminal icon in the launcher (this is defined in QML!) – you can read off the source of the image, the x,y coordinates, width & height, and a whole lot more.

All these properties, methods, signals, etc, are scriptable via Ruby. Thus we can emulate every interaction that a user can make with the application, and determine the reaction matches our expectation.

And all this power comes with almost no changes* to the source code! (* = just need to add object names, see later)

This forms the foundation for the Unity 2D User Experience testing suite.

How Testability Works

First some definitions:
“target” = the machine with the software being tested
“host” = the machine running the test suite, and controlling the target

Testability works as follows

  • any Qt application using Qt4.6+ can be tested by executing it with the “-testability” switch.
  • a standalone server “qttasserver” runs in the background.
  • With the -testability switch, the Qt library tries to load a “libtestability.so” plugin which establishes a connection between the application and qttasserver, giving qttasserver access to the root node of the QObject tree.
  • qttasserver then climbs the QObject tree, reads all the info it can and converts it to an XML format. It also can receive commands and cause the application to react upon them (click here, type, etc..).
  • A series of Ruby scripts connect to qttasserver, receive and parse this XML and allow us to script tests and interactions with the application.

Image

Note that there is a clear divide between the machine of the target and of the host.

This means Testability is great for testing software on embedded devices too. (Indeed Meego has been using it for that exact task!). It also reduces the risk that packages used to run the test suite could interfere with the software being tested.

However there’s nothing stopping you having the target and host the same machine.

[The Unity 2D test framework also includes an extra helper library to control the X server, to control the mouse and keyboard, and to manipulate windows on the desktop. As far as X is concerned, a human is controlling it.]

With the ability to fake any form of user input, and directly read the output from the shell applications
themselves, we can test almost every behaviour of the desktop shell. And as a result, the quality of the user’s experience will only go up.

How to get Testability and start writing tests

Unfortunately Testability is not available in Ubuntu’s repositories just yet, but I have it packaged in a PPA. Installation takes a few steps, so I suggest that interested people consult this wiki page.

Tests are just Ruby scripts, so running tests is just a matter of running a Ruby script!

I think it easiest to show how to use Testability with an example:

# Example test for Unity 2D with Testability Driver.
#
# Note: this probably won't succeed on your installation. Only works
# on unity-2d trunk *right now*, something that will change soon.# This is only a taster!require 'tdriver'
include TDriverVerify

# Establish connection to 'qttasserver' on target
@sut = TDriver.connect_sut(:Id => 'sut_qt')

# Execute the application to test, supplying '-testability' flag
@app = @sut.run( :name => 'unity-2d-launcher',
                 :arguments => '-testability' )
# ------------- Start of Testing Code ----------------

# Check Launcher is 65 pixels wide
verify_true(1, 'Launcher is not 65 pixels wide') {
  @app.LauncherView()['width'] == '65'
}

# Check that Terminal tile is using correct icon source
verify_true(1, 'Terminal icon in Launcher is wrong') {
  @app.LauncherList( :name => 'main' ) \
      .QDeclarativeItem( :name => 'Terminal' ) \
      .QDeclarativeImage( :name => 'icon' )['source']     == 'image://icons/utilities-terminal'
}
# this is a bad test, as the name "Terminal" can be # localised, meaning a fail if you're using a non-English # installation. Choosing good objectNames is important!

# ------------------- Clean up -----------------------
# This closes (kills actually) the launcher when we're done.
@app.kill

There is some boiler-plate code there, but the bits I want to point out are:

@app.LauncherView()['width']

This causes Testability to search for an object of type "LauncherView()" in the QObject tree, and if found reads its "width" property and returns it. Then in Ruby, we can check this value matches what we expect (65).

The second test does a similar operation, but needs a little more help. As there are many tiles, we need to help Testability to find the exact tile we want. We do this by adding some object names ("main", "Terminal") to the C++/QML.

By consulting with the Visualizer image above, maybe you can see how Testability navigates the tree to find the icon source for the Terminal tile!

Once you can track down objects uniquely, you can then start interacting with them, send mouse clicks, set properties, call methods, etc. This power gives you great flexibility in testing your application!

Demo

As a demo, here is a video of a part of the Unity 2D test suite in action. On the right is Ubuntu Oneiric running inside VirtualBox, where the Launcher will be tested. On the left is my host machine terminal running the test suite.

Various hide/show tests are being performed, with windows in the way, mouse being controlled, keyboard shortcuts being pressed etc.

Our policy is that every new feature and bug fix in Unity 2D will now be tested like this. You can see our growing test suite here. This will ensure Unity 2D remains reliable, consistent and fast. Thanks to Testability!

Tests will improve your project's quality too. Will you give it a try?

- Gerry Boland

 
5 Comments

Posted by on February 1, 2012 in Uncategorized

 

Tags: , , ,

5 responses to “Automated Testing of Qt apps with Testability Driver

  1. rasjani

    February 3, 2012 at 10:33 am

    I’ve been a user (and propably provided some patches to it too) of Testability Driver for past +3 years now and while i do think “testability driver” or tdriver as we used to refer to it has its merits, it also has its bad points.

    First off, *real* touch event emulation is non-existent, main method of delivering events within tdriver is to inject then directly into qt’s event loop and this will be slightly problematic in certain cases like actually testing touch event recognizers with qt. Also, tdriver (at the moment afaik) does not support multitouch event delivery…

    Also, something that might not be evident or even usefull for everyone is how fixtures and traversers work together. Consider that tdriver does a good job of exposing all the widgets from toplevel window down to the chain of parent/child tree but in some cases one needs to extend the visibility of certain objects which do not fall to this. This could be due to fact that certain objects are not part of the c/p tree or their base object type is not supported by default by tdriver itself. So, for this reason, you need to write a traverser plugin. Writing a one is easy but it will lead a problems. Reason is that when you are using fixture plugins, the objects exposed via traversers are unusable because fixture implementation does not use traversers. Ok, you might think that since im not using fixtures, this does not concern me but you are wrong. Good portion tdriver api’s are actually just mapped method calls in tdriver’s objects to appropriate fixtures to just make the api look nicer..

    One of the most painfull things in tdriver was signal verifications. I know this is hard to come up with in a way that it would work 100% time but still. Consider an case where application widget elements are created during runtime and one would want to verify that animation framework signals are checked to be coming in proper order .. or just that creating a new window has happened it has been changed to visible state. Allmost impossible (won’t go into details but if you are qt dev, you know you can’t use QSignalSpy on object that isnt constructed yet .. )

    And one more thing that comes to my mind is a bit in-coherent api’s of tdriver api itself. Some take ruby hashes as parameters giving you ability to pass the stuff you only need to pass and some has fixed parameters and some cases these are mixed with no predictable pattern.

    And finally, the documentation, while not utter bullshit, has its downsides. Consider this example: https://projects.developer.nokia.com/dav/Testabilitydriver/doc/api/qt_linux/#FileTransfer:list_files_from_sut

    Nowhere does it mentation that if you pass a directory into this api, it will crawl through that directory and all of its sub directories without no possibility to limit it to just do that particular directory. And there are alot of similar things within the api and docs.

    Ok, that was a bit ranty post about negative sides .. In the end, we managed to utilize tdriver quite well but it required alot of workarounds and creative ways to do our testing. There are still some stuff left which i personally blame on tdriver and development team blames on our platform but we did it, 3 years of work and product was shipped in sane condition with *very minimal* people and that was a feat and tdriver has its share on archiving that.. But would i personally recommend it to anyone, no. Unless you have a people with actual *development/coding* skills (both ruby and qt/c++) to do that work for you.

     
    • rasjani

      February 3, 2012 at 10:41 am

      PS. I didnt even go to sync issues and proper usage of verifies in order to avoid unnecessery waiting time – both which arent quite well explained in the documents ;)

       
    • gerryboland

      February 3, 2012 at 1:02 pm

      Hey rasjani,
      thanks for your frank input. You bring up good points all. We are still in the early stages of utilizing the tool, but so far I can see no other open-source tool which matches its feature set.

      You are correct that the gesture & multitouch (docs claim to support it, but I’ve not had reason to try it) emulation is just inserting events into the event queue – not perfect. I’ve hacked up an xdotool-based library to properly fake mouse & keyboard inputs, in time we may extend this to gestures & multitouch too.

      Signal verifications we’ve only used sparingly. And we’ve not had the pleasure of dealing with objects tdriver doesn’t understand. As you said, I expect we’ll have to hack around some of the deficiencies, and hopefully push fixes upstream.

      Do I consider it polished enough for the masses: not quite. For simple interactions and property checks, it’s great and everyone can use it. But more advanced usage requires a deeper understanding of the tool and how it works with Qt.

       

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

 
Follow

Get every new post delivered to your Inbox.

%d bloggers like this: