Android/Testing

Introduction

Testing is a broad subject, with various methodologies combining various approaches, but it embodies attempts to ensure that the user's experience is satisfactory: intuitive, reliable, responsive, secure, etc. Practices such as Test-Driven Development and Behaviour-Driven Development are popular at the time of writing, using means such as Unit Testing and Integration Testing, but wider testing areas include checking that the user can perceive how to achieve his/her goals without frustration or cultural misunderstandings through to checking that the server against which an application runs can sustain the expected average throughput, gracefully handles peak load, is robust against attacks, and so on.

Unit Testing

Unit tests are tests that exercise a class in isolation from (most of) the rest of the application, and form a significant element of Test Driven Development where failing tests are written to allow the application to be 'debugged into existence'.

Where the code under development is uninvolved with Android, standard Java unit test techniques apply, but where code rubs against the Android system a degree of working around (similar to that required while working with legacy code) is necessary to overcome issues that include:

• The android.jar on the development machine (against which applications are built) throws exceptions in response to attempts to execute its methods
• Android declares various of these exception-throwing methods as final, rendering them impossible to include in tested code directly
• Android makes use of static methods, again impossible to test directly.

The section on Unit Testing gives various approaches that may help with these issues; you will need to evaluate how well they work for you.

The following tools may also prove useful in combination with the aforementioned approaches:

• Robolectric - provides unit-test friendly simulations of Android classes that run on the hosting PC with a faster compile/execute cycle than running on a phone or an emulator with non-native architecture^[1].
• Hamcrest - provides useful and readable abstractions for building assertion conditions with useful diagnostics
• awaitility - helps with asynchronous testing
• Object mocking libraries provides a powerful way of building mock objects
  • JMock - notably, JUnitRuleMockery
  • mockito
  • powermock
• Dependency injection, which allows you to request objects based on interface and a mapping system rather than directly by class, so you can substitute test classes in without disrupting the compiled code. This may simplify your code, with less code to read... or make it harder to track down quite what's calling what, and it can degrade start-up time unacceptably.
  • RoboGuice
  • Spring
  • Related:
    • Android Annotations - Promising as a means of reducing boilerplate (and hence coding error), but not directly testing related. Its unit testing section mentions Robolectric, which is good.
    • Android Binding isn't quite dependency injections, but seeks to separate concerns with MVVM, and may be worth checking out.

Integration Testing

Whereas unit testing tests classes (largely) in isolation from each other, and system testing checks that the entire assembly works in the target platform when operated as a user, integration testing checks that selected groups of parts fit together properly - in controlled conditions that are easier to set up and manipulate with direct access to the program code.

The official Android documentation tends to focus on tests based on the AndroidTestCase and its subclasses, with commentators often calling them 'unit tests'. These tests are emphatically not unit tests: since they integrate with the Android system, they are at least integration tests, but while they are too embedded and slow to execute as unit tests, they are too tightly coupled to the code itself to be system testing. (Which said, on an Intel-based development machine, the emulator boots the Intel Atom Android images in under 15s, as opposed to 40s for ARM-based images, which makes it much more useful.)

System Testing

System tests exercise the system end-to-end: the application will be on the device and will talk to your server if you use one. They are slower to run than unit tests and are based on driving the application through the UI in a 'black box' way. They form part of Behaviour Driven Development.

Because they run on a phone, the compile/execute cycle of the unit test becomes a compile/package/install/execute cycle, but they run against the real implementation and thus don't suffer from the danger of implementation inaccuracy unit test stubs and mocks present.

The following tools may prove useful:

• Robotium - like Selenium for Android
• Sikuli - a screen-scraping execution system
• Monkeyrunner - a scripted UI driving tool
• Android SDK testing - the old test system provided with the Android SDK.

For random, long-running tests:

• Monkey - a random event generator, part of the Android SDK

Some commercial tools operate on your own equipment:

• SeeTest, EggPlant (TestPlant), Apphance

Some commercial tools operate on the tool provider's equipment:

• DeviceAnywhere

Web testing

Some applications are written for the browser, rather than against the SDK. For these, Selenium is supported - particularly under ICS with WebDriver.

Commercial testing

There are commercial companies to which you can submit your applications for testing on a variety of devices, so you don't have to own all the various tablets, phones, and web-enabled refrigerators people might want to run your application on. The good ones will offer implicit consulting by trying to sell you a variety of testing approaches in addition to the ones covered in this book: security, performance, usability, and localisation. The functional testing they will be able to offer will typically be restricted to black box testing, although it's not inconceivable someone might start to offer static checking and architectural advice.

A simple web search for 'Android testing' should bring up several adverts for such companies.

Testing testing

Tests are code, and can have bugs! Tools such as mutation testing, and condition, branch, or line coverage help determine how much of your code is actually and/or only seemingly tested.

Screenshots

In-app screenshots can be a useful debugging tool.^[2]