6 New Test Automation Frameworks for Android (2022)
New year, new tools! Just the right time to look into what’s new around test automation frameworks.
Top on the list, of the most used testing tools for Android, is still Espresso. So I’ll give a brief overview of developments, and then also introduce some new test automation frameworks.
Section overview
What’s new in Android Espresso?
Since Android Espresso is already very well known, I will not go into the basic description of the framework here. Instead, I have compiled a list of the features that have been released in the last few months (Jan 2020 —Feb 2022):
Espresso: 3.5.0 Alpha (October 4, 2021):
- Espresso’s DefaultFailureHandler now saves a screenshot on test failures to TestStorage
- Add new experimental APIs for screenshots: View.captureToBitmap, Window.captureRegionToBitmap, takeScreenshot()
- Add experimental Bitmap.writeToTestStorage API
- Save view hierarchy to a file on failures
- Enhance error handling in exception scenarios (app crashes, etc)
- Reports the exception when an error is thrown out of the test executor.
- Support for android API 31
Espresso: 3.4.0 (June 30, 2021):
- Add Espresso ViewMatcher APIs for negative conditions
- Allow unregistering Loopers from Espresso IdlingRegistry
- Support specifying junit RunListeners via java.util.ServiceLoader
- Support filtering by prerelease SDKs via SdkSuppress#codeName
- Add truth extensions for Location
- Add truth assertions for Bundle string and parceable arrays.
- Support ActivityOptions in ActivityScenario
- Make activity lifecycle transition timeout configurable
- Add a BundleMatchers.isEmpty() and isEmptyOrNull() methods
- Allow Intents.release without Intents.init
- Add ViewMatchers.isFocused()
On top of that there is a great cheat sheet available for download here.
Nothing has changed about the tiresome topic of idling resources. These are still necessary to keep espresso up to date about whether it needs to wait before executing the next step, for example because a background process has not yet been completed.
Kaspresso
Kaspresso is a test automation framework built on top of Espresso and UIAutomator. With a lot of useful additions. It was developed to improve some of the known weaknesses of Espresso:
- flakiness
- no ADB support
- difficult to read code
Let’s take a look how Kakao compares to pure Espresso. Here is a simple Espresso test example:
And here the same test in Kakao DSL syntax:
Because longer tests can become confusing, Kaspresso’s team has extended the Kakao DSL. This way we don’t have to care about what is under the hood of our tests, whether it’s Espresso or UI Automator.
A full test written with Kaspresso could look like this:
The speed comparison with a conventional UIAutomator (Appium) Test is also remarkable:
Kaspresso chooses a very sophisticated strategy (compared to Espresso or Barista) to improve the stability (avoid flakiness) of tests: Android’s event queue is monitored and commands are only sent when Android is in a calm state and there is a low likeliness that UI commands will be skipped because of lack of computation resources.
In addition, the testing framework also automatically closes system dialogs that sometimes cause a test to fail because they cover important elements of the app.
If you want to know more, the project is really well documented.
Strengths of Kaspresso:
- Greatly improved stability for tests compared to all other test automation frameworks
- Improved speed
- Abstraction and thus better maintainability for tests
- Open Source (110 PullRequests & 887 stars on Github)
Weaknesses of Kaspresso:
- DSL adds a further level of abstraction and requires some time to get to know and familiarize yourself with
- DSL can sometimes lead to confusing code. See an example here.
- Documentation and some issues are partly in Russian
Barista
Barista is a big extension of espresso. The stability (flakiness) is improved with a retry strategy: For certain tests, you can specify how often a test should be repeated if it fails. In addition, like Kaspresso, Barista uses an abstraction applied to PageObjects for UI components and test routines.
A simple example:
Fairly legible right? Obviously, some more work is needed to implement those methods. But still, the structure is very clean and readable:
Benefit: In case of a feature change in the app you might be able to avoid changing all of your 50 tests and instead only change the implementation of a single test method.
Strengths of Barista:
- Improved stability through auto retry and auto scroll
- Good readability thanks to an additional level of abstraction (Page objects)
- Very good documentation, easy to use
- Open source and active community (220 pull requests & 1222 stars on Github)
Weaknesses of Barista:
- Idling resources still have to be implemented manually
BusyBee
BusyBee is also an extension of Espresso. More specifically, an alternative for CountingIdlingResource is offered, which has some known difficulties.
One of them being debugging: If a test fails because CountingIdlingResource does not signal an idling state, it is often not clear why the exception happened.
Instead of just tracking the number of ongoing processes, BusyBee logs the specific, currently running processes. This is accomplished by tracking a Java object, a character string or an ID.
If a timeout occurs, the list of running processes can be output. This way debugging is getting easier.
Strengths of BusyBee:
- Simple tool that simplifies the handling of tiresome IdlingResources
- Open Source (11 pull requests, 143 stars)
Weaknesses of BusyBee:
- Small tool, no substitute for barista or kaspresso
Blueprint
Blueprint is a collection of Kotlin-based toolkits for bootstrapping modern Android codebases.
Doing a lot of Android projects, the Author of this framework has collected the most useful testing tools that work great in modern Android codebases. They follow a clean architecture and reactive paradigm, with a strong emphasis on automated UI testing.
Blueprint provides a number of independent artifacts that span a variety of use cases. However, they are all “optional,” meaning you can choose to utilize only the ones that are appropriate for your codebase.
Since we are covering mostly UI test automation in this article, I want to mention that blueprint offers dedicated extensions and widgets for working with the Android UI toolkit. But also RxJava v1–v3, Kotlin Coroutines, and the Robot Pattern are supported.
The Blueprint Testing Robot uses Espresso to perform view actions and assertions on platform UI widgets as well as the Material components, but it abstracts from the Espresso framework, the Android UI toolkit, and material components:
Strengths of Blueprint:
- Minimal footprint due to its well-structured library code with optional dependencies
- Support for a lot of modern components such as Material UI and Coroutines
- Great documentation
- Open Source (56 pull requests, 36 stars)
Weaknesses of Blueprint:
- A relatively small tool, still not very well known
Repeato
Repeato is a new NoCode test automation framework for Android. Tests can be created using a recorder and then played back. More specifically: During record, small “fingerprints” are recorded in the vicinity of the mouse pointer. On replay, these are used to find the corresponding UI element again.
There is also the possibility of creating “Script Steps” which allow you to execute ADB or other terminal commands.
And that’s how you record a test:
Afterwards, the recorded steps can then be played back:
Tests created with Repeato are stored in the file system and can also be versioned via Git as well as shared and edited within your team.
There is a free version, but advanced features such as continuous integration support or workspaces are not included.
The tool is particularly helpful as an extension for manual testing: Since developer resources are often scarce, it makes sense to outsource part of the test automation to the QA team.
Strengths of Repeato:
- Very easy to use, can be used by non-developers (QA team)
- Repeato does not need access to the source code of the app
- Free version available
- Visual reporting simplifies troubleshooting
Weaknesses of Repeato:
- no open source
- extended version is commercial (49 EUR per month)
- complex (if / else) assertions are not possible
Waldo
Waldo is a new cloud service that allows you to record tests using a recorder. It’s not yet available for Android, but I still wanted to give it a try, cause it looks very promising.
The developers have focused on offering a simple interface that also allows non-developers to create tests. The whole thing works, just as with Repeato, without a line of code.
A visual comparison of the original test run (everything worked) with the faulty test run (exception happened) makes debugging a breeze:
Strengths:
- Very clear and intuitive UI, easy to use
- Great animation, the llama signup is really fun!
Weaknesses:
- Price ($ 1999 USD / month)
- more complex use cases or data driven testing not possible
- Currently no Android support (“Coming soon”)
- The service does not seem to be over beta stage yet
Conclusio
Even with Espresso continuously being improved, there is still enough potential to make test automation more practicable, maintainable and intuitiv. Thus, new tools still emerge and also older alternative test automation frameworks are developed further. Unfortunately, there is no clear test winner. Each of the frameworks clearly has its strengths and weaknesses.
Like this article? there’s more where that came from.
Originally published at https://www.repeato.app on November 12, 2020.
