5 Snapshot Testing Tools for Android

While researching “Screenshot testing” and “Snapshot testing” for Android, I discovered that there are plenty of tools that enable you to get your job done.

I also discovered that “Snapshot testing” and “Screenshot testing” are used quite interchangeably.

So, is there a difference between “Snapshot testing” and “Screenshot testing”?

Yes, there is. “Snapshotting” is a more generic term. You could snapshot any type of content, such as text, images, structural information (JSON, XML), or even video. Screenshotting refers to only one kind of content, namely images.

So both techniques work in a very similar way:
First, the test engineer records the “good state” (AKA “golden value”) of the app. This could be done manually or automatically creating screenshots of UI elements in case of screenshot testing. Or saving text files in case of snapshot testing.

Then, later on, as the app needs to be validated, the test framework compares the “good state” of the app with the “current state”.

This could be a lot of work if done manually. Just give it a try. There are 10 differences in this pair of images:

Comparing images can be fun, but is a lot of work. Solution: Jump to the end of this article. Credits: https://www.boredpanda.com/quiz-spot-the-differences-images-langweiledich

For a computer, this task is quite straightforward: It’s usually done with a simple diff algorithm. Each “current pixel” is compared with its “good pixel”-bro. If there is a deviance, the test framework will notify the tester about the changes.

Fortunately, we don’t need to know how exactly the comparison works. That’s taken care of by the tools available for snapshot testing.

1. Repeato

Repeato is a mobile test automation tool that allows users to create screenshot tests without coding skills. It comes as a dedicated desktop application that can be used to record and run tests.

Here is a video that shows how screenshot tests can be created:

Creating a screenshot test with Repeato

The report looks like this:

Screenshot testing report generated with Repeato

And this is what a screenshot comparison looks like when you hover over the green button of the individual test step:

Screenshot testing result in Repeato: Visual diff between the original version, and the last run version

Pros:

• very easy to use for non-technical people like QAs
• possibilities to set comparison regions and thresholds for each check individually or for whole tests
• works also on iOS and application frameworks like Flutter

Cons:

• not Open Source (Free version available though)

2. DropBox Dropshots

Dropshots is a Gradle plugin that simplifies on-device screenshot testing.

Why “on-device”? Other snapshot testing libraries capture screenshots while running your instrumentation tests on the device, download the images to your host computer, and then compare the new images to the reference images. This introduces a couple of complexities since other tests are typically run directly on the device and some sort of synchronization mechanism is needed to run all sorts of tests from within your IDE.

By running your screenshot assertions together with all of your other tests in your suite, Dropshots streamlines this procedure. The Gradle plugin makes sure that your version-controlled reference photos are accessible on your test device so that your test screenshots can be compared to those reference images immediately within your test.

This is what a test looks like:

class MyTest {
  @get:Rule
  val activityScenarioRule = ActivityScenarioRule(TestActivity::class.java)
  
  @get:Rule
  val dropshots = Dropshots()
  @Before
  fun setup() {
    activityScenarioRule.scenario.onActivity {
      it.supportFragmentManager.beginTransaction()
          .add(android.R.id.content, ScreenshotTestFragment())
          .commitNow()
    }
  }
  @Test
  fun testMatchesActivityScreenshot() {
    activityScenarioRule.scenario.onActivity {
      dropshots.assertSnapshot(it, "MatchesActivityScreenshot")
    }
  }
  @Test
  fun testMatchesViewScreenshot() {
    activityScenarioRule.scenario.onActivity {
      dropshots.assertSnapshot(
        it.findViewById<View>(android.R.id.content),
          name = "MatchesViewScreenshot"
      )
    }
  }
}

Pros:

• Simplifies screenshot handling quite a bit
• Open source
• Can be run on physical devices

Cons:

• Less dev power than e.g. Paparazzi or Shot (127 commits, 245 ⭐️)

3. CashApp Paparazzi

What a great name! Paparazzi is a Gradle library to render your application screens without a physical device or emulator.

This is done by using the true rendering pipeline of Android on a JVM.

The obvious big benefit of this approach is: It’s crazy fast. You don’t need to build your app and you don’t need to start an AVD. That’s probably 5–10 minutes of time, each time you run your tests.

And this is what a Paparazzi test could look like:

class LaunchViewTest {
  @get:Rule
  val paparazzi = Paparazzi(
    deviceConfig = PIXEL_5,
    theme = "android:Theme.Material.Light.NoActionBar"
  )

  @Test
  fun launchView() {
    val view = paparazzi.inflate<LaunchView>(R.layout.launch)
    // or...
    // val view = LaunchView(paparazzi.context)
    view.setModel(LaunchModel(title = "paparazzi"))
    paparazzi.snapshot(view)
  }

  @Test
  fun launchComposable() {
    paparazzi.snapshot {
      MyComposable()
    }
  }
}

As you can see, you can simply specify the device configuration of the desired device. The tool takes this information, plus the configured dummy data of the model, to render the views accordingly before saving the screenshot under a specific name.

The documentation is quite minimal, but here is a great talk by John Rodriguez, one of the authors of Paparazzi. There are also a couple of more test examples here.

Pros:

• Fast, incredibly fast
• 100% Kotlin
• Compatible with Jetpack Compose
• Open source
• Strong dev team (816 commits, 1400 ⭐️)

Cons:

• limited to the most important device configs (Pixel series)
• real devices might render in fact differently
• developments of the Android rendering stack might break your test setup

4. Shot

Shot is an Android project that allows you to write screenshot tests for your apps in a simple and friendly way.

Shot provides a handy interface called `ScreenshotTest` and a ready-to-use `ShotTestRunner` that you can use to write tests. Here is an example of how to use it:

// If you are using regular Android views
@Test
fun theActivityIsShownProperly() {
  val mainActivity = startMainActivity()
  compareScreenshot(activity)
}

// If you are using Jetpack Compose
@Test
fun rendersGreetingMessageForTheSpecifiedPerson() {
  composeRule.setContent { Greeting(greeting) }
  compareScreenshot(composeRule)
}

To get started with Shot, you need to add the Shot plugin to your Gradle file and include the Shot library in your app’s dependencies.

Here you can see Shot in action:

And this is how an error result looks like:

Error message showing a screenshot testing result

Pros:

• Compatible with Jetpack Compose
• Tests can be run on many devices simultaneously
• Open source
• Popular: (314 commits, 1100 ⭐️)

Cons:

• If you want to use Shot on devices running API >= 28, you will have to enable access to non-SDK interfaces. That requires ADB access to your device

5. Android Testify

Android Testify is a Gradle plugin that allows you to add screenshots to your existing Android tests, expanding test coverage and monitoring the quality of the UI experience. It provides an easy way to review changes to the UI and establish a comprehensive set of screenshots for the application.

Here is an example:

@RunWith(AndroidJUnit4::class)
class MainActivityScreenshotTest {

    @get:Rule val rule = ScreenshotRule(MainActivity::class.java)

    @ScreenshotInstrumentation
    @Test
    fun default() {
        rule.assertSame()
    }
}

Testify operates by using a PNG baseline located in your androidTest/assets directory for every test case you create. As you develop and execute tests, a refreshed baseline image is kept on your device or emulator. To modify the baseline, it is necessary to transfer or retrieve the image from the device to your local development setting. Testify provides several Gradle tasks to make the transfer of your baseline images more straightforward.

This is how a report looks like in Android Testify

Pros:

• Open source
• well maintained (558 commits)

Cons:

• smaller community than other frameworks (68 ⭐️)

Conclusion

There is a variety of tools to do the job. Mostly it depends on whether you have technical staff available to make an additional effort of setting up a screenshot-testing pipeline.

While I was doing the research for this article, I didn’t find any more useful screenshot-testing tools for Android than the ones mentioned here. But you might know one. Please let me know in the comments, I’d be happy to add it to the list.

Oh, and by the way. The solution to the “find the difference star track image” is this one:

1. Geordi’s commbadge is missing 2. Data’s ear is too pointy 3. The wall near Worf’s head is missing an indentation 4. Deanna’s hair has an extra set of curls (this one took forever to spot!) 5. Deanna’s cleavage is more modest 6. Picard’s got an extra pip on his collar 7. Beverly’s hair is missing a curl (also took forever to spot!) 8. Riker’s right hand is missing 9. Riker’s commbadge is upside-down 10. There’s some detailing missing on the lower-right step of Ten Forward