Appium Image Recognition Using Java — Devstringx

We are going to identify the graphics content of a mobile app using the Appium image recognition library.

We are using an image element (a base64-encoded image file) as a template to compare it with the image captured from a device screen.

Let’s see how this approach works.

Prerequisites

  • Appium
  • Java
  • Nodejs
  • Android Studio
  • IDE

Get Started

Install the latest version of Appium

npmi -g appium

Install OpenCV image comparison library

npmi -g opencv4nodejs

Detailed installation guide for OpenCV library here

Methods Used

  • Locate Images

publicMobileElementgetAllImage(AppiumDriver<MobileElement>driver, intimgCount) throws Exception {

By img = By.xpath(“//android.widget.Image”);

waitForElementVisibility(driver, img);

MobileElementcurrentImg = null;

List<MobileElement>imgs = driver.findElements(img);

for (inti = 0; i<imgs.size(); i++) {

if (imgCount == i) {

currentImg = imgs.get(i);

}

}

returncurrentImg;

}

  • Capture Images From the App to Compare

public File doVisualCheckAllImages(AppiumDriver<MobileElement>driver, intimgCount) throws Exception {

String baselineFilename = VALIDATION_PATH + “/screenshots/app/img” + imgCount + “.png”;

File baselineImg = new File(baselineFilename);

File newBaseline = driver.getScreenshotAs(OutputType.FILE);

BufferedImagefullImg = ImageIO.read(newBaseline);

// Get the location of element on the pageto capture screenshot

Point point = getAllImage(driver, imgCount).getLocation();

// Get width and height of the element to capture screenshot

inteleWidth = getAllImage(driver, imgCount).getSize().getWidth();

inteleHeight = getAllImage(driver, imgCount).getSize().getHeight();

// Crop the entire page screenshot to get only element screenshot

BufferedImageeleScreenshot = fullImg.getSubimage(point.getX(), point.getY(), eleWidth, eleHeight);

ImageIO.write(eleScreenshot, “png”, newBaseline);

FileUtils.copyFile(newBaseline, new File(baselineFilename));

returnbaselineImg;

}

  • Compare Result

publicvoidcompareResult(AppiumDriver<MobileElement>driver, String checkName, String checkName2)throws Exception {

SimilarityMatchingOptionsopts = newSimilarityMatchingOptions();

opts.withEnabledVisualization();

File Img2 = newFile(VALIDATION_PATH + “/screenshots/app/” + checkName2 + “.png”);

File Img1 = newFile(VALIDATION_PATH + “/screenshots/app/” + checkName + “.png”);

try {

SimilarityMatchingResultres = driver.getImagesSimilarity(Img1, Img2, opts);

// If the similarity is not high enough, consider the check to be failed

if (res.getScore() <MATCH_THRESHOLD) {

File failViz = newFile(VALIDATION_PATH + “/FAIL_APP_” + checkName + “.png”);

res.storeVisualization(failViz);

System.err.println(String.format(

“Visual check of ‘%s’ with ‘%s’ failed; similarity match is only %f, and below the threshold of %f. Visualization written to %s.”,

checkName, checkName2, res.getScore(), MATCH_THRESHOLD, failViz.getAbsolutePath()));

} else

// Otherwise, it passed!

System.out.println(String.format(“Visual check of ‘%s’ with ‘%s’ passed; similarity match is %f”,

checkName, checkName2, res.getScore()));

} catch (

Exception e) {

System.err.println(“Visual check of “ + checkName + “ with “ + checkName2+ “ failed; Both images are expected to have the same size in order to calculate the similarity score.”);

}

}

Test Run:

publicvoidverifyPictures() throws Exception {

app.doVisualCheckAllImages(driver, 0);

app.doVisualCheckAllImages(driver, 1);

app.compareResult(driver, “img0”, “img1”);

}

Recommended To Read — Mobile App Automation Testing Using Appium and Python

If you are interested in even more software testing-related articles and information from us here at Devstringx, then we have a lot to choose from for you.

Originally published at https://www.devstringx.com on September 26, 2022

--

--

Devstringx Technologies is highly recommended IT company for custom software development, mobile app development and automation testing services

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Devstringx Technologies

Devstringx Technologies is highly recommended IT company for custom software development, mobile app development and automation testing services