Appium Image Recognition Using Java — Devstringx
We are going to identify the graphics content of a mobile app using the Appium image recognition library.
We are using an image element (a base64-encoded image file) as a template to compare it with the image captured from a device screen.
Let’s see how this approach works.
Prerequisites
- Appium
- Java
- Nodejs
- Android Studio
- IDE
Get Started
Install the latest version of Appium
npmi -g appium
Install OpenCV image comparison library
npmi -g opencv4nodejs
Detailed installation guide for OpenCV library here
Methods Used
- Locate Images
publicMobileElementgetAllImage(AppiumDriver<MobileElement>driver, intimgCount) throws Exception {
By img = By.xpath(“//android.widget.Image”);
waitForElementVisibility(driver, img);
MobileElementcurrentImg = null;
List<MobileElement>imgs = driver.findElements(img);
for (inti = 0; i<imgs.size(); i++) {
if (imgCount == i) {
currentImg = imgs.get(i);
}
}
returncurrentImg;
}
- Capture Images From the App to Compare
public File doVisualCheckAllImages(AppiumDriver<MobileElement>driver, intimgCount) throws Exception {
String baselineFilename = VALIDATION_PATH + “/screenshots/app/img” + imgCount + “.png”;
File baselineImg = new File(baselineFilename);
File newBaseline = driver.getScreenshotAs(OutputType.FILE);
BufferedImagefullImg = ImageIO.read(newBaseline);
// Get the location of element on the pageto capture screenshot
Point point = getAllImage(driver, imgCount).getLocation();
// Get width and height of the element to capture screenshot
inteleWidth = getAllImage(driver, imgCount).getSize().getWidth();
inteleHeight = getAllImage(driver, imgCount).getSize().getHeight();
// Crop the entire page screenshot to get only element screenshot
BufferedImageeleScreenshot = fullImg.getSubimage(point.getX(), point.getY(), eleWidth, eleHeight);
ImageIO.write(eleScreenshot, “png”, newBaseline);
FileUtils.copyFile(newBaseline, new File(baselineFilename));
returnbaselineImg;
}
- Compare Result
publicvoidcompareResult(AppiumDriver<MobileElement>driver, String checkName, String checkName2)throws Exception {
SimilarityMatchingOptionsopts = newSimilarityMatchingOptions();
opts.withEnabledVisualization();
File Img2 = newFile(VALIDATION_PATH + “/screenshots/app/” + checkName2 + “.png”);
File Img1 = newFile(VALIDATION_PATH + “/screenshots/app/” + checkName + “.png”);
try {
SimilarityMatchingResultres = driver.getImagesSimilarity(Img1, Img2, opts);
// If the similarity is not high enough, consider the check to be failed
if (res.getScore() <MATCH_THRESHOLD) {
File failViz = newFile(VALIDATION_PATH + “/FAIL_APP_” + checkName + “.png”);
res.storeVisualization(failViz);
System.err.println(String.format(
“Visual check of ‘%s’ with ‘%s’ failed; similarity match is only %f, and below the threshold of %f. Visualization written to %s.”,
checkName, checkName2, res.getScore(), MATCH_THRESHOLD, failViz.getAbsolutePath()));
} else
// Otherwise, it passed!
System.out.println(String.format(“Visual check of ‘%s’ with ‘%s’ passed; similarity match is %f”,
checkName, checkName2, res.getScore()));
} catch (
Exception e) {
System.err.println(“Visual check of “ + checkName + “ with “ + checkName2+ “ failed; Both images are expected to have the same size in order to calculate the similarity score.”);
}
}
Test Run:
publicvoidverifyPictures() throws Exception {
app.doVisualCheckAllImages(driver, 0);
app.doVisualCheckAllImages(driver, 1);
app.compareResult(driver, “img0”, “img1”);
}
Recommended To Read — Mobile App Automation Testing Using Appium and Python
If you are interested in even more software testing-related articles and information from us here at Devstringx, then we have a lot to choose from for you.
Originally published at https://www.devstringx.com on September 26, 2022
