Visual Testing | Bondar Academy
Course: Mastering testRigor
Module: Advanced Tricks and Techniques
Instructor: Artem Bondar
Lesson Summary
In this lesson, we explore how to use Test Trigger for visual testing . This powerful tool simplifies the process of comparing application screenshots to identify changes. Main Commands There are four primary commands used for visual testing: compare screen compare screenshots These commands allow you to create a baseline screenshot and compare it with future screenshots to detect changes. Example Workflow We demonstrate the process using a login flow: Create a base screenshot of the application. Make modifications (e.g., add a new article). Run the test to compare the new screenshot with the baseline. If changes are detected, the test will fail, allowing you to see the differences. Adjusting Sensitivity Test Trigger is sensitive to minor changes, detecting differences as small as 0.03% . You can adjust the sensitivity threshold to avoid flaky tests: Set a threshold for acceptable differences (e.g., 0.002 ). Configure the severity of failures (e.g., minor, major). Resetting Baseline Screenshots To update the baseline screenshot: Comment out the existing baseline command. Run the test to create a new baseline. Uncomment the command to use the new baseline. Conclusion In summary, use compare screen or compare screenshots commands to establish baselines and detect changes. Adjust sensitivity settings to manage test failures effectively. This approach ensures reliable visual testing without unnecessary flakiness.
Video Transcript
In this lesson, we're gonna talk about how to use Test Trigger for visual testing. So this is very convenient and powerful tool, which is very easy to use. So let's get into it. Here in Test Trigger documentation, here's the section for visual testing. And these are basically main four commands that you're gonna need to use. These four two are basically identical. They are doing the same and I will show you later how to do this one. So for this example, we will use our login flow. Here quick edit and this is the login to application. And now using a comparison of the screenshot, we will make a modification to our application and then visually compare what was changed. For example, we will run the test creating the base screenshot, then creating new article, and then run this test again. So I copy this command, compare the screenshot, edit into the test, and run update and retest. So when I do this, Test Trigger will create a base screenshot for the homepage of our application. And then if something will be changed later on the homepage, this will be compared to the base screenshot. All right, test is passed, so we can see over here, compare screen was executed. And now going back to application and let's say I'll create a new article. Test, article, test, test. Publish the article, going back to the homepage and right now we have a new article created on the homepage. And now going back to the test, if I click retest, my application, now the homepage is a little bit different, because we have a new article added over here. And Test Trigger will compare the initial screenshot where this article did not exist with a new screenshot with this new article. And here we go, and test expectedly failed. So now if we click over here, we see the comparison. So step five, old screen, this is how it looked initially. And step five, new screen, this is how it looks right now. And by looking into the screenshots, you can see what's the difference and understand, okay, why the test is failed. And every time you rerun this test, Test Trigger will compare to the initial baseline screenshot all new screenshots related to the entire page. So how to reset the screenshot? You need to go to Quick Edit, and let's say that adding a new article to the application, this article, is the new expected result. And you want this to be a new baseline for your application. All you need to do is just to remove this step, update and retest, and create a new baseline screenshot that will be saved somewhere in the Test Trigger. All right, and now we can run it and create a new base screenshot. The new screenshot is created, now we go back to the application, and let's say we delete this article. Delete this article, and current base screenshot includes this article. And if we retest this test one more time, it should expectedly fail, because we deleted the article and UI changed on the home page. All right, and test expectedly failed, this is the old screen, and new screen does not have this article, so the test failed. Let me show you one more example, so I create a new base screenshot real quick. Base screenshot real quick without the article. Okay, now I uncomment it to create a new base screenshot, update and retest. All right, new base screenshot was created. And now I want to show you that Test Trigger is actually very, very sensitive to little, even tiny changes on the page. So it detects the changes by the pixel. For example, if I click on the Like button over here, you can see the color is changed and counter is also changed. And if I run this test one more time, click Retest. And the test has failed, so we compare in the screenshot. This is the old screenshot, this is a new one. So Test Trigger identified that the change was in this little counter. But of course, we don't want to have this level of sensitivity, right? Because it often can break the changes and make the test flaky. And you can adjust the sensitivity to make tests more stable. And even make the test to be not failed at all if the changes in the screenshot is identified, so how to do that? If I click on More Details and Show Error, see Extra, here we go. So this is the difference that Test Trigger identified between the screenshots. It's 0.03% difference between the images. And you can configure the threshold, how precise comparison of the images have to be. And each of the threshold, you can configure, okay, what's the level of severity for the failed test? This little change is just 0.03%. And we can adjust our comparison using this command. I'm copying this prompt, quick edit, and in here. So compare screen to previous version with allowance of 0, 0, let's say 2. So everything what will be above 0, 0, 2 will fail. For us, we can specify the level of the failure. So by default, when you use compare screen, Test Trigger will use a critical level of the failure. And this means that test failed and will be marked as failed. But if you want this little allowance and just be notified that, hey, we have some difference in the screenshot, and you want to be just informed about that, but not to create the failure for the test case, you can configure this kind of allowance. So in the current example, our screenshot has a difference of 0, 0, 3. But we set the threshold a little bit lower of 0, 0, 2. And when we run this test one more time, since the allowance is just 0, 0, 2, and our actual failure is a little bit higher, this failure will be marked as the minor error. And I comment this and run this test. Okay, this passed. A new base screenshot was created because we used a new command. I'm going back to application, remove the Like button, and rerun this test one more time. Retest. Here we go. And this is the result that we expected. So the test marked as passed, but still, it is showing you the difference between the screenshots. This is the old screenshot, and this is a new screenshot. You can see the difference in the Like counter was changed. But the test actually did not fail. And if you go to the errors over here, click on the Errors tab, you can see the screenshot difference, the severity is minor, and then you can review this difference. If you click on that, you can then review all your tests one by one, reviewing those minor differences. So this can be very convenient because if you have a dynamic UI, you don't want to be your test very flaky, and you want to add this extra level of the accuracy of how TestRigor will detect your screenshots. And as I mentioned before, by default, TestRigor set this CompareScreenshot command to a critical. So if any little change was detected, it will be marked as critical. So let's say you don't want any of your tests to be failed because of the visual testing. So you can override the default behavior by this prompt. Compare screen to previous version, 13 error as minor. So the test will be executed and screenshots that did not have a match, and for those that you didn't provide the allowance, will be just marked as a minor and will not fail all your tests. And then you can just review manually if you need to update the baseline or something like that. And the final option that you have, you can compare the screenshot with a stored screenshot from your application using this kind of prompt. But I would probably not recommend doing this because remember, screenshots are compared with accuracy by pixels. So if your screenshot that you want to compare with just a few pixels different to the screenshot of your real application, you either need to set up this allowance or you will have just a flaky test. So the best way would be to use one of these four commands to set up the allowance for the accuracy or overriding the entire thing instead of using the critical type of error, using minor or major type of severity. All right, so let's quickly summarize what we did in this lesson. So you can use commands compare screen or compare screenshots to create a baseline for the screenshots that you want to compare against. And for the next test run, if anything changed on the application, Test Trigger will automatically compare with the base screenshot and if something didn't match, the test will fail. To reestablish the base screenshot, you need to comment this line, run the test and then uncomment this line one more time to create a new baseline. You can set up the allowance of the accuracy and creating the threshold how severe the failure should be. If you don't want the test to be failed, you can configure this threading error as minor or major and then later review your screenshots manually. All right, that's it guys. And see you in the next lesson.