Visual Testing | Bondar Academy
Course: Playwright UI Testing with TypeScript
Module: Advanced Tricks and Techniques
Instructor: Artem Bondar
Lesson Summary
Visual Testing Overview In this lesson, we explore visual testing or visual validation using Playwright. This process ensures that the layout and colors of web elements are as expected by comparing screenshots. Setting Up Visual Testing Modify playwright-config.ts to include the default HTML reporter. Focus on UI components tests, specifically testing radio buttons. Set test retries to zero and use test.only to run a single test. Creating and Comparing Snapshots To validate the selection of radio buttons visually: Remove traditional assertions and use visual assertions. Create a snapshot using await expect(locator).toHaveScreenshot() . Run the test to generate a baseline screenshot. Subsequent runs will compare the current screenshot with the baseline. Handling Test Failures If the test fails (e.g., selecting a different option), Playwright will indicate the pixel difference. The report includes: Actual screenshot Expected screenshot Difference visualization Adjusting Screenshot Comparison To manage precision in comparisons: Set a maximum pixel difference (e.g., maxDiffPixel ). Use --update-snapshots to refresh all baseline images after major UI updates. In summary, visual testing with Playwright simplifies UI validation by using snapshots, allowing for flexible comparison settings to accommodate UI changes.
Video Transcript
In this lesson, we will talk about visual testing or visual validation. For example, for your application, it is very important to validate the layout of the particular page, that certain elements on the page located at the places where they should be, and the colors of those elements are exactly as expected. In this example, you can use the comparison between two screenshots or snapshots. And Playwright can do this comparison and make a validation if the layout of your application looks exactly as it should. So in this lesson, we will talk about this. Let's get into it. So in order to start, let me modify playwright-config.ts. In the reporters option, I will add back our default reporter to show you capabilities of visual testing in the report. So I add HTML reporter back and temporarily remove Allure reporter in order not to generate the excessive number of files. And for this demonstration, we will go back to UI components tests and we will use this test radio buttons, where we select different radio buttons option one or option two inside of the using the grid form. So for this test, I will change the retries, put it just to zero for this demonstration. And we will run just this single test, so I put test.only, something like that. So also, I will remove this second portion of the test where we change the option two. And instead of making the assertion using the normal way to be checked or not to be checked, let's use a visual testing to validate if the option one was actually selected. So how to do that? So I will remove these two assertions. We will not use it and we will use a visual assertion, visual testing. So in order to make a visual comparison against the snapshot, first, we need to create the snapshot of the web element that we want to make an assertion. So I type await, expect, put the web element locator, which is using the grid form, and call the method toPathScreenshot, that's it. So when you use this method for the first time and on the first run, this method will generate a screenshot as a baseline or a golden screenshot that will be used later for the comparison. So first of all, let's run this test first time. I'm running this test, option one is selected. And we can see the message is published over here that snapshot does not exist. And it's writing the actual snapshot to the folder. So closing this, and here under the test folder, created a new folder with the name of the spec file. And under the spec file, we have a new picture. So this is a snapshot of the desired behavior of our test. And if we run this test right now one more time, PlayWrite will make a comparison of the existing screenshot with a new screenshot that it's gonna get through the test and compare with the existing file. So running this test one more time, and right now test is passed. Over here, it's compared the current behavior with existing screenshot. And now let's make this test fail. What we need to do is change the desired behavior. For example, instead of selecting option one, we want to select option two. And run this test one more time. Running this test, and test is failed. And it says that screenshot comparison failed. And the difference is 192 pixels. If we go right now to the test results folder, here the folder test UI components forms layout, we have a few screenshots. The actual screenshot, which was like that. The expected screenshot, which is expected to select option one. And the different screenshot, which is showing, okay, what's the difference between the golden screenshot, that was our baseline and the actual result. And we can see the difference. And now let me run the same exact test using command line. So the HTML report will be generated. So I type npx play write test dash dash project Chromium. Run this test, it should fail. Write test is failed. And here is my report. The report shows exactly the same thing, that there is a difference in the pixels. And if we scroll this down, we see this picture, actual and expected. And what is nice in this report, that you have a such slider that shows you the difference. Okay, this is our initial result. And this is our actual result. And playwright shows the difference in the image. And you can even see that the rendering was slightly different. Using the grid looks a little bit different. But we have a different view of the page and using the slider, we can easy to understand, okay, what is the difference? All right. And also one more note, how playwright generate the name of the files. So as you can see, the screenshot is generated in the format, the spec file, name of the test, name of the project, which is Chromium and PNG. So this is created because if you run this in the different projects or in the different browser, the screenshot may look a little bit different. So let me show you that. For example, I trigger this in the Firefox. This will generate a new snapshot. Let me show you that. So the test failed because it's generated a new snapshot. And under the folder, we have a new screenshot and look at this. So it looks slightly different because slightly different fonts, slightly different position of the web elements and name of the file is also different. So based on the project name, it creates a new snapshot. So this is how it works. And sometimes you may have a situation when the test fails because of the too precise comparison of the screenshots. And you can add a setting right here to have a screenshot, to have a max pixel difference. And let's say we add 150 pixels. And if I run this test right now, for example, we are expecting what? This is the option one. And let's say we expect here the option two to be different with the screenshot. So I run this test in the default Chromium project. This test should fail because the screenshot is different and it shows the difference is in 204 pixels. And if we want to make the test not that precise, let's say we can put here 250 pixels accuracy of our screenshot comparison. In this case, if I run this test right now, this test will not fail because we said, okay, our comparison accuracy should be up to 250 pixels. This can be useful if you're dealing with unstable UI or if you're dealing with some kind of animation. And that so precise comparison between the screenshots is not necessary. And additionally, you can add the settings inside of the general configuration right here. Inside of the expect timeout, you can add additional settings to have a screenshot and let's say max different pixels. Let's say we set to 50 pixels. And the one last thing, let's say you have a major project update with a major UI update and you have a lot of screenshots that were created as a baseline for your projects. In order to quickly update all those screenshots to a new baseline, you can use this command, mpxplaywritetest, and use dash dash update snapshots. Running this command, you will execute all your tests and all the snapshots that were created as a baseline will be updated as a new assertion for your entire projects. So we are right now running this for all three projects. One failed, two passed. Which one has failed? Because the dev project was trying to navigate the URL that did not exist. Okay, let me update this to a more correct one. Run the same command one more time. And this should generate three snapshots as a baseline. And here we go. You can see under the folder, we created a baseline for the Chromium project, for dev project, and for Firefox project, which will be a baseline for a future run of the test framework. All right, so let's quickly summarize what we did in this lesson. Using a snapshot is a very easy way to compare the UI view of your web elements. When you run the test for the first time, it creates a golden screenshot or a base screenshot that will be compared against the next test run. Those screenshots are saved into appropriate folder under the test folder, and the name of the screenshot match the name of the project. You can additionally set the max different pixels comparison or accuracy of your screenshots in the pixels to not make your test really flaky. And additionally, you can set the setting in the general settings of the playwright.config to set as a global setting for the entire framework. If you want to update all snapshots in case the major update in the UI happened to your framework, you can run the test with the flag dash dash update screenshots to update all baselines for all the projects inside of your framework. All right, that's it, guys, and see you in the next lesson.