Shadow Root and Canvas using OCR | Bondar Academy
Course: Mastering testRigor
Module: Interation with Web Elements
Instructor: Artem Bondar
Lesson Summary
This lesson focuses on the optical character recognition (OCR) feature in test automation, particularly for handling closed shadow DOM elements and canvas images. Key Concepts Closed Shadow DOM : A web design technique that isolates code, making it difficult for automation frameworks to interact with elements inside it. Optical Character Recognition (OCR) : A method used to recognize text within images, which can be essential for interacting with elements that are not directly accessible through standard automation techniques. Demonstration The lesson demonstrates how to use OCR to click a button labeled "open it" that resides within a closed shadow DOM. The initial attempt to click the button using a standard text descriptor fails due to the shadow DOM's restrictions. click open it using OCR, only using the mouse This command allows the test to recognize the button's text as an image and successfully click it. Additional Example Another example involves interacting with a pie chart rendered on a canvas. The values cannot be selected by text directly, but OCR can be used to recognize and validate these values: check that page contains France: 310 using OCR Summary In summary, when dealing with closed shadow DOMs or canvas elements, using OCR is a powerful technique in test automation. Always specify actions with OCR when the text is part of an image or not directly accessible.
Video Transcript
In this lesson, we will talk about test trigger feature called optical character recognition. Sometimes you may have a text on the page which is not actually a text. This text can be part of the image. And also you may have a situation such as closed shadow DOM, which is very difficult to handle for any test automation framework. It's very tricky. And using optical character recognition or OCR, you can handle the situations like this. So let's get into it. This is our models and overlays and the window page. And here we have a button open it. So with the title closed shadow DOM window. So if I click on this button, it works as expected, right? So it's open, some pops up and it works just fine. But what's the tricky part about this, that it will not work if you just try to click on this button by the text, because this button is located inside of the closed shadow DOM. So let me show you how to find this out. If I click inspect and look into the DOM, we have a shadow root with closed in parentheses. So shadow DOM and shadow root is a type of the web design where the web developer can isolate his code as much as possible from the entire application. So the CSS styles and the code itself will be kind of independent and embedded code inside of the existing application. And all the styles will be independent and will not be impacted by other style sheets and so on. So this technique is very rarely used actually in the real web design. But sometimes when it happens, it's actually a pain to automate this kind of stuff, but not in test trigger. So using OCR, I will show you how to do this. So let's start a new test. And let me copy the previous steps to navigate to the window page. And first of all, let me demonstrate you that the open it will actually not be clicked if I type something like click, open it. And let's run this test. All right. And our test expectedly failed. We have error, can't find element by descriptor, open it. But we clearly see that this text is displayed on the page. This is because of the closed shadow DOM. So how to handle this? So you need to update your current prompt with additional descriptor. So click open it using OCR, only using the mouse. So this is how it would look like. Using OCR is using optical character recognition and only using the mouse. We specify that we want to perform a mouse click instead of the JavaScript click. By default, test trigger will use a JavaScript clicking on the web elements just because it's a little bit faster. But in this type of situation, we need to use a natural mouse click. And let's run this test one more time. In this kind of situation, test trigger will look on open it button, not as a text. It will recognize the image displayed on this button as a text. So it will recognize this image, read it as a text, identify it and click on this button. And here we go, test pass successfully. It took a little bit longer because OCR a little bit slower. So we can see that open it was identified. And the next step, the shadow DOM window was opened. So everything worked as expected. Let me show you one more example where OCR can be used. So we navigate to the charts and e-charts page. So we have here a very nice pie chart. If I hover over different countries, we have a pop-up with whatever value and the percentiles that we have. Or if I hover over here, you can see it also pops up. Or if I click on the certain countries, it disappears. But if I make a right click and inspect, what I see here is just the canvas. So I do not see the actual values of the France, Germany, Canada. And so on. So I cannot select the values by text on this canvas as well. And this is another example when you're dealing with a canvas and situations like this, when you can use the optical character recognition to recognize the particular values on these types of images. So let's, for example, interact with this France and let's click on this France or this France. I don't know which one will be picked. And then we'll try to validate that this pops up, which is countries, you see it's floating, pops up if we can validate the value inside of it. So let's continue our scenario. So I need to navigate, click on charts and then e-charts. Okay, click charts and then click e-charts. And now let's just try to click on France. So click France using OCR, only using the mouse. And let's run this test. All right, something didn't work. Let's check it out. So click on charts, but charts was not open. And it was not open because we have this window displayed from the previous. So we need to close this window before clicking on the charts. And let's update the script real quick. So what do we need to do? Go to the window. And let's update the script. So let's update the script real quick. So what do we need to do? Go to the window, click on this. We need to click on this icon, inspect. And there is a data name close, and we can click close to the right of shadow DOM window. So let's add this step. So click close to the right of, and to the right of shadow DOM window. Here we go. And let's run this test one more time. All right, so let's check the screenshots. Now we were navigated to the canvas page. Click to the right. We can see this France was identified using OCR. And the next step, and we can see this pop-up with a value France and 310 is displayed on the screen. So now let's run this test one more time. So let's run this test one more time. And let's run this test one more time. And let's run this test one more time. So now let's try to validate this value using OCR as well. So now let's try to validate this value using OCR as well. So going back to the test and I type check that page contains. So going back to the test and I type check that page contains. And the value is, we need to exactly provide the value how it's displayed on the page. So it's France space, colon space 310. So it's France space, colon space 310. France space, colon 310 using OCR. And in this situation, we don't need to provide using the mouse because we are not performing any clicks or something. We just performing that validation. So let's run this test. So let's run this test. So let's run this test. So let's run this test. So let's run this test. So let's run this test. So let's run this test. So let's run this test. All right, test pass successfully. Let's check the screenshots. So the France was clicked over here. And then you can see the highlighted box with the value that we are asserting is also highlighting. So OCR recognized this value and made the assertion. So test pass successfully. All right, let's quickly summarize what we did in this lesson. When you're dealing with a closed shadow DOM or canvas, the optical character recognition is very useful technique to use in the test trigger. So if you want to perform any action, you need to use the prompt using OCR only using the mouse. And provide the text that you are looking for on the page using OCR. And if you want to make the assertions, you need to use OCR as well. If the text that you are asserting is not the text on the page and more of the text displayed on the image. All right, that's it guys and see you in the next lesson.