Automatic API Test Generation from UI Test | Bondar Academy
Course: Playwright API Testing with TypeScript
Module: AI-Assisted Scripting with Copilot
Instructor: Artem Bondar
Lesson Summary
In this lesson, we explore how to generate API tests based on UI tests using Playwright's CodeGen feature. This feature allows automatic generation of UI scripts by recording browser actions and capturing network activity, which can be saved in a HAR file. Key Steps in the Process Generate HAR File: Use the command npx playwright codegen --save-har networking.har to create a HAR file while performing actions in the browser. Record Actions: Sign in, create an article, and post a comment to capture the necessary API requests. Analyze HAR File: The generated HAR file contains extensive network data, including request details and responses. Clean Up HAR File: Filter out unnecessary data using a JavaScript function called HAR converter to retain only relevant API requests and responses. Prepare Instructions: Create a markdown file with processing instructions for the AI to generate the API tests based on the cleaned HAR file. Generate API Tests: Use the filtered HAR file and instructions to prompt the AI to create the API tests. Considerations While AI can assist in generating tests, it's important to evaluate whether crafting a perfect prompt is worth the time compared to manual test writing. AI is best utilized for repetitive tasks, while more complex scenarios may require manual intervention. In summary, this approach allows for efficient test generation by leveraging existing UI tests and automating the API testing process.
Video Transcript
Hey guys, welcome back and in this lesson, I will show you a pretty cool approach, how you can generate API test based on your UI tests. So Playwright has a feature of CodeGen, when Playwright can automatically generate the UI script for you based on the operation that you perform in the browser. You know that when you run the test in the browser, in the networking tab, you also have all network activity saved over there, what actually application sent to the API and what responses are coming back. When you use this CodeGen feature in the Playwright, it also can generate for you a HAR file that would save for you all this network information. So we can take this HAR file, modify it, and then feed it into the AI to generate API test based on this file. So that's the main idea. Well, let's try to do it. So here's our project. First of all, let's generate the HAR file based on the scenario that we're going to run in the browser. To trigger the CodeGen, we need to execute command npx playwright codegen, then dash, dash, and we need to generate the HAR file. Save HAR, and let's call this file, for example, networking.har, like this. We need to provide the URL, what application do we want to use? It will be our Conduit Boundary Academy app. So I copy the URL and paste it right here. So I think that's it. So yeah, let's run this. When we run this for the first time, Playwright will open this Playwright inspector. We don't care about the inspector, it will just show the steps of the UI test that will be generated, and also the browser like this. So now what we need to do is just to execute the scenario that we want to be recorded. So first of all, we need to do what? We need to sign in. So let's sign in with our account, pwapiuserattest.com, and password. We're logged in. Then let's say we want to create a new article. I provide some details about this article, publish the article, and then let's say that we want to add the new comment to the article. So I type something like, this is a first comment, and I click post comment, and this comment should be posted over here. So when we post the comment, a new API request to post the comment should be triggered by the browser, and this is probably the test that we want to automate. Create the article, post the comment, and then verify that article was created. So that's it. We completed the scenario. Now, when I close the browser, we see some notice. All right, that's fine. Going back to the File Explorer, and look, we have a networking.har has been generated. So let's look into this file. This is a huge file. If I scroll it all the way down, it has 2,800 lines of code, but it has everything we need related to the network. So if I make a quick search, API, for example, no, I make conduit-API. Here we go. This is the request, conduit-API to make a login. Then the next request was to make a user login, and then to make again this user login, then we make a GET request to get the articles. Look, we have all the details, headers, we have query string parameters, we have a response detail, status code. In the text over here, this is our response body, and all this stuff is recorded. So we can use this file to generate the API test based on this feeding it into LLM. But there is a little problem. You see that this file is giant. It's like 2,800 lines of code. It's too much. We can easily overload the context window for the LLM, and we need to clean this up, this file before. So for example, we don't need all these headers. This is unnecessary information for us because by default, we're going to use authenticated request. We don't need those headers. Also, there are some API calls that not related to our API requests as well. So let me try to show you fonts. For example, here, HTTP request to fonts.gstatic.com with some giant response that's completely unrelated to our API testing. So what we want to do before feeding this file into LLM, we want to clean this up and then only keep the networking responses and requests exactly to our flow and only then feed it to LLM. How are we going to do that? For that, we need to process it using just JavaScript. So I created this little nice JavaScript function, which I called HAR converter. What this guy does is basically read the input file over here, and then clean this up to making sure that only Conduit API URLs are used and do not use any request that has a JPEG at the end and so on. So we need to filter those out. Then we need to delete everything what is not relevant like cookies, HTTP version, body size, status text. So all those properties are irrelevant for the API testing. We want to clean this up. At the end, we want just to save this file on our computer to process this later. Then I call this function. Input is a networking.har and output file will be named filterhar.json. How to call it? Just call the new terminal and execute this file. So using Node.js, node and the name of the file, har-converter.js. All right, I ran this file and new file was generated. Look, filteredhar.json. Look, this file is significantly shorter. It has just 700 lines, so three times smaller than the original file. Well, and we can work with this file. So now we have filtered file that we can use and the next step, we need the instruction and we need a prompt, how to process all this stuff. I also prepared this in advance and here we go. So this is har-processing-instructions.md. That describes how do we want to process this file. It explains, okay, the request method located is over here. The URL is over here. Headers is over here. The request body is located under post data text and this is a JSON string. So how to read the data if needed and so on. Then request processing parameters. This is how to read all those values, URL, body, status code, response, and so on. This is how we extract the path from the URL because inside of our test, we just provide the path since the URL provided as the environment variable. Then how we instruct the query parameters as well. Then what else? Important resource fields, slug ID, comment ID, user ID, and the dynamic values. Any field that might be used in subsequent requests. So what we want to create is to create the test in the same order of the request that are in the HAR file. And if any of the fields in the response needed for the next request, we want the AI to use those values in the subsequent request. Then rules for variables naming, then how we test structure, then the example of the test, path, body, params method request. Then we need to add schema validation. Then how we extract values for the next request. We need to skip login request because we don't need authentication. Remove authorization headers. Dynamic value usage, again, for the articles request, for example, or the comments, we want to dynamically generate the data. And test data generation, we're gonna use Faker library. We also mentioned that we want to use it and how to use it. Then test quality standards, that should be schema, that should be response instruction, descriptive variables, error handling. The name of the spec file, how we want to name it, and the overall organization that we want, step one, step two, step three. So subsequent request based on the HAR file with all this user flow. All right, that's the instruction and the prompt. The prompt is generate test from the HAR, pretty simple. Convert each request to the framework format with schema validation. Identify response values needed to subsequent request. One test converted per complete API flow sequence. And the initial prompt, process the provided HAR file and create a single comprehensive API test. We also applied the instruction over here. And by the way, speaking about the instruction, previously one of the previous lessons for schema validation. We use this annotation, apply to spec.ts. So previously I mentioned that it is defining the scope where to apply these instructions. Actually, it's not. So this apply to means so if you add spec file into the context of LLM, of copilot, copilot will automatically will add this instruction. So if you will not use spec files or any of the spec files in the context of your LLM request, of copilot request, then the instruction will not be added. And when we create just a reference from the prompt, copilot will read this instruction and for some reason, it's not adding this explicitly into the copilot chat. So for the safe side, better to add this instruction to the copilot by yourself. Yeah, I think we're done. So let's go to the copilot and let's configure the task that we want to execute. So we want to do what? We need to add to the context filter HAR file. We need to add the instruction how to process the HAR file. I think we exclude from the context smoke test. And we need to call the prompt to generate test from HAR file. And I think that's it. Let's see what's gonna happen. So running this thing. And look, you see the filtered HAR file that JSON highlighted, because it's saying, well, we are exceeded the context window. Well, even if we cleaned this up, still we're exceeding the context window. Because Sana 3.5 had quite a limited context window. You might need to use more advanced and more expensive models which have a larger context window if you want to feed a bigger file. But I still think it should be able to do the work for us. So it's great, we found. And look, it's already generated the spec file. And here we go, this is our test. So let's check it out. So it's imported test, it's imported custom expect, why it's highlighted. And interesting that I found that all this time I had a misspell of how I named my custom expect file. Wow, that's interesting. By the end of the course, pretty much. So custom expect, okay, it's fixed. So and we have initial load of the articles, get the tags, create new article. All right, so create new article. Look, it's created the article with dynamically generated data, how we asked it to do, to use faker. Then it's extracted the slug ID over here, good. Then get created article using the slug ID, perfect. Should schema validation, validation that should match the title. Then get article comments. So it found a new API endpoint in the comments and add comment to the article. It's created a new body for the article and used this body over here. And also it used faker to generate this thing. So I think half of the work done, so we don't need this initial step so we can delete createx. We can remove this and start from the step number three. Create the article, get created article, get article comments, and add the comment. I think the last step we need to add something like this. So we can remove this from the context, add this to the context. I will keep it keep. And we need to add a validation that after we created the new comment, we need to make sure that this comment was created to get the article comments one more time. So I would say repeat step five one more time to validate that created article on the step six is exists in the response, add the assertion. And yeah, let's do this. All right, look, and also it worked, seems fine. So it made the get request, schema validation, and updated article response body to equal comment request, comment body. So let's keep this and let's run this test to make sure that it's working. So running the test, and test passed successfully. So I think we're gonna stop here. There are ways how to improve it, clean it up, update, and so on. But I think you got the idea, right? So you can take your existing UI test, run it, generate HAR file, and then convert this HAR file into the test. And then clean this up to make it actually the result that you are looking for. And also I want quickly to talk about what you should automate with AI and what you should not. So writing a perfect, well-crafted prompt and instruction take time. So ideally, you probably can craft the prompt and instruction that would nail the generation of the API test immediately from the first attempt. But the question, is it worth it? Is it worth the time spending on figuring out the perfect wording for your prompt to generate the test, or maybe just write the test manually, right? So you need to find the right tool for the right task. Something that AI can quickly help you and you don't need spend a lot of time on creating the instructions. Yeah, definitely. If you have a very repetitive workflows or some maintenance workflow or something like this, it's also really, really worth creating a very well-crafted instructions and prompts that you will be able to reuse and that way speed up your work. But in the situations like this, when you generate a new API test based on the UI test, maybe it's not worth spending a lot of time on creating ideal instruction and prompt. But maybe create a prompt like this that can give you initial boilerplate for your API test and the rest you can tweak by yourself. Adding the right assertions, adding the right schema validations, configuring how the request object should be done, dynamic data generation and all this stuff. So keep this in mind. Don't try to do everything through the AI. Use AI as a smart assistance for the repetitive in the routine task that otherwise would take you longer to do it manually. So that's my final point about this module. All right, I hope you liked it and see you in the next one.