Test Execution | Bondar Academy
Course: Playwright API Testing with TypeScript
Module: API Testing Basics
Instructor: Artem Bondar
Lesson Summary
In this lesson, we explored various methods to run tests in Playwright . We previously utilized the Playwright extension in Visual Studio Code , which allows for convenient test execution through a user interface. Here are the key points discussed: Running Tests in Playwright You can run tests individually or all at once using the Test Explorer. Tests can also be executed via the terminal using the command: npx playwright test Understanding Test Failures During execution, we encountered failures due to concurrency issues when running tests in parallel across multiple browsers. Playwright defaults to running tests in Chromium , Firefox , and WebKit , which can lead to conflicts when tests modify shared data. Configuring Test Execution To avoid concurrency issues, we can disable parallel execution by setting fullyParallel to false in the Playwright configuration file. Tests can be executed sequentially, ensuring no conflicts arise from shared test data. Additional Terminal Commands Run a specific project: --project API testing Run a specific spec file: example.spec.ts Run a specific test by name: npx playwright test -g 'test name' Re-run only failed tests: npx playwright test --last-failed Test Annotations Utilize annotations to control test execution: test.only to run only specific tests. test.skip to skip a test. test.fixme for tests that need fixing later. This lesson provided a comprehensive overview of running tests in Playwright, focusing on both the graphical interface and terminal commands. Further exploration of the Playwright configuration will be covered in future lessons.
Video Transcript
In this lesson, we're gonna step back a little bit from coding, and I will show you different options how you can run tests in the Playwright. Okay, so let's get into it. So previously, we were using a Playwright extension to run our test. So there is a little checkbox right next to each and every test, and you can click on that and run the test. Also, on the left, you have the Test Explorer with the list of your tests. You can run your test individually by clicking the icon next to the test. You can run all the tests at once within the spec file like this, and all tests are executed. And if you have several spec files, you can go to the upper level and run everything in all spec files like that. So pretty useful and convenient interface, especially when you work in Visual Studio Code, debugging your test, working on writing new code, and so on. But that's not the only option. You can also use a terminal using the built-in terminal commands to run tests in Playwright. So let me show you some of those. So go back to the terminal, open a new terminal session. And the default command to run all Playwright tests is npx playwright test. So when you execute this command, Playwright will execute all tests that you have currently in your project. Okay, so let's run this and let's see what's gonna happen. So running this, and it's starting executing, and okay, whoa, what happened? So let's check it out. So let's go back first to the terminal. We see that a lot of tests failed. We have 422 error. We see nine passed and three failed. Even we have just four tests, but 12 tests were executed in total. So why did that happen? And also the report was generated automatically. It's open in default browser. We see some tests failed. And look, we have those marks, Chromium, Firefox, WebKit. Chromium, Firefox, and WebKit. Where did that came from? So going back to the Playwright, I open a Playwright configuration file. In Playwright configuration file, we have a section with the projects. So by default, Playwright configured to run UI tests. And by default, the projects are configured to run in different browsers, like Chrome browser, Firefox, and Safari. But we don't need to use browsers. We run just API requests. We don't need browsers anymore. And Playwright, by default, trying to run our test suite in all three browsers, running them in parallel. And we have all those issues just because we have a concurrency issues with the test data. So if I open any of those, look, we have expected new article modified, but received test to test article. I go back here, and I see some articles is created. So we just have a concurrency issue with the article names that we're creating during the test because we run tests in parallel in a different browser, and it just creates a mess. So let's clean this up. I delete the article, delete the test data, close this report, go back here. And as I mentioned before, we don't need browsers. So we don't need Firefox and WebKit. And I just comment this out on the Mac. It's command forward slash the hotkey to comment the block of the code. So let's keep only Chromium, and let's run this one more time. To stop serving the session here in terminal, click Ctrl C. All right, and now you can re-execute this command. How to quickly run the last command in terminal? Again, just simply click up arrow button on the keyboard, and the last executed command in the terminal, you will see it over there. And then click Enter, running it again, running test. Now we see only four tests, and still we have an issue, one test failed. Let's check it out, and we still have a concurrency issue. So why did that happen right now? So if we go back to the test, we see that, okay, which test failed, it's create, update, and delete article. So create and delete, create, update, and delete article. And it happened, again, because of the concurrency issue. So here, put request on this get call. We expect that the first article in the list should have a title, what is in our report, expected new article modified. But instead, we got test to test article, which is related to the previous test. We created right here in create and delete article. And this happened because by default, Playwright was running our test in parallel, so all four tests within a single test file were executed in parallel. So these two tests ran in parallel and created the concurrency issue creating two articles at the same time. That's why the order of the article was messed up, the order of the test, and the second test failed. So if I delete this article right now, so clean up the test data, close the reporting, and how we can fix this? So in Playwright configuration, also by default right here, fully parallel set to true. It means that tests within a spec file have to be executed in parallel. And let's check this as false, put the false flag. And parallel execution for UI testing kind of makes sense, because UI testing is slower. But for API testing, while it will give you some performance and execution of the speed, on the other side, you may have an issue with the concurrency, with the test data, or something like this. And since API tests are very, very fast, just naturally, in most of the cases, you will not need to run your tests in parallel. Because your test, even in sequential order, will be executed pretty fast. But if you need to run tests in parallel, you have to make sure that you don't have any concurrency with your test data. This is the most often case, while the tests become flaky. So now, we disabled the fully parallel, and now our test should be executed sequentially, one by one, as we normally did it before. So npx.playwright.test, run the test, and one, two, three, four. Here we go, now everything works correctly. Test executed one by one, we don't have any issues with the test data, everything works correctly. And I click arrow up, hit it again, and one, two, three, four. Everything works, so very convenient. So by using the terminal, you even don't need to run the VS code to execute your test. So on your computer, if you just open a new terminal session in the folder where your project is located, you just simply type npx.playwright.test, hit Enter, and tests are executed. And then if you want to see the report, you execute this command, npx.playwright.showReport, hit Enter, and report will be opened in the default browser like this. All right, so going back to the Visual Studio Code, and now let me show you some other commands that you can execute. So for example, you want to run tests for the specific project. So let's say in your playwright configuration file, you have several projects like Chromium, maybe some other projects. And by the way, let's rename this project to something more meaningful. It's not Chromium, for us it will be API testing, it's more meaningful. And also, we don't need a browser though, we don't use browser, so we just keep a project name, okay? After that, you can see the API testing project also was renamed in the test explorer, okay? And now we can run this project, dash, dash, project, and name of the project, API testing. Run it, and we successfully execute the project. So it can be useful if you, for example, group your test files by different projects, for example, project for regression test or project for smoke test, and you can run different projects like that. And about the playwright configuration, we're gonna talk a little bit later in details. What else you can do? You can run tests, for example, for the specific spec file. So currently, we have just single spec file, which is example.spec.ts. And you can type example.spec.ts, hit Enter. And all files related to the spec file is there. Sometimes your project may have more files than that, and most likely you will have many spec files grouping your test based on the meaning, or the flow, or the coverage, or the scenario. So you can run different tests by the spec file that way using command line. What else? You can also run a specific test by the test name, like this. So let's, for example, run the test, get test text. So the command would be npx playwright-test-g. And then I need to provide the name of the test, test-text, okay? And the single test is executed. And the last useful command is to run only failed tests. So let me show you how that works. So for example, I intentionally fail this test with 201 and execute just my default command to run all the tests. Running the test, and the first test expectedly failed, the report opened automatically, all right? Going back, and now I'm fixing this test back to 100. And I execute command last failed. So Playwright will automatically pick all tests that failed for the previous run. In our example, it's just a single test, and just rerun those tests. Execute, and just a single test was executed. So this is useful. So imagine you have, let's say, 100 tests. You executed your test suite on your computer, and found that 10 tests, for example, failed. And they are just randomly located in different spec files or something. So instead of manually picking those tests and trying to figure out why they didn't work, you can trigger this command, last failed, and give your test a second chance. If from the second attempt, your test passed, then most likely you have some data issue or something deeper to examine why your tests are flaky. So very useful command as well. And one more thing, what you can do with additional test annotations inside of the test. So let's say you have a test file, and you want to run only a specific test within this test file. You can use test annotation only, test.only. And in this case, when you execute just the general command npx.playwritetest, like this, only single test will be executed. If you will add the same annotation to the second test, only like this, then only those two tests will be executed. Another example, if you want, let's say, to skip the test from the test run, you have annotation test.skip. All right, and you can run the test. And now three tests executed inside of this file, and one is skipped. And if I open the playwright report, open in default browser, you can see the three tests executed, and the skip test is put into the separate tab as a skip. So it's kind of excluded from the general report. And maybe this test is flaky or something, or it's not relevant right now for your test, so you can just skip it. Or another useful annotation that behaves exactly the same as skip is fixme. So for example, you identify the test that for some reason is not stable, or you need later to come back to this test and fix it. And you don't want just to skip it, right? You want to be included in the test run, but just temporarily, you want to remove it from the execution with the annotation fixme. It will work exactly the same as a skip, just a more meaningful annotation for you later to review the code. You run the test, and this test also skipped. All right, so this was a quick overview how you can run your test using Terminal in Playwright. Later in the course, we will come back to the Playwright configuration file, and I will show you some other examples how you can configure the execution and the scope for your execution using projects and using the terminal commands. But for now, we are good to go, to move forward. All right, see you in the next lesson.