Multiple Test Reports | Bondar Academy
Course: Cypress UI Testing with JavaScript
Module: Advanced Features
Instructor: Artem Bondar
Lesson Summary
In this lesson, we discussed configuring Cypress reporters to improve test failure visibility. Running tests in open mode allows easy debugging, but using run mode limits visibility into failures. By default, Cypress provides minimal output in the terminal, making it challenging to diagnose issues. Key Reporters Covered JUnit Reporter : Highly compatible with CI/CD pipelines (e.g., Azure DevOps, GitHub Actions). It generates XML files for each spec file. Cypress Mocha Awesome Reporter : An HTML reporter that provides a user-friendly interface to review test results. We opted for an optimized version that automatically attaches screenshots. Configuration Steps Install necessary packages: Cypress multiple reporter and Mocha JUnit reporter . Update the cypress.config.js file to include the reporters. Create a reporter-config.json file to configure the reporters and specify where to save results. Add scripts to manage report cleanup and merging. We also created a custom script, cy run all , to automate the process of deleting old reports, running tests, and merging results. It is crucial to ensure that the reports folder is included in .gitignore to prevent accidental commits. In summary, we successfully set up a robust reporting system in Cypress, enhancing our ability to diagnose test failures efficiently.
Video Transcript
Hey guys, welcome back. In this lesson, we will talk about the reporters. So I think you noticed that when you run test in the open mode, everything is fine. When the test fails, you can easily debug your tests, scroll up and down, look into the logs, and understand why your test failed. But if you run your tests using just the run mode, and if something fails, you have kind of a lack of visibility of understanding what went wrong. So what Cypress provides you out of the box, just the output in the terminal with just some logs, but there are no any meaningful HTML report or any other report that helps you to understand actually what went wrong. You just have the terminal output, you have the screenshot in the screenshot folder, and the video if you activated the video, that's it. So in this lesson, we will configure the reporter. So in case the test fails, you will have a better understanding what went wrong. So let's jump into it. So going back to the Cypress documentation, where they describe how to configure their reporters. And the built-in capabilities, we have just this spec reporter that is built in into the Cypress in the command line. And also they say that they have also built-in reporters for TeamCity in JUnit, which I found a little bit confusing because JUnit worked for me only if I install this package separately into my project. And the TeamCity reporter is the CI-CD system, also quite popular in the past, not sure how about now. And this is also a reporter that is kind of built in. What we want to install in our project is a JUnit reporter because this reporter is highly compatible with all CI-CD pipeline whenever you run it in the Azure DevOps, GitHub Actions, or CircleCI. JUnit report will be pulled out by those systems easily and displayed on the dashboard. And second type of the reporter is the Mocha Awesome Reporter. This is a very nice HTML reporter that you can open in the browser and review everything what happened during the test. But we will not install exactly the clear version of the Mocha Awesome Reporter because of two reasons. First of all, it's harder to configure this reporter and also by default it does not attach screenshots to the reporter. Instead, we will use this optimized Cypress Mocha Awesome Reporter that is easier to configure. It's the same thing as original Mocha Awesome Reporter, but it attaches screenshots automatically. So let's begin. And since we want to use two reporters at the same time all the time when we run the test, I go back to multiple reporters section here in documentation and we configure this section. So let's go line by line and let's install needed packages into the projects. First of all, we need to install Cypress multiple reporter and Mocha JUnit reporter packages. So I'm copying this, going back to the project. And I install everything into conduit Cypress project that we used before. So opening the terminal and pasting this into terminal to install the packages. So packages are installed, you see two new packages added to the package.json. Cool, moving on. The next step, we need to update Cypress configuration. In the reporter, we need to add Cypress multi reporters and config file reporter config.json. So let me copy this and I paste it into config file. So let me put my reporter configuration settings just under the environments. Sounds good. So the next thing we need to create the reporter config.json. This file does not exist yet. So I create it in the root of the application. All right, and this file is empty. This is where we configure the reporters. Moving on, so these examples of the command line is just the alternative. If you don't configure in the config file, you can trigger the same exact config, just providing the flags in the command line. We configure everything just like on the configuration level, so we don't need this. The next step, reporter config.json is the reporters that we want to use, spec mochaj-unit-reporter, so two reporters at the same time. And where we want to save the results. So I'd simply copy this thing and paste it into reporter-config.json. All right, so we have spec-reporter and mochaj-unit-reporter. And results will be saved on the Cypress, results and results.hash. So why is it like this? When we run the test and Cypress generate, JUnit will generate reporters, it will generate XML file per spec file. So currently, we have two files in the project, so two files will be generated. And this hash is just the random number to make sure that this file is unique. But later, we need to combine those files together into a single reporter, so it will be a single, single file. So we will come into that in a second. Also, we need to add these two scripts. So these scripts is to delete Cypress results folder, delete previous reports. Because if you will not do this, the list of HTML reports, of XML reports of the JUnit, they will just accumulate in that folder. And that will create basically a confusion about your future test execution and past execution will be all combined together. We don't want this, so we want to control it by the scripts. So I'm adding this, so let me remove this dummy script. I'm adding this into my scripts file. I will use this scripts later to clean up the reports folder. So moving on, this is where they offer to use JUnit reporter merger to combine the reports together. But to be honest, I don't like this report merger, the way how it merge the reports. The thing is that it merged all reports into a single suite without differentiating between the spec files. I don't like it. Better to use a different package, which is JUnit merge, which doesn't have this issue, do the same thing but with correct structure in between the spec files. So let's install this package as well. So going back, Command V, and I added also the dash save dash def like this. Okay, another package is installed. And then let's create a new script. I call it JUnit merge. And how we gonna use it, so let's see. I need to call it npx JUnit-merge. Then let's look into documentation. So we need to specify which directory we need to use to merge all the reports. So dash d, I type dash d. And path to the folder is this one, cypress-results. Going back and cypress-results. And the next thing I want to, the file output, so dash o. And how I want my final result to be named and where I want it to be saved. So then I type dash o. And I want to be saved in the same folder like this. And I want to call it, I don't know, results.xml. This will be the final report. So I think that's it. Let's try to run this and we should see at least first JUnit reporter to be generated. So I run just npx cypress run. And two spec files should be executed right now. And JUnit report should be generated in the folder under the results folder. Let's see. So the first spec file was executed. Now running the second spec file. All right, all pass. We see a folder results generated and we have two XML files with the result. All right, this is working. Let's try the new script JUnit merge if it is working as well. So I call npm run and call the name of the script JUnit merge like this. Okay, and results.xml that combines both of those reports also generated successfully, perfect. Now let's check the pre-report that should delete the folder with the results. Npm run and I run pre-report script. And this should delete the results folder. All right, it's working perfectly. So we configured first reporter and we can continue to the next step. Now let's install the second HTML reporter which is Cypress Mocha Awesome Reporter. So going back and first of all, I need to install it into the project. So npm i save dev and I pasted it over here. So all right, new dependency was added into the project. Moving to the next step. The next step, I need to add this reporter into the list of the reporters like this. And we are using reporter config, so I'm adding this as an additional reporter over here in the reporter config.json, looks good. Then we need to add the require plug-in into setup node events in configuration file. Where's our configuration file? Setup node events right here. All right, this step is done. What's next? The next, we don't need this, we don't need, yeah, we need this. So add cypress support into e2e.js. I'm copying this thing. So e2e.js, we adding this guy, okay? This is also completed. What else? This is Cucumber preprocessor, we don't use it. This is the custom configuration, but I'm looking for how to use it with Cypress multi-reporters. I'm looking for this guide, cypress-config.js. And that's what I'm looking for. So if I use multiple reporters, I need to add this configuration into the reporter.config over here. So that's the second configuration. Okay, I need to add double quotes, double quotes, double quotes, and custom title, double quotes as well. And this one is not needed, this one is not needed. What else? I need a little bit more configuration, so let me go back. There was a nice example somewhere here. So these, I want to get all these options also as part of my configuration. Not just two properties, I want more properties. So double quotes, double quotes, double quotes, double quotes, double quotes. Removing this one, fixing all the compiling issues. Yeah, you see, initial configuration is quite, quite painful. All right, so why do I need this? Because I want the screenshots to be embedded into the report. You see, just by providing a single flag true, or if I want to save all the screenshots for all other events, either true or false, I can easily configure this over here. And for the custom title, let's change it to CanDoItReport. Something like this, CanDoItReport. And I guess that's all we need to do. So all this is configured, package.json, we added it over here. This is our scripts. So yeah, let's see, if we did everything right, so if I run the test, the report, the HTML report right now should be generated as well. Let's see, so running the test, test number one. All right, test number two, and everything is completed. And look, and for some reason, by the way, it's printing twice into the report. Probably we need to remove from the reporter configuration, we need to remove this one reporter. Probably CypressMockAwesomeReporter, also generating the inline reporter. And we have a new folder, which is reports folder. And in the results folder, we have the JUnit files. In the reports folder, we have HTML report. We will fix it in a second to be in just everything in a single place. But first of all, let's look into this report. Right-click Inspect, Open in Default Browser. And by the way, if you don't have this option, Open in Default Browser, you can install this package, Open in Default Browser. This is the extension I have in Visual Studio Code to have this right-click and open the HTML files right away. Okay, so right-click Inspect and Open in Default Browser. And here we go, this is our reporter. So look how nice it is. When you click on the report, it is showing the code that was executed. We have two spec files and it's pretty nice. So you can filter, show only past reports or show only failed reports. This one is inactive because we don't have failed reports yet. Looks good. So now let's make a little optimization. So instead of having two folders, reports folder and results folder, let's have everything under the reports folder. How about that? So going back to configuration and I am changing the JUnit reporter settings. I want to call it reports and under it, I will create a JUnit folder. So reports will have two subfolders, HTML folder with HTML report and JUnit folder with JUnit report. So JUnit over here, then package.json, we need to fix it right here. Results, here I'm changing from the results to reports. Reports, I'm changing to the reports here as well. And this will be JUnit, okay? And this will be reports and also the folder JUnit, JUnit like this. Okay, so the JUnit reports will be under reports folder. What else do we need? We need to create a new script that will be responsible for the entire process. So it will delete the initial folders. It will run all the tests and then it will merge JUnit reports. So let's create this script as well. So I will call it, for example, sy run all, let's call it like this. And then I call the npm run, first script will be pre-report. I want to make sure that nothing exists over there. I forgot colon here. So the next step, and I want to run npx cypress run. And also I want to add this or true. So what does this mean? Or true is needed, is that when this process is failed, I want the next step in the script still to continue. Otherwise, if the test fails, the next step in the script, which will be merge the report, will not be executed. So I add npx cypress run or true. And then the next step in the command will be npm run and JUnit merge. So I want to merge the JUnit reports after the execution. All right, I think we all set right now. Let me delete results folder. Delete, move to trash. Let me delete reports folders. Delete, move to trash. Screenshots, do we have something? We have some over there as well. Delete as well, move to trash. And let's try to run our new script. Cy run all, npm run sunrise all. And this guy should delete all the folders. It should run the test and it should generate two reports. Let's see. All right, no such file or directory. It's all right, running the tests. Execution, reports folder is created. Some of the tests fail and let's see. So reports have HTML and JUnit. JUnit has results that is merged and HTML. Let's see what we have in HTML. Open in default browser. And one of the tests failed for some reason. Yeah, and why did it fail? Okay, because our application has the user already created and assertion did not match. And look, isn't it cool that screenshot now is attached to the reporter. You can see the log, you can understand the error message. You can see the actual code that was executed. And now it is more clear that the test failed and why did it fail? So let me sign in to the app. Probably there is some test data left, why this test is failed. So I press Config, this is my account. Yeah, some test data is left over here. So going back. And now if I run this test one more time, reporters folder should be deleted automatically. No, yeah, it was deleted. It was deleted, nice. It's running test one more time. And now all tests pass, open in the reporters and we have nice index HTML report and JUnit report. No, it actually did not delete. So we have some mistake. Let's see. I see where's the problem. So, rm command would work only to delete the files within the reports folder. But currently, reports folder also have embedded folders inside. So to delete this, we need to add extra flag-r like this, and it will work. So let's try. Okay, now it was working correctly. Running the test. And reports generated HTML report, JUnit report, combined report, and this report open in default browser. All tests pass successfully. All right, guys, it was a long video. You see, you need to configure it just once. But once you did it, it works as expected. And by the way, one more thing, this command, rm-r, would work only on Linux, Mac, or Docker environments. Because Docker environments also run on Linux. If you use Windows, you need to use a different command like removedir or something to delete directory if you run this in the Windows shell. Otherwise, this command will not work if you are on Windows. The rest of this stuff will work exactly the same. So let's quickly summarize what we did in this lesson. We installed two reporters, multiple reporters, in the configuration file using Cypress multiple reporters and created reporter-config.json. And reporter-config.json, we configured two reporters, Mocha JUnit reporter and Cypress Mocha Awesome reporter. All reports generated under the reports folder, HTML and JUnit reporter. And also keep in mind that to use all this, you need to install all these packages into your project to actually be compatible. And then as a result, we created a custom script, cyrunall that delete previous reporters from our project. Then run npx cypress run to generate the report. And then the last step to merge JUnit reports. And I believe the final step that you want to do is add the reports folder into the gitignore reports. So you will not put this folder accidentally and not commit this folder accidentally into the source control. So it will be only on your computer or on the machine where the test was executed. All right, that's it guys, and I'll see you in the next lesson.