In Memory Automated UI Testing ASP.NET Core

Introduction

In this article we look at how to run in-memory automated UI tests for an ASP.NET Core web app using Playwright and NUnit.

The article provides demo code and solutions to issues found along the way.

Automated UI Testing

Automated testing of any web application is essential to ensure it functions correctly.  On the top of the “testing pyramid” proudly sits UI testing or end-to-end testing, above integration and unit testing.  Automated UI testing involves launching and controlling a browser to drive through a set of interactions that a user would perform.  Assertions are made to see if the web app and browser behave as expected.  Browsers are controlled by testing tools such as Selenium,  Puppeteer or the new kid on the block, Playwright.

Normally, tests are run against a web application deployed to a test environment which has been configured with the same infrastructure as the production environment.  However, in this post we will be looking at running our UI tests against an in-memory instance of a web application.  By running the test in this way, we can:

• Develop tests faster.
• Reconfigure the app with test configuration.
• Catch some errors earlier in the dev/test process.
• Still run the tests against a deployed environment later.

Before we get started, we need to take a small step back.

Integration testing with WebApplicationFactory

Microsoft provide a neat way to run integration tests using an in-memory web server called TestServer.

Using the WebApplicationFactory class will provide you with a running TestServer.  It requires a type in the entry point assembly of the application. Typically the Startup or Program classes can be used.  The WebApplicationFactory class also provides a virtual ConfigureWebHost() method, which allows for the web host to be manipulated.  This is powerful and means services can be removed, added, or swapped.  For example, a SQL Server database can be removed and replaced with an in-memory database.

A unit test framework such as xUnit or NUnit can be used to write and run the tests targeting the app hosted in the in-memory TestServer.  When tests run they use an HttpClient connection given by the TestServer.  The HttpClient is a C# class and is not a browser, which means its a bit limited and inconvenient to use.

Using the TestServer approach, we have an in-memory web server at our disposal.  Not only that, it provides a way to override the way the webhost is built so that we can customise our application for our testing needs.  That sounds great! Unfortunately, when using a browser automation tool, such as Selenium or Playwright, the TestServer approach does not quite work.

Modifying Test Server to Work with Selenium and Playwright

After reading some really useful articles by Ben Day, Bertrand Thomas  and an oldie from Scott Hanselman (thanks guys!), I’ve derived a class from WebApplicationFactory which works with browser testing tools including Selenium and Playwright. In the demo project, this is the AutomatedTestServerFactory class.

One of the main issues when using a TestServer with a browser testing tool is that ConfigureWebHost() is not called.  This means the app cannot be changed at start up to use a test configuration.  In the “traditional” way of using TestServer, we are given an HttpClient instance for our request/response interaction with the server.  The HttpClient instance is created using the CreateClient() method in WebApplicationFactory, which in turn runs the ConfigureWebHost() method.  So, the really useful re-configuration we need is only done when an HttpClient is created! Browser testing tools effectively replace the client and are controlled independently of the TestServer.

To get round this issue, AutomatedTestServerFactory  sets the configuration when the webhost is built.  The work is mainly done in two methods CreateServer() and CreateWebHostBuilder(), which are called by the constructor.

CreateWebHostBuilder() is called first and sets the web host configuration, which is passed through as a constructor parameter.  Also optionally set is the environment, otherwise the default environment is Production, which may not be ideal.

CreateServer() builds and starts the web application and begins listening for requests. This method returns a null TestServer.  I didn’t see any impact of this and it avoids a new and different instance of the web app from being created.

With these changes in place, we have a WebApplicationFactory that creates an in-memory TestServer hosting our re-configured web application and is suitable for use with Selenium or Playwright.   Super!

Using Selenium or Playwright

When developing this project, I started using Selenium as the browser testing tool, however I noticed in October 2021, that Microsoft updated the integration test article (see above) to recommend Playwright instead of Selenium.  So I took a look, I liked it, and I have used Playwright for the remainder of the project.  Playwright is a relatively new end-to-end testing tool developed and maintained by Microsoft.

The reasons I liked Playwright include its simplification of browser driver management, that it works with Chromium, Firefox and Webkit browsers and the ease of use of the API compared to Selenium’s.  Out of interest, I found migrating tests from Selenium to Playwright to be straightforward too!

The decision to use Playwright also led to selecting NUnit as the test runner.  Typically, I use xUnit for testing, but it doesn't support running Playwright tests in parallel.

Configuring Playwright on local PC

To get Playwright to work, there are two steps required using Package Manager Console:

1. Install Playwright CLI as a global tool.
   • Run the command: install -g Microsoft.Playwright.CLI
2. Install browsers.
   • Navigate to the path of the test project and run the command: playwright install

Code Time!

In order to demonstrate this working, I’ve created a demo application available on GitHub.  The codebase has three parts as described below.

Employee System

This app is a very simple ASP.NET Core MVC application using EntityFrameworkCore with SqlClient. The only functionality is a homepage and a page returning a list of employees.  This is enough for our purposes though!  It has a migration script, so remember to run update-database before launching it.

This is what the Employee List page looks like:

Test code

The test code is comprised of two libraries, AutomatedTesting and Employees.UITests.  The AutomatedTesting library contains the custom AutomatedTestServerFactory class to enable in-memory testing with Selenium or Playwright. 

Employees.UITests is an NUnit test library.  It contains a base class, BaseTestClass, which is responsible for creating the web server and passing in the required configuration, in this case changing the database to in-memory and adding test data.  The base class is decorated with the attribute [Parallelizable(ParallelScope.Children)] to allow child tests to run in parallel.

The EmployeeListTests class inherits from BaseTestClass and focusses on the tests, so the code is quite clean. There are three tests, each one is the same, but using different browsers.  Running the test project will cause all three tests to run in parallel.  Finally, I’ve used FluentAssertions (as I like it!) for evaluating the outcomes of tests.

In the excerpt below, the Employee menu option is clicked and assertions are made on the output.

I have started to pull out some common tasks in TestHelper, such as taking a screenshot and attaching it to the test report.

After running the tests, you can see the results in the test log by right-clicking on a test and opening the log.

In Visual Studio 2019, the results include both the log messages and the screenshots.

Devops

I have included a yaml build pipeline for Azure DevOps.  This configuration deals with the Playwright requirements, builds the code and executes the tests in a cloud-hosted Windows server.  It can target a self-hosted Windows target too.

 

Resolved Issues

Developing this project was not without its issues.  Apart from getting the AutomatedTestServerFactory to work, here are the issues I found and overcame.

Issues found on Developer PC

Issue 1:

The web application launches, but is missing styling, JavaScript and images. This was because the test server would launch in the bin directory of the test project and not have a reference to the wwwroot folder. 

Solution: This was resolved by adding a post-build target to the Employees.UITests project file.  The target copies the wwwroot folder recursively from the Employees.UI project to the Employees.UITests\bin folder. 

Microsoft have an example of how to copy files recursively during after build.

 

Issue 2:

Microsoft.Playwright.PlaywrightException : Executable doesn't exist

Solution: One or more of the browsers is missing.  Run playwright install command in the Package Manager Console, but significantly make sure this is done in the directory of the test project.

Issues found in Azure Devops

Issue 3:

Install step of Playwright Tool causes error when re-run.  The following two lines in the log file indicate the issue.

Tool 'microsoft.playwright.cli' is already installed.
Error: The process 'C:\Program Files\dotnet\dotnet.exe' failed with exit code 1

Solution: Instead of calling dotnet install, run the dotnet update command as this removes and clean installs the tool.

Issue 4:

Error: Cannot find a tool in the manifest file that has a command named 'playwright'.

Solution: Make sure the playwright install command runs in the directory of the test project.

Issue 5:

Cloud-hosted Chrome fails with exception: Microsoft.Playwright.PlaywrightException : net::ERR_CERT_AUTHORITY_INVALID at https://localhost:5001/

Solution: Get a new browser instance to get round this issue using the code below.  This effectively runs the tests in incognito mode.

await browser.NewContextAsync(new BrowserNewContextOptions { IgnoreHTTPSErrors = true });

Issue 6:

Firefox does not navigate to test site.  Instead a warning is displayed in the browser

“Warning: Potential Security Risk Ahead”.

Solution: Firefox prevents navigation to another port on localhost.  The warning requires manual intervention, but the settings can be overridden so that the automated test can run.  To do so, set the following capabilities (these settings are briefly covered here):

caps.Add("security.insecure_field_warning.contextual.enabled", false);

caps.Add("security.certerrors.permanentOverride", false);

caps.Add("network.stricttransportsecurity.preloadlist", false);

caps.Add("security.enterprise_roots.enabled", true);

Other points:

1. Firefox not waiting for AJAX request to complete.

When waiting for a page to load that fires off an AJAX request, for example fetching data on page load, Firefox does not seem wait.  To resolve add await WaitForLoadStateAsync() after the command to load the page.

2. Azure DevOps only includes NUnit log on test failure.

NUnit provides the convenient TestContext.Out to log messages during testing.  These will always appear using the test runner in Visual Studio 2019, but these messages are only present in Azure Devops when a test fails.

Conclusion

In this article we have explored through the code, configuration and gotchas of running automated UI tests of an ASP.NET Core web app using Microsoft’s TestServer.  We have been able to perform UI testing using Playwright with multiple browsers running in parallel using NUnit.

We have seen a number of issues, mainly arising in Azure DevOps, and how to overcome them.

In-memory automated UI testing could be great for smaller systems and to help develop test code. It's not an approach for all cases, but certainly has its merits.

Finally, if you’ve not used Playwright for automated testing, perhaps it is a good opportunity to try it out!