---
marp: true
paginate: true
math: mathjax
theme: buutti
title: 8. Testing
---
# Testing
## Contents
- [Introduction to testing](#introduction-to-testing)
- [C# Testing Frameworks](#c-testing-frameworks)
- [An extensive unit test example](#an-extensive-unit-test-example)
- [Writing testable code](#writing-testable-code)
- [Testing with Postman](#testing-with-postman)
- [CI/CD/CT](#cicdct)
## Introduction to testing
### What is testing?
* Testing is much more than just trying out manually if the program works!
* [Manual testing](https://en.wikipedia.org/wiki/Manual_testing): acting as an end user of the application
* [Automated testing](https://en.wikipedia.org/wiki/Test_automation): using software to emulate the end user, or testing the functionality of the code itself
* Proving that the program is working as intended during development on a...
* ...***micro level***: individual pieces of code, such as methods
* ...***macro level***: larger functional components, such as services
* *All possible paths in a piece of code should be tested*
* i.e., [test coverage](https://en.wikipedia.org/wiki/Code_coverage) should be close to 100%
### Automated testing: Why?
* Why do automated testing?
* Sets ground truths that can be relied on during development:
* *"With this kind of input, this should happen"*
* (No need to take the programmers word for it)
* Ensures code functionality before moving on
* Ideally finding bugs before starting to debug
* Guides developers to create more testable and thus more readable, interchangeable and overall better code
* [Test-driven development (TDD)](https://en.wikipedia.org/wiki/Test-driven_development) is based on writing tests first and code later!
### Types of automated testing
* Unit testing
* Testing individual units (methods, classes) within a project
* *"input x produces output y"*
* Each unit has to be entirely independent and ***deterministic*** for it to be unit testable!
* Deterministic: Always the same result with the same input
* Integration testing
* Testing complete modules and the interaction between them within a project
* Closer to real life scenarios
* End-to-end (E2E) testing
* Testing the whole program by emulating the end user's behaviour
### Test type comparison
* Unit tests are faster to write, E2E tests slowest
## C# Testing Frameworks
* Three main testing frameworks:
* `MSTest`
* `NUnit`
* `XUnit`
* You can install these using NuGet, or in the newer versions of Visual Studio you can create a new testing project
* NUnit is the most popular of the three, and will be used in this lecture
### MSTest
* Built-in to most versions of Visual Studio since 2005
* Simple to use, easy structure
* `TestClass` for denoting classes containing unit tests
* `TestMethod` for denoting unit test methods
### NUnit
* Most popular C# unit testing framework
* Very similar structure to MSTest tests
* Instead of `TestClass`, we use `TestFixture`
* Instead of `TestMethod`, we use `Test` / `TestCase`
* Has some advantages to MSTest or XUnit tests, for example:
* Ability to test a single test method using multiple different arguments
### XUnit
* Free, open-source unit testing tool for .NET framework and .NET core
* Made by the creator of NUnit v2
* You can test C#, F#, VB.NET and other .NET languages
* Part of the .NET Foundation
* Unique and different style
### Note about test case naming
* In C# unit testing, the most commonly followed naming convention for test cases/methods is as follows:
```csharp
FunctionNameThatIsTested_Condition_ShouldDoX
```
* For example:
```csharp
Fibonacci_WhenNIs0_ShouldReturn0
```
## An extensive unit test Example
### Creating a solution & sample project
First, we need a solution and a sample project.
1) In Visual Studio, create a ***Class library*** that targets *__.NET Standard__*.
2) Name the project `Fibonacci.Library` and the solution `Fibonacci`. Remember to uncheck _Place solution and project in the same directory_ if it isn't unchecked already.
### Adding the function to be tested
3) Rename the default `Class1.cs` file to `Fibonacci.cs`.
4) Allow Visual Studio to also rename the class itself to `Fibonacci`.
* Make the class `public` and `static`.
5) Add a *recursive fibonacci function* shown here.
```csharp
using System;
namespace Fibonacci.Library
{
public static class Fibonacci
{
public static long Recursive(
int n,
long previous = 0,
long current = 0,
int counter = 0
)
{
return counter == n ?
previous + current :
Recursive(
n,
current,
Math.Max(previous + current, 1),
counter + 1
);
}
}
}
```
### Adding a testing project
* Now we need to add a testing project to our solution.
6) Right click your solution, and add a new project. Select the NUnit project for **_.NET Core_**, and call it `Fibonacci.UnitTests`.
7) Rename the created class file to `Fibonacci_Tests.cs`, and the class to `Fibonacci_Tests`
### Linking the testing project
8) We then need to link the `Fibonacci.Library` project to `Fibonacci.UnitTests`.
9) Right click `Fibonacci.UnitTests`, and select *Add > Reference...*, and under *Projects* select `Fibonacci.Library`. Press OK.
* Now we have linked `Fibonacci.Library` to `Fibonacci.UnitTests`, and we can begin testing our `Recursive` function in the `Fibonacci` class.
### Importing the class
10) Since we have many things called "Fibonacci", we have to import our `Fibonacci` class by using an alias
* Put this to the top of your file `Fibonacci_Tests.cs`, underneath
`using NUnit.Framework;`
```csharp
using FibonacciLib = Fibonacci.Library.Fibonacci;
```
### Test fixture for a class
11) Now, add `[TestFixture]` on top of our class, like so:
```csharp
using NUnit.Framework;
using FibonacciLib = Fibonacci.Library.Fibonacci;
namespace Fibonacci.UnitTests {
[TestFixture]
public class Fibonacci_Tests {
```
* This denotes that this class contains our unit tests.
### Adding the unit test method
12) Then, inside our class `Fibonacci_Tests`, let's create our first method to test the `Recursive` function, as shown here:
```csharp
using NUnit.Framework;
using FibonacciLib = Fibonacci.Library.Fibonacci;
namespace Fibonacci.UnitTests {
[TestFixture]
public class Fibonacci_Tests {
[Test]
public void Recursive_WhenNIs0_ShouldReturn0()
=> Assert.That(0, Is.EqualTo(FibonacciLib.Recursive(n: 0)));
}
}
```
### Things to take into consideration
* When passing in function arguments, to avoid ***magic numbers*** and to improve code readability, use the syntax `argumentName: value`
* Magic numbers = using numbers as input as-is without explanation, for example without assigning to a well-named variable, or without using a descriptive type
* `Assert.That` is one of many functions that can determine whether the test passes
* Its first argument is the ***expected value***. In the second argument you give a ***constraint*** for the actual output, i.e., value is equal to, greater than, starts with, [etc](https://docs.nunit.org/articles/nunit/writing-tests/constraints/Constraints.html).
* *Do not mix the arguments up!* If you do, the errors you get do not make any sense.
### Running Tests
* Now, let's open the built-in test runner in Visual Studio to run our tests.
* On the top of Visual Studio, click *Test*, then *Test Explorer*.
* In the Test Explorer, you should see the test case we created. If not, build the solution, and make sure there are no errors.
* Either way, let's run our test case. Press *Run All Tests* in the top-left corner of Test Explorer. You should see our test case *__passing__*.
### Adding More Test Cases
* While there's nothing inherently wrong about our test case, it doesn't cover much of anything.
* We need to add more test cases to test different values and outputs.
* In other testing frameworks, you would do this by creating more methods.
* But with `NUnit`, we can use our *existing* method.
### Adding a new test
1) Do the following modifications to our test method:
```csharp
[TestCase(0, 0)]
public void Recursive_Always_ShouldReturnNthFibonacci(
int n,
long expected
)
=> Assert.That(expected, Is.EqualTo(FibonacciLib.Recursive(n)));
```
* Using `TestCase` instead of `Test`, we can pass in arguments to our test case method, and we can use those passed in arguments in our actual tests.
### Adding more tests
2) Run the test cases again, and they should *__pass__*.
* But as you might notice, we haven't really changed anything yet.
3) Let's add more test cases, like so:
```csharp
[TestCase(0, 0)]
[TestCase(1, 1)]
public void Recursive_Always_ShouldReturnNthFibonacci(
```
4) Now we have 2 test cases with only one test method. Run the tests again, and both cases should *__pass__*.
### Adding a failing test
5) Let's add one more TestCase, like so:
```csharp
[TestCase(2, 1)]
```
6) Run the tests again, and you should see the test case *__fail__*.
* This is to be expected: the 2nd fibonacci number should be 1, but the actual value we got from our function was 2.
* This means our function does not work as expected.
### Fixing our code
7) Let's fix our function, as seen here.
8) Run the tests again. You should see all *__pass__* now.
9) Add a few more test cases
* (If you don't know the fibonacci sequence, you can check it from the internet.)
```csharp
public static long Recursive(
int n,
long previous = 0,
long current = 0,
int counter = 0
)
{
return counter == n ?
current : // changed from previous + current
Recursive(
n,
current,
Math.Max(previous + current, 1),
counter + 1
);
}
```
### Exercise 1: Unit testing in practice
Answer the following questions:
1) How many test cases should we ideally have to test our `Recursive` function?
2) What happens if we pass in `n = int.MaxValue (2147483647)`?
* What about other huge numbers?
* How would you fix the function so it works as expected?
3) What happens if we pass in `n = -1`?
* What about other negative numbers?
* How would you fix the function so it works as expected?
### Answers to Exercise 1: Unit testing in practice
1) We should cover all the edge cases and code paths with our test cases. We do not need to have many test cases for the same code path.
2) `Stack Overflow`, `Int64 Overflow`, or the test case just won't run.
* We should have a maximum allowed value for `n`, throw the ***argument out of range*** exception if above
3) `Stack Overflow`, since `current` can never equal `n` that is less than `0`
* We should have a minimum allowed value for `n`, throw the ***argument out of range*** exception if below
### Testing exceptions
* To expect exceptions to be thrown in test cases, we can use the method `Assert.Throws`
* Syntax is as follows:
```csharp
Assert.Throws(delegate);
```
* In practice:
```csharp
Assert.Throws(() => TestedFunction(n));
```
### Handling border cases
* Continuing our Fibonacci test case, we need to add handling for `n` that is less than `0` or more than a set amount. Then we need to add another test method and two test cases to test our new functionality.
1) Let's start with modifying our existing `Recursive` function, as shown here:
```csharp
public const int MinDepth = 0;
public const int MaxDepth = 10000;
public static long Recursive(
int n,
long previous = 0,
long current = 0,
int counter = 0
)
{
if (n < MinDepth || n > MaxDepth)
throw new ArgumentOutOfRangeException(
"n is out of bounds!"
);
return counter == n ?
current :
Recursive(
n,
current,
Math.Max(previous + current, 1),
counter + 1
);
}
```
### Testing that everything works
* Now our `Recursive` function throws an `ArgumentOutOfRange` exception when `n` is less than `0` or more than `10000`.
2) Let's test that everything works as expected by adding a new test method and a few test cases:
```csharp
[TestCase(int.MaxValue)]
[TestCase(-1)]
public void Recursive_WhenNIsOutOfBounds_ShouldThrowArgumentOutOfRange(
int n
)
=> Assert.Throws(() => FibonacciLib.Recursive(n));
```
3) Rename the other method to `Recursive_WhenNIsInBounds_ShouldReturnNthFibonacci`
### Unnecessary arguments
* We have one more problem:
* Our `Recursive` function is a `public` method that has in total 4 arguments it takes, yet we only test 1 argument (`n`).
* In order for our tests to be complete, we need to test all the arguments a `public` method takes in.
* So, how do we fix this?
### Removing unnecessary arguments
* The naive way would be to test all the arguments with more test cases.
* It's important to realize that *__we don't have to__*.
* Since we never pass in any other arguments than `n` from outside `Recursive`, we could just create another `private` function that has those arguments, and remove them from our public function.
```csharp
public static long Recursive(int nStartingValue)
{
if (nStartingValue < MinDepth
|| nStartingValue > MaxDepth)
throw new ArgumentOutOfRangeException(
"n is out of bounds!"
);
long getRecursive(
int n,
long previous = 0,
long current = 0,
int counter = 0)
=> counter == n ?
current :
getRecursive(
n,
current,
Math.Max(previous + current, 1),
counter + 1
);
return getRecursive(nStartingValue);
}
```
### End Result
* Now we have tested all the arguments in our public `Recursive` function.
* We do not need to test private functions such as `getRecursive`, only the public API has to be tested.
* All tests should *__pass__* at this point.
### Exercise 2: Unit Testing
```csharp
public static long DoX(int nStartingValue) {
if (nStartingValue < 0 || nStartingValue > 150)
throw new ArgumentOutOfRangeException("n is out of bounds!");
long doY(int n) => n > 0 ? n * doY(n - 1) : 1;
return doY(nStartingValue);
}
```
* Following the previously used architecture as closely as possible, unit test the attached function using `NUnit`. Answer the following questions:
1) What does the function that we are testing actually *do*?
2) Do we have to test `doY` as well?
3) How would you name `DoX` so it is descriptive? How about `doY`?
### Answers to exercise 2: Unit Testing
1) It gets the nth factorial (`n!`)
2) `doY` is a private method that gets tested when we test `DoX`, so no.
3) Example names: `DoX` $\Rightarrow$ `TryGetFactorial`, `doY` $\Rightarrow$ `getFactorial`
### Unit Testing: Final testing code
*
```csharp
[TestCase(0, 1)]
[TestCase(1, 1)]
[TestCase(2, 2)]
[TestCase(3, 6)]
[TestCase(15, 1307674368000)]
public void TryGetFactorial_WhenNIsInBounds_ShouldReturnNthFactorial(
int n, long expected
)
=> Assert.That(expected, Is.EqualTo(TestExerciseLib.TryGetFactorial(n)));
```
```csharp
[TestCase(-1)]
[TestCase(151)]
public void TryGetFactorial_NIsOutOfBounds_ShouldThrowArgumentOutOfRange(
int n
)
=> Assert.Throws(
() => TestExerciseLib.TryGetFactorial(n)
);
```
## Writing testable code
* Writing testable code often means that you're writing good code
* Testable units should
* be deterministic (functional): same input always produces the same output
* be [loosely coupled](https://en.wikipedia.org/wiki/Loose_coupling): components have a loose and limited connection to other components in the system, meaning changes in one component should not require changes to many others
* abide to the *__Single Responsibility Principle__*: a function, method or component is only responsible for one purpose
### Writing testable code: Example 1
* Can `GetCurrentSeason()` be unit tested? Why / why not?
* How to fix the problems, if any?
```csharp
public static class Season
{
public static string GetCurrentSeason() {
DateTime currentTime = DateTime.Now;
if (currentTime.Month > 11 && currentTime.Month < 3)
return "Winter";
else if (currentTime.Month < 6)
return "Spring";
else if (currentTime.Month < 9)
return "Summer";
else
return "Fall";
}
}
```
### Writing testable code: Example 1 fixed
* Now the method is purely functional and deterministic
* Will return the same value for every call with equal arguments
* `GetCurrentSeason()` can now easily be tested as shown next
```csharp
public static class Season {
public static string GetCurrentSeason(
DateTime currentTime
) {
if (currentTime.Month > 11 && currentTime.Month < 3)
return "Winter";
else if (currentTime.Month < 6)
return "Spring";
else if (currentTime.Month < 9)
return "Summer";
else
return "Fall";
}
}
```
---
```csharp
public class GetCurrentSeasonTest
{
[TestCase(12, "Winter")]
[TestCase(3, "Spring")]
[TestCase(6, "Summer")]
[TestCase(9, "Fall")]
public void GetCurrentSeason_ForFirstMonthsOfSeasons_ReturnsCorrectSeason(
int month,
string season
)
{
Assert.That(season,
Is.EqualTo(
Season.GetCurrentSeason(new DateTime(2020, month, 1))
)
);
}
}
```
---
* The tests fail, as shown above.
---
* Test results indicate there is something wrong with the code path that should return `Winter`
```csharp
if (currentTime.Month > 11 && currentTime.Month < 3)
```
* The line should be changed to
```csharp
if (currentTime.Month > 11 || currentTime.Month < 3)
```
### Writing testable code: Example 2
* Let's look at a `HeatingUnit` class that returns a heating setting based on the current season. The current date problem is back again:
```csharp
public class HeatingUnit
{
public string GetHeatingSetting()
{
DateTime currentDate = DateTime.Now;
if (Season.GetCurrentSeason(currentDate) == "Winter")
return "HEAT_SETTING_HIGH";
else if (Season.GetCurrentSeason(currentDate) == "Summer")
return "HEAT_SETTING_OFF";
else
return "HEAT_SETTING_MEDIUM";
}
}
```
### Writing testable code: Example 2 continued
* Instead of taking the current date initialization even higher in the class hierarchy, let's make a ***service*** that returns the current date
```csharp
public interface IDateTimeProvider
{
DateTime GetDateTime();
}
```
* After this the service can be injected into `HeatingUnit` with constructor injection
* Code example shown next
---
```csharp
public class HeatingUnit
{
private readonly IDateTimeProvider _dateTimeProvider;
public HeatingUnit(IDateTimeProvider dateTimeProvider)
{
_dateTimeProvider = dateTimeProvider;
}
public string GetHeatingSetting() {
if (Season.GetCurrentSeason(_dateTimeProvider.GetDateTime()) == "Winter")
return "HEAT_SETTING_HIGH";
else if (Season.GetCurrentSeason(_dateTimeProvider.GetDateTime()) == "Summer")
return "HEAT_SETTING_OFF";
else
return "HEAT_SETTING_MEDIUM";
}
}
```
---
* For testing, a fake `DateTimeProvider` service can be used for injecting the `HeatingUnit` with any date:
```csharp
class FakeDateTimeProvider : IDateTimeProvider
{
public DateTime Date { get; set; }
public DateTime GetDateTime() => Date;
}
```
* Using this kind of a fake service instead of the real one is called [mocking](https://en.wikipedia.org/wiki/Mock_object)
* In the real application the provider would return the real current date
---
```csharp
class SetHeatingSettingTest
{
[TestCase(12, "HEAT_SETTING_HIGH")]
[TestCase(3, "HEAT_SETTING_MEDIUM")]
[TestCase(6, "HEAT_SETTING_OFF")]
[TestCase(9, "HEAT_SETTING_MEDIUM")]
public void SetHeatingSetting_ForFirstMonthsOfSeasons_ShouldReturnCorrectSetting(
int month,
string setting
)
{
FakeDateTimeProvider timeProvider
= new FakeDateTimeProvider { Date = new DateTime(2020, month, 1) };
HeatingUnit heatingUnit = new HeatingUnit(timeProvider);
Assert.That(setting, Is.EqualTo(heatingUnit.GetHeatingSetting()));
}
}
```
---
Tests should pass!
## Testing with Postman
* So far, we have created individual request with Postman to see what the response of an API is for each request
* This is not testing an API, this is *__exploring__* an API
* Tests in Postman are inserted in the _Tests_ tab
* Postman uses the *__Chai.js__* testing library for creating tests
---
* To get started immediately, you can select *snippets* from the right
* Let's select the *Status code: Code is 200* snippet
---
* Successfully run tests show up in green in the *Test Result* tab of the response:
---
* [The official Postman web page](https://learning.postman.com/docs/writing-scripts/test-scripts/) is a good starting point for learning testing with Postman
* Using `pm.expect` will give a bit more info about the test:
---
* The response to /serverinfo has the following body:
```json
{
"appSettings": {
"applicationUrl": "http://localhost:63741",
"aspNetCoreEnvironment": "Development"
},
"serverStatus": {
"dbConnectionStatus": "Connected"
}
}
```
* Let's make a test to see whether the server is connected to the database
---
* Variables can be declared within the test:
### Exercise 3: Testing with Postman
1) Create a new collection in Postman by selecting the *Collections* tab and clicking *New Collection*. Name it `CourseAPI Tests`. Launch the CourseAPI you have developed during the lectures, and make sure the `course_db` database is connected
2) Create a new `GET` request for the URI `/api/courses`. Add a test: status code of the request should be 200. Save the request to the `CourseAPI Tests` collection.
3) Create a new `GET` request for the URI `/api/courses/999999999`. Add a test: status code of the request should be 404. Save the request to the `CourseAPI Tests` collection.
---
4) Create a new `POST` request for the URI `/api/courses`. Set the headers and the content correctly, but set the value of credits to 100. Add a test; status code should be 400. Save the request to the CourseAPI Tests collection.
5) Hover on CourseAPI Tests collection, click the arrow and click *Run*. From the opened window, scroll down and click *Run CourseAPI Tests*. This will create all the requests in your collection.
## CI/CD/CT
* CI stands for *__Continuous Integration__*
* Each change in code triggers a build-and-test sequence for the given project
* Goal is to have a consistent and automated way to build, package, and test applications
* CD stands for *__Continuous Delivery__*
* Automates the delivery of applications to selected environments, such as Azure, AWS, GCP
* Both require *__Continuous Testing__*, which includes automated regression, performance and other tests