---
marp: true
paginate: true
math: mathjax
theme: buutti
title: 8. Testing
---

# Testing

<!-- headingDivider: 5 -->
<!-- class: invert -->

## Contents

- [Introduction to testing](#introduction-to-testing)
- [C# Testing Frameworks](#c-testing-frameworks)
- [An extensive unit test example](#an-extensive-unit-test-example)
- [Writing testable code](#writing-testable-code)
- [Testing with Postman](#testing-with-postman)
- [CI/CD/CT](#cicdct)

## Introduction to testing

### What is testing?

* Testing is much more than just trying out manually if the program works!
* [Manual testing](https://en.wikipedia.org/wiki/Manual_testing): acting as an end user of the application
* [Automated testing](https://en.wikipedia.org/wiki/Test_automation): using software to emulate the end user, or testing the functionality of the code itself
* Proving that the program is working as intended during development on a...
  * ...***micro level***: individual pieces of code, such as methods
  * ...***macro level***: larger functional components, such as services
* *All possible paths in a piece of code should be tested*
  * i.e., [test coverage](https://en.wikipedia.org/wiki/Code_coverage) should be close to 100%

### Automated testing: Why?

* Why do automated testing?
  * Sets ground truths that can be relied on during development: 
    * *"With this kind of input, this should happen"*
    * (No need to take the programmers word for it)
  * Ensures code functionality before moving on
    * Ideally finding bugs before starting to debug
  * Guides developers to create more testable and thus more readable, interchangeable and overall better code
  * [Test-driven development (TDD)](https://en.wikipedia.org/wiki/Test-driven_development) is based on writing tests first and code later!

### Types of automated testing

* Unit testing
  * Testing individual units (methods, classes) within a project
  * *"input x produces output y"*
  * Each unit has to be entirely independent and ***deterministic*** for it to be unit testable!
    * Deterministic: Always the same result with the same input
* Integration testing
  * Testing complete modules and the interaction between them within a project
  * Closer to real life scenarios
* End-to-end (E2E) testing
  * Testing the whole program by emulating the end user's behaviour 

### Test type comparison

* Unit tests are faster to write, E2E tests slowest

<div class='centered'>

![w:800px](https://microsoft.github.io/code-with-engineering-playbook/automated-testing/cdc-testing/images/testing-pyramid.png)

</div>

## C# Testing Frameworks

* Three main testing frameworks:
  * `MSTest`
  * `NUnit`
  * `XUnit`
* You can install these using NuGet, or in the newer versions of Visual Studio you can create a new testing project
* NUnit is the most popular of the three, and will be used in this lecture

### MSTest

* Built-in to most versions of Visual Studio since 2005
* Simple to use, easy structure
  * `TestClass` for denoting classes containing unit tests
  * `TestMethod` for denoting unit test methods

### NUnit

* Most popular C# unit testing framework
* Very similar structure to MSTest tests
  * Instead of `TestClass`, we use `TestFixture`
  * Instead of `TestMethod`, we use `Test` / `TestCase`
* Has some advantages to MSTest or XUnit tests, for example:
  * Ability to test a single test method using multiple different arguments

### XUnit

* Free, open-source unit testing tool for .NET framework and .NET core
* Made by the creator of NUnit v2
* You can test C#, F#, VB.NET and other .NET languages
* Part of the .NET Foundation
* Unique and different style

### Note about test case naming

* In C# unit testing, the most commonly followed naming convention for test cases/methods is as follows:
  ```csharp
  FunctionNameThatIsTested_Condition_ShouldDoX
  ```
* For example:
  ```csharp
  Fibonacci_WhenNIs0_ShouldReturn0
  ```

## An extensive unit test Example

### Creating a solution & sample project

First, we need a solution and a sample project.

<div class='columns23' markdown='1'>
<div markdown='1'>

1) In Visual Studio, create a ***Class library*** that targets *__.NET Standard__*.
2) Name the project `Fibonacci.Library` and the solution `Fibonacci`. Remember to uncheck _Place solution and project in the same directory_ if it isn't unchecked already.

</div>
<div markdown='1'>

![](imgs/9-testing_1.png)

</div>
</div>

### Adding the function to be tested

<div class='columns' markdown='1'>
<div markdown='1'>

3) Rename the default `Class1.cs` file to `Fibonacci.cs`.
4) Allow Visual Studio to also rename the class itself to `Fibonacci`.
    * Make the class `public` and `static`.
5) Add a *recursive fibonacci function* shown here.

</div>
<div markdown='1'>

```csharp
using System;

namespace Fibonacci.Library
{
    public static class Fibonacci
    {
        public static long Recursive(
            int n,
            long previous = 0,
            long current = 0,
            int counter = 0
        )
        {
            return counter == n ?
                previous + current :
                Recursive(
                  n,
                  current,
                  Math.Max(previous + current, 1),
                  counter + 1
                );
        }
    }
}
```

</div>
</div>


### Adding a testing project

* Now we need to add a testing project to our solution.
6) Right click your solution, and add a new project. Select the NUnit project for **_.NET Core_**, and call it `Fibonacci.UnitTests`.
7) Rename the created class file to `Fibonacci_Tests.cs`, and the class to `Fibonacci_Tests`

### Linking the testing project

8) We then need to link the `Fibonacci.Library` project to `Fibonacci.UnitTests`.
9) Right click `Fibonacci.UnitTests`, and select *Add > Reference...*, and under *Projects* select `Fibonacci.Library`. Press OK.

* Now we have linked `Fibonacci.Library` to `Fibonacci.UnitTests`, and we can begin testing our `Recursive` function in the `Fibonacci` class.

### Importing the class

10)  Since we have many things called "Fibonacci", we have to import our `Fibonacci` class by using an alias
      * Put this to the top of your file `Fibonacci_Tests.cs`, underneath<br>`using NUnit.Framework;`
        ```csharp
        using FibonacciLib = Fibonacci.Library.Fibonacci;
        ```

### Test fixture for a class

11) Now, add `[TestFixture]` on top of our class, like so:
  ```csharp
  using NUnit.Framework;
  using FibonacciLib = Fibonacci.Library.Fibonacci;

  namespace Fibonacci.UnitTests {
      [TestFixture]
      public class Fibonacci_Tests {
  ```

* This denotes that this class contains our unit tests.

### Adding the unit test method

12) Then, inside our class `Fibonacci_Tests`, let's create our first method to test the `Recursive` function, as shown here:
    ```csharp
    using NUnit.Framework;
    using FibonacciLib = Fibonacci.Library.Fibonacci;

    namespace Fibonacci.UnitTests {
        [TestFixture]
        public class Fibonacci_Tests {
            [Test]
            public void Recursive_WhenNIs0_ShouldReturn0()
                => Assert.That(0, Is.EqualTo(FibonacciLib.Recursive(n: 0)));
        }
    }
    ```

### Things to take into consideration

* When passing in function arguments, to avoid ***magic numbers*** and to improve code readability, use the syntax `argumentName: value`
  * Magic numbers = using numbers as input as-is without explanation, for example without assigning to a well-named variable, or without using a descriptive type
* `Assert.That` is one of many functions that can determine whether the test passes
  * Its first argument is the ***expected value***. In the second argument you give a ***constraint*** for the actual output, i.e., value is equal to, greater than, starts with, [etc](https://docs.nunit.org/articles/nunit/writing-tests/constraints/Constraints.html).
  * *Do not mix the arguments up!* If you do, the errors you get do not make any sense.

### Running Tests

* Now, let's open the built-in test runner in Visual Studio to run our tests.

* On the top of Visual Studio, click *Test*, then *Test Explorer*.
* In the Test Explorer, you should see the test case we created. If not, build the solution, and make sure there are no errors.
* Either way, let's run our test case. Press *Run All Tests* in the top-left corner of Test Explorer. You should see our test case *__passing__*.

### Adding More Test Cases

* While there's nothing inherently wrong about our test case, it doesn't cover much of anything.
* We need to add more test cases to test different values and outputs.
* In other testing frameworks, you would do this by creating more methods.
* But with `NUnit`, we can use our *existing* method.

### Adding a new test

1) Do the following modifications to our test method:
    ```csharp
    [TestCase(0, 0)]
    public void Recursive_Always_ShouldReturnNthFibonacci(
        int n,
        long expected
    )
        => Assert.That(expected, Is.EqualTo(FibonacciLib.Recursive(n)));
    ```
  * Using `TestCase` instead of `Test`, we can pass in arguments to our test case method, and we can use those passed in arguments in our actual tests.

### Adding more tests

2) Run the test cases again, and they should *__pass__*.
   * But as you might notice, we haven't really changed anything yet.
3) Let's add more test cases, like so:
    ```csharp
    [TestCase(0, 0)]
    [TestCase(1, 1)]
    public void Recursive_Always_ShouldReturnNthFibonacci(
    ```

4) Now we have 2 test cases with only one test method. Run the tests again, and both cases should *__pass__*.

### Adding a failing test

5) Let's add one more TestCase, like so:
    ```csharp
    [TestCase(2, 1)]
    ```
6) Run the tests again, and you should see the test case *__fail__*.
   * This is to be expected: the 2nd fibonacci number should be 1, but the actual value we got from our function was 2.
   * This means our function does not work as expected.

### Fixing our code

<div class='columns' markdown='1'>
<div markdown='1'>

7) Let's fix our function, as seen here.
8) Run the tests again. You should see all *__pass__* now.
9) Add a few more test cases
    * (If you don't know the fibonacci sequence, you can check it from the internet.)

</div>
<div markdown='1'>

```csharp
public static long Recursive(
    int n,
    long previous = 0,
    long current = 0,
    int counter = 0
)
{
    return counter == n ?
        current : // changed from previous + current
        Recursive(
          n,
          current,
          Math.Max(previous + current, 1),
          counter + 1
        );
}
```

</div>
</div>

### Exercise 1: Unit testing in practice
<!--_class: "exercise invert" -->

Answer the following questions:

1) How many test cases should we ideally have to test our `Recursive` function?
2) What happens if we pass in `n = int.MaxValue (2147483647)`?
     * What about other huge numbers?
     * How would you fix the function so it works as expected?
3) What happens if we pass in `n = -1`?
     * What about other negative numbers?
     * How would you fix the function so it works as expected?

### Answers to Exercise 1: Unit testing in practice
<!--_class: "exercise invert" -->

1) We should cover all the edge cases and code paths with our test cases. We do not need to have many test cases for the same code path.
2) `Stack Overflow`, `Int64 Overflow`, or the test case just won't run.
    * We should have a maximum allowed value for `n`, throw the ***argument out of range*** exception if above
3) `Stack Overflow`, since `current` can never equal `n` that is less than `0`
    * We should have a minimum allowed value for `n`, throw the ***argument out of range*** exception if below

### Testing exceptions

* To expect exceptions to be thrown in test cases, we can use the method `Assert.Throws`
* Syntax is as follows:
  ```csharp
  Assert.Throws<ExceptionType>(delegate);
  ```
* In practice:
  ```csharp
  Assert.Throws<NullReferenceException>(() => TestedFunction(n));
  ```
### Handling border cases

<div class='columns' markdown='1'>
<div markdown='1'>

* Continuing our Fibonacci test case, we need to add handling for `n` that is less than `0` or more than a set amount. Then we need to add another test method and two test cases to test our new functionality.
1) Let's start with modifying our existing `Recursive` function, as shown here:

</div>
<div markdown='1'>

```csharp
public const int MinDepth = 0;
public const int MaxDepth = 10000;

public static long Recursive(
    int n,
    long previous = 0,
    long current = 0,
    int counter = 0
)
{
    if (n < MinDepth || n > MaxDepth)
        throw new ArgumentOutOfRangeException(
          "n is out of bounds!"
        );
    
    return counter == n ?
        current :
        Recursive(
          n,
          current,
          Math.Max(previous + current, 1),
          counter + 1
        );
}
```

</div>
</div>

### Testing that everything works

* Now our `Recursive` function throws an `ArgumentOutOfRange` exception when `n` is less than `0` or more than `10000`.
2) Let's test that everything works as expected by adding a new test method and a few test cases:
    ```csharp
    [TestCase(int.MaxValue)]
    [TestCase(-1)]
    public void Recursive_WhenNIsOutOfBounds_ShouldThrowArgumentOutOfRange(
        int n
    )
        => Assert.Throws<ArgumentOutOfRangeException>(() => FibonacciLib.Recursive(n));
    ```

3) Rename the other method to `Recursive_WhenNIsInBounds_ShouldReturnNthFibonacci`

### Unnecessary arguments

* We have one more problem:
* Our `Recursive` function is a `public` method that has in total 4 arguments it takes, yet we only test 1 argument (`n`).
* In order for our tests to be complete, we need to test all the arguments a `public` method takes in.
* So, how do we fix this?

### Removing unnecessary arguments

<div class='columns' markdown='1'>
<div markdown='1'>

* The naive way would be to test all the arguments with more test cases.
* It's important to realize that *__we don't have to__*.
* Since we never pass in any other arguments than `n` from outside `Recursive`, we could just create another `private` function that has those arguments, and remove them from our public function.

</div>
<div markdown='1'>

```csharp
public static long Recursive(int nStartingValue)
{
    if (nStartingValue < MinDepth
    || nStartingValue > MaxDepth)
        throw new ArgumentOutOfRangeException(
          "n is out of bounds!"
        );
    
    long getRecursive(
        int n,
        long previous = 0,
        long current = 0,
        int counter = 0)
        => counter == n ?
            current :
            getRecursive(
              n,
              current,
              Math.Max(previous + current, 1),
              counter + 1
            );
    return getRecursive(nStartingValue);
}
```

</div>
</div>

### End Result

* Now we have tested all the arguments in our public `Recursive` function.
* We do not need to test private functions such as `getRecursive`, only the public API has to be tested.
* All tests should *__pass__* at this point.

### Exercise 2: Unit Testing
<!--_class: "exercise invert" -->

```csharp
public static long DoX(int nStartingValue) {
    if (nStartingValue < 0 || nStartingValue > 150)
        throw new ArgumentOutOfRangeException("n is out of bounds!");
    
    long doY(int n) => n > 0 ? n * doY(n - 1) : 1;
    
    return doY(nStartingValue);
}
```

* Following the previously used architecture as closely as possible, unit test the attached function using `NUnit`. Answer the following questions:
1) What does the function that we are testing actually *do*?
2) Do we have to test `doY` as well?
3) How would you name `DoX` so it is descriptive? How about `doY`?

### Answers to exercise 2: Unit Testing
<!--_class: "exercise invert" -->

1) It gets the nth factorial (`n!`)
2) `doY` is a private method that gets tested when we test `DoX`, so no.
3) Example names: `DoX` $\Rightarrow$ `TryGetFactorial`, `doY` $\Rightarrow$ `getFactorial`
  

### Unit Testing: Final testing code

*
  ```csharp
  [TestCase(0, 1)]
  [TestCase(1, 1)]
  [TestCase(2, 2)]
  [TestCase(3, 6)]
  [TestCase(15, 1307674368000)]
  public void TryGetFactorial_WhenNIsInBounds_ShouldReturnNthFactorial(
      int n, long expected
  )
      => Assert.That(expected, Is.EqualTo(TestExerciseLib.TryGetFactorial(n)));
  ```
  
  ```csharp
  [TestCase(-1)]
  [TestCase(151)]
  public void TryGetFactorial_NIsOutOfBounds_ShouldThrowArgumentOutOfRange(
      int n
  )
      => Assert.Throws<ArgumentOutOfRangeException>(
          () => TestExerciseLib.TryGetFactorial(n)
      );
  ```

## Writing testable code

* Writing testable code often means that you're writing good code
* Testable units should
  * be deterministic (functional): same input always produces the same output
  * be [loosely coupled](https://en.wikipedia.org/wiki/Loose_coupling): components have a loose and limited connection to other components in the system, meaning changes in one component should not require changes to many others
  * abide to the *__Single Responsibility Principle__*: a function, method or component is only responsible for one purpose

### Writing testable code: Example 1

<div class='columns23' markdown='1'>
<div markdown='1'>

* Can `GetCurrentSeason()` be unit tested? Why / why not?
* How to fix the problems, if any?

</div>
<div markdown='1'>

```csharp
public static class Season
{
    public static string GetCurrentSeason() {
        DateTime currentTime = DateTime.Now;
        if (currentTime.Month > 11 && currentTime.Month < 3)
            return "Winter";
        else if (currentTime.Month < 6)
            return "Spring";
        else if (currentTime.Month < 9)
            return "Summer";
        else
            return "Fall";
    }
}
```

</div>
</div>

### Writing testable code: Example 1 fixed

<div class='columns23' markdown='1'>
<div markdown='1'>

* Now the method is purely functional and deterministic
  * Will return the same value for every call with equal arguments
* `GetCurrentSeason()` can now easily be tested as shown next

</div>
<div markdown='1'>

```csharp
public static class Season {
    public static string GetCurrentSeason(
        DateTime currentTime
    ) {
        if (currentTime.Month > 11 && currentTime.Month < 3)
            return "Winter";
        else if (currentTime.Month < 6)
            return "Spring";
        else if (currentTime.Month < 9)
            return "Summer";
        else
            return "Fall";
    }
}
```

</div>
</div>

---

```csharp
public class GetCurrentSeasonTest
{
    [TestCase(12, "Winter")]
    [TestCase(3, "Spring")]
    [TestCase(6, "Summer")]
    [TestCase(9, "Fall")]
    public void GetCurrentSeason_ForFirstMonthsOfSeasons_ReturnsCorrectSeason(
        int month,
        string season
    )
    {
        Assert.That(season,
          Is.EqualTo(
            Season.GetCurrentSeason(new DateTime(2020, month, 1))
          )
        );
    }
}
```

---

<div class='centered'>

![](imgs/9-testing_2.png)

</div>

* The tests fail, as shown above.

---

<div class='columns32' markdown='1'>
<div markdown='1'>

* Test results indicate there is something wrong with the code path that should return `Winter`
  ```csharp
  if (currentTime.Month > 11 && currentTime.Month < 3)
  ```
* The line should be changed to
  ```csharp
  if (currentTime.Month > 11 || currentTime.Month < 3)
  ```

</div>
<div markdown='1'>

![](imgs/9-testing_3.png)

</div>
</div>

### Writing testable code: Example 2

* Let's look at a `HeatingUnit` class that returns a heating setting based on the current season. The current date problem is back again:
  ```csharp
  public class HeatingUnit
  {
      public string GetHeatingSetting()
      {
          DateTime currentDate = DateTime.Now;
          if (Season.GetCurrentSeason(currentDate) == "Winter")
              return "HEAT_SETTING_HIGH";
          else if (Season.GetCurrentSeason(currentDate) == "Summer")
              return "HEAT_SETTING_OFF";
          else
              return "HEAT_SETTING_MEDIUM";
      }
  }
  ```

### Writing testable code: Example 2 continued

* Instead of taking the current date initialization even higher in the class hierarchy, let's make a ***service*** that returns the current date
  ```csharp
  public interface IDateTimeProvider
  {
      DateTime GetDateTime();
  }
  ```
* After this the service can be injected into `HeatingUnit` with constructor injection
* Code example shown next

---

```csharp
public class HeatingUnit
{
    private readonly IDateTimeProvider _dateTimeProvider;
    
    public HeatingUnit(IDateTimeProvider dateTimeProvider)
    {
        _dateTimeProvider = dateTimeProvider;
    }
    
    public string GetHeatingSetting() {
        if (Season.GetCurrentSeason(_dateTimeProvider.GetDateTime()) == "Winter")
            return "HEAT_SETTING_HIGH";
        else if (Season.GetCurrentSeason(_dateTimeProvider.GetDateTime()) == "Summer")
            return "HEAT_SETTING_OFF";
        else
            return "HEAT_SETTING_MEDIUM";
    }
}
```

---

* For testing, a fake `DateTimeProvider` service can be used for injecting the `HeatingUnit` with any date:

  ```csharp
  class FakeDateTimeProvider : IDateTimeProvider
  {
      public DateTime Date { get; set; }
      public DateTime GetDateTime() => Date;
  }
  ```

* Using this kind of a fake service instead of the real one is called [mocking](https://en.wikipedia.org/wiki/Mock_object)
* In the real application the provider would return the real current date

---

```csharp
class SetHeatingSettingTest
{
    [TestCase(12, "HEAT_SETTING_HIGH")]
    [TestCase(3, "HEAT_SETTING_MEDIUM")]
    [TestCase(6, "HEAT_SETTING_OFF")]
    [TestCase(9, "HEAT_SETTING_MEDIUM")]
    public void SetHeatingSetting_ForFirstMonthsOfSeasons_ShouldReturnCorrectSetting(
        int month,
        string setting
    )
    {
        FakeDateTimeProvider timeProvider
            = new FakeDateTimeProvider { Date = new DateTime(2020, month, 1) };
        HeatingUnit heatingUnit = new HeatingUnit(timeProvider);

        Assert.That(setting, Is.EqualTo(heatingUnit.GetHeatingSetting()));
    }
}
```

---

<div class='centered'>

![](imgs/9-testing_4.png)
Tests should pass!

</div>

## Testing with Postman

* So far, we have created individual request with Postman to see what the response of an API is for each request
* This is not testing an API, this is *__exploring__* an API
* Tests in Postman are inserted in the _Tests_ tab
  * Postman uses the *__Chai.js__* testing library for creating tests
    ![](imgs/9-testing_5.png)

---

* To get started immediately, you can select *snippets* from the right
* Let's select the *Status code: Code is 200* snippet
  ![](imgs/9-testing_6.png)

---

* Successfully run tests show up in green in the *Test Result* tab of the response:
  ![](imgs/9-testing_7.png)

---

* [The official Postman web page](https://learning.postman.com/docs/writing-scripts/test-scripts/) is a good starting point for learning testing with Postman
* Using `pm.expect` will give a bit more info about the test:
  ![](imgs/9-testing_8.png)

---

* The response to /serverinfo has the following body:
  ```json
  {
      "appSettings": {
          "applicationUrl": "http://localhost:63741",
          "aspNetCoreEnvironment": "Development"
      },
      "serverStatus": {
          "dbConnectionStatus": "Connected"
      }
  }
  ```
* Let's make a test to see whether the server is connected to the database

---

* Variables can be declared within the test:
  ![](imgs/9-testing_9.png)

### Exercise 3: Testing with Postman
<!--_class: "exercise invert" -->

1) Create a new collection in Postman by selecting the *Collections* tab and clicking *New Collection*. Name it `CourseAPI Tests`. Launch the CourseAPI you have developed during the lectures, and make sure the `course_db` database is connected
2) Create a new `GET` request for the URI `/api/courses`. Add a test: status code of the request should be 200. Save the request to the `CourseAPI Tests` collection.
3) Create a new `GET` request for the URI `/api/courses/999999999`. Add a test: status code of the request should be 404. Save the request to the `CourseAPI Tests` collection.

---

<!--_class: "exercise invert" -->

4) Create a new `POST` request for the URI `/api/courses`. Set the headers and the content correctly, but set the value of credits to 100. Add a test; status code should be 400. Save the request to the CourseAPI Tests collection.
5) Hover on CourseAPI Tests collection, click the arrow and click *Run*. From the opened window, scroll down and click *Run CourseAPI Tests*. This will create all the requests in your collection.

## CI/CD/CT

* CI stands for *__Continuous Integration__*
  * Each change in code triggers a build-and-test sequence for the given project
  * Goal is to have a consistent and automated way to build, package, and test applications
* CD stands for *__Continuous Delivery__*
  * Automates the delivery of applications to selected environments, such as Azure, AWS, GCP
* Both require *__Continuous Testing__*, which includes automated regression, performance and other tests
