Testing

Contents

Introduction to testing

What is testing?

  • Testing is much more than just trying out manually if the program works!
  • Manual testing: acting as an end user of the application
  • Automated testing: using software to emulate the end user, or testing the functionality of the code itself
  • Proving that the program is working as intended during development on a...
    • ...micro level: individual pieces of code, such as methods
    • ...macro level: larger functional components, such as services
  • All possible paths in a piece of code should be tested

Automated testing: Why?

  • Why do automated testing?
    • Sets ground truths that can be relied on during development:
      • "With this kind of input, this should happen"
      • (No need to take the programmers word for it)
    • Ensures code functionality before moving on
      • Ideally finding bugs before starting to debug
    • Guides developers to create more testable and thus more readable, interchangeable and overall better code
    • Test-driven development (TDD) is based on writing tests first and code later!

Types of automated testing

  • Unit testing
    • Testing individual units (methods, classes) within a project
    • "input x produces output y"
    • Each unit has to be entirely independent and deterministic for it to be unit testable!
      • Deterministic: Always the same result with the same input
  • Integration testing
    • Testing complete modules and the interaction between them within a project
    • Closer to real life scenarios
  • End-to-end (E2E) testing
    • Testing the whole program by emulating the end user's behaviour

Test type comparison

  • Unit tests are faster to write, E2E tests slowest

C# Testing Frameworks

  • Three main testing frameworks:
    • MSTest
    • NUnit
    • XUnit
  • You can install these using NuGet, or in the newer versions of Visual Studio you can create a new testing project
  • NUnit is the most popular of the three, and will be used in this lecture

MSTest

  • Built-in to most versions of Visual Studio since 2005
  • Simple to use, easy structure
    • TestClass for denoting classes containing unit tests
    • TestMethod for denoting unit test methods

NUnit

  • Most popular C# unit testing framework
  • Very similar structure to MSTest tests
    • Instead of TestClass, we use TestFixture
    • Instead of TestMethod, we use Test / TestCase
  • Has some advantages to MSTest or XUnit tests, for example:
    • Ability to test a single test method using multiple different arguments

XUnit

  • Free, open-source unit testing tool for .NET framework and .NET core
  • Made by the creator of NUnit v2
  • You can test C#, F#, VB.NET and other .NET languages
  • Part of the .NET Foundation
  • Unique and different style

Note about test case naming

  • In C# unit testing, the most commonly followed naming convention for test cases/methods is as follows:
    FunctionNameThatIsTested_Condition_ShouldDoX
  • For example:
    Fibonacci_WhenNIs0_ShouldReturn0

An extensive unit test Example

Creating a solution & sample project

First, we need a solution and a sample project.

  1. In Visual Studio, create a Class library that targets .NET Standard.
  2. Name the project Fibonacci.Library and the solution Fibonacci. Remember to uncheck Place solution and project in the same directory if it isn't unchecked already.

Adding the function to be tested

  1. Rename the default Class1.cs file to Fibonacci.cs.
  2. Allow Visual Studio to also rename the class itself to Fibonacci.
    • Make the class public and static.
  3. Add a recursive fibonacci function shown here.
using System;

namespace Fibonacci.Library
{
    public static class Fibonacci
    {
        public static long Recursive(
            int n,
            long previous = 0,
            long current = 0,
            int counter = 0
        )
        {
            return counter == n ?
                previous + current :
                Recursive(
                  n,
                  current,
                  Math.Max(previous + current, 1),
                  counter + 1
                );
        }
    }
}

Adding a testing project

  • Now we need to add a testing project to our solution.
  1. Right click your solution, and add a new project. Select the NUnit project for .NET Core, and call it Fibonacci.UnitTests.
  2. Rename the created class file to Fibonacci_Tests.cs, and the class to Fibonacci_Tests

Linking the testing project

  1. We then need to link the Fibonacci.Library project to Fibonacci.UnitTests.
  2. Right click Fibonacci.UnitTests, and select Add > Reference..., and under Projects select Fibonacci.Library. Press OK.
  • Now we have linked Fibonacci.Library to Fibonacci.UnitTests, and we can begin testing our Recursive function in the Fibonacci class.

Importing the class

  1. Since we have many things called "Fibonacci", we have to import our Fibonacci class by using an alias
    • Put this to the top of your file Fibonacci_Tests.cs, underneath
      using NUnit.Framework;
      using FibonacciLib = Fibonacci.Library.Fibonacci;

Test fixture for a class

  1. Now, add [TestFixture] on top of our class, like so:
using NUnit.Framework;
using FibonacciLib = Fibonacci.Library.Fibonacci;

namespace Fibonacci.UnitTests {
    [TestFixture]
    public class Fibonacci_Tests {
  • This denotes that this class contains our unit tests.

Adding the unit test method

  1. Then, inside our class Fibonacci_Tests, let's create our first method to test the Recursive function, as shown here:
    using NUnit.Framework;
using FibonacciLib = Fibonacci.Library.Fibonacci;

namespace Fibonacci.UnitTests {
    [TestFixture]
    public class Fibonacci_Tests {
        [Test]
        public void Recursive_WhenNIs0_ShouldReturn0()
            => Assert.That(0, Is.EqualTo(FibonacciLib.Recursive(n: 0)));
    }
}

Things to take into consideration

  • When passing in function arguments, to avoid magic numbers and to improve code readability, use the syntax argumentName: value
    • Magic numbers = using numbers as input as-is without explanation, for example without assigning to a well-named variable, or without using a descriptive type
  • Assert.That is one of many functions that can determine whether the test passes
    • Its first argument is the expected value. In the second argument you give a constraint for the actual output, i.e., value is equal to, greater than, starts with, etc.
    • Do not mix the arguments up! If you do, the errors you get do not make any sense.

Running Tests

  • Now, let's open the built-in test runner in Visual Studio to run our tests.

  • On the top of Visual Studio, click Test, then Test Explorer.

  • In the Test Explorer, you should see the test case we created. If not, build the solution, and make sure there are no errors.

  • Either way, let's run our test case. Press Run All Tests in the top-left corner of Test Explorer. You should see our test case passing.

Adding More Test Cases

  • While there's nothing inherently wrong about our test case, it doesn't cover much of anything.
  • We need to add more test cases to test different values and outputs.
  • In other testing frameworks, you would do this by creating more methods.
  • But with NUnit, we can use our existing method.

Adding a new test

  1. Do the following modifications to our test method:
    [TestCase(0, 0)]
public void Recursive_Always_ShouldReturnNthFibonacci(
    int n,
    long expected
)
    => Assert.That(expected, Is.EqualTo(FibonacciLib.Recursive(n)));
  • Using TestCase instead of Test, we can pass in arguments to our test case method, and we can use those passed in arguments in our actual tests.

Adding more tests

  1. Run the test cases again, and they should pass.

    • But as you might notice, we haven't really changed anything yet.

  2. Let's add more test cases, like so:

    [TestCase(0, 0)]
[TestCase(1, 1)]
public void Recursive_Always_ShouldReturnNthFibonacci(

  3. Now we have 2 test cases with only one test method. Run the tests again, and both cases should pass.

Adding a failing test

  1. Let's add one more TestCase, like so:
    [TestCase(2, 1)]
  2. Run the tests again, and you should see the test case fail.
    • This is to be expected: the 2nd fibonacci number should be 1, but the actual value we got from our function was 2.
    • This means our function does not work as expected.

Fixing our code

  1. Let's fix our function, as seen here.
  2. Run the tests again. You should see all pass now.
  3. Add a few more test cases
    • (If you don't know the fibonacci sequence, you can check it from the internet.)
public static long Recursive(
    int n,
    long previous = 0,
    long current = 0,
    int counter = 0
)
{
    return counter == n ?
        current : // changed from previous + current
        Recursive(
          n,
          current,
          Math.Max(previous + current, 1),
          counter + 1
        );
}

Exercise 1: Unit testing in practice

Answer the following questions:

  1. How many test cases should we ideally have to test our Recursive function?
  2. What happens if we pass in n = int.MaxValue (2147483647)?
    • What about other huge numbers?
    • How would you fix the function so it works as expected?
  3. What happens if we pass in n = -1?
    • What about other negative numbers?
    • How would you fix the function so it works as expected?

Answers to Exercise 1: Unit testing in practice

  1. We should cover all the edge cases and code paths with our test cases. We do not need to have many test cases for the same code path.
  2. Stack Overflow, Int64 Overflow, or the test case just won't run.
    • We should have a maximum allowed value for n, throw the argument out of range exception if above
  3. Stack Overflow, since current can never equal n that is less than 0
    • We should have a minimum allowed value for n, throw the argument out of range exception if below

Testing exceptions

  • To expect exceptions to be thrown in test cases, we can use the method Assert.Throws
  • Syntax is as follows:
    Assert.Throws<ExceptionType>(delegate);
  • In practice:
    Assert.Throws<NullReferenceException>(() => TestedFunction(n));

Handling border cases

  • Continuing our Fibonacci test case, we need to add handling for n that is less than 0 or more than a set amount. Then we need to add another test method and two test cases to test our new functionality.
  1. Let's start with modifying our existing Recursive function, as shown here:
public const int MinDepth = 0;
public const int MaxDepth = 10000;

public static long Recursive(
    int n,
    long previous = 0,
    long current = 0,
    int counter = 0
)
{
    if (n < MinDepth || n > MaxDepth)
        throw new ArgumentOutOfRangeException(
          "n is out of bounds!"
        );
    
    return counter == n ?
        current :
        Recursive(
          n,
          current,
          Math.Max(previous + current, 1),
          counter + 1
        );
}

Testing that everything works

  • Now our Recursive function throws an ArgumentOutOfRange exception when n is less than 0 or more than 10000.

  1. Let's test that everything works as expected by adding a new test method and a few test cases:

    [TestCase(int.MaxValue)]
[TestCase(-1)]
public void Recursive_WhenNIsOutOfBounds_ShouldThrowArgumentOutOfRange(
    int n
)
    => Assert.Throws<ArgumentOutOfRangeException>(() => FibonacciLib.Recursive(n));

  2. Rename the other method to Recursive_WhenNIsInBounds_ShouldReturnNthFibonacci

Unnecessary arguments

  • We have one more problem:
  • Our Recursive function is a public method that has in total 4 arguments it takes, yet we only test 1 argument (n).
  • In order for our tests to be complete, we need to test all the arguments a public method takes in.
  • So, how do we fix this?

Removing unnecessary arguments

  • The naive way would be to test all the arguments with more test cases.
  • It's important to realize that we don't have to.
  • Since we never pass in any other arguments than n from outside Recursive, we could just create another private function that has those arguments, and remove them from our public function.
public static long Recursive(int nStartingValue)
{
    if (nStartingValue < MinDepth
    || nStartingValue > MaxDepth)
        throw new ArgumentOutOfRangeException(
          "n is out of bounds!"
        );
    
    long getRecursive(
        int n,
        long previous = 0,
        long current = 0,
        int counter = 0)
        => counter == n ?
            current :
            getRecursive(
              n,
              current,
              Math.Max(previous + current, 1),
              counter + 1
            );
    return getRecursive(nStartingValue);
}

End Result

  • Now we have tested all the arguments in our public Recursive function.
  • We do not need to test private functions such as getRecursive, only the public API has to be tested.
  • All tests should pass at this point.

Exercise 2: Unit Testing

public static long DoX(int nStartingValue) {
    if (nStartingValue < 0 || nStartingValue > 150)
        throw new ArgumentOutOfRangeException("n is out of bounds!");
    
    long doY(int n) => n > 0 ? n * doY(n - 1) : 1;
    
    return doY(nStartingValue);
}
  • Following the previously used architecture as closely as possible, unit test the attached function using NUnit. Answer the following questions:
  1. What does the function that we are testing actually do?
  2. Do we have to test doY as well?
  3. How would you name DoX so it is descriptive? How about doY?

Answers to exercise 2: Unit Testing

  1. It gets the nth factorial (n!)
  2. doY is a private method that gets tested when we test DoX, so no.
  3. Example names: DoX TryGetFactorial, doY getFactorial

Unit Testing: Final testing code

  • [TestCase(0, 1)]
[TestCase(1, 1)]
[TestCase(2, 2)]
[TestCase(3, 6)]
[TestCase(15, 1307674368000)]
public void TryGetFactorial_WhenNIsInBounds_ShouldReturnNthFactorial(
    int n, long expected
)
    => Assert.That(expected, Is.EqualTo(TestExerciseLib.TryGetFactorial(n)));

    [TestCase(-1)]
[TestCase(151)]
public void TryGetFactorial_NIsOutOfBounds_ShouldThrowArgumentOutOfRange(
    int n
)
    => Assert.Throws<ArgumentOutOfRangeException>(
        () => TestExerciseLib.TryGetFactorial(n)
    );

Writing testable code

  • Writing testable code often means that you're writing good code
  • Testable units should
    • be deterministic (functional): same input always produces the same output
    • be loosely coupled: components have a loose and limited connection to other components in the system, meaning changes in one component should not require changes to many others
    • abide to the Single Responsibility Principle: a function, method or component is only responsible for one purpose

Writing testable code: Example 1

  • Can GetCurrentSeason() be unit tested? Why / why not?
  • How to fix the problems, if any?
public static class Season
{
    public static string GetCurrentSeason() {
        DateTime currentTime = DateTime.Now;
        if (currentTime.Month > 11 && currentTime.Month < 3)
            return "Winter";
        else if (currentTime.Month < 6)
            return "Spring";
        else if (currentTime.Month < 9)
            return "Summer";
        else
            return "Fall";
    }
}

Writing testable code: Example 1 fixed

  • Now the method is purely functional and deterministic
    • Will return the same value for every call with equal arguments
  • GetCurrentSeason() can now easily be tested as shown next
public static class Season {
    public static string GetCurrentSeason(
        DateTime currentTime
    ) {
        if (currentTime.Month > 11 && currentTime.Month < 3)
            return "Winter";
        else if (currentTime.Month < 6)
            return "Spring";
        else if (currentTime.Month < 9)
            return "Summer";
        else
            return "Fall";
    }
}

public class GetCurrentSeasonTest
{
    [TestCase(12, "Winter")]
    [TestCase(3, "Spring")]
    [TestCase(6, "Summer")]
    [TestCase(9, "Fall")]
    public void GetCurrentSeason_ForFirstMonthsOfSeasons_ReturnsCorrectSeason(
        int month,
        string season
    )
    {
        Assert.That(season,
          Is.EqualTo(
            Season.GetCurrentSeason(new DateTime(2020, month, 1))
          )
        );
    }
}

  • The tests fail, as shown above.
  • Test results indicate there is something wrong with the code path that should return Winter
    if (currentTime.Month > 11 && currentTime.Month < 3)
  • The line should be changed to
    if (currentTime.Month > 11 || currentTime.Month < 3)

Writing testable code: Example 2

  • Let's look at a HeatingUnit class that returns a heating setting based on the current season. The current date problem is back again:
    public class HeatingUnit
{
    public string GetHeatingSetting()
    {
        DateTime currentDate = DateTime.Now;
        if (Season.GetCurrentSeason(currentDate) == "Winter")
            return "HEAT_SETTING_HIGH";
        else if (Season.GetCurrentSeason(currentDate) == "Summer")
            return "HEAT_SETTING_OFF";
        else
            return "HEAT_SETTING_MEDIUM";
    }
}

Writing testable code: Example 2 continued

  • Instead of taking the current date initialization even higher in the class hierarchy, let's make a service that returns the current date
    public interface IDateTimeProvider
{
    DateTime GetDateTime();
}
  • After this the service can be injected into HeatingUnit with constructor injection
  • Code example shown next
public class HeatingUnit
{
    private readonly IDateTimeProvider _dateTimeProvider;
    
    public HeatingUnit(IDateTimeProvider dateTimeProvider)
    {
        _dateTimeProvider = dateTimeProvider;
    }
    
    public string GetHeatingSetting() {
        if (Season.GetCurrentSeason(_dateTimeProvider.GetDateTime()) == "Winter")
            return "HEAT_SETTING_HIGH";
        else if (Season.GetCurrentSeason(_dateTimeProvider.GetDateTime()) == "Summer")
            return "HEAT_SETTING_OFF";
        else
            return "HEAT_SETTING_MEDIUM";
    }
}

  • For testing, a fake DateTimeProvider service can be used for injecting the HeatingUnit with any date:

    class FakeDateTimeProvider : IDateTimeProvider
{
    public DateTime Date { get; set; }
    public DateTime GetDateTime() => Date;
}

  • Using this kind of a fake service instead of the real one is called mocking

  • In the real application the provider would return the real current date

class SetHeatingSettingTest
{
    [TestCase(12, "HEAT_SETTING_HIGH")]
    [TestCase(3, "HEAT_SETTING_MEDIUM")]
    [TestCase(6, "HEAT_SETTING_OFF")]
    [TestCase(9, "HEAT_SETTING_MEDIUM")]
    public void SetHeatingSetting_ForFirstMonthsOfSeasons_ShouldReturnCorrectSetting(
        int month,
        string setting
    )
    {
        FakeDateTimeProvider timeProvider
            = new FakeDateTimeProvider { Date = new DateTime(2020, month, 1) };
        HeatingUnit heatingUnit = new HeatingUnit(timeProvider);

        Assert.That(setting, Is.EqualTo(heatingUnit.GetHeatingSetting()));
    }
}


Tests should pass!

Testing with Postman

  • So far, we have created individual request with Postman to see what the response of an API is for each request
  • This is not testing an API, this is exploring an API
  • Tests in Postman are inserted in the Tests tab
    • Postman uses the Chai.js testing library for creating tests
  • To get started immediately, you can select snippets from the right
  • Let's select the Status code: Code is 200 snippet
  • Successfully run tests show up in green in the Test Result tab of the response:
  • The official Postman web page is a good starting point for learning testing with Postman
  • Using pm.expect will give a bit more info about the test:
  • The response to /serverinfo has the following body:
    {
    "appSettings": {
        "applicationUrl": "http://localhost:63741",
        "aspNetCoreEnvironment": "Development"
    },
    "serverStatus": {
        "dbConnectionStatus": "Connected"
    }
}
  • Let's make a test to see whether the server is connected to the database
  • Variables can be declared within the test:

Exercise 3: Testing with Postman

  1. Create a new collection in Postman by selecting the Collections tab and clicking New Collection. Name it CourseAPI Tests. Launch the CourseAPI you have developed during the lectures, and make sure the course_db database is connected
  2. Create a new GET request for the URI /api/courses. Add a test: status code of the request should be 200. Save the request to the CourseAPI Tests collection.
  3. Create a new GET request for the URI /api/courses/999999999. Add a test: status code of the request should be 404. Save the request to the CourseAPI Tests collection.
  1. Create a new POST request for the URI /api/courses. Set the headers and the content correctly, but set the value of credits to 100. Add a test; status code should be 400. Save the request to the CourseAPI Tests collection.
  2. Hover on CourseAPI Tests collection, click the arrow and click Run. From the opened window, scroll down and click Run CourseAPI Tests. This will create all the requests in your collection.

CI/CD/CT

  • CI stands for Continuous Integration
    • Each change in code triggers a build-and-test sequence for the given project
    • Goal is to have a consistent and automated way to build, package, and test applications
  • CD stands for Continuous Delivery
    • Automates the delivery of applications to selected environments, such as Azure, AWS, GCP
  • Both require Continuous Testing, which includes automated regression, performance and other tests