How to test your app?¶

Learning Objectives

By the end of this section (if you work on the assignments), you should know how to:

Create basic tests for your CLI with pytest and typer
Test your APIs with doctest (via pytest)
Add a CI/CD pipeline for automatic testing

You made it to the last section of the learning guide! Kudos!

Testing software is a wonderful but complicated endeavor. A thorough discussion of all the facets of testing is beyond the scope of this section.

But we do want to emphasize the crucial role that testing plays in software development. So we at least provide you with some basic examples and then point to resources for further learning.

Since this is the last chapter of the learning guide, we are going to challenge you a bit more: we leave out extensive explanations and instead let you work through assignments. We hope this will help you deepening your understanding and prepare you for your next steps on your journey to become a great Pythonista and open-source contributor!

Testing your CLI with `pytest` and `Typer`¶

Assignment 1

a) Read through: https://typer.tiangolo.com/tutorial/testing/
b) Read through: https://docs.pytest.org/en/7.0.x/getting-started.html
c) Add useful tests to tests/test_quizli.py (shown below) to improve the code coverage.

test_quizli.py

""" 
Basic testing for the `quizli` app.

See: https://typer.tiangolo.com/tutorial/testing/
"""


import pytest
from typer.testing import CliRunner

from quizli.main import app

runner = CliRunner()


#  ╭──────────────────────────────────────────────────────────╮
#  │ Test Help Commands                                       │
#  ╰──────────────────────────────────────────────────────────╯
help_commands = [
    "--help",
    "demo --help",
    "start --help",
]


@pytest.mark.parametrize("test_input", help_commands)
def test_help_commands(
    test_input,
):
    result = runner.invoke(app, test_input)
    assert result.exit_code == 0


#  ╭──────────────────────────────────────────────────────────╮
#  │ Test Demo Command                                        │
#  ╰──────────────────────────────────────────────────────────╯
def test_demo():
    result = runner.invoke(app, "demo")
    assert result.exit_code == 0


#  ╭──────────────────────────────────────────────────────────╮
#  │ Test Start Command                                       │
#  ╰──────────────────────────────────────────────────────────╯
@pytest.mark.parametrize(
    "test_cmd, test_inp, exp_out",
    [
        (
            "start --quiz-name python_quiz",
            ["modules\n", "Answer\n", "n\n", "n\n"],
            "See you soon",
        ),
        (
            "start --quiz-name python_quiz",
            ["objects\n", "quizli\n", "Answer\n", "n\n", "n\n"],
            "See you soon",
        ),
    ],
)
def test_start_builtin_quiz(test_cmd, test_inp, exp_out):
    result = runner.invoke(app, test_cmd, input="".join(test_inp))

    assert result.exit_code == 0
    assert exp_out in result.stdout


@pytest.mark.parametrize(
    "test_cmd, test_inp, exp_out",
    [
        (
            "start --from-csv examples/quiz.csv --mode sudden_death --in-order",
            ["pip install quizli", "yes", "2", "monty python", "42", "yes", ""],
            "🏆 You won! 🏆 See you soon, champion!",
        ),
        (
            "start --from-csv examples/quiz.csv --mode sudden_death --in-order",
            ["pip install quizli", "yes", "2", "monty python", "42", "no", "n"],
            "Game Over! 😞 See you soon !",
        ),
    ],
)
def test_start_quiz_from_csv(test_cmd, test_inp, exp_out):
    result = runner.invoke(app, test_cmd, input="\n\n".join(test_inp))

    assert result.exit_code == 0
    assert exp_out in result.stdout

Testing your docstring examples with `doctest`¶

Assignment 2

Read through: doctests - Test interactive Python examples
Read through: pytest - Doctest integration for modules and test files

What is doctest good for?

Quote

The doctest module searches for pieces of text that look like interactive Python sessions, and then executes those sessions to verify that they work exactly as shown. There are several common ways to use doctest:

To check that a module’s docstrings are up-to-date by verifying that all interactive examples still work as documented.
To perform regression testing by verifying that interactive examples from a test file or a test object work as expected.
To write tutorial documentation for a package, liberally illustrated with input-output examples. Depending on whether the examples or the expository text are emphasized, this has the flavor of “literate testing” or “executable documentation”.

Example

@dataclass
class QuizItem:
    """Represents a quiz item - a question-answer pair

    Attributes:
        question: The question of the quiz item
        answer: The answer of the quiz item


    Examples:
        >>> quiz_item = QuizItem(question='What is the meaning of life?', answer=42)

        >>> quiz_item
        QuizItem(question='What is the meaning of life?', answer=42)

        >>> quiz_item.question
        'What is the meaning of life?'

        >>> quiz_item.answer
        42
    """


(...)

Hint

We can runpytest with the --doctest-modules flag to test whether all docstring examples are correct and up-to-date.

Automate testing with a CI/CD pipeline¶

Assignment 3

a) Read through Github's CI/CD: The what, why, and how
b) Read through: https://github.com/snok/install-poetry
c) Read through .github/workflows/test_package.yml (shown below) and then explain it to someone.

CI/CD workflow

name: test_package

on:
  push:
    branches:
      - master
      - main
      - Add_testing
jobs:
  linting:
    runs-on: ubuntu-latest
    steps:
#  ╭──────────────────────────────────────────────────────────╮
#  │             Check-out repo and set-up python             │
#  ╰──────────────────────────────────────────────────────────╯
      - uses: actions/checkout@v2
      - uses: actions/setup-python@v2
#  ╭──────────────────────────────────────────────────────────╮
#  │              Load pip cache if cache exists              │
#  ╰──────────────────────────────────────────────────────────╯
      - uses: actions/cache@v2
        with:
          path: ~/.cache/pip
          key: ${{ runner.os }}-pip
          restore-keys: ${{ runner.os }}-pip
#  ╭──────────────────────────────────────────────────────────╮
#  │                  Install and run linters                 │
#  ╰──────────────────────────────────────────────────────────╯
      - run: python -m pip install black isort
      - run: |
          black . --check
          isort .
  test:
    strategy:
      matrix:
        # NOTE: In the real world you should test on other
        # operating systems as well!
        os: [ "ubuntu-latest"] 
        python-version: ["3.7", "3.8", "3.9", "3.10" ]
    defaults:
      run:
        shell: bash
    runs-on: ${{ matrix.os }}
    steps:
#  ╭──────────────────────────────────────────────────────────╮
#  │             Check-out repo and set-up python             │
#  ╰──────────────────────────────────────────────────────────╯
      - name: Check out repository
        uses: actions/checkout@v2
      - name: Set up python ${{ matrix.python-version }}
        id: setup-python
        uses: actions/setup-python@v2
        with:
          python-version: ${{ matrix.python-version }}
#  ╭──────────────────────────────────────────────────────────╮
#  │                Install & configure poetry                │
#  ╰──────────────────────────────────────────────────────────╯
      - name: Install Poetry
        uses: snok/install-poetry@v1
        with:
          virtualenvs-create: false
#  ╭──────────────────────────────────────────────────────────╮
#  │                      Install quizli                      │
#  ╰──────────────────────────────────────────────────────────╯
      - name: Install quizli
        run: poetry install --no-interaction
        if: steps.cached-poetry-dependencies.outputs.cache-hit != 'true'
#  ╭──────────────────────────────────────────────────────────╮
#  │          Run test suite and output coverage file         │
#  ╰──────────────────────────────────────────────────────────╯
      - name: Run tests
        run: poetry run pytest --doctest-modules -vvv --cov-report=xml --cov=. 
#  ╭──────────────────────────────────────────────────────────╮
#  │                   Upload coverage stats                  │
#  ╰──────────────────────────────────────────────────────────╯
      - name: Upload coverage
        uses: codecov/codecov-action@v1

#  ╭──────────────────────────────────────────────────────────╮
#  │   NOTE: In a real-world example you would add a another  │
#  │   stage that runs if the test stage has been successful  │
#  │                 and publishes the package                │
#  ╰──────────────────────────────────────────────────────────╯

Real-world testing¶

We already told you that testing is hard. And there is a lot to learn to do it right. Before we part, here is a list with key concepts you can explore to ease your way into the world of testing.

Assignment 4: Key Concepts in the testing world

a) What is test-driven development?
b) What is property-based testing?
c) How can you utilize tests as documentation?
d) What is unit testing? And what is integration testing?
e) What is code coverage?

Last update: March 11, 2022

How to test your app?¶

Testing your CLI with pytest and Typer¶

Testing your docstring examples with doctest¶

Automate testing with a CI/CD pipeline¶

Real-world testing¶

Testing your CLI with `pytest` and `Typer`¶

Testing your docstring examples with `doctest`¶