Making a coverage badge

Monday 19 September 2022

This is a sketch of how to use GitHub actions to get a total combined coverage number, and create a badge for your README. There are other approaches too, but this uses some commonly used tools to get the job done.

We’ll use tox to run tests, and GitHub actions to run tox. A GitHub gist will be used as a scratch file to store parameters for the badge, which will be rendered by shields.io.

Start with the tox.ini that runs your test suite, and also includes a “coverage” environment that combines, reports, and produces a JSON data file:

tox.ini

[tox]
envlist = py37,py38,py39,py310,coverage

[testenv]
commands =
    python -m coverage run -p -m pytest

[testenv:coverage]
basepython = python3.10
commands =
    python -m coverage combine
    python -m coverage report -m --skip-covered
    python -m coverage json

[gh-actions]
python =
    3.7: py37
    3.8: py38
    3.9: py39
    3.10: py310

We’ll use a GitHub action to run tox, but before we get to that, we need two bits of infrastructure. Go to https://gist.github.com and make an empty secret gist. Copy the id of the gist. Here we’ll call it 123abc456def789.

Next we’ll create a personal access token for updating the gist. Go to your GitHub personal access tokens page and click “Generate new token.” Select the “gist” scope, and click “Generate token.” Copy the value displayed, it will look like “ghp_FSfkCeFblahblahblah”. You can’t get the value again, so be careful with it.

In your repo on GitHub, go to Settings - Secrets - Actions, click “New repository secret.” Use “GIST_TOKEN” as the Name, and paste the ghp_etc token as the Secret, then “Add secret.”

Now we’re ready to create the GitHub action. It will run the test suite on many versions of Python, then run the coverage step to combine all the data files. It uses the JSON report to extract a displayable percentage, then uses a third-party GitHub action to create the JSON data in the Gist so that shields.io can display the badge.

The badge is automatically colored: 50% or lower is red, 90% or higher is green, with a gradient between the two, like this:

The spectrum of badge colors.

As a bonus, there’s an action job summary with the coverage total. Here’s the workflow file:

.github/workflows/tests.yaml

# Run tests

name: "Test Suite"

on:
  push:
  pull_request:

defaults:
  run:
    shell: bash

jobs:
  tests:
    name: "Python ${{ matrix.python-version }} on ${{ matrix.os }}"
    runs-on: "${{ matrix.os }}"

    strategy:
      fail-fast: false
      matrix:
        os:
          - ubuntu-latest
          - macos-latest
          - windows-latest
        python-version:
          - "3.7"
          - "3.8"
          - "3.9"
          - "3.10"

    steps:
      - name: "Check out the repo"
        uses: "actions/checkout@v2"

      - name: "Set up Python"
        uses: "actions/setup-python@v2"
        with:
          python-version: "${{ matrix.python-version }}"

      - name: "Install dependencies"
        run: |
          python -m pip install tox tox-gh-actions

      - name: "Run tox for ${{ matrix.python-version }}"
        run: |
          python -m tox

      - name: "Upload coverage data"
        uses: actions/upload-artifact@v3
        with:
          name: covdata
          path: .coverage.*

  coverage:
    name: Coverage
    needs: tests
    runs-on: ubuntu-latest
    steps:
      - name: "Check out the repo"
        uses: "actions/checkout@v2"

      - name: "Set up Python"
        uses: "actions/setup-python@v2"
        with:
          python-version: "3.10"

      - name: "Install dependencies"
        run: |
          python -m pip install tox tox-gh-actions

      - name: "Download coverage data"
        uses: actions/download-artifact@v3
        with:
          name: covdata

      - name: "Combine"
        run: |
          python -m tox -e coverage
          export TOTAL=$(python -c "import json;print(json.load(open('coverage.json'))['totals']['percent_covered_display'])")
          echo "total=$TOTAL" >> $GITHUB_ENV
          echo "### Total coverage: ${TOTAL}%" >> $GITHUB_STEP_SUMMARY

      - name: "Make badge"
        uses: schneegans/dynamic-badges-action@v1.4.0
        with:
          # GIST_TOKEN is a GitHub personal access token with scope "gist".
          auth: ${{ secrets.GIST_TOKEN }}
          gistID: 123abc456def789   # replace with your real Gist id.
          filename: covbadge.json
          label: Coverage
          message: ${{ env.total }}%
          minColorRange: 50
          maxColorRange: 90
          valColorRange: ${{ env.total }}

Now the badge can be displayed with a URL like this, but replace YOUR_GITHUB_NAME with your GitHub name, and 123abc456def789 with your real Gist id:

https://img.shields.io/endpoint?url=https://gist.githubusercontent.com/YOUR_GITHUB_NAME/123abc456def789/raw/covbadge.json

Consult the docs for your markup language of choice for how to use the image URL to display the badge.

BTW: the files here are simplified versions of the action and tox.ini from scriv, if you are interested.

Comments

[gravatar]

Neat!

Incidentally, this sort of trick (coverage combine from multiple tox environments) breaks if the user runs tox -p auto to run the tox environments in parallel.

I like tox -p auto, so here’s how to un-break it: in [testenv:coverage] add two options. One is

depends = py37,py38,py39,py310

and the other is

parallel_show_output = true

and now tox -p auto will run everything but the coverage combine step in parallel.

[gravatar]

Thanks, I’ve never used tox -p, this works nicely.

[gravatar]

Thanks for the great article!

I run into the error No source for code at the coverage report step. I’ve opened issue #1459 for this on coveragepy.

Any help would be much appreciated!

[gravatar]

That’s an interesting idea. It occurred to me that you could push a banner to GitHub Pages instead. And this way, there would be no need for a user-account token — having ${{ secrets.GITHUB_TOKEN }} would be more than enough. Similar to how a static website can be published: https://github.com/webknjaz/.me/blob/cff03ba/.github/workflows/build-gh-pages.yml#L44-L56. Generate an SVG with Jinja2 and push it to gh-pages. This can also be combined with a normal static website, if there’s a workflow building it. This way shields.io would also be unnecessary — you’d just point to the SVG directly.

[gravatar]

See also this article from 2020.

[gravatar]

And this article from April of this year.

Add a comment:

Ignore this:
Leave this empty:
Name is required. Either email or web are required. Email won't be displayed and I won't spam you. Your web site won't be indexed by search engines.
Don't put anything here:
Leave this empty:
Comment text is Markdown.