This is a sketch of how to use GitHub actions to get a total combined coverage number, and create a badge for your README. There are other approaches too, but this uses some commonly used tools to get the job done.
We’ll use tox to run tests, and GitHub actions to run tox. A GitHub gist will be used as a scratch file to store parameters for the badge, which will be rendered by shields.io.
Start with the tox.ini that runs your test suite, and also includes a “coverage” environment that combines, reports, and produces a JSON data file:
tox.ini
[tox]
envlist = py37,py38,py39,py310,coverage
[testenv]
commands =
python -m coverage run -p -m pytest
[testenv:coverage]
basepython = python3.10
commands =
python -m coverage combine
python -m coverage report -m --skip-covered
python -m coverage json
[gh-actions]
python =
3.7: py37
3.8: py38
3.9: py39
3.10: py310
We’ll use a GitHub action to run tox, but before we get to that, we need two bits of infrastructure. Go to https://gist.github.com and make an empty secret gist. Copy the id of the gist. Here we’ll call it 123abc456def789.
Next we’ll create a personal access token for updating the gist. Go to your GitHub personal access tokens page and click “Generate new token.” Select the “gist” scope, and click “Generate token.” Copy the value displayed, it will look like “ghp_FSfkCeFblahblahblah”. You can’t get the value again, so be careful with it.
In your repo on GitHub, go to Settings - Secrets - Actions, click “New repository secret.” Use “GIST_TOKEN” as the Name, and paste the ghp_etc token as the Secret, then “Add secret.”
Now we’re ready to create the GitHub action. It will run the test suite on many versions of Python, then run the coverage step to combine all the data files. It uses the JSON report to extract a displayable percentage, then uses a third-party GitHub action to create the JSON data in the Gist so that shields.io can display the badge.
The badge is automatically colored: 50% or lower is red, 90% or higher is green, with a gradient between the two, like this:
As a bonus, there’s an action job summary with the coverage total. Here’s the workflow file:
.github/workflows/tests.yaml
# Run tests
name: "Test Suite"
on:
push:
pull_request:
defaults:
run:
shell: bash
jobs:
tests:
name: "Python ${{ matrix.python-version }} on ${{ matrix.os }}"
runs-on: "${{ matrix.os }}"
strategy:
fail-fast: false
matrix:
os:
- ubuntu-latest
- macos-latest
- windows-latest
python-version:
- "3.7"
- "3.8"
- "3.9"
- "3.10"
steps:
- name: "Check out the repo"
uses: "actions/checkout@v2"
- name: "Set up Python"
uses: "actions/setup-python@v2"
with:
python-version: "${{ matrix.python-version }}"
- name: "Install dependencies"
run: |
python -m pip install tox tox-gh-actions
- name: "Run tox for ${{ matrix.python-version }}"
run: |
python -m tox
- name: "Upload coverage data"
uses: actions/upload-artifact@v3
with:
name: covdata
path: .coverage.*
coverage:
name: Coverage
needs: tests
runs-on: ubuntu-latest
steps:
- name: "Check out the repo"
uses: "actions/checkout@v2"
- name: "Set up Python"
uses: "actions/setup-python@v2"
with:
python-version: "3.10"
- name: "Install dependencies"
run: |
python -m pip install tox tox-gh-actions
- name: "Download coverage data"
uses: actions/download-artifact@v3
with:
name: covdata
- name: "Combine"
run: |
python -m tox -e coverage
export TOTAL=$(python -c "import json;print(json.load(open('coverage.json'))['totals']['percent_covered_display'])")
echo "total=$TOTAL" >> $GITHUB_ENV
echo "### Total coverage: ${TOTAL}%" >> $GITHUB_STEP_SUMMARY
- name: "Make badge"
uses: schneegans/dynamic-badges-action@v1.4.0
with:
# GIST_TOKEN is a GitHub personal access token with scope "gist".
auth: ${{ secrets.GIST_TOKEN }}
gistID: 123abc456def789 # replace with your real Gist id.
filename: covbadge.json
label: Coverage
message: ${{ env.total }}%
minColorRange: 50
maxColorRange: 90
valColorRange: ${{ env.total }}
Now the badge can be displayed with a URL like this, but replace YOUR_GITHUB_NAME with your GitHub name, and 123abc456def789 with your real Gist id:
https://img.shields.io/endpoint?url=https://gist.githubusercontent.com/YOUR_GITHUB_NAME/123abc456def789/raw/covbadge.json
Consult the docs for your markup language of choice for how to use the image URL to display the badge.
BTW: the files here are simplified versions of the action and tox.ini from scriv, if you are interested.
Comments
Neat!
Incidentally, this sort of trick (coverage combine from multiple tox environments) breaks if the user runs tox -p auto to run the tox environments in parallel.
I like tox -p auto, so here’s how to un-break it: in [testenv:coverage] add two options. One is
and the other is
and now tox -p auto will run everything but the coverage combine step in parallel.
Thanks, I’ve never used
tox -p
, this works nicely.Thanks for the great article!
I run into the error
No source for code
at thecoverage report
step. I’ve opened issue #1459 for this on coveragepy.Any help would be much appreciated!
That’s an interesting idea. It occurred to me that you could push a banner to GitHub Pages instead. And this way, there would be no need for a user-account token — having
${{ secrets.GITHUB_TOKEN }}
would be more than enough. Similar to how a static website can be published: https://github.com/webknjaz/.me/blob/cff03ba/.github/workflows/build-gh-pages.yml#L44-L56. Generate an SVG with Jinja2 and push it togh-pages
. This can also be combined with a normal static website, if there’s a workflow building it. This way shields.io would also be unnecessary — you’d just point to the SVG directly.See also this article from 2020.
And this article from April of this year.
Add a comment: