| 1 | The Automake test suite
|
|---|
| 2 |
|
|---|
| 3 |
|
|---|
| 4 | User interface
|
|---|
| 5 | ==============
|
|---|
| 6 |
|
|---|
| 7 |
|
|---|
| 8 | Running all tests
|
|---|
| 9 | -----------------
|
|---|
| 10 |
|
|---|
| 11 | make check
|
|---|
| 12 |
|
|---|
| 13 |
|
|---|
| 14 | Interpretation
|
|---|
| 15 | --------------
|
|---|
| 16 |
|
|---|
| 17 | Successes:
|
|---|
| 18 | PASS - success
|
|---|
| 19 | XFAIL - expected failure
|
|---|
| 20 |
|
|---|
| 21 | Failures:
|
|---|
| 22 | FAIL - failure
|
|---|
| 23 | XPASS - unexpected success
|
|---|
| 24 |
|
|---|
| 25 | Other:
|
|---|
| 26 | SKIP - skipped tests (third party tools not available)
|
|---|
| 27 |
|
|---|
| 28 |
|
|---|
| 29 | Getting details from failures
|
|---|
| 30 | -----------------------------
|
|---|
| 31 |
|
|---|
| 32 | Each test is a script. In a non-VPATH build you can run them
|
|---|
| 33 | directly, they will be verbose.
|
|---|
| 34 |
|
|---|
| 35 | Otherwise, you can invoke make as follows, just replace the list
|
|---|
| 36 | of tests by those you want to check.
|
|---|
| 37 |
|
|---|
| 38 | env VERBOSE=x TESTS='first.test second.test ...' make -e check
|
|---|
| 39 |
|
|---|
| 40 |
|
|---|
| 41 | Reporting failures
|
|---|
| 42 | ------------------
|
|---|
| 43 |
|
|---|
| 44 | Send verbose output of failing tests to <bug-automake@gnu.org>,
|
|---|
| 45 | along with the usual version numbers (which Automake, which
|
|---|
| 46 | Autoconf, which operating system, which make version, which shell,
|
|---|
| 47 | etc.)
|
|---|
| 48 |
|
|---|
| 49 |
|
|---|
| 50 |
|
|---|
| 51 | Writing test cases
|
|---|
| 52 | ==================
|
|---|
| 53 |
|
|---|
| 54 |
|
|---|
| 55 | Do
|
|---|
| 56 | --
|
|---|
| 57 |
|
|---|
| 58 | If you plan to fix a bug, write the test case first. This way you'll
|
|---|
| 59 | make sure the test catches the bug, and that it succeeds once you have
|
|---|
| 60 | fixed the bug.
|
|---|
| 61 |
|
|---|
| 62 | Add a copyright/license paragraph.
|
|---|
| 63 |
|
|---|
| 64 | Explain what the test does.
|
|---|
| 65 |
|
|---|
| 66 | Cite the PR number (if any), and the original reporter (if any), so
|
|---|
| 67 | we can find or ask for information if needed.
|
|---|
| 68 |
|
|---|
| 69 | Use `required=...' for required tools.
|
|---|
| 70 |
|
|---|
| 71 | Include ./defs (see other tests).
|
|---|
| 72 |
|
|---|
| 73 | Use `set -e' to catch failures you might not have thought of.
|
|---|
| 74 |
|
|---|
| 75 | ./defs sets a skeleton configure.in. If possible, append to this
|
|---|
| 76 | file. In some cases you'll have to overwrite it, but this should
|
|---|
| 77 | be the exception. Note that configure.in registers Makefile.in
|
|---|
| 78 | but do not output anything by default. If you need ./configure
|
|---|
| 79 | to create Makefile, append AC_OUTPUT to configure.in.
|
|---|
| 80 |
|
|---|
| 81 | Use $ACLOCAL, $AUTOMAKE, $AUTOCONF, $AUTOUPDATE, $AUTOHEADER,
|
|---|
| 82 | $PERL, $MAKE, $EGREP, and $FGREP, instead of the corresponding
|
|---|
| 83 | commands.
|
|---|
| 84 |
|
|---|
| 85 | Use $sleep when you have to make sure that some file is newer
|
|---|
| 86 | than another.
|
|---|
| 87 |
|
|---|
| 88 | Use `cat' or `grep' to display (part of) files that may be
|
|---|
| 89 | interesting for debugging, so that when a user send a verbose
|
|---|
| 90 | output we don't have to ask him for more details.
|
|---|
| 91 |
|
|---|
| 92 | It's more important to make sure that a feature works, than
|
|---|
| 93 | make sure that Automake's output looks correct. It might look
|
|---|
| 94 | correct and still fails to work. In other words, prefer
|
|---|
| 95 | running `make' over grepping `Makefile.in' (or do both).
|
|---|
| 96 |
|
|---|
| 97 | If you run $AUTOMAKE or $AUTOCONF several times in the same test
|
|---|
| 98 | and change `configure.in' by the meantime, do
|
|---|
| 99 | rm -rf autom4te.cache
|
|---|
| 100 | before the following runs. On fast machines the new `configure.in'
|
|---|
| 101 | could otherwise have the same timestamp as the old `autom4te.cache'.
|
|---|
| 102 |
|
|---|
| 103 | Use filenames with two consecutive spaces when testing that some
|
|---|
| 104 | code preserves filenames with spaces. This will catch errors like
|
|---|
| 105 | `echo $filename | ...`.
|
|---|
| 106 |
|
|---|
| 107 | Before commit: make sure the test is executable, add the tests to
|
|---|
| 108 | TESTS in Makefile.am, add it to XFAIL_TESTS in addition if needed,
|
|---|
| 109 | write a ChangeLog entry, send the diff to <automake-patches@gnu.org>.
|
|---|
| 110 |
|
|---|
| 111 |
|
|---|
| 112 | Do not
|
|---|
| 113 | ------
|
|---|
| 114 |
|
|---|
| 115 | Do not test an Automake error with `$AUTOMAKE && exit 1', or in three
|
|---|
| 116 | years we'll discover that this test failed for some other bogus reason.
|
|---|
| 117 | This happened many times. Better use something like
|
|---|
| 118 | AUTOMAKE_fails
|
|---|
| 119 | grep 'expected diagnostic' stderr
|
|---|
| 120 | (Note this doesn't prevent the test from failing for another
|
|---|
| 121 | reason, but at least it makes sure the original error is still
|
|---|
| 122 | here.)
|
|---|
| 123 |
|
|---|
| 124 | Do not override Makefile variables using make arguments, as in
|
|---|
| 125 | $MAKE ANSI2KNR=./ansi2knr U=_ all
|
|---|
| 126 | this is not portable for recursive targets (targets that
|
|---|
| 127 | call a sub-make may not pass `ANSI2KNR=./ansi2knr U=_' along).
|
|---|
| 128 | Use the following instead.
|
|---|
| 129 | ANSI2KNR=./ansi2knr U=_ $MAKE -e all
|
|---|
| 130 |
|
|---|
| 131 | Do not send a test case without signing a copyright disclaimer.
|
|---|
| 132 | See http://sources.redhat.com/automake/contribute.html or
|
|---|
| 133 | ask <automake@gnu.org> for details.
|
|---|
