@admiralAwkbar basically what happens is:

1. we have an "expected" TAP report in the test directory.
2. Generate a TAP on the fly for each language.
3. Diff the generated tap against the expected tap.

Example of an expected TAP:

TAP version 13
1..2
not ok 1 - shell_bad_1.sh
  ---
  message: \nIn /tmp/lint/.automation/test/bash/shell_bad_1.sh line 10 \nif [ $ERROR_CODE -ne 0]; then\n^-- SC1009  The mentioned syntax error was in this if expression.\n   ^-- SC1073  Couldn't parse this test expression. Fix to allow more checks.\n                       ^-- SC1020  You need a space before the ].\n                       ^-- SC1072  Missing space before ]. Fix any mentioned problems and try again.\n\nFor more information \n  https //www.shellcheck.net/wiki/SC1020 -- You need a space before the ].\n  https //www.shellcheck.net/wiki/SC1072 -- Missing space before ]. Fix any m...\n  https //www.shellcheck.net/wiki/SC1073 -- Couldn't parse this test expressi...\n
  ...
ok 2 - /tmp/lint/.automation/test/bash/shell_good_1.sh

1. TAP version 13: tap version, mandated by the spec

2. 1..2: test plan. Tells how many tests will be run, or how many tests have run. In this example we see two tests: one for the good file, and one for the bad file in the shell test directory.

3. not ok 1 - shell_bad_1.sh and ok 2 - /tmp/lint/.automation/test/bash/shell_good_1.sh: test lines. Tell that the test 1 fails (the bad file), and the test 2 runs successfully (the good file).

4. The YAML block. This is not standardized in the spec, so we have complete freedom. In super-linter, someone made the choice to use the format:

   ---
   message: ABCDEFG.....
   ...

   We then use this YAML to save the expected output message, or the actual output message.

There are issues in this approach when the output has dynamic parts. I suspect that's the reason why some languages (such as arm-ttk) have a TAP file, but with an ignored extension.

PS: found some info here #421

PPS: we can tweak the TransformTAPDetails function to handle this.

@admiralAwkbar almost there. I'm missing a regex and fixing a weird formatting issue with the powershell test.

@admiralAwkbar finally ready for review.

Great work @ferrarimarco ! Dunno what it all means to be honest (TAP?) but hopefully will help catch things in future and looks like a huge amount of work, so thanks!

LOL. TAP = Test Anything Protocol. It's a semi-structured protocol for test reporting. We're using it in super-linter to check if the test results match our expectations.