mirror of https://github.com/tc39/test262.git
CI: Do not report test failures as errors
Prior to this patch, the CircleCI continuous integration environment was configured to report test failures in a negative light, displaying red cross-marks and reporting that "some checks were not successful" in commits and GitHub Pull Requests which included them. The passing/failing status of tests does not influence their desirability for Test262. (In practice, engines very commonly fail newly-contributed tests.) Although these conflicting interpretations does not technically interfere with the maintainers' ability to merge new contributions, it does create confusion for many contributors who interpreted the UI as a rejection of their work. In addition, this behavior made it impossible to distinguish between the benign test failures and disruptive infrastructural problems (e.g. the crashing of engines). Reconfigure the continuous integration environment to accept passing and failing tests equally, and to only report a problem when the Test262 project's testing infrastructure behaves unexpectedly.
This commit is contained in:
parent
d454b8389b
commit
1ebd34b53b
|
@ -14,7 +14,7 @@
|
|||
"homepage": "https://github.com/tc39/test262#readme",
|
||||
"devDependencies": {
|
||||
"esvu": "^1.0.0",
|
||||
"test262-harness": "^7.3.0"
|
||||
"test262-harness": "^8.0.0"
|
||||
},
|
||||
"scripts": {
|
||||
"ci": "./tools/scripts/ci_test.sh",
|
||||
|
|
|
@ -13,12 +13,5 @@ if [ "$CIRCLE_PULL_REQUEST" != "" ]; then
|
|||
echo ""
|
||||
|
||||
echo "Running the tests with test262-harness"
|
||||
test262-harness -t 1 --hostType=$hostType --hostPath=$hostPath --hostArgs="$hostArgs" -- $paths | tee exec.out
|
||||
|
||||
if grep -q '^[1-9][0-9]* failed$' exec.out; then
|
||||
rm exec.out
|
||||
exit 1
|
||||
fi
|
||||
|
||||
rm exec.out
|
||||
test262-harness -t 1 --hostType=$hostType --hostPath=$hostPath --hostArgs="$hostArgs" -- $paths
|
||||
fi
|
||||
|
|
Loading…
Reference in New Issue