For asynchronous tests, the contract between test file and test runner
is implicit: runners are expected to inspect the source code for
references to a global `$DONE` identifier.
Promote a more explicit contract between test file and test runner by
introducing a new frontmatter "tag", `async`. This brings asynchronous
test configuration in-line with other configuration mechanisms and also
provides a more natural means of test filtering.
The modifications to test files was made programatically using the
`grep` and `sed` utilities:
$ grep "\$DONE" test/ -r --files-with-match --null | \
xargs -0 sed -i 's/^\(flags:\s*\)\[/\1[async, /g'
$ grep "\$DONE" test/ -rl --null | \
xargs -0 grep -E '^flags:' --files-without-match --null | \
xargs -0 sed -i 's/^---\*\//flags: [async]\n---*\//'
When executing multiple tests in parallel, each "child" thread would
write to the process's standard output buffer immediately upon test
completion. Because thread execution order and instruction interleaving
is non-deterministic, this made it possible for characters to be emitted
out-of-order.
When extended to support multiple concurrent threads, the runner was
outfitted with a "log lock" dedicated to sharing access to the output
file (when applicable). Re-use this lock when writing to standard out,
ensuring proper ordering of test result messages.
A recent extension to the test runner introduced support for running
tests in parallel using multi-threading. Following this, the runner
would incorrectly emit the "final report" before all individual test
results.
In order to emit the "final report" at the end of the output stream, the
parent thread would initialize all children and wait for availability of
a "log lock" shared by all children.
According to the documentation on the "threading" module's Lock object
[1]:
> When more than one thread is blocked in acquire() waiting for the state
> to turn to unlocked, only one thread proceeds when a release() call
> resets the state to unlocked; which one of the waiting threads proceeds
> is not defined, and may vary across implementations.
This means the primitive cannot be used by the parent thread to reliably
detect completion of all child threads.
Update the parent to maintain a reference for each child thread, and to
explicitly wait for every child thread to complete before emitting the
final result.
[1] https://docs.python.org/2/library/threading.html#lock-objects
Adds a `-j`/`--workers-count` parameter to `tools/packaging/test262.py`, defaulting to `[number of cores] - 1`.
Speeds up running the test suite by about ~3x on my 4-core machine, with the SpiderMonkey shell. This could certainly be optimized more by just appending test results to per-thread lists and merging them at the end, but it's better than nothing.
Some tests involving the directive prologue are invalidated by source
text transformations that insert executable code in the beginning of the
script. Implement a `raw` flag that allows these tests to opt-out of
this transformation. Update the relevant tests to use this flag (and
remove references to globals only available when code is injected).
Update the Python runner accordingly:
- Do not run tests marked as "raw" in strict mode
- Reject invalid test configurations
Update the browser runner accordingly:
- Do not modify the script body of tests marked as "raw"
Although test files once expressed dependencies on external files using
a global `$INCLUDE` function, that pattern was removed in favor of
declarative meta-data [1].
Remove the associated logic from the Python runner and the browser.
[1] See commit d4354d14d5.
add test to repro #114
fix issue #114
factor out PercentFormat function
new fn write; start factoring summary output
fix tests to match
consolidate "Failed Tests" message
add test for successful run
consolidate all_succeeded message
consolidate summary line
consolidate ntests
failing negative tests
squeeze whitespace
pilot directory for replacing flags: [negative] with
negative: errorname
ch07-7.2,7.3: add error names
ch07-rest: insert error names rest of ch07
ch08: add expected error name
ch11: add error name to negative
ch12: put error name in negative
ch13: add error name to negative
only one test, did it manually
ch15: add error name to negative:
ch07: add error name to negative
these tests used flags: \n - negative and so
were not caught by the earlier naive grep for flags: [negative]
--- changes suggested by @anba
bestPractice - remove added NotEarlyError
ch07,ch11 - expect ReferenceError (req'd by ES6)
correct test descriptions to expect ReferenceError only
enforce parsing as block statement by adding `;`
remove needless include of $FAIL.js
effectively, a rebase of all changes onto master
Merge branch 'negative-addErrorName-work' into negative-addErrorName-bestPractice
Merge branch 'saved-bestPractice-negative' into negative-addErrorName-work
Conflicts:
test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T2.js
test/suite/ch12/12.1/S12.1_A4_T1.js
test/suite/ch12/12.5/S12.5_A2.js
incorporate changes made on PR branch
re-remove added NotEarlyError
Merge branch 'negative-addErrorName-work' into negative-addErrorName-bestPractice
Conflicts:
test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T2.js
test/suite/ch12/12.1/S12.1_A4_T1.js
test/suite/ch12/12.5/S12.5_A2.js
ch15: correct negative regexes
ch10: avoid use of not-NotEarlyError regex
ch14 error regexps
ch13 error regexps
ch12: negative error matching
Merge branch 'console-runner-checkError' into negative-addErrorName-work
test262.py: check negative tests with regex
implement checking of negative tests =>
negative: SyntaxError
means that /SyntaxError/ must match stderr
or test is reported as failure
Fixes#78
expect real errors
bestPractice: supply error names to negative
pilot directory for replacing flags: [negative] with
negative: errorname
ch07-7.2,7.3: add error names
ch07-rest: insert error names rest of ch07
ch08: add expected error name
ch11: add error name to negative
ch12: put error name in negative
ch13: add error name to negative
only one test, did it manually
ch15: add error name to negative:
ch07: add error name to negative
these tests used flags: \n - negative and so
were not caught by the earlier naive grep for flags: [negative]
--- changes suggested by @anba
bestPractice - remove added NotEarlyError
ch07,ch11 - expect ReferenceError (req'd by ES6)
correct test descriptions to expect ReferenceError only
enforce parsing as block statement by adding `;`
remove needless include of $FAIL.js
Merge branch 'negative-addErrorName-work' into negative-addErrorName-bestPractice
resolved Conflicts:
test/suite/ch07/7.9/S7.9_A5.7_T1.js
test/suite/ch11/11.13/11.13.1/S11.13.1_A2.1_T3.js
test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T1.js
test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T10.js
test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T11.js
test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T2.js
test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T3.js
test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T4.js
test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T5.js
test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T6.js
test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T7.js
test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T8.js
test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T9.js
test/suite/ch11/11.3/11.3.1/S11.3.1_A2.1_T3.js
test/suite/ch11/11.3/11.3.2/S11.3.2_A2.1_T3.js
test/suite/ch11/11.4/11.4.4/S11.4.4_A2.1_T3.js
test/suite/ch11/11.4/11.4.5/S11.4.5_A2.1_T3.js
test/suite/ch12/12.5/S12.5_A2.js
test/suite/ch12/12.6/12.6.3/S12.6.3_A4_T2.js
negative: second half of @anba notes
correct test descriptions to expect ReferenceError only
enforce parse as block statement by adding `;`
remove needless include of $FAIL.js
ch12: specify type
negative: @anba fixes
bestPractice - remove added NotEarlyError
ch07,ch11 - expect ReferenceError (req'd by ES6)
bestPractice: supply error names to negative
pilot directory for replacing flags: [negative] with
negative: errorname
ch07-7.2,7.3: add error names
ch07-rest: insert error names rest of ch07
ch08: add expected error name
ch11: add error name to negative
ch12: put error name in negative
ch13: add error name to negative
only one test, did it manually
ch15: add error name to negative:
ch07: add error name to negative
these tests used flags: \n - negative and so
were not caught by the earlier naive grep for flags: [negative]
expect a SyntaxError
ch07: add error name to negative
these tests used flags: \n - negative and so
were not caught by the earlier naive grep for flags: [negative]
ch15: add error name to negative:
ch13: add error name to negative
only one test, did it manually
ch12: put error name in negative
ch11: add error name to negative
ch08: add expected error name
ch07-rest: insert error names rest of ch07
ch07-7.2,7.3: add error names
negative-errorname: pilot
pilot directory for replacing flags: [negative] with
negative: errorname
bestPractice: supply error names to negative
bestPractice: supply error names to negative
pilot directory for replacing flags: [negative] with
negative: errorname
ch07-7.2,7.3: add error names
ch07-rest: insert error names rest of ch07
ch08: add expected error name
ch11: add error name to negative
ch12: put error name in negative
ch13: add error name to negative
only one test, did it manually
ch15: add error name to negative:
ch07: add error name to negative
these tests used flags: \n - negative and so
were not caught by the earlier naive grep for flags: [negative]
bestPractice: supply error names to negative
negative-errorname: pilot
pilot directory for replacing flags: [negative] with
negative: errorname
ch07-7.2,7.3: add error names
ch07-rest: insert error names rest of ch07
ch08: add expected error name
ch11: add error name to negative
ch12: put error name in negative
ch13: add error name to negative
only one test, did it manually
ch15: add error name to negative:
ch07: add error name to negative
these tests used flags: \n - negative and so
were not caught by the earlier naive grep for flags: [negative]
expect a SyntaxError
negative: @anba fixes
bestPractice - remove added NotEarlyError
ch07,ch11 - expect ReferenceError (req'd by ES6)
ch12: specify type
negative: second half of @anba notes
correct test descriptions to expect ReferenceError only
enforce parse as block statement by adding `;`
remove needless include of $FAIL.js
expect real errors
ch12: negative error matching
ch13 error regexps
ch14 error regexps
add "description" header back
remove now-unused includes
remove needless include
sta.js:
slight change to Test262Error() semantics; message property now always set (default "")
make $ERROR a var
set $ERROR to function $ERROR so it can be overridden if needed
remove 2009 copyright in favor of 2012 copyright
REVERTED: remove never-used fn testFailed
testFailed actually used by $FAIL
remove obsolete fn $INCLUDE
ed.js:
remove commented-out obsolete code
test262.py:
remove always-included harness file that provides no functions used by any extant test
add comment line to nonstrict
cth.js: define `print` for node, cscript
use cth (console test harness) to define functions
for console runner. V8, Spidermonkey (js) and JavaScriptCore (jsc)
provide a function `print`. Provide a default `print` for node
and cscript
set print_handle to 'print' by default (can still override)
for cscript: wrap tests in try/catch so we get syntax errors
parseTestRecord: add support for YAML frontmatter
parseTestRecord: initial unit test for test record parser
parseTestRecord: refactor for testing
factor old parsing; add YAML parsing
runner: support "includes" from YAML frontmatter
support frontmatter "includes" in python runner
use test.includes if present instead of scanning test code with regex
harness: factor individual functions out into files
tools: handle YAML errors
tolerate missing keys in dictionary (flags, includes)
report filename when empty frontmatter block
new option --list-includes to test262.py
harness: factor helper functions into separate files
sth: remove extra close-paren (syntax error)
test_common: TDD; failing parse of YAML
common: use parseTestRecord (YAML-aware)
test262.py: only supply async helper scripts when test is async
sth.js: factor out function isAsyncTest()
timer.js: improve workaround for async tests when Promise is defined but setTimeout is noot
timer.js emulates setTimeout using Promise by doing a busy loop that checks
if `timeout` milliseconds have elapsed. Modified check to (timeLeft > 0) instead
of (!timeLeft) to prevent infinite loop when check does not happen to run
at precise millisecond timeout expires.
Because test262.py did not support the $INCLUDE directive, some helper
scripts were added to every test -- notably testIntl, timer, and donePrintHandle
Now that $INCLUDE is supported, these can be dropped, speeding overall test run time
packager.py cannot run due to syntax errors in a few script files
packagerConfig: use git instead of hg
test262: use pop instead of delete to avoid throw if property missing
S22.1.2.1_T*: fix docString header comment: s/b /**
S22.1.2.1_T3: fix docString header comment: s/b /**, fix end of docstring * / => */
doneprintHandle.js: make $DONE accept any falsy argument as meaning 'pass'
PromiseHelper.js: checkSequence: new helper fn for async tests
.gitignore: port .hgignore to .gitignore syntax
test262.py: support $INCLUDE directive in python test runner
S25.4.4.1*: tests to cover Section 25.4.4.1, Promise.all( iterable )
A1.1: Promise.all is callable
A1.2: Promise.all expects 1 argument
A2.1: Promise.all([]) is a Promise
A2.2: Promise.all([]) is resolved immediately
A2.3: Promise.all([]) is resolved with a new empty array
A3.1: Promise.all expects an iterable argument
- Removed a few old test cases that were redundant with new, more comprehensive ones.
- Added testIntl.js as standard include for all console tests in test262.py – see related bug 574.
- Added .jshintrc file for settings for the JSHint tool.
- Removed some old tests that were redundant with the new tests.
- Added testBuiltInObject.js as standard include for all console tests in test262.py – see related bug 574.
Leo Balter
Mike Pennisi
Till Schneidereit
Brian Terlson
Gregory Brail
Brian Terlson
smikes
Domenic Denicola
NikSurya
Yusuke Suzuki
Prashanth Srinivasan
Dan Schaffer
Norbert Lindenberg
Mark Miller