Previously, test consumers were encouraged to insert a `throw` statement
as the first statement of tests for early errors. This recommendation
made tests harder to consume, and as an optional transformation,
consumers may have ignored it or simply been unaware it was made. By
explicitly including such a `throw` statement, the tests become more
literal, making them easier to consume and more transparent in their
expectations.
Document expectation for all tests for early errors to include an
explicit `throw` statement. Extend linting script to verify that
contributors are automatically notified of violations and to ensure that
future contributions satisfy this expectation.
A recent commit introduced a document that enumerated acceptable values
for the test "features" metadata tag. However, this list was incomplete,
and maintaining it placed extra burden on the project owners.
Restructure the document into a machine-readable format. Add entries for
all previously-omitted values. Add in-line documentation with
recommendations for maintenance of the file. Extend the project's
linting tool to validate tests according to the document's contents.
This script is intended to identify common test file formatting errors
prior to their acceptance into the project. It is designed to support
future extensions for additional validation rules.
Introduce configuration to allow build servers provided by the Travis CI
service to execute the test generation tool and commit the resultant
files to the canonical upstream repository.
Enabling this workflow required additional administrative work:
1. Create an account with TravisCI
2. Install the `travis` command-line utility
3. Create a "deploy key" and an encrypted version using the command
`./make.py github_deploy_key_enc`
4. Register the deploy key with the project's GitHub account
5. Check the encrypted deploy key to the repository
6. Configure the TravisCI service to automatically build this project
Because expectations regarding error "phase" are now expressed via test
meta-data, the test runner may now enforce this requirement on negative
tests.
Remove the "NotEarlyError" from the project source. This reduces the
amount of domain knowledge required to author tests and lessens the
potential for inconsistencies between tests.
Unlike PHP, in JavaScript ! has higher precedence than instanceof, thus !smth instanceof TypeError will never (unless @@hasInstance is defined) be true.
As written, the example for asserting runtime errors is written with an
early error. Because the error is expected to be reported prior to
program execution, the `assert.throws` function cannot be used to detect
it.
Demonstrate the usage of the helper function with a runtime error.
For asynchronous tests, the contract between test file and test runner
is implicit: runners are expected to inspect the source code for
references to a global `$DONE` identifier.
Promote a more explicit contract between test file and test runner by
introducing a new frontmatter "tag", `async`. This brings asynchronous
test configuration in-line with other configuration mechanisms and also
provides a more natural means of test filtering.
The modifications to test files was made programatically using the
`grep` and `sed` utilities:
$ grep "\$DONE" test/ -r --files-with-match --null | \
xargs -0 sed -i 's/^\(flags:\s*\)\[/\1[async, /g'
$ grep "\$DONE" test/ -rl --null | \
xargs -0 grep -E '^flags:' --files-without-match --null | \
xargs -0 sed -i 's/^---\*\//flags: [async]\n---*\//'
It was recently decided to prefer the new `id` tag over the existing
`es5id` and `es6id` tag when authoring tests. Update the contribution
guidelines to reference the new tag.
Some tests involving the directive prologue are invalidated by source
text transformations that insert executable code in the beginning of the
script. Implement a `raw` flag that allows these tests to opt-out of
this transformation. Update the relevant tests to use this flag (and
remove references to globals only available when code is injected).
Update the Python runner accordingly:
- Do not run tests marked as "raw" in strict mode
- Reject invalid test configurations
Update the browser runner accordingly:
- Do not modify the script body of tests marked as "raw"
Unlike the console runner, the browser runner does not modify the
strictness of tests prior to running them. Regardless of a given test's
metadata, it runs every test exactly once, and it never enables strict
mode. This means that tests intended to function in strict mode must
declare the "use strict"; directive prologue in addition to the
`onlyStrict` flag.
For any test that specifies the `onlyStrict` metadata flag, transform
the source code by injecting a "use strict" directive prologue prior to
running the test.
- Remove trailing white space
- Streamline documentation of test tags
- Do not reference obsolete tags
- Document `features` frontmatter tag
- Document `es6id` frontmatter tag
- Omit unnecessary detail about test262 website generation. This is not
directly useful to potential test contributors. Implementation details
like these can be taken for granted by that audience.
- Remove documentation on YAML syntax. Details on YAML may be helpful
for some new contributors, but this document should not attempt to
cover the topic (especially not from the description of a specific
frontmatter entry). Replace with a link to a more comprehensive source
as this will be more generally useful to those who need it (and less
obtrusive for those who do not).
- Consolidate information on test helpers
- Document `assert` helpers
- Update instructions for asserting errors. Since the introduction of
`assert.throws` in gh-22, the preferred means of expressing
expectations regarding errors has changed. Update the CONTRIBUTING.md
file to reflect the latest approach. Explain purpose of `throw
NotEarlyError;` in example test.
- Re-order information on file names. The inconsistency in the project's
file names should not go unmentioned, but neither should it not
preceed instructions for the accepted approach to namine tests.
- More clearly document required frontmatter tags. Explicitly list
`description` as a required frontmatter tag, implicitly identifying
all other tags as optional.
CONTRIBUTING.md
- document `timeout` tag
- reorder tags in frontmatter doc
- minor cleanups
- minor fixes
- add style note
- reformat flags
- remove discussion of obsolete $INCLUDE
- incorporate line notes from @domenic
- integrate additional comments
- add links back, move arg notes down
- Raise outline level by one
README.md
- link to CONTRIBUTING
Add section on test environment
Add section on custom helpers
describe YAML frontmatter
Fix minor formatting errors
document $INCLUDE as obsolete
Change documentation of negative error
Move test environment and custom helpers down
indent copyright and frontmatter sections
better description of the async calls between a promise
and the functions in its `.then`
Correct Early Error example: don't throw a string
CONSOLE-RUNNER: split runner doc into new file
add troubleshooting section
add a table showing which print handle to use when
running async tests through test262.py runner
@anba contributed information about async tests in
SpiderMonkey and JavaScriptCore
add a section on requirements for using console test runner test262.py
add a section on asynchronous tests
add subsection to authoring guidelines with suggestion for test names
Mike Pennisi
Rick Waldron
Mathias Bynens
Leo Balter
Tim Disney
Aleksey Shvayka
Gorkem Yakin
Brian Terlson
Domenic Denicola
Sam Mikes
Murat Sutunc