tyler.durden
/
test262
mirror of https://github.com/tc39/test262.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
test262/CONTRIBUTING.md

451 lines
26 KiB
Markdown
Raw Permalink Normal View History

document `timeout` and clean up CONTRIBUTING.md - document `timeout` tag - reorder tags in frontmatter doc - minor cleanups - minor fixes - add style note - reformat flags - remove discussion of obsolete $INCLUDE - incorporate line notes from @domenic - integrate additional comments - add links back, move arg notes down - Raise outline level by one README.md - link to CONTRIBUTING
2014-08-22 17:47:18 +02:00
# Test262 Authoring Guidelines
CONTRIBUTING: document test262.py, async tests add a section on requirements for using console test runner test262.py add a section on asynchronous tests add subsection to authoring guidelines with suggestion for test names
2014-07-21 17:39:10 +02:00
CONTRIBUTING.md: Adds 'Source Material' section with 'needs tests' pr list (#1054)
2017-06-13 00:29:58 +02:00
## Source Material
- https://tc39.github.io/ecma262/
- https://github.com/tc39/ecma262/pulls?q=is%3Apr+is%3Aopen+label%3A%22needs+tests%22
+ This is a list of the "needs tests" PRs for changes to the specification.
Acceptable Tests, Unacceptable Tests
2020-09-11 19:41:08 +02:00
## Acceptable Tests
Update CONTRIBUTING.md (#3005)
2021-06-10 17:36:04 +02:00
Any test that exercises observable grammar or semantics, originating with citable, normative text in the latest draft of the [ECMAScript Language Specification](https://tc39.es/ecma262/), the [ECMAScript Internationalization API Specification](https://tc39.es/ecma402/), the [The JSON Data
Acceptable Tests, Unacceptable Tests
2020-09-11 19:41:08 +02:00
Interchange Syntax](https://ecma-international.org/publications/standards/Ecma-404.htm), a [Stage 3](https://github.com/tc39/proposals#stage-3) proposal or a Pull Request which makes a normative change to any of those specifications.
## Unacceptable Tests
Any test that restricts potentially valid extensions to the ECMAScript Language will not be accepted. Implementations are allowed to extend the language in any way that does not contradict the normative grammar specification, nor violate the specification's [Forbidden Extensions](https://tc39.es/ecma262/#sec-forbidden-extensions) section, which clearly lists the only exceptions to this rule.
document `timeout` and clean up CONTRIBUTING.md - document `timeout` tag - reorder tags in frontmatter doc - minor cleanups - minor fixes - add style note - reformat flags - remove discussion of obsolete $INCLUDE - incorporate line notes from @domenic - integrate additional comments - add links back, move arg notes down - Raise outline level by one README.md - link to CONTRIBUTING
2014-08-22 17:47:18 +02:00
## Test Case Names
CONTRIBUTING: document test262.py, async tests add a section on requirements for using console test runner test262.py add a section on asynchronous tests add subsection to authoring guidelines with suggestion for test names
2014-07-21 17:39:10 +02:00
Update contribution guidelines - Remove trailing white space - Streamline documentation of test tags - Do not reference obsolete tags - Document `features` frontmatter tag - Document `es6id` frontmatter tag - Omit unnecessary detail about test262 website generation. This is not directly useful to potential test contributors. Implementation details like these can be taken for granted by that audience. - Remove documentation on YAML syntax. Details on YAML may be helpful for some new contributors, but this document should not attempt to cover the topic (especially not from the description of a specific frontmatter entry). Replace with a link to a more comprehensive source as this will be more generally useful to those who need it (and less obtrusive for those who do not). - Consolidate information on test helpers - Document `assert` helpers - Update instructions for asserting errors. Since the introduction of `assert.throws` in gh-22, the preferred means of expressing expectations regarding errors has changed. Update the CONTRIBUTING.md file to reflect the latest approach. Explain purpose of `throw NotEarlyError;` in example test. - Re-order information on file names. The inconsistency in the project's file names should not go unmentioned, but neither should it not preceed instructions for the accepted approach to namine tests. - More clearly document required frontmatter tags. Explicitly list `description` as a required frontmatter tag, implicitly identifying all other tags as optional.
2015-05-25 23:51:55 +02:00
Test cases should be created in files that are named to identify the feature or API that's being tested.
CONTRIBUTING: document test262.py, async tests add a section on requirements for using console test runner test262.py add a section on asynchronous tests add subsection to authoring guidelines with suggestion for test names
2014-07-21 17:39:10 +02:00
Update filename character restriction in documention, linter and a few filenames
2018-05-02 22:14:31 +02:00
The names should use alphanumeric characters and `.`, `-`, `_`. Otherwise, there is no strict naming convention, but the file names should be human readable, helpful and, ideally, consistent within a single directory. For examples:
CONTRIBUTING: document test262.py, async tests add a section on requirements for using console test runner test262.py add a section on asynchronous tests add subsection to authoring guidelines with suggestion for test names
2014-07-21 17:39:10 +02:00
Contributing: updates the Test Case Names section. Closes gh-150. Closes gh-136
2015-02-13 23:39:17 +01:00
- `Math.fround` handling of `Infinity`: `test/built-ins/Math/fround/Math.fround_Infinity.js`
update CONTRIBUTING.md to reflect current practices
2017-10-06 22:09:39 +02:00
- `Array.prototype.find` use with `Proxy`: `test/built-ins/Array/prototype/find/Array.prototype.find_callable-Proxy-1.js`
Contributing: updates the Test Case Names section. Closes gh-150. Closes gh-136
2015-02-13 23:39:17 +01:00
- `arguments` implements an `iterator` interface: `test/language/arguments-object/iterator-interface.js`
CONTRIBUTING: document test262.py, async tests add a section on requirements for using console test runner test262.py add a section on asynchronous tests add subsection to authoring guidelines with suggestion for test names
2014-07-21 17:39:10 +02:00
update CONTRIBUTING.md to reflect current practices
2017-10-06 22:09:39 +02:00
See the following directory trees for further recommended examples:
- [test/built-ins/Math/fround](./test/built-ins/Math/fround/)
- [test/built-ins/WeakMap](./test/built-ins/WeakMap/)
- [test/language/arguments-object](./test/language/arguments-object/)
Update CONTRIBUTING.md vocabulary for consistency with YAML "tag" has a specific and different meaning; the proper term for this page is "key" cf. https://yaml.org/spec/1.2/spec.html#id2761292
2019-03-29 19:13:51 +01:00
**Note** The project is currently transitioning from a naming system based on specification section numbers. There remains a substantial number of tests that conform to this outdated convention; contributors should ignore that approach when introducing new tests and instead encode this information using [the `esid` frontmatter key](#esid).
CONTRIBUTING: more notes on test writing Add section on test environment Add section on custom helpers describe YAML frontmatter Fix minor formatting errors document $INCLUDE as obsolete Change documentation of negative error Move test environment and custom helpers down indent copyright and frontmatter sections better description of the async calls between a promise and the functions in its `.then` Correct Early Error example: don't throw a string CONSOLE-RUNNER: split runner doc into new file add troubleshooting section
2014-07-30 00:27:20 +02:00
document `timeout` and clean up CONTRIBUTING.md - document `timeout` tag - reorder tags in frontmatter doc - minor cleanups - minor fixes - add style note - reformat flags - remove discussion of obsolete $INCLUDE - incorporate line notes from @domenic - integrate additional comments - add links back, move arg notes down - Raise outline level by one README.md - link to CONTRIBUTING
2014-08-22 17:47:18 +02:00
## Test Case Style
CONTRIBUTING: more notes on test writing Add section on test environment Add section on custom helpers describe YAML frontmatter Fix minor formatting errors document $INCLUDE as obsolete Change documentation of negative error Move test environment and custom helpers down indent copyright and frontmatter sections better description of the async calls between a promise and the functions in its `.then` Correct Early Error example: don't throw a string CONSOLE-RUNNER: split runner doc into new file add troubleshooting section
2014-07-30 00:27:20 +02:00
A test file has three sections: Copyright, Frontmatter, and Body. A test looks roughly like this:
Added contributing.md and improved README
2014-07-04 02:13:42 +02:00
```javascript
Provide more useful and realistic test examples in CONTRIBUTING.md Per advice from @leobalter, based upon content from test/language/expressions/dynamic-import
2019-03-29 19:22:40 +01:00
// Copyright (C) $Year $ContributorName. All rights reserved.
Added contributing.md and improved README
2014-07-04 02:13:42 +02:00
// This code is governed by the BSD license found in the LICENSE file.
Update contribution guidelines - Remove trailing white space - Streamline documentation of test tags - Do not reference obsolete tags - Document `features` frontmatter tag - Document `es6id` frontmatter tag - Omit unnecessary detail about test262 website generation. This is not directly useful to potential test contributors. Implementation details like these can be taken for granted by that audience. - Remove documentation on YAML syntax. Details on YAML may be helpful for some new contributors, but this document should not attempt to cover the topic (especially not from the description of a specific frontmatter entry). Replace with a link to a more comprehensive source as this will be more generally useful to those who need it (and less obtrusive for those who do not). - Consolidate information on test helpers - Document `assert` helpers - Update instructions for asserting errors. Since the introduction of `assert.throws` in gh-22, the preferred means of expressing expectations regarding errors has changed. Update the CONTRIBUTING.md file to reflect the latest approach. Explain purpose of `throw NotEarlyError;` in example test. - Re-order information on file names. The inconsistency in the project's file names should not go unmentioned, but neither should it not preceed instructions for the accepted approach to namine tests. - More clearly document required frontmatter tags. Explicitly list `description` as a required frontmatter tag, implicitly identifying all other tags as optional.
2015-05-25 23:51:55 +02:00
/*---
Provide more useful and realistic test examples in CONTRIBUTING.md Per advice from @leobalter, based upon content from test/language/expressions/dynamic-import
2019-03-29 19:22:40 +01:00
description: >
brief description, e.g. "Non-numeric input must be rejected with a TypeError"
esid: reference to spec section, e.g. "sec-well-known-symbols"
info: |
verbose test description, multiple lines OK.
(info should contain relevant, direct quotes from ECMAScript if possible)
For example:
String.fromCodePoint ( ..._codePoints_ )
5. Repeat, while _nextIndex_ < _length_
a. Let _next_ be _codePoints_[_nextIndex_].
b. Let _nextCP_ be ? ToNumber(_next_).
d. If _nextCP_ < 0 or _nextCP_ > 0x10FFFF, throw a *RangeError* exception.
features: [example]
CONTRIBUTING: more notes on test writing Add section on test environment Add section on custom helpers describe YAML frontmatter Fix minor formatting errors document $INCLUDE as obsolete Change documentation of negative error Move test environment and custom helpers down indent copyright and frontmatter sections better description of the async calls between a promise and the functions in its `.then` Correct Early Error example: don't throw a string CONSOLE-RUNNER: split runner doc into new file add troubleshooting section
2014-07-30 00:27:20 +02:00
---*/
Update contribution guidelines - Remove trailing white space - Streamline documentation of test tags - Do not reference obsolete tags - Document `features` frontmatter tag - Document `es6id` frontmatter tag - Omit unnecessary detail about test262 website generation. This is not directly useful to potential test contributors. Implementation details like these can be taken for granted by that audience. - Remove documentation on YAML syntax. Details on YAML may be helpful for some new contributors, but this document should not attempt to cover the topic (especially not from the description of a specific frontmatter entry). Replace with a link to a more comprehensive source as this will be more generally useful to those who need it (and less obtrusive for those who do not). - Consolidate information on test helpers - Document `assert` helpers - Update instructions for asserting errors. Since the introduction of `assert.throws` in gh-22, the preferred means of expressing expectations regarding errors has changed. Update the CONTRIBUTING.md file to reflect the latest approach. Explain purpose of `throw NotEarlyError;` in example test. - Re-order information on file names. The inconsistency in the project's file names should not go unmentioned, but neither should it not preceed instructions for the accepted approach to namine tests. - More clearly document required frontmatter tags. Explicitly list `description` as a required frontmatter tag, implicitly identifying all other tags as optional.
2015-05-25 23:51:55 +02:00
Provide more useful and realistic test examples in CONTRIBUTING.md Per advice from @leobalter, based upon content from test/language/expressions/dynamic-import
2019-03-29 19:22:40 +01:00
var maxCodePoint = 0x10FFFF;
var maxCodePointString = '\u{10FFFF}';
assert.sameValue(String.fromCodePoint(maxCodePoint), maxCodePointString,
'String.fromCodePoint can produce a string including the highest code point');
assert.throws(
RangeError,
function() {
String.fromCodePoint(maxCodePoint + 1);
},
'String.fromCodePoint throws a RangeError when an argument is greater than the highest code point value'
);
Added contributing.md and improved README
2014-07-04 02:13:42 +02:00
```
CONTRIBUTING: document test262.py, async tests add a section on requirements for using console test runner test262.py add a section on asynchronous tests add subsection to authoring guidelines with suggestion for test names
2014-07-21 17:39:10 +02:00
document `timeout` and clean up CONTRIBUTING.md - document `timeout` tag - reorder tags in frontmatter doc - minor cleanups - minor fixes - add style note - reformat flags - remove discussion of obsolete $INCLUDE - incorporate line notes from @domenic - integrate additional comments - add links back, move arg notes down - Raise outline level by one README.md - link to CONTRIBUTING
2014-08-22 17:47:18 +02:00
### Copyright
CONTRIBUTING: more notes on test writing Add section on test environment Add section on custom helpers describe YAML frontmatter Fix minor formatting errors document $INCLUDE as obsolete Change documentation of negative error Move test environment and custom helpers down indent copyright and frontmatter sections better description of the async calls between a promise and the functions in its `.then` Correct Early Error example: don't throw a string CONSOLE-RUNNER: split runner doc into new file add troubleshooting section
2014-07-30 00:27:20 +02:00
Update contributing file to describe current requirements for the copyright header
2019-07-15 19:40:27 +02:00
The copyright block should be the first section of the test. The copyright block must use `//` style comments.
The copyright should follow this format:
```js
// Copyright (C) $Year $ContributorName. All rights reserved.
// This code is governed by the BSD license found in the LICENSE file.
```
Where
Update CONTRIBUTING.md (#3004) Fixed some grammatical issues in the document.
2021-06-10 17:34:28 +02:00
- `$Year` must be a 4 digit single year. It should not be a year range. If it's extremely necessary to add multiple years, repeat line for each year.
Update contributing file to describe current requirements for the copyright header
2019-07-15 19:40:27 +02:00
- `$ContributorName` must be a legal (natural or juridical) person's name.
The code must be a BSD or BSD-style compatible with the license of this project. Therefore, the line following the year and name parts should be written as in the example above.
CONTRIBUTING: more notes on test writing Add section on test environment Add section on custom helpers describe YAML frontmatter Fix minor formatting errors document $INCLUDE as obsolete Change documentation of negative error Move test environment and custom helpers down indent copyright and frontmatter sections better description of the async calls between a promise and the functions in its `.then` Correct Early Error example: don't throw a string CONSOLE-RUNNER: split runner doc into new file add troubleshooting section
2014-07-30 00:27:20 +02:00
document `timeout` and clean up CONTRIBUTING.md - document `timeout` tag - reorder tags in frontmatter doc - minor cleanups - minor fixes - add style note - reformat flags - remove discussion of obsolete $INCLUDE - incorporate line notes from @domenic - integrate additional comments - add links back, move arg notes down - Raise outline level by one README.md - link to CONTRIBUTING
2014-08-22 17:47:18 +02:00
### Frontmatter
CONTRIBUTING: more notes on test writing Add section on test environment Add section on custom helpers describe YAML frontmatter Fix minor formatting errors document $INCLUDE as obsolete Change documentation of negative error Move test environment and custom helpers down indent copyright and frontmatter sections better description of the async calls between a promise and the functions in its `.then` Correct Early Error example: don't throw a string CONSOLE-RUNNER: split runner doc into new file add troubleshooting section
2014-07-30 00:27:20 +02:00
tools: enforce restriction on YAML Some consumers have reported difficulty parsing an uncommon YAML construction [1] [2] [3]. Extend the linter to help ensure that construction is not used in future contributions. [1] https://github.com/tc39/test262/issues/1997 [2] https://github.com/tc39/test262/pull/2505 [3] https://github.com/tc39/test262/issues/3171
2021-09-03 20:16:21 +02:00
The Test262 frontmatter is a string of [YAML](https://en.wikipedia.org/wiki/YAML) enclosed by the comment start tag `/*---` and end tag `---*/`. In order to simplify parsing, scalar values spanning multiple lines may not be expressed using "flow" notation.
There must be exactly one Frontmatter per test.
CONTRIBUTING: more notes on test writing Add section on test environment Add section on custom helpers describe YAML frontmatter Fix minor formatting errors document $INCLUDE as obsolete Change documentation of negative error Move test environment and custom helpers down indent copyright and frontmatter sections better description of the async calls between a promise and the functions in its `.then` Correct Early Error example: don't throw a string CONSOLE-RUNNER: split runner doc into new file add troubleshooting section
2014-07-30 00:27:20 +02:00
Update CONTRIBUTING.md vocabulary for consistency with YAML "tag" has a specific and different meaning; the proper term for this page is "key" cf. https://yaml.org/spec/1.2/spec.html#id2761292
2019-03-29 19:13:51 +01:00
Test262 supports the following keys:
CONTRIBUTING: more notes on test writing Add section on test environment Add section on custom helpers describe YAML frontmatter Fix minor formatting errors document $INCLUDE as obsolete Change documentation of negative error Move test environment and custom helpers down indent copyright and frontmatter sections better description of the async calls between a promise and the functions in its `.then` Correct Early Error example: don't throw a string CONSOLE-RUNNER: split runner doc into new file add troubleshooting section
2014-07-30 00:27:20 +02:00
Update contribution guidelines - Remove trailing white space - Streamline documentation of test tags - Do not reference obsolete tags - Document `features` frontmatter tag - Document `es6id` frontmatter tag - Omit unnecessary detail about test262 website generation. This is not directly useful to potential test contributors. Implementation details like these can be taken for granted by that audience. - Remove documentation on YAML syntax. Details on YAML may be helpful for some new contributors, but this document should not attempt to cover the topic (especially not from the description of a specific frontmatter entry). Replace with a link to a more comprehensive source as this will be more generally useful to those who need it (and less obtrusive for those who do not). - Consolidate information on test helpers - Document `assert` helpers - Update instructions for asserting errors. Since the introduction of `assert.throws` in gh-22, the preferred means of expressing expectations regarding errors has changed. Update the CONTRIBUTING.md file to reflect the latest approach. Explain purpose of `throw NotEarlyError;` in example test. - Re-order information on file names. The inconsistency in the project's file names should not go unmentioned, but neither should it not preceed instructions for the accepted approach to namine tests. - More clearly document required frontmatter tags. Explicitly list `description` as a required frontmatter tag, implicitly identifying all other tags as optional.
2015-05-25 23:51:55 +02:00
- [**description**](#description) (required)
Use consistent ordering for documenting test frontmatter keys
2019-03-29 19:07:48 +01:00
- [**esid**](#esid) (required for new tests)
document `timeout` and clean up CONTRIBUTING.md - document `timeout` tag - reorder tags in frontmatter doc - minor cleanups - minor fixes - add style note - reformat flags - remove discussion of obsolete $INCLUDE - incorporate line notes from @domenic - integrate additional comments - add links back, move arg notes down - Raise outline level by one README.md - link to CONTRIBUTING
2014-08-22 17:47:18 +02:00
- [**info**](#info)
- [**negative**](#negative)
- [**includes**](#includes)
- [**author**](#author)
- [**flags**](#flags)
CONTRIBUTING.md: More detail about "features"
2017-10-17 22:38:39 +02:00
- [**features**](#features) (required for new tests written for new features)
Document more test metadata in contribution guide
2021-11-12 20:17:44 +01:00
- [**locale**](#locale)
document `timeout` and clean up CONTRIBUTING.md - document `timeout` tag - reorder tags in frontmatter doc - minor cleanups - minor fixes - add style note - reformat flags - remove discussion of obsolete $INCLUDE - incorporate line notes from @domenic - integrate additional comments - add links back, move arg notes down - Raise outline level by one README.md - link to CONTRIBUTING
2014-08-22 17:47:18 +02:00
Update CONTRIBUTING.md vocabulary for consistency with YAML "tag" has a specific and different meaning; the proper term for this page is "key" cf. https://yaml.org/spec/1.2/spec.html#id2761292
2019-03-29 19:13:51 +01:00
The following keys are deprecated, but exist in old tests:
update CONTRIBUTING.md to reflect current practices
2017-10-06 22:09:39 +02:00
- [**es5id**](#es5id) (deprecated)
- [**es6id**](#es6id) (deprecated)
document `timeout` and clean up CONTRIBUTING.md - document `timeout` tag - reorder tags in frontmatter doc - minor cleanups - minor fixes - add style note - reformat flags - remove discussion of obsolete $INCLUDE - incorporate line notes from @domenic - integrate additional comments - add links back, move arg notes down - Raise outline level by one README.md - link to CONTRIBUTING
2014-08-22 17:47:18 +02:00
#### description
Improve CONTRIBUTING.md formatting
2019-03-29 19:20:16 +01:00
`description: [string]`
document `timeout` and clean up CONTRIBUTING.md - document `timeout` tag - reorder tags in frontmatter doc - minor cleanups - minor fixes - add style note - reformat flags - remove discussion of obsolete $INCLUDE - incorporate line notes from @domenic - integrate additional comments - add links back, move arg notes down - Raise outline level by one README.md - link to CONTRIBUTING
2014-08-22 17:47:18 +02:00
Improve CONTRIBUTING.md formatting
2019-03-29 19:20:16 +01:00
This is one of two required frontmatter keys. The description should be a short, one-line description of the purpose of this testcase. We suggest that the description be kept to less than 100 characters, but clarity is preferred over brevity.
CONTRIBUTING: document test262.py, async tests add a section on requirements for using console test runner test262.py add a section on asynchronous tests add subsection to authoring guidelines with suggestion for test names
2014-07-21 17:39:10 +02:00
Added contributing.md and improved README
2014-07-04 02:13:42 +02:00
Eg: Insert <LS> between chunks of one string
CONTRIBUTING: document test262.py, async tests add a section on requirements for using console test runner test262.py add a section on asynchronous tests add subsection to authoring guidelines with suggestion for test names
2014-07-21 17:39:10 +02:00
Use consistent ordering for documenting test frontmatter keys
2019-03-29 19:07:48 +01:00
#### esid
Improve CONTRIBUTING.md formatting
2019-03-29 19:20:16 +01:00
`esid: [spec-id]`
Use consistent ordering for documenting test frontmatter keys
2019-03-29 19:07:48 +01:00
Update CONTRIBUTING.md vocabulary for consistency with YAML "tag" has a specific and different meaning; the proper term for this page is "key" cf. https://yaml.org/spec/1.2/spec.html#id2761292
2019-03-29 19:13:51 +01:00
This key is required for all new feature tests. This key identifies the hash ID from the portion of the ECMAScript draft which is most recent to the date the test was added. It represents the anchors on the generated HTML version of the specs. E.g.: `esid: sec-typedarray-length`. This key might be used to replace a `es6id` or further.
Use consistent ordering for documenting test frontmatter keys
2019-03-29 19:07:48 +01:00
When writing a new test for a Stage 3+ spec not yet published on the draft, the `pending` value can be used while a hash ID is not available.
document `timeout` and clean up CONTRIBUTING.md - document `timeout` tag - reorder tags in frontmatter doc - minor cleanups - minor fixes - add style note - reformat flags - remove discussion of obsolete $INCLUDE - incorporate line notes from @domenic - integrate additional comments - add links back, move arg notes down - Raise outline level by one README.md - link to CONTRIBUTING
2014-08-22 17:47:18 +02:00
#### info
Improve CONTRIBUTING.md formatting
2019-03-29 19:20:16 +01:00
`info: [multiline string]`
document `timeout` and clean up CONTRIBUTING.md - document `timeout` tag - reorder tags in frontmatter doc - minor cleanups - minor fixes - add style note - reformat flags - remove discussion of obsolete $INCLUDE - incorporate line notes from @domenic - integrate additional comments - add links back, move arg notes down - Raise outline level by one README.md - link to CONTRIBUTING
2014-08-22 17:47:18 +02:00
Fix CONTRIBUTING.md description of the info frontmatter tag
2017-10-04 23:45:03 +02:00
This allows a long, free-form comment. The comment is almost always a direct
quote from ECMAScript. It is used to indicate the observable being tested
within the file.
document `timeout` and clean up CONTRIBUTING.md - document `timeout` tag - reorder tags in frontmatter doc - minor cleanups - minor fixes - add style note - reformat flags - remove discussion of obsolete $INCLUDE - incorporate line notes from @domenic - integrate additional comments - add links back, move arg notes down - Raise outline level by one README.md - link to CONTRIBUTING
2014-08-22 17:47:18 +02:00
Fix CONTRIBUTING.md description of the info frontmatter tag
2017-10-04 23:45:03 +02:00
For example:
CONTRIBUTING.md: Fix code block
2018-09-12 20:54:31 +02:00
```
Fix CONTRIBUTING.md description of the info frontmatter tag
2017-10-04 23:45:03 +02:00
/*---
esid: sec-weakset.prototype.has
description: Throws TypeError if `this` is not Object.
CONTRIBUTING.md: Fix code block
2018-09-12 20:54:31 +02:00
info: |
Provide more useful and realistic test examples in CONTRIBUTING.md Per advice from @leobalter, based upon content from test/language/expressions/dynamic-import
2019-03-29 19:22:40 +01:00
String.fromCodePoint ( ..._codePoints_ )
Fix CONTRIBUTING.md description of the info frontmatter tag
2017-10-04 23:45:03 +02:00
Provide more useful and realistic test examples in CONTRIBUTING.md Per advice from @leobalter, based upon content from test/language/expressions/dynamic-import
2019-03-29 19:22:40 +01:00
5. Repeat, while _nextIndex_ < _length_
a. Let _next_ be _codePoints_[_nextIndex_].
b. Let _nextCP_ be ? ToNumber(_next_).
d. If _nextCP_ < 0 or _nextCP_ > 0x10FFFF, throw a *RangeError* exception.
Fix CONTRIBUTING.md description of the info frontmatter tag
2017-10-04 23:45:03 +02:00
---*/
CONTRIBUTING.md: Fix code block
2018-09-12 20:54:31 +02:00
```
Fix CONTRIBUTING.md description of the info frontmatter tag
2017-10-04 23:45:03 +02:00
Note: Adding more context than the direct quote from ECMAScript should rarely
be necessary. If you must add context to the quote, use the JavaScript
single line comment syntax.
document `timeout` and clean up CONTRIBUTING.md - document `timeout` tag - reorder tags in frontmatter doc - minor cleanups - minor fixes - add style note - reformat flags - remove discussion of obsolete $INCLUDE - incorporate line notes from @domenic - integrate additional comments - add links back, move arg notes down - Raise outline level by one README.md - link to CONTRIBUTING
2014-08-22 17:47:18 +02:00
#### negative
Improve CONTRIBUTING.md formatting
2019-03-29 19:20:16 +01:00
`negative: [dictionary containing "phase" and "type" keys]`
CONTRIBUTING: document test262.py, async tests add a section on requirements for using console test runner test262.py add a section on asynchronous tests add subsection to authoring guidelines with suggestion for test names
2014-07-21 17:39:10 +02:00
tools: enforce restriction on YAML negative key The phase field must precede the type field for negative tests to have a consistent style and be able to parse easier. Related to the goal of https://github.com/tc39/test262/issues/1997 Added this check to the linting script and updated tests accordingly.
2022-01-06 13:01:51 +01:00
This means the test is expected to throw an error of the given type. If no error is thrown, a test failure is reported. The **phase** field must precede the **type** field.
CONTRIBUTING: document test262.py, async tests add a section on requirements for using console test runner test262.py add a section on asynchronous tests add subsection to authoring guidelines with suggestion for test names
2014-07-21 17:39:10 +02:00
Rename negative test "phase" for module resolution
2017-12-03 06:20:06 +01:00
- **phase** - Negative tests whose **phase** value is "parse" must produce the specified error prior to executing code. The value "resolution" indicates that the error is expected to result while performing ES2015 module resolution. The value "runtime" dictates that the error is expected to be produced as a result of executing the test code.
tools: enforce restriction on YAML negative key The phase field must precede the type field for negative tests to have a consistent style and be able to parse easier. Related to the goal of https://github.com/tc39/test262/issues/1997 Added this check to the linting script and updated tests accordingly.
2022-01-06 13:01:51 +01:00
- **type** - If an error is thrown, it is implicitly converted to a string. In order for the test to pass, this value must match the name of the error constructor.
CONTRIBUTING: more notes on test writing Add section on test environment Add section on custom helpers describe YAML frontmatter Fix minor formatting errors document $INCLUDE as obsolete Change documentation of negative error Move test environment and custom helpers down indent copyright and frontmatter sections better description of the async calls between a promise and the functions in its `.then` Correct Early Error example: don't throw a string CONSOLE-RUNNER: split runner doc into new file add troubleshooting section
2014-07-30 00:27:20 +02:00
Update CONTRIBUTING.md vocabulary for consistency with YAML "tag" has a specific and different meaning; the proper term for this page is "key" cf. https://yaml.org/spec/1.2/spec.html#id2761292
2019-03-29 19:13:51 +01:00
For best practices on how to use the negative key please see [Handling Errors and Negative Test Cases](#handling-errors-and-negative-test-cases), below.
CONTRIBUTING: more notes on test writing Add section on test environment Add section on custom helpers describe YAML frontmatter Fix minor formatting errors document $INCLUDE as obsolete Change documentation of negative error Move test environment and custom helpers down indent copyright and frontmatter sections better description of the async calls between a promise and the functions in its `.then` Correct Early Error example: don't throw a string CONSOLE-RUNNER: split runner doc into new file add troubleshooting section
2014-07-30 00:27:20 +02:00
Update documentation
2016-03-13 18:26:10 +01:00
For example:
Improve CONTRIBUTING.md formatting
2019-03-29 19:20:16 +01:00
```
negative:
phase: parse
Test update for letting all early errors be SyntaxErrors
2019-05-03 08:56:41 +02:00
type: SyntaxError
Improve CONTRIBUTING.md formatting
2019-03-29 19:20:16 +01:00
```
Update documentation
2016-03-13 18:26:10 +01:00
document `timeout` and clean up CONTRIBUTING.md - document `timeout` tag - reorder tags in frontmatter doc - minor cleanups - minor fixes - add style note - reformat flags - remove discussion of obsolete $INCLUDE - incorporate line notes from @domenic - integrate additional comments - add links back, move arg notes down - Raise outline level by one README.md - link to CONTRIBUTING
2014-08-22 17:47:18 +02:00
#### includes
Improve CONTRIBUTING.md formatting
2019-03-29 19:20:16 +01:00
`includes: [file-list]`
document `timeout` and clean up CONTRIBUTING.md - document `timeout` tag - reorder tags in frontmatter doc - minor cleanups - minor fixes - add style note - reformat flags - remove discussion of obsolete $INCLUDE - incorporate line notes from @domenic - integrate additional comments - add links back, move arg notes down - Raise outline level by one README.md - link to CONTRIBUTING
2014-08-22 17:47:18 +02:00
Improve documentation of harness files & raw flag Prior to this commit, the term "harness" was used to refer both to the like-named directory in the repository and the external projects which consume the tests. Reserve the term "harness" to refer solely to the files maintained within this project's `harness/` directory, and use the term "runner" to refer to the external projects. Prior to this commit, the term "helper" was not defined and therefore introduced ambiguity in text that described harness files. Replace the term with more explicit wording in terms of the harness files. Because test runners have some leeway in how they include harness files, the requirement that "[t]he test source code must not be modified in any way" does not necessarily preclude the use of harness files. Update the instructions for interpreting the `raw` frontmatter flag to explicitly disallow harness files.
2021-11-19 21:08:18 +01:00
This key names a list of files in the `harness/` directory that will be included in the test environment prior to running the test. Filenames **must** include the `.js` extension.
tools: enforce restriction on YAML includes key Includes key should use flow notation to be able parsed easier as suggested in https://github.com/tc39/test262/issues/1997 Added this check to the linting script and updated tests accordingly.
2022-01-04 15:29:10 +01:00
Includes should use flow style without line break (separating items by commas and enclosing them in square brackets), block style (one line for each item, each line starting with a dash) isn't allowed.
Improve documentation of harness files & raw flag Prior to this commit, the term "harness" was used to refer both to the like-named directory in the repository and the external projects which consume the tests. Reserve the term "harness" to refer solely to the files maintained within this project's `harness/` directory, and use the term "runner" to refer to the external projects. Prior to this commit, the term "helper" was not defined and therefore introduced ambiguity in text that described harness files. Replace the term with more explicit wording in terms of the harness files. Because test runners have some leeway in how they include harness files, the requirement that "[t]he test source code must not be modified in any way" does not necessarily preclude the use of harness files. Update the instructions for interpreting the `raw` frontmatter flag to explicitly disallow harness files.
2021-11-19 21:08:18 +01:00
When some code is used repeatedly across a group of tests, it may be appropriate to define it in a harness file. This practice increases test complexity, so it should be applied sparingly.
CONTRIBUTING.md: minor fixes Minor fixes: typos, formatting, verb tense&agreement Add example of short-list `includes: []` and long-list `includes: \n - \n -`
2014-07-31 22:41:08 +02:00
tools: enforce restriction on YAML includes key Includes key should use flow notation to be able parsed easier as suggested in https://github.com/tc39/test262/issues/1997 Added this check to the linting script and updated tests accordingly.
2022-01-04 15:29:10 +01:00
For example:
```
includes: [include1.js, include2.js]
```
document `timeout` and clean up CONTRIBUTING.md - document `timeout` tag - reorder tags in frontmatter doc - minor cleanups - minor fixes - add style note - reformat flags - remove discussion of obsolete $INCLUDE - incorporate line notes from @domenic - integrate additional comments - add links back, move arg notes down - Raise outline level by one README.md - link to CONTRIBUTING
2014-08-22 17:47:18 +02:00
#### author
Improve CONTRIBUTING.md formatting
2019-03-29 19:20:16 +01:00
`author: [string]`
document `timeout` and clean up CONTRIBUTING.md - document `timeout` tag - reorder tags in frontmatter doc - minor cleanups - minor fixes - add style note - reformat flags - remove discussion of obsolete $INCLUDE - incorporate line notes from @domenic - integrate additional comments - add links back, move arg notes down - Raise outline level by one README.md - link to CONTRIBUTING
2014-08-22 17:47:18 +02:00
Update CONTRIBUTING.md vocabulary for consistency with YAML "tag" has a specific and different meaning; the proper term for this page is "key" cf. https://yaml.org/spec/1.2/spec.html#id2761292
2019-03-29 19:13:51 +01:00
This key is used to identify the author of a test case.
document `timeout` and clean up CONTRIBUTING.md - document `timeout` tag - reorder tags in frontmatter doc - minor cleanups - minor fixes - add style note - reformat flags - remove discussion of obsolete $INCLUDE - incorporate line notes from @domenic - integrate additional comments - add links back, move arg notes down - Raise outline level by one README.md - link to CONTRIBUTING
2014-08-22 17:47:18 +02:00
#### flags
Improve CONTRIBUTING.md formatting
2019-03-29 19:20:16 +01:00
`flags: [list]`
document `timeout` and clean up CONTRIBUTING.md - document `timeout` tag - reorder tags in frontmatter doc - minor cleanups - minor fixes - add style note - reformat flags - remove discussion of obsolete $INCLUDE - incorporate line notes from @domenic - integrate additional comments - add links back, move arg notes down - Raise outline level by one README.md - link to CONTRIBUTING
2014-08-22 17:47:18 +02:00
Update CONTRIBUTING.md vocabulary for consistency with YAML "tag" has a specific and different meaning; the proper term for this page is "key" cf. https://yaml.org/spec/1.2/spec.html#id2761292
2019-03-29 19:13:51 +01:00
This key is for boolean properties associated with the test.
CONTRIBUTING: more notes on test writing Add section on test environment Add section on custom helpers describe YAML frontmatter Fix minor formatting errors document $INCLUDE as obsolete Change documentation of negative error Move test environment and custom helpers down indent copyright and frontmatter sections better description of the async calls between a promise and the functions in its `.then` Correct Early Error example: don't throw a string CONSOLE-RUNNER: split runner doc into new file add troubleshooting section
2014-07-30 00:27:20 +02:00
Improve CONTRIBUTING.md formatting
2019-03-29 19:20:16 +01:00
- **onlyStrict** - only run the test in strict mode
- **noStrict** - only run the test in "sloppy" mode
- **module** - interpret the source text as
[module code](https://tc39.github.io/ecma262/#sec-modules)
Improve documentation of harness files & raw flag Prior to this commit, the term "harness" was used to refer both to the like-named directory in the repository and the external projects which consume the tests. Reserve the term "harness" to refer solely to the files maintained within this project's `harness/` directory, and use the term "runner" to refer to the external projects. Prior to this commit, the term "helper" was not defined and therefore introduced ambiguity in text that described harness files. Replace the term with more explicit wording in terms of the harness files. Because test runners have some leeway in how they include harness files, the requirement that "[t]he test source code must not be modified in any way" does not necessarily preclude the use of harness files. Update the instructions for interpreting the `raw` frontmatter flag to explicitly disallow harness files.
2021-11-19 21:08:18 +01:00
- **raw** - execute the test without any modification (no harness files will be
included); necessary to test the behavior of directive prologue; implies
Implement `raw` flag Some tests involving the directive prologue are invalidated by source text transformations that insert executable code in the beginning of the script. Implement a `raw` flag that allows these tests to opt-out of this transformation. Update the relevant tests to use this flag (and remove references to globals only available when code is injected). Update the Python runner accordingly: - Do not run tests marked as "raw" in strict mode - Reject invalid test configurations Update the browser runner accordingly: - Do not modify the script body of tests marked as "raw"
2015-06-08 21:35:16 +02:00
`noStrict`
Improve CONTRIBUTING.md formatting
2019-03-29 19:20:16 +01:00
- **async** - defer interpretation of test results until after the invocation
Fix markdown typo (#763)
2016-09-27 23:05:08 +02:00
of the global `$DONE` function
Improve CONTRIBUTING.md formatting
2019-03-29 19:20:16 +01:00
- **generated** - informative flag used to denote test files that were
Improve "Procedurally-generated tests" documentation
2021-07-28 01:43:23 +02:00
created procedurally using the project's test generation tool; refer to
[Procedurally-generated tests](#procedurally-generated-tests)
for more information on this process
Improve CONTRIBUTING.md formatting
2019-03-29 19:20:16 +01:00
- **CanBlockIsFalse** - only run the test when the [[CanBlock]] property of the [Agent Record](https://tc39.github.io/ecma262/#sec-agents) executing the test file is `false`
- **CanBlockIsTrue** - only run the test when the [[CanBlock]] property of the [Agent Record](https://tc39.github.io/ecma262/#sec-agents) executing the test file is `true`
Document more test metadata in contribution guide
2021-11-12 20:17:44 +01:00
- **non-deterministic** - informative flag used to communicate that the semantics under test are intentionally under-specified, so the test's passing or failing status is neither reliable nor an indication of conformance
document `timeout` and clean up CONTRIBUTING.md - document `timeout` tag - reorder tags in frontmatter doc - minor cleanups - minor fixes - add style note - reformat flags - remove discussion of obsolete $INCLUDE - incorporate line notes from @domenic - integrate additional comments - add links back, move arg notes down - Raise outline level by one README.md - link to CONTRIBUTING
2014-08-22 17:47:18 +02:00
Update contribution guidelines - Remove trailing white space - Streamline documentation of test tags - Do not reference obsolete tags - Document `features` frontmatter tag - Document `es6id` frontmatter tag - Omit unnecessary detail about test262 website generation. This is not directly useful to potential test contributors. Implementation details like these can be taken for granted by that audience. - Remove documentation on YAML syntax. Details on YAML may be helpful for some new contributors, but this document should not attempt to cover the topic (especially not from the description of a specific frontmatter entry). Replace with a link to a more comprehensive source as this will be more generally useful to those who need it (and less obtrusive for those who do not). - Consolidate information on test helpers - Document `assert` helpers - Update instructions for asserting errors. Since the introduction of `assert.throws` in gh-22, the preferred means of expressing expectations regarding errors has changed. Update the CONTRIBUTING.md file to reflect the latest approach. Explain purpose of `throw NotEarlyError;` in example test. - Re-order information on file names. The inconsistency in the project's file names should not go unmentioned, but neither should it not preceed instructions for the accepted approach to namine tests. - More clearly document required frontmatter tags. Explicitly list `description` as a required frontmatter tag, implicitly identifying all other tags as optional.
2015-05-25 23:51:55 +02:00
#### features
Improve CONTRIBUTING.md formatting
2019-03-29 19:20:16 +01:00
`features: [list]`
CONTRIBUTING: document test262.py, async tests add a section on requirements for using console test runner test262.py add a section on asynchronous tests add subsection to authoring guidelines with suggestion for test names
2014-07-21 17:39:10 +02:00
Update CONTRIBUTING.md (#3004) Fixed some grammatical issues in the document.
2021-06-10 17:34:28 +02:00
Some tests require the use of language features that are not directly described by the test file's location in the directory structure. These features should be specified with this key. See the [`features.txt`](features.txt) file for a complete list of available values. This key is required for new tests written for new features, but contributions will not be "blocked" if the key is missing from frontmatter. The committing maintainer is required to ensure that the key is present and contains the correct feature names; this can be done in a follow up commit.
CONTRIBUTING: more notes on test writing Add section on test environment Add section on custom helpers describe YAML frontmatter Fix minor formatting errors document $INCLUDE as obsolete Change documentation of negative error Move test environment and custom helpers down indent copyright and frontmatter sections better description of the async calls between a promise and the functions in its `.then` Correct Early Error example: don't throw a string CONSOLE-RUNNER: split runner doc into new file add troubleshooting section
2014-07-30 00:27:20 +02:00
Document more test metadata in contribution guide
2021-11-12 20:17:44 +01:00
#### locale
`locale: [list]`
Some tests require the use of one or more specific human languages as exposed by [ECMA402](https://tc39.es/ecma402/) as a means to verify semantics which cannot be observed in the abstract. Sch tests must declare their requirements by using this key to define an array of one or more valid language tags or subtags.
update CONTRIBUTING.md to reflect current practices
2017-10-06 22:09:39 +02:00
#### es5id
Improve CONTRIBUTING.md formatting
2019-03-29 19:20:16 +01:00
`es5id: [es5-test-id]`
update CONTRIBUTING.md to reflect current practices
2017-10-06 22:09:39 +02:00
Improve CONTRIBUTING.md formatting
2019-03-29 19:20:16 +01:00
**_DEPRECATED._**
update CONTRIBUTING.md to reflect current practices
2017-10-06 22:09:39 +02:00
Update CONTRIBUTING.md vocabulary for consistency with YAML "tag" has a specific and different meaning; the proper term for this page is "key" cf. https://yaml.org/spec/1.2/spec.html#id2761292
2019-03-29 19:13:51 +01:00
This key identifies the section number from the portion of the ECMAScript 5.1 or ECMAScript 3 standard that is tested by this test. It was automatically generated for tests that were originally written for the ES5 (or earlier) version of the test suite and are now part of the ES6 version. You can use the es5id to discover the relevant portion of the ECMAScript standard by looking up the section number in [previous publications of the specification](https://www.ecma-international.org/publications/standards/Ecma-262-arch.htm). Unfortunately, there is no way to identify which version of ECMAScript (specifically, 3 or 5.1) without looking up the section number and deciding whether it covers the observable in the test.
update CONTRIBUTING.md to reflect current practices
2017-10-06 22:09:39 +02:00
Make Atomics tests more resilient against intermittent failures (#1617) Also fixes multiple issues and bugs in various Atomics tests. CONTRIBUTING.md and INTERPRETING.md: - Add missing description for `CanBlockIsFalse` flag and also introduce `CanBlockIsTrue` flag which is needed for some tests (see below). harness/atomicsHelper.js: - Updated `$262.agent.waitUntil` to clarify it can also be called with BigInt64Array objects. - Added `$262.agent.timeouts` constants to unify the various, inconsistently used timeouts in the Atomics tests. Each timeout constant has a brief description and a usage example to clarify when it should or should not be used. The default values were tested in SpiderMonkey (locally and CI) under various system load levels and should hopefully also be valid for other engines. - Added `$262.agent.tryYield` to yield control from the main thread. The default implementation simply calls `$262.agent.sleep`, but test262 hosts could theoretically provide a different implementation. - Added `$262.agent.trySleep` to replace direct calls to `$262.agent.sleep` from the main thread. Motivation for this function: Some test262 hosts (like browsers) may be able to pause and sleep on the main thread, so they could provide their implementation for `trySleep` which performs a busy-wait or something do nothing. harness/testAtomics.js: - Replace `let`, `arrow-functions`, and `for-of` loops to avoid creating extremely long `features` lists in tests using this helper. Removed `Map`, `WeakMap`, `Set`, and `WeakSet` (these weren't even listed in features.yaml) for the same reason and added the missing `Symbol` entry for this file to features.yaml. - Updated all files including "harness/testAtomics.js" to match the new feature requirements. test/built-ins/Atomics/store/good-views.js and test/built-ins/Atomics/store/bigint/good-views.js: - Replace `arrow-functions` and `for-of` loops with ES5 alternatives. test/built-ins/Atomics/wait/bigint/*.js and test/built-ins/Atomics/wake/bigint/*.js - Some tests were using `BigInt64Array.BYTES_PER_ELEMENT * 8` for the SharedArrayBuffer length, but their non-BigInt counterparts are using `Int32Array.BYTES_PER_ELEMENT * 4`. For consistency and to make it easier to compare the BigInt against the non-BigInt versions, I've changed it to `BigInt64Array.BYTES_PER_ELEMENT * 4`. - Also aligned formatting and statement placement when they differed between the non-BigInt and the BigInt version of a test file. (I've diffed some of the non-BigInt and BigInt files against each other and different formatting was a nuisance.) Test files using `$262.agent.monotonicNow()`: - Moved `$262.agent.report()` calls outside of the block of code measured `$262.agent.monotonicNow()` to avoid measuring how long it takes to execute `$262.agent.report()`. - Without this change some tests failed intermittently in certain test configurations in SpiderMonkey. For example with the flags `--ion-eager -- ion-offthread-compile=off` which forces early Ion compilation on the main thread. The `$262.agent.report()` implementation in the SpiderMonkey test262 host embedding uses a for-loop which was forcefully Ion compiled under these settings. And because Ion compilation can take some time, the test case ran longer than `$262.agent.MAX_TIME_EPSILON` which lead to intermittent failures. Test files using `CanBlockIsFalse` / `CanBlockIsTrue`: - Some of these tests actually expected that the main thread can wait and [[CanBlock]] is `true` for the agent record executing the test. Therefore I've added a new `CanBlockIsTrue` flag and replaced the flags where needed. test/built-ins/Atomics/wait/**/*.js and test/built-ins/Atomics/wake/**/*.js: - Use an atomic counter `RUNNING` in more tests to have better control when a worker agent was actually started. - Replace the various `$262.agent.sleep(/* Sleep X ms to ensure worker actually sleeps */)` calls with the new `$262.agent.tryYield()` function. This `X` was sometimes as low as 10 milliseconds, which is definitely too short for CI systems under heavy load (observed by intermittent CI failures for SpiderMonkey) and sometimes as high as 500 milliseconds, which is probably much longer than needed even when the system is under heavy load. - Removed duplicate strings in assertion messages, presumably from copy-pasting the messages between different files. - Removed extra empty lines at the end of multiple files. test/built-ins/Atomics/wake/bad-range.js, test/built-ins/Atomics/wake/bigint/bad-range.js, and test/built-ins/Atomics/wait/bigint/non-bigint64-typedarray-throws.js: - Removed unnecessary `features` and `includes` from this file. test/built-ins/Atomics/wait/waiterlist-order-of-operations-is-fifo.js and test/built-ins/Atomics/wait/bigint/waiterlist-order-of-operations-is-fifo.js: - The test was actually broken and didn't test what it said it does. This probably explains #1530. - The test wants to ensure the waiterlist is implemented as a FIFO structure. This requires that the waiting agents all wait on the same index position, because the waiterlists are defined by each index. But if the agents wait on different indices, each agent is inserted into a different FIFO structure and therefore we can't observe any FIFO ordering between the agents when they're woken up. - All this requires a bit of synchronization between the main agent and the waiting agent, I hope the added comments help to review these changes. test/built-ins/Atomics/wait/good-views.js: - The agent sends multiple reports to the main agent, but only the first one was read. - This error was introduced during previous refactorings. I've changed it to back to use a while-loop as used in the first version of this file. test/built-ins/Atomics/wait/no-spurious-wakeup-no-operation.js: - Only the BigInt version of this test was present, copied it so we also get code coverage for the non-BigInt case. test/built-ins/Atomics/wait/waiterlist-block-indexedposition-wake.js: - Added extra while loops to avoid intermittent failures when the agent worker haven't started to wait. - This should help to avoid some of the intermittent failures we saw for SpiderMonkey. test/built-ins/Atomics/wake/wake-all-on-loc.js and test/built-ins/Atomics/wake/bigint/wake-all-on-loc.js: - This test was also no longer after previous refactoring sessions. - The "B" agent only waited for 10 milliseconds, which made it likely that it already timed out before the main agent was able to call `Atomics.wake`, which in turn rendered the test useless, because the test case wants to ensure that `Atomics.wake` cannot wake "B". But if "B" was already timed out, it can trivially not be woken by `Atomics.wake`. - Added some safety measure to catch the case when "B" timed out before `Atomics.wake` was called and made it a test error if that happens. test/built-ins/Atomics/wake/count-defaults-to-infinity-missing.js, test/built-ins/Atomics/wake/count-defaults-to-infinity-undefined.js, and test/built-ins/Atomics/wake/undefined-index-defaults-to-zero.js: - Changed the `$262.agent.start()` calls to use a for-loop to avoid code duplication. - (Forgot to undo the code formatting around `assert.sameValue`, still need to change it back.) - Also more while-loops around `Atomics.wake`. - These changes should fix #1529 and #1566. test/built-ins/Atomics/wake/wake-all.js: - Removed "B" worker agent. - Without this removal the test case would be exactly equal to test/built-ins/Atomics/wake/wake-all-on-loc.js. test/built-ins/Atomics/wake/wake-in-order-one-time.js: - Add for-loops to avoid code duplication and make the test more readable. - Make the `Atomics.wake` assertion messages unique by adding the current loop counter. - Add `$262.agent.tryYield()` to give the worker agents enough time to actually start waiting. test/built-ins/Atomics/wake/wake-in-order.js: - Removed the outer loop `attempt < 10` because it uses `$262.agent` in a way currently not required to work. And which actually also doesn't work in SpiderMonkey's implementation of `$262.agent`. - According to INTERPRETING.md `$262.agent.broadcast()` broadcasts its message to all agents ever started by `$262.agent.start()` and then blocks until all agents received the broadcast. It is not required that the agents started by the first `$262.agent.broadcast()` call will all be disabled/destroyed/whatever when the second `$262.agent.broadcast()` call occurs, which then means the second `$262.agent.broadcast()` call still tries to reach the agents started in the first loop iteration, but these may no longer accept broadcasts and therefore won't acknowledge they've received the broadcast. Which then means the second `$262.agent.broadcast()` call waits forever.
2018-07-03 18:34:54 +02:00
Read the [Test262 Technical Rationale Report](https://github.com/tc39/test262/wiki/Test262-Technical-Rationale-Report,-October-2017#specification-reference-ids) for reasoning behind deprecation.
update CONTRIBUTING.md to reflect current practices
2017-10-06 22:09:39 +02:00
#### es6id
Improve CONTRIBUTING.md formatting
2019-03-29 19:20:16 +01:00
`es6id: [es6-test-id]`
update CONTRIBUTING.md to reflect current practices
2017-10-06 22:09:39 +02:00
Improve CONTRIBUTING.md formatting
2019-03-29 19:20:16 +01:00
**_DEPRECATED._**
update CONTRIBUTING.md to reflect current practices
2017-10-06 22:09:39 +02:00
Update CONTRIBUTING.md vocabulary for consistency with YAML "tag" has a specific and different meaning; the proper term for this page is "key" cf. https://yaml.org/spec/1.2/spec.html#id2761292
2019-03-29 19:13:51 +01:00
This key identifies the section number from the portion of the ES6 standard that is tested by this test _at the time the test was written_. The es6ids might not correspond to the correction section numbers in the ES6 (or later) specification because routine edits to the specification will change section numbering. For this reason, only the esid is required for new tests.
update CONTRIBUTING.md to reflect current practices
2017-10-06 22:09:39 +02:00
Make Atomics tests more resilient against intermittent failures (#1617) Also fixes multiple issues and bugs in various Atomics tests. CONTRIBUTING.md and INTERPRETING.md: - Add missing description for `CanBlockIsFalse` flag and also introduce `CanBlockIsTrue` flag which is needed for some tests (see below). harness/atomicsHelper.js: - Updated `$262.agent.waitUntil` to clarify it can also be called with BigInt64Array objects. - Added `$262.agent.timeouts` constants to unify the various, inconsistently used timeouts in the Atomics tests. Each timeout constant has a brief description and a usage example to clarify when it should or should not be used. The default values were tested in SpiderMonkey (locally and CI) under various system load levels and should hopefully also be valid for other engines. - Added `$262.agent.tryYield` to yield control from the main thread. The default implementation simply calls `$262.agent.sleep`, but test262 hosts could theoretically provide a different implementation. - Added `$262.agent.trySleep` to replace direct calls to `$262.agent.sleep` from the main thread. Motivation for this function: Some test262 hosts (like browsers) may be able to pause and sleep on the main thread, so they could provide their implementation for `trySleep` which performs a busy-wait or something do nothing. harness/testAtomics.js: - Replace `let`, `arrow-functions`, and `for-of` loops to avoid creating extremely long `features` lists in tests using this helper. Removed `Map`, `WeakMap`, `Set`, and `WeakSet` (these weren't even listed in features.yaml) for the same reason and added the missing `Symbol` entry for this file to features.yaml. - Updated all files including "harness/testAtomics.js" to match the new feature requirements. test/built-ins/Atomics/store/good-views.js and test/built-ins/Atomics/store/bigint/good-views.js: - Replace `arrow-functions` and `for-of` loops with ES5 alternatives. test/built-ins/Atomics/wait/bigint/*.js and test/built-ins/Atomics/wake/bigint/*.js - Some tests were using `BigInt64Array.BYTES_PER_ELEMENT * 8` for the SharedArrayBuffer length, but their non-BigInt counterparts are using `Int32Array.BYTES_PER_ELEMENT * 4`. For consistency and to make it easier to compare the BigInt against the non-BigInt versions, I've changed it to `BigInt64Array.BYTES_PER_ELEMENT * 4`. - Also aligned formatting and statement placement when they differed between the non-BigInt and the BigInt version of a test file. (I've diffed some of the non-BigInt and BigInt files against each other and different formatting was a nuisance.) Test files using `$262.agent.monotonicNow()`: - Moved `$262.agent.report()` calls outside of the block of code measured `$262.agent.monotonicNow()` to avoid measuring how long it takes to execute `$262.agent.report()`. - Without this change some tests failed intermittently in certain test configurations in SpiderMonkey. For example with the flags `--ion-eager -- ion-offthread-compile=off` which forces early Ion compilation on the main thread. The `$262.agent.report()` implementation in the SpiderMonkey test262 host embedding uses a for-loop which was forcefully Ion compiled under these settings. And because Ion compilation can take some time, the test case ran longer than `$262.agent.MAX_TIME_EPSILON` which lead to intermittent failures. Test files using `CanBlockIsFalse` / `CanBlockIsTrue`: - Some of these tests actually expected that the main thread can wait and [[CanBlock]] is `true` for the agent record executing the test. Therefore I've added a new `CanBlockIsTrue` flag and replaced the flags where needed. test/built-ins/Atomics/wait/**/*.js and test/built-ins/Atomics/wake/**/*.js: - Use an atomic counter `RUNNING` in more tests to have better control when a worker agent was actually started. - Replace the various `$262.agent.sleep(/* Sleep X ms to ensure worker actually sleeps */)` calls with the new `$262.agent.tryYield()` function. This `X` was sometimes as low as 10 milliseconds, which is definitely too short for CI systems under heavy load (observed by intermittent CI failures for SpiderMonkey) and sometimes as high as 500 milliseconds, which is probably much longer than needed even when the system is under heavy load. - Removed duplicate strings in assertion messages, presumably from copy-pasting the messages between different files. - Removed extra empty lines at the end of multiple files. test/built-ins/Atomics/wake/bad-range.js, test/built-ins/Atomics/wake/bigint/bad-range.js, and test/built-ins/Atomics/wait/bigint/non-bigint64-typedarray-throws.js: - Removed unnecessary `features` and `includes` from this file. test/built-ins/Atomics/wait/waiterlist-order-of-operations-is-fifo.js and test/built-ins/Atomics/wait/bigint/waiterlist-order-of-operations-is-fifo.js: - The test was actually broken and didn't test what it said it does. This probably explains #1530. - The test wants to ensure the waiterlist is implemented as a FIFO structure. This requires that the waiting agents all wait on the same index position, because the waiterlists are defined by each index. But if the agents wait on different indices, each agent is inserted into a different FIFO structure and therefore we can't observe any FIFO ordering between the agents when they're woken up. - All this requires a bit of synchronization between the main agent and the waiting agent, I hope the added comments help to review these changes. test/built-ins/Atomics/wait/good-views.js: - The agent sends multiple reports to the main agent, but only the first one was read. - This error was introduced during previous refactorings. I've changed it to back to use a while-loop as used in the first version of this file. test/built-ins/Atomics/wait/no-spurious-wakeup-no-operation.js: - Only the BigInt version of this test was present, copied it so we also get code coverage for the non-BigInt case. test/built-ins/Atomics/wait/waiterlist-block-indexedposition-wake.js: - Added extra while loops to avoid intermittent failures when the agent worker haven't started to wait. - This should help to avoid some of the intermittent failures we saw for SpiderMonkey. test/built-ins/Atomics/wake/wake-all-on-loc.js and test/built-ins/Atomics/wake/bigint/wake-all-on-loc.js: - This test was also no longer after previous refactoring sessions. - The "B" agent only waited for 10 milliseconds, which made it likely that it already timed out before the main agent was able to call `Atomics.wake`, which in turn rendered the test useless, because the test case wants to ensure that `Atomics.wake` cannot wake "B". But if "B" was already timed out, it can trivially not be woken by `Atomics.wake`. - Added some safety measure to catch the case when "B" timed out before `Atomics.wake` was called and made it a test error if that happens. test/built-ins/Atomics/wake/count-defaults-to-infinity-missing.js, test/built-ins/Atomics/wake/count-defaults-to-infinity-undefined.js, and test/built-ins/Atomics/wake/undefined-index-defaults-to-zero.js: - Changed the `$262.agent.start()` calls to use a for-loop to avoid code duplication. - (Forgot to undo the code formatting around `assert.sameValue`, still need to change it back.) - Also more while-loops around `Atomics.wake`. - These changes should fix #1529 and #1566. test/built-ins/Atomics/wake/wake-all.js: - Removed "B" worker agent. - Without this removal the test case would be exactly equal to test/built-ins/Atomics/wake/wake-all-on-loc.js. test/built-ins/Atomics/wake/wake-in-order-one-time.js: - Add for-loops to avoid code duplication and make the test more readable. - Make the `Atomics.wake` assertion messages unique by adding the current loop counter. - Add `$262.agent.tryYield()` to give the worker agents enough time to actually start waiting. test/built-ins/Atomics/wake/wake-in-order.js: - Removed the outer loop `attempt < 10` because it uses `$262.agent` in a way currently not required to work. And which actually also doesn't work in SpiderMonkey's implementation of `$262.agent`. - According to INTERPRETING.md `$262.agent.broadcast()` broadcasts its message to all agents ever started by `$262.agent.start()` and then blocks until all agents received the broadcast. It is not required that the agents started by the first `$262.agent.broadcast()` call will all be disabled/destroyed/whatever when the second `$262.agent.broadcast()` call occurs, which then means the second `$262.agent.broadcast()` call still tries to reach the agents started in the first loop iteration, but these may no longer accept broadcasts and therefore won't acknowledge they've received the broadcast. Which then means the second `$262.agent.broadcast()` call waits forever.
2018-07-03 18:34:54 +02:00
Read the [Test262 Technical Rationale Report](https://github.com/tc39/test262/wiki/Test262-Technical-Rationale-Report,-October-2017#specification-reference-ids) for reasoning behind deprecation.
update CONTRIBUTING.md to reflect current practices
2017-10-06 22:09:39 +02:00
document `timeout` and clean up CONTRIBUTING.md - document `timeout` tag - reorder tags in frontmatter doc - minor cleanups - minor fixes - add style note - reformat flags - remove discussion of obsolete $INCLUDE - incorporate line notes from @domenic - integrate additional comments - add links back, move arg notes down - Raise outline level by one README.md - link to CONTRIBUTING
2014-08-22 17:47:18 +02:00
## Test Environment
CONTRIBUTING: more notes on test writing Add section on test environment Add section on custom helpers describe YAML frontmatter Fix minor formatting errors document $INCLUDE as obsolete Change documentation of negative error Move test environment and custom helpers down indent copyright and frontmatter sections better description of the async calls between a promise and the functions in its `.then` Correct Early Error example: don't throw a string CONSOLE-RUNNER: split runner doc into new file add troubleshooting section
2014-07-30 00:27:20 +02:00
Improve documentation of harness files & raw flag Prior to this commit, the term "harness" was used to refer both to the like-named directory in the repository and the external projects which consume the tests. Reserve the term "harness" to refer solely to the files maintained within this project's `harness/` directory, and use the term "runner" to refer to the external projects. Prior to this commit, the term "helper" was not defined and therefore introduced ambiguity in text that described harness files. Replace the term with more explicit wording in terms of the harness files. Because test runners have some leeway in how they include harness files, the requirement that "[t]he test source code must not be modified in any way" does not necessarily preclude the use of harness files. Update the instructions for interpreting the `raw` frontmatter flag to explicitly disallow harness files.
2021-11-19 21:08:18 +01:00
Each test case is run in a fresh JavaScript environment; in a browser, this will be a new &lt;iframe&gt;; for a console runner, this will be a new process.
Before the test is executed (and unless the test uses the `raw` frontmatter flag), the test runner will evaluate a number of files in the `harness/` directory. At a minimum, this procedure will define the following functions:
CONTRIBUTING: more notes on test writing Add section on test environment Add section on custom helpers describe YAML frontmatter Fix minor formatting errors document $INCLUDE as obsolete Change documentation of negative error Move test environment and custom helpers down indent copyright and frontmatter sections better description of the async calls between a promise and the functions in its `.then` Correct Early Error example: don't throw a string CONSOLE-RUNNER: split runner doc into new file add troubleshooting section
2014-07-30 00:27:20 +02:00
Function | Purpose
---------|--------
Rename to $DONOTEVALUATE per @leobalter's suggestion
2018-10-11 16:59:24 +02:00
`Test262Error(message)` | constructor for an error object that indicates a test failure
`$DONE(arg)` | see [Writing Asynchronous Tests](#writing-asynchronous-tests), below
Explain the purpose of `assert` helper message parameters The convention in some testing libraries is an explanation of what went wrong; in this one it is an explanation of what should happen.
2021-09-11 17:44:37 +02:00
`assert(value, message)` | throw a new Test262Error instance if the specified value is not strictly equal to the JavaScript `true` value; accepts an optional string message explaining the scenario and what should have happened
`assert.sameValue(actual, expected, message)` | throw a new Test262Error instance if the first two arguments are not [the same value](https://tc39.github.io/ecma262/#sec-samevalue); accepts an optional string message explaining the scenario and what should have happened
`assert.notSameValue(actual, unexpected, message)` | throw a new Test262Error instance if the first two arguments are [the same value](https://tc39.github.io/ecma262/#sec-samevalue); accepts an optional string message explaining the scenario and what should have happened
`assert.throws(expectedErrorConstructor, fn, message)` | throw a new Test262Error instance if the provided function does not throw an error or if the constructor of the value thrown does not match the provided constructor; accepts an optional string message explaining the scenario and what should have happened
Interpreting: explain $DONOTEVALUATE. (#2735) No version change is necessary, this is purely informative.
2020-08-20 23:11:10 +02:00
`$DONOTEVALUATE()` | throw an exception if the code gets evaluated. This may only be used in [negative test cases for parsing errors](#handling-errors-and-negative-test-cases).
Document raw negative test reqs
2019-02-05 21:38:12 +01:00
`throw "Test262: This statement should not be evaluated.";` | throw an exception if the code gets evaluated. Use this if the test file has the `raw` flag and it's a negative test case for parsing error.
Update contribution guidelines - Remove trailing white space - Streamline documentation of test tags - Do not reference obsolete tags - Document `features` frontmatter tag - Document `es6id` frontmatter tag - Omit unnecessary detail about test262 website generation. This is not directly useful to potential test contributors. Implementation details like these can be taken for granted by that audience. - Remove documentation on YAML syntax. Details on YAML may be helpful for some new contributors, but this document should not attempt to cover the topic (especially not from the description of a specific frontmatter entry). Replace with a link to a more comprehensive source as this will be more generally useful to those who need it (and less obtrusive for those who do not). - Consolidate information on test helpers - Document `assert` helpers - Update instructions for asserting errors. Since the introduction of `assert.throws` in gh-22, the preferred means of expressing expectations regarding errors has changed. Update the CONTRIBUTING.md file to reflect the latest approach. Explain purpose of `throw NotEarlyError;` in example test. - Re-order information on file names. The inconsistency in the project's file names should not go unmentioned, but neither should it not preceed instructions for the accepted approach to namine tests. - More clearly document required frontmatter tags. Explicitly list `description` as a required frontmatter tag, implicitly identifying all other tags as optional.
2015-05-25 23:51:55 +02:00
Rename to $DONOTEVALUATE per @leobalter's suggestion
2018-10-11 16:59:24 +02:00
```javascript
CONTRIBUTING: more notes on test writing Add section on test environment Add section on custom helpers describe YAML frontmatter Fix minor formatting errors document $INCLUDE as obsolete Change documentation of negative error Move test environment and custom helpers down indent copyright and frontmatter sections better description of the async calls between a promise and the functions in its `.then` Correct Early Error example: don't throw a string CONSOLE-RUNNER: split runner doc into new file add troubleshooting section
2014-07-30 00:27:20 +02:00
/// error class
function Test262Error(message) {
Rename to $DONOTEVALUATE per @leobalter's suggestion
2018-10-11 16:59:24 +02:00
// [omitted body]
CONTRIBUTING: more notes on test writing Add section on test environment Add section on custom helpers describe YAML frontmatter Fix minor formatting errors document $INCLUDE as obsolete Change documentation of negative error Move test environment and custom helpers down indent copyright and frontmatter sections better description of the async calls between a promise and the functions in its `.then` Correct Early Error example: don't throw a string CONSOLE-RUNNER: split runner doc into new file add troubleshooting section
2014-07-30 00:27:20 +02:00
}
```
Update project structure to support non-JS files This change is in service of forthcoming tests for the "JSON modules" language proposal [1]. Verifying the semantics of that proposal requires modules whose source text is not valid ECMAScript; this change updates the guidelines for contributing and interpreting tests so that such test material can be handled consistently. Differentiating JSON files with a distinct file name suffice will assist consumers which require special handling of such files (e.g. web browsers). Change the pattern used to designate "fixture" files so that it may be applied to files used for JSON modules. Increment the project version number to alert consumers of this change in interpreting instructions. [1] https://github.com/tc39/proposal-json-modules
2021-05-29 00:53:04 +02:00
## Rules For Module `_FIXTURE` Files
INTERPRETING.md/CONTRIBUTING.md: revised rules for _FIXTURE.js files (#2135)
2019-04-24 16:59:54 +02:00
Update CONTRIBUTING.md (#3004) Fixed some grammatical issues in the document.
2021-06-10 17:34:28 +02:00
The [Module section of INTERPRETING.md](https://github.com/tc39/test262/blob/HEAD/INTERPRETING.md#modules) states that `_FIXTURE` files will not have Realm modifications applied. In practice, this means that code in `_FIXTURE` files must abide by the following rules:
INTERPRETING.md/CONTRIBUTING.md: revised rules for _FIXTURE.js files (#2135)
2019-04-24 16:59:54 +02:00
Rename usage of master to main Ref #2699
2020-07-15 20:53:19 +02:00
- **MUST NOT** refer to, or make use of any [Test262-Defined Bindings](https://github.com/tc39/test262/blob/HEAD/INTERPRETING.md#test262-defined-bindings) in any way.
- **MUST NOT** refer to, or make use of any [Host-Defined Functions](https://github.com/tc39/test262/blob/HEAD/INTERPRETING.md#host-defined-functions) in any way.
INTERPRETING.md/CONTRIBUTING.md: revised rules for _FIXTURE.js files (#2135)
2019-04-24 16:59:54 +02:00
document `timeout` and clean up CONTRIBUTING.md - document `timeout` tag - reorder tags in frontmatter doc - minor cleanups - minor fixes - add style note - reformat flags - remove discussion of obsolete $INCLUDE - incorporate line notes from @domenic - integrate additional comments - add links back, move arg notes down - Raise outline level by one README.md - link to CONTRIBUTING
2014-08-22 17:47:18 +02:00
## Handling Errors and Negative Test Cases
CONTRIBUTING: more notes on test writing Add section on test environment Add section on custom helpers describe YAML frontmatter Fix minor formatting errors document $INCLUDE as obsolete Change documentation of negative error Move test environment and custom helpers down indent copyright and frontmatter sections better description of the async calls between a promise and the functions in its `.then` Correct Early Error example: don't throw a string CONSOLE-RUNNER: split runner doc into new file add troubleshooting section
2014-07-30 00:27:20 +02:00
Update CONTRIBUTING.md per @bakkot’s feedback
2018-11-15 00:12:08 +01:00
Expectations for **parsing errors** should be declared using [the `negative` frontmatter flag](#negative). They must also include `$DONOTEVALUATE();` (in order to guarantee that implementations do not execute the code):
CONTRIBUTING: more notes on test writing Add section on test environment Add section on custom helpers describe YAML frontmatter Fix minor formatting errors document $INCLUDE as obsolete Change documentation of negative error Move test environment and custom helpers down indent copyright and frontmatter sections better description of the async calls between a promise and the functions in its `.then` Correct Early Error example: don't throw a string CONSOLE-RUNNER: split runner doc into new file add troubleshooting section
2014-07-30 00:27:20 +02:00
Added contributing.md and improved README
2014-07-04 02:13:42 +02:00
```javascript
CONTRIBUTING: more notes on test writing Add section on test environment Add section on custom helpers describe YAML frontmatter Fix minor formatting errors document $INCLUDE as obsolete Change documentation of negative error Move test environment and custom helpers down indent copyright and frontmatter sections better description of the async calls between a promise and the functions in its `.then` Correct Early Error example: don't throw a string CONSOLE-RUNNER: split runner doc into new file add troubleshooting section
2014-07-30 00:27:20 +02:00
/*---
Update documentation
2016-03-13 18:26:10 +01:00
negative:
Rename negative test "phase" for parsing Early errors may result from parsing the source text of a test file, but they may also result from parsing some other source text as referenced through the ES2015 module syntax. The latter form of early error is not necessarily detectable by ECMAScript parsers, however. Because of this, the label "early" is not sufficiently precise for all Test262 consumers to correctly interpret all tests. Update the "phase" name of "early" to "parse" for all those negative tests that describe errors resulting from parsing of the file's source text directly. A forthcoming commit will update the remaining tests to use a "phase" name that is more specific to module resolution.
2017-12-03 06:06:42 +01:00
phase: parse
Update documentation
2016-03-13 18:26:10 +01:00
type: SyntaxError
CONTRIBUTING: more notes on test writing Add section on test environment Add section on custom helpers describe YAML frontmatter Fix minor formatting errors document $INCLUDE as obsolete Change documentation of negative error Move test environment and custom helpers down indent copyright and frontmatter sections better description of the async calls between a promise and the functions in its `.then` Correct Early Error example: don't throw a string CONSOLE-RUNNER: split runner doc into new file add troubleshooting section
2014-07-30 00:27:20 +02:00
---*/
Added contributing.md and improved README
2014-07-04 02:13:42 +02:00
Rename to $DONOTEVALUATE per @leobalter's suggestion
2018-10-11 16:59:24 +02:00
$DONOTEVALUATE();
Enforce use of `throw` stmt in early error tests Previously, test consumers were encouraged to insert a `throw` statement as the first statement of tests for early errors. This recommendation made tests harder to consume, and as an optional transformation, consumers may have ignored it or simply been unaware it was made. By explicitly including such a `throw` statement, the tests become more literal, making them easier to consume and more transparent in their expectations. Document expectation for all tests for early errors to include an explicit `throw` statement. Extend linting script to verify that contributors are automatically notified of violations and to ensure that future contributions satisfy this expectation.
2017-04-29 21:33:06 +02:00
Update contribution guidelines - Remove trailing white space - Streamline documentation of test tags - Do not reference obsolete tags - Document `features` frontmatter tag - Document `es6id` frontmatter tag - Omit unnecessary detail about test262 website generation. This is not directly useful to potential test contributors. Implementation details like these can be taken for granted by that audience. - Remove documentation on YAML syntax. Details on YAML may be helpful for some new contributors, but this document should not attempt to cover the topic (especially not from the description of a specific frontmatter entry). Replace with a link to a more comprehensive source as this will be more generally useful to those who need it (and less obtrusive for those who do not). - Consolidate information on test helpers - Document `assert` helpers - Update instructions for asserting errors. Since the introduction of `assert.throws` in gh-22, the preferred means of expressing expectations regarding errors has changed. Update the CONTRIBUTING.md file to reflect the latest approach. Explain purpose of `throw NotEarlyError;` in example test. - Re-order information on file names. The inconsistency in the project's file names should not go unmentioned, but neither should it not preceed instructions for the accepted approach to namine tests. - More clearly document required frontmatter tags. Explicitly list `description` as a required frontmatter tag, implicitly identifying all other tags as optional.
2015-05-25 23:51:55 +02:00
var var = var;
Added contributing.md and improved README
2014-07-04 02:13:42 +02:00
```
Improve documentation of harness files & raw flag Prior to this commit, the term "harness" was used to refer both to the like-named directory in the repository and the external projects which consume the tests. Reserve the term "harness" to refer solely to the files maintained within this project's `harness/` directory, and use the term "runner" to refer to the external projects. Prior to this commit, the term "helper" was not defined and therefore introduced ambiguity in text that described harness files. Replace the term with more explicit wording in terms of the harness files. Because test runners have some leeway in how they include harness files, the requirement that "[t]he test source code must not be modified in any way" does not necessarily preclude the use of harness files. Update the instructions for interpreting the `raw` frontmatter flag to explicitly disallow harness files.
2021-11-19 21:08:18 +01:00
If the test case has the `raw` flag, the test runner will not load any harness file including the file which defines `$DONOTEVALUATE`. In this case, include a direct `throw "Test262: This statement should not be evaluated.";` statement:
Document raw negative test reqs
2019-02-05 21:38:12 +01:00
```javascript
/*---
flags: [raw]
negative:
phase: parse
type: SyntaxError
---*/
throw "Test262: This statement should not be evaluated.";
var var = var;
```
Update contribution guidelines - Remove trailing white space - Streamline documentation of test tags - Do not reference obsolete tags - Document `features` frontmatter tag - Document `es6id` frontmatter tag - Omit unnecessary detail about test262 website generation. This is not directly useful to potential test contributors. Implementation details like these can be taken for granted by that audience. - Remove documentation on YAML syntax. Details on YAML may be helpful for some new contributors, but this document should not attempt to cover the topic (especially not from the description of a specific frontmatter entry). Replace with a link to a more comprehensive source as this will be more generally useful to those who need it (and less obtrusive for those who do not). - Consolidate information on test helpers - Document `assert` helpers - Update instructions for asserting errors. Since the introduction of `assert.throws` in gh-22, the preferred means of expressing expectations regarding errors has changed. Update the CONTRIBUTING.md file to reflect the latest approach. Explain purpose of `throw NotEarlyError;` in example test. - Re-order information on file names. The inconsistency in the project's file names should not go unmentioned, but neither should it not preceed instructions for the accepted approach to namine tests. - More clearly document required frontmatter tags. Explicitly list `description` as a required frontmatter tag, implicitly identifying all other tags as optional.
2015-05-25 23:51:55 +02:00
Expectations for **runtime errors** should be defined using the `assert.throws` method and the appropriate JavaScript Error constructor function:
document `timeout` and clean up CONTRIBUTING.md - document `timeout` tag - reorder tags in frontmatter doc - minor cleanups - minor fixes - add style note - reformat flags - remove discussion of obsolete $INCLUDE - incorporate line notes from @domenic - integrate additional comments - add links back, move arg notes down - Raise outline level by one README.md - link to CONTRIBUTING
2014-08-22 17:47:18 +02:00
Added contributing.md and improved README
2014-07-04 02:13:42 +02:00
```javascript
Correct example in contribution guidelines As written, the example for asserting runtime errors is written with an early error. Because the error is expected to be reported prior to program execution, the `assert.throws` function cannot be used to detect it. Demonstrate the usage of the helper function with a runtime error.
2016-03-22 14:51:05 +01:00
assert.throws(TypeError, function() {
null(); // expect this statement to throw a TypeError
Update contribution guidelines - Remove trailing white space - Streamline documentation of test tags - Do not reference obsolete tags - Document `features` frontmatter tag - Document `es6id` frontmatter tag - Omit unnecessary detail about test262 website generation. This is not directly useful to potential test contributors. Implementation details like these can be taken for granted by that audience. - Remove documentation on YAML syntax. Details on YAML may be helpful for some new contributors, but this document should not attempt to cover the topic (especially not from the description of a specific frontmatter entry). Replace with a link to a more comprehensive source as this will be more generally useful to those who need it (and less obtrusive for those who do not). - Consolidate information on test helpers - Document `assert` helpers - Update instructions for asserting errors. Since the introduction of `assert.throws` in gh-22, the preferred means of expressing expectations regarding errors has changed. Update the CONTRIBUTING.md file to reflect the latest approach. Explain purpose of `throw NotEarlyError;` in example test. - Re-order information on file names. The inconsistency in the project's file names should not go unmentioned, but neither should it not preceed instructions for the accepted approach to namine tests. - More clearly document required frontmatter tags. Explicitly list `description` as a required frontmatter tag, implicitly identifying all other tags as optional.
2015-05-25 23:51:55 +02:00
});
Added contributing.md and improved README
2014-07-04 02:13:42 +02:00
```
CONTRIBUTING: document test262.py, async tests add a section on requirements for using console test runner test262.py add a section on asynchronous tests add subsection to authoring guidelines with suggestion for test names
2014-07-21 17:39:10 +02:00
Update {CONTRIBUTING,INTERPRETING}.md per @jugglinmike's feedback
2018-10-19 09:44:47 +02:00
Consumers that violate the spec by throwing exceptions for parsing errors at runtime instead of at parse time can still get value out of Test262's negative tests by conditionally overriding `$DONOTEVALUATE` to be a no-op for tests that are known to fail. This way, the consumer can still verify that they throw an exception of the expected type, even if it happens during the wrong phase, which prevents regressions and further deviations from the spec.
document `timeout` and clean up CONTRIBUTING.md - document `timeout` tag - reorder tags in frontmatter doc - minor cleanups - minor fixes - add style note - reformat flags - remove discussion of obsolete $INCLUDE - incorporate line notes from @domenic - integrate additional comments - add links back, move arg notes down - Raise outline level by one README.md - link to CONTRIBUTING
2014-08-22 17:47:18 +02:00
## Writing Asynchronous Tests
CONTRIBUTING: document test262.py, async tests add a section on requirements for using console test runner test262.py add a section on asynchronous tests add subsection to authoring guidelines with suggestion for test names
2014-07-21 17:39:10 +02:00
Call $DONE function exactly once per test Also make a note in CONTRIBUTING.md that `$DONE` shouldn't be called multiple times.
2019-07-22 11:12:24 +02:00
An asynchronous test is any test that include the `async` frontmatter flag. When executing such tests, the runner expects that the global `$DONE()` function will be called **exactly once** to signal test completion.
CONTRIBUTING: document test262.py, async tests add a section on requirements for using console test runner test262.py add a section on asynchronous tests add subsection to authoring guidelines with suggestion for test names
2014-07-21 17:39:10 +02:00
* If the argument to `$DONE` is omitted, is `undefined`, or is any other falsy value, the test is considered to have passed.
* If the argument to `$DONE` is a truthy value, the test is considered to have failed and the argument is displayed as the failure reason.
A common idiom when writing asynchronous tests is the following:
```js
var p = new Promise(function () { /* some test code */ });
p.then(function checkAssertions(arg) {
CONTRIBUTING: Link to the latest spec draft (#978)
2017-04-14 18:19:39 +02:00
if (!expected_condition) {
update CONTRIBUTING.md to reflect current practices
2017-10-06 22:09:39 +02:00
throw new Test262Error("failure message");
CONTRIBUTING: Link to the latest spec draft (#978)
2017-04-14 18:19:39 +02:00
}
CONTRIBUTING: document test262.py, async tests add a section on requirements for using console test runner test262.py add a section on asynchronous tests add subsection to authoring guidelines with suggestion for test names
2014-07-21 17:39:10 +02:00
}).then($DONE, $DONE);
```
CONTRIBUTING: more notes on test writing Add section on test environment Add section on custom helpers describe YAML frontmatter Fix minor formatting errors document $INCLUDE as obsolete Change documentation of negative error Move test environment and custom helpers down indent copyright and frontmatter sections better description of the async calls between a promise and the functions in its `.then` Correct Early Error example: don't throw a string CONSOLE-RUNNER: split runner doc into new file add troubleshooting section
2014-07-30 00:27:20 +02:00
Function `checkAssertions` implicitly returns `undefined` if the expected condition is observed. The return value of function `checkAssertions` is then used to asynchronously invoke the first function of the final `then` call, resulting in a call to `$DONE(undefined)`, which signals a passing test.
CONTRIBUTING: document test262.py, async tests add a section on requirements for using console test runner test262.py add a section on asynchronous tests add subsection to authoring guidelines with suggestion for test names
2014-07-21 17:39:10 +02:00
update CONTRIBUTING.md to reflect current practices
2017-10-06 22:09:39 +02:00
If the expected condition is not observed, function `checkAssertions` throws a `Test262Error`. This is caught by the Promise and then used to asynchronously invoke the second function in the call -- which is also `$DONE` -- resulting in a call to `$DONE(error_object)`, which signals a failing test.
CONTRIBUTING: document test262.py, async tests add a section on requirements for using console test runner test262.py add a section on asynchronous tests add subsection to authoring guidelines with suggestion for test names
2014-07-21 17:39:10 +02:00
document `timeout` and clean up CONTRIBUTING.md - document `timeout` tag - reorder tags in frontmatter doc - minor cleanups - minor fixes - add style note - reformat flags - remove discussion of obsolete $INCLUDE - incorporate line notes from @domenic - integrate additional comments - add links back, move arg notes down - Raise outline level by one README.md - link to CONTRIBUTING
2014-08-22 17:47:18 +02:00
### Checking Exception Type and Message in Asynchronous Tests
CONTRIBUTING: document test262.py, async tests add a section on requirements for using console test runner test262.py add a section on asynchronous tests add subsection to authoring guidelines with suggestion for test names
2014-07-21 17:39:10 +02:00
This idiom can be extended to check for specific exception types or messages:
```js
p.then(function () {
CONTRIBUTING: Link to the latest spec draft (#978)
2017-04-14 18:19:39 +02:00
// some code that is expected to throw a TypeError
CONTRIBUTING: document test262.py, async tests add a section on requirements for using console test runner test262.py add a section on asynchronous tests add subsection to authoring guidelines with suggestion for test names
2014-07-21 17:39:10 +02:00
CONTRIBUTING: Link to the latest spec draft (#978)
2017-04-14 18:19:39 +02:00
return "Expected exception to be thrown";
CONTRIBUTING: document test262.py, async tests add a section on requirements for using console test runner test262.py add a section on asynchronous tests add subsection to authoring guidelines with suggestion for test names
2014-07-21 17:39:10 +02:00
}).then($DONE, function (e) {
CONTRIBUTING: Link to the latest spec draft (#978)
2017-04-14 18:19:39 +02:00
if (!(e instanceof TypeError)) {
update CONTRIBUTING.md to reflect current practices
2017-10-06 22:09:39 +02:00
throw new Test262Error("Expected TypeError but got " + e);
CONTRIBUTING: Link to the latest spec draft (#978)
2017-04-14 18:19:39 +02:00
}
CONTRIBUTING: document test262.py, async tests add a section on requirements for using console test runner test262.py add a section on asynchronous tests add subsection to authoring guidelines with suggestion for test names
2014-07-21 17:39:10 +02:00
CONTRIBUTING: Link to the latest spec draft (#978)
2017-04-14 18:19:39 +02:00
if (!/expected message/.test(e.message)) {
update CONTRIBUTING.md to reflect current practices
2017-10-06 22:09:39 +02:00
throw new Test262Error("Expected message to contain 'expected message' but found " + e.message);
CONTRIBUTING: Link to the latest spec draft (#978)
2017-04-14 18:19:39 +02:00
}
CONTRIBUTING: document test262.py, async tests add a section on requirements for using console test runner test262.py add a section on asynchronous tests add subsection to authoring guidelines with suggestion for test names
2014-07-21 17:39:10 +02:00
}).then($DONE, $DONE);
```
Update contribution guidelines - Remove trailing white space - Streamline documentation of test tags - Do not reference obsolete tags - Document `features` frontmatter tag - Document `es6id` frontmatter tag - Omit unnecessary detail about test262 website generation. This is not directly useful to potential test contributors. Implementation details like these can be taken for granted by that audience. - Remove documentation on YAML syntax. Details on YAML may be helpful for some new contributors, but this document should not attempt to cover the topic (especially not from the description of a specific frontmatter entry). Replace with a link to a more comprehensive source as this will be more generally useful to those who need it (and less obtrusive for those who do not). - Consolidate information on test helpers - Document `assert` helpers - Update instructions for asserting errors. Since the introduction of `assert.throws` in gh-22, the preferred means of expressing expectations regarding errors has changed. Update the CONTRIBUTING.md file to reflect the latest approach. Explain purpose of `throw NotEarlyError;` in example test. - Re-order information on file names. The inconsistency in the project's file names should not go unmentioned, but neither should it not preceed instructions for the accepted approach to namine tests. - More clearly document required frontmatter tags. Explicitly list `description` as a required frontmatter tag, implicitly identifying all other tags as optional.
2015-05-25 23:51:55 +02:00
As above, exceptions that are thrown from a `then` clause are passed to a later `$DONE` function and reported asynchronously.
Add initial set of generated test sources
2016-03-16 18:58:29 +01:00
Improve documentation for Python usage (#1703) Update the recommended invocation of `pip` to use the `python` command, avoiding discrepancies on systems where the `pip` and `python` commands are not provided by the same Python installation. In the interest of consistency, also update the configuration for the project's continuous integration service. Recommend additional tooling for users with more advanced needs.
2018-09-04 16:56:51 +02:00
## A Note on Python-based tools
This project's internal tooling is built with Python. Contributors seeking to interact with these tools should begin by installing [Python version 2](https://www.python.org/) and [PIP](https://pypi.org/project/pip/). This guide includes instructions for installing packages using PIP directly, but contributors are welcomed to use utilities such as [virtualenv](https://pypi.org/project/virtualenv/), [pyenv](https://github.com/pyenv/pyenv), or [pipenv](https://pypi.org/project/pipenv/) should they have more advanced package management needs.
Introduce automated validation for test format (#994) This script is intended to identify common test file formatting errors prior to their acceptance into the project. It is designed to support future extensions for additional validation rules.
2017-05-01 18:04:05 +02:00
## Linting
Document steps for satisfying tool dependencies
2017-12-03 05:09:24 +01:00
Some of the expectations documented here are enforced via a "linting" script. This script is used to validate patches automatically at submission time, but it may also be invoked locally. To do so, first install the required Python packages via the following command:
Improve documentation for Python usage (#1703) Update the recommended invocation of `pip` to use the `python` command, avoiding discrepancies on systems where the `pip` and `python` commands are not provided by the same Python installation. In the interest of consistency, also update the configuration for the project's continuous integration service. Recommend additional tooling for users with more advanced needs.
2018-09-04 16:56:51 +02:00
python -m pip install --requirement tools/lint/requirements.txt
Document steps for satisfying tool dependencies
2017-12-03 05:09:24 +01:00
Then invoke the following command:
Introduce automated validation for test format (#994) This script is intended to identify common test file formatting errors prior to their acceptance into the project. It is designed to support future extensions for additional validation rules.
2017-05-01 18:04:05 +02:00
rename whitelist to exceptions in the linter tool (#2004)
2018-12-18 18:39:57 +01:00
python tools/lint/lint.py --exceptions lint.exceptions [paths to tests]
Introduce automated validation for test format (#994) This script is intended to identify common test file formatting errors prior to their acceptance into the project. It is designed to support future extensions for additional validation rules.
2017-05-01 18:04:05 +02:00
...where `[paths to tests]` is a list of one or more paths to test files or directories containing test files.
rename whitelist to exceptions in the linter tool (#2004)
2018-12-18 18:39:57 +01:00
In some cases, it may be necessary for a test to intentionally violate the rules enforced by the linting tool. Such violations can be allowed by including the path of the test(s) in the `lint.exceptions` file. Each path must appear on a dedicated line in that file, and a space-separated list of rules to ignore must follow each path. Lines beginning with the pound sign (`#`) will be ignored. For example:
Introduce automated validation for test format (#994) This script is intended to identify common test file formatting errors prior to their acceptance into the project. It is designed to support future extensions for additional validation rules.
2017-05-01 18:04:05 +02:00
# This file documents authorship information and is not itself a test
test/built-ins/Simd/AUTHORS FRONTMATTER LICENSE
Add initial set of generated test sources
2016-03-16 18:58:29 +01:00
## Procedurally-generated tests
Improve "Procedurally-generated tests" documentation
2021-07-28 01:43:23 +02:00
Some language features are expressed through a number of distinct syntactic forms. To ensure equivalent coverage across all of them, Test262 includes "test templates" and "test cases" in sub-directories under the `src/` directory. Test cases and test templates have the same sections as so-called "static" (i.e. non-generated) tests, although the expected metadata keys differ between test cases and test templates and the Body of each is subject to special interpretation as described below.
Add initial set of generated test sources
2016-03-16 18:58:29 +01:00
Improve "Procedurally-generated tests" documentation
2021-07-28 01:43:23 +02:00
### test templates (`*.template`)
The Body of a test template matches that of a non-generated test in which an arbitrary number of "placeholder" comments have replaced input elements. A placeholder comment is one whose content consists of an identifier wrapped in curly braces with optional interior whitespace, e.g. `/*{name}*/` or `/*{ name }*/`, which is replaced in each generated file with data from a test case.
Expected [Frontmatter](#frontmatter) keys:
Key | Description
------|-------------
Fix path example in generated test instructions We should not have an absolute path (starting with a slash) in this field, because that will mess up the test generation program.
2022-02-12 01:58:28 +01:00
`path` | location within the published test hierarchy to output files created from this template, each with a path formed by appending the name of the corresponding test case file. For example, a template with `path` "language/template1-" used by test case file case1.js will generate a test file for that case at "language/template1-case1.js".
Improve "Procedurally-generated tests" documentation
2021-07-28 01:43:23 +02:00
`name` | human-readable name of the syntactic form described by this template. Each generated test will have a `description` that is the result of appending the test template `name`, in parentheses, to the test case `desc` field.
`esid` | see the general definition of the [`esid` frontmatter key](#esid).
`info` | see the general definition of the [`info` frontmatter key](#info). Each generated test will have `info` that is the concatenation of the test template `info` field and the test case `info` field.
`features` | see the general definition of the [`features` frontmatter key](#features). Each generated test will have a feature list that is the union of the test template `features` and the test case `features`.
any other valid frontmatter field | see the general definitions.
Add initial set of generated test sources
2016-03-16 18:58:29 +01:00
Fix generate test explination
2017-10-17 18:57:44 +02:00
### test cases (`*.case`)
Improve "Procedurally-generated tests" documentation
2021-07-28 01:43:23 +02:00
The Body of a test case consists of a sequence of "value" comments, each followed by a line break and then content for use in a template. A value comment is one whose content consists of a whitespace-separated sequence of an initial ASCII dash, an identifier, and any number of optional tags, e.g. `//- name` or `//- character codepoints`. Line(s) after the value comment are trimmed of leading whitespace and then interpreted according to its tags and used as the replacement for any template placeholder comment sharing its identifier. The special identifiers "setup" and "teardown" always become the start and end (respectively) of the template Body.
Currently-defined value comment tags:
* `codepoints` indicates that the following line(s) of content are to be interpreted as a sequence of hexadecimal-encoded Unicode code points separated by whitespace. This is useful for definining content that includes whitespace or other invisible characters, e.g. `0000` represents a NULL character and `2028 2029` represents a line separator followed by a paragraph separator.
Expected [Frontmatter](#frontmatter) keys:
Key | Description
Fix generate test explination
2017-10-17 18:57:44 +02:00
------|-------------
Enable configuration of templates for generated tests. Fixes gh-1049
2020-09-24 22:51:49 +02:00
`template` | a template file, directory or glob expression.
`templates` | a list of template file, directory or glob expressions.
Improve "Procedurally-generated tests" documentation
2021-07-28 01:43:23 +02:00
`desc` | see the general definition of the [`description` frontmatter key](#description). Each generated test will have a `description` that is the result of appending the test template `name`, in parentheses, to the test case `desc` field.
`info` | see the general definition of the [`info` frontmatter key](#info). Each generated test will have `info` that is the concatenation of the test template `info` field and the test case `info` field.
`features` | see the general definition of the [`features` frontmatter key](#features). Each generated test will have a feature list that is the union of the test template `features` and the test case `features`.
Add initial set of generated test sources
2016-03-16 18:58:29 +01:00
Improve "Procedurally-generated tests" documentation
2021-07-28 01:43:23 +02:00
### Generated test management
Document steps for satisfying tool dependencies
2017-12-03 05:09:24 +01:00
Generated files are managed using the `make.py` Python script located in the root of this repository. To use it, first install the required Python packages via the following command:
Improve documentation for Python usage (#1703) Update the recommended invocation of `pip` to use the `python` command, avoiding discrepancies on systems where the `pip` and `python` commands are not provided by the same Python installation. In the interest of consistency, also update the configuration for the project's continuous integration service. Recommend additional tooling for users with more advanced needs.
2018-09-04 16:56:51 +02:00
python -m pip install --requirement tools/generation/requirements.txt
Add initial set of generated test sources
2016-03-16 18:58:29 +01:00
Document steps for satisfying tool dependencies
2017-12-03 05:09:24 +01:00
And then issue the following command to create files:
Add initial set of generated test sources
2016-03-16 18:58:29 +01:00
Fix docs for using make.py (#2181)
2019-05-29 20:51:20 +02:00
./make.py
Add initial set of generated test sources
2016-03-16 18:58:29 +01:00
To remove all generated files:
Fix docs for using make.py (#2181)
2019-05-29 20:51:20 +02:00
./make.py clean
Add initial set of generated test sources
2016-03-16 18:58:29 +01:00
The executable located at `tools/generation/generator.py` offers additional control over the generation procedure.
./tools/generation/generator.py --help
Automate deployment (#546) Introduce configuration to allow build servers provided by the Travis CI service to execute the test generation tool and commit the resultant files to the canonical upstream repository. Enabling this workflow required additional administrative work: 1. Create an account with TravisCI 2. Install the `travis` command-line utility 3. Create a "deploy key" and an encrypted version using the command `./make.py github_deploy_key_enc` 4. Register the deploy key with the project's GitHub account 5. Check the encrypted deploy key to the repository 6. Configure the TravisCI service to automatically build this project
2017-04-10 21:31:19 +02:00
Tests expressed with this convention are built automatically following the source files' acceptance into the project. Patches should **not** include assets built from these sources.
Contributing: add list of links to implementation bug trackers. Fixes gh-2121
2020-09-14 18:13:41 +02:00
## Reporting Bugs to Implementers
- [ChakraCore](https://github.com/microsoft/ChakraCore/issues/new)
- [engine262](https://github.com/engine262/engine262/issues/new)
- [Hermes](https://github.com/facebook/hermes/issues/new?labels%5B%5D=need+triage&labels%5B%5D=bug&template=01_bug_report.md&title=)
- [JavaScriptCore](https://bugs.webkit.org/enter_bug.cgi?product=WebKit&component=JavaScriptCore)
- [Moddable XS](https://github.com/Moddable-OpenSource/moddable/issues/new?assignees=&labels=&template=bug_report.md&title=)
- [QuickJS](https://github.com/bellard/quickjs/issues/new)
- [SpiderMonkey](https://bugzilla.mozilla.org/enter_bug.cgi?product=Core&component=JavaScript%20Engine)
- [V8](https://bugs.chromium.org/p/v8/issues/entry)