tyler.durden
/
test262
mirror of https://github.com/tc39/test262.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
test262/CONTRIBUTING.md

331 lines
17 KiB
Markdown
Raw Normal View History

document `timeout` and clean up CONTRIBUTING.md - document `timeout` tag - reorder tags in frontmatter doc - minor cleanups - minor fixes - add style note - reformat flags - remove discussion of obsolete $INCLUDE - incorporate line notes from @domenic - integrate additional comments - add links back, move arg notes down - Raise outline level by one README.md - link to CONTRIBUTING
2014-08-22 17:47:18 +02:00
# Test262 Authoring Guidelines
CONTRIBUTING: document test262.py, async tests add a section on requirements for using console test runner test262.py add a section on asynchronous tests add subsection to authoring guidelines with suggestion for test names
2014-07-21 17:39:10 +02:00
CONTRIBUTING.md: Adds 'Source Material' section with 'needs tests' pr list (#1054)
2017-06-13 00:29:58 +02:00
## Source Material
- https://tc39.github.io/ecma262/
- https://github.com/tc39/ecma262/pulls?q=is%3Apr+is%3Aopen+label%3A%22needs+tests%22
+ This is a list of the "needs tests" PRs for changes to the specification.
document `timeout` and clean up CONTRIBUTING.md - document `timeout` tag - reorder tags in frontmatter doc - minor cleanups - minor fixes - add style note - reformat flags - remove discussion of obsolete $INCLUDE - incorporate line notes from @domenic - integrate additional comments - add links back, move arg notes down - Raise outline level by one README.md - link to CONTRIBUTING
2014-08-22 17:47:18 +02:00
## Test Case Names
CONTRIBUTING: document test262.py, async tests add a section on requirements for using console test runner test262.py add a section on asynchronous tests add subsection to authoring guidelines with suggestion for test names
2014-07-21 17:39:10 +02:00
Update contribution guidelines - Remove trailing white space - Streamline documentation of test tags - Do not reference obsolete tags - Document `features` frontmatter tag - Document `es6id` frontmatter tag - Omit unnecessary detail about test262 website generation. This is not directly useful to potential test contributors. Implementation details like these can be taken for granted by that audience. - Remove documentation on YAML syntax. Details on YAML may be helpful for some new contributors, but this document should not attempt to cover the topic (especially not from the description of a specific frontmatter entry). Replace with a link to a more comprehensive source as this will be more generally useful to those who need it (and less obtrusive for those who do not). - Consolidate information on test helpers - Document `assert` helpers - Update instructions for asserting errors. Since the introduction of `assert.throws` in gh-22, the preferred means of expressing expectations regarding errors has changed. Update the CONTRIBUTING.md file to reflect the latest approach. Explain purpose of `throw NotEarlyError;` in example test. - Re-order information on file names. The inconsistency in the project's file names should not go unmentioned, but neither should it not preceed instructions for the accepted approach to namine tests. - More clearly document required frontmatter tags. Explicitly list `description` as a required frontmatter tag, implicitly identifying all other tags as optional.
2015-05-25 23:51:55 +02:00
Test cases should be created in files that are named to identify the feature or API that's being tested.
CONTRIBUTING: document test262.py, async tests add a section on requirements for using console test runner test262.py add a section on asynchronous tests add subsection to authoring guidelines with suggestion for test names
2014-07-21 17:39:10 +02:00
update CONTRIBUTING.md to reflect current practices
2017-10-06 22:09:39 +02:00
There is no strict naming convention. The file names should be human readable, helpful and, ideally, consistent within a single directory. For examples:
CONTRIBUTING: document test262.py, async tests add a section on requirements for using console test runner test262.py add a section on asynchronous tests add subsection to authoring guidelines with suggestion for test names
2014-07-21 17:39:10 +02:00
Contributing: updates the Test Case Names section. Closes gh-150. Closes gh-136
2015-02-13 23:39:17 +01:00
- `Math.fround` handling of `Infinity`: `test/built-ins/Math/fround/Math.fround_Infinity.js`
update CONTRIBUTING.md to reflect current practices
2017-10-06 22:09:39 +02:00
- `Array.prototype.find` use with `Proxy`: `test/built-ins/Array/prototype/find/Array.prototype.find_callable-Proxy-1.js`
Contributing: updates the Test Case Names section. Closes gh-150. Closes gh-136
2015-02-13 23:39:17 +01:00
- `arguments` implements an `iterator` interface: `test/language/arguments-object/iterator-interface.js`
CONTRIBUTING: document test262.py, async tests add a section on requirements for using console test runner test262.py add a section on asynchronous tests add subsection to authoring guidelines with suggestion for test names
2014-07-21 17:39:10 +02:00
update CONTRIBUTING.md to reflect current practices
2017-10-06 22:09:39 +02:00
See the following directory trees for further recommended examples:
- [test/built-ins/Math/fround](./test/built-ins/Math/fround/)
- [test/built-ins/WeakMap](./test/built-ins/WeakMap/)
- [test/language/arguments-object](./test/language/arguments-object/)
Update contribution guidelines It was recently decided to prefer the new `id` tag over the existing `es5id` and `es6id` tag when authoring tests. Update the contribution guidelines to reference the new tag.
2016-02-04 21:37:02 +01:00
**Note** The project is currently transitioning from a naming system based on specification section numbers. There remains a substantial number of tests that conform to this outdated convention; contributors should ignore that approach when introducing new tests and instead encode this information using the [id](#id) frontmatter tag.
CONTRIBUTING: more notes on test writing Add section on test environment Add section on custom helpers describe YAML frontmatter Fix minor formatting errors document $INCLUDE as obsolete Change documentation of negative error Move test environment and custom helpers down indent copyright and frontmatter sections better description of the async calls between a promise and the functions in its `.then` Correct Early Error example: don't throw a string CONSOLE-RUNNER: split runner doc into new file add troubleshooting section
2014-07-30 00:27:20 +02:00
document `timeout` and clean up CONTRIBUTING.md - document `timeout` tag - reorder tags in frontmatter doc - minor cleanups - minor fixes - add style note - reformat flags - remove discussion of obsolete $INCLUDE - incorporate line notes from @domenic - integrate additional comments - add links back, move arg notes down - Raise outline level by one README.md - link to CONTRIBUTING
2014-08-22 17:47:18 +02:00
## Test Case Style
CONTRIBUTING: more notes on test writing Add section on test environment Add section on custom helpers describe YAML frontmatter Fix minor formatting errors document $INCLUDE as obsolete Change documentation of negative error Move test environment and custom helpers down indent copyright and frontmatter sections better description of the async calls between a promise and the functions in its `.then` Correct Early Error example: don't throw a string CONSOLE-RUNNER: split runner doc into new file add troubleshooting section
2014-07-30 00:27:20 +02:00
A test file has three sections: Copyright, Frontmatter, and Body. A test looks roughly like this:
Added contributing.md and improved README
2014-07-04 02:13:42 +02:00
```javascript
Update copyright year in sample copyright
2015-02-17 17:54:51 +01:00
// Copyright (C) 2015 [Contributor Name]. All rights reserved.
Added contributing.md and improved README
2014-07-04 02:13:42 +02:00
// This code is governed by the BSD license found in the LICENSE file.
Update contribution guidelines - Remove trailing white space - Streamline documentation of test tags - Do not reference obsolete tags - Document `features` frontmatter tag - Document `es6id` frontmatter tag - Omit unnecessary detail about test262 website generation. This is not directly useful to potential test contributors. Implementation details like these can be taken for granted by that audience. - Remove documentation on YAML syntax. Details on YAML may be helpful for some new contributors, but this document should not attempt to cover the topic (especially not from the description of a specific frontmatter entry). Replace with a link to a more comprehensive source as this will be more generally useful to those who need it (and less obtrusive for those who do not). - Consolidate information on test helpers - Document `assert` helpers - Update instructions for asserting errors. Since the introduction of `assert.throws` in gh-22, the preferred means of expressing expectations regarding errors has changed. Update the CONTRIBUTING.md file to reflect the latest approach. Explain purpose of `throw NotEarlyError;` in example test. - Re-order information on file names. The inconsistency in the project's file names should not go unmentioned, but neither should it not preceed instructions for the accepted approach to namine tests. - More clearly document required frontmatter tags. Explicitly list `description` as a required frontmatter tag, implicitly identifying all other tags as optional.
2015-05-25 23:51:55 +02:00
/*---
document `timeout` and clean up CONTRIBUTING.md - document `timeout` tag - reorder tags in frontmatter doc - minor cleanups - minor fixes - add style note - reformat flags - remove discussion of obsolete $INCLUDE - incorporate line notes from @domenic - integrate additional comments - add links back, move arg notes down - Raise outline level by one README.md - link to CONTRIBUTING
2014-08-22 17:47:18 +02:00
description: brief description
CONTRIBUTING: more notes on test writing Add section on test environment Add section on custom helpers describe YAML frontmatter Fix minor formatting errors document $INCLUDE as obsolete Change documentation of negative error Move test environment and custom helpers down indent copyright and frontmatter sections better description of the async calls between a promise and the functions in its `.then` Correct Early Error example: don't throw a string CONSOLE-RUNNER: split runner doc into new file add troubleshooting section
2014-07-30 00:27:20 +02:00
info: >
verbose test description, multiple lines OK.
Fix CONTRIBUTING.md description of the info frontmatter tag
2017-10-04 23:45:03 +02:00
(info typically contains relevant, direct quote from ECMAScript)
CONTRIBUTING: more notes on test writing Add section on test environment Add section on custom helpers describe YAML frontmatter Fix minor formatting errors document $INCLUDE as obsolete Change documentation of negative error Move test environment and custom helpers down indent copyright and frontmatter sections better description of the async calls between a promise and the functions in its `.then` Correct Early Error example: don't throw a string CONSOLE-RUNNER: split runner doc into new file add troubleshooting section
2014-07-30 00:27:20 +02:00
---*/
Update contribution guidelines - Remove trailing white space - Streamline documentation of test tags - Do not reference obsolete tags - Document `features` frontmatter tag - Document `es6id` frontmatter tag - Omit unnecessary detail about test262 website generation. This is not directly useful to potential test contributors. Implementation details like these can be taken for granted by that audience. - Remove documentation on YAML syntax. Details on YAML may be helpful for some new contributors, but this document should not attempt to cover the topic (especially not from the description of a specific frontmatter entry). Replace with a link to a more comprehensive source as this will be more generally useful to those who need it (and less obtrusive for those who do not). - Consolidate information on test helpers - Document `assert` helpers - Update instructions for asserting errors. Since the introduction of `assert.throws` in gh-22, the preferred means of expressing expectations regarding errors has changed. Update the CONTRIBUTING.md file to reflect the latest approach. Explain purpose of `throw NotEarlyError;` in example test. - Re-order information on file names. The inconsistency in the project's file names should not go unmentioned, but neither should it not preceed instructions for the accepted approach to namine tests. - More clearly document required frontmatter tags. Explicitly list `description` as a required frontmatter tag, implicitly identifying all other tags as optional.
2015-05-25 23:51:55 +02:00
Added contributing.md and improved README
2014-07-04 02:13:42 +02:00
[Test Code]
```
CONTRIBUTING: document test262.py, async tests add a section on requirements for using console test runner test262.py add a section on asynchronous tests add subsection to authoring guidelines with suggestion for test names
2014-07-21 17:39:10 +02:00
document `timeout` and clean up CONTRIBUTING.md - document `timeout` tag - reorder tags in frontmatter doc - minor cleanups - minor fixes - add style note - reformat flags - remove discussion of obsolete $INCLUDE - incorporate line notes from @domenic - integrate additional comments - add links back, move arg notes down - Raise outline level by one README.md - link to CONTRIBUTING
2014-08-22 17:47:18 +02:00
### Copyright
CONTRIBUTING: more notes on test writing Add section on test environment Add section on custom helpers describe YAML frontmatter Fix minor formatting errors document $INCLUDE as obsolete Change documentation of negative error Move test environment and custom helpers down indent copyright and frontmatter sections better description of the async calls between a promise and the functions in its `.then` Correct Early Error example: don't throw a string CONSOLE-RUNNER: split runner doc into new file add troubleshooting section
2014-07-30 00:27:20 +02:00
The copyright block must be the first section of the test. The copyright block must use `//` style comments.
document `timeout` and clean up CONTRIBUTING.md - document `timeout` tag - reorder tags in frontmatter doc - minor cleanups - minor fixes - add style note - reformat flags - remove discussion of obsolete $INCLUDE - incorporate line notes from @domenic - integrate additional comments - add links back, move arg notes down - Raise outline level by one README.md - link to CONTRIBUTING
2014-08-22 17:47:18 +02:00
### Frontmatter
CONTRIBUTING: more notes on test writing Add section on test environment Add section on custom helpers describe YAML frontmatter Fix minor formatting errors document $INCLUDE as obsolete Change documentation of negative error Move test environment and custom helpers down indent copyright and frontmatter sections better description of the async calls between a promise and the functions in its `.then` Correct Early Error example: don't throw a string CONSOLE-RUNNER: split runner doc into new file add troubleshooting section
2014-07-30 00:27:20 +02:00
Update contribution guidelines - Remove trailing white space - Streamline documentation of test tags - Do not reference obsolete tags - Document `features` frontmatter tag - Document `es6id` frontmatter tag - Omit unnecessary detail about test262 website generation. This is not directly useful to potential test contributors. Implementation details like these can be taken for granted by that audience. - Remove documentation on YAML syntax. Details on YAML may be helpful for some new contributors, but this document should not attempt to cover the topic (especially not from the description of a specific frontmatter entry). Replace with a link to a more comprehensive source as this will be more generally useful to those who need it (and less obtrusive for those who do not). - Consolidate information on test helpers - Document `assert` helpers - Update instructions for asserting errors. Since the introduction of `assert.throws` in gh-22, the preferred means of expressing expectations regarding errors has changed. Update the CONTRIBUTING.md file to reflect the latest approach. Explain purpose of `throw NotEarlyError;` in example test. - Re-order information on file names. The inconsistency in the project's file names should not go unmentioned, but neither should it not preceed instructions for the accepted approach to namine tests. - More clearly document required frontmatter tags. Explicitly list `description` as a required frontmatter tag, implicitly identifying all other tags as optional.
2015-05-25 23:51:55 +02:00
The Test262 frontmatter is a string of [YAML](https://en.wikipedia.org/wiki/YAML) enclosed by the comment start tag `/*---` and end tag `---*/`. There must be exactly one Frontmatter per test.
CONTRIBUTING: more notes on test writing Add section on test environment Add section on custom helpers describe YAML frontmatter Fix minor formatting errors document $INCLUDE as obsolete Change documentation of negative error Move test environment and custom helpers down indent copyright and frontmatter sections better description of the async calls between a promise and the functions in its `.then` Correct Early Error example: don't throw a string CONSOLE-RUNNER: split runner doc into new file add troubleshooting section
2014-07-30 00:27:20 +02:00
Update contribution guidelines - Remove trailing white space - Streamline documentation of test tags - Do not reference obsolete tags - Document `features` frontmatter tag - Document `es6id` frontmatter tag - Omit unnecessary detail about test262 website generation. This is not directly useful to potential test contributors. Implementation details like these can be taken for granted by that audience. - Remove documentation on YAML syntax. Details on YAML may be helpful for some new contributors, but this document should not attempt to cover the topic (especially not from the description of a specific frontmatter entry). Replace with a link to a more comprehensive source as this will be more generally useful to those who need it (and less obtrusive for those who do not). - Consolidate information on test helpers - Document `assert` helpers - Update instructions for asserting errors. Since the introduction of `assert.throws` in gh-22, the preferred means of expressing expectations regarding errors has changed. Update the CONTRIBUTING.md file to reflect the latest approach. Explain purpose of `throw NotEarlyError;` in example test. - Re-order information on file names. The inconsistency in the project's file names should not go unmentioned, but neither should it not preceed instructions for the accepted approach to namine tests. - More clearly document required frontmatter tags. Explicitly list `description` as a required frontmatter tag, implicitly identifying all other tags as optional.
2015-05-25 23:51:55 +02:00
Test262 supports the following tags:
CONTRIBUTING: more notes on test writing Add section on test environment Add section on custom helpers describe YAML frontmatter Fix minor formatting errors document $INCLUDE as obsolete Change documentation of negative error Move test environment and custom helpers down indent copyright and frontmatter sections better description of the async calls between a promise and the functions in its `.then` Correct Early Error example: don't throw a string CONSOLE-RUNNER: split runner doc into new file add troubleshooting section
2014-07-30 00:27:20 +02:00
Update contribution guidelines - Remove trailing white space - Streamline documentation of test tags - Do not reference obsolete tags - Document `features` frontmatter tag - Document `es6id` frontmatter tag - Omit unnecessary detail about test262 website generation. This is not directly useful to potential test contributors. Implementation details like these can be taken for granted by that audience. - Remove documentation on YAML syntax. Details on YAML may be helpful for some new contributors, but this document should not attempt to cover the topic (especially not from the description of a specific frontmatter entry). Replace with a link to a more comprehensive source as this will be more generally useful to those who need it (and less obtrusive for those who do not). - Consolidate information on test helpers - Document `assert` helpers - Update instructions for asserting errors. Since the introduction of `assert.throws` in gh-22, the preferred means of expressing expectations regarding errors has changed. Update the CONTRIBUTING.md file to reflect the latest approach. Explain purpose of `throw NotEarlyError;` in example test. - Re-order information on file names. The inconsistency in the project's file names should not go unmentioned, but neither should it not preceed instructions for the accepted approach to namine tests. - More clearly document required frontmatter tags. Explicitly list `description` as a required frontmatter tag, implicitly identifying all other tags as optional.
2015-05-25 23:51:55 +02:00
- [**description**](#description) (required)
document `timeout` and clean up CONTRIBUTING.md - document `timeout` tag - reorder tags in frontmatter doc - minor cleanups - minor fixes - add style note - reformat flags - remove discussion of obsolete $INCLUDE - incorporate line notes from @domenic - integrate additional comments - add links back, move arg notes down - Raise outline level by one README.md - link to CONTRIBUTING
2014-08-22 17:47:18 +02:00
- [**info**](#info)
- [**negative**](#negative)
add note to list that esid is required for new tests only
2017-06-28 18:45:25 +02:00
- [**esid**](#esid) (required for new tests)
document `timeout` and clean up CONTRIBUTING.md - document `timeout` tag - reorder tags in frontmatter doc - minor cleanups - minor fixes - add style note - reformat flags - remove discussion of obsolete $INCLUDE - incorporate line notes from @domenic - integrate additional comments - add links back, move arg notes down - Raise outline level by one README.md - link to CONTRIBUTING
2014-08-22 17:47:18 +02:00
- [**includes**](#includes)
- [**timeout**](#timeout)
- [**author**](#author)
- [**flags**](#flags)
Update contribution guidelines - Remove trailing white space - Streamline documentation of test tags - Do not reference obsolete tags - Document `features` frontmatter tag - Document `es6id` frontmatter tag - Omit unnecessary detail about test262 website generation. This is not directly useful to potential test contributors. Implementation details like these can be taken for granted by that audience. - Remove documentation on YAML syntax. Details on YAML may be helpful for some new contributors, but this document should not attempt to cover the topic (especially not from the description of a specific frontmatter entry). Replace with a link to a more comprehensive source as this will be more generally useful to those who need it (and less obtrusive for those who do not). - Consolidate information on test helpers - Document `assert` helpers - Update instructions for asserting errors. Since the introduction of `assert.throws` in gh-22, the preferred means of expressing expectations regarding errors has changed. Update the CONTRIBUTING.md file to reflect the latest approach. Explain purpose of `throw NotEarlyError;` in example test. - Re-order information on file names. The inconsistency in the project's file names should not go unmentioned, but neither should it not preceed instructions for the accepted approach to namine tests. - More clearly document required frontmatter tags. Explicitly list `description` as a required frontmatter tag, implicitly identifying all other tags as optional.
2015-05-25 23:51:55 +02:00
- [**features**](#features)
document `timeout` and clean up CONTRIBUTING.md - document `timeout` tag - reorder tags in frontmatter doc - minor cleanups - minor fixes - add style note - reformat flags - remove discussion of obsolete $INCLUDE - incorporate line notes from @domenic - integrate additional comments - add links back, move arg notes down - Raise outline level by one README.md - link to CONTRIBUTING
2014-08-22 17:47:18 +02:00
update CONTRIBUTING.md to reflect current practices
2017-10-06 22:09:39 +02:00
The following tags are deprecated, but exist in old tests:
- [**es5id**](#es5id) (deprecated)
- [**es6id**](#es6id) (deprecated)
document `timeout` and clean up CONTRIBUTING.md - document `timeout` tag - reorder tags in frontmatter doc - minor cleanups - minor fixes - add style note - reformat flags - remove discussion of obsolete $INCLUDE - incorporate line notes from @domenic - integrate additional comments - add links back, move arg notes down - Raise outline level by one README.md - link to CONTRIBUTING
2014-08-22 17:47:18 +02:00
#### description
**description**: [string]
update CONTRIBUTING.md to reflect current practices
2017-10-06 22:09:39 +02:00
This is one of two required frontmatter tags. The description should be a short, one-line description of the purpose of this testcase. We suggested that the description be kept to less than 100 characters, but clarity is preferred over brevity.
CONTRIBUTING: document test262.py, async tests add a section on requirements for using console test runner test262.py add a section on asynchronous tests add subsection to authoring guidelines with suggestion for test names
2014-07-21 17:39:10 +02:00
Added contributing.md and improved README
2014-07-04 02:13:42 +02:00
Eg: Insert <LS> between chunks of one string
CONTRIBUTING: document test262.py, async tests add a section on requirements for using console test runner test262.py add a section on asynchronous tests add subsection to authoring guidelines with suggestion for test names
2014-07-21 17:39:10 +02:00
document `timeout` and clean up CONTRIBUTING.md - document `timeout` tag - reorder tags in frontmatter doc - minor cleanups - minor fixes - add style note - reformat flags - remove discussion of obsolete $INCLUDE - incorporate line notes from @domenic - integrate additional comments - add links back, move arg notes down - Raise outline level by one README.md - link to CONTRIBUTING
2014-08-22 17:47:18 +02:00
#### info
**info**: [multiline string]
Fix CONTRIBUTING.md description of the info frontmatter tag
2017-10-04 23:45:03 +02:00
This allows a long, free-form comment. The comment is almost always a direct
quote from ECMAScript. It is used to indicate the observable being tested
within the file.
document `timeout` and clean up CONTRIBUTING.md - document `timeout` tag - reorder tags in frontmatter doc - minor cleanups - minor fixes - add style note - reformat flags - remove discussion of obsolete $INCLUDE - incorporate line notes from @domenic - integrate additional comments - add links back, move arg notes down - Raise outline level by one README.md - link to CONTRIBUTING
2014-08-22 17:47:18 +02:00
Fix CONTRIBUTING.md description of the info frontmatter tag
2017-10-04 23:45:03 +02:00
For example:
/*---
esid: sec-weakset.prototype.has
description: Throws TypeError if `this` is not Object.
info: >
WeakSet.prototype.has ( value )
1. Let S be the this value.
2. If Type(S) is not Object, throw a TypeError exception.
---*/
Note: Adding more context than the direct quote from ECMAScript should rarely
be necessary. If you must add context to the quote, use the JavaScript
single line comment syntax.
document `timeout` and clean up CONTRIBUTING.md - document `timeout` tag - reorder tags in frontmatter doc - minor cleanups - minor fixes - add style note - reformat flags - remove discussion of obsolete $INCLUDE - incorporate line notes from @domenic - integrate additional comments - add links back, move arg notes down - Raise outline level by one README.md - link to CONTRIBUTING
2014-08-22 17:47:18 +02:00
#### negative
Update documentation
2016-03-13 18:26:10 +01:00
**negative**: [dictionary containing **phase** and **type**]
CONTRIBUTING: document test262.py, async tests add a section on requirements for using console test runner test262.py add a section on asynchronous tests add subsection to authoring guidelines with suggestion for test names
2014-07-21 17:39:10 +02:00
CONTRIBUTING.md: minor fixes Minor fixes: typos, formatting, verb tense&agreement Add example of short-list `includes: []` and long-list `includes: \n - \n -`
2014-07-31 22:41:08 +02:00
This means the test is expected to throw an error of the given type. If no error is thrown, a test failure is reported.
CONTRIBUTING: document test262.py, async tests add a section on requirements for using console test runner test262.py add a section on asynchronous tests add subsection to authoring guidelines with suggestion for test names
2014-07-21 17:39:10 +02:00
Update documentation
2016-03-13 18:26:10 +01:00
- **type**- If an error is thrown, it is implicitly converted to a string. In order for the test to pass, this value must match the name of the error constructor.
- **phase** - Negative tests whose **phase** value is "early" must produce the specified error prior to executing code. The value "runtime" dictates that the error is expected to be produced as a result of executing the test code.
CONTRIBUTING: more notes on test writing Add section on test environment Add section on custom helpers describe YAML frontmatter Fix minor formatting errors document $INCLUDE as obsolete Change documentation of negative error Move test environment and custom helpers down indent copyright and frontmatter sections better description of the async calls between a promise and the functions in its `.then` Correct Early Error example: don't throw a string CONSOLE-RUNNER: split runner doc into new file add troubleshooting section
2014-07-30 00:27:20 +02:00
CONTRIBUTING.md: Link #handling-errors-and-negative-test-cases (#1295)
2017-10-17 22:45:37 +02:00
For best practices on how to use the negative tag please see [Handling Errors and Negative Test Cases](#handling-errors-and-negative-test-cases), below.
CONTRIBUTING: more notes on test writing Add section on test environment Add section on custom helpers describe YAML frontmatter Fix minor formatting errors document $INCLUDE as obsolete Change documentation of negative error Move test environment and custom helpers down indent copyright and frontmatter sections better description of the async calls between a promise and the functions in its `.then` Correct Early Error example: don't throw a string CONSOLE-RUNNER: split runner doc into new file add troubleshooting section
2014-07-30 00:27:20 +02:00
Update documentation
2016-03-13 18:26:10 +01:00
For example:
negative:
phase: early
type: ReferenceError
s/id/esid Fixes #477
2016-02-22 15:47:03 +01:00
#### esid
**esid**: [spec-id]
Document frontmatter's id tag Fixes #477
2016-01-28 21:55:50 +01:00
update contributing.md to make esid required for new feature tests
2017-06-27 23:29:22 +02:00
This tag is required for all new feature tests. This tag identifies the hash ID from the portion of the ECMAScript draft which is most recent to the date the test was added. It represents the anchors on the generated HTML version of the specs. E.g.: `esid: sec-typedarray-length`. This tag might be used to replace a `es6id` or further.
Document frontmatter's id tag Fixes #477
2016-01-28 21:55:50 +01:00
Replace "id" with "ID"
2016-01-29 18:29:09 +01:00
When writing a new test for a Stage 3+ spec not yet published on the draft, the `pending` value can be used while a hash ID is not available.
CONTRIBUTING: more notes on test writing Add section on test environment Add section on custom helpers describe YAML frontmatter Fix minor formatting errors document $INCLUDE as obsolete Change documentation of negative error Move test environment and custom helpers down indent copyright and frontmatter sections better description of the async calls between a promise and the functions in its `.then` Correct Early Error example: don't throw a string CONSOLE-RUNNER: split runner doc into new file add troubleshooting section
2014-07-30 00:27:20 +02:00
document `timeout` and clean up CONTRIBUTING.md - document `timeout` tag - reorder tags in frontmatter doc - minor cleanups - minor fixes - add style note - reformat flags - remove discussion of obsolete $INCLUDE - incorporate line notes from @domenic - integrate additional comments - add links back, move arg notes down - Raise outline level by one README.md - link to CONTRIBUTING
2014-08-22 17:47:18 +02:00
#### includes
**includes**: [file-list]
This tag names a list of helper files that will be included in the test environment prior to running the test. Filenames **must** include the `.js` extension.
CONTRIBUTING: more notes on test writing Add section on test environment Add section on custom helpers describe YAML frontmatter Fix minor formatting errors document $INCLUDE as obsolete Change documentation of negative error Move test environment and custom helpers down indent copyright and frontmatter sections better description of the async calls between a promise and the functions in its `.then` Correct Early Error example: don't throw a string CONSOLE-RUNNER: split runner doc into new file add troubleshooting section
2014-07-30 00:27:20 +02:00
Update contribution guidelines - Remove trailing white space - Streamline documentation of test tags - Do not reference obsolete tags - Document `features` frontmatter tag - Document `es6id` frontmatter tag - Omit unnecessary detail about test262 website generation. This is not directly useful to potential test contributors. Implementation details like these can be taken for granted by that audience. - Remove documentation on YAML syntax. Details on YAML may be helpful for some new contributors, but this document should not attempt to cover the topic (especially not from the description of a specific frontmatter entry). Replace with a link to a more comprehensive source as this will be more generally useful to those who need it (and less obtrusive for those who do not). - Consolidate information on test helpers - Document `assert` helpers - Update instructions for asserting errors. Since the introduction of `assert.throws` in gh-22, the preferred means of expressing expectations regarding errors has changed. Update the CONTRIBUTING.md file to reflect the latest approach. Explain purpose of `throw NotEarlyError;` in example test. - Re-order information on file names. The inconsistency in the project's file names should not go unmentioned, but neither should it not preceed instructions for the accepted approach to namine tests. - More clearly document required frontmatter tags. Explicitly list `description` as a required frontmatter tag, implicitly identifying all other tags as optional.
2015-05-25 23:51:55 +02:00
The helper files are found in the `test/harness/` directory. When some code is used repeatedly across a group of tests, a new helper function (or group of helpers) can be defined. Helpers increase test complexity, so they should be created and used sparingly.
CONTRIBUTING.md: minor fixes Minor fixes: typos, formatting, verb tense&agreement Add example of short-list `includes: []` and long-list `includes: \n - \n -`
2014-07-31 22:41:08 +02:00
document `timeout` and clean up CONTRIBUTING.md - document `timeout` tag - reorder tags in frontmatter doc - minor cleanups - minor fixes - add style note - reformat flags - remove discussion of obsolete $INCLUDE - incorporate line notes from @domenic - integrate additional comments - add links back, move arg notes down - Raise outline level by one README.md - link to CONTRIBUTING
2014-08-22 17:47:18 +02:00
#### timeout
**timeout**: [integer]
This tag specifies the number of milliseconds to wait before the test runner declares an [asynchronous test](#writing-asynchronous-tests) to have timed out. It has no effect on synchronous tests.
Update contribution guidelines - Remove trailing white space - Streamline documentation of test tags - Do not reference obsolete tags - Document `features` frontmatter tag - Document `es6id` frontmatter tag - Omit unnecessary detail about test262 website generation. This is not directly useful to potential test contributors. Implementation details like these can be taken for granted by that audience. - Remove documentation on YAML syntax. Details on YAML may be helpful for some new contributors, but this document should not attempt to cover the topic (especially not from the description of a specific frontmatter entry). Replace with a link to a more comprehensive source as this will be more generally useful to those who need it (and less obtrusive for those who do not). - Consolidate information on test helpers - Document `assert` helpers - Update instructions for asserting errors. Since the introduction of `assert.throws` in gh-22, the preferred means of expressing expectations regarding errors has changed. Update the CONTRIBUTING.md file to reflect the latest approach. Explain purpose of `throw NotEarlyError;` in example test. - Re-order information on file names. The inconsistency in the project's file names should not go unmentioned, but neither should it not preceed instructions for the accepted approach to namine tests. - More clearly document required frontmatter tags. Explicitly list `description` as a required frontmatter tag, implicitly identifying all other tags as optional.
2015-05-25 23:51:55 +02:00
Test authors **should not** use this tag except as a last resort. Each runner is allowed to provide its own default timeout, and the user may be permitted to override this in order to account for unusually fast or slow hardware, network delays, etc.
document `timeout` and clean up CONTRIBUTING.md - document `timeout` tag - reorder tags in frontmatter doc - minor cleanups - minor fixes - add style note - reformat flags - remove discussion of obsolete $INCLUDE - incorporate line notes from @domenic - integrate additional comments - add links back, move arg notes down - Raise outline level by one README.md - link to CONTRIBUTING
2014-08-22 17:47:18 +02:00
#### author
**author**: [string]
Update contribution guidelines - Remove trailing white space - Streamline documentation of test tags - Do not reference obsolete tags - Document `features` frontmatter tag - Document `es6id` frontmatter tag - Omit unnecessary detail about test262 website generation. This is not directly useful to potential test contributors. Implementation details like these can be taken for granted by that audience. - Remove documentation on YAML syntax. Details on YAML may be helpful for some new contributors, but this document should not attempt to cover the topic (especially not from the description of a specific frontmatter entry). Replace with a link to a more comprehensive source as this will be more generally useful to those who need it (and less obtrusive for those who do not). - Consolidate information on test helpers - Document `assert` helpers - Update instructions for asserting errors. Since the introduction of `assert.throws` in gh-22, the preferred means of expressing expectations regarding errors has changed. Update the CONTRIBUTING.md file to reflect the latest approach. Explain purpose of `throw NotEarlyError;` in example test. - Re-order information on file names. The inconsistency in the project's file names should not go unmentioned, but neither should it not preceed instructions for the accepted approach to namine tests. - More clearly document required frontmatter tags. Explicitly list `description` as a required frontmatter tag, implicitly identifying all other tags as optional.
2015-05-25 23:51:55 +02:00
This tag is used to identify the author of a test case.
document `timeout` and clean up CONTRIBUTING.md - document `timeout` tag - reorder tags in frontmatter doc - minor cleanups - minor fixes - add style note - reformat flags - remove discussion of obsolete $INCLUDE - incorporate line notes from @domenic - integrate additional comments - add links back, move arg notes down - Raise outline level by one README.md - link to CONTRIBUTING
2014-08-22 17:47:18 +02:00
#### flags
**flags**: [list]
CONTRIBUTING: more notes on test writing Add section on test environment Add section on custom helpers describe YAML frontmatter Fix minor formatting errors document $INCLUDE as obsolete Change documentation of negative error Move test environment and custom helpers down indent copyright and frontmatter sections better description of the async calls between a promise and the functions in its `.then` Correct Early Error example: don't throw a string CONSOLE-RUNNER: split runner doc into new file add troubleshooting section
2014-07-30 00:27:20 +02:00
This tag is for boolean properties associated with the test.
Update browser runner to honor `onlyStrict` flag Unlike the console runner, the browser runner does not modify the strictness of tests prior to running them. Regardless of a given test's metadata, it runs every test exactly once, and it never enables strict mode. This means that tests intended to function in strict mode must declare the "use strict"; directive prologue in addition to the `onlyStrict` flag. For any test that specifies the `onlyStrict` metadata flag, transform the source code by injecting a "use strict" directive prologue prior to running the test.
2015-06-08 23:05:22 +02:00
- **`onlyStrict`** - only run the test in strict mode
Update contribution guidelines - Remove trailing white space - Streamline documentation of test tags - Do not reference obsolete tags - Document `features` frontmatter tag - Document `es6id` frontmatter tag - Omit unnecessary detail about test262 website generation. This is not directly useful to potential test contributors. Implementation details like these can be taken for granted by that audience. - Remove documentation on YAML syntax. Details on YAML may be helpful for some new contributors, but this document should not attempt to cover the topic (especially not from the description of a specific frontmatter entry). Replace with a link to a more comprehensive source as this will be more generally useful to those who need it (and less obtrusive for those who do not). - Consolidate information on test helpers - Document `assert` helpers - Update instructions for asserting errors. Since the introduction of `assert.throws` in gh-22, the preferred means of expressing expectations regarding errors has changed. Update the CONTRIBUTING.md file to reflect the latest approach. Explain purpose of `throw NotEarlyError;` in example test. - Re-order information on file names. The inconsistency in the project's file names should not go unmentioned, but neither should it not preceed instructions for the accepted approach to namine tests. - More clearly document required frontmatter tags. Explicitly list `description` as a required frontmatter tag, implicitly identifying all other tags as optional.
2015-05-25 23:51:55 +02:00
- **`noStrict`** - only run the test in "sloppy" mode
Add tests for early errors in module syntax Introduce the `module` flag to unambiguously identify tests that are intended to be interpreted as module code.
2015-05-28 20:41:22 +02:00
- **`module`** - interpret the source text as [module
CONTRIBUTING: Link to the latest spec draft (#978)
2017-04-14 18:19:39 +02:00
code](https://tc39.github.io/ecma262/#sec-modules)
Implement `raw` flag Some tests involving the directive prologue are invalidated by source text transformations that insert executable code in the beginning of the script. Implement a `raw` flag that allows these tests to opt-out of this transformation. Update the relevant tests to use this flag (and remove references to globals only available when code is injected). Update the Python runner accordingly: - Do not run tests marked as "raw" in strict mode - Reject invalid test configurations Update the browser runner accordingly: - Do not modify the script body of tests marked as "raw"
2015-06-08 21:35:16 +02:00
- **`raw`** - execute the test without any modification (no helpers will be
available); necessary to test the behavior of directive prologue; implies
`noStrict`
Make asynchronous test configuration explicit For asynchronous tests, the contract between test file and test runner is implicit: runners are expected to inspect the source code for references to a global `$DONE` identifier. Promote a more explicit contract between test file and test runner by introducing a new frontmatter "tag", `async`. This brings asynchronous test configuration in-line with other configuration mechanisms and also provides a more natural means of test filtering. The modifications to test files was made programatically using the `grep` and `sed` utilities: $ grep "\$DONE" test/ -r --files-with-match --null | \ xargs -0 sed -i 's/^\(flags:\s*\)\[/\1[async, /g' $ grep "\$DONE" test/ -rl --null | \ xargs -0 grep -E '^flags:' --files-without-match --null | \ xargs -0 sed -i 's/^---\*\//flags: [async]\n---*\//'
2016-02-12 18:59:51 +01:00
- **`async`** - defer interpretation of test results until after the invocation
Fix markdown typo (#763)
2016-09-27 23:05:08 +02:00
of the global `$DONE` function
Document test flag: `generated` (#886) Resolves gh-885
2017-03-06 18:12:42 +01:00
- **`generated`** - informative flag used to denote test files that were
created procedurally using the project's test generation tool; refer to the
section titled "Procedurally-generated tests" for more information on this
process
document `timeout` and clean up CONTRIBUTING.md - document `timeout` tag - reorder tags in frontmatter doc - minor cleanups - minor fixes - add style note - reformat flags - remove discussion of obsolete $INCLUDE - incorporate line notes from @domenic - integrate additional comments - add links back, move arg notes down - Raise outline level by one README.md - link to CONTRIBUTING
2014-08-22 17:47:18 +02:00
Update contribution guidelines - Remove trailing white space - Streamline documentation of test tags - Do not reference obsolete tags - Document `features` frontmatter tag - Document `es6id` frontmatter tag - Omit unnecessary detail about test262 website generation. This is not directly useful to potential test contributors. Implementation details like these can be taken for granted by that audience. - Remove documentation on YAML syntax. Details on YAML may be helpful for some new contributors, but this document should not attempt to cover the topic (especially not from the description of a specific frontmatter entry). Replace with a link to a more comprehensive source as this will be more generally useful to those who need it (and less obtrusive for those who do not). - Consolidate information on test helpers - Document `assert` helpers - Update instructions for asserting errors. Since the introduction of `assert.throws` in gh-22, the preferred means of expressing expectations regarding errors has changed. Update the CONTRIBUTING.md file to reflect the latest approach. Explain purpose of `throw NotEarlyError;` in example test. - Re-order information on file names. The inconsistency in the project's file names should not go unmentioned, but neither should it not preceed instructions for the accepted approach to namine tests. - More clearly document required frontmatter tags. Explicitly list `description` as a required frontmatter tag, implicitly identifying all other tags as optional.
2015-05-25 23:51:55 +02:00
#### features
**features**: [list]
CONTRIBUTING: document test262.py, async tests add a section on requirements for using console test runner test262.py add a section on asynchronous tests add subsection to authoring guidelines with suggestion for test names
2014-07-21 17:39:10 +02:00
Lint test `features` tag (#1060) A recent commit introduced a document that enumerated acceptable values for the test "features" metadata tag. However, this list was incomplete, and maintaining it placed extra burden on the project owners. Restructure the document into a machine-readable format. Add entries for all previously-omitted values. Add in-line documentation with recommendations for maintenance of the file. Extend the project's linting tool to validate tests according to the document's contents.
2017-06-12 23:59:14 +02:00
Some tests require the use of language features that are not directly described by the test file's location in the directory structure. These features should be specified with this tag. See the `features.txt` file for a complete list of available values.
CONTRIBUTING: more notes on test writing Add section on test environment Add section on custom helpers describe YAML frontmatter Fix minor formatting errors document $INCLUDE as obsolete Change documentation of negative error Move test environment and custom helpers down indent copyright and frontmatter sections better description of the async calls between a promise and the functions in its `.then` Correct Early Error example: don't throw a string CONSOLE-RUNNER: split runner doc into new file add troubleshooting section
2014-07-30 00:27:20 +02:00
update CONTRIBUTING.md to reflect current practices
2017-10-06 22:09:39 +02:00
#### es5id
**es5id**: [es5-test-id]
**Deprecated.**
This tag identifies the section number from the portion of the ECMAScript 5.1 or ECMAScript 3 standard that is tested by this test. It was automatically generated for tests that were originally written for the ES5 (or earlier) version of the test suite and are now part of the ES6 version. You can use the es5id to discover the relevant portion of the ECMAScript standard by looking up the section number in [previous publications of the specification](https://www.ecma-international.org/publications/standards/Ecma-262-arch.htm). Unfortunately, there is no way to identify which version of ECMAScript (specifically, 3 or 5.1) without looking up the section number and deciding whether it covers the observable in the test.
Read the (Test262 Technical Rationale Report)[https://github.com/tc39/test262/wiki/Test262-Technical-Rationale-Report,-October-2017#specification-reference-ids] for reasoning behind deprecation.
#### es6id
**es6id**: [es6-test-id]
**Deprecated.**
This tag identifies the section number from the portion of the ES6 standard that is tested by this test _at the time the test was written_. The es6ids might not correspond to the correction section numbers in the ES6 (or later) specifcation because routine edits to the specification will change section numbering. For this reason, only the esid is required for new tests.
Read the (Test262 Technical Rationale Report)[https://github.com/tc39/test262/wiki/Test262-Technical-Rationale-Report,-October-2017#specification-reference-ids] for reasoning behind deprecation.
document `timeout` and clean up CONTRIBUTING.md - document `timeout` tag - reorder tags in frontmatter doc - minor cleanups - minor fixes - add style note - reformat flags - remove discussion of obsolete $INCLUDE - incorporate line notes from @domenic - integrate additional comments - add links back, move arg notes down - Raise outline level by one README.md - link to CONTRIBUTING
2014-08-22 17:47:18 +02:00
## Test Environment
CONTRIBUTING: more notes on test writing Add section on test environment Add section on custom helpers describe YAML frontmatter Fix minor formatting errors document $INCLUDE as obsolete Change documentation of negative error Move test environment and custom helpers down indent copyright and frontmatter sections better description of the async calls between a promise and the functions in its `.then` Correct Early Error example: don't throw a string CONSOLE-RUNNER: split runner doc into new file add troubleshooting section
2014-07-30 00:27:20 +02:00
Each test case is run in a fresh JavaScript environment; in a browser, this will be a new `IFRAME`; for a console runner, this will be a new process. The test harness code is loaded before the test is run. The test harness defines the following helper functions:
Function | Purpose
---------|--------
Test262Error(message) | constructor for an error object that indicates a test failure
Update contribution guidelines - Remove trailing white space - Streamline documentation of test tags - Do not reference obsolete tags - Document `features` frontmatter tag - Document `es6id` frontmatter tag - Omit unnecessary detail about test262 website generation. This is not directly useful to potential test contributors. Implementation details like these can be taken for granted by that audience. - Remove documentation on YAML syntax. Details on YAML may be helpful for some new contributors, but this document should not attempt to cover the topic (especially not from the description of a specific frontmatter entry). Replace with a link to a more comprehensive source as this will be more generally useful to those who need it (and less obtrusive for those who do not). - Consolidate information on test helpers - Document `assert` helpers - Update instructions for asserting errors. Since the introduction of `assert.throws` in gh-22, the preferred means of expressing expectations regarding errors has changed. Update the CONTRIBUTING.md file to reflect the latest approach. Explain purpose of `throw NotEarlyError;` in example test. - Re-order information on file names. The inconsistency in the project's file names should not go unmentioned, but neither should it not preceed instructions for the accepted approach to namine tests. - More clearly document required frontmatter tags. Explicitly list `description` as a required frontmatter tag, implicitly identifying all other tags as optional.
2015-05-25 23:51:55 +02:00
$DONE(arg) | see Writing Asynchronous Tests, below
assert(value, message) | throw a new Test262Error instance if the specified value is not strictly equal to the JavaScript `true` value; accepts an optional string message for use in creating the error
CONTRIBUTING: Link to the latest spec draft (#978)
2017-04-14 18:19:39 +02:00
assert.sameValue(actual, expected, message) | throw a new Test262Error instance if the first two arguments are not [the same value](https://tc39.github.io/ecma262/#sec-samevalue); accepts an optional string message for use in creating the error
assert.notSameValue(actual, unexpected, message) | throw a new Test262Error instance if the first two arguments are [the same value](https://tc39.github.io/ecma262/#sec-samevalue); accepts an optional string message for use in creating the error
Add assert.throws.early
2017-05-01 18:29:52 +02:00
assert.throws(expectedErrorConstructor, fn, message) | throw a new Test262Error instance if the provided function does not throw an error, or if the constructor of the value thrown does not match the provided constructor
CONTRIBUTING: Fix typo (#1013)
2017-05-01 19:22:49 +02:00
assert.throws.early(expectedErrorConstructor, code) | throw a new Test262Error instance if the provided code does not throw an early error, or if the constructor of the value thrown does not match the provided constructor. This assertion catches only errors that will be parsed through `Function(code)`.
update CONTRIBUTING.md to reflect current practices
2017-10-06 22:09:39 +02:00
$ERROR(message) | construct a Test262Error object and throw it <br>**DEPRECATED** -- Do not use in new tests. Use `assert`, `assert.*`, or `throw new Test252Error` instead.
Update contribution guidelines - Remove trailing white space - Streamline documentation of test tags - Do not reference obsolete tags - Document `features` frontmatter tag - Document `es6id` frontmatter tag - Omit unnecessary detail about test262 website generation. This is not directly useful to potential test contributors. Implementation details like these can be taken for granted by that audience. - Remove documentation on YAML syntax. Details on YAML may be helpful for some new contributors, but this document should not attempt to cover the topic (especially not from the description of a specific frontmatter entry). Replace with a link to a more comprehensive source as this will be more generally useful to those who need it (and less obtrusive for those who do not). - Consolidate information on test helpers - Document `assert` helpers - Update instructions for asserting errors. Since the introduction of `assert.throws` in gh-22, the preferred means of expressing expectations regarding errors has changed. Update the CONTRIBUTING.md file to reflect the latest approach. Explain purpose of `throw NotEarlyError;` in example test. - Re-order information on file names. The inconsistency in the project's file names should not go unmentioned, but neither should it not preceed instructions for the accepted approach to namine tests. - More clearly document required frontmatter tags. Explicitly list `description` as a required frontmatter tag, implicitly identifying all other tags as optional.
2015-05-25 23:51:55 +02:00
CONTRIBUTING: more notes on test writing Add section on test environment Add section on custom helpers describe YAML frontmatter Fix minor formatting errors document $INCLUDE as obsolete Change documentation of negative error Move test environment and custom helpers down indent copyright and frontmatter sections better description of the async calls between a promise and the functions in its `.then` Correct Early Error example: don't throw a string CONSOLE-RUNNER: split runner doc into new file add troubleshooting section
2014-07-30 00:27:20 +02:00
```
/// error class
function Test262Error(message) {
//[omitted body]
}
```
document `timeout` and clean up CONTRIBUTING.md - document `timeout` tag - reorder tags in frontmatter doc - minor cleanups - minor fixes - add style note - reformat flags - remove discussion of obsolete $INCLUDE - incorporate line notes from @domenic - integrate additional comments - add links back, move arg notes down - Raise outline level by one README.md - link to CONTRIBUTING
2014-08-22 17:47:18 +02:00
## Handling Errors and Negative Test Cases
CONTRIBUTING: more notes on test writing Add section on test environment Add section on custom helpers describe YAML frontmatter Fix minor formatting errors document $INCLUDE as obsolete Change documentation of negative error Move test environment and custom helpers down indent copyright and frontmatter sections better description of the async calls between a promise and the functions in its `.then` Correct Early Error example: don't throw a string CONSOLE-RUNNER: split runner doc into new file add troubleshooting section
2014-07-30 00:27:20 +02:00
Enforce use of `throw` stmt in early error tests Previously, test consumers were encouraged to insert a `throw` statement as the first statement of tests for early errors. This recommendation made tests harder to consume, and as an optional transformation, consumers may have ignored it or simply been unaware it was made. By explicitly including such a `throw` statement, the tests become more literal, making them easier to consume and more transparent in their expectations. Document expectation for all tests for early errors to include an explicit `throw` statement. Extend linting script to verify that contributors are automatically notified of violations and to ensure that future contributions satisfy this expectation.
2017-04-29 21:33:06 +02:00
Expectations for **parsing errors** should be declared using [the `negative` frontmatter flag](#negative). They must also include the exact `throw` statement specified in this example (in order to guarantee that implementations do not execute the code):
CONTRIBUTING: more notes on test writing Add section on test environment Add section on custom helpers describe YAML frontmatter Fix minor formatting errors document $INCLUDE as obsolete Change documentation of negative error Move test environment and custom helpers down indent copyright and frontmatter sections better description of the async calls between a promise and the functions in its `.then` Correct Early Error example: don't throw a string CONSOLE-RUNNER: split runner doc into new file add troubleshooting section
2014-07-30 00:27:20 +02:00
Added contributing.md and improved README
2014-07-04 02:13:42 +02:00
```javascript
CONTRIBUTING: more notes on test writing Add section on test environment Add section on custom helpers describe YAML frontmatter Fix minor formatting errors document $INCLUDE as obsolete Change documentation of negative error Move test environment and custom helpers down indent copyright and frontmatter sections better description of the async calls between a promise and the functions in its `.then` Correct Early Error example: don't throw a string CONSOLE-RUNNER: split runner doc into new file add troubleshooting section
2014-07-30 00:27:20 +02:00
/*---
Update documentation
2016-03-13 18:26:10 +01:00
negative:
phase: early
type: SyntaxError
CONTRIBUTING: more notes on test writing Add section on test environment Add section on custom helpers describe YAML frontmatter Fix minor formatting errors document $INCLUDE as obsolete Change documentation of negative error Move test environment and custom helpers down indent copyright and frontmatter sections better description of the async calls between a promise and the functions in its `.then` Correct Early Error example: don't throw a string CONSOLE-RUNNER: split runner doc into new file add troubleshooting section
2014-07-30 00:27:20 +02:00
---*/
Added contributing.md and improved README
2014-07-04 02:13:42 +02:00
Enforce use of `throw` stmt in early error tests Previously, test consumers were encouraged to insert a `throw` statement as the first statement of tests for early errors. This recommendation made tests harder to consume, and as an optional transformation, consumers may have ignored it or simply been unaware it was made. By explicitly including such a `throw` statement, the tests become more literal, making them easier to consume and more transparent in their expectations. Document expectation for all tests for early errors to include an explicit `throw` statement. Extend linting script to verify that contributors are automatically notified of violations and to ensure that future contributions satisfy this expectation.
2017-04-29 21:33:06 +02:00
throw "Test262: This statement should not be evaluated.";
Update contribution guidelines - Remove trailing white space - Streamline documentation of test tags - Do not reference obsolete tags - Document `features` frontmatter tag - Document `es6id` frontmatter tag - Omit unnecessary detail about test262 website generation. This is not directly useful to potential test contributors. Implementation details like these can be taken for granted by that audience. - Remove documentation on YAML syntax. Details on YAML may be helpful for some new contributors, but this document should not attempt to cover the topic (especially not from the description of a specific frontmatter entry). Replace with a link to a more comprehensive source as this will be more generally useful to those who need it (and less obtrusive for those who do not). - Consolidate information on test helpers - Document `assert` helpers - Update instructions for asserting errors. Since the introduction of `assert.throws` in gh-22, the preferred means of expressing expectations regarding errors has changed. Update the CONTRIBUTING.md file to reflect the latest approach. Explain purpose of `throw NotEarlyError;` in example test. - Re-order information on file names. The inconsistency in the project's file names should not go unmentioned, but neither should it not preceed instructions for the accepted approach to namine tests. - More clearly document required frontmatter tags. Explicitly list `description` as a required frontmatter tag, implicitly identifying all other tags as optional.
2015-05-25 23:51:55 +02:00
var var = var;
Added contributing.md and improved README
2014-07-04 02:13:42 +02:00
```
Update contribution guidelines - Remove trailing white space - Streamline documentation of test tags - Do not reference obsolete tags - Document `features` frontmatter tag - Document `es6id` frontmatter tag - Omit unnecessary detail about test262 website generation. This is not directly useful to potential test contributors. Implementation details like these can be taken for granted by that audience. - Remove documentation on YAML syntax. Details on YAML may be helpful for some new contributors, but this document should not attempt to cover the topic (especially not from the description of a specific frontmatter entry). Replace with a link to a more comprehensive source as this will be more generally useful to those who need it (and less obtrusive for those who do not). - Consolidate information on test helpers - Document `assert` helpers - Update instructions for asserting errors. Since the introduction of `assert.throws` in gh-22, the preferred means of expressing expectations regarding errors has changed. Update the CONTRIBUTING.md file to reflect the latest approach. Explain purpose of `throw NotEarlyError;` in example test. - Re-order information on file names. The inconsistency in the project's file names should not go unmentioned, but neither should it not preceed instructions for the accepted approach to namine tests. - More clearly document required frontmatter tags. Explicitly list `description` as a required frontmatter tag, implicitly identifying all other tags as optional.
2015-05-25 23:51:55 +02:00
Expectations for **runtime errors** should be defined using the `assert.throws` method and the appropriate JavaScript Error constructor function:
document `timeout` and clean up CONTRIBUTING.md - document `timeout` tag - reorder tags in frontmatter doc - minor cleanups - minor fixes - add style note - reformat flags - remove discussion of obsolete $INCLUDE - incorporate line notes from @domenic - integrate additional comments - add links back, move arg notes down - Raise outline level by one README.md - link to CONTRIBUTING
2014-08-22 17:47:18 +02:00
Added contributing.md and improved README
2014-07-04 02:13:42 +02:00
```javascript
Correct example in contribution guidelines As written, the example for asserting runtime errors is written with an early error. Because the error is expected to be reported prior to program execution, the `assert.throws` function cannot be used to detect it. Demonstrate the usage of the helper function with a runtime error.
2016-03-22 14:51:05 +01:00
assert.throws(TypeError, function() {
null(); // expect this statement to throw a TypeError
Update contribution guidelines - Remove trailing white space - Streamline documentation of test tags - Do not reference obsolete tags - Document `features` frontmatter tag - Document `es6id` frontmatter tag - Omit unnecessary detail about test262 website generation. This is not directly useful to potential test contributors. Implementation details like these can be taken for granted by that audience. - Remove documentation on YAML syntax. Details on YAML may be helpful for some new contributors, but this document should not attempt to cover the topic (especially not from the description of a specific frontmatter entry). Replace with a link to a more comprehensive source as this will be more generally useful to those who need it (and less obtrusive for those who do not). - Consolidate information on test helpers - Document `assert` helpers - Update instructions for asserting errors. Since the introduction of `assert.throws` in gh-22, the preferred means of expressing expectations regarding errors has changed. Update the CONTRIBUTING.md file to reflect the latest approach. Explain purpose of `throw NotEarlyError;` in example test. - Re-order information on file names. The inconsistency in the project's file names should not go unmentioned, but neither should it not preceed instructions for the accepted approach to namine tests. - More clearly document required frontmatter tags. Explicitly list `description` as a required frontmatter tag, implicitly identifying all other tags as optional.
2015-05-25 23:51:55 +02:00
});
Added contributing.md and improved README
2014-07-04 02:13:42 +02:00
```
CONTRIBUTING: document test262.py, async tests add a section on requirements for using console test runner test262.py add a section on asynchronous tests add subsection to authoring guidelines with suggestion for test names
2014-07-21 17:39:10 +02:00
document `timeout` and clean up CONTRIBUTING.md - document `timeout` tag - reorder tags in frontmatter doc - minor cleanups - minor fixes - add style note - reformat flags - remove discussion of obsolete $INCLUDE - incorporate line notes from @domenic - integrate additional comments - add links back, move arg notes down - Raise outline level by one README.md - link to CONTRIBUTING
2014-08-22 17:47:18 +02:00
## Writing Asynchronous Tests
CONTRIBUTING: document test262.py, async tests add a section on requirements for using console test runner test262.py add a section on asynchronous tests add subsection to authoring guidelines with suggestion for test names
2014-07-21 17:39:10 +02:00
Make asynchronous test configuration explicit For asynchronous tests, the contract between test file and test runner is implicit: runners are expected to inspect the source code for references to a global `$DONE` identifier. Promote a more explicit contract between test file and test runner by introducing a new frontmatter "tag", `async`. This brings asynchronous test configuration in-line with other configuration mechanisms and also provides a more natural means of test filtering. The modifications to test files was made programatically using the `grep` and `sed` utilities: $ grep "\$DONE" test/ -r --files-with-match --null | \ xargs -0 sed -i 's/^\(flags:\s*\)\[/\1[async, /g' $ grep "\$DONE" test/ -rl --null | \ xargs -0 grep -E '^flags:' --files-without-match --null | \ xargs -0 sed -i 's/^---\*\//flags: [async]\n---*\//'
2016-02-12 18:59:51 +01:00
An asynchronous test is any test that include the `async` frontmatter flag. When executing such tests, the runner expects that the global `$DONE()` function will be called to signal test completion.
CONTRIBUTING: document test262.py, async tests add a section on requirements for using console test runner test262.py add a section on asynchronous tests add subsection to authoring guidelines with suggestion for test names
2014-07-21 17:39:10 +02:00
* If the argument to `$DONE` is omitted, is `undefined`, or is any other falsy value, the test is considered to have passed.
* If the argument to `$DONE` is a truthy value, the test is considered to have failed and the argument is displayed as the failure reason.
A common idiom when writing asynchronous tests is the following:
```js
var p = new Promise(function () { /* some test code */ });
p.then(function checkAssertions(arg) {
CONTRIBUTING: Link to the latest spec draft (#978)
2017-04-14 18:19:39 +02:00
if (!expected_condition) {
update CONTRIBUTING.md to reflect current practices
2017-10-06 22:09:39 +02:00
throw new Test262Error("failure message");
CONTRIBUTING: Link to the latest spec draft (#978)
2017-04-14 18:19:39 +02:00
}
CONTRIBUTING: document test262.py, async tests add a section on requirements for using console test runner test262.py add a section on asynchronous tests add subsection to authoring guidelines with suggestion for test names
2014-07-21 17:39:10 +02:00
}).then($DONE, $DONE);
```
CONTRIBUTING: more notes on test writing Add section on test environment Add section on custom helpers describe YAML frontmatter Fix minor formatting errors document $INCLUDE as obsolete Change documentation of negative error Move test environment and custom helpers down indent copyright and frontmatter sections better description of the async calls between a promise and the functions in its `.then` Correct Early Error example: don't throw a string CONSOLE-RUNNER: split runner doc into new file add troubleshooting section
2014-07-30 00:27:20 +02:00
Function `checkAssertions` implicitly returns `undefined` if the expected condition is observed. The return value of function `checkAssertions` is then used to asynchronously invoke the first function of the final `then` call, resulting in a call to `$DONE(undefined)`, which signals a passing test.
CONTRIBUTING: document test262.py, async tests add a section on requirements for using console test runner test262.py add a section on asynchronous tests add subsection to authoring guidelines with suggestion for test names
2014-07-21 17:39:10 +02:00
update CONTRIBUTING.md to reflect current practices
2017-10-06 22:09:39 +02:00
If the expected condition is not observed, function `checkAssertions` throws a `Test262Error`. This is caught by the Promise and then used to asynchronously invoke the second function in the call -- which is also `$DONE` -- resulting in a call to `$DONE(error_object)`, which signals a failing test.
CONTRIBUTING: document test262.py, async tests add a section on requirements for using console test runner test262.py add a section on asynchronous tests add subsection to authoring guidelines with suggestion for test names
2014-07-21 17:39:10 +02:00
document `timeout` and clean up CONTRIBUTING.md - document `timeout` tag - reorder tags in frontmatter doc - minor cleanups - minor fixes - add style note - reformat flags - remove discussion of obsolete $INCLUDE - incorporate line notes from @domenic - integrate additional comments - add links back, move arg notes down - Raise outline level by one README.md - link to CONTRIBUTING
2014-08-22 17:47:18 +02:00
### Checking Exception Type and Message in Asynchronous Tests
CONTRIBUTING: document test262.py, async tests add a section on requirements for using console test runner test262.py add a section on asynchronous tests add subsection to authoring guidelines with suggestion for test names
2014-07-21 17:39:10 +02:00
This idiom can be extended to check for specific exception types or messages:
```js
p.then(function () {
CONTRIBUTING: Link to the latest spec draft (#978)
2017-04-14 18:19:39 +02:00
// some code that is expected to throw a TypeError
CONTRIBUTING: document test262.py, async tests add a section on requirements for using console test runner test262.py add a section on asynchronous tests add subsection to authoring guidelines with suggestion for test names
2014-07-21 17:39:10 +02:00
CONTRIBUTING: Link to the latest spec draft (#978)
2017-04-14 18:19:39 +02:00
return "Expected exception to be thrown";
CONTRIBUTING: document test262.py, async tests add a section on requirements for using console test runner test262.py add a section on asynchronous tests add subsection to authoring guidelines with suggestion for test names
2014-07-21 17:39:10 +02:00
}).then($DONE, function (e) {
CONTRIBUTING: Link to the latest spec draft (#978)
2017-04-14 18:19:39 +02:00
if (!(e instanceof TypeError)) {
update CONTRIBUTING.md to reflect current practices
2017-10-06 22:09:39 +02:00
throw new Test262Error("Expected TypeError but got " + e);
CONTRIBUTING: Link to the latest spec draft (#978)
2017-04-14 18:19:39 +02:00
}
CONTRIBUTING: document test262.py, async tests add a section on requirements for using console test runner test262.py add a section on asynchronous tests add subsection to authoring guidelines with suggestion for test names
2014-07-21 17:39:10 +02:00
CONTRIBUTING: Link to the latest spec draft (#978)
2017-04-14 18:19:39 +02:00
if (!/expected message/.test(e.message)) {
update CONTRIBUTING.md to reflect current practices
2017-10-06 22:09:39 +02:00
throw new Test262Error("Expected message to contain 'expected message' but found " + e.message);
CONTRIBUTING: Link to the latest spec draft (#978)
2017-04-14 18:19:39 +02:00
}
CONTRIBUTING: document test262.py, async tests add a section on requirements for using console test runner test262.py add a section on asynchronous tests add subsection to authoring guidelines with suggestion for test names
2014-07-21 17:39:10 +02:00
}).then($DONE, $DONE);
```
Update contribution guidelines - Remove trailing white space - Streamline documentation of test tags - Do not reference obsolete tags - Document `features` frontmatter tag - Document `es6id` frontmatter tag - Omit unnecessary detail about test262 website generation. This is not directly useful to potential test contributors. Implementation details like these can be taken for granted by that audience. - Remove documentation on YAML syntax. Details on YAML may be helpful for some new contributors, but this document should not attempt to cover the topic (especially not from the description of a specific frontmatter entry). Replace with a link to a more comprehensive source as this will be more generally useful to those who need it (and less obtrusive for those who do not). - Consolidate information on test helpers - Document `assert` helpers - Update instructions for asserting errors. Since the introduction of `assert.throws` in gh-22, the preferred means of expressing expectations regarding errors has changed. Update the CONTRIBUTING.md file to reflect the latest approach. Explain purpose of `throw NotEarlyError;` in example test. - Re-order information on file names. The inconsistency in the project's file names should not go unmentioned, but neither should it not preceed instructions for the accepted approach to namine tests. - More clearly document required frontmatter tags. Explicitly list `description` as a required frontmatter tag, implicitly identifying all other tags as optional.
2015-05-25 23:51:55 +02:00
As above, exceptions that are thrown from a `then` clause are passed to a later `$DONE` function and reported asynchronously.
Add initial set of generated test sources
2016-03-16 18:58:29 +01:00
Introduce automated validation for test format (#994) This script is intended to identify common test file formatting errors prior to their acceptance into the project. It is designed to support future extensions for additional validation rules.
2017-05-01 18:04:05 +02:00
## Linting
Some of the expectations documented here are enforced via a "linting" script. This script is used to validate patches automatically at submission time, but it may also be invoked locally via the following command:
python tools/lint/lint.py --whitelist lint.whitelist [paths to tests]
...where `[paths to tests]` is a list of one or more paths to test files or directories containing test files.
In some cases, it may be necessary for a test to intentionally violate the rules enforced by the linting tool. Such violations can be allowed by including the path of the test(s) in the `lint.whitelist` file. Each path must appear on a dedicated line in that file, and a space-separated list of rules to ignore must follow each path. Lines beginning with the pound sign (`#`) will be ignored. For example:
# This file documents authorship information and is not itself a test
test/built-ins/Simd/AUTHORS FRONTMATTER LICENSE
Add initial set of generated test sources
2016-03-16 18:58:29 +01:00
## Procedurally-generated tests
Some language features are expressed through a number of distinct syntactic forms. Test262 maintains these tests as a set of "test cases" and "test templates" in order to ensure equivalent coverage across all forms. The sub-directories within the `src/` directory describe the various language features that benefit from this approach.
Test cases and test templates specify meta-data using the same YAML frontmatter pattern as so-called "static" (i.e. non-generated) tests. The expected attributes differ between test cases and test templates:
Fix generate test explination
2017-10-17 18:57:44 +02:00
### test cases (`*.case`)
Field | Description
------|-------------
`template` | name of the sub-directory to locate templates for this test
`desc` | see the frontmatter definition of the "desc" field. The generated test will have a have final "desc" value which is this text appended with the test template's "name" field in parentheses.
`info` | see the frontmatter definition of the "info" field. The generated test will have a have final "info" value which is this text appended with the test template's "info" text.
`features` | see the frontmatter definition of the "features" field. The generated test will have a final feature list in combination with the template's feature field.
### test templates (`*.template`)
Field | Description
------|-------------
`path` | location within the published test hierarchy to output files created from this template. This path will be ended with the name of the test case file. If path is "/test/language/template1-" and the test case is "cast1.js", the final location of the file will be "/test/language/template1-case1.js"
`name` | human-readable name of the syntactic form described by this template. This text will be appended, in parentheses, to the end of the test cases `desc` field.
`esid` | see the frontmatter definition of the "info" tag.
`info` | see the frontmatter definition of the "info" tag. The generated test will have a have final "info" value which is this text concatenated at the end of the test case's "info" text.
`features` | see the frontmatter definition of the "features" field. The generated test will have a final feature list in combination with the test case's feature field.
any other valid frontmatter field | see the frontmatter definitions.
Add initial set of generated test sources
2016-03-16 18:58:29 +01:00
Generated files are managed using the `make.py` Python script located in the root of this repository.
To create files:
make.py
To remove all generated files:
make.py clean
The executable located at `tools/generation/generator.py` offers additional control over the generation procedure.
./tools/generation/generator.py --help
Automate deployment (#546) Introduce configuration to allow build servers provided by the Travis CI service to execute the test generation tool and commit the resultant files to the canonical upstream repository. Enabling this workflow required additional administrative work: 1. Create an account with TravisCI 2. Install the `travis` command-line utility 3. Create a "deploy key" and an encrypted version using the command `./make.py github_deploy_key_enc` 4. Register the deploy key with the project's GitHub account 5. Check the encrypted deploy key to the repository 6. Configure the TravisCI service to automatically build this project
2017-04-10 21:31:19 +02:00
Tests expressed with this convention are built automatically following the source files' acceptance into the project. Patches should **not** include assets built from these sources.