mirror of https://github.com/tc39/test262.git
Add documentation on interpreting tests (#508)
The project's CONTRIBUTING.md was written with test authors in mind. It contains details on non-technical metadata (e.g. "author" and "es6id"), helper function usage, and preferred code structure. In addition, it elides certain low-level technical details on the requirements for the runtime environment. Introduce a new document targeted towards those executing the tests. Formalize all expectations regarding how the runtime environment should be defined, how metadata should be interpreted, and how results should be understood. This information has overlap with the CONTRIBUTING.md file, but it also contains details that are irrelevant to test authors. This document can serve as a more formal contract between Test262 and the implementors who consume it. This allows Test262 to unambiguously document future modifications to the formal requirements which in turn supports consumers who maintain their own test harnesses.
This commit is contained in:
parent
20d37a6f2f
commit
a657b64ae4
|
@ -0,0 +1,191 @@
|
|||
# Interpreting Test262 Tests
|
||||
|
||||
All tests are declared as text files located within this project's `test`
|
||||
directory. In order to execute Test262 tests, runtimes must observe the
|
||||
following semantics.
|
||||
|
||||
## Test Execution
|
||||
|
||||
Test262 tests are only valid under the runtime environment conditions described
|
||||
here. Test environments may be further modified according to the metadata
|
||||
contained with each test--refer to the "Metadata" section for more details.
|
||||
|
||||
### Realm Isolation
|
||||
|
||||
Each test must be executed in a new [ECMAScript
|
||||
realm](https://tc39.github.io/ecma262/#sec-code-realms) dedicated to that test.
|
||||
Unless configured otherwise (via the `module` flag), source text must be
|
||||
interpreted as [global
|
||||
code](http://www.ecma-international.org/ecma-262/6.0/#sec-types-of-source-code).
|
||||
|
||||
### Test262-Defined Bindings
|
||||
|
||||
The contents of the following files must be evaluated in the test realm's
|
||||
global scope prior to test execution:
|
||||
|
||||
1. `harness/assert.js`
|
||||
2. `harness/sta.js`
|
||||
|
||||
### Host-Defined Functions
|
||||
|
||||
The following values must be defined as writable, configurable, non-enumerable
|
||||
properties of the global scope prior to test execution.
|
||||
|
||||
- **`print`** A function that exposes the string value of its first argument to
|
||||
the test runner. This is used as a communication mechanism for asynchronous
|
||||
tests (via the `async` flag, described below).
|
||||
|
||||
### Strict Mode
|
||||
|
||||
Unless configured otherwise (via the `noStrict`, `onlyStrict`, `module`, or
|
||||
`raw` flags), each test must be executed twice: once in ECMAScript's non-strict
|
||||
mode, and again in ECMAScript's strict mode. To run in strict mode, the test
|
||||
contents must be modified prior to execution--[a "use strict"
|
||||
directive](https://tc39.github.io/ecma262/#sec-directive-prologues-and-the-use-strict-directive)
|
||||
must be inserted as the initial character sequence of the file, followed by a
|
||||
semicolon (`;`) and newline character (`\n`):
|
||||
|
||||
"use strict";
|
||||
|
||||
This must precede any additional text modifications described by test metadata.
|
||||
|
||||
## Test Results
|
||||
|
||||
By default, tests signal failure by generating an uncaught exception. If
|
||||
execution completes without generating an exception, the test must be
|
||||
interpreted as "passing." Any uncaught exception must be interpreted as test
|
||||
failure. These semantics may be modified by any test according to the metadata
|
||||
declared within the test itself (via the `negative` attribute and the `async`
|
||||
flag, described below).
|
||||
|
||||
## Metadata
|
||||
|
||||
Each test file may define metadata that describe additional requirements. This
|
||||
information is delimited by the token sequence `/*---` and `---*/` and is
|
||||
structured as [YAML](http://yaml.org/).
|
||||
|
||||
### `negative`
|
||||
|
||||
These tests are expected to generate an uncaught exception. The value of this
|
||||
attribute is the name of the constructor of the expected error. If a test
|
||||
configured with the `negative` attribute completes without throwing an
|
||||
exception, or if the name of the thrown exception's constructor does not match
|
||||
the specified constructor name, the test must be interpreted as "failing."
|
||||
|
||||
*Example:*
|
||||
|
||||
```js
|
||||
/*---
|
||||
negative: ReferenceError
|
||||
---*/
|
||||
unresolvable;
|
||||
```
|
||||
|
||||
### `includes`
|
||||
|
||||
One or more files whose content must be evaluated in the test realm's global
|
||||
scope prior to test execution. These files are located within the `harness/`
|
||||
directory of the Test262 project.
|
||||
|
||||
*Example*
|
||||
|
||||
```js
|
||||
/*---
|
||||
includes: [testBuildInObject.js]
|
||||
---*/
|
||||
testBuiltInObject(Number.prototype.toLocaleString, true, false, [], 0);
|
||||
```
|
||||
|
||||
### `flags`
|
||||
|
||||
The `flags` attribute is an optional value that specifies one or more of the
|
||||
following strings:
|
||||
|
||||
- **`onlyStrict`** The test must be executed just once--in strict mode, only.
|
||||
This must be accomplished using the transformation described in the section
|
||||
titled "Strict Mode".
|
||||
|
||||
*Example*
|
||||
|
||||
```js
|
||||
/*---
|
||||
flags: [onlyStrict]
|
||||
---*/
|
||||
var thisVal = null;
|
||||
[null].forEach(function() {
|
||||
thisVal = this;
|
||||
});
|
||||
assert.sameValue(thisVal, undefined);
|
||||
```
|
||||
|
||||
- **`noStrict`** The test must be executed just once--in non-strict mode, only.
|
||||
In other words, the transformation described by the section titled "Strict
|
||||
Mode" must **not** be applied to these tests.
|
||||
|
||||
*Example*
|
||||
|
||||
```js
|
||||
/*---
|
||||
flags: [noStrict]
|
||||
---*/
|
||||
var thisVal = null;
|
||||
[null].forEach(function() {
|
||||
thisVal = this;
|
||||
});
|
||||
assert.notSameValue(thisVal, undefined);
|
||||
assert.sameValue(thisVal, this);
|
||||
```
|
||||
|
||||
- **`module`** The test source code must be interpreted as [module
|
||||
code](http://www.ecma-international.org/ecma-262/6.0/#sec-types-of-source-code).
|
||||
In addition, this flag negates the default requirement to execute the test
|
||||
both in strict mode and in non-strict mode. In other words, the
|
||||
transformation described by the section titled "Strict Mode" must **not** be
|
||||
applied to these tests.
|
||||
|
||||
*Example*
|
||||
|
||||
```js
|
||||
/*---
|
||||
flags: [module]
|
||||
---*/
|
||||
export default function* g() {}
|
||||
```
|
||||
|
||||
- **`raw`** The test source code must not be modified in any way, and the test
|
||||
must be executed just once (in non-strict mode, only).
|
||||
|
||||
*Example*
|
||||
|
||||
```js
|
||||
/*---
|
||||
flags: [raw]
|
||||
---*/
|
||||
'use strict'
|
||||
[0]
|
||||
's'.p = null;
|
||||
```
|
||||
|
||||
- **`async`** The file `harness/donePrintHandle.js` must be evaluated in the
|
||||
test realm's global scope prior to test execution. The test must not be
|
||||
considered complete until the implementation-defined `print` function has
|
||||
been invoked or some length of time has passed without any such invocation.
|
||||
In the event of a passing test run, this function will be invoked with the
|
||||
string `'Test262:AsyncTestComplete'`. If invoked with any other value, the
|
||||
test must be interpreted as failed. The implementation is free to select an
|
||||
appropriate length of time to wait before considering the test "timed out"
|
||||
and failing.
|
||||
|
||||
*Example*
|
||||
|
||||
```js
|
||||
/*---
|
||||
flags: [async]
|
||||
---*/
|
||||
Promise.resolve()
|
||||
.then(function() {
|
||||
print('Test262:AsyncTestComplete');
|
||||
}, function(reason) {
|
||||
print('Error: ' + reason);
|
||||
});
|
||||
```
|
|
@ -7,3 +7,7 @@ Test262 tests conformance to the continually maintained draft future ECMAScript
|
|||
|
||||
1. Sign the [Test262 CLA](http://tc39.github.io/test262-cla).
|
||||
2. Send a pull request. Please make sure you have one commit per pull request. If you have multiple commits, squash them before sending the pull request.
|
||||
|
||||
### Running Test262
|
||||
|
||||
> See [INTERPRETING.md](./INTERPRETING.md)
|
||||
|
|
Loading…
Reference in New Issue