2016-04-18 20:31:51 +02:00
|
|
|
# Interpreting Test262 Tests
|
|
|
|
|
|
|
|
All tests are declared as text files located within this project's `test`
|
|
|
|
directory. In order to execute Test262 tests, runtimes must observe the
|
|
|
|
following semantics.
|
|
|
|
|
2017-10-02 17:12:14 +02:00
|
|
|
**Note** When these instructions change in any substantive way, the `version`
|
|
|
|
property of the JSON-formatted `package.json` file will be incremented. In this
|
|
|
|
way, consumers who are transitioning between revisions of Test262 can more
|
|
|
|
easily determine the cause of new test failures.
|
|
|
|
|
2016-04-18 20:31:51 +02:00
|
|
|
## Test Execution
|
|
|
|
|
|
|
|
Test262 tests are only valid under the runtime environment conditions described
|
|
|
|
here. Test environments may be further modified according to the metadata
|
|
|
|
contained with each test--refer to the "Metadata" section for more details.
|
|
|
|
|
|
|
|
### Realm Isolation
|
|
|
|
|
|
|
|
Each test must be executed in a new [ECMAScript
|
|
|
|
realm](https://tc39.github.io/ecma262/#sec-code-realms) dedicated to that test.
|
|
|
|
Unless configured otherwise (via the `module` flag), source text must be
|
|
|
|
interpreted as [global
|
2017-04-28 01:38:14 +02:00
|
|
|
code](https://tc39.github.io/ecma262/#sec-types-of-source-code).
|
2016-04-18 20:31:51 +02:00
|
|
|
|
|
|
|
### Test262-Defined Bindings
|
|
|
|
|
|
|
|
The contents of the following files must be evaluated in the test realm's
|
|
|
|
global scope prior to test execution:
|
|
|
|
|
|
|
|
1. `harness/assert.js`
|
|
|
|
2. `harness/sta.js`
|
|
|
|
|
|
|
|
### Host-Defined Functions
|
|
|
|
|
|
|
|
The following values must be defined as writable, configurable, non-enumerable
|
|
|
|
properties of the global scope prior to test execution.
|
|
|
|
|
|
|
|
- **`print`** A function that exposes the string value of its first argument to
|
|
|
|
the test runner. This is used as a communication mechanism for asynchronous
|
|
|
|
tests (via the `async` flag, described below).
|
2017-03-01 22:34:15 +01:00
|
|
|
- **`$262`** An ordinary object with the following properties:
|
2016-06-21 18:11:18 +02:00
|
|
|
- **`createRealm`** - a function which creates a new [ECMAScript
|
2017-04-28 01:38:14 +02:00
|
|
|
Realm](https://tc39.github.io/ecma262/#sec-code-realms),
|
2017-04-20 19:54:02 +02:00
|
|
|
defines this API on the new realm's global object, and returns the `$262`
|
2016-06-21 18:11:18 +02:00
|
|
|
property of the new realm's global object
|
|
|
|
- **`detachArrayBuffer`** - a function which implements [the
|
|
|
|
DetachArrayBuffer abstract
|
2017-04-28 01:38:14 +02:00
|
|
|
operation](https://tc39.github.io/ecma262/#sec-detacharraybuffer)
|
2016-06-21 18:11:18 +02:00
|
|
|
- **`evalScript`** - a function which accepts a string value as its first
|
|
|
|
argument and executes is as [an ECMAScript
|
2017-04-28 01:38:14 +02:00
|
|
|
script](https://tc39.github.io/ecma262/#sec-scripts) according to the
|
2016-06-21 18:11:18 +02:00
|
|
|
following algorithm:
|
|
|
|
|
|
|
|
1. Let hostDefined be any host-defined values for the provided
|
|
|
|
sourceText (obtained in an implementation dependent manner)
|
|
|
|
2. Let realm be the current Realm Record.
|
|
|
|
3. Let s be ParseScript(sourceText, realm, hostDefined).
|
|
|
|
4. If s is a List of errors, then
|
|
|
|
a. Let error be the first element of s.
|
|
|
|
b. Return
|
|
|
|
Completion{[[Type]]: throw, [[Value]]: error, [[Target]]: empty}.
|
|
|
|
5. Let status be ScriptEvaluation(s).
|
|
|
|
6. Return Completion(status).
|
|
|
|
|
2017-03-01 22:34:15 +01:00
|
|
|
- **`global`** - a reference to the global object on which `$262` was initially defined
|
2017-10-19 21:03:21 +02:00
|
|
|
- **`IsHTMLDDA`** - (present only in implementations that can provide it) an
|
|
|
|
object that 1) has an [[IsHTMLDDA]] internal slot, and 2) when called with
|
|
|
|
no arguments or with the single argument `""` returns `null`. Use this
|
|
|
|
property to test that ECMAScript algorithms aren't mis-implemented to treat
|
|
|
|
`document.all` as being `undefined` or of type Undefined (instead of
|
|
|
|
Object). (The peculiar second requirement permits testing algorithms when
|
|
|
|
they also call `document.all` with such arguments, so that testing for
|
|
|
|
correct behavior requires knowing how the call behaves. This is rarely
|
|
|
|
necessary.) Tests using this function must be tagged with the `IsHTMLDDA`
|
|
|
|
feature so that only hosts supporting this property will run them.
|
2017-02-07 17:17:31 +01:00
|
|
|
- **`agent`** - an ordinary object with the following properties:
|
|
|
|
- **`start`** - a function that takes a script source string and runs
|
|
|
|
the script in a concurrent agent. Will block until that agent is
|
|
|
|
running. The agent has no representation. The agent script will be
|
2017-04-20 19:54:02 +02:00
|
|
|
run in an environment that has an object `$262` with a property `agent`
|
2017-02-07 17:17:31 +01:00
|
|
|
with the following properties:
|
2017-04-28 01:38:14 +02:00
|
|
|
- **`receiveBroadcast`** - a function that takes a function and
|
2017-02-07 17:17:31 +01:00
|
|
|
calls the function when it has received a broadcast from the parent,
|
|
|
|
passing it the broadcast as two arguments, a SharedArrayBuffer and
|
|
|
|
an Int32. This function may return before a broadcast is received
|
|
|
|
(eg to return to an event loop to await a message) and no code should
|
|
|
|
follow the call to this function.
|
|
|
|
- **`report`** - a function that takes a string and places it in a
|
|
|
|
transmit queue whence the parent will retrieve it. Messages
|
|
|
|
should be short.
|
|
|
|
- **`sleep`** - a function that takes a millisecond argument and
|
|
|
|
sleeps the agent for approximately that duration.
|
|
|
|
- **`leaving`** - a function that signals that the agent is done and
|
|
|
|
may be terminated (if possible).
|
|
|
|
- **`broadcast`** - a function that takes a SharedArrayBuffer and an Int32
|
|
|
|
and broadcasts the two values to all concurrent agents. The function
|
|
|
|
blocks until all agents have retrieved the message. Note, this assumes
|
|
|
|
that all agents that were started are still running.
|
|
|
|
- **`getReport`** - a function that reads an incoming string from any agent,
|
|
|
|
and returns it if it exists, or returns `null` otherwise.
|
|
|
|
- **`sleep`** - a function that takes a millisecond argument and
|
|
|
|
sleeps the execution for approximately that duration.
|
2016-04-18 20:31:51 +02:00
|
|
|
|
|
|
|
### Strict Mode
|
|
|
|
|
|
|
|
Unless configured otherwise (via the `noStrict`, `onlyStrict`, `module`, or
|
|
|
|
`raw` flags), each test must be executed twice: once in ECMAScript's non-strict
|
|
|
|
mode, and again in ECMAScript's strict mode. To run in strict mode, the test
|
|
|
|
contents must be modified prior to execution--[a "use strict"
|
|
|
|
directive](https://tc39.github.io/ecma262/#sec-directive-prologues-and-the-use-strict-directive)
|
|
|
|
must be inserted as the initial character sequence of the file, followed by a
|
|
|
|
semicolon (`;`) and newline character (`\n`):
|
|
|
|
|
|
|
|
"use strict";
|
|
|
|
|
|
|
|
This must precede any additional text modifications described by test metadata.
|
|
|
|
|
2016-06-21 18:17:51 +02:00
|
|
|
### Modules
|
|
|
|
|
|
|
|
Test262 includes tests for ECMAScript 2015 module code, denoted by the "module"
|
|
|
|
metadata flag. Files bearing a name ending in `_FIXTURE.js` should not be
|
|
|
|
interpreted as standalone tests; they are intended to be referenced by test
|
|
|
|
files.
|
|
|
|
|
|
|
|
All module specifiers used by Test262 begin with the character sequence `./`.
|
|
|
|
The remaining characters should be interpreted as the name of a file within the
|
|
|
|
same directory as the file under test. The contents of this file must be
|
|
|
|
interpreted as UTF-8-encoded text and supplied to the Source Text Module
|
|
|
|
Record's ParseModule abstract operation. The result of that operation must be
|
|
|
|
returned by the implementation-defined HostResolveImportedModule directly.
|
|
|
|
|
|
|
|
For example, consider a test file located at
|
|
|
|
`test/language/import/nested/index.js` with the following contents:
|
|
|
|
|
|
|
|
```js
|
|
|
|
import * as ns from './dep.js';
|
|
|
|
```
|
|
|
|
|
|
|
|
Implementers should attempt to resolve this module specifier by loading a file
|
|
|
|
located at `test/language/import/nested/dep.js`.
|
|
|
|
|
2016-04-18 20:31:51 +02:00
|
|
|
## Test Results
|
|
|
|
|
|
|
|
By default, tests signal failure by generating an uncaught exception. If
|
|
|
|
execution completes without generating an exception, the test must be
|
|
|
|
interpreted as "passing." Any uncaught exception must be interpreted as test
|
|
|
|
failure. These semantics may be modified by any test according to the metadata
|
|
|
|
declared within the test itself (via the `negative` attribute and the `async`
|
|
|
|
flag, described below).
|
|
|
|
|
|
|
|
## Metadata
|
|
|
|
|
|
|
|
Each test file may define metadata that describe additional requirements. This
|
|
|
|
information is delimited by the token sequence `/*---` and `---*/` and is
|
|
|
|
structured as [YAML](http://yaml.org/).
|
|
|
|
|
|
|
|
### `negative`
|
|
|
|
|
|
|
|
These tests are expected to generate an uncaught exception. The value of this
|
2016-03-13 18:26:10 +01:00
|
|
|
attribute is a YAML dictonary with two keys:
|
|
|
|
|
|
|
|
- `phase` - the stage of the test interpretation process that the error is
|
|
|
|
expected to be produced; either "early" (meaning, "prior to evaluation") or
|
2017-04-29 21:33:06 +02:00
|
|
|
"runtime" (meaning, "during evaluation")
|
2016-03-13 18:26:10 +01:00
|
|
|
- `type` - the name of the constructor of the expected error
|
|
|
|
|
|
|
|
If a test configured with the `negative` attribute completes without throwing
|
|
|
|
an exception, or if the name of the thrown exception's constructor does not
|
|
|
|
match the specified constructor name, or if the error occurs at a phase that
|
|
|
|
differs from the indicated phase, the test must be interpreted as "failing."
|
2016-04-18 20:31:51 +02:00
|
|
|
|
|
|
|
*Example:*
|
|
|
|
|
|
|
|
```js
|
|
|
|
/*---
|
2016-03-13 18:26:10 +01:00
|
|
|
negative:
|
|
|
|
phase: runtime
|
|
|
|
type: ReferenceError
|
2016-04-18 20:31:51 +02:00
|
|
|
---*/
|
|
|
|
unresolvable;
|
|
|
|
```
|
|
|
|
|
|
|
|
### `includes`
|
|
|
|
|
|
|
|
One or more files whose content must be evaluated in the test realm's global
|
|
|
|
scope prior to test execution. These files are located within the `harness/`
|
|
|
|
directory of the Test262 project.
|
|
|
|
|
|
|
|
*Example*
|
|
|
|
|
|
|
|
```js
|
|
|
|
/*---
|
|
|
|
includes: [testBuildInObject.js]
|
|
|
|
---*/
|
|
|
|
testBuiltInObject(Number.prototype.toLocaleString, true, false, [], 0);
|
|
|
|
```
|
|
|
|
|
|
|
|
### `flags`
|
|
|
|
|
|
|
|
The `flags` attribute is an optional value that specifies one or more of the
|
|
|
|
following strings:
|
|
|
|
|
|
|
|
- **`onlyStrict`** The test must be executed just once--in strict mode, only.
|
|
|
|
This must be accomplished using the transformation described in the section
|
|
|
|
titled "Strict Mode".
|
|
|
|
|
|
|
|
*Example*
|
|
|
|
|
|
|
|
```js
|
|
|
|
/*---
|
|
|
|
flags: [onlyStrict]
|
|
|
|
---*/
|
|
|
|
var thisVal = null;
|
|
|
|
[null].forEach(function() {
|
|
|
|
thisVal = this;
|
|
|
|
});
|
|
|
|
assert.sameValue(thisVal, undefined);
|
|
|
|
```
|
|
|
|
|
|
|
|
- **`noStrict`** The test must be executed just once--in non-strict mode, only.
|
|
|
|
In other words, the transformation described by the section titled "Strict
|
|
|
|
Mode" must **not** be applied to these tests.
|
|
|
|
|
|
|
|
*Example*
|
|
|
|
|
|
|
|
```js
|
|
|
|
/*---
|
|
|
|
flags: [noStrict]
|
|
|
|
---*/
|
|
|
|
var thisVal = null;
|
|
|
|
[null].forEach(function() {
|
|
|
|
thisVal = this;
|
|
|
|
});
|
|
|
|
assert.notSameValue(thisVal, undefined);
|
|
|
|
assert.sameValue(thisVal, this);
|
|
|
|
```
|
|
|
|
|
|
|
|
- **`module`** The test source code must be interpreted as [module
|
2017-04-28 01:38:14 +02:00
|
|
|
code](https://tc39.github.io/ecma262/#sec-types-of-source-code).
|
2016-04-18 20:31:51 +02:00
|
|
|
In addition, this flag negates the default requirement to execute the test
|
|
|
|
both in strict mode and in non-strict mode. In other words, the
|
|
|
|
transformation described by the section titled "Strict Mode" must **not** be
|
2016-06-21 18:17:51 +02:00
|
|
|
applied to these tests. Refer to the section titled "Modules" for more
|
|
|
|
information on interpreting these tests.
|
2016-04-18 20:31:51 +02:00
|
|
|
|
|
|
|
*Example*
|
|
|
|
|
|
|
|
```js
|
|
|
|
/*---
|
|
|
|
flags: [module]
|
|
|
|
---*/
|
|
|
|
export default function* g() {}
|
|
|
|
```
|
|
|
|
|
|
|
|
- **`raw`** The test source code must not be modified in any way, and the test
|
|
|
|
must be executed just once (in non-strict mode, only).
|
|
|
|
|
|
|
|
*Example*
|
|
|
|
|
|
|
|
```js
|
|
|
|
/*---
|
|
|
|
flags: [raw]
|
|
|
|
---*/
|
|
|
|
'use strict'
|
|
|
|
[0]
|
|
|
|
's'.p = null;
|
|
|
|
```
|
|
|
|
|
2016-11-22 17:46:51 +01:00
|
|
|
- **`async`** The file `harness/doneprintHandle.js` must be evaluated in the
|
2016-04-18 20:31:51 +02:00
|
|
|
test realm's global scope prior to test execution. The test must not be
|
|
|
|
considered complete until the implementation-defined `print` function has
|
|
|
|
been invoked or some length of time has passed without any such invocation.
|
|
|
|
In the event of a passing test run, this function will be invoked with the
|
|
|
|
string `'Test262:AsyncTestComplete'`. If invoked with any other value, the
|
|
|
|
test must be interpreted as failed. The implementation is free to select an
|
|
|
|
appropriate length of time to wait before considering the test "timed out"
|
|
|
|
and failing.
|
|
|
|
|
|
|
|
*Example*
|
|
|
|
|
|
|
|
```js
|
|
|
|
/*---
|
|
|
|
flags: [async]
|
|
|
|
---*/
|
|
|
|
Promise.resolve()
|
|
|
|
.then(function() {
|
|
|
|
print('Test262:AsyncTestComplete');
|
|
|
|
}, function(reason) {
|
|
|
|
print('Error: ' + reason);
|
|
|
|
});
|
|
|
|
```
|
2017-03-06 18:12:42 +01:00
|
|
|
|
|
|
|
- **`generated`** The test file was created procedurally using the project's
|
|
|
|
test generation tool. This flag is specified for informational purposes only
|
|
|
|
and has no bearing on how the test should be interpreted.
|