mirror of https://github.com/tc39/test262.git
Improve documentation of harness files & raw flag
Prior to this commit, the term "harness" was used to refer both to the like-named directory in the repository and the external projects which consume the tests. Reserve the term "harness" to refer solely to the files maintained within this project's `harness/` directory, and use the term "runner" to refer to the external projects. Prior to this commit, the term "helper" was not defined and therefore introduced ambiguity in text that described harness files. Replace the term with more explicit wording in terms of the harness files. Because test runners have some leeway in how they include harness files, the requirement that "[t]he test source code must not be modified in any way" does not necessarily preclude the use of harness files. Update the instructions for interpreting the `raw` frontmatter flag to explicitly disallow harness files.
This commit is contained in:
Mike Pennisi
rwaldron
parent
46f165ae49
commit
74b12b3777
|
|
@ -177,9 +177,9 @@ negative:
|
|||
#### includes
|
||||
`includes: [file-list]`
|
||||
|
||||
This key names a list of helper files that will be included in the test environment prior to running the test. Filenames **must** include the `.js` extension.
|
||||
This key names a list of files in the `harness/` directory that will be included in the test environment prior to running the test. Filenames **must** include the `.js` extension.
|
||||
|
||||
The helper files are found in the `harness/` directory. When some code is used repeatedly across a group of tests, a new helper function (or group of helpers) can be defined. Helpers increase test complexity, so they should be created and used sparingly.
|
||||
When some code is used repeatedly across a group of tests, it may be appropriate to define it in a harness file. This practice increases test complexity, so it should be applied sparingly.
|
||||
|
||||
#### author
|
||||
`author: [string]`
|
||||
|
|
@ -195,8 +195,8 @@ This key is for boolean properties associated with the test.
|
|||
- **noStrict** - only run the test in "sloppy" mode
|
||||
- **module** - interpret the source text as
|
||||
[module code](https://tc39.github.io/ecma262/#sec-modules)
|
||||
- **raw** - execute the test without any modification (no helpers will be
|
||||
available); necessary to test the behavior of directive prologue; implies
|
||||
- **raw** - execute the test without any modification (no harness files will be
|
||||
included); necessary to test the behavior of directive prologue; implies
|
||||
`noStrict`
|
||||
- **async** - defer interpretation of test results until after the invocation
|
||||
of the global `$DONE` function
|
||||
|
|
@ -238,7 +238,9 @@ Read the [Test262 Technical Rationale Report](https://github.com/tc39/test262/wi
|
|||
|
||||
## Test Environment
|
||||
|
||||
Each test case is run in a fresh JavaScript environment; in a browser, this will be a new <iframe>; for a console runner, this will be a new process. The test harness code is loaded before the test is run unless the test file has the `raw` flag. The test harness defines the following helper functions:
|
||||
Each test case is run in a fresh JavaScript environment; in a browser, this will be a new <iframe>; for a console runner, this will be a new process.
|
||||
|
||||
Before the test is executed (and unless the test uses the `raw` frontmatter flag), the test runner will evaluate a number of files in the `harness/` directory. At a minimum, this procedure will define the following functions:
|
||||
|
||||
Function | Purpose
|
||||
---------|--------
|
||||
|
|
@ -281,7 +283,7 @@ $DONOTEVALUATE();
|
|||
var var = var;
|
||||
```
|
||||
|
||||
If the test case has the `raw` flag, this disallows the test to load any harness file including `$DONOTEVALUATE`. In this case, include a direct `throw "Test262: This statement should not be evaluated.";` statement:
|
||||
If the test case has the `raw` flag, the test runner will not load any harness file including the file which defines `$DONOTEVALUATE`. In this case, include a direct `throw "Test262: This statement should not be evaluated.";` statement:
|
||||
|
||||
```javascript
|
||||
/*---
|
||||
|
|
|
|||
|
|
@ -26,8 +26,9 @@ code](https://tc39.github.io/ecma262/#sec-types-of-source-code).
|
|||
|
||||
### Test262-Defined Bindings
|
||||
|
||||
The contents of the following files must be evaluated in the test realm's
|
||||
global scope prior to test execution:
|
||||
The contents of the following harness files must be evaluated in the test
|
||||
realm's global scope prior to test execution unless the test uses the `raw`
|
||||
frontmatter flag:
|
||||
|
||||
1. `harness/assert.js`
|
||||
2. `harness/sta.js`
|
||||
|
|
@ -107,7 +108,7 @@ properties of the global scope prior to test execution.
|
|||
sleeps the execution for approximately that duration.
|
||||
- **`monotonicNow`** - a function that returns a value that conforms to [`DOMHighResTimeStamp`][] and is produced in such a way that its semantics conform to **[Monotonic Clock][]**.
|
||||
|
||||
In addition, consumers may choose to override any of the [the available test harness helper functions](https://github.com/tc39/test262/blob/HEAD/CONTRIBUTING.md#test-environment) as they see fit. See [the documentation on handling errors and negative test cases](https://github.com/tc39/test262/blob/HEAD/CONTRIBUTING.md#handling-errors-and-negative-test-cases) for a useful example of this.
|
||||
In addition, consumers may choose to override any of [the functions defined by test harness files](https://github.com/tc39/test262/blob/HEAD/CONTRIBUTING.md#test-environment) as they see fit. See [the documentation on handling errors and negative test cases](https://github.com/tc39/test262/blob/HEAD/CONTRIBUTING.md#handling-errors-and-negative-test-cases) for a useful example of this.
|
||||
|
||||
|
||||
#### Normative references
|
||||
|
|
@ -315,8 +316,9 @@ following strings:
|
|||
export default function* g() {}
|
||||
```
|
||||
|
||||
- **`raw`** The test source code must not be modified in any way, and the test
|
||||
must be executed just once (in non-strict mode, only).
|
||||
- **`raw`** The test source code must not be modified in any way, files from
|
||||
the `harness/` directory must not be evaluated, and the test must be executed
|
||||
just once (in non-strict mode, only).
|
||||
|
||||
*Example*
|
||||
|
||||
|
|
|
|||
|
|
@ -32,10 +32,10 @@ Guidance for contributing to Test262 can be found in [CONTRIBUTING.md](./CONTRIB
|
|||
|
||||
### Running Test262
|
||||
|
||||
Guidance for running Test262 and explanations of how a test file is interpreted by a test runner is in [INTERPRETING](./INTERPRETING.md)
|
||||
Guidance for running Test262 and explanations of how a test file must be interpreted by a test runner is in [INTERPRETING](./INTERPRETING.md)
|
||||
|
||||
|
||||
### Test262 Runners & Harnesses
|
||||
### Test262 Runners
|
||||
|
||||
Volunteer-maintained projects that may be used to execute Test262 in various ECMAScript hosts:
|
||||
|
||||
|
|
|
|||
Loading…
Reference in New Issue