The console test runner is used to test browserless implementations of ECMAScript, e.g., [v8](http://en.wikipedia.org/wiki/V8_(JavaScript_engine), [node](http://en.wikipedia.org/wiki/Node.js), or [js24](http://packages.ubuntu.com/trusty/libmozjs-24-bin)
### Requirements
To use the `test262.py` runner, you must have the following:
* a checkout of the [test262 project](https://github.com/tc39/test262/)
* Python 2.7
* the Python YAML library [PyYAML](http://www.pyyaml.org)
* the javascript engine you intend to test (node, v8, etc.)
### Quick Start
To confirm the console test runner is working on a UNIX-like system
--command=COMMAND | **required** command which invokes javascript engine to be tested
--tests=TESTS | path to the test suite; default is current directory
--cat | don't execute tests, just print code that would be run
--summary | generate a summary at end of execution
--full-summary | generate a longer summary with details of test failures
--strict_only | run only tests that are marked **@onlyStrict**
--non_strict_only | run only tests that are marked **@noStrict**
--unmarked_default=MODE | mode to use for tests that are not marked **@onlyStrict** or **@noStrict** ; MODE can be `strict` or `non_strict` or `both`
--logname=LOGNAME | write output to file (in addition to stdout)
--junitname=JUNITNAME | write test results to file in JUnit XML format
--loglevel=LOGLEVEL | set log level, primarily useful for debugging `test262.py`
--print-handle=FUNC | enable async test logging via javascript function e.g., `console.log`
### Usage Notes
Non-option arguments are used as filters to match test names. If no filters are found, the whole test suite is run.
Example | Result
-|-
test262.py --command="node" | run all tests
test262.py --command="node" ch07 ch11 | run tests from chapters 7 and 11
test262.py --command="node" 4.4 | run all tests with "4.4" in the name
The COMMAND argument can be a quoted string. This is useful when testing ECMAScript6 features in node, because node requires the command-line argument `--harmony` to enable ES6:
Async tests require a 'print' function to be supplied to the test runner. Here are some good defaults:
Engine | Filename | Print Function
-------|----------|---------------
V8/Node | node | console.log
V8/shell | shell | print
SpiderMonkey<sup>1</sup> | js | print
JavaScriptCore<sup>2</sup> | jsc | print
***Notes:***
1. As of 2014-Jul-23, SpiderMonkey does not support Promise in the `js` executable ([see bug 911216](https://bugzilla.mozilla.org/show_bug.cgi?id=911216) )
2. As of 2014-Jul-23, JavaScriptCore does not support Promise in the `jsc` executable
These are guidelines. There is a substantial amount of variation in existing test names.
Test cases are often named by the portion of the spec or draft spec they reference. A test for ECMAScript 5.1 section [8.7.2 PutValue](http://www.ecma-international.org/ecma-262/5.1/#sec-8.7.2) should be placed in the directory `test262/suite/ch08/8.7/8.7.2/` and have a filename which contains the substring `8.7.2`. A test for ECMAScript 6.0 draft section [25.4.4.1 Promise.all](https://people.mozilla.org/~jorendorff/es6-draft.html#sec-promise.all) should be placed in directory `test262/suite/es6/ch25/25.4/25.4.4/25.4.4.1` and have a filename that contains the substring `25.4.4.1`.
One common convention is to break down a section into a number of testable assertions (A) and to test each assertion with one or more test cases (T). For example, section 25.4.4.1 of ES6 could be broken down like this:
* A1 Promise.all is a function of one argument
- A1.1 Promise.all is callable
- A1.2 Promise.all expects 1 argument
* A2 Promise.all([]) is a Promise immediately resolved with []
- A2.1 Promise.all([]) returns a Promise
- A2.2 Promise.all([]) is resolved immediately
- A2.3 Promise.all([]) is resolved with a new empty array
with test case `S25.4.4.1_A1.1_T1.js` being the first test for assertion A1.1, and `S25.4.4.1_A2.3_T3.js` being the third test for assertion A2.3.
##### @negative [.] \(not supported by Python console harness)
Negative means the test will throw an error and given the error there will be a string comparison on the error message. It has an optional second parameter (it's a *regex-dot* if not provided) that will be used to compare. For best practices on how to use negative tag please see Handling Errors and Negative Test Cases.
There are *very* few cases where a syntax error is **not** an early error. In those cases use the Runtime Error pattern but wrap the test code in an eval statement. Be careful, eval code is not global code!
An asynchronous test is any test that includes the string `$DONE` anywhere in the test file. The test runner checks for the presence of this string; if it is found, the runner expects that the `$DONE()` function will be called to signal test completion.
* If the argument to `$DONE` is omitted, is `undefined`, or is any other falsy value, the test is considered to have passed.
* If the argument to `$DONE` is a truthy value, the test is considered to have failed and the argument is displayed as the failure reason.
A common idiom when writing asynchronous tests is the following:
```js
var p = new Promise(function () { /* some test code */ });
p.then(function checkAssertions(arg) {
if (!expected_condition) {
$ERROR("failure message");
}
}).then($DONE, $DONE);
```
Function `checkAssertions` implicitly returns `undefined` if the expected condition is observed. The return value of function `checkAssertions` is then passed to the first function of the final `then` call, resulting in a call to `$DONE(undefined)`, which signals a passing test.
If the expected condition is not observed, function `checkAssertions` throws a `Test262Error` via function $ERROR. This is caught by the Promise and passed to the second function in the call -- which is also `$DONE` -- resulting in a call to `$DONE(error_object)`, which signals
a failing test.
#### Checking Exception Type and Message in Asynchronous Tests
This idiom can be extended to check for specific exception types or messages:
```js
p.then(function () {
// some code that is expected to throw a TypeError
return "Expected exception to be thrown";
}).then($DONE, function (e) {
if (!e instanceof TypeError) {
$ERROR("Expected TypeError but got " + e);
}
if (!/expected message/.test(e.message)) {
$ERROR("Expected message to contain 'expected message' but found " + e.message);
$INCLUDE will load an external Javascript file in the same context before executing a test. In most cases usage of this method should be avoided. It's a good practice only when a large amount of tests need a special check that's not provided by the default harness. If that's the case, please explain the use case in detail.
