Commit Graph

37 Commits

Author SHA1 Message Date
Claude Pache ec6239fc65
typo (#2512) 2020-02-26 16:59:26 -08:00
Leo Balter a0e3f37134 Documentation 2019-09-23 12:26:02 -04:00
Leo Balter 7134634aa4 Revert "Remove tests based on the $262.gc hook"
This reverts commit d6d37f2f43.
2019-09-23 12:26:02 -04:00
Leo Balter d6d37f2f43 Remove tests based on the $262.gc hook
Fixes #2239
Closes #2329
2019-09-05 19:38:03 +01:00
Rick Waldron b223ac4e1f INTERPRETING.md: Add $262.gc() and "host-gc-required" feature (#2208) 2019-06-21 11:18:10 -04:00
Ross Kirsling 1859ec66f4 Test update for letting all early errors be SyntaxErrors 2019-06-12 14:09:03 -04:00
Rick Waldron cfe59eec64 INTERPRETING.md/CONTRIBUTING.md: revised rules for _FIXTURE.js files (#2135) 2019-04-24 10:59:54 -04:00
Rick Waldron ba013cf7fc INTERPRETING.md: explicitly disallow use of Test262-Defined Bindings and Host-Defined Functions in _FIXTURE.js files (#2130) 2019-04-17 15:44:42 -04:00
Rick Waldron e4883091b9 Introduce $262.agent.safeBroadcast and migrate Atomics tests.
Migrating all tests to this API is necessary to prevent tests from hanging indefinitely when a SAB is sent to a worker but the code in the worker attempts to create a non-sharable TypedArray (something that is not Int32Array or BigInt64Array). When that scenario occurs, an exception is thrown and the agent worker can no longer communicate with any other threads that control the SAB. If the main thread happens to be spinning in the $262.agent.waitUntil() while loop, it will never meet its termination condition and the test will hang indefinitely.

Because we've defined $262.agent.broadcast(SAB) in https://github.com/tc39/test262/blob/master/INTERPRETING.md, there are host implementations that assume compatibility, which must be maintained.
2018-11-20 15:17:47 -05:00
Mathias Bynens c59f671bda Update {CONTRIBUTING,INTERPRETING}.md per @jugglinmike's feedback 2018-10-23 13:51:17 +02:00
Mathias Bynens b9daa57dcb Rename to $DONOTEVALUATE per @leobalter's suggestion 2018-10-23 13:51:17 +02:00
Mathias Bynens e3feffb01e Move `throw "…"` for negative parsing errors to a helper function
Closes #1634.
2018-10-23 13:51:16 +02:00
Rick Waldron aba414caad Documentation: $262.agent.report() is expected to convert its argument to a string.
Fixes #1640
Closes #1652
2018-07-30 11:50:14 -04:00
André Bargull 8bc4e38a48 Make Atomics tests more resilient against intermittent failures (#1617)
Also fixes multiple issues and bugs in various Atomics tests.

CONTRIBUTING.md and INTERPRETING.md:
- Add missing description for `CanBlockIsFalse` flag and also introduce `CanBlockIsTrue` flag which is needed for some tests (see below).

harness/atomicsHelper.js:
- Updated `$262.agent.waitUntil` to clarify it can also be called with BigInt64Array objects.
- Added `$262.agent.timeouts` constants to unify the various, inconsistently used timeouts in the Atomics tests. Each timeout constant has a brief description and a usage example to clarify when it should or should not be used. The default values were tested in SpiderMonkey (locally and CI) under various system load levels and should hopefully also be valid for other engines. 
- Added `$262.agent.tryYield` to yield control from the main thread. The default implementation simply calls `$262.agent.sleep`, but test262 hosts could theoretically provide a different implementation.
- Added `$262.agent.trySleep` to replace direct calls to `$262.agent.sleep` from the main thread. Motivation for this function: Some test262 hosts (like browsers) may be able to pause and sleep on the main thread, so they could provide their implementation for `trySleep` which performs a busy-wait or something do nothing.

harness/testAtomics.js:
- Replace `let`, `arrow-functions`, and `for-of` loops to avoid creating extremely long `features` lists in tests using this helper. Removed `Map`, `WeakMap`, `Set`, and `WeakSet` (these weren't even listed in features.yaml) for the same reason and added the missing `Symbol` entry for this file to features.yaml.
- Updated all files including "harness/testAtomics.js" to match the new feature requirements.


test/built-ins/Atomics/store/good-views.js and test/built-ins/Atomics/store/bigint/good-views.js:
- Replace `arrow-functions` and `for-of` loops with ES5 alternatives.


test/built-ins/Atomics/wait/bigint/*.js and test/built-ins/Atomics/wake/bigint/*.js
- Some tests were using `BigInt64Array.BYTES_PER_ELEMENT * 8` for the SharedArrayBuffer length, but their non-BigInt counterparts are using `Int32Array.BYTES_PER_ELEMENT * 4`. For consistency and to make it easier to compare the BigInt against the non-BigInt versions, I've changed it to `BigInt64Array.BYTES_PER_ELEMENT * 4`.
- Also aligned formatting and statement placement when they differed between the non-BigInt and the BigInt version of a test file. (I've diffed some of the non-BigInt and BigInt files against each other and different formatting was a nuisance.)

Test files using `$262.agent.monotonicNow()`:
- Moved `$262.agent.report()` calls outside of the block of code measured `$262.agent.monotonicNow()` to avoid measuring how long it takes to execute `$262.agent.report()`. 
- Without this change some tests failed intermittently in certain test configurations in SpiderMonkey. For example with the flags `--ion-eager -- ion-offthread-compile=off` which forces early Ion compilation on the main thread. The `$262.agent.report()` implementation in the SpiderMonkey test262 host embedding uses a for-loop which was forcefully Ion compiled under these settings. And because Ion compilation can take some time, the test case ran longer than `$262.agent.MAX_TIME_EPSILON` which lead to intermittent failures.

Test files using `CanBlockIsFalse` / `CanBlockIsTrue`:
- Some of these tests actually expected that the main thread can wait and [[CanBlock]] is `true` for the agent record executing the test. Therefore I've added a new `CanBlockIsTrue` flag and replaced the flags where needed.

test/built-ins/Atomics/wait/**/*.js and test/built-ins/Atomics/wake/**/*.js:
- Use an atomic counter `RUNNING` in more tests to have better control when a worker agent was actually started.
- Replace the various `$262.agent.sleep(/* Sleep X ms to ensure worker actually sleeps */)` calls with the new `$262.agent.tryYield()` function. This `X` was sometimes as low as 10 milliseconds, which is definitely too short for CI systems under heavy load (observed by intermittent CI failures for SpiderMonkey) and sometimes as high as 500 milliseconds, which is probably much longer than needed even when the system is under heavy load. 
- Removed duplicate strings in assertion messages, presumably from copy-pasting the messages between different files.
- Removed extra empty lines at the end of multiple files.

test/built-ins/Atomics/wake/bad-range.js, test/built-ins/Atomics/wake/bigint/bad-range.js, and test/built-ins/Atomics/wait/bigint/non-bigint64-typedarray-throws.js:
- Removed unnecessary `features` and `includes` from this file.

test/built-ins/Atomics/wait/waiterlist-order-of-operations-is-fifo.js and test/built-ins/Atomics/wait/bigint/waiterlist-order-of-operations-is-fifo.js:
- The test was actually broken and didn't test what it said it does. This probably explains #1530.
- The test wants to ensure the waiterlist is implemented as a FIFO structure. This requires that the waiting agents all wait on the same index position, because the waiterlists are defined by each index. But if the agents wait on different indices, each agent is inserted into a different FIFO structure and therefore we can't observe any FIFO ordering between the agents when they're woken up.
- All this requires a bit of synchronization between the main agent and the waiting agent, I hope the added comments help to review these changes.

test/built-ins/Atomics/wait/good-views.js:
- The agent sends multiple reports to the main agent, but only the first one was read.
- This error was introduced during previous refactorings. I've changed it to back to use a while-loop as used in the first version of this file.

test/built-ins/Atomics/wait/no-spurious-wakeup-no-operation.js:
- Only the BigInt version of this test was present, copied it so we also get code coverage for the non-BigInt case.

test/built-ins/Atomics/wait/waiterlist-block-indexedposition-wake.js:
- Added extra while loops to avoid intermittent failures when the agent worker haven't started to wait.
- This should help to avoid some of the intermittent failures we saw for SpiderMonkey.

test/built-ins/Atomics/wake/wake-all-on-loc.js and test/built-ins/Atomics/wake/bigint/wake-all-on-loc.js:
- This test was also no longer after previous refactoring sessions.
- The "B" agent only waited for 10 milliseconds, which made it likely that it already timed out before the main agent was able to call `Atomics.wake`, which in turn rendered the test useless, because the test case wants to ensure that `Atomics.wake` cannot wake "B". But if "B" was already timed out, it can trivially not be woken by `Atomics.wake`.
- Added some safety measure to catch the case when "B" timed out before `Atomics.wake` was called and made it a test error if that happens.

test/built-ins/Atomics/wake/count-defaults-to-infinity-missing.js, test/built-ins/Atomics/wake/count-defaults-to-infinity-undefined.js, and test/built-ins/Atomics/wake/undefined-index-defaults-to-zero.js:
- Changed the `$262.agent.start()` calls to use a for-loop to avoid code duplication.
- (Forgot to undo the code formatting around `assert.sameValue`, still need to change it back.)
- Also more while-loops around `Atomics.wake`.
- These changes should fix #1529 and #1566.

test/built-ins/Atomics/wake/wake-all.js:
- Removed "B" worker agent.
- Without this removal the test case would be exactly equal to test/built-ins/Atomics/wake/wake-all-on-loc.js.

test/built-ins/Atomics/wake/wake-in-order-one-time.js:
- Add for-loops to avoid code duplication and make the test more readable.
- Make the `Atomics.wake` assertion messages unique by adding the current loop counter.
- Add `$262.agent.tryYield()` to give the worker agents enough time to actually start waiting.

test/built-ins/Atomics/wake/wake-in-order.js:
- Removed the outer loop `attempt < 10` because it uses `$262.agent` in a way currently not required to work. And which actually also doesn't work in SpiderMonkey's implementation of `$262.agent`.
- According to INTERPRETING.md `$262.agent.broadcast()` broadcasts its message to all agents ever started by `$262.agent.start()` and then blocks until all agents received the broadcast. It is not required that the agents started by the first `$262.agent.broadcast()` call will all be disabled/destroyed/whatever when the second `$262.agent.broadcast()` call occurs, which then means the second `$262.agent.broadcast()` call still tries to reach the agents started in the first loop iteration, but these may no longer accept broadcasts and therefore won't acknowledge they've received the broadcast. Which then means the second `$262.agent.broadcast()` call waits forever.
2018-07-03 12:34:54 -04:00
Rick Waldron 7dc92154af Frontmatter: Adds top level "locale" attribute. Fixes gh-1515 (#1561) 2018-06-19 14:56:30 -04:00
Mike Pennisi 51bbcf59be Formalize asynchronous failure conditions
Previously, the error message generated by failed asynchronous tests was
generic and underspecified. Improve the format and explicitly document
it in project's interpreting guidelines.
2018-05-20 15:24:55 -04:00
Rick Waldron dff37af3f1 Defines $262.agent.monotonicNow() 2018-05-03 13:28:46 -04:00
mitchcurtis cc6f2e352e Link to Metadata section in INTERPRETING.md (#1482) 2018-03-09 10:16:45 -05:00
Rick Waldron b0338941f7 INTERPRETING.md: Make negative phase values easier to read 2018-01-10 15:20:22 -05:00
Rick Waldron f68488ffad Fix typo in INTERPRETING.md 2018-01-05 15:38:34 -05:00
Mike Pennisi e6d674ef7a Rename negative test "phase" for module resolution 2018-01-05 15:17:51 -05:00
Mike Pennisi 136110378b Rename negative test "phase" for parsing
Early errors may result from parsing the source text of a test file, but
they may also result from parsing some other source text as referenced
through the ES2015 module syntax. The latter form of early error is not
necessarily detectable by ECMAScript parsers, however. Because of this,
the label "early" is not sufficiently precise for all Test262 consumers
to correctly interpret all tests.

Update the "phase" name of "early" to "parse" for all those negative
tests that describe errors resulting from parsing of the file's source
text directly. A forthcoming commit will update the remaining tests to
use a "phase" name that is more specific to module resolution.
2018-01-05 15:17:50 -05:00
André Bargull 266e0ffb66 Remove harness/testBuiltInObject.js file 2017-12-21 16:46:56 -05:00
Jeff Walden c05138b44c Modify $262.uncallableAndIsHTMLDDA() to $262.IsHTMLDDA (#1305)
Modify $262.uncallableAndIsHTMLDDA() to an optional $262.IsHTMLDDA (whose use must be guarded by a feature of the same name), and narrowly/correctly prescribe its requirements consistent with `document.all`'s behavior in HTML.
2017-10-19 15:03:21 -04:00
Jeff Walden 76c1a8fa3a Add tests for the case of <iterator>.return, as used in the iteration protocol, being an object that's uncallable and compares equal to `undefined`. 2017-10-17 12:21:43 -07:00
jugglinmike a7f5616713 Introduce version number (#1255)
Changes to the instructions for interpreting tests will likely produce
new failures for consumers who are updating between revisions of
Test262. Introduce a machine-readable convention for signaling
substantive changes.
2017-10-02 11:12:14 -04:00
Mike Pennisi 53842533b7 Enforce use of `throw` stmt in early error tests
Previously, test consumers were encouraged to insert a `throw` statement
as the first statement of tests for early errors. This recommendation
made tests harder to consume, and as an optional transformation,
consumers may have ignored it or simply been unaware it was made. By
explicitly including such a `throw` statement, the tests become more
literal, making them easier to consume and more transparent in their
expectations.

Document expectation for all tests for early errors to include an
explicit `throw` statement. Extend linting script to verify that
contributors are automatically notified of violations and to ensure that
future contributions satisfy this expectation.
2017-06-28 11:24:36 -04:00
Leo Balter f77a406ebc Update links to the specs (#1005) 2017-04-27 19:38:14 -04:00
Thomas Wood 2f64a621cd INTERPRETING.md missed two `$` => `$262` changes.
References #875, #802
2017-04-20 18:54:02 +01:00
jugglinmike 56fd7712d2 Document test flag: `generated` (#886)
Resolves gh-885
2017-03-06 12:12:42 -05:00
Rick Waldron fdd2fa9b1c Rename $ => $262. Fixes gh-802 (#823)
Signed-off-by: Rick Waldron <waldron.rick@gmail.com>
2017-03-01 16:34:15 -05:00
Shu-yu Guo a72ee6d912 SharedArrayBuffer and Atomics tests (#839) 2017-02-07 11:17:31 -05:00
jugglinmike 13954ed1c4 Correct file reference in "interpreting" document (#799) 2016-11-22 11:46:51 -05:00
Mike Pennisi 0d4a07ba8c Update documentation 2016-10-19 15:24:21 -04:00
jugglinmike 11d5b4f18b Document requirements for module tests 2016-06-21 12:17:51 -04:00
jugglinmike 10e4bdf0b6 Document host-defined realm/script API (#644)
Define the expected behavior of new host-defined utilities. These will
facilitate forthcoming tests that concern cross-realm and cross-script
semantics (which are currently untestable using standard ECMAScript
alone).
2016-06-21 12:11:18 -04:00
jugglinmike a657b64ae4 Add documentation on interpreting tests (#508)
The project's CONTRIBUTING.md was written with test authors in mind. It
contains details on non-technical metadata (e.g. "author" and "es6id"),
helper function usage, and preferred code structure. In addition, it
elides certain low-level technical details on the requirements for the
runtime environment.

Introduce a new document targeted towards those executing the tests.
Formalize all expectations regarding how the runtime environment should
be defined, how metadata should be interpreted, and how results should
be understood. This information has overlap with the CONTRIBUTING.md
file, but it also contains details that are irrelevant to test authors.

This document can serve as a more formal contract between Test262 and
the implementors who consume it. This allows Test262 to unambiguously
document future modifications to the formal requirements which in turn
supports consumers who maintain their own test harnesses.
2016-04-18 14:31:51 -04:00