Commit Graph

89 Commits

Author SHA1 Message Date
André Bargull 9e915f9b3c Update parseTestRecord and monkeyYaml to handle new tests
Changes:
- Allow zero indent for lists
- Handle list with zero elements
- Handle public domain license block
2024-12-17 17:52:36 +01:00
André Bargull b218cf6dec Replace "imp" module with "importlib" in parseTestRecord
The "imp" module is deprecated in Python 3.4 and will be removed in
Python 3.12. Use it only if the current Python version is too old to use
the "importlib" module.
2023-11-15 09:15:47 -08:00
Leo Balter 0277ed4260 fixup! Rename usage of master to main 2020-07-15 15:47:15 -04:00
Leo Balter 2e127ec40e Rename usage of master to main
Ref #2699
2020-07-15 15:47:15 -04:00
Leo Balter 8b28871eca Update header regexp to prevent year ranges 2019-07-15 14:03:25 -04:00
André Bargull 41edfcebce Remove old test format parser and simplify yaml frontmatter parsing 2019-02-28 06:39:59 -08:00
Leo Balter b4e15b3d5c fix parseTestRecord for optional license header at the top 2019-02-25 15:23:40 -05:00
Leo Balter 0f64e61451 update tests for packaging tools 2019-02-25 15:23:40 -05:00
André Bargull 4daa3de9bf Make parseTestRecord.py python3 compatible 2018-04-24 09:52:57 -07:00
Mike Pennisi 2915fe8527 Update test harness to support new negative format 2016-10-19 15:24:21 -04:00
jugglinmike ba32580be5 Remove Python-based test runner (#748)
TC-39 agreed to remove this tool from the Test262 repository in May of
2016 [1]. It has since been re-published as a standalone project to
facilitate independent development by interested parties [2]. Remove it
from the Test262 repository.

[1] https://github.com/tc39/tc39-notes/blob/master/es7/2016-05/may-25.md
[2] https://github.com/test262-utils/test262-harness-py
2016-08-22 14:23:26 -07:00
Sam Saccone cb4e1bcbe9 Switch to non-positional argument.
Since the argument is required, we mark it as so. Using this approach
gives the user a much nicer error message, as compared to just the "not
enough args" message.
2016-06-18 13:50:56 -07:00
jugglinmike 330cea98f5 [runner] Add support for "folding" block delimiter (#654) 2016-05-31 18:04:31 -04:00
Leo Balter 5357b1585b Merge pull request #504 from bocoup/explicit-async
Make asynchronous test configuration explicit
2016-03-17 15:28:08 -04:00
Mike Pennisi 3771cb3acc Revert "Enable parallel test execution in console runner"
This reverts commit 7ae29d49ae.
2016-02-19 10:46:18 -05:00
Mike Pennisi 4dd257d7e6 Revert "Test runner: Avoid race condition"
This reverts commit 217812891c.
2016-02-19 10:46:17 -05:00
Mike Pennisi ccf0adfc62 Revert "Runner: Re-use lock to share access to stdout"
This reverts commit b791cc4fbe.
2016-02-19 10:46:14 -05:00
Mike Pennisi 23d566209a Make asynchronous test configuration explicit
For asynchronous tests, the contract between test file and test runner
is implicit: runners are expected to inspect the source code for
references to a global `$DONE` identifier.

Promote a more explicit contract between test file and test runner by
introducing a new frontmatter "tag", `async`. This brings asynchronous
test configuration in-line with other configuration mechanisms and also
provides a more natural means of test filtering.

The modifications to test files was made programatically using the
`grep` and `sed` utilities:

    $ grep "\$DONE" test/ -r --files-with-match --null | \
        xargs -0 sed -i 's/^\(flags:\s*\)\[/\1[async, /g'
    $ grep "\$DONE" test/ -rl --null | \
        xargs -0 grep -E '^flags:' --files-without-match --null | \
        xargs -0 sed -i 's/^---\*\//flags: [async]\n---*\//'
2016-02-12 13:03:19 -05:00
Mike Pennisi b791cc4fbe Runner: Re-use lock to share access to stdout
When executing multiple tests in parallel, each "child" thread would
write to the process's standard output buffer immediately upon test
completion. Because thread execution order and instruction interleaving
is non-deterministic, this made it possible for characters to be emitted
out-of-order.

When extended to support multiple concurrent threads, the runner was
outfitted with a "log lock" dedicated to sharing access to the output
file (when applicable). Re-use this lock when writing to standard out,
ensuring proper ordering of test result messages.
2016-02-10 17:15:49 -05:00
Mike Pennisi 217812891c Test runner: Avoid race condition
A recent extension to the test runner introduced support for running
tests in parallel using multi-threading. Following this, the runner
would incorrectly emit the "final report" before all individual test
results.

In order to emit the "final report" at the end of the output stream, the
parent thread would initialize all children and wait for availability of
a "log lock" shared by all children.

According to the documentation on the "threading" module's Lock object
[1]:

> When more than one thread is blocked in acquire() waiting for the state
> to turn to unlocked, only one thread proceeds when a release() call
> resets the state to unlocked; which one of the waiting threads proceeds
> is not defined, and may vary across implementations.

This means the primitive cannot be used by the parent thread to reliably
detect completion of all child threads.

Update the parent to maintain a reference for each child thread, and to
explicitly wait for every child thread to complete before emitting the
final result.

[1] https://docs.python.org/2/library/threading.html#lock-objects
2016-02-10 16:47:01 -05:00
Till Schneidereit 7ae29d49ae Enable parallel test execution in console runner
Adds a `-j`/`--workers-count` parameter to `tools/packaging/test262.py`, defaulting to `[number of cores] - 1`.

Speeds up running the test suite by about ~3x on my 4-core machine, with the SpiderMonkey shell. This could certainly be optimized more by just appending test results to per-thread lists and merging them at the end, but it's better than nothing.
2016-02-08 17:40:16 +01:00
André Bargull a85f5039e0 Remove links to hg.ecmascript.org 2015-07-17 17:43:09 +02:00
Brian Terlson d9f1758ef4 Merge pull request #340 from bocoup/raw
Implement `raw` flag
2015-07-07 14:35:11 -05:00
Mike Pennisi ab7617dedd Implement `raw` flag
Some tests involving the directive prologue are invalidated by source
text transformations that insert executable code in the beginning of the
script. Implement a `raw` flag that allows these tests to opt-out of
this transformation. Update the relevant tests to use this flag (and
remove references to globals only available when code is injected).

Update the Python runner accordingly:

- Do not run tests marked as "raw" in strict mode
- Reject invalid test configurations

Update the browser runner accordingly:

- Do not modify the script body of tests marked as "raw"
2015-07-07 13:18:55 -04:00
Brian Terlson 284c30646c Merge pull request #351 from arv/fix-indentation
Unify indentation in test_monkeyYaml.py
2015-07-06 17:21:31 -05:00
Brian Terlson 26faeed5c7 Merge pull request #347 from bocoup/all-monkey-all-the-time
Use "monkeyYaml" in all environments
2015-07-06 16:51:00 -05:00
Erik Arvidsson 8dd6aa22dd Unify indentation in test_monkeyYaml.py 2015-07-06 10:09:26 -04:00
Mike Pennisi 1303ef0d05 Use "monkeyYaml" in all environments
The "monkeyYaml" parser is intended to serve as a lightweight fallback
to Python's standard YAML parser in contexts where the latter is not
available. Any intentionally-simplified implementation will necessarily
exhibit non-standard behavior for different input, so not all input
accepted by the standard parser will be accepted by "monkeyYaml". If
loaded exclusively in fallback situations, these edge cases can only be
identified (and debugged) in the environments that require the fallback.
This has allowed developers to unknowingly author tests that cause
errors.

Update the test runner to use "monkeyYaml" in all cases, ensuring more
consistent behavior across contexts and precluding this class of
regression.
2015-07-01 14:49:54 -04:00
Erik Arvidsson 37b1d7a7a8 monkeyYaml: Add support for line folding
Fixes #345
2015-07-01 11:09:08 -04:00
Mike Pennisi b974c13751 Move website scripts to a dedicated directory
Some JavaScript source files are only relevant in the context of the
Test262 website. They should not be explicitly included by individual
tests, so their presence in the `harness/` directory alongside "include"
files is misleading.

Move the scripts to a location within the `website/` directory to
better-reflect their intended use. Update the relevant HTML templates
with the new locations.
2015-06-26 13:29:07 -04:00
Mike Pennisi 86c7e272a6 Remove support for legacy `$INCLUDE` syntax
Although test files once expressed dependencies on external files using
a global `$INCLUDE` function, that pattern was removed in favor of
declarative meta-data [1].

Remove the associated logic from the Python runner and the browser.

[1] See commit d4354d14d5.
2015-06-26 13:29:07 -04:00
Mike Pennisi 73aa06275a Update browser runner to inject `assert.js`
Since the Python runner was updated to include `assert.js` in all tests
unconditionally, a number of tests have been written that implicitly
rely on its presence. The browser runner does not currently provide this
file's contents to these tests, so they fail unconditionally.

Update the browser runner to inject that file's contents into every test
context.

Note: the existing approach to file retrieval (namely loading via
synchronous XHR requests) is inefficient and deprecated in some
browsers. It is honored here for the sake of consistency and to minimize
the changeset necessary to fix the browser runner.
2015-06-18 13:53:09 -04:00
Erik Arvidsson be26179719 Fix monkey yaml's handling of carriage return
monkeyYaml didn't split lines correctly leading to \r in resulting
values.

Fixes #295
2015-06-10 10:45:23 -04:00
Mike Pennisi 7c4b2e9ef5 Update contribution information
Test262 now accepts community contributions. Update the copy on the
website to reflect this.
2015-05-25 17:41:13 -04:00
Brian Terlson 5ce9712be0 Python harness: run tests in both modes by default 2015-05-13 18:52:01 -07:00
smikes 107d786ec5 candidate fix for #215
additional test, tests should have unique names
improve python style thx to arv
2015-03-30 15:37:59 -06:00
Gregory Brail 41ed1257d7 Fix JUnit output so that it can be actually be parsed by JUnit and
Jenkins. Wrap XML output correctly and
post-process failure messages to avoid invalid XML characters.
2015-02-26 16:46:59 -08:00
Brian Terlson b81b71eac2 Fix packager.py for updated locations 2014-12-09 14:52:12 -08:00
Brian Terlson 47821a8bd0 Update python scripts for new paths 2014-12-07 15:42:12 -08:00
Brian Terlson 3883a2e906 Merge pull request #127 from smikes/asserts
add "assert.js" to python, website runners too
2014-12-01 20:14:47 -08:00
smikes e0fdc30078 add "assert.js" to python, website runners too 2014-12-01 20:04:35 -07:00
Domenic Denicola 2a01b54bb4 Fix whitespace for async test detection
Presumably since Python is whitespace-sensitive, this is actually quite important?
2014-12-01 15:53:27 -05:00
Domenic Denicola dd243a757c Remove trailing whitespace from the Python 2014-12-01 15:52:19 -05:00
smikes 64093f79d3 add some unit tests
add test to repro #114
fix issue #114
factor out PercentFormat function
new fn write; start factoring summary output
fix tests to match
consolidate "Failed Tests" message
add test for successful run
consolidate all_succeeded message
consolidate summary line
consolidate ntests
failing negative tests
squeeze whitespace
2014-11-11 17:07:48 -07:00
Brian Terlson 1f2812adeb Merge pull request #104 from smikes/monkeyYaml-loadfix
load monkeyYaml without assuming it is on path
2014-10-24 14:41:40 -07:00
Sam Mikes 9dc29da897 load monkeyYaml without assuming it is on path
use imp to import monkeyYaml locally
make monkeyYaml the backup again
2014-10-24 22:36:03 +01:00
Sam Mikes 89d71f56b2 additional fixes suggested by @anba 2014-10-24 15:18:37 +01:00
Sam Mikes 4f9cec2fa8 provide fallback parser if YAML not installed
parseTestRecord.py:
- recover from ImportError when YAML not defined
- use monkeyYaml as backup

monkeyYaml.py:
- parser for subset of YAML used in test262 frontmatter

test_monkeyYaml.py:
- unit tests
2014-08-28 12:47:58 +01:00
Sam Mikes 6b7e5d7747 bestPractice: supply error names to negative
pilot directory for replacing flags: [negative] with
negative: errorname

ch07-7.2,7.3: add error names
ch07-rest: insert error names rest of ch07
ch08: add expected error name
ch11: add error name to negative
ch12: put error name in negative

ch13: add error name to negative
only one test, did it manually

ch15: add error name to negative:

ch07: add error name to negative

these tests used flags: \n - negative and so
were not caught by the earlier naive grep for flags: [negative]

--- changes suggested by @anba

bestPractice - remove added NotEarlyError
ch07,ch11 - expect ReferenceError (req'd by ES6)
correct test descriptions to expect ReferenceError only
enforce parsing as block statement by adding `;`
remove needless include of $FAIL.js

effectively, a rebase of all changes onto master

    Merge branch 'negative-addErrorName-work' into negative-addErrorName-bestPractice

    Merge branch 'saved-bestPractice-negative' into negative-addErrorName-work

    Conflicts:
    	test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T2.js
    	test/suite/ch12/12.1/S12.1_A4_T1.js
    	test/suite/ch12/12.5/S12.5_A2.js

    incorporate changes made on PR branch

    re-remove added NotEarlyError

    Merge branch 'negative-addErrorName-work' into negative-addErrorName-bestPractice

    Conflicts:
    	test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T2.js
    	test/suite/ch12/12.1/S12.1_A4_T1.js
    	test/suite/ch12/12.5/S12.5_A2.js

    ch15: correct negative regexes

    ch10: avoid use of not-NotEarlyError regex

    ch14 error regexps

    ch13 error regexps

    ch12: negative error matching

    Merge branch 'console-runner-checkError' into negative-addErrorName-work

    test262.py: check negative tests with regex

    implement checking of negative tests =>

    negative: SyntaxError

    means that /SyntaxError/ must match stderr
    or test is reported as failure

    Fixes #78

    expect real errors

    bestPractice: supply error names to negative

    pilot directory for replacing flags: [negative] with
    negative: errorname

    ch07-7.2,7.3: add error names
    ch07-rest: insert error names rest of ch07
    ch08: add expected error name
    ch11: add error name to negative
    ch12: put error name in negative

    ch13: add error name to negative
    only one test, did it manually

    ch15: add error name to negative:

    ch07: add error name to negative

    these tests used flags: \n - negative and so
    were not caught by the earlier naive grep for flags: [negative]

    --- changes suggested by @anba

    bestPractice - remove added NotEarlyError
    ch07,ch11 - expect ReferenceError (req'd by ES6)
    correct test descriptions to expect ReferenceError only
    enforce parsing as block statement by adding `;`
    remove needless include of $FAIL.js

    Merge branch 'negative-addErrorName-work' into negative-addErrorName-bestPractice

    resolved Conflicts:
    	test/suite/ch07/7.9/S7.9_A5.7_T1.js
    	test/suite/ch11/11.13/11.13.1/S11.13.1_A2.1_T3.js
    	test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T1.js
    	test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T10.js
    	test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T11.js
    	test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T2.js
    	test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T3.js
    	test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T4.js
    	test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T5.js
    	test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T6.js
    	test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T7.js
    	test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T8.js
    	test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T9.js
    	test/suite/ch11/11.3/11.3.1/S11.3.1_A2.1_T3.js
    	test/suite/ch11/11.3/11.3.2/S11.3.2_A2.1_T3.js
    	test/suite/ch11/11.4/11.4.4/S11.4.4_A2.1_T3.js
    	test/suite/ch11/11.4/11.4.5/S11.4.5_A2.1_T3.js
    	test/suite/ch12/12.5/S12.5_A2.js
    	test/suite/ch12/12.6/12.6.3/S12.6.3_A4_T2.js

    negative: second half of @anba notes

    correct test descriptions to expect ReferenceError only
    enforce parse as block statement by adding `;`
    remove needless include of $FAIL.js

    ch12: specify type

    negative: @anba fixes

    bestPractice - remove added NotEarlyError
    ch07,ch11 - expect ReferenceError (req'd by ES6)

    bestPractice: supply error names to negative

    pilot directory for replacing flags: [negative] with
    negative: errorname

    ch07-7.2,7.3: add error names
    ch07-rest: insert error names rest of ch07
    ch08: add expected error name
    ch11: add error name to negative
    ch12: put error name in negative

    ch13: add error name to negative
    only one test, did it manually

    ch15: add error name to negative:

    ch07: add error name to negative

    these tests used flags: \n - negative and so
    were not caught by the earlier naive grep for flags: [negative]

    expect a SyntaxError

    ch07: add error name to negative

    these tests used flags: \n - negative and so
    were not caught by the earlier naive grep for flags: [negative]

    ch15: add error name to negative:

    ch13: add error name to negative

    only one test, did it manually

    ch12: put error name in negative

    ch11: add error name to negative

    ch08: add expected error name
    ch07-rest: insert error names rest of ch07

    ch07-7.2,7.3: add error names

    negative-errorname: pilot

    pilot directory for replacing flags: [negative] with
    negative: errorname

    bestPractice: supply error names to negative

bestPractice: supply error names to negative

pilot directory for replacing flags: [negative] with
negative: errorname

ch07-7.2,7.3: add error names
ch07-rest: insert error names rest of ch07
ch08: add expected error name
ch11: add error name to negative
ch12: put error name in negative

ch13: add error name to negative
only one test, did it manually

ch15: add error name to negative:

ch07: add error name to negative

these tests used flags: \n - negative and so
were not caught by the earlier naive grep for flags: [negative]

bestPractice: supply error names to negative

negative-errorname: pilot

pilot directory for replacing flags: [negative] with
negative: errorname

ch07-7.2,7.3: add error names

ch07-rest: insert error names rest of ch07

ch08: add expected error name

ch11: add error name to negative

ch12: put error name in negative

ch13: add error name to negative

only one test, did it manually

ch15: add error name to negative:

ch07: add error name to negative

these tests used flags: \n - negative and so
were not caught by the earlier naive grep for flags: [negative]

expect a SyntaxError

negative: @anba fixes

bestPractice - remove added NotEarlyError
ch07,ch11 - expect ReferenceError (req'd by ES6)

ch12: specify type

negative: second half of @anba notes

correct test descriptions to expect ReferenceError only
enforce parse as block statement by adding `;`
remove needless include of $FAIL.js

expect real errors

ch12: negative error matching

ch13 error regexps

ch14 error regexps

add "description" header back

remove now-unused includes

remove needless include
2014-08-13 00:19:11 +01:00
Sam Mikes 4205a1da64 harness: remove unused code
sta.js:
slight change to Test262Error() semantics; message property now always set (default "")
make $ERROR a var
set $ERROR to function $ERROR so it can be overridden if needed
remove 2009 copyright in favor of 2012 copyright
REVERTED: remove never-used fn testFailed
testFailed actually used by $FAIL
remove obsolete fn $INCLUDE

ed.js:
remove commented-out obsolete code

test262.py:
remove always-included harness file that provides no functions used by any extant test
add comment line to nonstrict

cth.js: define `print` for node, cscript

use cth (console test harness) to define functions
for console runner.  V8, Spidermonkey (js) and JavaScriptCore (jsc)
provide a function `print`.  Provide a default `print` for node
and cscript

set print_handle to 'print' by default (can still override)
for cscript: wrap tests in try/catch so we get syntax errors
2014-08-03 00:33:08 +01:00