Commit Graph

107 Commits

Author SHA1 Message Date
Gregory Brail 41ed1257d7 Fix JUnit output so that it can be actually be parsed by JUnit and
Jenkins. Wrap XML output correctly and
post-process failure messages to avoid invalid XML characters.
2015-02-26 16:46:59 -08:00
Brian Terlson b81b71eac2 Fix packager.py for updated locations 2014-12-09 14:52:12 -08:00
Brian Terlson 47821a8bd0 Update python scripts for new paths 2014-12-07 15:42:12 -08:00
Brian Terlson 3883a2e906 Merge pull request #127 from smikes/asserts
add "assert.js" to python, website runners too
2014-12-01 20:14:47 -08:00
smikes e0fdc30078 add "assert.js" to python, website runners too 2014-12-01 20:04:35 -07:00
Domenic Denicola 2a01b54bb4 Fix whitespace for async test detection
Presumably since Python is whitespace-sensitive, this is actually quite important?
2014-12-01 15:53:27 -05:00
Domenic Denicola dd243a757c Remove trailing whitespace from the Python 2014-12-01 15:52:19 -05:00
smikes 64093f79d3 add some unit tests
add test to repro #114
fix issue #114
factor out PercentFormat function
new fn write; start factoring summary output
fix tests to match
consolidate "Failed Tests" message
add test for successful run
consolidate all_succeeded message
consolidate summary line
consolidate ntests
failing negative tests
squeeze whitespace
2014-11-11 17:07:48 -07:00
Brian Terlson 1f2812adeb Merge pull request #104 from smikes/monkeyYaml-loadfix
load monkeyYaml without assuming it is on path
2014-10-24 14:41:40 -07:00
Sam Mikes 9dc29da897 load monkeyYaml without assuming it is on path
use imp to import monkeyYaml locally
make monkeyYaml the backup again
2014-10-24 22:36:03 +01:00
Sam Mikes 89d71f56b2 additional fixes suggested by @anba 2014-10-24 15:18:37 +01:00
Sam Mikes 4f9cec2fa8 provide fallback parser if YAML not installed
parseTestRecord.py:
- recover from ImportError when YAML not defined
- use monkeyYaml as backup

monkeyYaml.py:
- parser for subset of YAML used in test262 frontmatter

test_monkeyYaml.py:
- unit tests
2014-08-28 12:47:58 +01:00
Sam Mikes 6b7e5d7747 bestPractice: supply error names to negative
pilot directory for replacing flags: [negative] with
negative: errorname

ch07-7.2,7.3: add error names
ch07-rest: insert error names rest of ch07
ch08: add expected error name
ch11: add error name to negative
ch12: put error name in negative

ch13: add error name to negative
only one test, did it manually

ch15: add error name to negative:

ch07: add error name to negative

these tests used flags: \n - negative and so
were not caught by the earlier naive grep for flags: [negative]

--- changes suggested by @anba

bestPractice - remove added NotEarlyError
ch07,ch11 - expect ReferenceError (req'd by ES6)
correct test descriptions to expect ReferenceError only
enforce parsing as block statement by adding `;`
remove needless include of $FAIL.js

effectively, a rebase of all changes onto master

    Merge branch 'negative-addErrorName-work' into negative-addErrorName-bestPractice

    Merge branch 'saved-bestPractice-negative' into negative-addErrorName-work

    Conflicts:
    	test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T2.js
    	test/suite/ch12/12.1/S12.1_A4_T1.js
    	test/suite/ch12/12.5/S12.5_A2.js

    incorporate changes made on PR branch

    re-remove added NotEarlyError

    Merge branch 'negative-addErrorName-work' into negative-addErrorName-bestPractice

    Conflicts:
    	test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T2.js
    	test/suite/ch12/12.1/S12.1_A4_T1.js
    	test/suite/ch12/12.5/S12.5_A2.js

    ch15: correct negative regexes

    ch10: avoid use of not-NotEarlyError regex

    ch14 error regexps

    ch13 error regexps

    ch12: negative error matching

    Merge branch 'console-runner-checkError' into negative-addErrorName-work

    test262.py: check negative tests with regex

    implement checking of negative tests =>

    negative: SyntaxError

    means that /SyntaxError/ must match stderr
    or test is reported as failure

    Fixes #78

    expect real errors

    bestPractice: supply error names to negative

    pilot directory for replacing flags: [negative] with
    negative: errorname

    ch07-7.2,7.3: add error names
    ch07-rest: insert error names rest of ch07
    ch08: add expected error name
    ch11: add error name to negative
    ch12: put error name in negative

    ch13: add error name to negative
    only one test, did it manually

    ch15: add error name to negative:

    ch07: add error name to negative

    these tests used flags: \n - negative and so
    were not caught by the earlier naive grep for flags: [negative]

    --- changes suggested by @anba

    bestPractice - remove added NotEarlyError
    ch07,ch11 - expect ReferenceError (req'd by ES6)
    correct test descriptions to expect ReferenceError only
    enforce parsing as block statement by adding `;`
    remove needless include of $FAIL.js

    Merge branch 'negative-addErrorName-work' into negative-addErrorName-bestPractice

    resolved Conflicts:
    	test/suite/ch07/7.9/S7.9_A5.7_T1.js
    	test/suite/ch11/11.13/11.13.1/S11.13.1_A2.1_T3.js
    	test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T1.js
    	test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T10.js
    	test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T11.js
    	test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T2.js
    	test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T3.js
    	test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T4.js
    	test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T5.js
    	test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T6.js
    	test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T7.js
    	test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T8.js
    	test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T9.js
    	test/suite/ch11/11.3/11.3.1/S11.3.1_A2.1_T3.js
    	test/suite/ch11/11.3/11.3.2/S11.3.2_A2.1_T3.js
    	test/suite/ch11/11.4/11.4.4/S11.4.4_A2.1_T3.js
    	test/suite/ch11/11.4/11.4.5/S11.4.5_A2.1_T3.js
    	test/suite/ch12/12.5/S12.5_A2.js
    	test/suite/ch12/12.6/12.6.3/S12.6.3_A4_T2.js

    negative: second half of @anba notes

    correct test descriptions to expect ReferenceError only
    enforce parse as block statement by adding `;`
    remove needless include of $FAIL.js

    ch12: specify type

    negative: @anba fixes

    bestPractice - remove added NotEarlyError
    ch07,ch11 - expect ReferenceError (req'd by ES6)

    bestPractice: supply error names to negative

    pilot directory for replacing flags: [negative] with
    negative: errorname

    ch07-7.2,7.3: add error names
    ch07-rest: insert error names rest of ch07
    ch08: add expected error name
    ch11: add error name to negative
    ch12: put error name in negative

    ch13: add error name to negative
    only one test, did it manually

    ch15: add error name to negative:

    ch07: add error name to negative

    these tests used flags: \n - negative and so
    were not caught by the earlier naive grep for flags: [negative]

    expect a SyntaxError

    ch07: add error name to negative

    these tests used flags: \n - negative and so
    were not caught by the earlier naive grep for flags: [negative]

    ch15: add error name to negative:

    ch13: add error name to negative

    only one test, did it manually

    ch12: put error name in negative

    ch11: add error name to negative

    ch08: add expected error name
    ch07-rest: insert error names rest of ch07

    ch07-7.2,7.3: add error names

    negative-errorname: pilot

    pilot directory for replacing flags: [negative] with
    negative: errorname

    bestPractice: supply error names to negative

bestPractice: supply error names to negative

pilot directory for replacing flags: [negative] with
negative: errorname

ch07-7.2,7.3: add error names
ch07-rest: insert error names rest of ch07
ch08: add expected error name
ch11: add error name to negative
ch12: put error name in negative

ch13: add error name to negative
only one test, did it manually

ch15: add error name to negative:

ch07: add error name to negative

these tests used flags: \n - negative and so
were not caught by the earlier naive grep for flags: [negative]

bestPractice: supply error names to negative

negative-errorname: pilot

pilot directory for replacing flags: [negative] with
negative: errorname

ch07-7.2,7.3: add error names

ch07-rest: insert error names rest of ch07

ch08: add expected error name

ch11: add error name to negative

ch12: put error name in negative

ch13: add error name to negative

only one test, did it manually

ch15: add error name to negative:

ch07: add error name to negative

these tests used flags: \n - negative and so
were not caught by the earlier naive grep for flags: [negative]

expect a SyntaxError

negative: @anba fixes

bestPractice - remove added NotEarlyError
ch07,ch11 - expect ReferenceError (req'd by ES6)

ch12: specify type

negative: second half of @anba notes

correct test descriptions to expect ReferenceError only
enforce parse as block statement by adding `;`
remove needless include of $FAIL.js

expect real errors

ch12: negative error matching

ch13 error regexps

ch14 error regexps

add "description" header back

remove now-unused includes

remove needless include
2014-08-13 00:19:11 +01:00
Sam Mikes 4205a1da64 harness: remove unused code
sta.js:
slight change to Test262Error() semantics; message property now always set (default "")
make $ERROR a var
set $ERROR to function $ERROR so it can be overridden if needed
remove 2009 copyright in favor of 2012 copyright
REVERTED: remove never-used fn testFailed
testFailed actually used by $FAIL
remove obsolete fn $INCLUDE

ed.js:
remove commented-out obsolete code

test262.py:
remove always-included harness file that provides no functions used by any extant test
add comment line to nonstrict

cth.js: define `print` for node, cscript

use cth (console test harness) to define functions
for console runner.  V8, Spidermonkey (js) and JavaScriptCore (jsc)
provide a function `print`.  Provide a default `print` for node
and cscript

set print_handle to 'print' by default (can still override)
for cscript: wrap tests in try/catch so we get syntax errors
2014-08-03 00:33:08 +01:00
Sam Mikes c33bf0e043 tools, harness: support new YAML frontmatter
parseTestRecord: add support for YAML frontmatter
parseTestRecord: initial unit test for test record parser
parseTestRecord: refactor for testing

factor old parsing; add YAML parsing

runner: support "includes" from YAML frontmatter

support frontmatter "includes" in python runner
use test.includes if present instead of scanning test code with regex

harness: factor individual functions out into files

tools: handle YAML errors

tolerate missing keys in dictionary (flags, includes)
report filename when empty frontmatter block
new option --list-includes to test262.py

harness: factor helper functions into separate files

sth: remove extra close-paren (syntax error)

test_common: TDD; failing parse of YAML

common: use parseTestRecord (YAML-aware)
2014-07-30 15:39:04 -07:00
Sam Mikes d4a3479a1e test262.py: only include helper scripts when needed
test262.py: only supply async helper scripts when test is async
sth.js: factor out function isAsyncTest()
timer.js: improve workaround for async tests when Promise is defined but setTimeout is noot

timer.js emulates setTimeout using Promise by doing a busy loop that checks
if `timeout` milliseconds have elapsed.  Modified check to (timeLeft > 0) instead
of (!timeLeft) to prevent infinite loop when check does not happen to run
at precise millisecond timeout expires.

Because test262.py did not support the $INCLUDE directive, some helper
scripts were added to every test -- notably testIntl, timer, and donePrintHandle
Now that $INCLUDE is supported, these can be dropped, speeding overall test run time
2014-07-21 15:15:04 +01:00
Brian Terlson 33c8399de8 Merge pull request #31 from Constellation/fix-error-code
Propagate error code when test cases are failed
2014-07-18 11:38:31 -07:00
Brian Terlson 96c6efe0ef Merge pull request #43 from smikes/initial-Promise-tests
Initial promise tests
2014-07-18 11:22:48 -07:00
smikes 7032af4db8 packager: make packager runnable
packager.py cannot run due to syntax errors in a few script files

packagerConfig: use git instead of hg
test262: use pop instead of delete to avoid throw if property missing
S22.1.2.1_T*: fix docString header comment: s/b /**
S22.1.2.1_T3: fix docString header comment: s/b /**, fix end of docstring * / => */
2014-07-18 08:53:29 +01:00
Sam Mikes 7e07cc138d async,promises: initial tests of Promises
doneprintHandle.js: make $DONE accept any falsy argument as meaning 'pass'
PromiseHelper.js: checkSequence: new helper fn for async tests
.gitignore: port .hgignore to .gitignore syntax
test262.py: support $INCLUDE directive in python test runner

S25.4.4.1*: tests to cover Section 25.4.4.1, Promise.all( iterable )
A1.1: Promise.all is callable
A1.2: Promise.all expects 1 argument
A2.1: Promise.all([]) is a Promise
A2.2: Promise.all([]) is resolved immediately
A2.3: Promise.all([]) is resolved with a new empty array
A3.1: Promise.all expects an iterable argument
2014-07-18 05:37:08 +01:00
NikSurya 9aa6b917fa Adding Support for Async Tests 2014-07-15 13:47:59 -07:00
Yusuke Suzuki d83a16f46c Propagate error code when test cases are failed 2014-03-03 03:32:26 +09:00
Prashanth Srinivasan 87605ba3b5 Bug 1472: Adding JUnit Compatible XML and Logfile support for Python test runner 2013-06-10 11:51:24 -04:00
Dan Schaffer 06879c044f Bug 1466: add test262.py parameter to set loglevel 2013-05-10 16:19:32 -04:00
Dan Schaffer 1cb9ed439f Bug 1457: fix for Windows to remove quotes in test262.py 2013-05-10 16:16:51 -04:00
Norbert Lindenberg 46359091c7 Updated test402 main page to reflect that ECMA-402 is now an approved standard.
Fixed a property access in 6.4_c so that the error message shows the correct expected value.
2012-12-17 23:49:06 -08:00
Norbert Lindenberg 1ba4238d9a Made web site packager insert line breaks into JSON files for saner diffs. 2012-09-12 15:09:36 -07:00
Norbert Lindenberg 1af2425075 Added new tests for chapters 6 and 9 of ECMAScript Internationalization API Specification.
- Removed a few old test cases that were redundant with new, more comprehensive ones.
- Added testIntl.js as standard include for all console tests in test262.py – see related bug 574.
- Added .jshintrc file for settings for the JSHint tool.
2012-08-26 20:49:25 -07:00
Norbert Lindenberg 8cad7d03ce Added tests for requirements imposed on the built-in objects of the ECMAScript Internationalization API Specification by the introduction of chapter 15 of the ECMAScript Language Specification.
- Removed some old tests that were redundant with the new tests.
- Added testBuiltInObject.js as standard include for all console tests in test262.py – see related bug 574.
2012-08-26 20:48:49 -07:00
Norbert Lindenberg c3111850e3 Updated existing tests for June 2012 spec changes; removed LocaleList tests; fixed minor issues.
- Removed LocaleList tests; updated other tests so they don't depend on LocaleList.
- Updated tests so they no longer assume that the prototype object of a constructor is an instance of that constructor.
- Updated tests so that jshint is happy.
- Removed @path attributes from test files; updated comment in packager.py explaining why they're unnecessary.
- Removed "use strict" statements, which interfere with strict/non-strict testing.
- Removed testcase functions, which are unnecessary.
2012-08-22 17:53:26 -07:00
Norbert Lindenberg 9f4a513496 Merged changes. 2012-08-22 16:12:54 -07:00
Bill Ticehurst ebdd0a99e6 Fixing Intl402 and BestPractice pages as per bug https://bugs.ecmascript.org/show_bug.cgi?id=611 2012-08-09 12:29:48 -07:00
Norbert Lindenberg 9a4b4011f8 Got test402 web site into usable shape. Unified headers of test262/402 sites.
- Added table of contents for ECMA-402.
- Added ability to load the right table of contents into results display.
- Added meaningful text to test402 web page.
- Updated test262 web pages for 5.1 edition of ECMA-262.
- Unified headers of test262/402 web pages to ECMAScript <area> <test>.
- Replaced text logos with styled text.
- Decapitalized some text; fixed a typo; expanded an abbreviation.
- Fixed a few issues found by W3 HTML validator.
2012-07-18 12:32:54 -07:00
Norbert Lindenberg d371b39cb6 Fixed bug 443: packager.py fails on Mac because of .DS_Store files.
Running packager.py on Mac unfortunately also changes some line terminators in generated files (bug 444), so lots of additional files in this commit.
2012-07-18 12:19:53 -07:00
t-adamre 385b4f7163 Add capability to run chapters individually from the website 2012-06-12 14:54:30 -07:00
David Fugate 87be1bf6ec https://bugs.ecmascript.org/show_bug.cgi?id=65 fixed. 2012-03-28 16:05:03 -07:00
David Fugate 23598a3c67 Renamed AddLicenseHeader.py to FixLicenseHeader.py.
Modified it to switch us over to the Ecma license for everything
except Sputnik-based sources (someone from Google should make
this change on line 60 of FixLicenseHeader.py).
2012-03-01 11:06:50 -08:00
David Fugate 0612ad7bb3 Added a LICENSE file to the root of the enlistment.
All Microsoft-contributed *.js and *.py sources now have Ecma's copyright header.
2012-02-29 14:23:47 -08:00
David Fugate c3b41309c7 Made it clear on testcases_bestPractice.html that it's not the same as default.html. 2012-02-29 09:02:11 -08:00
David Fugate 4fb5071cd4 Migrated latest external\contributions\Microsoft\ietcLatest changes over to test\suite\*. 2012-02-27 15:46:23 -08:00
David Fugate 394e82c6c2 Fixed more license headers. 2012-02-08 15:33:30 -08:00
David Fugate e0cf97e344 A conversion tool ripped Microsoft license headers out of globally scoped tests. Fixed 2012-02-07 09:34:36 -08:00
David Fugate 0ff258fa1d Experimental test402 (internationalization testing) page added. 2012-01-25 14:29:00 -08:00
David Fugate 279825ea44 Added experimental test402 support (JS internationalization standard). 2012-01-25 14:04:40 -08:00
David Fugate d7619456fa Generate one *.html per *.json in addition to the default.html which includes all tests.
Added support to create testcases_bestPractice.html without also adding them to default.html.

Preliminary support for tests under test/suite/internationalization has been added.
2012-01-19 16:03:21 -08:00
David Fugate 43b58c99d0 Preserve only the first line of each test case's copyright header
when encoding to JSON which coincidentally preserves the BOM as well.
This shaved ~10 megs off of chapter15.json alone.
2011-11-04 12:04:46 -07:00
David Fugate 6d83f405b8 The Sputnik/IETC => test262 converter is no longer useful. 2011-11-04 11:16:13 -07:00
David Fugate edd1628de5 FindTestCaseIssues.py is a new helper script which can be used to detect issues with the format of test cases (e.g., runTestCase not being called).
FixPathsAndIds.py now preserves line endings and the BOM.
2011-11-04 11:13:23 -07:00
David Fugate dc9bd9601b Fewer Json files (experiment...). 2011-10-03 16:01:47 -07:00
David Fugate 5a665767e6 Re-gen'ed website using new Sputnik directory names. 2011-10-03 13:31:15 -07:00