Commit Graph

108 Commits

Author SHA1 Message Date
smikes 107d786ec5 candidate fix for #215
additional test, tests should have unique names
improve python style thx to arv
2015-03-30 15:37:59 -06:00
Gregory Brail 41ed1257d7 Fix JUnit output so that it can be actually be parsed by JUnit and
Jenkins. Wrap XML output correctly and
post-process failure messages to avoid invalid XML characters.
2015-02-26 16:46:59 -08:00
Brian Terlson b81b71eac2 Fix packager.py for updated locations 2014-12-09 14:52:12 -08:00
Brian Terlson 47821a8bd0 Update python scripts for new paths 2014-12-07 15:42:12 -08:00
Brian Terlson 3883a2e906 Merge pull request #127 from smikes/asserts
add "assert.js" to python, website runners too
2014-12-01 20:14:47 -08:00
smikes e0fdc30078 add "assert.js" to python, website runners too 2014-12-01 20:04:35 -07:00
Domenic Denicola 2a01b54bb4 Fix whitespace for async test detection
Presumably since Python is whitespace-sensitive, this is actually quite important?
2014-12-01 15:53:27 -05:00
Domenic Denicola dd243a757c Remove trailing whitespace from the Python 2014-12-01 15:52:19 -05:00
smikes 64093f79d3 add some unit tests
add test to repro #114
fix issue #114
factor out PercentFormat function
new fn write; start factoring summary output
fix tests to match
consolidate "Failed Tests" message
add test for successful run
consolidate all_succeeded message
consolidate summary line
consolidate ntests
failing negative tests
squeeze whitespace
2014-11-11 17:07:48 -07:00
Brian Terlson 1f2812adeb Merge pull request #104 from smikes/monkeyYaml-loadfix
load monkeyYaml without assuming it is on path
2014-10-24 14:41:40 -07:00
Sam Mikes 9dc29da897 load monkeyYaml without assuming it is on path
use imp to import monkeyYaml locally
make monkeyYaml the backup again
2014-10-24 22:36:03 +01:00
Sam Mikes 89d71f56b2 additional fixes suggested by @anba 2014-10-24 15:18:37 +01:00
Sam Mikes 4f9cec2fa8 provide fallback parser if YAML not installed
parseTestRecord.py:
- recover from ImportError when YAML not defined
- use monkeyYaml as backup

monkeyYaml.py:
- parser for subset of YAML used in test262 frontmatter

test_monkeyYaml.py:
- unit tests
2014-08-28 12:47:58 +01:00
Sam Mikes 6b7e5d7747 bestPractice: supply error names to negative
pilot directory for replacing flags: [negative] with
negative: errorname

ch07-7.2,7.3: add error names
ch07-rest: insert error names rest of ch07
ch08: add expected error name
ch11: add error name to negative
ch12: put error name in negative

ch13: add error name to negative
only one test, did it manually

ch15: add error name to negative:

ch07: add error name to negative

these tests used flags: \n - negative and so
were not caught by the earlier naive grep for flags: [negative]

--- changes suggested by @anba

bestPractice - remove added NotEarlyError
ch07,ch11 - expect ReferenceError (req'd by ES6)
correct test descriptions to expect ReferenceError only
enforce parsing as block statement by adding `;`
remove needless include of $FAIL.js

effectively, a rebase of all changes onto master

    Merge branch 'negative-addErrorName-work' into negative-addErrorName-bestPractice

    Merge branch 'saved-bestPractice-negative' into negative-addErrorName-work

    Conflicts:
    	test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T2.js
    	test/suite/ch12/12.1/S12.1_A4_T1.js
    	test/suite/ch12/12.5/S12.5_A2.js

    incorporate changes made on PR branch

    re-remove added NotEarlyError

    Merge branch 'negative-addErrorName-work' into negative-addErrorName-bestPractice

    Conflicts:
    	test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T2.js
    	test/suite/ch12/12.1/S12.1_A4_T1.js
    	test/suite/ch12/12.5/S12.5_A2.js

    ch15: correct negative regexes

    ch10: avoid use of not-NotEarlyError regex

    ch14 error regexps

    ch13 error regexps

    ch12: negative error matching

    Merge branch 'console-runner-checkError' into negative-addErrorName-work

    test262.py: check negative tests with regex

    implement checking of negative tests =>

    negative: SyntaxError

    means that /SyntaxError/ must match stderr
    or test is reported as failure

    Fixes #78

    expect real errors

    bestPractice: supply error names to negative

    pilot directory for replacing flags: [negative] with
    negative: errorname

    ch07-7.2,7.3: add error names
    ch07-rest: insert error names rest of ch07
    ch08: add expected error name
    ch11: add error name to negative
    ch12: put error name in negative

    ch13: add error name to negative
    only one test, did it manually

    ch15: add error name to negative:

    ch07: add error name to negative

    these tests used flags: \n - negative and so
    were not caught by the earlier naive grep for flags: [negative]

    --- changes suggested by @anba

    bestPractice - remove added NotEarlyError
    ch07,ch11 - expect ReferenceError (req'd by ES6)
    correct test descriptions to expect ReferenceError only
    enforce parsing as block statement by adding `;`
    remove needless include of $FAIL.js

    Merge branch 'negative-addErrorName-work' into negative-addErrorName-bestPractice

    resolved Conflicts:
    	test/suite/ch07/7.9/S7.9_A5.7_T1.js
    	test/suite/ch11/11.13/11.13.1/S11.13.1_A2.1_T3.js
    	test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T1.js
    	test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T10.js
    	test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T11.js
    	test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T2.js
    	test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T3.js
    	test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T4.js
    	test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T5.js
    	test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T6.js
    	test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T7.js
    	test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T8.js
    	test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T9.js
    	test/suite/ch11/11.3/11.3.1/S11.3.1_A2.1_T3.js
    	test/suite/ch11/11.3/11.3.2/S11.3.2_A2.1_T3.js
    	test/suite/ch11/11.4/11.4.4/S11.4.4_A2.1_T3.js
    	test/suite/ch11/11.4/11.4.5/S11.4.5_A2.1_T3.js
    	test/suite/ch12/12.5/S12.5_A2.js
    	test/suite/ch12/12.6/12.6.3/S12.6.3_A4_T2.js

    negative: second half of @anba notes

    correct test descriptions to expect ReferenceError only
    enforce parse as block statement by adding `;`
    remove needless include of $FAIL.js

    ch12: specify type

    negative: @anba fixes

    bestPractice - remove added NotEarlyError
    ch07,ch11 - expect ReferenceError (req'd by ES6)

    bestPractice: supply error names to negative

    pilot directory for replacing flags: [negative] with
    negative: errorname

    ch07-7.2,7.3: add error names
    ch07-rest: insert error names rest of ch07
    ch08: add expected error name
    ch11: add error name to negative
    ch12: put error name in negative

    ch13: add error name to negative
    only one test, did it manually

    ch15: add error name to negative:

    ch07: add error name to negative

    these tests used flags: \n - negative and so
    were not caught by the earlier naive grep for flags: [negative]

    expect a SyntaxError

    ch07: add error name to negative

    these tests used flags: \n - negative and so
    were not caught by the earlier naive grep for flags: [negative]

    ch15: add error name to negative:

    ch13: add error name to negative

    only one test, did it manually

    ch12: put error name in negative

    ch11: add error name to negative

    ch08: add expected error name
    ch07-rest: insert error names rest of ch07

    ch07-7.2,7.3: add error names

    negative-errorname: pilot

    pilot directory for replacing flags: [negative] with
    negative: errorname

    bestPractice: supply error names to negative

bestPractice: supply error names to negative

pilot directory for replacing flags: [negative] with
negative: errorname

ch07-7.2,7.3: add error names
ch07-rest: insert error names rest of ch07
ch08: add expected error name
ch11: add error name to negative
ch12: put error name in negative

ch13: add error name to negative
only one test, did it manually

ch15: add error name to negative:

ch07: add error name to negative

these tests used flags: \n - negative and so
were not caught by the earlier naive grep for flags: [negative]

bestPractice: supply error names to negative

negative-errorname: pilot

pilot directory for replacing flags: [negative] with
negative: errorname

ch07-7.2,7.3: add error names

ch07-rest: insert error names rest of ch07

ch08: add expected error name

ch11: add error name to negative

ch12: put error name in negative

ch13: add error name to negative

only one test, did it manually

ch15: add error name to negative:

ch07: add error name to negative

these tests used flags: \n - negative and so
were not caught by the earlier naive grep for flags: [negative]

expect a SyntaxError

negative: @anba fixes

bestPractice - remove added NotEarlyError
ch07,ch11 - expect ReferenceError (req'd by ES6)

ch12: specify type

negative: second half of @anba notes

correct test descriptions to expect ReferenceError only
enforce parse as block statement by adding `;`
remove needless include of $FAIL.js

expect real errors

ch12: negative error matching

ch13 error regexps

ch14 error regexps

add "description" header back

remove now-unused includes

remove needless include
2014-08-13 00:19:11 +01:00
Sam Mikes 4205a1da64 harness: remove unused code
sta.js:
slight change to Test262Error() semantics; message property now always set (default "")
make $ERROR a var
set $ERROR to function $ERROR so it can be overridden if needed
remove 2009 copyright in favor of 2012 copyright
REVERTED: remove never-used fn testFailed
testFailed actually used by $FAIL
remove obsolete fn $INCLUDE

ed.js:
remove commented-out obsolete code

test262.py:
remove always-included harness file that provides no functions used by any extant test
add comment line to nonstrict

cth.js: define `print` for node, cscript

use cth (console test harness) to define functions
for console runner.  V8, Spidermonkey (js) and JavaScriptCore (jsc)
provide a function `print`.  Provide a default `print` for node
and cscript

set print_handle to 'print' by default (can still override)
for cscript: wrap tests in try/catch so we get syntax errors
2014-08-03 00:33:08 +01:00
Sam Mikes c33bf0e043 tools, harness: support new YAML frontmatter
parseTestRecord: add support for YAML frontmatter
parseTestRecord: initial unit test for test record parser
parseTestRecord: refactor for testing

factor old parsing; add YAML parsing

runner: support "includes" from YAML frontmatter

support frontmatter "includes" in python runner
use test.includes if present instead of scanning test code with regex

harness: factor individual functions out into files

tools: handle YAML errors

tolerate missing keys in dictionary (flags, includes)
report filename when empty frontmatter block
new option --list-includes to test262.py

harness: factor helper functions into separate files

sth: remove extra close-paren (syntax error)

test_common: TDD; failing parse of YAML

common: use parseTestRecord (YAML-aware)
2014-07-30 15:39:04 -07:00
Sam Mikes d4a3479a1e test262.py: only include helper scripts when needed
test262.py: only supply async helper scripts when test is async
sth.js: factor out function isAsyncTest()
timer.js: improve workaround for async tests when Promise is defined but setTimeout is noot

timer.js emulates setTimeout using Promise by doing a busy loop that checks
if `timeout` milliseconds have elapsed.  Modified check to (timeLeft > 0) instead
of (!timeLeft) to prevent infinite loop when check does not happen to run
at precise millisecond timeout expires.

Because test262.py did not support the $INCLUDE directive, some helper
scripts were added to every test -- notably testIntl, timer, and donePrintHandle
Now that $INCLUDE is supported, these can be dropped, speeding overall test run time
2014-07-21 15:15:04 +01:00
Brian Terlson 33c8399de8 Merge pull request #31 from Constellation/fix-error-code
Propagate error code when test cases are failed
2014-07-18 11:38:31 -07:00
Brian Terlson 96c6efe0ef Merge pull request #43 from smikes/initial-Promise-tests
Initial promise tests
2014-07-18 11:22:48 -07:00
smikes 7032af4db8 packager: make packager runnable
packager.py cannot run due to syntax errors in a few script files

packagerConfig: use git instead of hg
test262: use pop instead of delete to avoid throw if property missing
S22.1.2.1_T*: fix docString header comment: s/b /**
S22.1.2.1_T3: fix docString header comment: s/b /**, fix end of docstring * / => */
2014-07-18 08:53:29 +01:00
Sam Mikes 7e07cc138d async,promises: initial tests of Promises
doneprintHandle.js: make $DONE accept any falsy argument as meaning 'pass'
PromiseHelper.js: checkSequence: new helper fn for async tests
.gitignore: port .hgignore to .gitignore syntax
test262.py: support $INCLUDE directive in python test runner

S25.4.4.1*: tests to cover Section 25.4.4.1, Promise.all( iterable )
A1.1: Promise.all is callable
A1.2: Promise.all expects 1 argument
A2.1: Promise.all([]) is a Promise
A2.2: Promise.all([]) is resolved immediately
A2.3: Promise.all([]) is resolved with a new empty array
A3.1: Promise.all expects an iterable argument
2014-07-18 05:37:08 +01:00
NikSurya 9aa6b917fa Adding Support for Async Tests 2014-07-15 13:47:59 -07:00
Yusuke Suzuki d83a16f46c Propagate error code when test cases are failed 2014-03-03 03:32:26 +09:00
Prashanth Srinivasan 87605ba3b5 Bug 1472: Adding JUnit Compatible XML and Logfile support for Python test runner 2013-06-10 11:51:24 -04:00
Dan Schaffer 06879c044f Bug 1466: add test262.py parameter to set loglevel 2013-05-10 16:19:32 -04:00
Dan Schaffer 1cb9ed439f Bug 1457: fix for Windows to remove quotes in test262.py 2013-05-10 16:16:51 -04:00
Norbert Lindenberg 46359091c7 Updated test402 main page to reflect that ECMA-402 is now an approved standard.
Fixed a property access in 6.4_c so that the error message shows the correct expected value.
2012-12-17 23:49:06 -08:00
Norbert Lindenberg 1ba4238d9a Made web site packager insert line breaks into JSON files for saner diffs. 2012-09-12 15:09:36 -07:00
Norbert Lindenberg 1af2425075 Added new tests for chapters 6 and 9 of ECMAScript Internationalization API Specification.
- Removed a few old test cases that were redundant with new, more comprehensive ones.
- Added testIntl.js as standard include for all console tests in test262.py – see related bug 574.
- Added .jshintrc file for settings for the JSHint tool.
2012-08-26 20:49:25 -07:00
Norbert Lindenberg 8cad7d03ce Added tests for requirements imposed on the built-in objects of the ECMAScript Internationalization API Specification by the introduction of chapter 15 of the ECMAScript Language Specification.
- Removed some old tests that were redundant with the new tests.
- Added testBuiltInObject.js as standard include for all console tests in test262.py – see related bug 574.
2012-08-26 20:48:49 -07:00
Norbert Lindenberg c3111850e3 Updated existing tests for June 2012 spec changes; removed LocaleList tests; fixed minor issues.
- Removed LocaleList tests; updated other tests so they don't depend on LocaleList.
- Updated tests so they no longer assume that the prototype object of a constructor is an instance of that constructor.
- Updated tests so that jshint is happy.
- Removed @path attributes from test files; updated comment in packager.py explaining why they're unnecessary.
- Removed "use strict" statements, which interfere with strict/non-strict testing.
- Removed testcase functions, which are unnecessary.
2012-08-22 17:53:26 -07:00
Norbert Lindenberg 9f4a513496 Merged changes. 2012-08-22 16:12:54 -07:00
Bill Ticehurst ebdd0a99e6 Fixing Intl402 and BestPractice pages as per bug https://bugs.ecmascript.org/show_bug.cgi?id=611 2012-08-09 12:29:48 -07:00
Norbert Lindenberg 9a4b4011f8 Got test402 web site into usable shape. Unified headers of test262/402 sites.
- Added table of contents for ECMA-402.
- Added ability to load the right table of contents into results display.
- Added meaningful text to test402 web page.
- Updated test262 web pages for 5.1 edition of ECMA-262.
- Unified headers of test262/402 web pages to ECMAScript <area> <test>.
- Replaced text logos with styled text.
- Decapitalized some text; fixed a typo; expanded an abbreviation.
- Fixed a few issues found by W3 HTML validator.
2012-07-18 12:32:54 -07:00
Norbert Lindenberg d371b39cb6 Fixed bug 443: packager.py fails on Mac because of .DS_Store files.
Running packager.py on Mac unfortunately also changes some line terminators in generated files (bug 444), so lots of additional files in this commit.
2012-07-18 12:19:53 -07:00
t-adamre 385b4f7163 Add capability to run chapters individually from the website 2012-06-12 14:54:30 -07:00
David Fugate 87be1bf6ec https://bugs.ecmascript.org/show_bug.cgi?id=65 fixed. 2012-03-28 16:05:03 -07:00
David Fugate 23598a3c67 Renamed AddLicenseHeader.py to FixLicenseHeader.py.
Modified it to switch us over to the Ecma license for everything
except Sputnik-based sources (someone from Google should make
this change on line 60 of FixLicenseHeader.py).
2012-03-01 11:06:50 -08:00
David Fugate 0612ad7bb3 Added a LICENSE file to the root of the enlistment.
All Microsoft-contributed *.js and *.py sources now have Ecma's copyright header.
2012-02-29 14:23:47 -08:00
David Fugate c3b41309c7 Made it clear on testcases_bestPractice.html that it's not the same as default.html. 2012-02-29 09:02:11 -08:00
David Fugate 4fb5071cd4 Migrated latest external\contributions\Microsoft\ietcLatest changes over to test\suite\*. 2012-02-27 15:46:23 -08:00
David Fugate 394e82c6c2 Fixed more license headers. 2012-02-08 15:33:30 -08:00
David Fugate e0cf97e344 A conversion tool ripped Microsoft license headers out of globally scoped tests. Fixed 2012-02-07 09:34:36 -08:00
David Fugate 0ff258fa1d Experimental test402 (internationalization testing) page added. 2012-01-25 14:29:00 -08:00
David Fugate 279825ea44 Added experimental test402 support (JS internationalization standard). 2012-01-25 14:04:40 -08:00
David Fugate d7619456fa Generate one *.html per *.json in addition to the default.html which includes all tests.
Added support to create testcases_bestPractice.html without also adding them to default.html.

Preliminary support for tests under test/suite/internationalization has been added.
2012-01-19 16:03:21 -08:00
David Fugate 43b58c99d0 Preserve only the first line of each test case's copyright header
when encoding to JSON which coincidentally preserves the BOM as well.
This shaved ~10 megs off of chapter15.json alone.
2011-11-04 12:04:46 -07:00
David Fugate 6d83f405b8 The Sputnik/IETC => test262 converter is no longer useful. 2011-11-04 11:16:13 -07:00
David Fugate edd1628de5 FindTestCaseIssues.py is a new helper script which can be used to detect issues with the format of test cases (e.g., runTestCase not being called).
FixPathsAndIds.py now preserves line endings and the BOM.
2011-11-04 11:13:23 -07:00
David Fugate dc9bd9601b Fewer Json files (experiment...). 2011-10-03 16:01:47 -07:00