add test to repro #114
fix issue #114
factor out PercentFormat function
new fn write; start factoring summary output
fix tests to match
consolidate "Failed Tests" message
add test for successful run
consolidate all_succeeded message
consolidate summary line
consolidate ntests
failing negative tests
squeeze whitespace
parseTestRecord.py:
- recover from ImportError when YAML not defined
- use monkeyYaml as backup
monkeyYaml.py:
- parser for subset of YAML used in test262 frontmatter
test_monkeyYaml.py:
- unit tests
pilot directory for replacing flags: [negative] with
negative: errorname
ch07-7.2,7.3: add error names
ch07-rest: insert error names rest of ch07
ch08: add expected error name
ch11: add error name to negative
ch12: put error name in negative
ch13: add error name to negative
only one test, did it manually
ch15: add error name to negative:
ch07: add error name to negative
these tests used flags: \n - negative and so
were not caught by the earlier naive grep for flags: [negative]
--- changes suggested by @anba
bestPractice - remove added NotEarlyError
ch07,ch11 - expect ReferenceError (req'd by ES6)
correct test descriptions to expect ReferenceError only
enforce parsing as block statement by adding `;`
remove needless include of $FAIL.js
effectively, a rebase of all changes onto master
Merge branch 'negative-addErrorName-work' into negative-addErrorName-bestPractice
Merge branch 'saved-bestPractice-negative' into negative-addErrorName-work
Conflicts:
test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T2.js
test/suite/ch12/12.1/S12.1_A4_T1.js
test/suite/ch12/12.5/S12.5_A2.js
incorporate changes made on PR branch
re-remove added NotEarlyError
Merge branch 'negative-addErrorName-work' into negative-addErrorName-bestPractice
Conflicts:
test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T2.js
test/suite/ch12/12.1/S12.1_A4_T1.js
test/suite/ch12/12.5/S12.5_A2.js
ch15: correct negative regexes
ch10: avoid use of not-NotEarlyError regex
ch14 error regexps
ch13 error regexps
ch12: negative error matching
Merge branch 'console-runner-checkError' into negative-addErrorName-work
test262.py: check negative tests with regex
implement checking of negative tests =>
negative: SyntaxError
means that /SyntaxError/ must match stderr
or test is reported as failure
Fixes#78
expect real errors
bestPractice: supply error names to negative
pilot directory for replacing flags: [negative] with
negative: errorname
ch07-7.2,7.3: add error names
ch07-rest: insert error names rest of ch07
ch08: add expected error name
ch11: add error name to negative
ch12: put error name in negative
ch13: add error name to negative
only one test, did it manually
ch15: add error name to negative:
ch07: add error name to negative
these tests used flags: \n - negative and so
were not caught by the earlier naive grep for flags: [negative]
--- changes suggested by @anba
bestPractice - remove added NotEarlyError
ch07,ch11 - expect ReferenceError (req'd by ES6)
correct test descriptions to expect ReferenceError only
enforce parsing as block statement by adding `;`
remove needless include of $FAIL.js
Merge branch 'negative-addErrorName-work' into negative-addErrorName-bestPractice
resolved Conflicts:
test/suite/ch07/7.9/S7.9_A5.7_T1.js
test/suite/ch11/11.13/11.13.1/S11.13.1_A2.1_T3.js
test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T1.js
test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T10.js
test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T11.js
test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T2.js
test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T3.js
test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T4.js
test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T5.js
test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T6.js
test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T7.js
test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T8.js
test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T9.js
test/suite/ch11/11.3/11.3.1/S11.3.1_A2.1_T3.js
test/suite/ch11/11.3/11.3.2/S11.3.2_A2.1_T3.js
test/suite/ch11/11.4/11.4.4/S11.4.4_A2.1_T3.js
test/suite/ch11/11.4/11.4.5/S11.4.5_A2.1_T3.js
test/suite/ch12/12.5/S12.5_A2.js
test/suite/ch12/12.6/12.6.3/S12.6.3_A4_T2.js
negative: second half of @anba notes
correct test descriptions to expect ReferenceError only
enforce parse as block statement by adding `;`
remove needless include of $FAIL.js
ch12: specify type
negative: @anba fixes
bestPractice - remove added NotEarlyError
ch07,ch11 - expect ReferenceError (req'd by ES6)
bestPractice: supply error names to negative
pilot directory for replacing flags: [negative] with
negative: errorname
ch07-7.2,7.3: add error names
ch07-rest: insert error names rest of ch07
ch08: add expected error name
ch11: add error name to negative
ch12: put error name in negative
ch13: add error name to negative
only one test, did it manually
ch15: add error name to negative:
ch07: add error name to negative
these tests used flags: \n - negative and so
were not caught by the earlier naive grep for flags: [negative]
expect a SyntaxError
ch07: add error name to negative
these tests used flags: \n - negative and so
were not caught by the earlier naive grep for flags: [negative]
ch15: add error name to negative:
ch13: add error name to negative
only one test, did it manually
ch12: put error name in negative
ch11: add error name to negative
ch08: add expected error name
ch07-rest: insert error names rest of ch07
ch07-7.2,7.3: add error names
negative-errorname: pilot
pilot directory for replacing flags: [negative] with
negative: errorname
bestPractice: supply error names to negative
bestPractice: supply error names to negative
pilot directory for replacing flags: [negative] with
negative: errorname
ch07-7.2,7.3: add error names
ch07-rest: insert error names rest of ch07
ch08: add expected error name
ch11: add error name to negative
ch12: put error name in negative
ch13: add error name to negative
only one test, did it manually
ch15: add error name to negative:
ch07: add error name to negative
these tests used flags: \n - negative and so
were not caught by the earlier naive grep for flags: [negative]
bestPractice: supply error names to negative
negative-errorname: pilot
pilot directory for replacing flags: [negative] with
negative: errorname
ch07-7.2,7.3: add error names
ch07-rest: insert error names rest of ch07
ch08: add expected error name
ch11: add error name to negative
ch12: put error name in negative
ch13: add error name to negative
only one test, did it manually
ch15: add error name to negative:
ch07: add error name to negative
these tests used flags: \n - negative and so
were not caught by the earlier naive grep for flags: [negative]
expect a SyntaxError
negative: @anba fixes
bestPractice - remove added NotEarlyError
ch07,ch11 - expect ReferenceError (req'd by ES6)
ch12: specify type
negative: second half of @anba notes
correct test descriptions to expect ReferenceError only
enforce parse as block statement by adding `;`
remove needless include of $FAIL.js
expect real errors
ch12: negative error matching
ch13 error regexps
ch14 error regexps
add "description" header back
remove now-unused includes
remove needless include
sta.js:
slight change to Test262Error() semantics; message property now always set (default "")
make $ERROR a var
set $ERROR to function $ERROR so it can be overridden if needed
remove 2009 copyright in favor of 2012 copyright
REVERTED: remove never-used fn testFailed
testFailed actually used by $FAIL
remove obsolete fn $INCLUDE
ed.js:
remove commented-out obsolete code
test262.py:
remove always-included harness file that provides no functions used by any extant test
add comment line to nonstrict
cth.js: define `print` for node, cscript
use cth (console test harness) to define functions
for console runner. V8, Spidermonkey (js) and JavaScriptCore (jsc)
provide a function `print`. Provide a default `print` for node
and cscript
set print_handle to 'print' by default (can still override)
for cscript: wrap tests in try/catch so we get syntax errors
parseTestRecord: add support for YAML frontmatter
parseTestRecord: initial unit test for test record parser
parseTestRecord: refactor for testing
factor old parsing; add YAML parsing
runner: support "includes" from YAML frontmatter
support frontmatter "includes" in python runner
use test.includes if present instead of scanning test code with regex
harness: factor individual functions out into files
tools: handle YAML errors
tolerate missing keys in dictionary (flags, includes)
report filename when empty frontmatter block
new option --list-includes to test262.py
harness: factor helper functions into separate files
sth: remove extra close-paren (syntax error)
test_common: TDD; failing parse of YAML
common: use parseTestRecord (YAML-aware)
test262.py: only supply async helper scripts when test is async
sth.js: factor out function isAsyncTest()
timer.js: improve workaround for async tests when Promise is defined but setTimeout is noot
timer.js emulates setTimeout using Promise by doing a busy loop that checks
if `timeout` milliseconds have elapsed. Modified check to (timeLeft > 0) instead
of (!timeLeft) to prevent infinite loop when check does not happen to run
at precise millisecond timeout expires.
Because test262.py did not support the $INCLUDE directive, some helper
scripts were added to every test -- notably testIntl, timer, and donePrintHandle
Now that $INCLUDE is supported, these can be dropped, speeding overall test run time
packager.py cannot run due to syntax errors in a few script files
packagerConfig: use git instead of hg
test262: use pop instead of delete to avoid throw if property missing
S22.1.2.1_T*: fix docString header comment: s/b /**
S22.1.2.1_T3: fix docString header comment: s/b /**, fix end of docstring * / => */
doneprintHandle.js: make $DONE accept any falsy argument as meaning 'pass'
PromiseHelper.js: checkSequence: new helper fn for async tests
.gitignore: port .hgignore to .gitignore syntax
test262.py: support $INCLUDE directive in python test runner
S25.4.4.1*: tests to cover Section 25.4.4.1, Promise.all( iterable )
A1.1: Promise.all is callable
A1.2: Promise.all expects 1 argument
A2.1: Promise.all([]) is a Promise
A2.2: Promise.all([]) is resolved immediately
A2.3: Promise.all([]) is resolved with a new empty array
A3.1: Promise.all expects an iterable argument
- Removed a few old test cases that were redundant with new, more comprehensive ones.
- Added testIntl.js as standard include for all console tests in test262.py – see related bug 574.
- Added .jshintrc file for settings for the JSHint tool.
- Removed some old tests that were redundant with the new tests.
- Added testBuiltInObject.js as standard include for all console tests in test262.py – see related bug 574.
- Removed LocaleList tests; updated other tests so they don't depend on LocaleList.
- Updated tests so they no longer assume that the prototype object of a constructor is an instance of that constructor.
- Updated tests so that jshint is happy.
- Removed @path attributes from test files; updated comment in packager.py explaining why they're unnecessary.
- Removed "use strict" statements, which interfere with strict/non-strict testing.
- Removed testcase functions, which are unnecessary.
- Added table of contents for ECMA-402.
- Added ability to load the right table of contents into results display.
- Added meaningful text to test402 web page.
- Updated test262 web pages for 5.1 edition of ECMA-262.
- Unified headers of test262/402 web pages to ECMAScript <area> <test>.
- Replaced text logos with styled text.
- Decapitalized some text; fixed a typo; expanded an abbreviation.
- Fixed a few issues found by W3 HTML validator.
Modified it to switch us over to the Ecma license for everything
except Sputnik-based sources (someone from Google should make
this change on line 60 of FixLicenseHeader.py).
Added support to create testcases_bestPractice.html without also adding them to default.html.
Preliminary support for tests under test/suite/internationalization has been added.
morning. The rewrite of "function testcase()..." now puts the
assertTrue at the bottom. Preconditions, names, and ids are
removed. And the path in the file is ignored, and is instead set
accurately according to where the file is found.
- a lot of JS harness code written in strings have been moved out to actual physical files
such as ed.js (syntax error detection for globally scoped tests) and gs.js (global scope test
case validator). This change makes it far easier to maintain the test harness code
- reorganized helper.js providing a clear indication which methods are used by external objects,
which are implementation details, and which are unequivocally test262-specific. I've also added,
openErrorWindow, which will be used to open a descriptive error message window for each test case
failure reported on the 'Run' tab
- improved the error message for syntax errors occurring when a test case fails to load
- sta.js no longer tries to pickle all helper functions it contains! Instead, we load the file
directly from sth.js. The performance of fnGlobalObject has been improved. Finally, the ES5Harness
object has been moved from sth.js (in a string) to here
- sth.js now has a browser implementer hook, controller.implementerHook, which allows browser implementers
to handle test case failures in their own way (e.g., log to the filesystem). The 'run' function was
basically re-written
Added 37 new test cases from the "IE Test Center" Build release. There were 14 modifications to
existing test cases as well. Refactored SputnikGlobalScope.js such that test case paths are now used
as indices into the GlobalScopeTests array.
TestCasePackager.py had the concept of templated test harnesses introduced - see templates\runner.test262.html.
Also added support for one HTML test harness per ES5 chapter. Last but not least, TestCasePackagerConfig.py
now has a 'source control' abstraction class which abstracts away source control adds|edits when dynamically
generating *.json and *.html test chapters.
tests covering a given chapter, N, without delving into subsections. This in turn
broken the Results page which was by default displaying individual results for chapters
13(.0) and 14(.0). Fixed.
directory next*
SputnikConverter:
- ES5TestCase.cs
* Case of JSON-based property names was wrong. Fixed
* Use the tests' 'id' instead of 'path' as the GlobalScopeTests keys
* Added 'id' and 'path' as Global Scope test metadata. The correct/clean solution here is to simply use 'path'
as the key to GlobalScopeTests, but this refactoring needs to be undertaken later when we can convert the
'normal' test cases over to use 'path' as the key as well
* Turns out we cannot depend on the message received by window.onerror to have some form of "syntax"
contained within it. Instead, we'll just rely on the regular expression ".", matches any one character,
for the short term
- SputnikTestCase.cs
* Same case issue as for ES5TestCases.cs
* Don't trust the Sputnik metadata for the ES5 section name or even test case id to be correct. Instead,
generate this information from the file path of the test case
TestCasePackager.py:
- added a new global, GLOBAL_SCOPE_FILES, which is a list of JS files found directly under test\suite\*.js
which include metadata for so-called globally scoped tests. These files are imported directly by the HTML
test harness
test\suite\*:
- regenerated Sputnik tests based on new converter
default.html:
- import SputnikGlobalScope.js. Really TestCasePackager.py should generate the global scope imports to
default.html automatically...
website\resources\scripts\testcases\*:
- test cases have shuffled from existing *.json files into globalscope.json
- restructured the generated GlobalScope.js file such that the HTML test harness
can import multiple such files without worrying about one blowing aways another's
metadata
- inline with the last change, GlobalScope.js has been renamed to SputnikGlobalScope.js
and generated directly to test\suite\*
- the path and ID properties for most entries in GlobalScope.js were incorrect. That is,
they were based on the metadata contained in the original Sputnik tests and not the
new test262 directories and filenames
- generate global scope tests to test\suite\GlobalScope\* instead of
test\suite\sputnik_converted\GlobalScope\*. Going forward we need to move towards
integrating all test sources into test262 directories (e.g., "GlobalScope"===good;
"Sputnik\GlobalScope"===not-so-good)
- a few hacks to the sources (e.g., private vars to public vars) I'll fix in a refactoring
later. Just trying to get everything working first
Still needs a bit of integration work into the test harness.
A number of Sputnik tests were written outside the context of a chapter's sections. E.g.,
"Chapter 13" versus "Chapter 13, Section 1". We now fake a section number ("13.0" for the
example above) for such cases when generating test case metadata.
their test case objects. Created an automated tool, FixPathsAndIds.py,
to fix these.
TestCasePackager.py no longer throws when it encounters a non-*.js
file where it expects test cases. Instead, it simply ignores it.
rationale behind the addition of this file is it's meant to replace
TestCasePackager.ps1 outright and will be runnable on all major
operating systems. There's still a bit of cleanup work needed as
the Python port was hardly optimized for the language.
A few side effects of this change:
- it now takes under ten seconds to generate the test262 website (PS
version was taking over a minute)
- generated *.json files are ~5% smaller in size (PS was forcing
'\r\n' and the Python version preserves the tests' newlines whether
they're '\r\n' or '\n')
to the following criteria:
1. If a test case directory directly contains *.js files at the root level, package
the entire directory as an XML regardless of how many *.js files are in
subdirectories
2. If the directory and subdirectories contain less than 1,000 test cases, package
the entire directory as an XML. Otherwise...
3. Perform steps 1 and 2 on each subdirectory of the directory
The net effect of this change is that no test case XMLs are larger than 1.5 megs any longer,
although the total number of test XMLs has jumped from about a dozen to ~75. This should
make running the test harness seem a little smoother across all browsers since we won't be
blocked downloading 4 meg files for several seconds any longer.
were actually unique across the test cases. This *was* needed prior to November as we weren't
running each test case in it's own private global environment. The situation now is that
we're running each test within it's own iframe => the modifications are no longer needed.
Few small improvements to SputnikConverter:
- App.config file locations have been fixed
- template files get pushed alongside generated tool binaries
- the root path for Sputnik conformance files is "Conformance", not "tests"
- allow the main exe to throw exceptions so they can be properly debugged with VS
prior to the usage of "ES5Harness". Specifically in cases like:
var abc = ...;
ES5Harness.registerTest( {
//...abc gets used here
the declaration of "abc" would've been ripped out thereby corrupting the test.
This change fixes the issue for all existing IE Test Center and Sputnik test cases. Note
however that it could potentially fail for lines like this: "/*abc*/ /*". If we ever add
cases including comments like this, the implementation of isTestStarted will need to be
revisited.
There were five such test cases that were being affected by this issue in IE Test
Center's chapter 10 and 15 tests. These have all been fixed via the updated script.
- The only parameter required for this script is now the test suite version. All other
parameters are generated relative to the location of this script
- This script now generates the XML'ized test cases directly to website\resources\scripts\testcases\.
Likewise, it copies the test harness taken from test\harness\* to website\resources\scripts\global\
- Added tons of error checking and we now emit the current generation status to stdout
- Made the script more generic in the sense that it can take arbitrary directories under test\suite\*
containing ES5 test chapters. Previous behavior was it was looking for test chapters directly under website\
Brian Terlson
smikes
Domenic Denicola
NikSurya
Yusuke Suzuki
Prashanth Srinivasan
Dan Schaffer
Norbert Lindenberg
Bill Ticehurst
t-adamre
David Fugate
Mark Miller