to the following criteria:
1. If a test case directory directly contains *.js files at the root level, package
the entire directory as an XML regardless of how many *.js files are in
subdirectories
2. If the directory and subdirectories contain less than 1,000 test cases, package
the entire directory as an XML. Otherwise...
3. Perform steps 1 and 2 on each subdirectory of the directory
The net effect of this change is that no test case XMLs are larger than 1.5 megs any longer,
although the total number of test XMLs has jumped from about a dozen to ~75. This should
make running the test harness seem a little smoother across all browsers since we won't be
blocked downloading 4 meg files for several seconds any longer.
- 26 test case modifications. Most of these involved replacing instances 'window' with the
global object
- removal of 47 test cases. The bulk of these were dependent upon 'window' methods or properties
such as 'document' and 'alert'
- addition of 1,234 new test cases. These cover the following ES5.1 sections:
chapter07\7.6\7.6.1
chapter07\7.8\7.8.5
chapter11\11.8\11.8.2
chapter11\11.8\11.8.3
chapter12\12.6\12.6.3
chapter12\12.6\12.6.4
chapter12\12.7
chapter12\12.8
chapter12\12.9
chapter13\13.2
chapter15\15.10\15.10.2
chapter15\15.10\15.10.4
chapter15\15.10\15.10.6\15.10.6.2
chapter15\15.11\15.11.4\15.11.4.4
chapter15\15.1\15.1.1\15.1.1.1
chapter15\15.1\15.1.1\15.1.1.2
chapter15\15.1\15.1.1\15.1.1.3
chapter15\15.1\15.1.2\15.1.2.2
chapter15\15.1\15.1.2\15.1.2.3
chapter15\15.2\15.2.4\15.2.4.2
chapter15\15.3\15.3.4\15.3.4.5.1
chapter15\15.3\15.3.4\15.3.4.5.2
chapter15\15.4\15.4.4\15.4.4.10
chapter15\15.4\15.4.4\15.4.4.12
chapter15\15.4\15.4.4\15.4.4.4
chapter15\15.9\15.9.1\15.9.1.15
were actually unique across the test cases. This *was* needed prior to November as we weren't
running each test case in it's own private global environment. The situation now is that
we're running each test within it's own iframe => the modifications are no longer needed.
Few small improvements to SputnikConverter:
- App.config file locations have been fixed
- template files get pushed alongside generated tool binaries
- the root path for Sputnik conformance files is "Conformance", not "tests"
- allow the main exe to throw exceptions so they can be properly debugged with VS
"?sitedebug" to the end of the Test262 URL.
There's a mismatch between the Test262 test harness and IE Test Center's harness
which invalidated 15.4.4.17-5-1.js, 15.4.4.18-5-1.js, and 15.4.4.20-5-1.js on Test262.
Essentially what's going on is Test262 uses the 'window' object as 'this' for the test
case functions whereas IE Test Center uses the test case object housing the test case
function as this. This causes problems for test cases pertaining to scoping. I've gone
ahead and adjusted the Test262 tests to expect 'window' to be 'this', and will be updating
IE Test Center's harness to use the 'window' object as well so we don't run into this
problem again.
of the tests. Emit this to the activity bar
test\harness\sta.js: too many years of Python had me thinking JavaScript arrays have an append method:) Fixed.
Also, added a pickled representation of all test helper functions found in this file
test\harness\sth.js: detached most test helper functions from the iframe's document object (as globals) and
inject these into the actual test cases. It's a bit slower and not as elegant, but it is
cleaner from an ES5 purist perspective. Still need to move Sputnik helper functions into
sta.js
Extended Controller such that it now measures overall test execution time. Such a change
is very useful for measuring performance-impact changes such as the aforementioned
improvement
- 12.14-7.js: cleaned up
- 15.2.3.3: 89 test case additions
- 15.2.3.4: 29 test case additions
- 15.2.3.6: 1,076 test case additions and 15 minor test case fixes
- 15.2.3.9-1.js: cleaned up
- 15.3.4.5-2-8.js: cleaned up
- 15.4.4.14: 14 test case additions and 3 minor test case fixes
- 15.4.4.15: 9 test case additions and 3 minor test case fixes
- 15.4.4.17: 1 test case additions and 1 minor test case fix
- 15.4.4.18: 1 minor test case fix
- 15.4.4.19: 1 test case addition and 2 minor test case fixes
- 15.4.4.20: 3 minor test case fixes
- 15.4.4.21: 1 test case addition and 1 minor test case fix
- 15.4.4.22: 2 minor test case fixes
- 15.9.4.4: 2 minor test case fixes
- 15.9.5.43-0-2.js: minor test case fix
- test\harness\sth.js: fixed a bug spotted by Mark Miller. In short, we were allowing
a test case return value of 'undefined' to be successful WRT IE Test Center tests because
Sputnik tests normally return undefined (successful or not). We now differentiate between
Sputnik/IE Test Center when evaluating return values
prior to the usage of "ES5Harness". Specifically in cases like:
var abc = ...;
ES5Harness.registerTest( {
//...abc gets used here
the declaration of "abc" would've been ripped out thereby corrupting the test.
This change fixes the issue for all existing IE Test Center and Sputnik test cases. Note
however that it could potentially fail for lines like this: "/*abc*/ /*". If we ever add
cases including comments like this, the implementation of isTestStarted will need to be
revisited.
There were five such test cases that were being affected by this issue in IE Test
Center's chapter 10 and 15 tests. These have all been fixed via the updated script.
- The only parameter required for this script is now the test suite version. All other
parameters are generated relative to the location of this script
- This script now generates the XML'ized test cases directly to website\resources\scripts\testcases\.
Likewise, it copies the test harness taken from test\harness\* to website\resources\scripts\global\
- Added tons of error checking and we now emit the current generation status to stdout
- Made the script more generic in the sense that it can take arbitrary directories under test\suite\*
containing ES5 test chapters. Previous behavior was it was looking for test chapters directly under website\
website\* out to test\*:
- Removed test\harness\ECMA-262-TOC.xml. The casing on this file was incorrect, but
more importantly it's a static file not generated by the harness
- Populated test\harness with the contents of website\resources\scripts\global\. In
the future, we need to update test\harness\* and propagate these changes out to
website\*
- Test\suite\ietestcenter is now a verbatim copy of the IE Test Center tests that
WERE under website\resources\scripts\testcases\*
- Moved all Sputnik tests from website\resources\scripts\testcases\* out to
test\suite\sputnik_converted
- Moved website\resources\scripts\testcases\excludelist.xml out to test\config\*. This
particular file was only used for the test conversion process to XML, and is not actually
needed by the website as best as I can tell
- Website\resources\scripts\testcases now only contains the XMLized test cases. This is
the right thing to do as the *.js files here weren't actually being used by the website
and the general public can now peruse the test cases directly via Mercurial
the Test262 XML conversion tool. These have been manually updated, and a bug needs to
be filed on the tool when bugs.ecmascript.org comes back online.
Fixed an issue involving the Back button on the results page.
which have resulted in it being able to run negative test cases which corrupt the global JavaScript
environment. This lets us enable fifty plus more Sputnik test cases. Another positive result from
this test harness refactoring is that Opera no longer hangs when running the tests.
David Fugate