Since 2017 the test suite has grown significantly and 50k is a significant number!
The command `git grep -lF '/*---' test | wc -l` found 50568 tests.
A count in May (commit 5c9ff876dcefd19a27ef735224bd69ac1d1ee8e9) is used so that the past tense still works.
I've developed a library for running and generating (currently) NUnit based test suites for test262. This is already being successfully used in two .NET projects, Jint (JS interpreter) and esprima-dotnet (JS parsing library).
Prior to this commit, the term "harness" was used to refer both to the
like-named directory in the repository and the external projects which
consume the tests. Reserve the term "harness" to refer solely to the
files maintained within this project's `harness/` directory, and use
the term "runner" to refer to the external projects.
Prior to this commit, the term "helper" was not defined and therefore
introduced ambiguity in text that described harness files. Replace the
term with more explicit wording in terms of the harness files.
Because test runners have some leeway in how they include harness files,
the requirement that "[t]he test source code must not be modified in any
way" does not necessarily preclude the use of harness files. Update the
instructions for interpreting the `raw` frontmatter flag to explicitly
disallow harness files.
Currently, we direct the readers to `compare` page. Which is odd, since readers
hope to land on the branch's page. Let's fix this by changing the link to point
to the branch's page.
The project's CONTRIBUTING.md was written with test authors in mind. It
contains details on non-technical metadata (e.g. "author" and "es6id"),
helper function usage, and preferred code structure. In addition, it
elides certain low-level technical details on the requirements for the
runtime environment.
Introduce a new document targeted towards those executing the tests.
Formalize all expectations regarding how the runtime environment should
be defined, how metadata should be interpreted, and how results should
be understood. This information has overlap with the CONTRIBUTING.md
file, but it also contains details that are irrelevant to test authors.
This document can serve as a more formal contract between Test262 and
the implementors who consume it. This allows Test262 to unambiguously
document future modifications to the formal requirements which in turn
supports consumers who maintain their own test harnesses.
CONTRIBUTING.md
- document `timeout` tag
- reorder tags in frontmatter doc
- minor cleanups
- minor fixes
- add style note
- reformat flags
- remove discussion of obsolete $INCLUDE
- incorporate line notes from @domenic
- integrate additional comments
- add links back, move arg notes down
- Raise outline level by one
README.md
- link to CONTRIBUTING
