# Test262 maintenance rationale Explanations behind the practices promoted by the project maintainers. ## Vestigial tests Test262 has been maintained for many years, and the practices used to write tests have evolved alongside the needs of its consumers. When conventions change, old tests are typically updated to accommodate the new practices. That doesn't always happen, though, and one doesn't have to look very far to find examples of tests which contradict the preferred patterns. For instance: - tests which express expectations with `throw` statements inside of conditional statements rather than the assertion API implemented by the harness files (though this explicitness will always be desirable when asserting the semantics of conditional statements and `throw` statements themselves) - tests with file names derived from section numbers in the 5th edition of ECMA262, e.g. `built-ins/Array/15.4.5-1.js` - tests which validate multiple behaviors, using elaborate comment blocks to designate sections - tests which use deprecated harness functions, e.g. `verifyEnumerable`, `verifyConfigurable`, and `verifyWritable` Since existing tests do not necessarily reflect the project's current best-practices, it's especially important for test authors to familiarize themselves with [the contribution guidelines](../CONTRIBUTING.md). ## Test generation The project includes a software tool for generating test material from abstract templates. The generator was designed to promote uniformity of coverage, particularly for parts of the grammar that are used in many productions (specifically: the destructuring assignment patterns introduced in the 6th edition of ECMA262). This tool makes it easy to introduce enormous numbers of tests. Introducing more tests is not a goal unto itself, though. Test262 prioritizes coherence in its coverage of the specification, and it recognizes that the value of tests, as measured by their likelihood to identify defects, varies. For these reasons, the maintainers urge restraint in the application of the test generation tool. ## File structure For practical reasons, tests are organized in a tree structure according to the conventions of modern file systems. Unfortunately, this structure is not expressive enough to model the semantics of a rich and evolving programming language like ECMAScript. This means the common and crucial task of coverage assessment will likely always be a challenge, but strong conventions around file organization can help. Tests for syntax-derived operations are organized according to the language grammar, with directories used to describe non-terminals. For example, tests for [the `if` statement](https://tc39.es/ecma262/#sec-if-statement) are located in [the `tests/language/statements/if` directory](https://github.com/tc39/test262/tree/main/test/language/statements/if), and tests for [the `instanceof` operator](https://tc39.es/ecma262/#sec-relational-operators) are located in [the `tests/language/expressions/instanceof` directory](https://github.com/tc39/test262/tree/main/test/language/expressions/instanceof). Tests for built-in APIs are organized within [the `tests/built-ins` directory](https://github.com/tc39/test262/tree/main/test/intl402) according to the identifiers by which they can be accessed. There, directories describe the sequence of properties that can be used from the global scope. For example, tests for [the `Array.prototype.reduce` method](https://tc39.es/ecma262/#sec-array.prototype.reduce) are located in [the `tests/built-ins/Array/prototype/reduce` directory](https://github.com/tc39/test262/tree/main/test/built-ins/Array/prototype/reduce), while tests for [the `isNaN` function](https://tc39.es/ecma262/#sec-isnan-number) are located in [the `tests/built-ins/isNan` directory(https://github.com/tc39/test262/tree/main/test/built-ins/isNaN)]. Built-ins which are defined only in [the ECMA-402 specification](https://tc39.es/ecma402/) follow a similar naming convention within [the `tests/intl402` directory](https://github.com/tc39/test262/tree/main/test/intl402). [The `tests/annexB` directory](https://github.com/tc39/test262/tree/main/test/annexB) holds tests for the semantics described by [Annex B of ECMA262](https://tc39.es/ecma262/#sec-additional-ecmascript-features-for-web-browsers). The conventions for syntax-derived operations and built-in APIs as described above are also applied within this directory. [The `tests/harness` directory](https://github.com/tc39/test262/tree/main/test/harness) stores tests for the "harness" files which Test262 maintains to assist in test writing. Directories are not generally applied beyond these limits; further differentiation is instead achieved through structured file names which follow ad-hoc conventions. This organization balances the need to group tests logically with the need to discover tests. See, for example, [the tests for template literals](https://github.com/tc39/test262/tree/main/test/language/expressions/template-literal). Many consumers use file names as a way to compare test results across revisions and between implementations. For this reason, tests files are rarely re-organized after being accepted. ## Regression tests It is possible to write tests for semantics which, while not explicitly specified by ECMA262, are nonetheless valid according to the normative text. Such tests are welcome in Test262, but their fitness is not a given. Consumers from many constituencies value the coherence and consistency of the test suite, and tests which disallow arbitrary extraneous behavior can degrade those qualities. Because Test262 is not maintained as a repository of regression tests, contributions which include these kinds of tests will be weighed against their likelihood of identifying error in a plurality of implementations. For example, assume that some runtime spuriously accesses the `toJSON` property of the value passed to [`String.prototype.repeat`](https://tc39.es/ecma262/#sec-string.prototype.repeat). While the maintainers of the engine may decide to include a regression test which disallows that behavior in their project, the maintainers of Test262 would not necessarily accept such a test. ## Large tests Test262 tests are typically very focused. The vast majority exercise just one algorithm step/grammar production, and some are even more granular than that! Some test contributors are uncomfortable splitting their work across files like this. It's certainly unlike the practices that are common in modern application development. In those settings, many tests are often grouped into the same file and separated by function boundaries. Test262 doesn't use the same approach as a typical application test suite in order to limit complexity. The guidance of "one test per file" means that consuming Test262 is relatively easy; there is no "test runner" API for consumers to implement, and interpreting results is likewise straightforward. It also lowers the barrier to entry for new contributors since there is no API to learn. ## Syntax tests When testing a syntactic feature of the language, it can be tempting to write tests which verify only that some bit of source text does *not* produce a syntax error. Contributors should try to push beyond verifying only the lack of a syntax error because such tests also have observable semantics. It's better for a test to assert that the expected semantics are followed. However, verifying semantics invariably requires inserting still more code, and that additional code may degrade the tests' precision for verifying syntax. For cases where this trade-off is significant, contributors may consider submitting simplified tests to [the test262-parser-tests project](https://github.com/tc39/test262-parser-tests). ## Avoiding abstraction Contributors will occasionally suggest introducing new abstractions to reduce duplication in tests. The maintainers set a relatively high bar for such enhancements, both due to their many drawbacks and due to the aspects of standards testing which limit their benefit. The drawbacks to abstraction include: - it degrades the tests by introducing unrelated semantics - it discourages contributors by requiring them to learn more - it frustrates implementers by making it harder to understand what's being tested and what has failed One of abstraction's common motivations is its tendency to reduce maintenance costs by limiting duplication. TC39 has a very high standard for compatibility between revisions of ECMA262. This gives us a certain assurance in Test262 that maintainers of other test suites do not enjoy: Test262's tests are very rarely invalidated. The project takes advantage of this by using a more declarative, readable, and verbose style. Abstraction has other motivations, so there will always be room for it to some extent. When the benefits of a specific proposal outweigh the drawbacks, then it should be well documented and also well-tested. Test262 maintains tests for its "harness" abstractions in [a dedicated directory within the test suite itself](https://github.com/tc39/test262/tree/main/test/harness).