This change is in service of forthcoming tests for the "JSON modules"
language proposal [1]. Verifying the semantics of that proposal requires
modules whose source text is not valid ECMAScript; this change updates
the guidelines for contributing and interpreting tests so that such test
material can be handled consistently.
Differentiating JSON files with a distinct file name suffice will assist
consumers which require special handling of such files (e.g. web
browsers).
Change the pattern used to designate "fixture" files so that it may be
applied to files used for JSON modules.
Increment the project version number to alert consumers of this change
in interpreting instructions.
[1] https://github.com/tc39/proposal-json-modules
Prior to this patch, the CircleCI continuous integration environment was
configured to report test failures in a negative light, displaying red
cross-marks and reporting that "some checks were not successful" in
commits and GitHub Pull Requests which included them.
The passing/failing status of tests does not influence their
desirability for Test262. (In practice, engines very commonly fail
newly-contributed tests.)
Although these conflicting interpretations does not technically
interfere with the maintainers' ability to merge new contributions, it
does create confusion for many contributors who interpreted the UI as a
rejection of their work.
In addition, this behavior made it impossible to distinguish between the
benign test failures and disruptive infrastructural problems (e.g. the
crashing of engines).
Reconfigure the continuous integration environment to accept passing and
failing tests equally, and to only report a problem when the Test262
project's testing infrastructure behaves unexpectedly.
Recent changes to the metadata for negative tests fundamentally alter
the way those tests are intended to be interpreted. Increment the
project's version number in order to signal this to project consumers.
Changes to the instructions for interpreting tests will likely produce
new failures for consumers who are updating between revisions of
Test262. Introduce a machine-readable convention for signaling
substantive changes.
Philip Chimento
Mike Pennisi
Leo Balter
Rick Waldron
Gus Caplan
Richard Gibson