Verify that every test file which references a harness file using the
"includes" directive also contains at least one reference to a value
defined in the harness file.
To support this check, extend each harness file with a list of values
which it defines.
Update the lint.py script to work with pip 10+ as pip.req was moved to
pip._internal.req in version 10 onwards and the existing code only works
on versions of pip under or equal to 9.0.3
Previously, test consumers were encouraged to insert a `throw` statement
as the first statement of tests for early errors. This recommendation
made tests harder to consume, and as an optional transformation,
consumers may have ignored it or simply been unaware it was made. By
explicitly including such a `throw` statement, the tests become more
literal, making them easier to consume and more transparent in their
expectations.
Document expectation for all tests for early errors to include an
explicit `throw` statement. Extend linting script to verify that
contributors are automatically notified of violations and to ensure that
future contributions satisfy this expectation.
A recent commit introduced a document that enumerated acceptable values
for the test "features" metadata tag. However, this list was incomplete,
and maintaining it placed extra burden on the project owners.
Restructure the document into a machine-readable format. Add entries for
all previously-omitted values. Add in-line documentation with
recommendations for maintenance of the file. Extend the project's
linting tool to validate tests according to the document's contents.
This script is intended to identify common test file formatting errors
prior to their acceptance into the project. It is designed to support
future extensions for additional validation rules.
Gus Caplan
Mike Pennisi
Ms2ger
André Bargull
Leo Balter
Ujjwal Sharma
Rick Waldron
Valerie R Young
Adrian Heine