notepad-plus-plus/lexilla/test

README for testing lexers with lexilla/test.

The TestLexers application is run to test the lexing and folding of a set of example
files and thus ensure that the lexers are working correctly.

Lexers are accessed through the Lexilla shared library which must be built first
in the lexilla/src directory.

TestLexers works on Windows, Linux, or macOS and requires a C++20 compiler.
MSVC 2019.4, GCC 9.0, Clang 9.0, and Apple Clang 11.0 are known to work.

MSVC is only available on Windows.

GCC and Clang work on Windows and Linux.

On macOS, only Apple Clang is available.

Lexilla requires some headers from Scintilla to build and expects a directory named
"scintilla" containing a copy of Scintilla 5+ to be a peer of the Lexilla top level
directory conventionally called "lexilla".

To use GCC run lexilla/test/makefile:
	make test

To use Clang run lexilla/test/makefile:
	make CLANG=1 test
On macOS, CLANG is set automatically so this can just be
	make test

To use MSVC:
	nmake -f testlexers.mak test
There is also a project file TestLexers.vcxproj that can be loaded into the Visual
C++ IDE.



Adding or Changing Tests

The lexilla/test/examples directory contains a set of tests located in a tree of
subdirectories.

Each directory contains example files along with control files called
SciTE.properties and expected result files with .styled and .folded suffixes.
If an unexpected result occurs then files with the additional suffix .new 
(that is .styled.new or .folded.new) may be created.

Each file in the examples tree that does not have an extension of .properties, .styled,
.folded or .new is an example file that will be lexed and folded according to settings
found in SciTE.properties.

The results of the lex will be compared to the corresponding .styled file and if different
the result will be saved to a .styled.new file for checking.
So, if x.cxx is the example, its lexed form will be checked against x.cxx.styled and a
x.cxx.styled.new file may be created. The .styled.new and .styled files contain the text
of the original file along with style number changes in {} like:
	{5}function{0} {11}first{10}(){0}
After checking that the .styled.new file is correct, it can be promoted to .styled and
committed to the repository.

The results of the fold will be compared to the corresponding .folded file and if different
the result will be saved to a .folded.new file for checking.
So, if x.cxx is the example, its folded form will be checked against x.cxx.folded and a
x.cxx.folded.new file may be created. The folded.new and .folded files contain the text
of the original file along with fold information to the left like:

 2 400   0 + --[[ coding:UTF-8
 0 402   0 | comment ]]

There are 4 columns before the file text representing the bits of the fold level:
[flags (0xF000), level (0x0FFF), other (0xFFFF0000), picture].
flags: may be 2 for header or 1 for whitespace.
level: hexadecimal level number starting at 0x400. 'negative' level numbers like 0x3FF
indicate errors in either the folder or in the input file, such as a C file that starts with #endif.
other: can be used as the folder wants. Often used to hold the level of the next line.
picture: gives a rough idea of the fold structure: '|' for level greater than 0x400,
'+' for header, ' ' otherwise.
After checking that the .folded.new file is correct, it can be promoted to .folded and
committed to the repository.

An interactive file comparison program like WinMerge (https://winmerge.org/) on
Windows or meld (https://meldmerge.org/) on Linux can help examine differences
between the .styled and .styled.new files or .folded and .folded.new files.

On Windows, the scripts/PromoteNew.bat script can be run to promote all .new result
files to their base names without .new.

Styling and folding tests are first performed on the file as a whole, then the file is lexed
and folded line-by-line. If there are differences between the whole file and line-by-line
then a message with 'per-line is different' for styling or 'per-line has different folds' will be
printed. Problems with line-by-line processing are often caused by local variables in the
lexer or folder that are incorrectly initialised. Sometimes extra state can be inferred, but it
may have to be stored between runs (possibly with SetLineState) or the code may have to
backtrack to a previous safe line - often something like a line that starts with a character
in the default style.

The SciTE.properties file is similar to properties files used for SciTE but are simpler.
The lexer to be run is defined with a lexer.{filepatterns} statement like:
	lexer.*.d=d

Keywords may be defined with keywords settings like:
	keywords.*.cxx;*.c=int char
	keywords2.*.cxx=open

Substyles and substyle identifiers may be defined with settings like:
	substyles.cpp.11=1
	substylewords.11.1.*.cxx=map string vector

Other settings are treated as lexer or folder properties and forwarded to the lexer/folder:
	lexer.cpp.track.preprocessor=1
	fold=1

It is often necessary to set 'fold' in SciTE.properties to cause folding.

Properties can be set for a particular file with an "if $(=" or "match" expression like so:
if $(= $(FileNameExt);HeaderEOLFill_1.md)
    lexer.markdown.header.eolfill=1
match Header*1.md
    lexer.markdown.header.eolfill=1

More complex tests with additional configurations of keywords or properties can be performed
by creating another subdirectory with the different settings in a new SciTE.properties.

There is some support for running benchmarks on lexers and folders. The properties
testlexers.repeat.lex and testlexers.repeat.fold specify the number of times example
documents are lexed or folded. Set to a large number like testlexers.repeat.lex=10000
then run with a profiler.

A list of styles used in a lex can be displayed with testlexers.list.styles=1.