Updated to Scintilla 5.4.1 & Lexilla 5.3.0

Scintilla 5.4.1
https://www.scintilla.org/scintilla541.zip
Released 27 December 2023.

1.  Add IDocumentEditable interface to allow efficient interaction with document objects which may not be visible in a Scintilla instance. This feature is provisonal and may change before being declared stable. For better type-safety, the ScintillaCall C++ API uses IDocumentEditable* where void* was used before which may require changes to client code that uses document pointer APIs DocPointer, SetDocPointer, CreateDocument, AddRefDocument, and ReleaseDocument.
2.  Ctrl-click on a selection deselects it in multiple selection mode.
3.  Add SCI_SELECTIONFROMPOINT for modifying multiple selections.
4.  Add SCI_SETMOVEEXTENDSSELECTION and SCI_CHANGESELECTIONMODE to simplify selection mode manipulation.
5.  Improve performance of global replace by reducing cache invalidation overhead. [Feature #1502](https://sourceforge.net/p/scintilla/feature-requests/1502/).
6.  Fix regular expression search for "\<" matching beginning of search when not beginning of word and for "\>" not matching line end. [Bug #2157](https://sourceforge.net/p/scintilla/bugs/2157/).
7.  Fix regular expression search failure when search for "\<" followed by search for "\>". [Bug #2413](https://sourceforge.net/p/scintilla/bugs/2413/).
8.  Fix regular expression assertion (^, $, \b. \B) failures when using SCFIND_CXX11REGEX. [Bug #2405](https://sourceforge.net/p/scintilla/bugs/2405/).
9.  Fix regular expression bug in reverse direction where shortened match returned. [Bug #2405](https://sourceforge.net/p/scintilla/bugs/2405/).
10. Avoid character fragments in regular expression search results. [Bug #2405](https://sourceforge.net/p/scintilla/bugs/2405/).
11. With a document that does not have the SC_DOCUMENTOPTION_TEXT_LARGE option set, allocating more than 2G (calling SCI_ALLOCATE or similar) will now fail with SC_STATUS_FAILURE.
12. Protect SCI_REPLACETARGET, SCI_REPLACETARGETMINIMAL, and SCI_REPLACETARGETRE from application changing target in notification handlers. [Bug #2289](https://sourceforge.net/p/scintilla/bugs/2289/).

Lexilla 5.3.0
https://www.scintilla.org/lexilla530.zip
Released 27 December 2023.

1. Fix calling AddStaticLexerModule by defining as C++ instead of C which matches header. [Bug #2421](https://sourceforge.net/p/scintilla/bugs/2421/).
2. Bash: Fix shift operator << incorrectly recognized as here-doc. [Issue #215](https://github.com/ScintillaOrg/lexilla/issues/215).
3. Bash: Fix termination of '${' with first unquoted '}' instead of nesting. [Issue #216](https://github.com/ScintillaOrg/lexilla/issues/216).
4. HTML: JavaScript double-quoted strings may escape line end with '\'. [Issue #214](https://github.com/ScintillaOrg/lexilla/issues/214).
5. Lua: recognize --- doc comments. Defined by [LDoc](https://github.com/lunarmodules/ldoc). Does not recognize --[[-- doc comments which seem less common.

Close #14375

lexilla/.gitignore

@@ -27,6 +27,7 @@ __pycache__
 *.bsc
 *.intermediate.manifest
 *.lastbuildstate
+*.nativecodeanalysis.xml
 *.cache
 *.ilk
 *.ncb
@@ -34,13 +35,13 @@ __pycache__
 *.sdf
 gtk/*.plist
 win32/*.plist
-*.nativecodeanalysis.xml
 *.opt
 *.plg
 *.pbxbtree
 *.mode1v3
 *.pbxuser
 *.pbproj
+*.tgz
 *.log
 *.xcbkptlist
 *.xcuserstate
@@ -49,8 +50,8 @@ xcuserdata/
 xcschememanagement.plist
 .DS_Store
 test/TestLexers
-Debug
 Release
+Debug
 x64
 ARM64
 cocoa/build
@@ -58,8 +59,9 @@ cocoa/ScintillaFramework/build
 cocoa/ScintillaTest/build
 macosx/SciTest/build
 *.cppcheck
+*.pro.user
+.qmake.stash
 cov-int
-*.tgz
 .vs
 meson-private
 meson-logs

lexilla/doc/Lexilla.html

@@ -9,7 +9,7 @@
     <meta name="keywords" content="Scintilla, SciTE, Editing Component, Text Editor" />
     <meta name="Description"
     content="www.scintilla.org is the home of the Scintilla editing component and SciTE text editor application." />
-    <meta name="Date.Modified" content="20231105" />
+    <meta name="Date.Modified" content="20231227" />
     <meta name="viewport" content="width=device-width, initial-scale=1" />
     <style type="text/css">
         .logo {
@@ -61,8 +61,8 @@
           <font color="#FFCC99" size="4"> A library of language lexers for use with Scintilla</font>
         </td>
         <td width="40%" align="right">
-          <font color="#FFCC99" size="3">Release version 5.2.8<br />
+          <font color="#FFCC99" size="3">Release version 5.3.0<br />
-           Site last modified November 5 2023</font>
+           Site last modified December 27 2023</font>
         </td>
         <td width="20%">
           &nbsp;
@@ -77,11 +77,11 @@
       </tr>
     </table>
     <ul id="versionlist">
+      <li>Version 5.3.0 improves Bash, HTML, and Lua.</li>
+      <li>Version 5.2.9 fixes potential problems on macOS 12 and older when built with Xcode 15.0.</li>
       <li>Version 5.2.8 improves Python and R.</li>
       <li>Version 5.2.7 improves Bash, F#, and HTML.</li>
       <li>Version 5.2.6 improves Bash, Errorlist, HTML, Matlab, and Visual Prolog.</li>
-      <li>Version 5.2.5 improves Bash, Batch, F#, and VB.</li>
-      <li>Version 5.2.4 improves C++ and GDScript.</li>
     </ul>
     <ul id="menu">
       <li id="remote1"><a href="https://www.scintilla.org/SciTEImage.html">Screenshot</a></li>

lexilla/doc/LexillaDownload.html

@@ -26,9 +26,9 @@
     <table bgcolor="#CCCCCC" width="100%" cellspacing="0" cellpadding="8" border="0">
       <tr>
         <td>
-          <font size="4"> <a href="https://www.scintilla.org/lexilla528.zip">
+          <font size="4"> <a href="https://www.scintilla.org/lexilla530.zip">
 	Windows</a>&nbsp;&nbsp;
-	<a href="https://www.scintilla.org/lexilla528.tgz">
+	<a href="https://www.scintilla.org/lexilla530.tgz">
           GTK/Linux</a>&nbsp;&nbsp;
 	</font>
         </td>
@@ -42,7 +42,7 @@
       containing very few restrictions.
     </p>
     <h3>
-       Release 5.2.8
+       Release 5.3.0
     </h3>
     <h4>
        Source Code
@@ -50,8 +50,8 @@
        The source code package contains all of the source code for Lexilla but no binary
 	executable code and is available in
        <ul>
-       <li><a href="https://www.scintilla.org/lexilla528.zip">zip format</a> (1.3M) commonly used on Windows</li>
+       <li><a href="https://www.scintilla.org/lexilla530.zip">zip format</a> (1.3M) commonly used on Windows</li>
-       <li><a href="https://www.scintilla.org/lexilla528.tgz">tgz format</a> (0.9M) commonly used on Linux and compatible operating systems</li>
+       <li><a href="https://www.scintilla.org/lexilla530.tgz">tgz format</a> (0.9M) commonly used on Linux and compatible operating systems</li>
        </ul>
        Instructions for building on both Windows and Linux are included in the readme file.
     <h4>

lexilla/doc/LexillaHistory.html

@@ -583,9 +583,50 @@
       </tr><tr>
 	<td>German Aizek</td>
 	<td>Tsuyoshi Miyake</td>
+	<td>Martin Schäfer</td>
     </tr>
     </table>
     <h2>Releases</h2>
+    <h3>
+       <a href="https://www.scintilla.org/lexilla530.zip">Release 5.3.0</a>
+    </h3>
+    <ul>
+	<li>
+	Released 27 December 2023.
+	</li>
+	<li>
+	Fix calling AddStaticLexerModule by defining as C++ instead of C which matches header.
+	<a href="https://sourceforge.net/p/scintilla/bugs/2421/">Bug #2421</a>.
+	</li>
+	<li>
+	Bash: Fix shift operator &lt;&lt; incorrectly recognized as here-doc.
+	<a href="https://github.com/ScintillaOrg/lexilla/issues/215">Issue #215</a>.
+	</li>
+	<li>
+	Bash: Fix termination of '${' with first unquoted '}' instead of nesting.
+	<a href="https://github.com/ScintillaOrg/lexilla/issues/216">Issue #216</a>.
+	</li>
+	<li>
+	HTML: JavaScript double-quoted strings may escape line end with '\'.
+	<a href="https://github.com/ScintillaOrg/lexilla/issues/214">Issue #214</a>.
+	</li>
+	<li>
+	Lua: recognize --- doc comments.
+	Defined by <a href="https://github.com/lunarmodules/ldoc">LDoc</a>.
+	Does not recognize --[[-- doc comments which seem less common.
+	</li>
+    </ul>
+    <h3>
+       <a href="https://www.scintilla.org/lexilla529.zip">Release 5.2.9</a>
+    </h3>
+    <ul>
+	<li>
+	Released 18 November 2023.
+	</li>
+	<li>
+	Xcode build settings changed to avoid problems with Xcode 15.0.
+	</li>
+    </ul>
     <h3>
        <a href="https://www.scintilla.org/lexilla528.zip">Release 5.2.8</a>
     </h3>

lexilla/include/Lexilla.h

@@ -48,20 +48,20 @@ using Scintilla::ILexer5;
 typedef void ILexer5;
 #endif
 
-typedef ILexer5 *(*LexerFactoryFunction)();
+typedef ILexer5 *(*LexerFactoryFunction)(void);
 
 #if defined(__cplusplus)
 namespace Lexilla {
 #endif
 
-typedef int (LEXILLA_CALL *GetLexerCountFn)();
+typedef int (LEXILLA_CALL *GetLexerCountFn)(void);
 typedef void (LEXILLA_CALL *GetLexerNameFn)(unsigned int Index, char *name, int buflength);
 typedef LexerFactoryFunction(LEXILLA_CALL *GetLexerFactoryFn)(unsigned int Index);
 typedef ILexer5*(LEXILLA_CALL *CreateLexerFn)(const char *name);
 DEPRECATE_DEFINITION typedef const char *(LEXILLA_CALL *LexerNameFromIDFn)(int identifier);
-typedef const char *(LEXILLA_CALL *GetLibraryPropertyNamesFn)();
+typedef const char *(LEXILLA_CALL *GetLibraryPropertyNamesFn)(void);
 typedef void(LEXILLA_CALL *SetLibraryPropertyFn)(const char *key, const char *value);
-typedef const char *(LEXILLA_CALL *GetNameSpaceFn)();
+typedef const char *(LEXILLA_CALL *GetNameSpaceFn)(void);
 
 #if defined(__cplusplus)
 }
@@ -85,13 +85,13 @@ extern "C" {
 #endif
 
 ILexer5 * LEXILLA_CALL CreateLexer(const char *name);
-int LEXILLA_CALL GetLexerCount();
+int LEXILLA_CALL GetLexerCount(void);
 void LEXILLA_CALL GetLexerName(unsigned int index, char *name, int buflength);
 LexerFactoryFunction LEXILLA_CALL GetLexerFactory(unsigned int index);
 DEPRECATE_DEFINITION const char *LEXILLA_CALL LexerNameFromID(int identifier);
-const char * LEXILLA_CALL GetLibraryPropertyNames();
+const char * LEXILLA_CALL GetLibraryPropertyNames(void);
 void LEXILLA_CALL SetLibraryProperty(const char *key, const char *value);
-const char *LEXILLA_CALL GetNameSpace();
+const char *LEXILLA_CALL GetNameSpace(void);
 
 #if defined(__cplusplus)
 }

lexilla/lexers/LexBash.cxx

@@ -333,7 +333,7 @@ public:
 	}
 	bool CountDown(StyleContext &sc, CmdState &cmdState) {
 		Current.Count--;
-		if (Current.Count == 1 && sc.Match(')', ')')) {
+		while (Current.Count > 0 && sc.chNext == Current.Down) {
 			Current.Count--;
 			sc.Forward();
 		}
@@ -380,10 +380,6 @@ public:
 					sc.ChangeState(SCE_SH_BACKTICKS);
 				}
 			}
-			if (current == CmdState::Body && sc.Match('(', '(') && state == SCE_SH_DEFAULT && Depth == 0) {
-				// optimized to avoid track nested delimiter pairs
-				style = QuoteStyle::Literal;
-			}
 		} else {
 			// scalar has no delimiter pair
 			if (!setParamStart.Contains(sc.ch)) {
@@ -939,7 +935,9 @@ void SCI_METHOD LexerBash::Lex(Sci_PositionU startPos, Sci_Position length, int
 						continue;
 					}
 				} else if (sc.ch == QuoteStack.Current.Up) {
+					if (QuoteStack.Current.Style != QuoteStyle::Parameter) {
 						QuoteStack.Current.Count++;
+					}
 				} else {
 					if (QuoteStack.Current.Style == QuoteStyle::String ||
 						QuoteStack.Current.Style == QuoteStyle::HereDoc ||

lexilla/lexers/LexHTML.cxx

@@ -2174,8 +2174,10 @@ void SCI_METHOD LexerHTML::Lex(Sci_PositionU startPos, Sci_Position length, int
 				state = SCE_HJ_DEFAULT;
 			} else if (isLineEnd(ch)) {
 				styler.ColourTo(i - 1, StateToPrint);
+				if (chPrev != '\\' && (chPrev2 != '\\' || chPrev != '\r' || ch != '\n')) {
 					state = SCE_HJ_STRINGEOL;
 				}
+			}
 			break;
 		case SCE_HJ_SINGLESTRING:
 			if (ch == '\\') {

lexilla/lexers/LexLua.cxx

@@ -7,15 +7,13 @@
  ** Modified by Marcos E. Wurzius & Philippe Lhoste
  **/
 
-#include <stdlib.h>
+#include <cstdlib>
-#include <string.h>
+#include <cassert>
-#include <stdio.h>
+#include <cstring>
-#include <stdarg.h>
-#include <assert.h>
-#include <ctype.h>
 
 #include <string>
 #include <string_view>
+#include <map>
 
 #include "ILexer.h"
 #include "Scintilla.h"
@@ -28,13 +26,18 @@
 #include "StyleContext.h"
 #include "CharacterSet.h"
 #include "LexerModule.h"
+#include "OptionSet.h"
+#include "DefaultLexer.h"
 
+using namespace Scintilla;
 using namespace Lexilla;
 
+namespace {
+
 // Test for [=[ ... ]=] delimiters, returns 0 if it's only a [ or ],
 // return 1 for [[ or ]], returns >=2 for [=[ or ]=] and so on.
 // The maximum number of '=' characters allowed is 254.
-static int LongDelimCheck(StyleContext &sc) {
+int LongDelimCheck(StyleContext &sc) {
 	int sep = 1;
 	while (sc.GetRelative(sep) == '=' && sep < 0xFF)
 		sep++;
@@ -43,31 +46,165 @@ static int LongDelimCheck(StyleContext &sc) {
 	return 0;
 }
 
-static void ColouriseLuaDoc(
+const char * const luaWordListDesc[] = {
-	Sci_PositionU startPos,
+	"Keywords",
-	Sci_Position length,
+	"Basic functions",
-	int initStyle,
+	"String, (table) & math functions",
-	WordList *keywordlists[],
+	"(coroutines), I/O & system facilities",
-	Accessor &styler) {
+	"user1",
+	"user2",
+	"user3",
+	"user4",
+    nullptr
+};
 
-	const WordList &keywords = *keywordlists[0];
+const LexicalClass lexicalClasses[] = {
-	const WordList &keywords2 = *keywordlists[1];
+	// Lexer Lua SCLEX_LUA SCE_LUA_:
-	const WordList &keywords3 = *keywordlists[2];
+	0, "SCE_LUA_DEFAULT", "default", "White space: Visible only in View Whitespace mode (or if it has a back colour)",
-	const WordList &keywords4 = *keywordlists[3];
+	1, "SCE_LUA_COMMENT", "comment", "Block comment (Lua 5.0)",
-	const WordList &keywords5 = *keywordlists[4];
+	2, "SCE_LUA_COMMENTLINE", "comment line", "Line comment",
-	const WordList &keywords6 = *keywordlists[5];
+	3, "SCE_LUA_COMMENTDOC", "comment documentation", "Doc comment",
-	const WordList &keywords7 = *keywordlists[6];
+	4, "SCE_LUA_NUMBER", "literal numeric", "Number",
-	const WordList &keywords8 = *keywordlists[7];
+	5, "SCE_LUA_WORD", "keyword", "Keyword",
+	6, "SCE_LUA_STRING", "literal string", "(Double quoted) String",
+	7, "SCE_LUA_CHARACTER", "literal string character", "Character (Single quoted string)",
+	8, "SCE_LUA_LITERALSTRING", "literal string", "Literal string",
+	9, "SCE_LUA_PREPROCESSOR", "preprocessor", "Preprocessor (obsolete in Lua 4.0 and up)",
+	10, "SCE_LUA_OPERATOR", "operator", "Operators",
+	11, "SCE_LUA_IDENTIFIER", "identifier", "Identifier (everything else...)",
+	12, "SCE_LUA_STRINGEOL", "error literal string", "End of line where string is not closed",
+	13, "SCE_LUA_WORD2", "identifier", "Other keywords",
+	14, "SCE_LUA_WORD3", "identifier", "Other keywords",
+	15, "SCE_LUA_WORD4", "identifier", "Other keywords",
+	16, "SCE_LUA_WORD5", "identifier", "Other keywords",
+	17, "SCE_LUA_WORD6", "identifier", "Other keywords",
+	18, "SCE_LUA_WORD7", "identifier", "Other keywords",
+	19, "SCE_LUA_WORD8", "identifier", "Other keywords",
+	20, "SCE_LUA_LABEL", "label", "Labels",
+};
+
+// Options used for LexerLua
+struct OptionsLua {
+	bool foldCompact = true;
+};
+
+struct OptionSetLua : public OptionSet<OptionsLua> {
+	OptionSetLua() {
+		DefineProperty("fold.compact", &OptionsLua::foldCompact);
+
+		DefineWordListSets(luaWordListDesc);
+	}
+};
+
+class LexerLua : public DefaultLexer {
+	WordList keywords;
+	WordList keywords2;
+	WordList keywords3;
+	WordList keywords4;
+	WordList keywords5;
+	WordList keywords6;
+	WordList keywords7;
+	WordList keywords8;
+	OptionsLua options;
+	OptionSetLua osLua;
+public:
+	explicit LexerLua() :
+		DefaultLexer("lua", SCLEX_LUA, lexicalClasses, ELEMENTS(lexicalClasses)) {
+	}
+	~LexerLua() override = default;
+	void SCI_METHOD Release() noexcept override {
+		delete this;
+	}
+	int SCI_METHOD Version() const noexcept override {
+		return lvRelease5;
+	}
+	const char *SCI_METHOD PropertyNames() noexcept override {
+		return osLua.PropertyNames();
+	}
+	int SCI_METHOD PropertyType(const char *name) override {
+		return osLua.PropertyType(name);
+	}
+	const char *SCI_METHOD DescribeProperty(const char *name) override {
+		return osLua.DescribeProperty(name);
+	}
+	Sci_Position SCI_METHOD PropertySet(const char *key, const char *val) override;
+	const char *SCI_METHOD PropertyGet(const char *key) override {
+		return osLua.PropertyGet(key);
+	}
+	const char *SCI_METHOD DescribeWordListSets() noexcept override {
+		return osLua.DescribeWordListSets();
+	}
+	Sci_Position SCI_METHOD WordListSet(int n, const char *wl) override;
+	void SCI_METHOD Lex(Sci_PositionU startPos, Sci_Position length, int initStyle, IDocument *pAccess) override;
+	void SCI_METHOD Fold(Sci_PositionU startPos, Sci_Position length, int initStyle, IDocument *pAccess) override;
+
+	static ILexer5 *LexerFactoryLua() {
+		return new LexerLua();
+	}
+};
+
+Sci_Position SCI_METHOD LexerLua::PropertySet(const char *key, const char *val) {
+	if (osLua.PropertySet(&options, key, val)) {
+		return 0;
+	}
+	return -1;
+}
+
+Sci_Position SCI_METHOD LexerLua::WordListSet(int n, const char *wl) {
+	WordList *wordListN = nullptr;
+	switch (n) {
+	case 0:
+		wordListN = &keywords;
+		break;
+	case 1:
+		wordListN = &keywords2;
+		break;
+	case 2:
+		wordListN = &keywords3;
+		break;
+	case 3:
+		wordListN = &keywords4;
+		break;
+	case 4:
+		wordListN = &keywords5;
+		break;
+	case 5:
+		wordListN = &keywords6;
+		break;
+	case 6:
+		wordListN = &keywords7;
+		break;
+	case 7:
+		wordListN = &keywords8;
+		break;
+	default:
+		break;
+	}
+	Sci_Position firstModification = -1;
+	if (wordListN) {
+		if (wordListN->Set(wl)) {
+			firstModification = 0;
+		}
+	}
+	return firstModification;
+}
+
+constexpr int maskSeparator = 0xFF;
+constexpr int maskStringWs = 0x100;
+constexpr int maskDocComment = 0x200;
+
+void LexerLua::Lex(Sci_PositionU startPos, Sci_Position length, int initStyle, IDocument *pAccess) {
+	LexAccessor styler(pAccess);
 
 	// Accepts accented characters
-	CharacterSet setWordStart(CharacterSet::setAlpha, "_", true);
+	const CharacterSet setWordStart(CharacterSet::setAlpha, "_", true);
-	CharacterSet setWord(CharacterSet::setAlphaNum, "_", true);
+	const CharacterSet setWord(CharacterSet::setAlphaNum, "_", true);
 	// Not exactly following number definition (several dots are seen as OK, etc.)
 	// but probably enough in most cases. [pP] is for hex floats.
-	CharacterSet setNumber(CharacterSet::setDigits, ".-+abcdefpABCDEFP");
+	const CharacterSet setNumber(CharacterSet::setDigits, ".-+abcdefpABCDEFP");
-	CharacterSet setExponent("eEpP");
+	const CharacterSet setExponent("eEpP");
-	CharacterSet setLuaOperator("*/-+()={}~[];<>,.^%:#&|");
+	const CharacterSet setLuaOperator("*/-+()={}~[];<>,.^%:#&|");
-	CharacterSet setEscapeSkip("\"'\\");
+	const CharacterSet setEscapeSkip("\"'\\");
 
 	Sci_Position currentLine = styler.GetLine(startPos);
 	// Initialize long string [[ ... ]] or block comment --[[ ... ]],
@@ -76,11 +213,13 @@ static void ColouriseLuaDoc(
 	// Continuation of a string (\z whitespace escaping) is controlled by stringWs.
 	int sepCount = 0;
 	int stringWs = 0;
-	if (initStyle == SCE_LUA_LITERALSTRING || initStyle == SCE_LUA_COMMENT ||
+	int lastLineDocComment = 0;
-		initStyle == SCE_LUA_STRING || initStyle == SCE_LUA_CHARACTER) {
+	if ((currentLine > 0) &&
+		AnyOf(initStyle, SCE_LUA_DEFAULT, SCE_LUA_LITERALSTRING, SCE_LUA_COMMENT, SCE_LUA_COMMENTDOC, SCE_LUA_STRING, SCE_LUA_CHARACTER)) {
 		const int lineState = styler.GetLineState(currentLine - 1);
-		sepCount = lineState & 0xFF;
+		sepCount = lineState & maskSeparator;
-		stringWs = lineState & 0x100;
+		stringWs = lineState & maskStringWs;
+		lastLineDocComment = lineState & maskDocComment;
 	}
 
 	// results of identifier/keyword matching
@@ -90,7 +229,7 @@ static void ColouriseLuaDoc(
 	bool foundGoto = false;
 
 	// Do not leak onto next line
-	if (initStyle == SCE_LUA_STRINGEOL || initStyle == SCE_LUA_COMMENTLINE || initStyle == SCE_LUA_PREPROCESSOR) {
+	if (AnyOf(initStyle, SCE_LUA_STRINGEOL, SCE_LUA_COMMENTLINE, SCE_LUA_COMMENTDOC, SCE_LUA_PREPROCESSOR)) {
 		initStyle = SCE_LUA_DEFAULT;
 	}
 
@@ -104,12 +243,14 @@ static void ColouriseLuaDoc(
 			// Update the line state, so it can be seen by next line
 			currentLine = styler.GetLine(sc.currentPos);
 			switch (sc.state) {
+			case SCE_LUA_DEFAULT:
 			case SCE_LUA_LITERALSTRING:
 			case SCE_LUA_COMMENT:
+			case SCE_LUA_COMMENTDOC:
 			case SCE_LUA_STRING:
 			case SCE_LUA_CHARACTER:
 				// Inside a literal string, block comment or string, we set the line state
-				styler.SetLineState(currentLine, stringWs | sepCount);
+				styler.SetLineState(currentLine, lastLineDocComment | stringWs | sepCount);
 				break;
 			default:
 				// Reset the line state
@@ -141,20 +282,19 @@ static void ColouriseLuaDoc(
 				Sci_Position ln = 0;
 				while (IsASpaceOrTab(sc.GetRelative(ln)))	// skip over spaces/tabs
 					ln++;
-				Sci_Position ws1 = ln;
+				const Sci_Position ws1 = ln;
 				if (setWordStart.Contains(sc.GetRelative(ln))) {
-					int c, i = 0;
+					int c = 0;
-					char s[100];
+					std::string s;
 					while (setWord.Contains(c = sc.GetRelative(ln))) {	// get potential label
-						if (i < 90)
+						s.push_back(static_cast<char>(c));
-							s[i++] = static_cast<char>(c);
 						ln++;
 					}
-					s[i] = '\0'; Sci_Position lbl = ln;
+					const Sci_Position lbl = ln;
 					if (!keywords.InList(s)) {
 						while (IsASpaceOrTab(sc.GetRelative(ln)))	// skip over spaces/tabs
 							ln++;
-						Sci_Position ws2 = ln - lbl;
+						const Sci_Position ws2 = ln - lbl;
 						if (sc.GetRelative(ln) == ':' && sc.GetRelative(ln + 1) == ':') {
 							// final :: found, complete valid label construct
 							sc.ChangeState(SCE_LUA_LABEL);
@@ -206,14 +346,14 @@ static void ColouriseLuaDoc(
 					sc.Forward();
 					while (setWord.Contains(sc.ch))
 						sc.Forward();
-					char s[100];
+					std::string s;
-					sc.GetCurrent(s, sizeof(s));
+					sc.GetCurrentString(s, StyleContext::Transform::none);
 					if (keywords.InList(s))		// labels cannot be keywords
 						sc.ChangeState(SCE_LUA_WORD);
 				}
 				sc.SetState(SCE_LUA_DEFAULT);
 			}
-		} else if (sc.state == SCE_LUA_COMMENTLINE || sc.state == SCE_LUA_PREPROCESSOR) {
+		} else if (AnyOf(sc.state, SCE_LUA_COMMENTLINE, SCE_LUA_COMMENTDOC, SCE_LUA_PREPROCESSOR)) {
 			if (sc.atLineEnd) {
 				sc.ForwardSetState(SCE_LUA_DEFAULT);
 			}
@@ -227,7 +367,7 @@ static void ColouriseLuaDoc(
 					sc.Forward();
 				} else if (sc.chNext == 'z') {
 					sc.Forward();
-					stringWs = 0x100;
+					stringWs = maskStringWs;
 				}
 			} else if (sc.ch == '\"') {
 				sc.ForwardSetState(SCE_LUA_DEFAULT);
@@ -245,7 +385,7 @@ static void ColouriseLuaDoc(
 					sc.Forward();
 				} else if (sc.chNext == 'z') {
 					sc.Forward();
-					stringWs = 0x100;
+					stringWs = maskStringWs;
 				}
 			} else if (sc.ch == '\'') {
 				sc.ForwardSetState(SCE_LUA_DEFAULT);
@@ -277,9 +417,9 @@ static void ColouriseLuaDoc(
 				idenWordPos = 0;
 				idenStyle = SCE_LUA_IDENTIFIER;
 				foundGoto = false;
-				int cNext;
+				int cNext = 0;
 				do {
-					int c;
+					int c = 0;
 					const Sci_Position idenPosOld = idenPos;
 					std::string identSeg;
 					identSeg += static_cast<char>(sc.GetRelative(idenPos++));
@@ -287,29 +427,28 @@ static void ColouriseLuaDoc(
 						identSeg += static_cast<char>(c);
 						idenPos++;
 					}
-					if (keywords.InList(identSeg.c_str()) && (idenPosOld > 0)) {
+					if (keywords.InList(identSeg) && (idenPosOld > 0)) {
 						idenPos = idenPosOld - 1;	// keywords cannot mix
 						ident.pop_back();
 						break;
 					}
 					ident += identSeg;
-					const char* s = ident.c_str();
 					int newStyle = SCE_LUA_IDENTIFIER;
-					if (keywords.InList(s)) {
+					if (keywords.InList(ident)) {
 						newStyle = SCE_LUA_WORD;
-					} else if (keywords2.InList(s)) {
+					} else if (keywords2.InList(ident)) {
 						newStyle = SCE_LUA_WORD2;
-					} else if (keywords3.InList(s)) {
+					} else if (keywords3.InList(ident)) {
 						newStyle = SCE_LUA_WORD3;
-					} else if (keywords4.InList(s)) {
+					} else if (keywords4.InList(ident)) {
 						newStyle = SCE_LUA_WORD4;
-					} else if (keywords5.InList(s)) {
+					} else if (keywords5.InList(ident)) {
 						newStyle = SCE_LUA_WORD5;
-					} else if (keywords6.InList(s)) {
+					} else if (keywords6.InList(ident)) {
 						newStyle = SCE_LUA_WORD6;
-					} else if (keywords7.InList(s)) {
+					} else if (keywords7.InList(ident)) {
 						newStyle = SCE_LUA_WORD7;
-					} else if (keywords8.InList(s)) {
+					} else if (keywords8.InList(ident)) {
 						newStyle = SCE_LUA_WORD8;
 					}
 					if (newStyle != SCE_LUA_IDENTIFIER) {
@@ -326,7 +465,7 @@ static void ColouriseLuaDoc(
 						cNext = 0;
 					}
 				} while (cNext);
-				if ((idenStyle == SCE_LUA_WORD) && (ident.compare("goto") == 0)) {
+                if ((idenStyle == SCE_LUA_WORD) && (ident == "goto")) {
 					foundGoto = true;
 				}
 				sc.SetState(SCE_LUA_IDENTIFIER);
@@ -345,7 +484,7 @@ static void ColouriseLuaDoc(
 					sc.Forward(sepCount);
 				}
 			} else if (sc.Match('-', '-')) {
-				sc.SetState(SCE_LUA_COMMENTLINE);
+				sc.SetState(lastLineDocComment ? SCE_LUA_COMMENTDOC : SCE_LUA_COMMENTLINE);
 				if (sc.Match("--[")) {
 					sc.Forward(2);
 					sepCount = LongDelimCheck(sc);
@@ -353,6 +492,9 @@ static void ColouriseLuaDoc(
 						sc.ChangeState(SCE_LUA_COMMENT);
 						sc.Forward(sepCount);
 					}
+				} else if (sc.Match("---")) {
+					sc.SetState(SCE_LUA_COMMENTDOC);
+					lastLineDocComment = maskDocComment;
 				} else {
 					sc.Forward();
 				}
@@ -361,46 +503,51 @@ static void ColouriseLuaDoc(
 			} else if (setLuaOperator.Contains(sc.ch)) {
 				sc.SetState(SCE_LUA_OPERATOR);
 			}
+			if (!AnyOf(sc.state, SCE_LUA_DEFAULT, SCE_LUA_COMMENTDOC)) {
+				lastLineDocComment = 0;
+			}
 		}
 	}
 
 	sc.Complete();
 }
 
-static void FoldLuaDoc(Sci_PositionU startPos, Sci_Position length, int initStyle, WordList *[],
+void LexerLua::Fold(Sci_PositionU startPos_, Sci_Position length, int initStyle, IDocument *pAccess) {
-                       Accessor &styler) {
+	LexAccessor styler(pAccess);
-	const Sci_PositionU lengthDoc = startPos + length;
+	const Sci_Position startPos = startPos_;
+	const Sci_Position lengthDoc = startPos + length;
 	int visibleChars = 0;
 	Sci_Position lineCurrent = styler.GetLine(startPos);
 	int levelPrev = styler.LevelAt(lineCurrent) & SC_FOLDLEVELNUMBERMASK;
 	int levelCurrent = levelPrev;
 	char chNext = styler[startPos];
-	const bool foldCompact = styler.GetPropertyInt("fold.compact", 1) != 0;
+	const bool foldCompact = options.foldCompact;
 	int style = initStyle;
-	int styleNext = styler.StyleAt(startPos);
+	int styleNext = styler.StyleIndexAt(startPos);
 
-	for (Sci_PositionU i = startPos; i < lengthDoc; i++) {
+	for (Sci_Position i = startPos; i < lengthDoc; i++) {
 		const char ch = chNext;
 		chNext = styler.SafeGetCharAt(i + 1);
 		const int stylePrev = style;
 		style = styleNext;
-		styleNext = styler.StyleAt(i + 1);
+		styleNext = styler.StyleIndexAt(i + 1);
 		const bool atEOL = (ch == '\r' && chNext != '\n') || (ch == '\n');
 		if (style == SCE_LUA_WORD) {
-			if (ch == 'i' || ch == 'd' || ch == 'f' || ch == 'e' || ch == 'r' || ch == 'u') {
+			// Fixed list of folding words: if, do, function, repeat, end, until
-				char s[10] = "";
+			// Must fix up next line with initial characters if any new words added.
-				for (Sci_PositionU j = 0; j < 8; j++) {
+			if ((style != stylePrev) && AnyOf(ch, 'i', 'd', 'f', 'e', 'r', 'u')) {
+				std::string s;
+				for (Sci_Position j = 0; j < 8; j++) {	// 8 is length of longest: function
 					if (!iswordchar(styler[i + j])) {
 						break;
 					}
-					s[j] = styler[i + j];
+					s.push_back(styler[i + j]);
-					s[j + 1] = '\0';
 				}
 
-				if ((strcmp(s, "if") == 0) || (strcmp(s, "do") == 0) || (strcmp(s, "function") == 0) || (strcmp(s, "repeat") == 0)) {
+				if (s == "if" || s == "do" || s == "function" || s == "repeat") {
 					levelCurrent++;
 				}
-				if ((strcmp(s, "end") == 0) || (strcmp(s, "elseif") == 0) || (strcmp(s, "until") == 0)) {
+				if (s == "end" || s == "until") {
 					levelCurrent--;
 				}
 			}
@@ -439,49 +586,10 @@ static void FoldLuaDoc(Sci_PositionU startPos, Sci_Position length, int initStyl
 	}
 	// Fill in the real level of the next line, keeping the current flags as they will be filled in later
 
-	int flagsNext = styler.LevelAt(lineCurrent) & ~SC_FOLDLEVELNUMBERMASK;
+	const int flagsNext = styler.LevelAt(lineCurrent) & ~SC_FOLDLEVELNUMBERMASK;
 	styler.SetLevel(lineCurrent, levelPrev | flagsNext);
 }
 
-static const char * const luaWordListDesc[] = {
-	"Keywords",
-	"Basic functions",
-	"String, (table) & math functions",
-	"(coroutines), I/O & system facilities",
-	"user1",
-	"user2",
-	"user3",
-	"user4",
-	0
-};
-
-namespace {
-
-LexicalClass lexicalClasses[] = {
-	// Lexer Lua SCLEX_LUA SCE_LUA_:
-	0, "SCE_LUA_DEFAULT", "default", "White space: Visible only in View Whitespace mode (or if it has a back colour)",
-	1, "SCE_LUA_COMMENT", "comment", "Block comment (Lua 5.0)",
-	2, "SCE_LUA_COMMENTLINE", "comment line", "Line comment",
-	3, "SCE_LUA_COMMENTDOC", "comment documentation", "Doc comment -- Not used in Lua (yet?)",
-	4, "SCE_LUA_NUMBER", "literal numeric", "Number",
-	5, "SCE_LUA_WORD", "keyword", "Keyword",
-	6, "SCE_LUA_STRING", "literal string", "(Double quoted) String",
-	7, "SCE_LUA_CHARACTER", "literal string character", "Character (Single quoted string)",
-	8, "SCE_LUA_LITERALSTRING", "literal string", "Literal string",
-	9, "SCE_LUA_PREPROCESSOR", "preprocessor", "Preprocessor (obsolete in Lua 4.0 and up)",
-	10, "SCE_LUA_OPERATOR", "operator", "Operators",
-	11, "SCE_LUA_IDENTIFIER", "identifier", "Identifier (everything else...)",
-	12, "SCE_LUA_STRINGEOL", "error literal string", "End of line where string is not closed",
-	13, "SCE_LUA_WORD2", "identifier", "Other keywords",
-	14, "SCE_LUA_WORD3", "identifier", "Other keywords",
-	15, "SCE_LUA_WORD4", "identifier", "Other keywords",
-	16, "SCE_LUA_WORD5", "identifier", "Other keywords",
-	17, "SCE_LUA_WORD6", "identifier", "Other keywords",
-	18, "SCE_LUA_WORD7", "identifier", "Other keywords",
-	19, "SCE_LUA_WORD8", "identifier", "Other keywords",
-	20, "SCE_LUA_LABEL", "label", "Labels",
-};
-
 }
 
-LexerModule lmLua(SCLEX_LUA, ColouriseLuaDoc, "lua", FoldLuaDoc, luaWordListDesc, lexicalClasses, ELEMENTS(lexicalClasses));
+LexerModule lmLua(SCLEX_LUA, LexerLua::LexerFactoryLua, "lua", luaWordListDesc);

lexilla/scripts/HeaderOrder.txt

@@ -97,7 +97,7 @@
 #include "LexerSimple.h"
 #include "LexerNoExceptions.h"
 
-// src
+// test
 
 #include "TestDocument.h"
 

lexilla/scripts/LexillaData.py

@@ -3,7 +3,10 @@
 # Released to the public domain.
 # Requires FileGenerator from Scintilla so scintilla must be a peer directory of lexilla.
 
-# Common code used by Lexilla and SciTE for source file regeneration.
+"""
+Common code used by Lexilla and SciTE for source file regeneration.
+"""
+
 # The LexillaData object exposes information about Lexilla as properties:
 # Version properties
 #     version
@@ -42,15 +45,16 @@
 
 import datetime, pathlib, sys, textwrap
 
-thisPath = pathlib.Path(__file__).resolve()
-
-sys.path.append(str(thisPath.parent.parent.parent / "scintilla" / "scripts"))
-
-import FileGenerator
-
 neutralEncoding = "iso-8859-1"	# Each byte value is valid in iso-8859-1
 
+def ReadFileAsList(path):
+    """Read all the lnes in the file and return as a list of strings without line ends.
+    """
+    with path.open(encoding="utf-8") as f:
+        return [line.rstrip('\n') for line in f]
+
 def FindModules(lexFile):
+    """ Return a list of modules found within a lexer implementation file. """
     modules = []
     partLine = ""
     with lexFile.open(encoding=neutralEncoding) as f:
@@ -69,7 +73,7 @@ def FindModules(lexFile):
                     lexerName = parts[4]
                     if not (lexerName.startswith('"') and lexerName.endswith('"')):
                         print(f"{lexFile}:{lineNum}: Bad LexerModule statement:\n{original}")
-                        exit(1)
+                        sys.exit(1)
                     lexerName = lexerName.strip('"')
                     modules.append([parts[1], parts[2], lexerName])
                     partLine = ""
@@ -77,14 +81,48 @@ def FindModules(lexFile):
                     partLine = partLine + line
     return modules
 
-def FindLexersInXcode(xCodeProject):
+def FindSectionInList(lines, markers):
-    lines = FileGenerator.ReadFileAsList(xCodeProject)
+    """Find a section defined by an initial start marker, an optional secondary
+    marker and an end marker.
+    The section is between the secondary/initial start and the end.
+    Report as a slice object so the section can be extracted or replaced.
+    Raises an exception if the markers can't be found.
+    Currently only used for Xcode project files.
+    """
+    start = -1
+    end = -1
+    state = 0
+    for i, line in enumerate(lines):
+        if markers[0] in line:
+            if markers[1]:
+                state = 1
+            else:
+                start = i+1
+                state = 2
+        elif state == 1:
+            if markers[1] in line:
+                start = i+1
+                state = 2
+        elif state == 2:
+            if markers[2] in line:
+                end = i
+                state = 3
+    # Check that section was found
+    if start == -1:
+        raise ValueError("Could not find start marker(s) |" + markers[0] + "|" + markers[1] + "|")
+    if end == -1:
+        raise ValueError("Could not find end marker " + markers[2])
+    return slice(start, end)
 
-    # PBXBuildFile section is a list of all buildable files in the project so extract the file basename and
+def FindLexersInXcode(xCodeProject):
-    # its build and file IDs
+    """ Return a dictionary { file name: [build UUID, file UUID] } of lexers in Xcode project. """
+    lines = ReadFileAsList(xCodeProject)
+
+    # PBXBuildFile section is a list of all buildable files in the project so extract the file
+    # basename and its build and file IDs
     uidsOfBuild = {}
     markersPBXBuildFile = ["Begin PBXBuildFile section", "", "End PBXBuildFile section"]
-    for buildLine in lines[FileGenerator.FindSectionInList(lines, markersPBXBuildFile)]:
+    for buildLine in lines[FindSectionInList(lines, markersPBXBuildFile)]:
         # Occurs for each file in the build. Find the UIDs used for the file.
         #\t\t[0-9A-F]+ /* [a-zA-Z]+.cxx in sources */ = {isa = PBXBuildFile; fileRef = [0-9A-F]+ /* [a-zA-Z]+ */; };
         pieces = buildLine.split()
@@ -96,7 +134,7 @@ def FindLexersInXcode(xCodeProject):
     # PBXGroup section contains the folders (Lexilla, Lexers, LexLib, ...) so is used to find the lexers
     lexers = {}
     markersLexers = ["/* Lexers */ =", "children", ");"]
-    for lexerLine in lines[FileGenerator.FindSectionInList(lines, markersLexers)]:
+    for lexerLine in lines[FindSectionInList(lines, markersLexers)]:
         #\t\t\t\t[0-9A-F]+ /* [a-zA-Z]+.cxx */,
         uid, _, rest = lexerLine.partition("/* ")
         uid = uid.strip()
@@ -121,7 +159,8 @@ knownIrregularProperties = [
 ]
 
 def FindProperties(lexFile):
-    properties = {}
+    """ Return a set of property names in a lexer implementation file. """
+    properties = set()
     with open(lexFile, encoding=neutralEncoding) as f:
         for s in f.readlines():
             if ("GetProperty" in s or "DefineProperty" in s) and "\"" in s:
@@ -133,10 +172,11 @@ def FindProperties(lexFile):
                         if propertyName in knownIrregularProperties or \
                             propertyName.startswith("fold.") or \
                             propertyName.startswith("lexer."):
-                            properties[propertyName] = 1
+                            properties.add(propertyName)
     return properties
 
 def FindPropertyDocumentation(lexFile):
+    """ Return a dictionary { name: document string } of property documentation in a lexer. """
     documents = {}
     with lexFile.open(encoding=neutralEncoding) as f:
         name = ""
@@ -178,7 +218,8 @@ def FindPropertyDocumentation(lexFile):
     return documents
 
 def FindCredits(historyFile):
-    credits = []
+    """ Return a list of contributors in a history file. """
+    creditList = []
     stage = 0
     with historyFile.open(encoding="utf-8") as f:
         for line in f.readlines():
@@ -190,7 +231,7 @@ def FindCredits(historyFile):
             if stage == 1 and line.startswith("<td>"):
                 credit = line[4:-5]
                 if "<a" in line:
-                    title, a, rest = credit.partition("<a href=")
+                    title, dummy, rest = credit.partition("<a href=")
                     urlplus, _bracket, end = rest.partition(">")
                     name = end.split("<")[0]
                     url = urlplus[1:-1]
@@ -198,16 +239,20 @@ def FindCredits(historyFile):
                     if credit:
                         credit += " "
                     credit += name + " " + url
-                credits.append(credit)
+                creditList.append(credit)
-    return credits
+    return creditList
 
 def ciKey(a):
+    """ Return a string lowered to be used when sorting. """
     return str(a).lower()
 
-def SortListInsensitive(list):
+def SortListInsensitive(l):
-    list.sort(key=ciKey)
+    """ Sort a list of strings case insensitively. """
+    l.sort(key=ciKey)
 
 class LexillaData:
+    """ Expose information about Lexilla as properties. """
+
     def __init__(self, scintillaRoot):
         # Discover version information
         self.version = (scintillaRoot / "version.txt").read_text().strip()
@@ -223,7 +268,7 @@ class LexillaData:
             dtModified = datetime.datetime.strptime(self.dateModified, "%Y%m%d")
             self.yearModified = self.dateModified[0:4]
             monthModified = dtModified.strftime("%B")
-            dayModified = "%d" % dtModified.day
+            dayModified = f"{dtModified.day}"
             self.mdyModified = monthModified + " " + dayModified + " " + self.yearModified
             # May 22 2013
             # Lexilla.html, SciTE.html
@@ -247,12 +292,12 @@ class LexillaData:
                 self.sclexFromName[module[2]] = module[1]
                 self.fileFromSclex[module[1]] = lexFile
                 self.lexerModules.append(module[0])
-            for k in FindProperties(lexFile).keys():
+            for prop in FindProperties(lexFile):
-                lexerProperties.add(k)
+                lexerProperties.add(prop)
             documents = FindPropertyDocumentation(lexFile)
-            for k in documents.keys():
+            for prop, doc in documents.items():
-                if k not in self.propertyDocuments:
+                if prop not in self.propertyDocuments:
-                    self.propertyDocuments[k] = documents[k]
+                    self.propertyDocuments[prop] = doc
         SortListInsensitive(self.lexerModules)
         self.lexerProperties = list(lexerProperties)
         SortListInsensitive(self.lexerProperties)
@@ -262,13 +307,14 @@ class LexillaData:
         self.credits = FindCredits(scintillaRoot / "doc" / "LexillaHistory.html")
 
 def printWrapped(text):
+    """ Print string wrapped with subsequent lines indented. """
     print(textwrap.fill(text, subsequent_indent="    "))
 
 if __name__=="__main__":
     sci = LexillaData(pathlib.Path(__file__).resolve().parent.parent)
-    print("Version   %s   %s   %s" % (sci.version, sci.versionDotted, sci.versionCommad))
+    print(f"Version   {sci.version}   {sci.versionDotted}   {sci.versionCommad}")
-    print("Date last modified    %s   %s   %s   %s   %s" % (
+    print(f"Date last modified    {sci.dateModified}   {sci.yearModified}   {sci.mdyModified}"
-        sci.dateModified, sci.yearModified, sci.mdyModified, sci.dmyModified, sci.myModified))
+        f"   {sci.dmyModified}   {sci.myModified}")
     printWrapped(str(len(sci.lexFiles)) + " lexer files: " + ", ".join(sci.lexFiles))
     printWrapped(str(len(sci.lexerModules)) + " lexer modules: " + ", ".join(sci.lexerModules))
     #~ printWrapped(str(len(sci.lexersXcode)) + " Xcode lexer references: " + ", ".join(

lexilla/scripts/LexillaGen.py

@@ -2,7 +2,10 @@
 # LexillaGen.py - implemented 2019 by Neil Hodgson neilh@scintilla.org
 # Released to the public domain.
 
-# Regenerate the Lexilla source files that list all the lexers.
+"""
+Regenerate the Lexilla source files that list all the lexers.
+"""
+
 # Should be run whenever a new lexer is added or removed.
 # Requires Python 3.6 or later
 # Files are regenerated in place with templates stored in comments.
@@ -15,8 +18,7 @@ thisPath = pathlib.Path(__file__).resolve()
 sys.path.append(str(thisPath.parent.parent.parent / "scintilla" / "scripts"))
 
 from FileGenerator import Regenerate, UpdateLineInFile, \
-    ReplaceREInFile, UpdateLineInPlistFile, ReadFileAsList, UpdateFileFromLines, \
+    ReplaceREInFile, UpdateLineInPlistFile, UpdateFileFromLines
-    FindSectionInList
 import LexillaData
 import LexFacer
 
@@ -25,11 +27,12 @@ import DepGen
 
 # RegenerateXcodeProject and assiciated functions is copied from scintilla/scripts/LexGen.py
 
-# Last 24 digits of UUID, used for item IDs in Xcode
 def uid24():
+    """ Last 24 digits of UUID, used for item IDs in Xcode. """
     return str(uuid.uuid4()).replace("-", "").upper()[-24:]
 
 def ciLexerKey(a):
+    """ Return 3rd element of string lowered to be used when sorting. """
     return a.split()[2].lower()
 
 
@@ -40,6 +43,7 @@ def ciLexerKey(a):
 				11F35FDB12AEFAF100F0236D /* LexA68k.cxx in Sources */,
 """
 def RegenerateXcodeProject(path, lexers, lexerReferences):
+    """ Regenerate project to include any new lexers. """
     # Build 4 blocks for insertion:
     # Each markers contains a unique section start, an optional wait string, and a section end
 
@@ -61,34 +65,35 @@ def RegenerateXcodeProject(path, lexers, lexerReferences):
             uid2 = uid24()
             print("Lexer", lexer, "is not in Xcode project. Use IDs", uid1, uid2)
             lexerReferences[lexer] = [uid1, uid2]
-            linePBXBuildFile = "\t\t{} /* {}.cxx in Sources */ = {{isa = PBXBuildFile; fileRef = {} /* {}.cxx */; }};".format(uid1, lexer, uid2, lexer)
+            linePBXBuildFile = f"\t\t{uid1} /* {lexer}.cxx in Sources */ = {{isa = PBXBuildFile; fileRef = {uid2} /* {lexer}.cxx */; }};"
-            linePBXFileReference = "\t\t{} /* {}.cxx */ = {{isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; name = {}.cxx; path = ../../lexers/{}.cxx; sourceTree = SOURCE_ROOT; }};".format(uid2, lexer, lexer, lexer)
+            linePBXFileReference = f"\t\t{uid2} /* {lexer}.cxx */ = {{isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; name = {lexer}.cxx; path = ../../lexers/{lexer}.cxx; sourceTree = SOURCE_ROOT; }};"
-            lineLexers = "\t\t\t\t{} /* {}.cxx */,".format(uid2, lexer)
+            lineLexers = f"\t\t\t\t{uid2} /* {lexer}.cxx */,"
-            linePBXSourcesBuildPhase = "\t\t\t\t{} /* {}.cxx in Sources */,".format(uid1, lexer)
+            linePBXSourcesBuildPhase = f"\t\t\t\t{uid1} /* {lexer}.cxx in Sources */,"
             sectionPBXBuildFile.append(linePBXBuildFile)
             sectionPBXFileReference.append(linePBXFileReference)
             sectionLexers.append(lineLexers)
             sectionPBXSourcesBuildPhase.append(linePBXSourcesBuildPhase)
 
-    lines = ReadFileAsList(path)
+    lines = LexillaData.ReadFileAsList(path)
 
-    sli = FindSectionInList(lines, markersPBXBuildFile)
+    sli = LexillaData.FindSectionInList(lines, markersPBXBuildFile)
     lines[sli.stop:sli.stop] = sectionPBXBuildFile
 
-    sli = FindSectionInList(lines, markersPBXFileReference)
+    sli = LexillaData.FindSectionInList(lines, markersPBXFileReference)
     lines[sli.stop:sli.stop] = sectionPBXFileReference
 
-    sli = FindSectionInList(lines, markersLexers)
+    sli = LexillaData.FindSectionInList(lines, markersLexers)
     # This section is shown in the project outline so sort it to make it easier to navigate.
     allLexers = sorted(lines[sli.start:sli.stop] + sectionLexers, key=ciLexerKey)
     lines[sli] = allLexers
 
-    sli = FindSectionInList(lines, markersPBXSourcesBuildPhase)
+    sli = LexillaData.FindSectionInList(lines, markersPBXSourcesBuildPhase)
     lines[sli.stop:sli.stop] = sectionPBXSourcesBuildPhase
 
-    UpdateFileFromLines(path, lines, "\n")
+    UpdateFileFromLines(path, lines, os.linesep)
 
 def RegenerateAll(rootDirectory):
+    """ Regenerate all the files. """
 
     root = pathlib.Path(rootDirectory)
 

lexilla/src/Lexilla.cxx

@@ -376,11 +376,11 @@ EXPORT_FUNCTION const char * CALLING_CONVENTION GetNameSpace() {
 	return "lexilla";
 }
 
+}
+
 // Not exported from binary as LexerModule must be built exactly the same as
 // modules listed above
 void AddStaticLexerModule(LexerModule *plm) {
 	AddEachLexer();
 	catalogueLexilla.AddLexerModule(plm);
 }
-
-}

lexilla/src/Lexilla/Info.plist

@@ -15,7 +15,7 @@
 	<key>CFBundlePackageType</key>
 	<string>$(PRODUCT_BUNDLE_PACKAGE_TYPE)</string>
 	<key>CFBundleShortVersionString</key>
-	<string>5.2.8</string>
+	<string>5.3.0</string>
 	<key>CFBundleVersion</key>
 	<string>$(CURRENT_PROJECT_VERSION)</string>
 	<key>NSHumanReadableCopyright</key>

lexilla/src/Lexilla/Lexilla.xcodeproj/project.pbxproj

@@ -792,6 +792,7 @@
 				MTL_ENABLE_DEBUG_INFO = INCLUDE_SOURCE;
 				MTL_FAST_MATH = YES;
 				ONLY_ACTIVE_ARCH = YES;
+				"OTHER_LDFLAGS[arch=*]" = "-Wl,-ld_classic";
 				SDKROOT = macosx;
 			};
 			name = Debug;
@@ -847,6 +848,7 @@
 				MACOSX_DEPLOYMENT_TARGET = 10.13;
 				MTL_ENABLE_DEBUG_INFO = NO;
 				MTL_FAST_MATH = YES;
+				"OTHER_LDFLAGS[arch=*]" = "-Wl,-ld_classic";
 				SDKROOT = macosx;
 			};
 			name = Release;
@@ -856,7 +858,7 @@
 			buildSettings = {
 				CLANG_CXX_LANGUAGE_STANDARD = "gnu++17";
 				CODE_SIGN_STYLE = Automatic;
-				CURRENT_PROJECT_VERSION = 5.2.8;
+				CURRENT_PROJECT_VERSION = 5.3.0;
 				DEAD_CODE_STRIPPING = YES;
 				DEVELOPMENT_TEAM = 4F446KW87E;
 				DYLIB_COMPATIBILITY_VERSION = 1;
@@ -884,7 +886,7 @@
 			buildSettings = {
 				CLANG_CXX_LANGUAGE_STANDARD = "gnu++17";
 				CODE_SIGN_STYLE = Automatic;
-				CURRENT_PROJECT_VERSION = 5.2.8;
+				CURRENT_PROJECT_VERSION = 5.3.0;
 				DEAD_CODE_STRIPPING = YES;
 				DEVELOPMENT_TEAM = 4F446KW87E;
 				DYLIB_COMPATIBILITY_VERSION = 1;

lexilla/src/LexillaVersion.rc

@@ -4,8 +4,8 @@
 
 #include <windows.h>
 
-#define VERSION_LEXILLA "5.2.8"
+#define VERSION_LEXILLA "5.3.0"
-#define VERSION_WORDS 5, 2, 8, 0
+#define VERSION_WORDS 5, 3, 0, 0
 
 VS_VERSION_INFO VERSIONINFO
 FILEVERSION	VERSION_WORDS

lexilla/src/deps.mak

@@ -860,7 +860,9 @@ $(DIR_O)/LexLua.o: \
 	../lexlib/Accessor.h \
 	../lexlib/StyleContext.h \
 	../lexlib/CharacterSet.h \
-	../lexlib/LexerModule.h
+	../lexlib/LexerModule.h \
+	../lexlib/OptionSet.h \
+	../lexlib/DefaultLexer.h
 $(DIR_O)/LexMagik.o: \
 	../lexers/LexMagik.cxx \
 	../../scintilla/include/ILexer.h \

lexilla/src/lexilla.mak

@@ -60,7 +60,7 @@ CXXFLAGS=$(CXXFLAGS) $(CXXNDEBUG)
 
 SCINTILLA_INCLUDE = ../../scintilla/include
 
-INCLUDEDIRS=-I../include -I$(SCINTILLA_INCLUDE) -I../src -I../lexlib
+INCLUDEDIRS=-I../include -I$(SCINTILLA_INCLUDE) -I../lexlib
 CXXFLAGS=$(CXXFLAGS) $(INCLUDEDIRS)
 
 all:	$(SCINTILLA_INCLUDE) $(LEXILLA) $(LIBLEXILLA)

lexilla/src/makefile

@@ -75,13 +75,13 @@ RANLIB ?= ranlib
 
 SCINTILLA_INCLUDE = ../../scintilla/include
 
-vpath %.h ../src ../include ../../scintilla/include ../lexlib
+vpath %.h ../include ../../scintilla/include ../lexlib
 vpath %.cxx ../src ../lexlib ../lexers
 
 DEFINES += -D$(if $(DEBUG),DEBUG,NDEBUG)
 BASE_FLAGS += $(if $(DEBUG),-g,-O3)
 
-INCLUDES = -I ../include -I $(SCINTILLA_INCLUDE) -I ../src -I ../lexlib
+INCLUDES = -I ../include -I $(SCINTILLA_INCLUDE) -I ../lexlib
 LDFLAGS += -shared
 
 BASE_FLAGS += $(WARNINGS)

lexilla/src/nmdeps.mak

@@ -860,7 +860,9 @@ $(DIR_O)/LexLua.obj: \
 	../lexlib/Accessor.h \
 	../lexlib/StyleContext.h \
 	../lexlib/CharacterSet.h \
-	../lexlib/LexerModule.h
+	../lexlib/LexerModule.h \
+	../lexlib/OptionSet.h \
+	../lexlib/DefaultLexer.h
 $(DIR_O)/LexMagik.obj: \
 	../lexers/LexMagik.cxx \
 	../../scintilla/include/ILexer.h \

lexilla/test/examples/bash/197ArithmeticOperator.bsh

@@ -14,3 +14,8 @@ done
 for ((i = 0; i < 2; ++i)); do
    echo prefix inc $i
 done
+
+# issue 215
+for ((i = 0; i < 2; i++)); do
+  echo $((((1)) << i))
+done

lexilla/test/examples/bash/197ArithmeticOperator.bsh.folded

@@ -14,4 +14,9 @@
  2 400   0 + for ((i = 0; i < 2; ++i)); do
  0 401   0 |    echo prefix inc $i
  0 401   0 | done
+ 1 400   0   
+ 0 400   0   # issue 215
+ 2 400   0 + for ((i = 0; i < 2; i++)); do
+ 0 401   0 |   echo $((((1)) << i))
+ 0 401   0 | done
  0 400   0   

lexilla/test/examples/bash/197ArithmeticOperator.bsh.styled

@@ -14,3 +14,8 @@
 {4}for{0} {7}(({8}i{0} {7}={0} {3}0{7};{0} {8}i{0} {7}<{0} {3}2{7};{0} {7}++{8}i{7}));{0} {4}do{0}
    {4}echo{0} {8}prefix{0} {8}inc{0} {9}$i{0}
 {4}done{0}
+
+{2}# issue 215{0}
+{4}for{0} {7}(({8}i{0} {7}={0} {3}0{7};{0} {8}i{0} {7}<{0} {3}2{7};{0} {8}i{7}++));{0} {4}do{0}
+  {4}echo{0} {7}$(((({3}1{7})){0} {7}<<{0} {8}i{7})){0}
+{4}done{0}

lexilla/test/examples/bash/Nested.bsh

@@ -28,6 +28,10 @@ var=abcdef
 sub=abc
 rep='& '
 echo ${var/$sub/"${rep}}"} #
+# issue 216
+option="no[foo]"
+option=${option%%[<{().[]*}
+echo $option
 
 # '$' in variable
 echo $$PID

lexilla/test/examples/bash/Nested.bsh.folded

@@ -28,6 +28,10 @@
  0 400   0   sub=abc
  0 400   0   rep='& '
  0 400   0   echo ${var/$sub/"${rep}}"} #
+ 0 400   0   # issue 216
+ 0 400   0   option="no[foo]"
+ 0 400   0   option=${option%%[<{().[]*}
+ 0 400   0   echo $option
  1 400   0   
  0 400   0   # '$' in variable
  0 400   0   echo $$PID

lexilla/test/examples/bash/Nested.bsh.styled

@@ -28,6 +28,10 @@
 {8}sub{7}={8}abc{0}
 {8}rep{7}={6}'& '{0}
 {4}echo{0} {10}${var/$sub/"${rep}}"}{0} {2}#{0}
+{2}# issue 216{0}
+{8}option{7}={5}"no[foo]"{0}
+{8}option{7}={10}${option%%[<{().[]*}{0}
+{4}echo{0} {9}$option{0}
 
 {2}# '$' in variable{0}
 {4}echo{0} {9}$${8}PID{0}

lexilla/test/examples/hypertext/x.html

@@ -4,6 +4,9 @@
 var b = /abc/i.test('abc');
 'x\
 </t>'
+// issue 214 fix to behave same as single quote escaped eol
+"x\
+</t>"
 </script>
 <head>
     <meta name="Date.Modified" content="20010515" />

lexilla/test/examples/hypertext/x.html.folded

@@ -4,6 +4,9 @@
  0 402   0 | var b = /abc/i.test('abc');
  0 402   0 | 'x\
  0 402   0 | </t>'
+ 0 402   0 | // issue 214 fix to behave same as single quote escaped eol
+ 0 402   0 | "x\
+ 0 402   0 | </t>"
  0 402   0 | </script>
  2 401   0 + <head>
  0 402   0 |     <meta name="Date.Modified" content="20010515" />

lexilla/test/examples/hypertext/x.html.styled

@@ -4,6 +4,9 @@
 {47}var{41} {46}b{41} {50}={41} {52}/abc/i{46}.test{50}({49}'abc'{50});{41}
 {49}'x\
 </t>'{41}
+{43}// issue 214 fix to behave same as single quote escaped eol{41}
+{48}"x\
+</t>"{41}
 {1}</script>{0}
 {1}<head>{0}
     {1}<meta{8} {3}name{8}={6}"Date.Modified"{8} {3}content{8}={6}"20010515"{8} {11}/>{0}

lexilla/test/examples/lua/AllStyles.lua

@@ -0,0 +1,66 @@
+-- Enumerate all styles: 0 to 20
+-- 3 (comment doc) is not currently produced by lexer
+
+--[[ comment=1 ]]
+
+--[[ whitespace=0 ]]
+	-- w
+
+-- comment line=2
+
+--- comment doc=3
+-- still comment doc
+
+-- still comment doc
+3	-- comment doc broken only by code
+
+-- number=4
+37
+
+-- keyword=5
+local a
+
+-- double-quoted-string=6
+"str"
+
+-- single-quoted-string=7
+'str'
+
+-- literal string=8
+[[ literal ]]
+
+-- unused preprocessor=9
+$if
+
+-- operator=10
+*
+
+-- identifier=11
+identifier=1
+
+-- string EOL=12
+"unclosed
+
+-- keyword 2=13
+print
+
+-- keyword 3=14
+keyword3
+
+-- keyword 4=15
+keyword4
+
+-- keyword 5=16
+keyword5
+
+-- keyword 6=17
+keyword6
+
+-- keyword 7=18
+keyword7
+
+-- keyword 8=19
+keyword8
+
+-- label=20
+::label::

lexilla/test/examples/lua/AllStyles.lua.folded

@@ -0,0 +1,67 @@
+ 0 400   0   -- Enumerate all styles: 0 to 20
+ 0 400   0   -- 3 (comment doc) is not currently produced by lexer
+ 1 400   0   
+ 0 400   0   --[[ comment=1 ]]
+ 1 400   0   
+ 0 400   0   --[[ whitespace=0 ]]
+ 0 400   0   	-- w
+ 1 400   0   
+ 0 400   0   -- comment line=2
+ 1 400   0   
+ 0 400   0   --- comment doc=3
+ 0 400   0   -- still comment doc
+ 1 400   0   
+ 0 400   0   -- still comment doc
+ 0 400   0   3	-- comment doc broken only by code
+ 1 400   0   
+ 0 400   0   -- number=4
+ 0 400   0   37
+ 1 400   0   
+ 0 400   0   -- keyword=5
+ 0 400   0   local a
+ 1 400   0   
+ 0 400   0   -- double-quoted-string=6
+ 0 400   0   "str"
+ 1 400   0   
+ 0 400   0   -- single-quoted-string=7
+ 0 400   0   'str'
+ 1 400   0   
+ 0 400   0   -- literal string=8
+ 0 400   0   [[ literal ]]
+ 1 400   0   
+ 0 400   0   -- unused preprocessor=9
+ 0 400   0   $if
+ 1 400   0   
+ 0 400   0   -- operator=10
+ 0 400   0   *
+ 1 400   0   
+ 0 400   0   -- identifier=11
+ 0 400   0   identifier=1
+ 1 400   0   
+ 0 400   0   -- string EOL=12
+ 0 400   0   "unclosed
+ 1 400   0   
+ 0 400   0   -- keyword 2=13
+ 0 400   0   print
+ 1 400   0   
+ 0 400   0   -- keyword 3=14
+ 0 400   0   keyword3
+ 1 400   0   
+ 0 400   0   -- keyword 4=15
+ 0 400   0   keyword4
+ 1 400   0   
+ 0 400   0   -- keyword 5=16
+ 0 400   0   keyword5
+ 1 400   0   
+ 0 400   0   -- keyword 6=17
+ 0 400   0   keyword6
+ 1 400   0   
+ 0 400   0   -- keyword 7=18
+ 0 400   0   keyword7
+ 1 400   0   
+ 0 400   0   -- keyword 8=19
+ 0 400   0   keyword8
+ 1 400   0   
+ 0 400   0   -- label=20
+ 0 400   0   ::label::
+ 0 400   0   

lexilla/test/examples/lua/AllStyles.lua.styled

@@ -0,0 +1,66 @@
+{2}-- Enumerate all styles: 0 to 20
+-- 3 (comment doc) is not currently produced by lexer
+{0}
+{1}--[[ comment=1 ]]{0}
+
+{1}--[[ whitespace=0 ]]{0}
+	{2}-- w
+{0}
+{2}-- comment line=2
+{0}
+{3}--- comment doc=3
+-- still comment doc
+{0}
+{3}-- still comment doc
+{4}3{0}	{2}-- comment doc broken only by code
+{0}
+{2}-- number=4
+{4}37{0}
+
+{2}-- keyword=5
+{5}local{0} {11}a{0}
+
+{2}-- double-quoted-string=6
+{6}"str"{0}
+
+{2}-- single-quoted-string=7
+{7}'str'{0}
+
+{2}-- literal string=8
+{8}[[ literal ]]{0}
+
+{2}-- unused preprocessor=9
+{9}$if
+{0}
+{2}-- operator=10
+{10}*{0}
+
+{2}-- identifier=11
+{11}identifier{10}={4}1{0}
+
+{2}-- string EOL=12
+{12}"unclosed
+{0}
+{2}-- keyword 2=13
+{13}print{0}
+
+{2}-- keyword 3=14
+{14}keyword3{0}
+
+{2}-- keyword 4=15
+{15}keyword4{0}
+
+{2}-- keyword 5=16
+{16}keyword5{0}
+
+{2}-- keyword 6=17
+{17}keyword6{0}
+
+{2}-- keyword 7=18
+{18}keyword7{0}
+
+{2}-- keyword 8=19
+{19}keyword8{0}
+
+{2}-- label=20
+{20}::label::{0}

lexilla/test/examples/lua/SciTE.properties

@@ -1,4 +1,10 @@
 lexer.*.lua=lua
-keywords.*.lua=function end
+keywords.*.lua=do else elseif end for function if local repeat then until while
 keywords2.*.lua=print
+keywords3.*.lua=keyword3
+keywords4.*.lua=keyword4
+keywords5.*.lua=keyword5
+keywords6.*.lua=keyword6
+keywords7.*.lua=keyword7
+keywords8.*.lua=keyword8
 fold=1

lexilla/test/examples/lua/folding.lua

@@ -0,0 +1,41 @@
+--[[ coding:UTF-8
+folding structure examples ]]
+
+-- Use all the folding keywords:
+--    do end function if repeat until while
+function first()
+   -- Comment
+   if op == "+" then
+      r = a + b
+    elseif op == "-" then
+      r = a - b
+    elseif op == "*" then
+      r = a*b
+    elseif op == "/" then
+      r = a/b
+    else
+      error("invalid operation")
+    end
+
+    for i=1,10 do
+      print(i)
+    end
+
+    while a[i] do
+      print(a[i])
+      i = i + 1
+    end
+
+    -- print the first non-empty line
+    repeat
+      line = io.read()
+    until line ~= ""
+    print(line)
+
+end
+
+-- { ... } folds
+markers = {
+     256,
+     128,
+}

lexilla/test/examples/lua/folding.lua.folded

@@ -0,0 +1,42 @@
+ 2 400   0 + --[[ coding:UTF-8
+ 0 401   0 | folding structure examples ]]
+ 1 400   0   
+ 0 400   0   -- Use all the folding keywords:
+ 0 400   0   --    do end function if repeat until while
+ 2 400   0 + function first()
+ 0 401   0 |    -- Comment
+ 2 401   0 +    if op == "+" then
+ 0 402   0 |       r = a + b
+ 0 402   0 |     elseif op == "-" then
+ 0 402   0 |       r = a - b
+ 0 402   0 |     elseif op == "*" then
+ 0 402   0 |       r = a*b
+ 0 402   0 |     elseif op == "/" then
+ 0 402   0 |       r = a/b
+ 0 402   0 |     else
+ 0 402   0 |       error("invalid operation")
+ 0 402   0 |     end
+ 1 401   0 | 
+ 2 401   0 +     for i=1,10 do
+ 0 402   0 |       print(i)
+ 0 402   0 |     end
+ 1 401   0 | 
+ 2 401   0 +     while a[i] do
+ 0 402   0 |       print(a[i])
+ 0 402   0 |       i = i + 1
+ 0 402   0 |     end
+ 1 401   0 | 
+ 0 401   0 |     -- print the first non-empty line
+ 2 401   0 +     repeat
+ 0 402   0 |       line = io.read()
+ 0 402   0 |     until line ~= ""
+ 0 401   0 |     print(line)
+ 1 401   0 | 
+ 0 401   0 | end
+ 1 400   0   
+ 0 400   0   -- { ... } folds
+ 2 400   0 + markers = {
+ 0 401   0 |      256,
+ 0 401   0 |      128,
+ 0 401   0 | }
+ 0 400   0   

lexilla/test/examples/lua/folding.lua.styled

@@ -0,0 +1,41 @@
+{1}--[[ coding:UTF-8
+folding structure examples ]]{0}
+
+{2}-- Use all the folding keywords:
+--    do end function if repeat until while
+{5}function{0} {11}first{10}(){0}
+   {2}-- Comment
+{0}   {5}if{0} {11}op{0} {10}=={0} {6}"+"{0} {5}then{0}
+      {11}r{0} {10}={0} {11}a{0} {10}+{0} {11}b{0}
+    {5}elseif{0} {11}op{0} {10}=={0} {6}"-"{0} {5}then{0}
+      {11}r{0} {10}={0} {11}a{0} {10}-{0} {11}b{0}
+    {5}elseif{0} {11}op{0} {10}=={0} {6}"*"{0} {5}then{0}
+      {11}r{0} {10}={0} {11}a{10}*{11}b{0}
+    {5}elseif{0} {11}op{0} {10}=={0} {6}"/"{0} {5}then{0}
+      {11}r{0} {10}={0} {11}a{10}/{11}b{0}
+    {5}else{0}
+      {11}error{10}({6}"invalid operation"{10}){0}
+    {5}end{0}
+
+    {5}for{0} {11}i{10}={4}1{10},{4}10{0} {5}do{0}
+      {13}print{10}({11}i{10}){0}
+    {5}end{0}
+
+    {5}while{0} {11}a{10}[{11}i{10}]{0} {5}do{0}
+      {13}print{10}({11}a{10}[{11}i{10}]){0}
+      {11}i{0} {10}={0} {11}i{0} {10}+{0} {4}1{0}
+    {5}end{0}
+
+    {2}-- print the first non-empty line
+{0}    {5}repeat{0}
+      {11}line{0} {10}={0} {11}io.read{10}(){0}
+    {5}until{0} {11}line{0} {10}~={0} {6}""{0}
+    {13}print{10}({11}line{10}){0}
+
+{5}end{0}
+
+{2}-- { ... } folds
+{11}markers{0} {10}={0} {10}{{0}
+     {4}256{10},{0}
+     {4}128{10},{0}
+{10}}{0}

lexilla/test/unit/UnitTester.vcxproj

@@ -88,7 +88,7 @@
       <WarningLevel>Level3</WarningLevel>
       <Optimization>Disabled</Optimization>
       <PreprocessorDefinitions>_CRT_SECURE_NO_WARNINGS=1;_HAS_AUTO_PTR_ETC=1;_SCL_SECURE_NO_WARNINGS=1;CHECK_CORRECTNESS;WIN32;_DEBUG;_CONSOLE;%(PreprocessorDefinitions)</PreprocessorDefinitions>
-      <AdditionalIncludeDirectories>..\..\include\;..\..\src\;..\..\lexlib\;..\..\..\scintilla\include\</AdditionalIncludeDirectories>
+      <AdditionalIncludeDirectories>..\..\include\;..\..\lexlib\;..\..\..\scintilla\include\</AdditionalIncludeDirectories>
       <LanguageStandard>stdcpp17</LanguageStandard>
     </ClCompile>
     <Link>
@@ -103,7 +103,7 @@
       <WarningLevel>Level3</WarningLevel>
       <Optimization>Disabled</Optimization>
       <PreprocessorDefinitions>_CRT_SECURE_NO_WARNINGS=1;_HAS_AUTO_PTR_ETC=1;_SCL_SECURE_NO_WARNINGS=1;CHECK_CORRECTNESS;_DEBUG;_CONSOLE;%(PreprocessorDefinitions)</PreprocessorDefinitions>
-      <AdditionalIncludeDirectories>..\..\include\;..\..\src\;..\..\lexlib\;..\..\..\scintilla\include\</AdditionalIncludeDirectories>
+      <AdditionalIncludeDirectories>..\..\include\;..\..\lexlib\;..\..\..\scintilla\include\</AdditionalIncludeDirectories>
       <LanguageStandard>stdcpp17</LanguageStandard>
     </ClCompile>
     <Link>
@@ -120,7 +120,7 @@
       <FunctionLevelLinking>true</FunctionLevelLinking>
       <IntrinsicFunctions>true</IntrinsicFunctions>
       <PreprocessorDefinitions>_CRT_SECURE_NO_WARNINGS=1;_HAS_AUTO_PTR_ETC=1;_SCL_SECURE_NO_WARNINGS=1;CHECK_CORRECTNESS;WIN32;NDEBUG;_CONSOLE;%(PreprocessorDefinitions)</PreprocessorDefinitions>
-      <AdditionalIncludeDirectories>..\..\include\;..\..\src\;..\..\lexlib\;..\..\..\scintilla\include\</AdditionalIncludeDirectories>
+      <AdditionalIncludeDirectories>..\..\include\;..\..\lexlib\;..\..\..\scintilla\include\</AdditionalIncludeDirectories>
       <LanguageStandard>stdcpp17</LanguageStandard>
     </ClCompile>
     <Link>
@@ -139,7 +139,7 @@
       <FunctionLevelLinking>true</FunctionLevelLinking>
       <IntrinsicFunctions>true</IntrinsicFunctions>
       <PreprocessorDefinitions>_CRT_SECURE_NO_WARNINGS=1;_HAS_AUTO_PTR_ETC=1;_SCL_SECURE_NO_WARNINGS=1;CHECK_CORRECTNESS;NDEBUG;_CONSOLE;%(PreprocessorDefinitions)</PreprocessorDefinitions>
-      <AdditionalIncludeDirectories>..\..\include\;..\..\src\;..\..\lexlib\;..\..\..\scintilla\include\</AdditionalIncludeDirectories>
+      <AdditionalIncludeDirectories>..\..\include\;..\..\lexlib\;..\..\..\scintilla\include\</AdditionalIncludeDirectories>
       <LanguageStandard>stdcpp17</LanguageStandard>
     </ClCompile>
     <Link>

lexilla/test/unit/makefile

@@ -22,7 +22,6 @@ CXX = clang++
 ifdef USELIBCPP
 # macOS, use libc++ but don't have sanitizers
 CXXFLAGS += --stdlib=libc++
-LINKFLAGS = -lc++
 else
 # Linux, have sanitizers
 SANITIZE = -fsanitize=address,undefined
@@ -40,22 +39,26 @@ DEL = rm -f
 EXE = unitTest
 endif
 
-INCLUDEDIRS = -I ../../include -I ../../src -I../../lexlib -I../../../scintilla/include
+vpath %.cxx ../../lexlib
+
+INCLUDEDIRS = -I ../../include -I../../lexlib -I../../../scintilla/include
 
 CPPFLAGS += $(INCLUDEDIRS)
 CXXFLAGS += -Wall -Wextra
 
 # Files in this directory containing tests
-TESTSRC=test*.cxx
+TESTSRC=$(wildcard test*.cxx)
-# Files being tested from scintilla/src directory
+TESTOBJ=$(TESTSRC:.cxx=.o)
-TESTEDSRC=\
+
- ../../lexlib/Accessor.cxx \
+# Files being tested from lexilla/lexlib directory
- ../../lexlib/CharacterSet.cxx \
+TESTEDOBJ=\
- ../../lexlib/LexerBase.cxx \
+ Accessor.o \
- ../../lexlib/LexerModule.cxx \
+ CharacterSet.o \
- ../../lexlib/LexerSimple.cxx \
+ LexerBase.o \
- ../../lexlib/PropSetSimple.cxx \
+ LexerModule.o \
- ../../lexlib/WordList.cxx
+ LexerSimple.o \
+ PropSetSimple.o \
+ WordList.o
 
 TESTS=$(EXE)
 
@@ -67,5 +70,8 @@ test: $(TESTS)
 clean:
 	$(DEL) $(TESTS) *.o *.obj *.exe
 
-$(EXE): $(TESTSRC) $(TESTEDSRC) unitTest.cxx
+%.o: %.cxx
+	$(CXX) $(CPPFLAGS) $(CXXFLAGS) -c $< -o $@
+
+$(EXE): $(TESTOBJ) $(TESTEDOBJ) unitTest.o
 	$(CXX) $(CPPFLAGS) $(CXXFLAGS) $(LINKFLAGS) $^ -o $@

lexilla/test/unit/test.mak

@@ -4,7 +4,7 @@
 DEL = del /q
 EXE = unitTest.exe
 
-INCLUDEDIRS = /I../../include /I../../src /I../../lexlib /I../../../scintilla/include
+INCLUDEDIRS = /I../../include /I../../lexlib /I../../../scintilla/include
 
 CXXFLAGS = /EHsc /std:c++17 /D_HAS_AUTO_PTR_ETC=1 /wd 4805 $(INCLUDEDIRS)
 

lexilla/version.txt

@@ -1 +1 @@
-528
+530

scintilla/call/ScintillaCall.cxx

@@ -2071,11 +2071,11 @@ void ScintillaCall::SetViewEOL(bool visible) {
 	Call(Message::SetViewEOL, visible);
 }
 
-void *ScintillaCall::DocPointer() {
+IDocumentEditable *ScintillaCall::DocPointer() {
-	return reinterpret_cast<void *>(Call(Message::GetDocPointer));
+	return reinterpret_cast<IDocumentEditable *>(Call(Message::GetDocPointer));
 }
 
-void ScintillaCall::SetDocPointer(void *doc) {
+void ScintillaCall::SetDocPointer(IDocumentEditable *doc) {
 	CallPointer(Message::SetDocPointer, 0, doc);
 }
 
@@ -2151,15 +2151,15 @@ int ScintillaCall::Zoom() {
 	return static_cast<int>(Call(Message::GetZoom));
 }
 
-void *ScintillaCall::CreateDocument(Position bytes, Scintilla::DocumentOption documentOptions) {
+IDocumentEditable *ScintillaCall::CreateDocument(Position bytes, Scintilla::DocumentOption documentOptions) {
-	return reinterpret_cast<void *>(Call(Message::CreateDocument, bytes, static_cast<intptr_t>(documentOptions)));
+	return reinterpret_cast<IDocumentEditable *>(Call(Message::CreateDocument, bytes, static_cast<intptr_t>(documentOptions)));
 }
 
-void ScintillaCall::AddRefDocument(void *doc) {
+void ScintillaCall::AddRefDocument(IDocumentEditable *doc) {
 	CallPointer(Message::AddRefDocument, 0, doc);
 }
 
-void ScintillaCall::ReleaseDocument(void *doc) {
+void ScintillaCall::ReleaseDocument(IDocumentEditable *doc) {
 	CallPointer(Message::ReleaseDocument, 0, doc);
 }
 
@@ -2363,10 +2363,18 @@ void ScintillaCall::SetSelectionMode(Scintilla::SelectionMode selectionMode) {
 	Call(Message::SetSelectionMode, static_cast<uintptr_t>(selectionMode));
 }
 
+void ScintillaCall::ChangeSelectionMode(Scintilla::SelectionMode selectionMode) {
+	Call(Message::ChangeSelectionMode, static_cast<uintptr_t>(selectionMode));
+}
+
 SelectionMode ScintillaCall::SelectionMode() {
 	return static_cast<Scintilla::SelectionMode>(Call(Message::GetSelectionMode));
 }
 
+void ScintillaCall::SetMoveExtendsSelection(bool moveExtendsSelection) {
+	Call(Message::SetMoveExtendsSelection, moveExtendsSelection);
+}
+
 bool ScintillaCall::MoveExtendsSelection() {
 	return Call(Message::GetMoveExtendsSelection);
 }
@@ -2879,6 +2887,10 @@ void ScintillaCall::AddSelection(Position caret, Position anchor) {
 	Call(Message::AddSelection, caret, anchor);
 }
 
+int ScintillaCall::SelectionFromPoint(int x, int y) {
+	return static_cast<int>(Call(Message::SelectionFromPoint, x, y));
+}
+
 void ScintillaCall::DropSelectionN(int selection) {
 	Call(Message::DropSelectionN, selection);
 }

scintilla/cocoa/Scintilla/Info.plist

@@ -15,7 +15,7 @@
 	<key>CFBundlePackageType</key>
 	<string>$(PRODUCT_BUNDLE_PACKAGE_TYPE)</string>
 	<key>CFBundleShortVersionString</key>
-	<string>5.3.8</string>
+	<string>5.4.1</string>
 	<key>CFBundleVersion</key>
 	<string>$(CURRENT_PROJECT_VERSION)</string>
 	<key>NSHumanReadableCopyright</key>

scintilla/cocoa/Scintilla/Scintilla.xcodeproj/project.pbxproj

@@ -574,7 +574,7 @@
 				CLANG_WARN_UNREACHABLE_CODE = YES;
 				CLANG_WARN__DUPLICATE_METHOD_MATCH = YES;
 				COPY_PHASE_STRIP = NO;
-				CURRENT_PROJECT_VERSION = 5.3.8;
+				CURRENT_PROJECT_VERSION = 5.4.1;
 				DEAD_CODE_STRIPPING = YES;
 				DEBUG_INFORMATION_FORMAT = dwarf;
 				ENABLE_STRICT_OBJC_MSGSEND = YES;
@@ -598,6 +598,7 @@
 				MTL_ENABLE_DEBUG_INFO = INCLUDE_SOURCE;
 				MTL_FAST_MATH = YES;
 				ONLY_ACTIVE_ARCH = YES;
+				"OTHER_LDFLAGS[arch=*]" = "-Wl,-ld_classic";
 				SDKROOT = macosx;
 				VERSIONING_SYSTEM = "apple-generic";
 				VERSION_INFO_PREFIX = "";
@@ -638,7 +639,7 @@
 				CLANG_WARN_UNREACHABLE_CODE = YES;
 				CLANG_WARN__DUPLICATE_METHOD_MATCH = YES;
 				COPY_PHASE_STRIP = NO;
-				CURRENT_PROJECT_VERSION = 5.3.8;
+				CURRENT_PROJECT_VERSION = 5.4.1;
 				DEAD_CODE_STRIPPING = YES;
 				DEBUG_INFORMATION_FORMAT = "dwarf-with-dsym";
 				ENABLE_NS_ASSERTIONS = NO;
@@ -656,6 +657,7 @@
 				MACOSX_DEPLOYMENT_TARGET = 10.13;
 				MTL_ENABLE_DEBUG_INFO = NO;
 				MTL_FAST_MATH = YES;
+				"OTHER_LDFLAGS[arch=*]" = "-Wl,-ld_classic";
 				SDKROOT = macosx;
 				VERSIONING_SYSTEM = "apple-generic";
 				VERSION_INFO_PREFIX = "";
@@ -670,7 +672,7 @@
 				CODE_SIGN_IDENTITY = "-";
 				CODE_SIGN_STYLE = Manual;
 				COMBINE_HIDPI_IMAGES = YES;
-				CURRENT_PROJECT_VERSION = 5.3.8;
+				CURRENT_PROJECT_VERSION = 5.4.1;
 				DEAD_CODE_STRIPPING = YES;
 				DEFINES_MODULE = YES;
 				DEVELOPMENT_TEAM = "";
@@ -705,7 +707,7 @@
 				CODE_SIGN_IDENTITY = "-";
 				CODE_SIGN_STYLE = Manual;
 				COMBINE_HIDPI_IMAGES = YES;
-				CURRENT_PROJECT_VERSION = 5.3.8;
+				CURRENT_PROJECT_VERSION = 5.4.1;
 				DEAD_CODE_STRIPPING = YES;
 				DEFINES_MODULE = YES;
 				DEVELOPMENT_TEAM = "";

scintilla/doc/ScintillaDoc.html

@@ -129,7 +129,7 @@
 
     <h1>Scintilla Documentation</h1>
 
-    <p>Last edited 1 November 2023 NH</p>
+    <p>Last edited 19 December 2023 NH</p>
 
     <p style="background:#90F0C0">Scintilla 5 has moved the lexers from Scintilla into a new
     <a href="Lexilla.html">Lexilla</a> project.<br />
@@ -402,34 +402,35 @@
         <tr>
           <td>&cir; <a class="toc" href="#MultipleViews">Multiple views</a></td>
           <td>&cir; <a class="toc" href="#BackgroundLoadSave">Background loading and saving</a></td>
-          <td>&cir; <a class="toc" href="#Folding">Folding</a></td>
+          <td>&cir; <a class="toc" href="#DocumentInterface">Document interface</a></td>
         </tr>
 
         <tr>
+          <td>&cir; <a class="toc" href="#Folding">Folding</a></td>
           <td>&cir; <a class="toc" href="#LineWrapping">Line wrapping</a></td>
           <td>&cir; <a class="toc" href="#Zooming">Zooming</a></td>
-          <td>&cir; <a class="toc" href="#LongLines">Long lines</a></td>
         </tr>
 
         <tr>
+          <td>&cir; <a class="toc" href="#LongLines">Long lines</a></td>
           <td>&cir; <a class="toc" href="#Accessibility">Accessibility</a></td>
           <td>&cir; <a class="toc" href="#Lexer">Lexer</a></td>
-          <td>&cir; <a class="toc" href="#LexerObjects">Lexer objects</a></td>
         </tr>
 
         <tr>
+          <td>&cir; <a class="toc" href="#LexerObjects">Lexer objects</a></td>
           <td>&cir; <a class="toc" href="#Notifications">Notifications</a></td>
           <td>&cir; <a class="toc" href="#Images">Images</a></td>
-          <td>&cir; <a class="toc" href="#GTK">GTK</a></td>
         </tr>
 
         <tr>
+          <td>&cir; <a class="toc" href="#GTK">GTK</a></td>
           <td>&cir; <a class="toc" href="#ProvisionalMessages"><span class="provisional">Provisional messages</span></a></td>
           <td>&cir; <a class="toc" href="#DeprecatedMessages">Deprecated messages</a></td>
-          <td>&cir; <a class="toc" href="#EditMessagesNeverSupportedByScintilla">Edit messages never supported by Scintilla</a></td>
         </tr>
 
         <tr>
+          <td>&cir; <a class="toc" href="#EditMessagesNeverSupportedByScintilla">Edit messages never supported by Scintilla</a></td>
           <td>&cir; <a class="toc" href="#RemovedFeatures">Removed features</a></td>
           <td>&cir; <a class="toc" href="#BuildingScintilla">Building Scintilla</a></td>
         </tr>
@@ -979,7 +980,9 @@ struct Sci_TextRangeFull {
      <a class="message" href="#SCI_GETCURLINE">SCI_GETCURLINE(position length, char *text) &rarr; position</a><br />
      <a class="message" href="#SCI_SELECTIONISRECTANGLE">SCI_SELECTIONISRECTANGLE &rarr; bool</a><br />
      <a class="message" href="#SCI_SETSELECTIONMODE">SCI_SETSELECTIONMODE(int selectionMode)</a><br />
+     <a class="message" href="#SCI_CHANGESELECTIONMODE">SCI_CHANGESELECTIONMODE(int selectionMode)</a><br />
      <a class="message" href="#SCI_GETSELECTIONMODE">SCI_GETSELECTIONMODE &rarr; int</a><br />
+     <a class="message" href="#SCI_SETMOVEEXTENDSSELECTION">SCI_SETMOVEEXTENDSSELECTION(bool moveExtendsSelection)</a><br />
      <a class="message" href="#SCI_GETMOVEEXTENDSSELECTION">SCI_GETMOVEEXTENDSSELECTION &rarr; bool</a><br />
      <a class="message" href="#SCI_GETLINESELSTARTPOSITION">SCI_GETLINESELSTARTPOSITION(line line) &rarr; position</a><br />
      <a class="message" href="#SCI_GETLINESELENDPOSITION">SCI_GETLINESELENDPOSITION(line line) &rarr; position</a><br />
@@ -1121,22 +1124,27 @@ struct Sci_TextRangeFull {
     <p><b id="SCI_SELECTIONISRECTANGLE">SCI_SELECTIONISRECTANGLE &rarr; bool</b><br />
      This returns 1 if the current selection is in rectangle mode, 0 if not.</p>
 
-    <p><b id="SCI_SETSELECTIONMODE">SCI_SETSELECTIONMODE(int selectionMode)</b><br />
+    <p>
+    <b id="SCI_SETSELECTIONMODE">SCI_SETSELECTIONMODE(int selectionMode)</b><br />
+    <b id="SCI_CHANGESELECTIONMODE">SCI_CHANGESELECTIONMODE(int selectionMode)</b><br />
     <b id="SCI_GETSELECTIONMODE">SCI_GETSELECTIONMODE &rarr; int</b><br />
-    The two functions set and get the selection mode, which can be
+    The functions set, change, and get the selection mode, which can be
      stream (<code>SC_SEL_STREAM</code>=0) or
      rectangular (<code>SC_SEL_RECTANGLE</code>=1) or
      by lines (<code>SC_SEL_LINES</code>=2)
      or thin rectangular (<code>SC_SEL_THIN</code>=3).
-     When set in these modes, regular caret moves will extend or reduce the selection,
+     When <code>SCI_SETSELECTIONMODE</code> sets these modes, regular caret moves will extend or reduce the selection,
-     until the mode is cancelled by a call with same value or with <code>SCI_CANCEL</code>.
+     until the mode is cancelled by a call with same value, or with <code>SCI_CANCEL</code>, or with <code>SCI_SETMOVEEXTENDSSELECTION</code>.
-     The get function returns the current mode even if the selection was made by mouse
+     <code>SCI_CHANGESELECTIONMODE</code> sets the mode but does not make regular caret moves extend or reduce the selection.</p>
-     or with regular extended moves.
+     <p>The get function returns the current mode even if the selection was made by mouse or with regular extended moves.
      <code>SC_SEL_THIN</code> is the mode after a rectangular selection has been typed into and ensures
      that no characters are selected.</p>
 
-    <p><b id="SCI_GETMOVEEXTENDSSELECTION">SCI_GETMOVEEXTENDSSELECTION &rarr; bool</b><br />
+    <p>
-     This returns 1 if regular caret moves will extend or reduce the selection, 0 if not.
+    <b id="SCI_SETMOVEEXTENDSSELECTION">SCI_SETMOVEEXTENDSSELECTION(bool moveExtendsSelection)</b><br />
+    <b id="SCI_GETMOVEEXTENDSSELECTION">SCI_GETMOVEEXTENDSSELECTION &rarr; bool</b><br />
+     This controls whether regular caret moves extends the selection leaving the anchor unchanged.
+     It is 1 if regular caret moves will extend or reduce the selection, 0 if not.
      <code>SCI_SETSELECTIONMODE</code> toggles this setting between on and off.</p>
 
     <p><b id="SCI_GETLINESELSTARTPOSITION">SCI_GETLINESELSTARTPOSITION(line line) &rarr; position</b><br />
@@ -1192,6 +1200,7 @@ struct Sci_TextRangeFull {
      <a class="message" href="#SCI_CLEARSELECTIONS">SCI_CLEARSELECTIONS</a><br />
      <a class="message" href="#SCI_SETSELECTION">SCI_SETSELECTION(position caret, position anchor)</a><br />
      <a class="message" href="#SCI_ADDSELECTION">SCI_ADDSELECTION(position caret, position anchor)</a><br />
+     <a class="message" href="#SCI_SELECTIONFROMPOINT">SCI_SELECTIONFROMPOINT(int x, int y) &rarr; int</a><br />
      <a class="message" href="#SCI_DROPSELECTIONN">SCI_DROPSELECTIONN(int selection)</a><br />
      <a class="message" href="#SCI_SETMAINSELECTION">SCI_SETMAINSELECTION(int selection)</a><br />
      <a class="message" href="#SCI_GETMAINSELECTION">SCI_GETMAINSELECTION &rarr; int</a><br />
@@ -1336,6 +1345,11 @@ struct Sci_TextRangeFull {
      Since there is always at least one selection, to set a list of selections, the first selection should be
      added with <code>SCI_SETSELECTION</code> and later selections added with <code>SCI_ADDSELECTION</code></p>
 
+    <p>
+    <b id="SCI_SELECTIONFROMPOINT">SCI_SELECTIONFROMPOINT(int x, int y) &rarr; int</b><br />
+     Return the index of the selection at the point. If there is no selection at the point, return -1.
+     This can be used to drop a selection or make it the main selection.</p>
+
     <p>
     <b id="SCI_DROPSELECTIONN">SCI_DROPSELECTIONN(int selection)</b><br />
      If there are multiple selections, remove the indicated selection.
@@ -7190,9 +7204,11 @@ sptr_t CallScintilla(unsigned int iMessage, uptr_t wParam, sptr_t lParam){
     windows (for use with splitter windows).</p>
 
     <p>These messages use <code>pointer</code> returns and arguments to refer to documents.
-    They point to internal objects inside Scintilla and should be treated as an opaque <code>void*</code>. That
+    They point to <a class="seealso" href="#IDocumentEditable" style="background:#FFB000">IDocumentEditable</a> objects inside
-    is, you can use and store the pointer as described in this section but you should not
+    Scintilla.
-    dereference it.</p>
+    The <code>IDocumentEditable</code> interface is provisional and may change.
+    Client code can call <code>IDocumentEditable</code> methods.
+    Clients may just treat these as opaque <code>void*</code> values that are received from and passed to Scintilla without dereferencing.</p>
     <code><a class="message" href="#SCI_GETDOCPOINTER">SCI_GETDOCPOINTER &rarr; pointer</a><br />
      <a class="message" href="#SCI_SETDOCPOINTER">SCI_SETDOCPOINTER(&lt;unused&gt;, pointer doc)</a><br />
      <a class="message" href="#SCI_CREATEDOCUMENT">SCI_CREATEDOCUMENT(position bytes, int documentOptions) &rarr; pointer</a><br />
@@ -7326,7 +7342,7 @@ sptr_t CallScintilla(unsigned int iMessage, uptr_t wParam, sptr_t lParam){
     <p>The <code class="parameter">documentOptions</code> argument
     is described in the <a class="seealso" href="#documentOptions"><code>SCI_CREATEDOCUMENT</code></a> section.</p>
 
-<h4>ILoader</h4>
+<h4 id="ILoader">ILoader</h4>
 
 <div class="highlighted">
 <span class="S5">class</span><span class="S0"> </span>ILoader<span class="S0"> </span><span class="S10">{</span><br />
@@ -7343,7 +7359,9 @@ sptr_t CallScintilla(unsigned int iMessage, uptr_t wParam, sptr_t lParam){
     If a failure occurs in <code>AddData</code> or in a file reading call then loading can be abandoned and the loader released with
     the <code>Release</code> call.
     When the whole file has been read, <code>ConvertToDocument</code> should be called to produce a Scintilla
-    document pointer. The newly created document will have a reference count of 1 in the same way as a document pointer
+    document pointer. This pointer can be treated as a <code>void*</code> cookie to pass to other APIs or cast to a
+    <a class="seealso" href="#IDocumentEditable" style="background:#FFB000">IDocumentEditable*</a> pointer.
+    The newly created document will have a reference count of 1 in the same way as a document pointer
     returned from
     <a class="seealso" href="#SCI_CREATEDOCUMENT">SCI_CREATEDOCUMENT</a>.
     There is no need to call <code>Release</code> after <code>ConvertToDocument</code>.</p>
@@ -7360,6 +7378,46 @@ sptr_t CallScintilla(unsigned int iMessage, uptr_t wParam, sptr_t lParam){
     The application may then decide to ignore the modification or to terminate the background saving thread and reenable
     modification before returning from the notification.</p>
 
+   <h2 id="DocumentInterface" class="provisional">Document interface</h2>
+
+    <p>Applications may want to manipulate documents that are not visible and the provisional <code>IDocumentEditable</code>
+    interface can be used for this.</p>
+
+    <p><code>IDocumentEditable</code> allows more direct access to functionality and is faster than calling Scintilla APIs.</p>
+
+    <p><code>IDocumentEditable</code> pointers are returned by
+    <a class="seealso" href="#SCI_CREATEDOCUMENT">SCI_CREATEDOCUMENT</a>,
+    <a class="seealso" href="#SCI_GETDOCPOINTER">SCI_GETDOCPOINTER</a>, and
+    <a class="seealso" href="#ILoader">ILoader::ConvertToDocument</a>.</p>
+
+    <p>They may be passed to
+    <a class="seealso" href="#SCI_ADDREFDOCUMENT">SCI_ADDREFDOCUMENT</a>,
+    <a class="seealso" href="#SCI_RELEASEDOCUMENT">SCI_RELEASEDOCUMENT</a>, and
+    <a class="seealso" href="#SCI_SETDOCPOINTER">SCI_SETDOCPOINTER</a>,
+    .</p>
+
+<h4 id="IDocumentEditable">IDocumentEditable</h4>
+
+<div class="highlighted" style="background:#FFD270;border: 1px solid #FFBB00">
+<span><span class="S5">class</span><span class="S0"> </span>IDocumentEditable<span class="S0"> </span><span class="S10">{</span><br />
+<span class="S5">public</span><span class="S10">:</span><br />
+<span class="S0">&nbsp; &nbsp; &nbsp; &nbsp; </span><span class="S2">// Allow this interface to add methods over time and discover whether new methods available.</span><br />
+<span class="S0">&nbsp; &nbsp; &nbsp; &nbsp; </span><span class="S5">virtual</span><span class="S0"> </span><span class="S5">int</span><span class="S0"> </span>SCI_METHOD<span class="S0"> </span>DEVersion<span class="S10">()</span><span class="S0"> </span><span class="S5">const</span><span class="S0"> </span><span class="S5">noexcept</span><span class="S0"> </span><span class="S10">=</span><span class="S0"> </span><span class="S4">0</span><span class="S10">;</span><br />
+<br />
+<span class="S0">&nbsp; &nbsp; &nbsp; &nbsp; </span><span class="S2">// Lifetime control</span><br />
+<span class="S0">&nbsp; &nbsp; &nbsp; &nbsp; </span><span class="S5">virtual</span><span class="S0"> </span><span class="S5">int</span><span class="S0"> </span>SCI_METHOD<span class="S0"> </span>AddRef<span class="S10">()</span><span class="S0"> </span><span class="S5">noexcept</span><span class="S0"> </span><span class="S10">=</span><span class="S0"> </span><span class="S4">0</span><span class="S10">;</span><br />
+<span class="S0">&nbsp; &nbsp; &nbsp; &nbsp; </span><span class="S5">virtual</span><span class="S0"> </span><span class="S5">int</span><span class="S0"> </span>SCI_METHOD<span class="S0"> </span>Release<span class="S10">()</span><span class="S0"> </span><span class="S10">=</span><span class="S0"> </span><span class="S4">0</span><span class="S10">;</span><br />
+<span class="S10">};</span><br />
+<span class="S0"></span></span>
+</div>
+
+    <p>The <code>IDocumentEditable</code> interface is being developed and more methods will be added in the future.
+    Its also possible that methods will change signatures or be removed.
+    Thus the feature is provisional and users should be aware that they may have to modify client code in response to these changes.</p>
+
+    <p><code>DEVersion</code> will return 0 while <code>IDocumentEditable</code> is provisional and will return 1
+    for the first stable release. After that, it will be incremented when new methods are added.</p>
+
     <h2 id="Folding">Folding</h2>
 
     <p>The fundamental operation in folding is making lines invisible or visible. Line visibility

scintilla/doc/ScintillaDownload.html

@@ -26,9 +26,9 @@
     <table bgcolor="#CCCCCC" width="100%" cellspacing="0" cellpadding="8" border="0">
       <tr>
         <td>
-          <font size="4"> <a href="https://www.scintilla.org/scintilla538.zip">
+          <font size="4"> <a href="https://www.scintilla.org/scintilla541.zip">
 	Windows</a>&nbsp;&nbsp;
-	<a href="https://www.scintilla.org/scintilla538.tgz">
+	<a href="https://www.scintilla.org/scintilla541.tgz">
           GTK/Linux</a>&nbsp;&nbsp;
 	</font>
         </td>
@@ -42,7 +42,7 @@
       containing very few restrictions.
     </p>
     <h3>
-       Release 5.3.8
+       Release 5.4.1
     </h3>
     <h4>
        Source Code
@@ -50,8 +50,8 @@
        The source code package contains all of the source code for Scintilla but no binary
 	executable code and is available in
        <ul>
-       <li><a href="https://www.scintilla.org/scintilla538.zip">zip format</a> (1.7M) commonly used on Windows</li>
+       <li><a href="https://www.scintilla.org/scintilla541.zip">zip format</a> (1.8M) commonly used on Windows</li>
-       <li><a href="https://www.scintilla.org/scintilla538.tgz">tgz format</a> (1.6M) commonly used on Linux and compatible operating systems</li>
+       <li><a href="https://www.scintilla.org/scintilla541.tgz">tgz format</a> (1.6M) commonly used on Linux and compatible operating systems</li>
        </ul>
        Instructions for building on both Windows and Linux are included in the readme file.
     <h4>

scintilla/doc/ScintillaHistory.html

@@ -583,6 +583,76 @@
     </tr>
     </table>
     <h2>Releases</h2>
+    <h3>
+       <a href="https://www.scintilla.org/scintilla541.zip">Release 5.4.1</a>
+    </h3>
+    <ul>
+	<li>
+	Released 27 December 2023.
+	</li>
+	<li>
+	Add IDocumentEditable interface to allow efficient interaction with document objects which may not be visible in
+	a Scintilla instance. This feature is provisonal and may change before being declared stable.
+	For better type-safety, the ScintillaCall C++ API uses IDocumentEditable* where void* was used before which may require
+	changes to client code that uses document pointer APIs
+	DocPointer, SetDocPointer, CreateDocument, AddRefDocument, and ReleaseDocument.
+	</li>
+	<li>
+	Ctrl-click on a selection deselects it in multiple selection mode.
+	</li>
+	<li>
+	Add SCI_SELECTIONFROMPOINT for modifying multiple selections.
+	</li>
+	<li>
+	Add SCI_SETMOVEEXTENDSSELECTION and SCI_CHANGESELECTIONMODE to
+	simplify selection mode manipulation.
+	</li>
+	<li>
+	Improve performance of global replace by reducing cache invalidation overhead.
+	<a href="https://sourceforge.net/p/scintilla/feature-requests/1502/">Feature #1502</a>.
+	</li>
+	<li>
+	Fix regular expression search for "\&lt;" matching beginning of search when not beginning of word and
+	for "\&gt;" not matching line end.
+	<a href="https://sourceforge.net/p/scintilla/bugs/2157/">Bug #2157</a>.
+	</li>
+	<li>
+	Fix regular expression search failure when search for "\&lt;" followed by search for "\&gt;".
+	<a href="https://sourceforge.net/p/scintilla/bugs/2413/">Bug #2413</a>.
+	</li>
+	<li>
+	Fix regular expression assertion (^, $, \b. \B) failures when using SCFIND_CXX11REGEX.
+	<a href="https://sourceforge.net/p/scintilla/bugs/2405/">Bug #2405</a>.
+	</li>
+	<li>
+	Fix regular expression bug in reverse direction where shortened match returned.
+	<a href="https://sourceforge.net/p/scintilla/bugs/2405/">Bug #2405</a>.
+	</li>
+	<li>
+	Avoid character fragments in regular expression search results.
+	<a href="https://sourceforge.net/p/scintilla/bugs/2405/">Bug #2405</a>.
+	</li>
+	<li>
+	With a document that does not have the SC_DOCUMENTOPTION_TEXT_LARGE option set,
+	allocating more than 2G (calling SCI_ALLOCATE or similar) will now fail with SC_STATUS_FAILURE.
+	</li>
+	<li>
+	Protect SCI_REPLACETARGET, SCI_REPLACETARGETMINIMAL, and SCI_REPLACETARGETRE from
+	application changing target in notification handlers.
+	<a href="https://sourceforge.net/p/scintilla/bugs/2289/">Bug #2289</a>.
+	</li>
+    </ul>
+    <h3>
+       <a href="https://www.scintilla.org/scintilla540.zip">Release 5.4.0</a>
+    </h3>
+    <ul>
+	<li>
+	Released 18 November 2023.
+	</li>
+	<li>
+	Fix crashes on macOS 12 and older when built with Xcode 15.0.
+	</li>
+    </ul>
     <h3>
        <a href="https://www.scintilla.org/scintilla538.zip">Release 5.3.8</a>
     </h3>

scintilla/doc/index.html

@@ -9,7 +9,7 @@
     <meta name="keywords" content="Scintilla, SciTE, Editing Component, Text Editor" />
     <meta name="Description"
     content="www.scintilla.org is the home of the Scintilla editing component and SciTE text editor application." />
-    <meta name="Date.Modified" content="20231105" />
+    <meta name="Date.Modified" content="20231227" />
     <meta name="viewport" content="width=device-width, initial-scale=1" />
     <style type="text/css">
         .logo {
@@ -60,8 +60,8 @@
           GTK, and macOS</font>
         </td>
         <td width="40%" align="right">
-          <font color="#FFCC99" size="3"> Release version 5.3.8<br />
+          <font color="#FFCC99" size="3"> Release version 5.4.1<br />
-           Site last modified November 5 2023</font>
+           Site last modified December 27 2023</font>
         </td>
         <td width="20%">
           &nbsp;
@@ -76,11 +76,11 @@
       </tr>
     </table>
     <ul id="versionlist">
+      <li>Version 5.4.1 adds IDocumentEditable interface to allow efficient interaction with document objects.</li>
+      <li>Version 5.4.0 fixes crashes on macOS 12 and older when built with Xcode 15.0.</li>
       <li>Version 5.3.8 fixes excesssive memory use when deleting contiguous ranges backwards and is compatible with new macOS 14.</li>
       <li>Version 5.3.7 fixes platform-specific issues on GTK, Qt, and Win32.</li>
       <li>Version 5.3.6 improves cursor behaviour on Win32 and IME support on Win32 and Qt.</li>
-      <li>Version 5.3.5 improves IME support on Win32 and Qt.</li>
-      <li>Version 5.3.4 adds multithreaded wrapping.</li>
     </ul>
     <ul id="menu">
       <li id="remote1"><a href="https://www.scintilla.org/SciTEImage.html">Screenshot</a></li>

scintilla/include/ILoader.h

@@ -1,6 +1,7 @@
 // Scintilla source code edit control
 /** @file ILoader.h
  ** Interface for loading into a Scintilla document from a background thread.
+ ** Interface for manipulating a document without a view.
  **/
 // Copyright 1998-2017 by Neil Hodgson <neilh@scintilla.org>
 // The License.txt file describes the conditions under which this software may be distributed.
@@ -20,6 +21,18 @@ public:
 	virtual void * SCI_METHOD ConvertToDocument() = 0;
 };
 
+static constexpr int deRelease0 = 0;
+
+class IDocumentEditable {
+public:
+	// Allow this interface to add methods over time and discover whether new methods available.
+	virtual int SCI_METHOD DEVersion() const noexcept = 0;
+
+	// Lifetime control
+	virtual int SCI_METHOD AddRef() noexcept = 0;
+	virtual int SCI_METHOD Release() = 0;
+};
+
 }
 
 #endif

scintilla/include/Scintilla.h

@@ -875,7 +875,9 @@ typedef sptr_t (*SciFnDirectStatus)(sptr_t ptr, unsigned int iMessage, uptr_t wP
 #define SC_SEL_LINES 2
 #define SC_SEL_THIN 3
 #define SCI_SETSELECTIONMODE 2422
+#define SCI_CHANGESELECTIONMODE 2659
 #define SCI_GETSELECTIONMODE 2423
+#define SCI_SETMOVEEXTENDSSELECTION 2719
 #define SCI_GETMOVEEXTENDSSELECTION 2706
 #define SCI_GETLINESELSTARTPOSITION 2424
 #define SCI_GETLINESELENDPOSITION 2425
@@ -1021,6 +1023,7 @@ typedef sptr_t (*SciFnDirectStatus)(sptr_t ptr, unsigned int iMessage, uptr_t wP
 #define SCI_CLEARSELECTIONS 2571
 #define SCI_SETSELECTION 2572
 #define SCI_ADDSELECTION 2573
+#define SCI_SELECTIONFROMPOINT 2474
 #define SCI_DROPSELECTIONN 2671
 #define SCI_SETMAINSELECTION 2574
 #define SCI_GETMAINSELECTION 2575

scintilla/include/Scintilla.iface

@@ -2359,9 +2359,16 @@ val SC_SEL_THIN=3
 # by lines (SC_SEL_LINES).
 set void SetSelectionMode=2422(SelectionMode selectionMode,)
 
+# Set the selection mode to stream (SC_SEL_STREAM) or rectangular (SC_SEL_RECTANGLE/SC_SEL_THIN) or
+# by lines (SC_SEL_LINES) without changing MoveExtendsSelection.
+fun void ChangeSelectionMode=2659(SelectionMode selectionMode,)
+
 # Get the mode of the current selection.
 get SelectionMode GetSelectionMode=2423(,)
 
+# Set whether or not regular caret moves will extend or reduce the selection.
+set void SetMoveExtendsSelection=2719(bool moveExtendsSelection,)
+
 # Get whether or not regular caret moves will extend or reduce the selection.
 get bool GetMoveExtendsSelection=2706(,)
 
@@ -2790,6 +2797,9 @@ fun void SetSelection=2572(position caret, position anchor)
 # Add a selection
 fun void AddSelection=2573(position caret, position anchor)
 
+# Find the selection index for a point. -1 when not at a selection.
+fun int SelectionFromPoint=2474(int x, int y)
+
 # Drop one selection
 fun void DropSelectionN=2671(int selection,)
 

scintilla/include/ScintillaCall.h

@@ -20,6 +20,8 @@ struct TextRangeFull;
 struct TextToFindFull;
 struct RangeToFormatFull;
 
+class IDocumentEditable;
+
 using FunctionDirect = intptr_t(*)(intptr_t ptr, unsigned int iMessage, uintptr_t wParam, intptr_t lParam, int *pStatus);
 
 struct Failure {
@@ -561,8 +563,8 @@ public:
 	Position BraceMatchNext(Position pos, Position startPos);
 	bool ViewEOL();
 	void SetViewEOL(bool visible);
-	void *DocPointer();
+	IDocumentEditable *DocPointer();
-	void SetDocPointer(void *doc);
+	void SetDocPointer(IDocumentEditable *doc);
 	void SetModEventMask(Scintilla::ModificationFlags eventMask);
 	Position EdgeColumn();
 	void SetEdgeColumn(Position column);
@@ -581,9 +583,9 @@ public:
 	bool SelectionIsRectangle();
 	void SetZoom(int zoomInPoints);
 	int Zoom();
-	void *CreateDocument(Position bytes, Scintilla::DocumentOption documentOptions);
+	IDocumentEditable *CreateDocument(Position bytes, Scintilla::DocumentOption documentOptions);
-	void AddRefDocument(void *doc);
+	void AddRefDocument(IDocumentEditable *doc);
-	void ReleaseDocument(void *doc);
+	void ReleaseDocument(IDocumentEditable *doc);
 	Scintilla::DocumentOption DocumentOptions();
 	Scintilla::ModificationFlags ModEventMask();
 	void SetCommandEvents(bool commandEvents);
@@ -634,7 +636,9 @@ public:
 	void CopyRange(Position start, Position end);
 	void CopyText(Position length, const char *text);
 	void SetSelectionMode(Scintilla::SelectionMode selectionMode);
+	void ChangeSelectionMode(Scintilla::SelectionMode selectionMode);
 	Scintilla::SelectionMode SelectionMode();
+	void SetMoveExtendsSelection(bool moveExtendsSelection);
 	bool MoveExtendsSelection();
 	Position GetLineSelStartPosition(Line line);
 	Position GetLineSelEndPosition(Line line);
@@ -763,6 +767,7 @@ public:
 	void ClearSelections();
 	void SetSelection(Position caret, Position anchor);
 	void AddSelection(Position caret, Position anchor);
+	int SelectionFromPoint(int x, int y);
 	void DropSelectionN(int selection);
 	void SetMainSelection(int selection);
 	int MainSelection();

scintilla/include/ScintillaMessages.h

@@ -556,7 +556,9 @@ enum class Message {
 	CopyRange = 2419,
 	CopyText = 2420,
 	SetSelectionMode = 2422,
+	ChangeSelectionMode = 2659,
 	GetSelectionMode = 2423,
+	SetMoveExtendsSelection = 2719,
 	GetMoveExtendsSelection = 2706,
 	GetLineSelStartPosition = 2424,
 	GetLineSelEndPosition = 2425,
@@ -676,6 +678,7 @@ enum class Message {
 	ClearSelections = 2571,
 	SetSelection = 2572,
 	AddSelection = 2573,
+	SelectionFromPoint = 2474,
 	DropSelectionN = 2671,
 	SetMainSelection = 2574,
 	GetMainSelection = 2575,

scintilla/qt/ScintillaEdit/ScintillaEdit.pro

@@ -13,7 +13,7 @@ TEMPLATE = lib
 CONFIG += lib_bundle
 CONFIG += c++1z
 
-VERSION = 5.3.8
+VERSION = 5.4.1
 
 SOURCES += \
     ScintillaEdit.cpp \

scintilla/qt/ScintillaEditBase/ScintillaEditBase.pro

@@ -6,13 +6,14 @@
 
 QT       += core gui
 greaterThan(QT_MAJOR_VERSION, 4): QT += widgets
+equals(QT_MAJOR_VERSION, 6): QT += core5compat
 
 TARGET = ScintillaEditBase
 TEMPLATE = lib
 CONFIG += lib_bundle
 CONFIG += c++1z
 
-VERSION = 5.3.8
+VERSION = 5.4.1
 
 SOURCES += \
     PlatQt.cpp \

scintilla/scripts/Face.py

@@ -5,9 +5,8 @@
 # Requires Python 2.7 or later
 
 def sanitiseLine(line):
-	if line[-1:] == '\n':
+	line = line.rstrip('\n')
-		line = line[:-1]
+	if "##" in line:
-	if line.find("##") != -1:
 		line = line[:line.find("##")]
 	line = line.strip()
 	return line

scintilla/scripts/FileGenerator.py

@@ -172,50 +172,12 @@ def UpdateLineInFile(path, linePrefix, lineReplace):
     contents = lineEnd.join(lines) + lineEnd
     UpdateFile(path, contents)
 
-def ReadFileAsList(path):
-    """Read all the lnes in the file and return as a list of strings without line ends.
-    """
-    with codecs.open(path, "r", "utf-8") as f:
-        return [line.rstrip('\n') for line in f]
-
 def UpdateFileFromLines(path, lines, lineEndToUse):
     """Join the lines with the lineEndToUse then update file if the result is different.
     """
     contents = lineEndToUse.join(lines) + lineEndToUse
     UpdateFile(path, contents)
 
-def FindSectionInList(lines, markers):
-    """Find a section defined by an initial start marker, an optional secondary
-    marker and an end marker.
-    The section is between the secondary/initial start and the end.
-    Report as a slice object so the section can be extracted or replaced.
-    Raises an exception if the markers can't be found.
-    """
-    start = -1
-    end = -1
-    state = 0
-    for i, line in enumerate(lines):
-        if markers[0] in line:
-            if markers[1]:
-                state = 1
-            else:
-                start = i+1
-                state = 2
-        elif state == 1:
-            if markers[1] in line:
-                start = i+1
-                state = 2
-        elif state == 2:
-            if markers[2] in line:
-                end = i
-                state = 3
-    # Check that section was found
-    if start == -1:
-        raise Exception("Could not find start marker(s) |" + markers[0] + "|" + markers[1] + "|")
-    if end == -1:
-        raise Exception("Could not find end marker " + markers[2])
-    return slice(start, end)
-
 def ReplaceREInFile(path, match, replace, count=1):
     with codecs.open(path, "r", "utf-8") as f:
         contents = f.read()

scintilla/scripts/ScintillaAPIFacer.py

@@ -55,7 +55,9 @@ deadValues = [
 ]
 
 def ActualTypeName(type, identifier=None):
-	if type in typeAliases:
+	if type == "pointer" and identifier in ["doc", "DocPointer", "CreateDocument"]:
+		return "IDocumentEditable *"
+	elif type in typeAliases:
 		return typeAliases[type]
 	else:
 		return type
@@ -63,6 +65,8 @@ def ActualTypeName(type, identifier=None):
 def IsEnumeration(s):
 	if s in ["Position", "Line", "Colour", "ColourAlpha"]:
 		return False
+	if s.endswith("*"):
+		return False
 	return s[:1].isupper()
 
 def JoinTypeAndIdentifier(type, identifier):
@@ -219,7 +223,7 @@ def HMethods(f):
 			if v["FeatureType"] in ["fun", "get", "set"]:
 				if v["FeatureType"] == "get" and name.startswith("Get"):
 					name = name[len("Get"):]
-				retType = ActualTypeName(v["ReturnType"])
+				retType = ActualTypeName(v["ReturnType"], name)
 				if IsEnumeration(retType):
 					retType = namespace + retType
 				parameters, args, callName = ParametersArgsCallname(v)
@@ -241,21 +245,21 @@ def CXXMethods(f):
 				msgName = "Message::" + name
 				if v["FeatureType"] == "get" and name.startswith("Get"):
 					name = name[len("Get"):]
-				retType = ActualTypeName(v["ReturnType"])
+				retType = ActualTypeName(v["ReturnType"], name)
 				parameters, args, callName = ParametersArgsCallname(v)
 				returnIfNeeded = "return " if retType != "void" else ""
 
 				out.append(JoinTypeAndIdentifier(retType, "ScintillaCall::" + name) + "(" + parameters + ")" + " {")
 				retCast = ""
 				retCastEnd = ""
-				if retType not in basicTypes or retType in ["int", "Colour", "ColourAlpha"]:
+				if retType.endswith("*"):
+					retCast = "reinterpret_cast<" + retType + ">("
+					retCastEnd = ")"
+				elif retType not in basicTypes or retType in ["int", "Colour", "ColourAlpha"]:
 					if IsEnumeration(retType):
 						retType = namespace + retType
 					retCast = "static_cast<" + retType + ">("
 					retCastEnd = ")"
-				elif retType in ["void *"]:
-					retCast = "reinterpret_cast<" + retType + ">("
-					retCastEnd = ")"
 				out.append("\t" + returnIfNeeded + retCast + callName + "(" + msgName + args + ")" + retCastEnd + ";")
 				out.append("}")
 				out.append("")

scintilla/src/CallTip.cxx

@@ -164,7 +164,7 @@ int CallTip::DrawChunk(Surface *surface, int x, std::string_view sv,
 	size_t startSeg = 0;
 	for (const size_t endSeg : ends) {
 		assert(endSeg > 0);
-		int xEnd;
+		int xEnd = 0;
 		if (IsArrowCharacter(sv[startSeg])) {
 			xEnd = x + widthArrow;
 			const bool upArrow = sv[startSeg] == '\001';
@@ -228,7 +228,8 @@ int CallTip::PaintContents(Surface *surfaceWindow, bool draw) {
 		chunkHighlight.start -= lineStart;
 		chunkHighlight.end -= lineStart;
 
-		rcClient.top = static_cast<XYPOSITION>(ytext - ascent - 1);
+		const int top = ytext - ascent - 1;
+		rcClient.top = top;
 
 		int x = insetX;     // start each line at this inset
 

scintilla/src/CellBuffer.cxx

@@ -11,6 +11,7 @@
 #include <cstring>
 #include <cstdio>
 #include <cstdarg>
+#include <climits>
 
 #include <stdexcept>
 #include <string>
@@ -405,7 +406,7 @@ const char *UndoHistory::AppendAction(ActionType at, Sci::Position position, con
 	} else if (detach && (*detach > currentAction)) {
 		detach = currentAction;
 	}
-	int oldCurrentAction = currentAction;
+	const int oldCurrentAction = currentAction;
 	if (currentAction >= 1) {
 		if (0 == undoSequenceDepth) {
 			// Top level actions may not always be coalesced
@@ -646,7 +647,7 @@ void CellBuffer::GetCharRange(char *buffer, Sci::Position position, Sci::Positio
 }
 
 char CellBuffer::StyleAt(Sci::Position position) const noexcept {
-	return hasStyles ? style.ValueAt(position) : 0;
+	return hasStyles ? style.ValueAt(position) : '\0';
 }
 
 void CellBuffer::GetStyleRange(unsigned char *buffer, Sci::Position position, Sci::Position lengthRetrieve) const {
@@ -773,6 +774,9 @@ Sci::Position CellBuffer::Length() const noexcept {
 }
 
 void CellBuffer::Allocate(Sci::Position newSize) {
+	if (!largeDocument && (newSize > INT32_MAX)) {
+		throw std::runtime_error("CellBuffer::Allocate: size of standard document limited to 2G.");
+	}
 	substance.ReAllocate(newSize);
 	if (hasStyles) {
 		style.ReAllocate(newSize);

scintilla/src/ChangeHistory.cxx

@@ -46,7 +46,7 @@ constexpr bool InsertionSpanSameDeletion(const ChangeSpan &is, Sci::Position pos
 		is.start == positionDeletion &&
 		is.length == 0 &&
 		is.edition == edition;
-};
+}
 
 }
 

scintilla/src/Document.cxx

@@ -184,7 +184,7 @@ Document::~Document() {
 }
 
 // Increase reference count and return its previous value.
-int Document::AddRef() {
+int SCI_METHOD Document::AddRef() noexcept {
 	return refCount++;
 }
 
@@ -461,6 +461,10 @@ Sci_Position SCI_METHOD Document::LineEnd(Sci_Position line) const {
 	return cb.LineEnd(line);
 }
 
+int SCI_METHOD Document::DEVersion() const noexcept {
+	return deRelease0;
+}
+
 void SCI_METHOD Document::SetErrorStatus(int status) {
 	// Tell the watchers an error has occurred.
 	for (const WatcherWithUserData &watcher : watchers) {
@@ -1341,8 +1345,12 @@ int SCI_METHOD Document::AddData(const char *data, Sci_Position length) {
 	return static_cast<int>(Status::Ok);
 }
 
+IDocumentEditable *Document::AsDocumentEditable() noexcept {
+	return static_cast<IDocumentEditable *>(this);
+}
+
 void * SCI_METHOD Document::ConvertToDocument() {
-	return this;
+	return AsDocumentEditable();
 }
 
 Sci::Position Document::Undo() {
@@ -2404,8 +2412,7 @@ void Document::EnsureStyledTo(Sci::Position pos) {
 	if ((enteredStyling == 0) && (pos > GetEndStyled())) {
 		IncrementStyleClock();
 		if (pli && !pli->UseContainerLexing()) {
-			const Sci::Line lineEndStyled = SciLineFromPosition(GetEndStyled());
+			const Sci::Position endStyledTo = LineStartPosition(GetEndStyled());
-			const Sci::Position endStyledTo = LineStart(lineEndStyled);
 			pli->Colourise(endStyledTo, pos);
 		} else {
 			// Ask the watchers to style, and stop as soon as one responds.
@@ -2844,8 +2851,8 @@ public:
 		lineRangeEnd = doc->SciLineFromPosition(endPos);
 		lineRangeBreak = lineRangeEnd + increment;
 	}
-	Range LineRange(Sci::Line line) const {
+	Range LineRange(Sci::Line line, Sci::Position lineStartPos, Sci::Position lineEndPos) const noexcept {
-		Range range(doc->LineStart(line), doc->LineEnd(line));
+		Range range(lineStartPos, lineEndPos);
 		if (increment == 1) {
 			if (line == lineRangeStart)
 				range.start = startPos;
@@ -2876,6 +2883,9 @@ public:
 		else
 			return pdoc->CharAt(index);
 	}
+	Sci::Position MovePositionOutsideChar(Sci::Position pos, Sci::Position moveDir) const noexcept override {
+		return pdoc->MovePositionOutsideChar(pos, moveDir, false);
+	}
 };
 
 #ifndef NO_CXX11_REGEX
@@ -3083,12 +3093,24 @@ public:
 
 #endif
 
-std::regex_constants::match_flag_type MatchFlags(const Document *doc, Sci::Position startPos, Sci::Position endPos) noexcept {
+std::regex_constants::match_flag_type MatchFlags(const Document *doc, Sci::Position startPos, Sci::Position endPos, Sci::Position lineStartPos, Sci::Position lineEndPos) {
 	std::regex_constants::match_flag_type flagsMatch = std::regex_constants::match_default;
-	if (!doc->IsLineStartPosition(startPos))
+	if (startPos != lineStartPos) {
+#ifdef _LIBCPP_VERSION
 		flagsMatch |= std::regex_constants::match_not_bol;
-	if (!doc->IsLineEndPosition(endPos))
+		if (!doc->IsWordStartAt(startPos)) {
+			flagsMatch |= std::regex_constants::match_not_bow;
+		}
+#else
+		flagsMatch |= std::regex_constants::match_prev_avail;
+#endif
+	}
+	if (endPos != lineEndPos) {
 		flagsMatch |= std::regex_constants::match_not_eol;
+		if (!doc->IsWordEndAt(endPos)) {
+			flagsMatch |= std::regex_constants::match_not_eow;
+		}
+	}
 	return flagsMatch;
 }
 
@@ -3102,39 +3124,33 @@ bool MatchOnLines(const Document *doc, const Regex &regexp, const RESearchRange
 	// has not been implemented by compiler runtimes with MSVC always in multiline
 	// mode and libc++ and libstdc++ always in single-line mode.
 	// If multiline regex worked well then the line by line iteration could be removed
-	// for the forwards case and replaced with the following 4 lines:
+	// for the forwards case and replaced with the following:
 #ifdef REGEX_MULTILINE
+	const Sci::Position lineStartPos = doc->LineStart(resr.lineRangeStart);
+	const Sci::Position lineEndPos = doc->LineEnd(resr.lineRangeEnd);
 	Iterator itStart(doc, resr.startPos);
 	Iterator itEnd(doc, resr.endPos);
-	const std::regex_constants::match_flag_type flagsMatch = MatchFlags(doc, resr.startPos, resr.endPos);
+	const std::regex_constants::match_flag_type flagsMatch = MatchFlags(doc, resr.startPos, resr.endPos, lineStartPos, lineEndPos);
 	const bool matched = std::regex_search(itStart, itEnd, match, regexp, flagsMatch);
 #else
 	// Line by line.
 	bool matched = false;
 	for (Sci::Line line = resr.lineRangeStart; line != resr.lineRangeBreak; line += resr.increment) {
-		const Range lineRange = resr.LineRange(line);
+		const Sci::Position lineStartPos = doc->LineStart(line);
+		const Sci::Position lineEndPos = doc->LineEnd(line);
+		const Range lineRange = resr.LineRange(line, lineStartPos, lineEndPos);
 		Iterator itStart(doc, lineRange.start);
 		Iterator itEnd(doc, lineRange.end);
-		std::regex_constants::match_flag_type flagsMatch = MatchFlags(doc, lineRange.start, lineRange.end);
+		const std::regex_constants::match_flag_type flagsMatch = MatchFlags(doc, lineRange.start, lineRange.end, lineStartPos, lineEndPos);
-		matched = std::regex_search(itStart, itEnd, match, regexp, flagsMatch);
+		std::regex_iterator<Iterator> it(itStart, itEnd, regexp, flagsMatch);
-		// Check for the last match on this line.
+		for (const std::regex_iterator<Iterator> last; it != last; ++it) {
-		if (matched) {
+			match = *it;
-			if (resr.increment == -1) {
-				while (matched) {
-					Iterator itNext(doc, match[0].second.PosRoundUp());
-					flagsMatch = MatchFlags(doc, itNext.Pos(), lineRange.end);
-					std::match_results<Iterator> matchNext;
-					matched = std::regex_search(itNext, itEnd, matchNext, regexp, flagsMatch);
-					if (matched) {
-						if (match[0].first == match[0].second) {
-							// Empty match means failure so exit
-							return false;
-						}
-						match = matchNext;
-					}
-				}
 			matched = true;
+			if (resr.increment > 0) {
+				break;
 			}
+		}
+		if (matched) {
 			break;
 		}
 	}
@@ -3172,7 +3188,6 @@ Sci::Position Cxx11RegexFindText(const Document *doc, Sci::Position minPos, Sci:
 			std::wregex regexp;
 			regexp.assign(ws, flagsRe);
 			matched = MatchOnLines<UTF8Iterator>(doc, regexp, resr, search);
-
 		} else {
 			std::regex regexp;
 			regexp.assign(s, flagsRe);
@@ -3233,8 +3248,10 @@ Sci::Position BuiltinRegex::FindText(Document *doc, Sci::Position minPos, Sci::P
 	const char searchEndPrev = (*length > 1) ? s[*length - 2] : '\0';
 	const bool searchforLineEnd = (searchEnd == '$') && (searchEndPrev != '\\');
 	for (Sci::Line line = resr.lineRangeStart; line != resr.lineRangeBreak; line += resr.increment) {
-		Sci::Position startOfLine = doc->LineStart(line);
+		const Sci::Position lineStartPos = doc->LineStart(line);
-		Sci::Position endOfLine = doc->LineEnd(line);
+		const Sci::Position lineEndPos = doc->LineEnd(line);
+		Sci::Position startOfLine = lineStartPos;
+		Sci::Position endOfLine = lineEndPos;
 		if (resr.increment == 1) {
 			if (line == resr.lineRangeStart) {
 				if ((resr.startPos != startOfLine) && searchforLineStart)
@@ -3260,36 +3277,32 @@ Sci::Position BuiltinRegex::FindText(Document *doc, Sci::Position minPos, Sci::P
 		}
 
 		const DocumentIndexer di(doc, endOfLine);
+		search.SetLineRange(lineStartPos, lineEndPos);
 		int success = search.Execute(di, startOfLine, endOfLine);
 		if (success) {
-			pos = search.bopat[0];
+			Sci::Position endPos = search.eopat[0];
-			// Ensure only whole characters selected
-			search.eopat[0] = doc->MovePositionOutsideChar(search.eopat[0], 1, false);
-			lenRet = search.eopat[0] - search.bopat[0];
 			// There can be only one start of a line, so no need to look for last match in line
 			if ((resr.increment == -1) && !searchforLineStart) {
 				// Check for the last match on this line.
-				int repetitions = 1000;	// Break out of infinite loop
+				while (success && (endPos < endOfLine)) {
-				RESearch::MatchPositions bopat{};
+					const RESearch::MatchPositions bopat = search.bopat;
-				RESearch::MatchPositions eopat{};
+					const RESearch::MatchPositions eopat = search.eopat;
-				while (success && (search.eopat[0] <= endOfLine) && (repetitions--)) {
+					pos = endPos;
-					bopat = search.bopat;
+					if (pos == bopat[0]) {
-					eopat = search.eopat;
+						// empty match
-					success = search.Execute(di, pos+1, endOfLine);
+						pos = doc->NextPosition(pos, 1);
+					}
+					success = search.Execute(di, pos, endOfLine);
 					if (success) {
-						if (search.eopat[0] <= minPos) {
+						endPos = search.eopat[0];
-							pos = search.bopat[0];
-							lenRet = search.eopat[0] - search.bopat[0];
 					} else {
-							success = 0;
-						}
-					}
-				}
-				if (!success) {
 						search.bopat = bopat;
 						search.eopat = eopat;
 					}
 				}
+			}
+			pos = search.bopat[0];
+			lenRet = endPos - pos;
 			break;
 		}
 	}

scintilla/src/Document.h

@@ -259,7 +259,7 @@ struct CharacterExtracted {
 
 /**
  */
-class Document : PerLine, public Scintilla::IDocument, public Scintilla::ILoader {
+class Document : PerLine, public Scintilla::IDocument, public Scintilla::ILoader, public Scintilla::IDocumentEditable {
 
 public:
 	/** Used to pair watcher pointer with user data. */
@@ -329,7 +329,7 @@ public:
 	Document &operator=(Document &&) = delete;
 	~Document() override;
 
-	int AddRef();
+	int SCI_METHOD AddRef() noexcept override;
 	int SCI_METHOD Release() override;
 
 	// From PerLine
@@ -347,6 +347,7 @@ public:
 	int SCI_METHOD Version() const override {
 		return Scintilla::dvRelease4;
 	}
+	int SCI_METHOD DEVersion() const noexcept override;
 
 	void SCI_METHOD SetErrorStatus(int status) override;
 
@@ -383,6 +384,7 @@ public:
 	Sci::Position InsertString(Sci::Position position, std::string_view sv);
 	void ChangeInsertion(const char *s, Sci::Position length);
 	int SCI_METHOD AddData(const char *data, Sci_Position length) override;
+	IDocumentEditable *AsDocumentEditable() noexcept;
 	void * SCI_METHOD ConvertToDocument() override;
 	Sci::Position Undo();
 	Sci::Position Redo();

scintilla/src/EditView.cxx

@@ -2060,17 +2060,11 @@ void DrawFoldLines(Surface *surface, const EditModel &model, const ViewStyle &vs
 		const ColourRGBA foldLineColour = vsDraw.ElementColour(Element::FoldLine).value_or(
 			vsDraw.styles[StyleDefault].fore);
 		// Paint the line above the fold
-		if ((subLine == 0) &&
+		if ((subLine == 0) && FlagSet(model.foldFlags, (expanded ? FoldFlag::LineBeforeExpanded: FoldFlag::LineBeforeContracted))) {
-			((expanded && (FlagSet(model.foldFlags, FoldFlag::LineBeforeExpanded)))
-				||
-				(!expanded && (FlagSet(model.foldFlags, FoldFlag::LineBeforeContracted))))) {
 			surface->FillRectangleAligned(Side(rcLine, Edge::top, 1.0), foldLineColour);
 		}
 		// Paint the line below the fold
-		if (lastSubLine &&
+		if (lastSubLine && FlagSet(model.foldFlags, (expanded ? FoldFlag::LineAfterExpanded : FoldFlag::LineAfterContracted))) {
-			((expanded && (FlagSet(model.foldFlags, FoldFlag::LineAfterExpanded)))
-				||
-				(!expanded && (FlagSet(model.foldFlags, FoldFlag::LineAfterContracted))))) {
 			surface->FillRectangleAligned(Side(rcLine, Edge::bottom, 1.0), foldLineColour);
 			// If contracted fold line drawn then don't overwrite with hidden line
 			// as fold lines are more specific then hidden lines.

scintilla/src/Editor.cxx

@@ -98,11 +98,12 @@ constexpr bool CanEliminate(const DocModification &mh) noexcept {
 	in a [possibly lengthy] multi-step Undo/Redo sequence
 */
 constexpr bool IsLastStep(const DocModification &mh) noexcept {
+	constexpr ModificationFlags finalMask = ModificationFlags::MultiStepUndoRedo
+		| ModificationFlags::LastStepInUndoRedo
+		| ModificationFlags::MultilineUndoRedo;
 	return
 		FlagSet(mh.modificationType, (ModificationFlags::Undo | ModificationFlags::Redo))
-	    && (FlagSet(mh.modificationType, ModificationFlags::MultiStepUndoRedo))
+	    && ((mh.modificationType & finalMask) == finalMask);
-	    && (FlagSet(mh.modificationType, ModificationFlags::LastStepInUndoRedo))
-	    && (FlagSet(mh.modificationType, ModificationFlags::MultilineUndoRedo));
 }
 
 }
@@ -2727,7 +2728,7 @@ void Editor::NotifyModified(Document *, DocModification mh, void *) {
 			if (FlagSet(mh.modificationType, ModificationFlags::BeforeInsert)) {
 				if (pdoc->ContainsLineEnd(mh.text, mh.length) && (mh.position != pdoc->LineStart(lineOfPos)))
 					endNeedShown = pdoc->LineStart(lineOfPos+1);
-			} else if (FlagSet(mh.modificationType, ModificationFlags::BeforeDelete)) {
+			} else {
 				// If the deletion includes any EOL then we extend the need shown area.
 				endNeedShown = mh.position + mh.length;
 				Sci::Line lineLast = pdoc->SciLineFromPosition(mh.position+mh.length);
@@ -2802,7 +2803,7 @@ void Editor::NotifyModified(Document *, DocModification mh, void *) {
 		SetScrollBars();
 	}
 
-	if ((FlagSet(mh.modificationType, ModificationFlags::ChangeMarker)) || (FlagSet(mh.modificationType, ModificationFlags::ChangeMargin))) {
+	if (FlagSet(mh.modificationType, (ModificationFlags::ChangeMarker | ModificationFlags::ChangeMargin))) {
 		if ((!willRedrawAll) && ((paintState == PaintState::notPainting) || !PaintContainsMargin())) {
 			if (FlagSet(mh.modificationType, ModificationFlags::ChangeFold)) {
 				// Fold changes can affect the drawing of following lines so redraw whole margin
@@ -3750,7 +3751,7 @@ int Editor::DelWordOrLine(Message iMessage) {
 		case Message::DelLineRight:
 			rangeDelete = Range(
 				sel.Range(r).caret.Position(),
-				pdoc->LineEnd(pdoc->LineFromPosition(sel.Range(r).caret.Position())));
+				pdoc->LineEndPosition(sel.Range(r).caret.Position()));
 			break;
 		default:
 			break;
@@ -4403,8 +4404,6 @@ bool Editor::DragThreshold(Point ptStart, Point ptNow) {
 
 void Editor::StartDrag() {
 	// Always handled by subclasses
-	//SetMouseCapture(true);
-	//DisplayCursor(Windows::Cursor::Arrow);
 }
 
 void Editor::DropAt(SelectionPosition position, const char *value, size_t lengthValue, bool moving, bool rectangular) {
@@ -4511,6 +4510,28 @@ bool Editor::PointInSelection(Point pt) {
 	return false;
 }
 
+ptrdiff_t Editor::SelectionFromPoint(Point pt) {
+	// Prioritize checking inside non-empty selections since each character will be inside only 1 
+	const SelectionPosition posChar = SPositionFromLocation(pt, true, true);
+	for (size_t r = 0; r < sel.Count(); r++) {
+		if (sel.Range(r).ContainsCharacter(posChar)) {
+			return r;
+		}
+	}
+
+	// Then check if near empty selections as may be near more than 1
+	const SelectionPosition pos = SPositionFromLocation(pt, true, false);
+	for (size_t r = 0; r < sel.Count(); r++) {
+		const SelectionRange &range = sel.Range(r);
+		if ((range.Empty()) && (pos == range.caret)) {
+			return r;
+		}
+	}
+
+	// No selection at point
+	return -1;
+}
+
 bool Editor::PointInSelMargin(Point pt) const {
 	// Really means: "Point in a margin"
 	if (vs.fixedColumnWidth > 0) {	// There is a margin
@@ -4535,6 +4556,12 @@ Window::Cursor Editor::GetMarginCursor(Point pt) const noexcept {
 	return Window::Cursor::reverseArrow;
 }
 
+void Editor::DropSelection(size_t part) {
+	sel.DropSelection(part);
+	ContainerNeedsUpdate(Update::Selection);
+	Redraw();
+}
+
 void Editor::TrimAndSetSelection(Sci::Position currentPos_, Sci::Position anchor_) {
 	sel.TrimSelection(SelectionRange(currentPos_, anchor_));
 	SetSelection(currentPos_, anchor_);
@@ -4620,8 +4647,7 @@ void Editor::MouseLeave() {
 }
 
 static constexpr bool AllowVirtualSpace(VirtualSpace virtualSpaceOptions, bool rectangular) noexcept {
-	return (!rectangular && (FlagSet(virtualSpaceOptions, VirtualSpace::UserAccessible)))
+	return FlagSet(virtualSpaceOptions, (rectangular ? VirtualSpace::RectangularSelection : VirtualSpace::UserAccessible));
-		|| (rectangular && (FlagSet(virtualSpaceOptions, VirtualSpace::RectangularSelection)));
 }
 
 void Editor::ButtonDownWithModifiers(Point pt, unsigned int curTime, KeyMod modifiers) {
@@ -4631,10 +4657,10 @@ void Editor::ButtonDownWithModifiers(Point pt, unsigned int curTime, KeyMod modi
 	const bool ctrl = FlagSet(modifiers, KeyMod::Ctrl);
 	const bool shift = FlagSet(modifiers, KeyMod::Shift);
 	const bool alt = FlagSet(modifiers, KeyMod::Alt);
-	SelectionPosition newPos = SPositionFromLocation(pt, false, false, AllowVirtualSpace(virtualSpaceOptions, alt));
+	const SelectionPosition clickPos = SPositionFromLocation(pt, false, false, AllowVirtualSpace(virtualSpaceOptions, alt));
-	newPos = MovePositionOutsideChar(newPos, sel.MainCaret() - newPos.Position());
+	const SelectionPosition newPos = MovePositionOutsideChar(clickPos, sel.MainCaret() - clickPos.Position());
-	SelectionPosition newCharPos = SPositionFromLocation(pt, false, true, false);
+	const SelectionPosition newCharPos = MovePositionOutsideChar(
-	newCharPos = MovePositionOutsideChar(newCharPos, -1);
+		SPositionFromLocation(pt, false, true, false), -1);
 	inDragDrop = DragDrop::none;
 	sel.SetMoveExtends(false);
 
@@ -4643,21 +4669,22 @@ void Editor::ButtonDownWithModifiers(Point pt, unsigned int curTime, KeyMod modi
 
 	NotifyIndicatorClick(true, newPos.Position(), modifiers);
 
+	const bool multiClick = (curTime < (lastClickTime + Platform::DoubleClickTime())) && Close(pt, lastClick, doubleClickCloseThreshold);
+	lastClickTime = curTime;
+	lastClick = pt;
+
 	const bool inSelMargin = PointInSelMargin(pt);
 	// In margin ctrl+(double)click should always select everything
 	if (ctrl && inSelMargin) {
 		SelectAll();
-		lastClickTime = curTime;
-		lastClick = pt;
 		return;
 	}
 	if (shift && !inSelMargin) {
 		SetSelection(newPos);
 	}
-	if ((curTime < (lastClickTime+Platform::DoubleClickTime())) && Close(pt, lastClick, doubleClickCloseThreshold)) {
+	if (multiClick) {
 		//Platform::DebugPrintf("Double click %d %d = %d\n", curTime, lastClickTime, curTime - lastClickTime);
-		SetMouseCapture(true);
+		ChangeMouseCapture(true);
-		FineTickerStart(TickReason::scroll, 100, 10);
 		if (!ctrl || !multipleSelection || (selectionUnit != TextUnit::character && selectionUnit != TextUnit::word))
 			SetEmptySelection(newPos.Position());
 		bool doubleClick = false;
@@ -4755,21 +4782,32 @@ void Editor::ButtonDownWithModifiers(Point pt, unsigned int curTime, KeyMod modi
 			}
 
 			SetDragPosition(SelectionPosition(Sci::invalidPosition));
-			SetMouseCapture(true);
+			ChangeMouseCapture(true);
-			FineTickerStart(TickReason::scroll, 100, 10);
 		} else {
 			if (PointIsHotspot(pt)) {
 				NotifyHotSpotClicked(newCharPos.Position(), modifiers);
 				hotSpotClickPos = newCharPos.Position();
 			}
 			if (!shift) {
-				if (PointInSelection(pt) && !SelectionEmpty())
+				const ptrdiff_t selectionPart = SelectionFromPoint(pt);
-					inDragDrop = DragDrop::initial;
+				if (selectionPart >= 0) {
-				else
+					if (multipleSelection && ctrl) {
-					inDragDrop = DragDrop::none;
+						// Deselect
+						if (sel.Count() > 1) {
+							DropSelection(selectionPart);
+							// Completed: don't want any more processing of this click
+							return;
+						} else {
+							// Switch to just the click position
+							SetSelection(newPos, newPos);
 						}
-			SetMouseCapture(true);
+					}
-			FineTickerStart(TickReason::scroll, 100, 10);
+					if (!sel.Range(selectionPart).Empty()) {
+						inDragDrop = DragDrop::initial;
+					}
+				}
+			}
+			ChangeMouseCapture(true);
 			if (inDragDrop != DragDrop::initial) {
 				SetDragPosition(SelectionPosition(Sci::invalidPosition));
 				if (!shift) {
@@ -4799,8 +4837,6 @@ void Editor::ButtonDownWithModifiers(Point pt, unsigned int curTime, KeyMod modi
 			}
 		}
 	}
-	lastClickTime = curTime;
-	lastClick = pt;
 	lastXChosen = static_cast<int>(pt.x) + xOffset;
 	ShowCaretAtCurrentPosition();
 }
@@ -4886,8 +4922,7 @@ void Editor::ButtonMoveWithModifiers(Point pt, unsigned int, KeyMod modifiers) {
 
 	if (inDragDrop == DragDrop::initial) {
 		if (DragThreshold(ptMouseLast, pt)) {
-			SetMouseCapture(false);
+			ChangeMouseCapture(false);
-			FineTickerCancel(TickReason::scroll);
 			SetDragPosition(movePos);
 			CopySelectionRange(&drag);
 			StartDrag();
@@ -5029,8 +5064,7 @@ void Editor::ButtonUpWithModifiers(Point pt, unsigned int curTime, KeyMod modifi
 			SetHotSpotRange(nullptr);
 		}
 		ptMouseLast = pt;
-		SetMouseCapture(false);
+		ChangeMouseCapture(false);
-		FineTickerCancel(TickReason::scroll);
 		NotifyIndicatorClick(false, newPos.Position(), modifiers);
 		if (inDragDrop == DragDrop::dragging) {
 			const SelectionPosition selStart = SelectionStart();
@@ -5163,6 +5197,16 @@ void Editor::FineTickerCancel(TickReason) {
 	assert(false);
 }
 
+void Editor::ChangeMouseCapture(bool on) {
+	SetMouseCapture(on);
+	// While mouse captured want timer to scroll automatically
+	if (on) {
+		FineTickerStart(TickReason::scroll, 100, 10);
+	} else {
+		FineTickerCancel(TickReason::scroll);
+	}
+}
+
 void Editor::SetFocusState(bool focusState) {
 	const bool changing = hasFocus != focusState;
 	hasFocus = focusState;
@@ -5732,13 +5776,17 @@ Sci::Position Editor::GetTag(char *tagValue, int tagNumber) {
 
 Sci::Position Editor::ReplaceTarget(ReplaceType replaceType, std::string_view text) {
 	UndoGroup ug(pdoc);
+
+	std::string substituted;	// Copy in case of re-entrance
+
 	if (replaceType == ReplaceType::patterns) {
 		Sci::Position length = text.length();
 		const char *p = pdoc->SubstituteByPosition(text.data(), &length);
 		if (!p) {
 			return 0;
 		}
-		text = std::string_view(p, length);
+		substituted.assign(p, length);
+		text = substituted;
 	}
 
 	if (replaceType == ReplaceType::minimal) {
@@ -5753,19 +5801,25 @@ Sci::Position Editor::ReplaceTarget(ReplaceType replaceType, std::string_view te
 		targetRange = SelectionSegment(start, SelectionPosition(range.end));
 	}
 
+	// Make a copy of targetRange in case callbacks use target
+	SelectionSegment replaceRange = targetRange;
+
 	// Remove the text inside the range
-	if (targetRange.Length() > 0)
+	if (replaceRange.Length() > 0)
-		pdoc->DeleteChars(targetRange.start.Position(), targetRange.Length());
+		pdoc->DeleteChars(replaceRange.start.Position(), replaceRange.Length());
-	targetRange.end = targetRange.start;
 
 	// Realize virtual space of target start
-	const Sci::Position startAfterSpaceInsertion = RealizeVirtualSpace(targetRange.start.Position(), targetRange.start.VirtualSpace());
+	const Sci::Position startAfterSpaceInsertion = RealizeVirtualSpace(replaceRange.start.Position(), replaceRange.start.VirtualSpace());
-	targetRange.start.SetPosition(startAfterSpaceInsertion);
+	replaceRange.start.SetPosition(startAfterSpaceInsertion);
-	targetRange.end = targetRange.start;
+	replaceRange.end = replaceRange.start;
 
 	// Insert the new text
-	const Sci::Position lengthInserted = pdoc->InsertString(targetRange.start.Position(), text);
+	const Sci::Position lengthInserted = pdoc->InsertString(replaceRange.start.Position(), text);
-	targetRange.end.SetPosition(targetRange.start.Position() + lengthInserted);
+	replaceRange.end.SetPosition(replaceRange.start.Position() + lengthInserted);
+
+	// Copy back to targetRange in case application is chaining modifications
+	targetRange = replaceRange;
+
 	return text.length();
 }
 
@@ -6009,6 +6063,47 @@ void Editor::SetSelectionNMessage(Message iMessage, uptr_t wParam, sptr_t lParam
 	ContainerNeedsUpdate(Update::Selection);
 }
 
+namespace {
+
+constexpr Selection::SelTypes SelTypeFromMode(SelectionMode mode) {
+	switch (mode) {
+	case SelectionMode::Rectangle:
+		return Selection::SelTypes::rectangle;
+	case SelectionMode::Lines:
+		return Selection::SelTypes::lines;
+	case SelectionMode::Thin:
+		return Selection::SelTypes::thin;
+	case SelectionMode::Stream:
+	default:
+		return Selection::SelTypes::stream;
+	}
+}
+
+sptr_t SPtrFromPtr(void *ptr) noexcept {
+	return reinterpret_cast<sptr_t>(ptr);
+}
+
+}
+
+void Editor::SetSelectionMode(uptr_t wParam, bool setMoveExtends) {
+	const Selection::SelTypes newSelType = SelTypeFromMode(static_cast<SelectionMode>(wParam));
+	if (setMoveExtends) {
+		sel.SetMoveExtends(!sel.MoveExtends() || (sel.selType != newSelType));
+	}
+	sel.selType = newSelType;
+	switch (sel.selType) {
+	case Selection::SelTypes::rectangle:
+		sel.Rectangular() = sel.RangeMain(); // adjust current selection
+		break;
+	case Selection::SelTypes::lines:
+		SetSelection(sel.RangeMain().caret, sel.RangeMain().anchor); // adjust current selection
+		break;
+	default:
+		break;
+	}
+	InvalidateWholeSelection();
+}
+
 sptr_t Editor::StringResult(sptr_t lParam, const char *val) noexcept {
 	const size_t len = val ? strlen(val) : 0;
 	if (lParam) {
@@ -8125,11 +8220,11 @@ sptr_t Editor::WndProc(Message iMessage, uptr_t wParam, sptr_t lParam) {
 		break;
 
 	case Message::GetDocPointer:
-		return reinterpret_cast<sptr_t>(pdoc);
+		return SPtrFromPtr(pdoc->AsDocumentEditable());
 
 	case Message::SetDocPointer:
 		CancelModes();
-		SetDocPointer(static_cast<Document *>(PtrFromSPtr(lParam)));
+		SetDocPointer(static_cast<Document *>(static_cast<IDocumentEditable *>(PtrFromSPtr(lParam))));
 		return 0;
 
 	case Message::CreateDocument: {
@@ -8137,15 +8232,15 @@ sptr_t Editor::WndProc(Message iMessage, uptr_t wParam, sptr_t lParam) {
 			doc->AddRef();
 			doc->Allocate(PositionFromUPtr(wParam));
 			pcs = ContractionStateCreate(pdoc->IsLarge());
-			return reinterpret_cast<sptr_t>(doc);
+			return SPtrFromPtr(doc->AsDocumentEditable());
 		}
 
 	case Message::AddRefDocument:
-		(static_cast<Document *>(PtrFromSPtr(lParam)))->AddRef();
+		(static_cast<IDocumentEditable *>(PtrFromSPtr(lParam)))->AddRef();
 		break;
 
 	case Message::ReleaseDocument:
-		(static_cast<Document *>(PtrFromSPtr(lParam)))->Release();
+		(static_cast<IDocumentEditable *>(PtrFromSPtr(lParam)))->Release();
 		break;
 
 	case Message::GetDocumentOptions:
@@ -8186,33 +8281,12 @@ sptr_t Editor::WndProc(Message iMessage, uptr_t wParam, sptr_t lParam) {
 	case Message::SelectionIsRectangle:
 		return sel.selType == Selection::SelTypes::rectangle ? 1 : 0;
 
-	case Message::SetSelectionMode: {
+	case Message::SetSelectionMode:
-			switch (static_cast<SelectionMode>(wParam)) {
+		SetSelectionMode(wParam, true);
-			case SelectionMode::Stream:
-				sel.SetMoveExtends(!sel.MoveExtends() || (sel.selType != Selection::SelTypes::stream));
-				sel.selType = Selection::SelTypes::stream;
 		break;
-			case SelectionMode::Rectangle:
+	case Message::ChangeSelectionMode:
-				sel.SetMoveExtends(!sel.MoveExtends() || (sel.selType != Selection::SelTypes::rectangle));
+		SetSelectionMode(wParam, false);
-				sel.selType = Selection::SelTypes::rectangle;
-				sel.Rectangular() = sel.RangeMain(); // adjust current selection
 		break;
-			case SelectionMode::Lines:
-				sel.SetMoveExtends(!sel.MoveExtends() || (sel.selType != Selection::SelTypes::lines));
-				sel.selType = Selection::SelTypes::lines;
-				SetSelection(sel.RangeMain().caret, sel.RangeMain().anchor); // adjust current selection
-				break;
-			case SelectionMode::Thin:
-				sel.SetMoveExtends(!sel.MoveExtends() || (sel.selType != Selection::SelTypes::thin));
-				sel.selType = Selection::SelTypes::thin;
-				break;
-			default:
-				sel.SetMoveExtends(!sel.MoveExtends() || (sel.selType != Selection::SelTypes::stream));
-				sel.selType = Selection::SelTypes::stream;
-			}
-			InvalidateWholeSelection();
-			break;
-		}
 	case Message::GetSelectionMode:
 		switch (sel.selType) {
 		case Selection::SelTypes::stream:
@@ -8226,6 +8300,9 @@ sptr_t Editor::WndProc(Message iMessage, uptr_t wParam, sptr_t lParam) {
 		default:	// ?!
 			return static_cast<sptr_t>(SelectionMode::Stream);
 		}
+	case Message::SetMoveExtendsSelection:
+		sel.SetMoveExtends(wParam != 0);
+		break;
 	case Message::GetMoveExtendsSelection:
 		return sel.MoveExtends();
 	case Message::GetLineSelStartPosition:
@@ -8657,10 +8734,11 @@ sptr_t Editor::WndProc(Message iMessage, uptr_t wParam, sptr_t lParam) {
 		Redraw();
 		break;
 
+	case Message::SelectionFromPoint:
+		return SelectionFromPoint(PointFromParameters(wParam, lParam));
+
 	case Message::DropSelectionN:
-		sel.DropSelection(wParam);
+		DropSelection(wParam);
-		ContainerNeedsUpdate(Update::Selection);
-		Redraw();
 		break;
 
 	case Message::SetMainSelection:

scintilla/src/Editor.h

@@ -527,8 +527,10 @@ protected:	// ScintillaBase subclass needs access to much of Editor
 	/** PositionInSelection returns true if position in selection. */
 	bool PositionInSelection(Sci::Position pos);
 	bool PointInSelection(Point pt);
+	ptrdiff_t SelectionFromPoint(Point pt);
 	bool PointInSelMargin(Point pt) const;
 	Window::Cursor GetMarginCursor(Point pt) const noexcept;
+	void DropSelection(size_t part);
 	void TrimAndSetSelection(Sci::Position currentPos_, Sci::Position anchor_);
 	void LineSelection(Sci::Position lineCurrentPos_, Sci::Position lineAnchorPos_, bool wholeLine);
 	void WordSelection(Sci::Position pos);
@@ -546,6 +548,7 @@ protected:	// ScintillaBase subclass needs access to much of Editor
 	virtual void FineTickerStart(TickReason reason, int millis, int tolerance);
 	virtual void FineTickerCancel(TickReason reason);
 	virtual bool SetIdle(bool) { return false; }
+	void ChangeMouseCapture(bool on);
 	virtual void SetMouseCapture(bool on) = 0;
 	virtual bool HaveMouseCapture() = 0;
 	void SetFocusState(bool focusState);
@@ -612,6 +615,7 @@ protected:	// ScintillaBase subclass needs access to much of Editor
 	void StyleSetMessage(Scintilla::Message iMessage, Scintilla::uptr_t wParam, Scintilla::sptr_t lParam);
 	Scintilla::sptr_t StyleGetMessage(Scintilla::Message iMessage, Scintilla::uptr_t wParam, Scintilla::sptr_t lParam);
 	void SetSelectionNMessage(Scintilla::Message iMessage, Scintilla::uptr_t wParam, Scintilla::sptr_t lParam);
+	void SetSelectionMode(uptr_t wParam, bool setMoveExtends);
 
 	// Coercion functions for transforming WndProc parameters into pointers
 	static void *PtrFromSPtr(Scintilla::sptr_t lParam) noexcept {

scintilla/src/Geometry.h

@@ -173,7 +173,7 @@ PRectangle PixelAlignOutside(const PRectangle &rc, int pixelDivisions) noexcept;
 constexpr float componentMaximum = 255.0F;
 constexpr unsigned int maximumByte = 0xffU;
 class ColourRGBA {
-	static constexpr float ComponentAsFloat(unsigned int component) {
+	static constexpr float ComponentAsFloat(unsigned char component) {
 		return component / componentMaximum;
 	}
 	static constexpr int rgbMask = 0xffffff;
@@ -195,7 +195,8 @@ public:
 	}
 
 	static constexpr ColourRGBA FromIpRGB(intptr_t co_) noexcept {
-		return ColourRGBA((co_ & rgbMask) | (maximumByte << 24));
+		const int rgb = co_ & rgbMask;
+		return ColourRGBA(rgb | (maximumByte << 24));
 	}
 
 	constexpr ColourRGBA WithoutAlpha() const noexcept {

scintilla/src/Indicator.h

@@ -13,7 +13,7 @@ namespace Scintilla::Internal {
 struct StyleAndColour {
 	Scintilla::IndicatorStyle style;
 	ColourRGBA fore;
-	StyleAndColour() noexcept : style(Scintilla::IndicatorStyle::Plain), fore(0, 0, 0) {
+	StyleAndColour() noexcept : style(Scintilla::IndicatorStyle::Plain), fore(black) {
 	}
 	StyleAndColour(Scintilla::IndicatorStyle style_, ColourRGBA fore_ = black) noexcept : style(style_), fore(fore_) {
 	}

scintilla/src/PositionCache.cxx

@@ -471,7 +471,7 @@ bool SignificantLines::LineMayCache(Sci::Line line) const noexcept {
 
 LineLayoutCache::LineLayoutCache() :
 	level(LineCache::None),
-	allInvalidated(false), styleClock(-1) {
+	maxValidity(LineLayout::ValidLevel::invalid), styleClock(-1) {
 }
 
 LineLayoutCache::~LineLayoutCache() = default;
@@ -520,7 +520,7 @@ void LineLayoutCache::AllocateForLevel(Sci::Line linesOnScreen, Sci::Line linesI
 	}
 
 	if (lengthForLevel != cache.size()) {
-		allInvalidated = false;
+		maxValidity = LineLayout::ValidLevel::lines;
 		cache.resize(lengthForLevel);
 		// Cache::none -> no entries
 		// Cache::caret -> 1 entry can take any line
@@ -563,26 +563,25 @@ void LineLayoutCache::AllocateForLevel(Sci::Line linesOnScreen, Sci::Line linesI
 }
 
 void LineLayoutCache::Deallocate() noexcept {
+	maxValidity = LineLayout::ValidLevel::invalid;
 	cache.clear();
 }
 
 void LineLayoutCache::Invalidate(LineLayout::ValidLevel validity_) noexcept {
-	if (!cache.empty() && !allInvalidated) {
+	if (maxValidity > validity_) {
+		maxValidity = validity_;
 		for (const std::shared_ptr<LineLayout> &ll : cache) {
 			if (ll) {
 				ll->Invalidate(validity_);
 			}
 		}
-		if (validity_ == LineLayout::ValidLevel::invalid) {
-			allInvalidated = true;
-		}
 	}
 }
 
 void LineLayoutCache::SetLevel(LineCache level_) noexcept {
 	if (level != level_) {
 		level = level_;
-		allInvalidated = false;
+		maxValidity = LineLayout::ValidLevel::invalid;
 		cache.clear();
 	}
 }
@@ -594,7 +593,7 @@ std::shared_ptr<LineLayout> LineLayoutCache::Retrieve(Sci::Line lineNumber, Sci:
 		Invalidate(LineLayout::ValidLevel::checkTextAndStyle);
 		styleClock = styleClock_;
 	}
-	allInvalidated = false;
+	maxValidity = LineLayout::ValidLevel::lines;
 	size_t pos = 0;
 	if (level == LineCache::Page) {
 		// If first entry is this line then just reuse it.

scintilla/src/PositionCache.h

@@ -66,7 +66,7 @@ public:
 	std::unique_ptr<char[]> chars;
 	std::unique_ptr<unsigned char[]> styles;
 	std::unique_ptr<XYPOSITION[]> positions;
-	char bracePreviousStyles[2];
+	unsigned char bracePreviousStyles[2];
 
 	std::unique_ptr<BidiData> bidiData;
 
@@ -156,7 +156,7 @@ public:
 private:
 	Scintilla::LineCache level;
 	std::vector<std::shared_ptr<LineLayout>>cache;
-	bool allInvalidated;
+	LineLayout::ValidLevel maxValidity;
 	int styleClock;
 	size_t EntryForLine(Sci::Line line) const noexcept;
 	void AllocateForLevel(Sci::Line linesOnScreen, Sci::Line linesInDoc);

scintilla/src/RESearch.cxx

@@ -253,11 +253,9 @@ RESearch::RESearch(CharClassify *charClassTable) {
 	failure = 0;
 	charClass = charClassTable;
 	sta = NOP;                  /* status of lastpat */
-	bol = 0;
+	lineStartPos = 0;
-	constexpr unsigned char nul = 0;
+	lineEndPos = 0;
-	std::fill(bittab, std::end(bittab), nul);
+	nfa[0] = END;
-	std::fill(tagstk, std::end(tagstk), 0);
-	std::fill(nfa, std::end(nfa), '\0');
 	Clear();
 }
 
@@ -334,14 +332,11 @@ constexpr int isinset(const char *ap, unsigned char c) noexcept {
  * @return the char if it resolves to a simple char,
  * or -1 for a char class. In this case, bittab is changed.
  */
-int RESearch::GetBackslashExpression(
+int RESearch::GetBackslashExpression(const char *pattern, int &incr) noexcept {
-    const char *pattern,
-    int &incr) noexcept {
 	// Since error reporting is primitive and messages are not used anyway,
 	// I choose to interpret unexpected syntax in a logical way instead
 	// of reporting errors. Otherwise, we can stick on, eg., PCRE behaviour.
 	incr = 0;	// Most of the time, will skip the char "naturally".
-	int c = 0;
 	int result = -1;
 	const unsigned char bsc = *pattern;
 	if (!bsc) {
@@ -373,12 +368,12 @@ int RESearch::GetBackslashExpression(
 		}
 		break;
 	case 'd':
-		for (c = '0'; c <= '9'; c++) {
+		for (int c = '0'; c <= '9'; c++) {
 			ChSet(static_cast<unsigned char>(c));
 		}
 		break;
 	case 'D':
-		for (c = 0; c < MAXCHR; c++) {
+		for (int c = 0; c < MAXCHR; c++) {
 			if (c < '0' || c > '9') {
 				ChSet(static_cast<unsigned char>(c));
 			}
@@ -393,21 +388,21 @@ int RESearch::GetBackslashExpression(
 		ChSet('\v');
 		break;
 	case 'S':
-		for (c = 0; c < MAXCHR; c++) {
+		for (int c = 0; c < MAXCHR; c++) {
 			if (c != ' ' && !(c >= 0x09 && c <= 0x0D)) {
 				ChSet(static_cast<unsigned char>(c));
 			}
 		}
 		break;
 	case 'w':
-		for (c = 0; c < MAXCHR; c++) {
+		for (int c = 0; c < MAXCHR; c++) {
 			if (iswordc(static_cast<unsigned char>(c))) {
 				ChSet(static_cast<unsigned char>(c));
 			}
 		}
 		break;
 	case 'W':
-		for (c = 0; c < MAXCHR; c++) {
+		for (int c = 0; c < MAXCHR; c++) {
 			if (!iswordc(static_cast<unsigned char>(c))) {
 				ChSet(static_cast<unsigned char>(c));
 			}
@@ -419,34 +414,32 @@ int RESearch::GetBackslashExpression(
 	return result;
 }
 
-const char *RESearch::Compile(const char *pattern, Sci::Position length, bool caseSensitive, bool posix) noexcept {
+const char *RESearch::Compile(const char *pattern, Sci::Position length, bool caseSensitive, bool posix) {
-	char *mp=nfa;          /* nfa pointer       */
-	char *lp=nullptr;      /* saved pointer     */
-	char *sp=nfa;          /* another one       */
-	const char * mpMax = mp + MAXNFA - BITBLK - 10;
-
-	int tagi = 0;          /* tag stack index   */
-	int tagc = 1;          /* actual tag count  */
-
-	int n = 0;
-	char mask = 0;         /* xor mask -CCL/NCL */
-	int c1 = 0;
-	int c2 = 0;
-	int prevChar = 0;
-
 	if (!pattern || !length) {
 		if (sta)
 			return nullptr;
 		else
 			return badpat("No previous regular expression");
 	}
+
+	bittab.fill(0);
+	nfa[0] = END;
+
+	char *mp=nfa;          /* nfa pointer       */
+	char *sp=nfa;          /* another saved pointer */
+	const char * const mpMax = mp + MAXNFA - BITBLK - 10;
+
+	int tagstk[MAXTAG]{};  /* subpat tag stack */
+	int tagi = 0;          /* tag stack index   */
+	int tagc = 1;          /* actual tag count  */
+
 	sta = NOP;
 
 	const char *p=pattern;     /* pattern pointer   */
 	for (int i=0; i<length; i++, p++) {
 		if (mp > mpMax)
 			return badpat("Pattern too long");
-		lp = mp;
+		char *lp = mp;			/* saved pointer     */
 		switch (*p) {
 
 		case '.':               /* match any char  */
@@ -463,7 +456,7 @@ const char *RESearch::Compile(const char *pattern, Sci::Position length, bool ca
 			break;
 
 		case '$':               /* match endofline */
-			if (!*(p+1)) {
+			if (!p[1]) {
 				*mp++ = EOL;
 			} else {
 				*mp++ = CHR;
@@ -471,17 +464,15 @@ const char *RESearch::Compile(const char *pattern, Sci::Position length, bool ca
 			}
 			break;
 
-		case '[':               /* match char class */
+		case '[': {               /* match char class */
-			*mp++ = CCL;
+			int prevChar = 0;
-			prevChar = 0;
+			char mask = 0;      /* xor mask -CCL/NCL */
 
 			i++;
 			if (*++p == '^') {
 				mask = '\377';
 				i++;
 				p++;
-			} else {
-				mask = 0;
 			}
 
 			if (*p == '-') {	/* real dash */
@@ -500,13 +491,13 @@ const char *RESearch::Compile(const char *pattern, Sci::Position length, bool ca
 						// Previous def. was a char class like \d, take dash literally
 						prevChar = *p;
 						ChSet(*p);
-					} else if (*(p+1)) {
+					} else if (p[1]) {
-						if (*(p+1) != ']') {
+						if (p[1] != ']') {
-							c1 = prevChar + 1;
+							int c1 = prevChar + 1;
 							i++;
-							c2 = static_cast<unsigned char>(*++p);
+							int c2 = static_cast<unsigned char>(*++p);
 							if (c2 == '\\') {
-								if (!*(p+1)) {	// End of RE
+								if (!p[1]) {	// End of RE
 									return badpat("Missing ]");
 								} else {
 									i++;
@@ -543,7 +534,7 @@ const char *RESearch::Compile(const char *pattern, Sci::Position length, bool ca
 					} else {
 						return badpat("Missing ]");
 					}
-				} else if (*p == '\\' && *(p+1)) {
+				} else if (*p == '\\' && p[1]) {
 					i++;
 					p++;
 					int incr;
@@ -568,10 +559,12 @@ const char *RESearch::Compile(const char *pattern, Sci::Position length, bool ca
 			if (!*p)
 				return badpat("Missing ]");
 
-			for (n = 0; n < BITBLK; bittab[n++] = 0)
+			*mp++ = CCL;
-				*mp++ = static_cast<char>(mask ^ bittab[n]);
+			for (const unsigned char byte : bittab) {
-
+				*mp++ = mask ^ byte;
-			break;
+			}
+			bittab.fill(0);
+		} break;
 
 		case '*':               /* match 0 or more... */
 		case '+':               /* match 1 or more... */
@@ -605,7 +598,7 @@ const char *RESearch::Compile(const char *pattern, Sci::Position length, bool ca
 			while (--mp > lp)
 				*mp = mp[-1];
 			if (*p == '?')          *mp = CLQ;
-			else if (*(p+1) == '?') *mp = LCLO;
+			else if (p[1] == '?') *mp = LCLO;
 			else                    *mp = CLO;
 
 			mp = sp;
@@ -630,8 +623,8 @@ const char *RESearch::Compile(const char *pattern, Sci::Position length, bool ca
 			case '6':
 			case '7':
 			case '8':
-			case '9':
+			case '9': {
-				n = *p-'0';
+				const int n = *p-'0';
 				if (tagi > 0 && tagstk[tagi] == n)
 					return badpat("Cyclical reference");
 				if (tagc > n) {
@@ -640,7 +633,7 @@ const char *RESearch::Compile(const char *pattern, Sci::Position length, bool ca
 				} else {
 					return badpat("Undetermined reference");
 				}
-				break;
+			} break;
 			default:
 				if (!posix && *p == '(') {
 					if (tagc < MAXTAG) {
@@ -669,9 +662,10 @@ const char *RESearch::Compile(const char *pattern, Sci::Position length, bool ca
 						*mp++ = static_cast<unsigned char>(c);
 					} else {
 						*mp++ = CCL;
-						mask = 0;
+						for (const unsigned char byte : bittab) {
-						for (n = 0; n < BITBLK; bittab[n++] = 0)
+							*mp++ = byte;
-							*mp++ = static_cast<char>(mask ^ bittab[n]);
+						}
+						bittab.fill(0);
 					}
 				}
 			}
@@ -703,11 +697,12 @@ const char *RESearch::Compile(const char *pattern, Sci::Position length, bool ca
 					*mp++ = CHR;
 					*mp++ = c;
 				} else {
-					*mp++ = CCL;
-					mask = 0;
 					ChSetWithCase(c, false);
-					for (n = 0; n < BITBLK; bittab[n++] = 0)
+					*mp++ = CCL;
-						*mp++ = static_cast<char>(mask ^ bittab[n]);
+					for (const unsigned char byte : bittab) {
+						*mp++ = byte;
+					}
+					bittab.fill(0);
 				}
 			}
 			break;
@@ -743,11 +738,9 @@ const char *RESearch::Compile(const char *pattern, Sci::Position length, bool ca
  *
  */
 int RESearch::Execute(const CharacterIndexer &ci, Sci::Position lp, Sci::Position endp) {
-	unsigned char c = 0;
 	Sci::Position ep = NOTFOUND;
-	char *ap = nfa;
+	const char * const ap = nfa;
 
-	bol = lp;
 	failure = 0;
 
 	Clear();
@@ -758,34 +751,54 @@ int RESearch::Execute(const CharacterIndexer &ci, Sci::Position lp, Sci::Positio
 		ep = PMatch(ci, lp, endp, ap);
 		break;
 	case EOL:			/* just searching for end of line normal path doesn't work */
-		if (*(ap+1) == END) {
+		if (endp == lineEndPos && ap[1] == END) {
 			lp = endp;
 			ep = lp;
 			break;
 		} else {
 			return 0;
 		}
-	case CHR:			/* ordinary char: locate it fast */
+	case CHR: {			/* ordinary char: locate it fast */
-		c = *(ap+1);
+		const unsigned char c = ap[1];
 		while ((lp < endp) && (static_cast<unsigned char>(ci.CharAt(lp)) != c))
 			lp++;
 		if (lp >= endp)	/* if EOS, fail, else fall through. */
 			return 0;
+	}
 	[[fallthrough]];
 	default:			/* regular matching all the way. */
 		while (lp < endp) {
 			ep = PMatch(ci, lp, endp, ap);
-			if (ep != NOTFOUND)
+			if (ep != NOTFOUND) {
+				// fix match started from middle of character like DBCS trailing ASCII byte
+				const Sci::Position pos = ci.MovePositionOutsideChar(lp, -1);
+				if (pos != lp) {
+					ep = NOTFOUND;
+				} else {
 					break;
+				}
+			}
 			lp++;
 		}
 		break;
 	case END:			/* munged automaton. fail always */
 		return 0;
 	}
-	if (ep == NOTFOUND)
+	if (ep == NOTFOUND) {
+		/* similar to EOL, match EOW at line end */
+		if (endp == lineEndPos && *ap == EOW) {
+			if ((ap[1] == END || ((ap[1] == EOL && ap[2] == END))) && iswordc(ci.CharAt(lp - 1))) {
+				lp = endp;
+				ep = lp;
+			} else {
 				return 0;
+			}
+		} else {
+			return 0;
+		}
+	}
 
+	ep = ci.MovePositionOutsideChar(ep, 1);
 	bopat[0] = lp;
 	eopat[0] = ep;
 	return 1;
@@ -830,15 +843,8 @@ int RESearch::Execute(const CharacterIndexer &ci, Sci::Position lp, Sci::Positio
 #define CHRSKIP 3	/* [CLO] CHR chr END      */
 #define CCLSKIP 34	/* [CLO] CCL 32 bytes END */
 
-Sci::Position RESearch::PMatch(const CharacterIndexer &ci, Sci::Position lp, Sci::Position endp, char *ap) {
+Sci::Position RESearch::PMatch(const CharacterIndexer &ci, Sci::Position lp, Sci::Position endp, const char *ap) {
-	int op = 0;
+	unsigned char op = 0;
-	int c = 0;
-	int n = 0;
-	Sci::Position e = 0;		/* extra pointer for CLO  */
-	Sci::Position bp = 0;		/* beginning of subpat... */
-	Sci::Position ep = 0;		/* ending of subpat...    */
-	Sci::Position are = 0;	/* to save the line ptr.  */
-	Sci::Position llp = 0;	/* lazy lp for LCLO       */
 
 	while ((op = *ap++) != END)
 		switch (op) {
@@ -859,39 +865,44 @@ Sci::Position RESearch::PMatch(const CharacterIndexer &ci, Sci::Position lp, Sci
 			ap += BITBLK;
 			break;
 		case BOL:
-			if (lp != bol)
+			if (lp != lineStartPos)
 				return NOTFOUND;
 			break;
 		case EOL:
-			if (lp < endp)
+			if (lp < lineEndPos)
 				return NOTFOUND;
 			break;
 		case BOT:
-			bopat[static_cast<int>(*ap++)] = lp;
+			if (lp != ci.MovePositionOutsideChar(lp, -1)) {
+				return NOTFOUND;
+			}
+			bopat[static_cast<unsigned char>(*ap++)] = lp;
 			break;
 		case EOT:
-			eopat[static_cast<int>(*ap++)] = lp;
+			lp = ci.MovePositionOutsideChar(lp, 1);
+			eopat[static_cast<unsigned char>(*ap++)] = lp;
 			break;
 		case BOW:
-			if ((lp!=bol && iswordc(ci.CharAt(lp-1))) || !iswordc(ci.CharAt(lp)))
+			if ((lp!=lineStartPos && iswordc(ci.CharAt(lp-1))) || !iswordc(ci.CharAt(lp)))
 				return NOTFOUND;
 			break;
 		case EOW:
-			if (lp==bol || !iswordc(ci.CharAt(lp-1)) || iswordc(ci.CharAt(lp)))
+			if (lp==lineStartPos || !iswordc(ci.CharAt(lp-1)) || iswordc(ci.CharAt(lp)))
 				return NOTFOUND;
 			break;
-		case REF:
+		case REF: {
-			n = *ap++;
+			const int n = static_cast<unsigned char>(*ap++);
-			bp = bopat[n];
+			Sci::Position bp = bopat[n];		/* beginning of subpat... */
-			ep = eopat[n];
+			const Sci::Position ep = eopat[n];	/* ending of subpat...    */
 			while (bp < ep)
 				if (ci.CharAt(bp++) != ci.CharAt(lp++))
 					return NOTFOUND;
-			break;
+		} break;
 		case LCLO:
 		case CLQ:
-		case CLO:
+		case CLO: {
-			are = lp;
+			int n = 0;
+			const Sci::Position are = lp;	/* to save the line ptr.  */
 			switch (*ap) {
 
 			case ANY:
@@ -903,15 +914,15 @@ Sci::Position RESearch::PMatch(const CharacterIndexer &ci, Sci::Position lp, Sci
 
 				n = ANYSKIP;
 				break;
-			case CHR:
+			case CHR: {
-				c = *(ap+1);
+				const char c = ap[1];
 				if (op == CLO || op == LCLO)
 					while ((lp < endp) && (c == ci.CharAt(lp)))
 						lp++;
 				else if ((lp < endp) && (c == ci.CharAt(lp)))
 					lp++;
 				n = CHRSKIP;
-				break;
+			} break;
 			case CCL:
 				while ((lp < endp) && isinset(ap+1, ci.CharAt(lp)))
 					lp++;
@@ -924,8 +935,8 @@ Sci::Position RESearch::PMatch(const CharacterIndexer &ci, Sci::Position lp, Sci
 			}
 			ap += n;
 
-			llp = lp;
+			Sci::Position llp = lp;		/* lazy lp for LCLO       */
-			e = NOTFOUND;
+			Sci::Position e = NOTFOUND; /* extra pointer for CLO  */
 			while (llp >= are) {
 				Sci::Position q;
 				if ((q = PMatch(ci, llp, endp, ap)) != NOTFOUND) {
@@ -939,11 +950,10 @@ Sci::Position RESearch::PMatch(const CharacterIndexer &ci, Sci::Position lp, Sci
 			if (*ap == EOT)
 				PMatch(ci, lp, endp, ap);
 			return e;
+		}
 		default:
 			//re_fail("RESearch::Execute: bad nfa.", static_cast<char>(op));
 			return NOTFOUND;
 		}
 	return lp;
 }
-
-

scintilla/src/RESearch.h

@@ -14,6 +14,7 @@ namespace Scintilla::Internal {
 class CharacterIndexer {
 public:
 	virtual char CharAt(Sci::Position index) const=0;
+	virtual Sci::Position MovePositionOutsideChar(Sci::Position pos, [[maybe_unused]] Sci::Position moveDir) const noexcept=0;
 };
 
 class RESearch {
@@ -22,8 +23,12 @@ public:
 	explicit RESearch(CharClassify *charClassTable);
 	// No dynamic allocation so default copy constructor and assignment operator are OK.
 	void Clear();
-	const char *Compile(const char *pattern, Sci::Position length, bool caseSensitive, bool posix) noexcept;
+	const char *Compile(const char *pattern, Sci::Position length, bool caseSensitive, bool posix);
 	int Execute(const CharacterIndexer &ci, Sci::Position lp, Sci::Position endp);
+	void SetLineRange(Sci::Position startPos, Sci::Position endPos) noexcept {
+		lineStartPos = startPos;
+		lineEndPos = endPos;
+	}
 
 	static constexpr int MAXTAG = 10;
 	static constexpr int NOTFOUND = -1;
@@ -45,14 +50,15 @@ private:
 	void ChSetWithCase(unsigned char c, bool caseSensitive) noexcept;
 	int GetBackslashExpression(const char *pattern, int &incr) noexcept;
 
-	Sci::Position PMatch(const CharacterIndexer &ci, Sci::Position lp, Sci::Position endp, char *ap);
+	Sci::Position PMatch(const CharacterIndexer &ci, Sci::Position lp, Sci::Position endp, const char *ap);
 
-	Sci::Position bol;
+	// positions to match line start and line end
-	Sci::Position tagstk[MAXTAG];  /* subpat tag stack */
+	Sci::Position lineStartPos;
+	Sci::Position lineEndPos;
 	char nfa[MAXNFA];    /* automaton */
 	int sta;
-	unsigned char bittab[BITBLK]; /* bit table for CCL pre-set bits */
 	int failure;
+	std::array<unsigned char, BITBLK> bittab {}; /* bit table for CCL pre-set bits */
 	CharClassify *charClass;
 	bool iswordc(unsigned char x) const noexcept {
 		return charClass->IsWord(x);

scintilla/src/ScintillaBase.cxx

@@ -804,10 +804,7 @@ const char *LexState::DescriptionOfStyle(int style) {
 
 void ScintillaBase::NotifyStyleToNeeded(Sci::Position endStyleNeeded) {
 	if (!DocumentLexState()->UseContainerLexing()) {
-		const Sci::Line lineEndStyled =
+		const Sci::Position endStyled = pdoc->LineStartPosition(pdoc->GetEndStyled());
-			pdoc->SciLineFromPosition(pdoc->GetEndStyled());
-		const Sci::Position endStyled =
-			pdoc->LineStart(lineEndStyled);
 		DocumentLexState()->Colourise(endStyled, endStyleNeeded);
 		return;
 	}

scintilla/src/Selection.cxx

@@ -123,6 +123,13 @@ bool SelectionRange::ContainsCharacter(Sci::Position posCharacter) const noexcep
 		return (posCharacter >= anchor.Position()) && (posCharacter < caret.Position());
 }
 
+bool SelectionRange::ContainsCharacter(SelectionPosition spCharacter) const noexcept {
+	if (anchor > caret)
+		return (spCharacter >= caret) && (spCharacter < anchor);
+	else
+		return (spCharacter >= anchor) && (spCharacter < caret);
+}
+
 SelectionSegment SelectionRange::Intersect(SelectionSegment check) const noexcept {
 	const SelectionSegment inOrder(caret, anchor);
 	if ((inOrder.start <= check.end) || (inOrder.end >= check.start)) {

scintilla/src/Selection.h

@@ -129,6 +129,7 @@ struct SelectionRange {
 	bool Contains(Sci::Position pos) const noexcept;
 	bool Contains(SelectionPosition sp) const noexcept;
 	bool ContainsCharacter(Sci::Position posCharacter) const noexcept;
+	bool ContainsCharacter(SelectionPosition spCharacter) const noexcept;
 	SelectionSegment Intersect(SelectionSegment check) const noexcept;
 	SelectionPosition Start() const noexcept {
 		return (anchor < caret) ? anchor : caret;

scintilla/src/ViewStyle.cxx

@@ -712,8 +712,7 @@ bool ViewStyle::SetWrapIndentMode(WrapIndentMode wrapIndentMode_) noexcept {
 
 bool ViewStyle::IsBlockCaretStyle() const noexcept {
 	return ((caret.style & CaretStyle::InsMask) == CaretStyle::Block) ||
-		FlagSet(caret.style, CaretStyle::OverstrikeBlock) ||
+		FlagSet(caret.style, (CaretStyle::OverstrikeBlock | CaretStyle::Curses));
-		FlagSet(caret.style, CaretStyle::Curses);
 }
 
 bool ViewStyle::IsCaretVisible(bool isMainSelection) const noexcept {

scintilla/test/performanceTests.py

@@ -28,24 +28,24 @@ class TestPerformance(unittest.TestCase):
 		start = timer()
 		for i in range(2000):
 			self.ed.AddText(len(data), data)
-			self.assertEquals(self.ed.LineCount, i + 2)
+			self.assertEqual(self.ed.LineCount, i + 2)
 		end = timer()
 		duration = end - start
 		print("%6.3f testAddLine" % duration)
 		self.xite.DoEvents()
-		self.assert_(self.ed.Length > 0)
+		self.assertTrue(self.ed.Length > 0)
 
 	def testAddLineMiddle(self):
 		data = (string.ascii_letters + string.digits + "\n").encode('utf-8')
 		start = timer()
 		for i in range(2000):
 			self.ed.AddText(len(data), data)
-			self.assertEquals(self.ed.LineCount, i + 2)
+			self.assertEqual(self.ed.LineCount, i + 2)
 		end = timer()
 		duration = end - start
 		print("%6.3f testAddLineMiddle" % duration)
 		self.xite.DoEvents()
-		self.assert_(self.ed.Length > 0)
+		self.assertTrue(self.ed.Length > 0)
 
 	def testHuge(self):
 		data = (string.ascii_letters + string.digits + "\n").encode('utf-8')
@@ -56,7 +56,7 @@ class TestPerformance(unittest.TestCase):
 		duration = end - start
 		print("%6.3f testHuge" % duration)
 		self.xite.DoEvents()
-		self.assert_(self.ed.Length > 0)
+		self.assertTrue(self.ed.Length > 0)
 
 	def testHugeInserts(self):
 		data = (string.ascii_letters + string.digits + "\n").encode('utf-8')
@@ -70,7 +70,7 @@ class TestPerformance(unittest.TestCase):
 		duration = end - start
 		print("%6.3f testHugeInserts" % duration)
 		self.xite.DoEvents()
-		self.assert_(self.ed.Length > 0)
+		self.assertTrue(self.ed.Length > 0)
 
 	def testHugeReplace(self):
 		oneLine = (string.ascii_letters + string.digits + "\n").encode('utf-8')
@@ -86,7 +86,7 @@ class TestPerformance(unittest.TestCase):
 		duration = end - start
 		print("%6.3f testHugeReplace" % duration)
 		self.xite.DoEvents()
-		self.assert_(self.ed.Length > 0)
+		self.assertTrue(self.ed.Length > 0)
 
 	def testUTF8CaseSearches(self):
 		self.ed.SetCodePage(65001)
@@ -101,7 +101,7 @@ class TestPerformance(unittest.TestCase):
 			self.ed.TargetEnd = self.ed.Length-1
 			self.ed.SearchFlags = self.ed.SCFIND_MATCHCASE
 			pos = self.ed.SearchInTarget(len(searchString), searchString)
-			self.assert_(pos > 0)
+			self.assertTrue(pos > 0)
 		end = timer()
 		duration = end - start
 		print("%6.3f testUTF8CaseSearches" % duration)
@@ -120,7 +120,7 @@ class TestPerformance(unittest.TestCase):
 			self.ed.TargetEnd = self.ed.Length-1
 			self.ed.SearchFlags = 0
 			pos = self.ed.SearchInTarget(len(searchString), searchString)
-			self.assert_(pos > 0)
+			self.assertTrue(pos > 0)
 		end = timer()
 		duration = end - start
 		print("%6.3f testUTF8Searches" % duration)
@@ -139,7 +139,7 @@ class TestPerformance(unittest.TestCase):
 			self.ed.TargetEnd = self.ed.Length-1
 			self.ed.SearchFlags = 0
 			pos = self.ed.SearchInTarget(len(searchString), searchString)
-			self.assert_(pos > 0)
+			self.assertTrue(pos > 0)
 		end = timer()
 		duration = end - start
 		print("%6.3f testUTF8AsciiSearches" % duration)

scintilla/test/simpleTests.py

@@ -1,5 +1,6 @@
 #!/usr/bin/env python3
 # -*- coding: utf-8 -*-
+# @file simpleTests.py
 # Requires Python 2.7 or later
 
 from __future__ import with_statement
@@ -808,11 +809,44 @@ class TestSimple(unittest.TestCase):
 		self.assertEqual(self.ed.TargetEndVirtualSpace, 0)
 
 	def testPointsAndPositions(self):
-		self.ed.AddText(1, b"x")
+		self.ed.SetContents(b"xyz")
+
+		# Inter-character positions
 		# Start of text
-		self.assertEqual(self.ed.PositionFromPoint(0,0), 0)
+		self.assertEqual(self.ed.PositionFromPoint(1,1), 0)
 		# End of text
-		self.assertEqual(self.ed.PositionFromPoint(0,100), 1)
+		self.assertEqual(self.ed.PositionFromPoint(100, 1), 3)
+		self.assertEqual(self.ed.PositionFromPointClose(100, 1), -1)
+		
+		# Character positions
+		# Start of text
+		self.assertEqual(self.ed.CharPositionFromPoint(0,0), 0)
+		# End of text
+		self.assertEqual(self.ed.CharPositionFromPoint(100, 0), 3)
+		self.assertEqual(self.ed.CharPositionFromPointClose(100, 0), -1)
+
+	def testSelectionFromPoint(self):
+		self.ed.SetContents(b"xxxxxx")
+		self.ed.SetSelection(3, 2)
+		self.ed.AddSelection(0, 1)
+		self.ed.AddSelection(5, 5)	# Empty
+		xStart = self.ed.PointXFromPosition(0, 0)
+		xEnd = self.ed.PointXFromPosition(0, 5)
+		width = xEnd-xStart
+		charWidth = width // 5
+		widthMid = charWidth // 2
+
+		posMid1 = xStart+widthMid
+		self.assertEqual(self.ed.SelectionFromPoint(posMid1, 1), 1)
+		posMid2 = xStart+charWidth+widthMid
+		self.assertEqual(self.ed.SelectionFromPoint(posMid2, 1), -1)
+		posMid3 = xStart+charWidth*2+widthMid
+		self.assertEqual(self.ed.SelectionFromPoint(posMid3, 1), 0)
+		# Empty selection at 5. Exact and then a few pixels either side
+		self.assertEqual(self.ed.SelectionFromPoint(xEnd, 1), 2)
+		self.assertEqual(self.ed.SelectionFromPoint(xEnd-2, 1), 2)
+		self.assertEqual(self.ed.SelectionFromPoint(xEnd+2, 1), 2)
+		self.assertEqual(self.ed.SelectionFromPoint(100, 0), -1)
 
 	def testLinePositions(self):
 		text = b"ab\ncd\nef"
@@ -1741,6 +1775,15 @@ def selectionRangeRepresentation(selectionRange):
 	anchor, caret = selectionRange
 	return selectionPositionRepresentation(anchor) + "-" + selectionPositionRepresentation(caret)
 
+def selectionRepresentation(ed, n):
+	anchor = (ed.GetSelectionNAnchor(n), ed.GetSelectionNAnchorVirtualSpace(n))
+	caret = (ed.GetSelectionNCaret(n), ed.GetSelectionNCaretVirtualSpace(n))
+	return selectionRangeRepresentation((anchor, caret))
+
+def allSelectionsRepresentation(ed):
+	reps = [selectionRepresentation(ed, i) for i in range(ed.Selections)]
+	return ';'.join(reps)
+
 class TestMultiSelection(unittest.TestCase):
 
 	def setUp(self):
@@ -1959,11 +2002,6 @@ class TestMultiSelection(unittest.TestCase):
 		self.assertEqual(self.textOfSelection(0), texts[1])
 		self.assertEqual(self.textOfSelection(1), texts[0])
 
-	def selectionRepresentation(self, n):
-		anchor = (self.ed.GetSelectionNAnchor(0), self.ed.GetSelectionNAnchorVirtualSpace(0))
-		caret = (self.ed.GetSelectionNCaret(0), self.ed.GetSelectionNCaretVirtualSpace(0))
-		return selectionRangeRepresentation((anchor, caret))
-
 	def testAdjacentSelections(self):
 		# For various permutations of selections, try swapping the text and ensure that the
 		# selections remain distinct
@@ -2007,14 +2045,14 @@ class TestMultiSelection(unittest.TestCase):
 		self.ed.SetSelection(1, 1)
 		self.ed.SetSelectionNAnchorVirtualSpace(0, 2)
 		self.ed.SetSelectionNCaretVirtualSpace(0, 2)
-		self.assertEqual(self.selectionRepresentation(0), "1+2v-1+2v")
+		self.assertEqual(selectionRepresentation(self.ed, 0), "1+2v-1+2v")
 		self.assertEqual(self.textOfSelection(0), b'')
 
 		# Append '1'
 		self.ed.SetTargetRange(1, 1)
 		self.ed.ReplaceTarget(1, b'1')
 		# Selection moved on 1, but still empty
-		self.assertEqual(self.selectionRepresentation(0), "2+1v-2+1v")
+		self.assertEqual(selectionRepresentation(self.ed, 0), "2+1v-2+1v")
 		self.assertEqual(self.ed.Contents(), b'a1')
 		self.assertEqual(self.textOfSelection(0), b'')
 
@@ -2023,7 +2061,7 @@ class TestMultiSelection(unittest.TestCase):
 		self.ed.SetSelection(1, 1)
 		self.ed.SetSelectionNAnchorVirtualSpace(0, 2)
 		self.ed.SetSelectionNCaretVirtualSpace(0, 3)
-		self.assertEqual(self.selectionRepresentation(0), "1+2v-1+3v")
+		self.assertEqual(selectionRepresentation(self.ed, 0), "1+2v-1+3v")
 		self.assertEqual(self.textOfSelection(0), b'')
 
 		# Append '1' past current virtual space
@@ -2032,7 +2070,7 @@ class TestMultiSelection(unittest.TestCase):
 		self.ed.SetTargetEndVirtualSpace(5)
 		self.ed.ReplaceTarget(1, b'1')
 		# Virtual space of selection all converted to real positions
-		self.assertEqual(self.selectionRepresentation(0), "3-4")
+		self.assertEqual(selectionRepresentation(self.ed, 0), "3-4")
 		self.assertEqual(self.ed.Contents(), b'a    1')
 		self.assertEqual(self.textOfSelection(0), b' ')
 
@@ -2050,26 +2088,44 @@ class TestModalSelection(unittest.TestCase):
 
 	def testCharacterSelection(self):
 		self.ed.SetSelection(1, 1)
-		self.assertEqual(self.ed.Selections, 1)
 		self.assertEqual(self.ed.MainSelection, 0)
-		self.assertEqual(self.ed.GetSelectionNCaret(0), 1)
+		self.assertEqual(allSelectionsRepresentation(self.ed), "1-1")
-		self.assertEqual(self.ed.GetSelectionNAnchor(0), 1)
 		self.ed.SelectionMode = self.ed.SC_SEL_STREAM
 		self.assertEqual(self.ed.GetSelectionMode(), self.ed.SC_SEL_STREAM)
-		self.assertEqual(self.ed.Selections, 1)
+		self.assertEqual(self.ed.MoveExtendsSelection, True)
-		self.assertEqual(self.ed.MainSelection, 0)
+		self.assertEqual(allSelectionsRepresentation(self.ed), "1-1")
-		self.assertEqual(self.ed.GetSelectionNCaret(0), 1)
-		self.assertEqual(self.ed.GetSelectionNAnchor(0), 1)
 		self.ed.CharRight()
-		self.assertEqual(self.ed.Selections, 1)
+		self.assertEqual(allSelectionsRepresentation(self.ed), "1-2")
-		self.assertEqual(self.ed.MainSelection, 0)
-		self.assertEqual(self.ed.GetSelectionNCaret(0), 2)
-		self.assertEqual(self.ed.GetSelectionNAnchor(0), 1)
 		self.ed.LineDown()
-		self.assertEqual(self.ed.Selections, 1)
+		self.assertEqual(allSelectionsRepresentation(self.ed), "1-6")
-		self.assertEqual(self.ed.MainSelection, 0)
+		self.ed.ClearSelections()
-		self.assertEqual(self.ed.GetSelectionNCaret(0), 6)
+
-		self.assertEqual(self.ed.GetSelectionNAnchor(0), 1)
+	def testChangeSelectionMode(self):
+		# Like testCharacterSelection but calling ChangeSelectionMode instead of SetSelectionMode
+		self.ed.SetSelection(1, 1)
+		self.assertEqual(allSelectionsRepresentation(self.ed), "1-1")
+		self.ed.ChangeSelectionMode(self.ed.SC_SEL_STREAM)
+		self.assertEqual(self.ed.GetSelectionMode(), self.ed.SC_SEL_STREAM)
+		self.assertEqual(self.ed.MoveExtendsSelection, False)
+		self.assertEqual(allSelectionsRepresentation(self.ed), "1-1")
+		self.ed.CharRight()
+		self.assertEqual(allSelectionsRepresentation(self.ed), "2-2")
+		self.ed.LineDown()
+		self.assertEqual(allSelectionsRepresentation(self.ed), "6-6")
+		self.ed.ClearSelections()
+
+	def testTurningOffMoveExtendsSelection(self):
+		self.ed.SetSelection(1, 1)
+		self.ed.SelectionMode = self.ed.SC_SEL_STREAM
+		self.ed.CharRight()
+		self.ed.LineDown()
+		self.assertEqual(self.ed.MoveExtendsSelection, True)
+		self.ed.MoveExtendsSelection = False
+		self.assertEqual(self.ed.MoveExtendsSelection, False)
+		self.ed.CharRight()
+		self.assertEqual(allSelectionsRepresentation(self.ed), "6-6")
+		self.ed.CharRight()
+		self.assertEqual(allSelectionsRepresentation(self.ed), "7-7")
 		self.ed.ClearSelections()
 
 	def testRectangleSelection(self):

scintilla/test/unit/makefile

@@ -23,7 +23,6 @@ CXX = clang++
 ifdef USELIBCPP
 # macOS, use libc++ but don't have sanitizers
 CXXFLAGS += --stdlib=libc++
-LINKFLAGS = -lc++
 else
 ifndef windir
 # Linux, have sanitizers
@@ -43,30 +42,34 @@ DEL = rm -f
 EXE = unitTest
 endif
 
+vpath %.cxx ../../src
+
 INCLUDEDIRS = -I ../../include -I ../../src
 
 CPPFLAGS += $(INCLUDEDIRS)
 CXXFLAGS += -Wall -Wextra
 
 # Files in this directory containing tests
-TESTSRC=test*.cxx
+TESTSRC=$(wildcard test*.cxx)
+TESTOBJ=$(TESTSRC:.cxx=.o)
+
 # Files being tested from scintilla/src directory
-TESTEDSRC=\
+TESTEDOBJ=\
- ../../src/CaseConvert.cxx \
+CaseConvert.o \
- ../../src/CaseFolder.cxx \
+CaseFolder.o \
- ../../src/CellBuffer.cxx \
+CellBuffer.o \
- ../../src/ChangeHistory.cxx \
+ChangeHistory.o \
- ../../src/CharacterCategoryMap.cxx \
+CharacterCategoryMap.o \
- ../../src/CharClassify.cxx \
+CharClassify.o \
- ../../src/ContractionState.cxx \
+ContractionState.o \
- ../../src/Decoration.cxx \
+Decoration.o \
- ../../src/Document.cxx \
+Document.o \
- ../../src/Geometry.cxx \
+Geometry.o \
- ../../src/PerLine.cxx \
+PerLine.o \
- ../../src/RESearch.cxx \
+RESearch.o \
- ../../src/RunStyles.cxx \
+RunStyles.o \
- ../../src/UniConversion.cxx \
+UniConversion.o \
- ../../src/UniqueString.cxx
+UniqueString.o
 
 TESTS=$(EXE)
 
@@ -78,5 +81,8 @@ test: $(TESTS)
 clean:
 	$(DEL) $(TESTS) *.o *.obj *.exe
 
-$(EXE): $(TESTSRC) $(TESTEDSRC) unitTest.cxx
+%.o: %.cxx
+	$(CXX) $(CPPFLAGS) $(CXXFLAGS) -c $< -o $@
+
+$(EXE): $(TESTOBJ) $(TESTEDOBJ) unitTest.o
 	$(CXX) $(CPPFLAGS) $(CXXFLAGS) $(LINKFLAGS) $^ -o $@

scintilla/test/unit/testCellBuffer.cxx

@@ -30,20 +30,23 @@ using namespace Scintilla;
 using namespace Scintilla::Internal;
 
 // Test CellBuffer.
+bool Equal(const char *ptr, std::string_view sv) noexcept {
+	return memcmp(ptr, sv.data(), sv.length()) == 0;
+}
 
 TEST_CASE("CellBuffer") {
 
-	const char sText[] = "Scintilla";
+	constexpr std::string_view sText = "Scintilla";
-	const Sci::Position sLength = static_cast<Sci::Position>(strlen(sText));
+	constexpr Sci::Position sLength = sText.length();
 
 	CellBuffer cb(true, false);
 
 	SECTION("InsertOneLine") {
 		bool startSequence = false;
-		const char *cpChange = cb.InsertString(0, sText, sLength, startSequence);
+		const char *cpChange = cb.InsertString(0, sText.data(), sLength, startSequence);
 		REQUIRE(startSequence);
 		REQUIRE(sLength == cb.Length());
-		REQUIRE(memcmp(cpChange, sText, sLength) == 0);
+		REQUIRE(Equal(cpChange, sText));
 		REQUIRE(1 == cb.Lines());
 		REQUIRE(0 == cb.LineStart(0));
 		REQUIRE(0 == cb.LineFromPosition(0));
@@ -54,13 +57,13 @@ TEST_CASE("CellBuffer") {
 	}
 
 	SECTION("InsertTwoLines") {
-		const char sText2[] = "Two\nLines";
+		constexpr std::string_view sText2 = "Two\nLines";
-		const Sci::Position sLength2 = static_cast<Sci::Position>(strlen(sText2));
+		constexpr Sci::Position sLength2 = sText2.length();
 		bool startSequence = false;
-		const char *cpChange = cb.InsertString(0, sText2, sLength2, startSequence);
+		const char *cpChange = cb.InsertString(0, sText2.data(), sLength2, startSequence);
 		REQUIRE(startSequence);
 		REQUIRE(sLength2 == cb.Length());
-		REQUIRE(memcmp(cpChange, sText2, sLength2) == 0);
+		REQUIRE(Equal(cpChange, sText2));
 		REQUIRE(2 == cb.Lines());
 		REQUIRE(0 == cb.LineStart(0));
 		REQUIRE(0 == cb.LineFromPosition(0));
@@ -78,45 +81,45 @@ TEST_CASE("CellBuffer") {
 		bool startSequence = false;
 		{
 			// Unix \n
-			const char sText2[] = "Two\nLines";
+			constexpr std::string_view sText2 = "Two\nLines";
-			const Sci::Position sLength2 = static_cast<Sci::Position>(strlen(sText2));
+			constexpr Sci::Position sLength2 = sText2.length();
-			cb.InsertString(0, sText2, strlen(sText2), startSequence);
+			cb.InsertString(0, sText2.data(), sLength2, startSequence);
 			REQUIRE(3 == cb.LineEnd(0));
 			REQUIRE(sLength2 == cb.LineEnd(1));
 			cb.DeleteChars(0, sLength2, startSequence);
 		}
 		{
 			// Windows \r\n
-			const char sText2[] = "Two\r\nLines";
+			constexpr std::string_view sText2 = "Two\r\nLines";
-			const Sci::Position sLength2 = static_cast<Sci::Position>(strlen(sText2));
+			constexpr Sci::Position sLength2 = sText2.length();
-			cb.InsertString(0, sText2, sLength2, startSequence);
+			cb.InsertString(0, sText2.data(), sLength2, startSequence);
 			REQUIRE(3 == cb.LineEnd(0));
 			REQUIRE(sLength2 == cb.LineEnd(1));
 			cb.DeleteChars(0, sLength2, startSequence);
 		}
 		{
 			// Old macOS \r
-			const char sText2[] = "Two\rLines";
+			constexpr std::string_view sText2 = "Two\rLines";
-			const Sci::Position sLength2 = static_cast<Sci::Position>(strlen(sText2));
+			constexpr Sci::Position sLength2 = sText2.length();
-			cb.InsertString(0, sText2, strlen(sText2), startSequence);
+			cb.InsertString(0, sText2.data(), sLength2, startSequence);
 			REQUIRE(3 == cb.LineEnd(0));
 			REQUIRE(sLength2 == cb.LineEnd(1));
 			cb.DeleteChars(0, sLength2, startSequence);
 		}
 		{
 			// Unicode NEL is U+0085 \xc2\x85
-			const char sText2[] = "Two\xc2\x85Lines";
+			constexpr std::string_view sText2 = "Two\xc2\x85Lines";
-			const Sci::Position sLength2 = static_cast<Sci::Position>(strlen(sText2));
+			constexpr Sci::Position sLength2 = sText2.length();
-			cb.InsertString(0, sText2, sLength2, startSequence);
+			cb.InsertString(0, sText2.data(), sLength2, startSequence);
 			REQUIRE(3 == cb.LineEnd(0));
 			REQUIRE(sLength2 == cb.LineEnd(1));
 			cb.DeleteChars(0, sLength2, startSequence);
 		}
 		{
 			// Unicode LS line separator is U+2028 \xe2\x80\xa8
-			const char sText2[] = "Two\xe2\x80\xa8Lines";
+			constexpr std::string_view sText2 = "Two\xe2\x80\xa8Lines";
-			const Sci::Position sLength2 = static_cast<Sci::Position>(strlen(sText2));
+			constexpr Sci::Position sLength2 = sText2.length();
-			cb.InsertString(0, sText2, sLength2, startSequence);
+			cb.InsertString(0, sText2.data(), sLength2, startSequence);
 			REQUIRE(3 == cb.LineEnd(0));
 			REQUIRE(sLength2 == cb.LineEnd(1));
 			cb.DeleteChars(0, sLength2, startSequence);
@@ -129,36 +132,36 @@ TEST_CASE("CellBuffer") {
 		cb.SetUndoCollection(false);
 		REQUIRE(!cb.IsCollectingUndo());
 		bool startSequence = false;
-		const char *cpChange = cb.InsertString(0, sText, sLength, startSequence);
+		const char *cpChange = cb.InsertString(0, sText.data(), sLength, startSequence);
 		REQUIRE(!startSequence);
 		REQUIRE(sLength == cb.Length());
-		REQUIRE(memcmp(cpChange, sText, sLength) == 0);
+		REQUIRE(Equal(cpChange, sText));
 		REQUIRE(!cb.CanUndo());
 		REQUIRE(!cb.CanRedo());
 	}
 
 	SECTION("UndoRedo") {
-		const char sTextDeleted[] = "ci";
+		constexpr std::string_view sTextDeleted = "ci";
-		const char sTextAfterDeletion[] = "Sntilla";
+		constexpr std::string_view sTextAfterDeletion = "Sntilla";
 		bool startSequence = false;
-		const char *cpChange = cb.InsertString(0, sText, sLength, startSequence);
+		const char *cpChange = cb.InsertString(0, sText.data(), sLength, startSequence);
 		REQUIRE(startSequence);
 		REQUIRE(sLength == cb.Length());
-		REQUIRE(memcmp(cpChange, sText, sLength) == 0);
+		REQUIRE(Equal(cpChange, sText));
-		REQUIRE(memcmp(cb.BufferPointer(), sText, sLength) == 0);
+		REQUIRE(Equal(cb.BufferPointer(), sText));
 		REQUIRE(cb.CanUndo());
 		REQUIRE(!cb.CanRedo());
 		const char *cpDeletion = cb.DeleteChars(1, 2, startSequence);
 		REQUIRE(startSequence);
-		REQUIRE(memcmp(cpDeletion, sTextDeleted, strlen(sTextDeleted)) == 0);
+		REQUIRE(Equal(cpDeletion, sTextDeleted));
-		REQUIRE(memcmp(cb.BufferPointer(), sTextAfterDeletion, strlen(sTextAfterDeletion)) == 0);
+		REQUIRE(Equal(cb.BufferPointer(), sTextAfterDeletion));
 		REQUIRE(cb.CanUndo());
 		REQUIRE(!cb.CanRedo());
 
 		int steps = cb.StartUndo();
 		REQUIRE(steps == 1);
 		cb.PerformUndoStep();
-		REQUIRE(memcmp(cb.BufferPointer(), sText, sLength) == 0);
+		REQUIRE(Equal(cb.BufferPointer(), sText));
 		REQUIRE(cb.CanUndo());
 		REQUIRE(cb.CanRedo());
 
@@ -172,14 +175,14 @@ TEST_CASE("CellBuffer") {
 		steps = cb.StartRedo();
 		REQUIRE(steps == 1);
 		cb.PerformRedoStep();
-		REQUIRE(memcmp(cb.BufferPointer(), sText, sLength) == 0);
+		REQUIRE(Equal(cb.BufferPointer(), sText));
 		REQUIRE(cb.CanUndo());
 		REQUIRE(cb.CanRedo());
 
 		steps = cb.StartRedo();
 		REQUIRE(steps == 1);
 		cb.PerformRedoStep();
-		REQUIRE(memcmp(cb.BufferPointer(), sTextAfterDeletion, strlen(sTextAfterDeletion)) == 0);
+		REQUIRE(Equal(cb.BufferPointer(), sTextAfterDeletion));
 		REQUIRE(cb.CanUndo());
 		REQUIRE(!cb.CanRedo());
 
@@ -201,7 +204,7 @@ TEST_CASE("CellBuffer") {
 		cb.SetReadOnly(true);
 		REQUIRE(cb.IsReadOnly());
 		bool startSequence = false;
-		cb.InsertString(0, sText, sLength, startSequence);
+		cb.InsertString(0, sText.data(), sLength, startSequence);
 		REQUIRE(cb.Length() == 0);
 	}
 
@@ -239,8 +242,8 @@ TEST_CASE("CharacterIndex") {
 		REQUIRE(cb.IndexLineStart(0, LineCharacterIndexType::Utf32) == 0);
 		REQUIRE(cb.IndexLineStart(1, LineCharacterIndexType::Utf32) == 1);
 
-		const char *hwair = "\xF0\x90\x8D\x88";
+		constexpr std::string_view hwair = "\xF0\x90\x8D\x88";
-		cb.InsertString(0, hwair, strlen(hwair), startSequence);
+		cb.InsertString(0, hwair.data(), hwair.length(), startSequence);
 		REQUIRE(cb.IndexLineStart(0, LineCharacterIndexType::Utf16) == 0);
 		REQUIRE(cb.IndexLineStart(1, LineCharacterIndexType::Utf16) == 3);
 		REQUIRE(cb.IndexLineStart(0, LineCharacterIndexType::Utf32) == 0);
@@ -253,8 +256,8 @@ TEST_CASE("CharacterIndex") {
 		cb.AllocateLineCharacterIndex(LineCharacterIndexType::Utf16 | LineCharacterIndexType::Utf32);
 
 		bool startSequence = false;
-		const char *hwair = "a\xF0\x90\x8D\x88z";
+		constexpr std::string_view hwair = "a\xF0\x90\x8D\x88z";
-		cb.InsertString(0, hwair, strlen(hwair), startSequence);
+		cb.InsertString(0, hwair.data(), hwair.length(), startSequence);
 
 		REQUIRE(cb.IndexLineStart(0, LineCharacterIndexType::Utf16) == 0);
 		REQUIRE(cb.IndexLineStart(1, LineCharacterIndexType::Utf16) == 4);
@@ -283,8 +286,8 @@ TEST_CASE("CharacterIndex") {
 
 		bool startSequence = false;
 		// 3 lines of text containing 8 bytes
-		const char *data = "a\n\xF0\x90\x8D\x88\nz";
+		constexpr std::string_view data = "a\n\xF0\x90\x8D\x88\nz";
-		cb.InsertString(0, data, strlen(data), startSequence);
+		cb.InsertString(0, data.data(), data.length(), startSequence);
 
 		REQUIRE(cb.IndexLineStart(0, LineCharacterIndexType::Utf16) == 0);
 		REQUIRE(cb.IndexLineStart(1, LineCharacterIndexType::Utf16) == 2);
@@ -298,7 +301,7 @@ TEST_CASE("CharacterIndex") {
 
 		// Insert a new line at end -> "a\n\xF0\x90\x8D\x88\nz\n" 4 lines
 		// Last line empty
-		cb.InsertString(strlen(data), "\n", 1, startSequence);
+		cb.InsertString(data.length(), "\n", 1, startSequence);
 
 		REQUIRE(cb.IndexLineStart(0, LineCharacterIndexType::Utf16) == 0);
 		REQUIRE(cb.IndexLineStart(1, LineCharacterIndexType::Utf16) == 2);
@@ -313,7 +316,7 @@ TEST_CASE("CharacterIndex") {
 		REQUIRE(cb.IndexLineStart(4, LineCharacterIndexType::Utf32) == 6);
 
 		// Insert a new line before end -> "a\n\xF0\x90\x8D\x88\nz\n\n" 5 lines
-		cb.InsertString(strlen(data), "\n", 1, startSequence);
+		cb.InsertString(data.length(), "\n", 1, startSequence);
 
 		REQUIRE(cb.IndexLineStart(0, LineCharacterIndexType::Utf16) == 0);
 		REQUIRE(cb.IndexLineStart(1, LineCharacterIndexType::Utf16) == 2);
@@ -332,8 +335,8 @@ TEST_CASE("CharacterIndex") {
 		// Insert a valid 3-byte UTF-8 character at start ->
 		// "\xE2\x82\xACa\n\xF0\x90\x8D\x88\nz\n\n" 5 lines
 
-		const char *euro = "\xE2\x82\xAC";
+		constexpr std::string_view euro = "\xE2\x82\xAC";
-		cb.InsertString(0, euro, strlen(euro), startSequence);
+		cb.InsertString(0, euro.data(), euro.length(), startSequence);
 
 		REQUIRE(cb.IndexLineStart(0, LineCharacterIndexType::Utf16) == 0);
 		REQUIRE(cb.IndexLineStart(1, LineCharacterIndexType::Utf16) == 3);
@@ -353,8 +356,8 @@ TEST_CASE("CharacterIndex") {
 		// "\xE2\x82\xACa\n\EF\xF0\x90\x8D\x88\nz\n\n" 5 lines
 		// Should be treated as a single byte character
 
-		const char *lead = "\xEF";
+		constexpr std::string_view lead = "\xEF";
-		cb.InsertString(5, lead, strlen(lead), startSequence);
+		cb.InsertString(5, lead.data(), lead.length(), startSequence);
 
 		REQUIRE(cb.IndexLineStart(0, LineCharacterIndexType::Utf16) == 0);
 		REQUIRE(cb.IndexLineStart(1, LineCharacterIndexType::Utf16) == 3);
@@ -372,8 +375,8 @@ TEST_CASE("CharacterIndex") {
 		// byte before and the 2 bytes after also be each treated as singles
 		// so 3 more characters on line 0.
 
-		const char *ascii = "!";
+		constexpr std::string_view ascii = "!";
-		cb.InsertString(1, ascii, strlen(ascii), startSequence);
+		cb.InsertString(1, ascii.data(), ascii.length(), startSequence);
 
 		REQUIRE(cb.IndexLineStart(0, LineCharacterIndexType::Utf16) == 0);
 		REQUIRE(cb.IndexLineStart(1, LineCharacterIndexType::Utf16) == 6);
@@ -386,8 +389,8 @@ TEST_CASE("CharacterIndex") {
 		// Insert a NEL after the '!' to trigger the utf8 line end case ->
 		// "\xE2!\xC2\x85 \x82\xACa\n \EF\xF0\x90\x8D\x88\n z\n\n" 5 lines
 
-		const char *nel = "\xC2\x85";
+		constexpr std::string_view nel = "\xC2\x85";
-		cb.InsertString(2, nel, strlen(nel), startSequence);
+		cb.InsertString(2, nel.data(), nel.length(), startSequence);
 
 		REQUIRE(cb.IndexLineStart(0, LineCharacterIndexType::Utf16) == 0);
 		REQUIRE(cb.IndexLineStart(1, LineCharacterIndexType::Utf16) == 3);
@@ -406,11 +409,11 @@ TEST_CASE("CharacterIndex") {
 
 		bool startSequence = false;
 		// 3 lines of text containing 8 bytes
-		const char *data = "a\n\xF0\x90\x8D\x88\nz\nc";
+		constexpr std::string_view data = "a\n\xF0\x90\x8D\x88\nz\nc";
-		cb.InsertString(0, data, strlen(data), startSequence);
+		cb.InsertString(0, data.data(), data.length(), startSequence);
 
 		// Delete first 2 new lines -> "az\nc"
-		cb.DeleteChars(1, strlen(data) - 4, startSequence);
+		cb.DeleteChars(1, data.length() - 4, startSequence);
 
 		REQUIRE(cb.IndexLineStart(0, LineCharacterIndexType::Utf16) == 0);
 		REQUIRE(cb.IndexLineStart(1, LineCharacterIndexType::Utf16) == 3);
@@ -427,8 +430,8 @@ TEST_CASE("CharacterIndex") {
 
 		bool startSequence = false;
 		// 3 lines of text containing 8 bytes
-		const char *data = "a\n\xF0\x90\x8D\x88\nz";
+		constexpr std::string_view data = "a\n\xF0\x90\x8D\x88\nz";
-		cb.InsertString(0, data, strlen(data), startSequence);
+		cb.InsertString(0, data.data(), data.length(), startSequence);
 
 		// Delete lead byte from character on line 1 ->
 		// "a\n\x90\x8D\x88\nz"
@@ -461,8 +464,8 @@ TEST_CASE("CharacterIndex") {
 		// Restore lead byte from character on line 0 making a 4-byte character ->
 		// "a\xF0\x90\x8D\x88\nz"
 
-		const char *lead4 = "\xF0";
+		constexpr std::string_view lead4 = "\xF0";
-		cb.InsertString(1, lead4, strlen(lead4), startSequence);
+		cb.InsertString(1, lead4.data(), lead4.length(), startSequence);
 
 		REQUIRE(cb.IndexLineStart(0, LineCharacterIndexType::Utf16) == 0);
 		REQUIRE(cb.IndexLineStart(1, LineCharacterIndexType::Utf16) == 4);
@@ -479,13 +482,13 @@ TEST_CASE("CharacterIndex") {
 
 		bool startSequence = false;
 		// 2 lines of text containing 4 bytes
-		const char *data = "a\r\nb";
+		constexpr std::string_view data = "a\r\nb";
-		cb.InsertString(0, data, strlen(data), startSequence);
+		cb.InsertString(0, data.data(), data.length(), startSequence);
 
 		// 3 lines of text containing 5 bytes ->
 		// "a\r!\nb"
-		const char *ascii = "!";
+		constexpr std::string_view ascii = "!";
-		cb.InsertString(2, ascii, strlen(ascii), startSequence);
+		cb.InsertString(2, ascii.data(), ascii.length(), startSequence);
 
 		REQUIRE(cb.IndexLineStart(0, LineCharacterIndexType::Utf16) == 0);
 		REQUIRE(cb.IndexLineStart(1, LineCharacterIndexType::Utf16) == 2);
@@ -862,9 +865,9 @@ TEST_CASE("CellBufferWithChangeHistory") {
 	SECTION("StraightUndoRedoSaveRevertRedo") {
 		CellBuffer cb(true, false);
 		cb.SetUndoCollection(false);
-		std::string sInsert = "abcdefghijklmnopqrstuvwxyz";
+		constexpr std::string_view sInsert = "abcdefghijklmnopqrstuvwxyz";
 		bool startSequence = false;
-		cb.InsertString(0, sInsert.c_str(), sInsert.length(), startSequence);
+		cb.InsertString(0, sInsert.data(), sInsert.length(), startSequence);
 		cb.SetUndoCollection(true);
 		cb.SetSavePoint();
 		cb.ChangeHistorySet(true);
@@ -1001,9 +1004,9 @@ TEST_CASE("CellBufferWithChangeHistory") {
 	SECTION("Detached") {
 		CellBuffer cb(true, false);
 		cb.SetUndoCollection(false);
-		std::string sInsert = "abcdefghijklmnopqrstuvwxyz";
+		constexpr std::string_view sInsert = "abcdefghijklmnopqrstuvwxyz";
 		bool startSequence = false;
-		cb.InsertString(0, sInsert.c_str(), sInsert.length(), startSequence);
+		cb.InsertString(0, sInsert.data(), sInsert.length(), startSequence);
 		cb.SetUndoCollection(true);
 		cb.SetSavePoint();
 		cb.ChangeHistorySet(true);

scintilla/test/unit/testDocument.cxx

@@ -38,7 +38,6 @@
 using namespace Scintilla;
 using namespace Scintilla::Internal;
 
-#if !defined(_WIN32) && !defined(NO_CXX11_REGEX)
 // set global locale to pass std::regex related tests
 // see https://gcc.gnu.org/bugzilla/show_bug.cgi?id=63776
 struct GlobalLocaleInitializer {
@@ -48,7 +47,6 @@ struct GlobalLocaleInitializer {
 		} catch (...) {}
 	}
 } globalLocaleInitializer;
-#endif
 
 // Test Document.
 
@@ -92,12 +90,28 @@ std::string ReadFile(std::string path) {
 	return content;
 }
 
+struct Match {
+	Sci::Position location = 0;
+	Sci::Position length = 0;
+	constexpr Match() = default;
+	constexpr Match(Sci::Position location_, Sci::Position length_=0) : location(location_), length(length_) {
+	}
+	constexpr bool operator==(const Match &other) const {
+		return location == other.location && length == other.length;
+	}
+};
+
+std::ostream &operator << (std::ostream &os, Match const &value) {
+	os << value.location << "," << value.length;
+	return os;
+}
+
 struct DocPlus {
 	Document document;
 
 	DocPlus(std::string_view svInitial, int codePage) : document(DocumentOption::Default) {
 		SetCodePage(codePage);
-		document.InsertString(0, svInitial.data(), svInitial.length());
+		document.InsertString(0, svInitial);
 	}
 
 	void SetCodePage(int codePage) {
@@ -122,14 +136,28 @@ struct DocPlus {
 		document.SetCaseFolder(std::move(pcft));
 	}
 
-	Sci::Position FindNeedle(const std::string &needle, FindOption options, Sci::Position *length) {
+	Sci::Position FindNeedle(std::string_view needle, FindOption options, Sci::Position *length) {
 		assert(*length == static_cast<Sci::Position>(needle.length()));
-		return document.FindText(0, document.Length(), needle.c_str(), options, length);
+		return document.FindText(0, document.Length(), needle.data(), options, length);
 	}
-	Sci::Position FindNeedleReverse(const std::string &needle, FindOption options, Sci::Position *length) {
+	Sci::Position FindNeedleReverse(std::string_view needle, FindOption options, Sci::Position *length) {
 		assert(*length == static_cast<Sci::Position>(needle.length()));
-		return document.FindText(document.Length(), 0, needle.c_str(), options, length);
+		return document.FindText(document.Length(), 0, needle.data(), options, length);
 	}
+
+	Match FindString(Sci::Position minPos, Sci::Position maxPos, std::string_view needle, FindOption flags) {
+		Sci::Position lengthFinding = needle.length();
+		const Sci::Position location = document.FindText(minPos, maxPos, needle.data(), flags, &lengthFinding);
+		return { location, lengthFinding };
+	}
+
+	std::string Substitute(std::string_view substituteText) {
+		Sci::Position lengthsubstitute = substituteText.length();
+		std::string substituted = document.SubstituteByPosition(substituteText.data(), &lengthsubstitute);
+		assert(lengthsubstitute == static_cast<Sci::Position>(substituted.length()));
+		return substituted;
+	}
+
 	void MoveGap(Sci::Position gapNew) {
 		// Move gap to gapNew by inserting
 		document.InsertString(gapNew, "!", 1);
@@ -144,12 +172,14 @@ void TimeTrace(std::string_view sv, const Catch::Timer &tikka) {
 
 TEST_CASE("Document") {
 
-	const char sText[] = "Scintilla";
+	constexpr std::string_view sText = "Scintilla";
-	const Sci::Position sLength = static_cast<Sci::Position>(strlen(sText));
+	constexpr Sci::Position sLength = sText.length();
+	constexpr FindOption rePosix = FindOption::RegExp | FindOption::Posix;
+	constexpr FindOption reCxx11 = FindOption::RegExp | FindOption::Cxx11RegEx;
 
 	SECTION("InsertOneLine") {
 		DocPlus doc("", 0);
-		const Sci::Position length = doc.document.InsertString(0, sText, sLength);
+		const Sci::Position length = doc.document.InsertString(0, sText);
 		REQUIRE(sLength == doc.document.Length());
 		REQUIRE(length == sLength);
 		REQUIRE(1 == doc.document.LinesTotal());
@@ -167,42 +197,42 @@ TEST_CASE("Document") {
 	// part way through a character.
 	SECTION("SearchInLatin") {
 		DocPlus doc("abcde", 0);	// a b c d e
-		std::string finding = "b";
+		constexpr std::string_view finding = "b";
 		Sci::Position lengthFinding = finding.length();
 		Sci::Position location = doc.FindNeedle(finding, FindOption::MatchCase, &lengthFinding);
 		REQUIRE(location == 1);
 		location = doc.FindNeedleReverse(finding, FindOption::MatchCase, &lengthFinding);
 		REQUIRE(location == 1);
-		location = doc.document.FindText(0, 2, finding.c_str(), FindOption::MatchCase, &lengthFinding);
+		location = doc.document.FindText(0, 2, finding.data(), FindOption::MatchCase, &lengthFinding);
 		REQUIRE(location == 1);
-		location = doc.document.FindText(0, 1, finding.c_str(), FindOption::MatchCase, &lengthFinding);
+		location = doc.document.FindText(0, 1, finding.data(), FindOption::MatchCase, &lengthFinding);
 		REQUIRE(location == -1);
 	}
 
 	SECTION("SearchInBothSegments") {
 		DocPlus doc("ab-ab", 0);	// a b - a b
-		std::string finding = "ab";
+		constexpr std::string_view finding = "ab";
 		for (int gapPos = 0; gapPos <= 5; gapPos++) {
 			doc.MoveGap(gapPos);
 			Sci::Position lengthFinding = finding.length();
-			Sci::Position location = doc.document.FindText(0, doc.document.Length(), finding.c_str(), FindOption::MatchCase, &lengthFinding);
+			Sci::Position location = doc.document.FindText(0, doc.document.Length(), finding.data(), FindOption::MatchCase, &lengthFinding);
 			REQUIRE(location == 0);
-			location = doc.document.FindText(2, doc.document.Length(), finding.c_str(), FindOption::MatchCase, &lengthFinding);
+			location = doc.document.FindText(2, doc.document.Length(), finding.data(), FindOption::MatchCase, &lengthFinding);
 			REQUIRE(location == 3);
 		}
 	}
 
 	SECTION("InsensitiveSearchInLatin") {
 		DocPlus doc("abcde", 0);	// a b c d e
-		std::string finding = "B";
+		constexpr std::string_view finding = "B";
 		Sci::Position lengthFinding = finding.length();
 		Sci::Position location = doc.FindNeedle(finding, FindOption::None, &lengthFinding);
 		REQUIRE(location == 1);
 		location = doc.FindNeedleReverse(finding, FindOption::None, &lengthFinding);
 		REQUIRE(location == 1);
-		location = doc.document.FindText(0, 2, finding.c_str(), FindOption::None, &lengthFinding);
+		location = doc.document.FindText(0, 2, finding.data(), FindOption::None, &lengthFinding);
 		REQUIRE(location == 1);
-		location = doc.document.FindText(0, 1, finding.c_str(), FindOption::None, &lengthFinding);
+		location = doc.document.FindText(0, 1, finding.data(), FindOption::None, &lengthFinding);
 		REQUIRE(location == -1);
 	}
 
@@ -212,11 +242,11 @@ TEST_CASE("Document") {
 		doc.SetSBCSFoldings(foldings1252, std::size(foldings1252));
 
 		// Search for upper-case AE
-		std::string finding = "\xc6";
+		std::string_view finding = "\xc6";
 		Sci::Position lengthFinding = finding.length();
 		Sci::Position location = doc.FindNeedle(finding, FindOption::None, &lengthFinding);
 		REQUIRE(location == 3);
-		location = doc.document.FindText(4, doc.document.Length(), finding.c_str(), FindOption::None, &lengthFinding);
+		location = doc.document.FindText(4, doc.document.Length(), finding.data(), FindOption::None, &lengthFinding);
 		REQUIRE(location == 5);
 		location = doc.FindNeedleReverse(finding, FindOption::None, &lengthFinding);
 		REQUIRE(location == 5);
@@ -225,7 +255,7 @@ TEST_CASE("Document") {
 		finding = "\xe6";
 		location = doc.FindNeedle(finding, FindOption::None, &lengthFinding);
 		REQUIRE(location == 3);
-		location = doc.document.FindText(4, doc.document.Length(), finding.c_str(), FindOption::None, &lengthFinding);
+		location = doc.document.FindText(4, doc.document.Length(), finding.data(), FindOption::None, &lengthFinding);
 		REQUIRE(location == 5);
 		location = doc.FindNeedleReverse(finding, FindOption::None, &lengthFinding);
 		REQUIRE(location == 5);
@@ -234,12 +264,12 @@ TEST_CASE("Document") {
 	SECTION("Search2InLatin") {
 		// Checks that the initial '_' and final 'f' are ignored since they are outside the search bounds
 		DocPlus doc("_abcdef", 0);	// _ a b c d e f
-		std::string finding = "cd";
+		constexpr std::string_view finding = "cd";
 		Sci::Position lengthFinding = finding.length();
 		size_t docLength = doc.document.Length() - 1;
-		Sci::Position location = doc.document.FindText(1, docLength, finding.c_str(), FindOption::MatchCase, &lengthFinding);
+		Sci::Position location = doc.document.FindText(1, docLength, finding.data(), FindOption::MatchCase, &lengthFinding);
 		REQUIRE(location == 3);
-		location = doc.document.FindText(docLength, 1, finding.c_str(), FindOption::MatchCase, &lengthFinding);
+		location = doc.document.FindText(docLength, 1, finding.data(), FindOption::MatchCase, &lengthFinding);
 		REQUIRE(location == 3);
 		location = doc.document.FindText(docLength, 1, "bc", FindOption::MatchCase, &lengthFinding);
 		REQUIRE(location == 2);
@@ -258,46 +288,46 @@ TEST_CASE("Document") {
 
 	SECTION("SearchInUTF8") {
 		DocPlus doc("ab\xCE\x93" "d", CpUtf8);	// a b gamma d
-		const std::string finding = "b";
+		constexpr std::string_view finding = "b";
 		Sci::Position lengthFinding = finding.length();
 		Sci::Position location = doc.FindNeedle(finding, FindOption::MatchCase, &lengthFinding);
 		REQUIRE(location == 1);
-		location = doc.document.FindText(doc.document.Length(), 0, finding.c_str(), FindOption::MatchCase, &lengthFinding);
+		location = doc.document.FindText(doc.document.Length(), 0, finding.data(), FindOption::MatchCase, &lengthFinding);
 		REQUIRE(location == 1);
-		location = doc.document.FindText(0, 1, finding.c_str(), FindOption::MatchCase, &lengthFinding);
+		location = doc.document.FindText(0, 1, finding.data(), FindOption::MatchCase, &lengthFinding);
 		REQUIRE(location == -1);
 		// Check doesn't try to follow a lead-byte past the search end
-		const std::string findingUTF = "\xCE\x93";
+		constexpr std::string_view findingUTF = "\xCE\x93";
 		lengthFinding = findingUTF.length();
-		location = doc.document.FindText(0, 4, findingUTF.c_str(), FindOption::MatchCase, &lengthFinding);
+		location = doc.document.FindText(0, 4, findingUTF.data(), FindOption::MatchCase, &lengthFinding);
 		REQUIRE(location == 2);
 		// Only succeeds as 3 is partway through character so adjusted to 4
-		location = doc.document.FindText(0, 3, findingUTF.c_str(), FindOption::MatchCase, &lengthFinding);
+		location = doc.document.FindText(0, 3, findingUTF.data(), FindOption::MatchCase, &lengthFinding);
 		REQUIRE(location == 2);
-		location = doc.document.FindText(0, 2, findingUTF.c_str(), FindOption::MatchCase, &lengthFinding);
+		location = doc.document.FindText(0, 2, findingUTF.data(), FindOption::MatchCase, &lengthFinding);
 		REQUIRE(location == -1);
 	}
 
 	SECTION("InsensitiveSearchInUTF8") {
 		DocPlus doc("ab\xCE\x93" "d", CpUtf8);	// a b gamma d
-		const std::string finding = "b";
+		constexpr std::string_view finding = "b";
 		Sci::Position lengthFinding = finding.length();
 		Sci::Position location = doc.FindNeedle(finding, FindOption::None, &lengthFinding);
 		REQUIRE(location == 1);
-		location = doc.document.FindText(doc.document.Length(), 0, finding.c_str(), FindOption::None, &lengthFinding);
+		location = doc.document.FindText(doc.document.Length(), 0, finding.data(), FindOption::None, &lengthFinding);
 		REQUIRE(location == 1);
-		const std::string findingUTF = "\xCE\x93";
+		constexpr std::string_view findingUTF = "\xCE\x93";
 		lengthFinding = findingUTF.length();
 		location = doc.FindNeedle(findingUTF, FindOption::None, &lengthFinding);
 		REQUIRE(location == 2);
-		location = doc.document.FindText(doc.document.Length(), 0, findingUTF.c_str(), FindOption::None, &lengthFinding);
+		location = doc.document.FindText(doc.document.Length(), 0, findingUTF.data(), FindOption::None, &lengthFinding);
 		REQUIRE(location == 2);
-		location = doc.document.FindText(0, 4, findingUTF.c_str(), FindOption::None, &lengthFinding);
+		location = doc.document.FindText(0, 4, findingUTF.data(), FindOption::None, &lengthFinding);
 		REQUIRE(location == 2);
 		// Only succeeds as 3 is partway through character so adjusted to 4
-		location = doc.document.FindText(0, 3, findingUTF.c_str(), FindOption::None, &lengthFinding);
+		location = doc.document.FindText(0, 3, findingUTF.data(), FindOption::None, &lengthFinding);
 		REQUIRE(location == 2);
-		location = doc.document.FindText(0, 2, findingUTF.c_str(), FindOption::None, &lengthFinding);
+		location = doc.document.FindText(0, 2, findingUTF.data(), FindOption::None, &lengthFinding);
 		REQUIRE(location == -1);
 	}
 
@@ -306,14 +336,14 @@ TEST_CASE("Document") {
 		// The 'b' can be incorrectly matched by the search string 'b' when the search
 		// does not iterate the text correctly.
 		DocPlus doc("ab\xe9" "b ", 932);	// a b {CJK UNIFIED IDEOGRAPH-9955} {space}
-		std::string finding = "b";
+		constexpr std::string_view finding = "b";
 		// Search forwards
 		Sci::Position lengthFinding = finding.length();
 		Sci::Position location = doc.FindNeedle(finding, FindOption::MatchCase, &lengthFinding);
 		REQUIRE(location == 1);
 		// Search backwards
 		lengthFinding = finding.length();
-		location = doc.document.FindText(doc.document.Length(), 0, finding.c_str(), FindOption::MatchCase, &lengthFinding);
+		location = doc.document.FindText(doc.document.Length(), 0, finding.data(), FindOption::MatchCase, &lengthFinding);
 		REQUIRE(location == 1);
 	}
 
@@ -322,27 +352,27 @@ TEST_CASE("Document") {
 		// The 'b' can be incorrectly matched by the search string 'b' when the search
 		// does not iterate the text correctly.
 		DocPlus doc("ab\xe9" "b ", 932);	// a b {CJK UNIFIED IDEOGRAPH-9955} {space}
-		std::string finding = "b";
+		constexpr std::string_view finding = "b";
 		// Search forwards
 		Sci::Position lengthFinding = finding.length();
 		Sci::Position location = doc.FindNeedle(finding, FindOption::None, &lengthFinding);
 		REQUIRE(location == 1);
 		// Search backwards
 		lengthFinding = finding.length();
-		location = doc.document.FindText(doc.document.Length(), 0, finding.c_str(), FindOption::None, &lengthFinding);
+		location = doc.document.FindText(doc.document.Length(), 0, finding.data(), FindOption::None, &lengthFinding);
 		REQUIRE(location == 1);
-		std::string finding932 = "\xe9" "b";
+		constexpr std::string_view finding932 = "\xe9" "b";
 		// Search forwards
 		lengthFinding = finding932.length();
 		location = doc.FindNeedle(finding932, FindOption::None, &lengthFinding);
 		REQUIRE(location == 2);
 		// Search backwards
 		lengthFinding = finding932.length();
-		location = doc.document.FindText(doc.document.Length(), 0, finding932.c_str(), FindOption::None, &lengthFinding);
+		location = doc.document.FindText(doc.document.Length(), 0, finding932.data(), FindOption::None, &lengthFinding);
 		REQUIRE(location == 2);
-		location = doc.document.FindText(0, 3, finding932.c_str(), FindOption::None, &lengthFinding);
+		location = doc.document.FindText(0, 3, finding932.data(), FindOption::None, &lengthFinding);
 		REQUIRE(location == 2);
-		location = doc.document.FindText(0, 2, finding932.c_str(), FindOption::None, &lengthFinding);
+		location = doc.document.FindText(0, 2, finding932.data(), FindOption::None, &lengthFinding);
 		REQUIRE(location == -1);
 		// Can not test case mapping of double byte text as folder available here does not implement this
 	}
@@ -427,8 +457,8 @@ TEST_CASE("Document") {
 		// O p e n = U+958B Ku ( O ) U+7DE8 -
 		// U+958B open
 		// U+7DE8 arrange
-		const std::string japaneseText = "Open=\x8aJ\x82\xad(O)\x95\xd2-";
+		constexpr std::string_view japaneseText = "Open=\x8aJ\x82\xad(O)\x95\xd2-";
-		const Sci::Position length = doc.InsertString(0, japaneseText.c_str(), japaneseText.length());
+		const Sci::Position length = doc.InsertString(0, japaneseText);
 		REQUIRE(length == 15);
 		// Forwards
 		REQUIRE(doc.NextPosition( 0, 1) == 1);
@@ -468,50 +498,229 @@ TEST_CASE("Document") {
 
 	SECTION("RegexSearchAndSubstitution") {
 		DocPlus doc("\n\r\r\n 1a\xCE\x93z \n\r\r\n 2b\xCE\x93y \n\r\r\n", CpUtf8);// 1a gamma z 2b gamma y
-		const std::string finding = R"(\d+(\w+))";
+		const Sci::Position docLength = doc.document.Length();
-		Sci::Position lengthFinding = finding.length();
+		Match match;
-		Sci::Position location = doc.FindNeedle(finding, FindOption::RegExp | FindOption::Posix, &lengthFinding);
-		REQUIRE(location == 5);
-		REQUIRE(lengthFinding == 5);
 
-		const std::string_view substituteText = R"(\t\1\n)";
+		constexpr std::string_view finding = R"(\d+(\w+))";
-		Sci::Position lengthsubstitute = substituteText.length();
+		constexpr std::string_view substituteText = R"(\t\1\n)";
-		std::string substituted = doc.document.SubstituteByPosition(substituteText.data(), &lengthsubstitute);
+		constexpr std::string_view longest = "\\w+";
-		REQUIRE(lengthsubstitute == 6);
+		std::string substituted;
+
+		match = doc.FindString(0, docLength, finding, rePosix);
+		REQUIRE(match == Match(5, 5));
+		substituted = doc.Substitute(substituteText);
 		REQUIRE(substituted == "\ta\xCE\x93z\n");
 
-		lengthFinding = finding.length();
+		match = doc.FindString(docLength, 0, finding, rePosix);
-		location = doc.FindNeedleReverse(finding, FindOption::RegExp | FindOption::Posix, &lengthFinding);
+		REQUIRE(match == Match(16, 5));
-		REQUIRE(location == 16);
+		substituted = doc.Substitute(substituteText);
-		REQUIRE(lengthFinding == 5);
-
-		lengthsubstitute = substituteText.length();
-		substituted = doc.document.SubstituteByPosition(substituteText.data(), &lengthsubstitute);
-		REQUIRE(lengthsubstitute == 6);
 		REQUIRE(substituted == "\tb\xCE\x93y\n");
 
+		match = doc.FindString(docLength, 0, longest, rePosix);
+		REQUIRE(match == Match(16, 5));
+
 		#ifndef NO_CXX11_REGEX
-		lengthFinding = finding.length();
+		match = doc.FindString(0, docLength, finding, reCxx11);
-		location = doc.FindNeedle(finding, FindOption::RegExp | FindOption::Cxx11RegEx, &lengthFinding);
+		REQUIRE(match == Match(5, 5));
-		REQUIRE(location == 5);
+		substituted = doc.Substitute(substituteText);
-		REQUIRE(lengthFinding == 5);
-
-		lengthsubstitute = substituteText.length();
-		substituted = doc.document.SubstituteByPosition(substituteText.data(), &lengthsubstitute);
-		REQUIRE(lengthsubstitute == 6);
 		REQUIRE(substituted == "\ta\xCE\x93z\n");
 
-		lengthFinding = finding.length();
+		match = doc.FindString(docLength, 0, finding, reCxx11);
-		location = doc.FindNeedleReverse(finding, FindOption::RegExp | FindOption::Cxx11RegEx, &lengthFinding);
+		REQUIRE(match == Match(16, 5));
-		REQUIRE(location == 16);
+		substituted = doc.Substitute(substituteText);
-		REQUIRE(lengthFinding == 5);
-
-		lengthsubstitute = substituteText.length();
-		substituted = doc.document.SubstituteByPosition(substituteText.data(), &lengthsubstitute);
-		REQUIRE(lengthsubstitute == 6);
 		REQUIRE(substituted == "\tb\xCE\x93y\n");
+
+		match = doc.FindString(docLength, 0, longest, reCxx11);
+		REQUIRE(match == Match(16, 5));
 		#endif
 	}
+
+	SECTION("RegexAssertion") {
+		DocPlus doc("ab cd ef\r\ngh ij kl", CpUtf8);
+		const Sci::Position docLength = doc.document.Length();
+		Match match;
+
+		constexpr std::string_view findingBOL = "^";
+		match = doc.FindString(0, docLength, findingBOL, rePosix);
+		REQUIRE(match == Match(0));
+		match = doc.FindString(1, docLength, findingBOL, rePosix);
+		REQUIRE(match == Match(10));
+		match = doc.FindString(docLength, 0, findingBOL, rePosix);
+		REQUIRE(match == Match(10));
+		match = doc.FindString(docLength - 1, 0, findingBOL, rePosix);
+		REQUIRE(match == Match(10));
+
+		#ifndef NO_CXX11_REGEX
+		match = doc.FindString(0, docLength, findingBOL, reCxx11);
+		REQUIRE(match == Match(0));
+		match = doc.FindString(1, docLength, findingBOL, reCxx11);
+		REQUIRE(match == Match(10));
+		match = doc.FindString(docLength, 0, findingBOL, reCxx11);
+		REQUIRE(match == Match(10));
+		match = doc.FindString(docLength - 1, 0, findingBOL, reCxx11);
+		REQUIRE(match == Match(10));
+		#endif
+
+		constexpr std::string_view findingEOL = "$";
+		match = doc.FindString(0, docLength, findingEOL, rePosix);
+		REQUIRE(match == Match(8));
+		match = doc.FindString(1, docLength, findingEOL, rePosix);
+		REQUIRE(match == Match(8));
+		match = doc.FindString(docLength, 0, findingEOL, rePosix);
+		REQUIRE(match == Match(18));
+		match = doc.FindString(docLength - 1, 0, findingEOL, rePosix);
+		REQUIRE(match == Match(8));
+
+		#if !defined(NO_CXX11_REGEX) && !defined(_LIBCPP_VERSION)
+		match = doc.FindString(0, docLength, findingEOL, reCxx11);
+		REQUIRE(match == Match(8));
+		match = doc.FindString(1, docLength, findingEOL, reCxx11);
+		REQUIRE(match == Match(8));
+		match = doc.FindString(docLength, 0, findingEOL, reCxx11);
+		REQUIRE(match == Match(18));
+		match = doc.FindString(docLength - 1, 0, findingEOL, reCxx11);
+		REQUIRE(match == Match(8));
+		#endif
+
+		constexpr std::string_view findingBOW = "\\<";
+		match = doc.FindString(0, docLength, findingBOW, rePosix);
+		REQUIRE(match == Match(0));
+		match = doc.FindString(1, docLength, findingBOW, rePosix);
+		REQUIRE(match == Match(3));
+		match = doc.FindString(docLength, 0, findingBOW, rePosix);
+		REQUIRE(match == Match(16));
+		match = doc.FindString(docLength - 1, 0, findingBOW, rePosix);
+		REQUIRE(match == Match(16));
+
+		constexpr std::string_view findingEOW = "\\>";
+		match = doc.FindString(0, docLength, findingEOW, rePosix);
+		REQUIRE(match == Match(2));
+		match = doc.FindString(1, docLength, findingEOW, rePosix);
+		REQUIRE(match == Match(2));
+		match = doc.FindString(docLength, 0, findingEOW, rePosix);
+		REQUIRE(match == Match(18));
+		match = doc.FindString(docLength - 1, 0, findingEOW, rePosix);
+		REQUIRE(match == Match(15));
+
+		constexpr std::string_view findingEOWEOL = "\\>$";
+		match = doc.FindString(0, docLength, findingEOWEOL, rePosix);
+		REQUIRE(match == Match(8));
+		match = doc.FindString(10, docLength, findingEOWEOL, rePosix);
+		REQUIRE(match == Match(18));
+
+		#ifndef NO_CXX11_REGEX
+		constexpr std::string_view findingWB = "\\b";
+		match = doc.FindString(0, docLength, findingWB, reCxx11);
+		REQUIRE(match == Match(0));
+		match = doc.FindString(1, docLength, findingWB, reCxx11);
+		REQUIRE(match == Match(2));
+		match = doc.FindString(docLength, 0, findingWB, reCxx11);
+		#ifdef _LIBCPP_VERSION
+		REQUIRE(match == Match(16));
+		#else
+		REQUIRE(match == Match(18));
+		#endif
+		match = doc.FindString(docLength - 1, 0, findingWB, reCxx11);
+		REQUIRE(match == Match(16));
+
+		constexpr std::string_view findingNWB = "\\B";
+		match = doc.FindString(0, docLength, findingNWB, reCxx11);
+		REQUIRE(match == Match(1));
+		match = doc.FindString(1, docLength, findingNWB, reCxx11);
+		REQUIRE(match == Match(1));
+		#ifdef _LIBCPP_VERSION
+		match = doc.FindString(docLength, 0, findingNWB, reCxx11);
+		REQUIRE(match == Match(18));
+		match = doc.FindString(docLength - 1, 0, findingNWB, reCxx11);
+		REQUIRE(match == Match(14));
+		#else
+		match = doc.FindString(docLength, 0, findingNWB, reCxx11);
+		REQUIRE(match == Match(17));
+		match = doc.FindString(docLength - 1, 0, findingNWB, reCxx11);
+		REQUIRE(match == Match(17));
+		#endif
+		#endif
+	}
+
+	SECTION("RegexContextualAssertion") {
+		// For std::regex, check the use of assertions next to text in forward direction
+		// These are more common than empty assertions
+		DocPlus doc("ab cd ef\r\ngh ij kl", CpUtf8);
+		const Sci::Position docLength = doc.document.Length();
+		Match match;
+
+		#ifndef NO_CXX11_REGEX
+
+		match = doc.FindString(0, docLength, "^[a-z]", reCxx11);
+		REQUIRE(match == Match(0, 1));
+		match = doc.FindString(1, docLength, "^[a-z]", reCxx11);
+		REQUIRE(match == Match(10, 1));
+
+		match = doc.FindString(0, docLength, "[a-z]$", reCxx11);
+		REQUIRE(match == Match(7, 1));
+		match = doc.FindString(10, docLength, "[a-z]$", reCxx11);
+		REQUIRE(match == Match(17, 1));
+
+		match = doc.FindString(0, docLength, "\\b[a-z]", reCxx11);
+		REQUIRE(match == Match(0, 1));
+		match = doc.FindString(1, docLength, "\\b[a-z]", reCxx11);
+		REQUIRE(match == Match(3, 1));
+		match = doc.FindString(0, docLength, "[a-z]\\b", reCxx11);
+		REQUIRE(match == Match(1, 1));
+		match = doc.FindString(2, docLength, "[a-z]\\b", reCxx11);
+		REQUIRE(match == Match(4, 1));
+
+		match = doc.FindString(0, docLength, "\\B[a-z]", reCxx11);
+		REQUIRE(match == Match(1, 1));
+		match = doc.FindString(1, docLength, "\\B[a-z]", reCxx11);
+		REQUIRE(match == Match(1, 1));
+		match = doc.FindString(0, docLength, "[a-z]\\B", reCxx11);
+		REQUIRE(match == Match(0, 1));
+		match = doc.FindString(2, docLength, "[a-z]\\B", reCxx11);
+		REQUIRE(match == Match(3, 1));
+
+		#endif
+	}
+
+	SECTION("RESearchMovePositionOutsideCharUTF8") {
+		DocPlus doc(" a\xCE\x93\xCE\x93z ", CpUtf8);// a gamma gamma z
+		const Sci::Position docLength = doc.document.Length();
+		constexpr std::string_view finding = R"([a-z](\w)\1)";
+
+		Match match = doc.FindString(0, docLength, finding, rePosix);
+		REQUIRE(match == Match(1, 5));
+
+		constexpr std::string_view substituteText = R"(\t\1\n)";
+		std::string substituted = doc.Substitute(substituteText);
+		REQUIRE(substituted == "\t\xCE\x93\n");
+
+		#ifndef NO_CXX11_REGEX
+		match = doc.FindString(0, docLength, finding, reCxx11);
+		REQUIRE(match == Match(1, 5));
+
+		substituted = doc.Substitute(substituteText);
+		REQUIRE(substituted == "\t\xCE\x93\n");
+		#endif
+	}
+
+	SECTION("RESearchMovePositionOutsideCharDBCS") {
+		DocPlus doc(" \x98\x61xx 1aa\x83\xA1\x83\xA1z ", 932);// U+548C xx 1aa gamma gamma z
+		const Sci::Position docLength = doc.document.Length();
+
+		Match match = doc.FindString(0, docLength, R"([a-z](\w)\1)", rePosix);
+		REQUIRE(match == Match(8, 5));
+
+		constexpr std::string_view substituteText = R"(\t\1\n)";
+		std::string substituted = doc.Substitute(substituteText);
+		REQUIRE(substituted == "\t\x83\xA1\n");
+
+		match = doc.FindString(0, docLength, R"(\w([a-z])\1)", rePosix);
+		REQUIRE(match == Match(6, 3));
+
+		substituted = doc.Substitute(substituteText);
+		REQUIRE(substituted == "\ta\n");
+	}
+
 }
 
 TEST_CASE("Words") {
@@ -549,7 +758,7 @@ TEST_CASE("SafeSegment") {
 	SECTION("Short") {
 		const DocPlus doc("", 0);
 		// all encoding: break before or after last space
-		const std::string_view text = "12 ";
+		constexpr std::string_view text = "12 ";
 		size_t length = doc.document.SafeSegment(text);
 		REQUIRE(length <= text.length());
 		REQUIRE(text[length - 1] == '2');

scintilla/test/unit/testRESearch.cxx

@@ -41,6 +41,9 @@ public:
 	char CharAt(Sci::Position index) const override {
 		return s.at(index);
 	}
+	Sci::Position MovePositionOutsideChar(Sci::Position pos, [[maybe_unused]] Sci::Position moveDir) const noexcept override {
+		return pos;
+	}
 	std::string GetCharRange(Sci::Position position, Sci::Position lengthRetrieve) const {
 		return s.substr(position, lengthRetrieve);
 	}
@@ -60,6 +63,17 @@ TEST_CASE("RESearch") {
 		REQUIRE(nullptr == msg);
 	}
 
+	SECTION("Bug2413") {
+		// Check for https://sourceforge.net/p/scintilla/bugs/2413/
+		std::unique_ptr<RESearch> re = std::make_unique<RESearch>(&cc);
+		constexpr std::string_view BOW = "\\<";
+		constexpr std::string_view EOW = "\\>";
+		const char *msg = re->Compile(BOW.data(), BOW.length(), true, false);
+		REQUIRE(nullptr == msg);
+		msg = re->Compile(EOW.data(), EOW.length(), true, false);
+		REQUIRE(nullptr == msg);
+	}
+
 	SECTION("Execute") {
 		std::unique_ptr<RESearch> re = std::make_unique<RESearch>(&cc);
 		re->Compile(pattern.data(), pattern.length(), true, false);

scintilla/test/unit/testUniConversion.cxx

@@ -255,8 +255,8 @@ TEST_CASE("UniConversion") {
 namespace {
 
 // Simple adapter to avoid casting
-int UTFClass(const char *s) noexcept {
+int UTFClass(std::string_view sv) noexcept {
-	return UTF8Classify(reinterpret_cast<const unsigned char *>(s), static_cast<int>(strlen(s)));
+	return UTF8Classify(sv);
 }
 
 }

scintilla/test/win32Tests.py

@@ -43,13 +43,13 @@ class TestWins(unittest.TestCase):
 		return self.Send("WM_GETTEXT", n, s)
 
 	def TextValue(self):
-		self.assertEquals(self.ed.GetStatus(), 0)
+		self.assertEqual(self.ed.GetStatus(), 0)
 		lenValue = self.GetTextLength()
 		lenValueWithNUL = lenValue + 1
 		value = ctypes.create_unicode_buffer(lenValueWithNUL)
 		lenData = self.GetText(lenValueWithNUL, value)
-		self.assertEquals(self.ed.GetStatus(), 0)
+		self.assertEqual(self.ed.GetStatus(), 0)
-		self.assertEquals(lenData, lenValue)
+		self.assertEqual(lenData, lenValue)
 		return value.value
 
 	def SetText(self, s):
@@ -59,19 +59,19 @@ class TestWins(unittest.TestCase):
 
 	def testSetText(self):
 		self.SetText(b"ab")
-		self.assertEquals(self.ed.Length, 2)
+		self.assertEqual(self.ed.Length, 2)
 
 	def testGetTextLength(self):
 		self.SetText(b"ab")
-		self.assertEquals(self.GetTextLength(), 2)
+		self.assertEqual(self.GetTextLength(), 2)
 
 	def testGetText(self):
 		self.SetText(b"ab")
 		data = ctypes.create_unicode_buffer(100)
 		lenData = self.GetText(100, data)
-		self.assertEquals(lenData, 2)
+		self.assertEqual(lenData, 2)
-		self.assertEquals(len(data.value), 2)
+		self.assertEqual(len(data.value), 2)
-		self.assertEquals(data.value, "ab")
+		self.assertEqual(data.value, "ab")
 
 	def testGetUTF8Text(self):
 		self.ed.SetCodePage(65001)
@@ -79,7 +79,7 @@ class TestWins(unittest.TestCase):
 		tu8 = t.encode("UTF-8")
 		self.SetText(tu8)
 		value = self.TextValue()
-		self.assertEquals(value, t)
+		self.assertEqual(value, t)
 
 	def testGetBadUTF8Text(self):
 		self.ed.SetCodePage(65001)
@@ -87,8 +87,8 @@ class TestWins(unittest.TestCase):
 		t = "t\xc2"
 		self.SetText(tu8)
 		value = self.TextValue()
-		self.assertEquals(len(value), 2)
+		self.assertEqual(len(value), 2)
-		self.assertEquals(value, t)
+		self.assertEqual(value, t)
 
 	def testGetJISText(self):
 		self.ed.SetCodePage(932)
@@ -96,8 +96,8 @@ class TestWins(unittest.TestCase):
 		tu8 = t.encode("shift-jis")
 		self.SetText(tu8)
 		value = self.TextValue()
-		self.assertEquals(len(value), 1)
+		self.assertEqual(len(value), 1)
-		self.assertEquals(value, t)
+		self.assertEqual(value, t)
 
 	def testGetBadJISText(self):
 		self.ed.SetCodePage(932)
@@ -110,24 +110,24 @@ class TestWins(unittest.TestCase):
 		privateBad = '[\uf8f3]'
 		self.SetText(tu8)
 		value = self.TextValue()
-		self.assertEquals(len(value), 3)
+		self.assertEqual(len(value), 3)
-		self.assertEquals(value, katakanaMiddleDot)
+		self.assertEqual(value, katakanaMiddleDot)
 
 		# This is even less valid Shift-JIS
 		tu8 = b'[\xff]'
 		self.SetText(tu8)
 		value = self.TextValue()
-		self.assertEquals(len(value), 3)
+		self.assertEqual(len(value), 3)
-		self.assertEquals(value, privateBad)
+		self.assertEqual(value, privateBad)
 
 	def testGetTextLong(self):
-		self.assertEquals(self.ed.GetStatus(), 0)
+		self.assertEqual(self.ed.GetStatus(), 0)
 		self.SetText(b"ab")
 		data = ctypes.create_unicode_buffer(100)
 		lenData = self.GetText(4, data)
-		self.assertEquals(self.ed.GetStatus(), 0)
+		self.assertEqual(self.ed.GetStatus(), 0)
-		self.assertEquals(lenData, 2)
+		self.assertEqual(lenData, 2)
-		self.assertEquals(data.value, "ab")
+		self.assertEqual(data.value, "ab")
 
 	def testGetTextLongNonASCII(self):
 		# With 1 multibyte character in document ask for 4 and ensure 1 character
@@ -138,38 +138,38 @@ class TestWins(unittest.TestCase):
 		self.SetText(tu8)
 		data = ctypes.create_unicode_buffer(100)
 		lenData = self.GetText(4, data)
-		self.assertEquals(self.ed.GetStatus(), 0)
+		self.assertEqual(self.ed.GetStatus(), 0)
-		self.assertEquals(lenData, 1)
+		self.assertEqual(lenData, 1)
-		self.assertEquals(data.value, t)
+		self.assertEqual(data.value, t)
 
 	def testGetTextShort(self):
-		self.assertEquals(self.ed.GetStatus(), 0)
+		self.assertEqual(self.ed.GetStatus(), 0)
 		self.SetText(b"ab")
 		data = ctypes.create_unicode_buffer(100)
 		lenData = self.GetText(2, data)
-		self.assertEquals(self.ed.GetStatus(), 0)
+		self.assertEqual(self.ed.GetStatus(), 0)
-		self.assertEquals(lenData, 1)
+		self.assertEqual(lenData, 1)
-		self.assertEquals(data.value, "a")
+		self.assertEqual(data.value, "a")
 
 	def testGetTextJustNUL(self):
-		self.assertEquals(self.ed.GetStatus(), 0)
+		self.assertEqual(self.ed.GetStatus(), 0)
 		self.SetText(b"ab")
 		data = ctypes.create_unicode_buffer(100)
 		lenData = self.GetText(1, data)
-		self.assertEquals(self.ed.GetStatus(), 0)
+		self.assertEqual(self.ed.GetStatus(), 0)
 		#~ print(data)
-		self.assertEquals(lenData, 0)
+		self.assertEqual(lenData, 0)
-		self.assertEquals(data.value, "")
+		self.assertEqual(data.value, "")
 
 	def testGetTextZeroLength(self):
-		self.assertEquals(self.ed.GetStatus(), 0)
+		self.assertEqual(self.ed.GetStatus(), 0)
 		self.SetText(b"ab")
 		data = ctypes.create_unicode_buffer(100)
 		lenData = self.GetText(0, data)
-		self.assertEquals(self.ed.GetStatus(), 0)
+		self.assertEqual(self.ed.GetStatus(), 0)
 		#~ print(data)
-		self.assertEquals(lenData, 0)
+		self.assertEqual(lenData, 0)
-		self.assertEquals(data.value, "")
+		self.assertEqual(data.value, "")
 
 if __name__ == '__main__':
 	uu = Xite.main("win32Tests")

scintilla/version.txt

@@ -1 +1 @@
-538
+541

scintilla/win32/ScintRes.rc

@@ -4,8 +4,8 @@
 
 #include <windows.h>
 
-#define VERSION_SCINTILLA "5.3.8"
+#define VERSION_SCINTILLA "5.4.1"
-#define VERSION_WORDS 5, 3, 8, 0
+#define VERSION_WORDS 5, 4, 1, 0
 
 VS_VERSION_INFO VERSIONINFO
 FILEVERSION	VERSION_WORDS

scintilla/win32/ScintillaWin.cxx

@@ -3196,7 +3196,7 @@ LRESULT ScintillaWin::ImeOnReconvert(LPARAM lParam) {
 		} else {
 			// Ensure docCompStart+docCompLen be not beyond lineEnd.
 			// since docCompLen by byte might break eol.
-			const Sci::Position lineEnd = pdoc->LineEnd(pdoc->LineFromPosition(rBase));
+			const Sci::Position lineEnd = pdoc->LineEndPosition(rBase);
 			const Sci::Position overflow = (docCompStart + docCompLen) - lineEnd;
 			if (overflow > 0) {
 				pdoc->DeleteChars(docCompStart, docCompLen - overflow);