ShellPkg: UefiShellDebug1CommandsLib: fix hex string parsing in SETVAR

The ShellCommandRunSetVar() function calls the ShellIsHexOrDecimalNumber()
UefiShellLib function to determine if the data string in the SETVAR
command is a string of hexadecimal bytes.

However, ShellIsHexOrDecimalNumber() calls ShellConvertStringToUint64()
for validation. Therefore SETVAR rejects hex strings that cannot be
interpreted as UINT64 values due to range errors, despite the fact that
the hexadecimal data string of the SETVAR command is supposed to describe
an arbitrary array of bytes, rather than UINT64 values.

The internal library function InternalShellIsHexOrDecimalNumber() comes
close, as the first idea for the fix, however it is not quite right
either; it removes a leading minus sign, plus it allows 0x / 0X prefixes.
This would not be correct for the SETVAR command.

Instead, add a trivial utility function that is specific to the SETVAR
implementation. IsStringOfHexNibbles() accepts empty strings for
simplicity, but where we call it we have already ensured that the data
string is not empty (because that is handled earlier by the "delete
variable" branch of SETVAR). An even number of nibbles is also enforced
near the call site.

Cc: Jaben Carsey <jaben.carsey@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.0
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>

git-svn-id: https://svn.code.sf.net/p/edk2/code/trunk/edk2@17128 6f19259b-4bc3-4df7-8a09-765794883524

ShellPkg/Library/UefiShellDebug1CommandsLib/SetVar.c

@@ -23,6 +23,35 @@ STATIC CONST SHELL_PARAM_ITEM ParamList[] = {
   {NULL, TypeMax}
   };
 
+
+/**
+  Check if the input is a (potentially empty) string of hexadecimal nibbles.
+
+  @param[in] String  The CHAR16 string to check.
+
+  @retval FALSE  A character has been found in String for which
+                 ShellIsHexaDecimalDigitCharacter() returned FALSE.
+
+  @retval TRUE   Otherwise. (Note that this covers the case when String is
+                 empty.)
+**/
+BOOLEAN
+EFIAPI
+IsStringOfHexNibbles (
+  IN CONST CHAR16  *String
+  )
+{
+  CONST CHAR16 *Pos;
+
+  for (Pos = String; *Pos != L'\0'; ++Pos) {
+    if (!ShellIsHexaDecimalDigitCharacter (*Pos)) {
+      return FALSE;
+    }
+  }
+  return TRUE;
+}
+
+
 /**
   Function for 'setvar' command.
 
@@ -135,6 +164,7 @@ ShellCommandRunSetVar (
 
         ASSERT(Data[0] == L'=');
         Data++;
+        ASSERT(Data[0] != L'\0');
 
         //
         // Determine if the variable exists and get the attributes
@@ -166,7 +196,7 @@ ShellCommandRunSetVar (
         //
         // What type is the new data.
         //
-        if (ShellIsHexOrDecimalNumber(Data, TRUE, FALSE)) {
+        if (IsStringOfHexNibbles(Data)) {
           if (StrLen(Data) % 2 != 0) {
             ShellPrintHiiEx(-1, -1, NULL, STRING_TOKEN (STR_GEN_PARAM_INV), gShellDebug1HiiHandle, L"setvar", Data);  
             ShellStatus = SHELL_INVALID_PARAMETER;