audk/CryptoPkg/Library/BaseCryptLib
Karyne Mayer 03805fc376 CryptoPkg: Fix function qsort for non 32-bit machines
Although the function qsort receives as an argument a "compare" function
which returns an "int", QuickSortWorker (the function used internally by
qsort to do its job) receives as an argument a "CompareFunction" which
returns an "INTN". In a 32-bit machine, "INTN" is defined as "INT32",
which is defined as "int" and everything works well. However, when qsort
is compiled for a 64-bit machine, "INTN" is defined as "INT64" and the
return values of the compare functions become incompatible ("int" for
qsort and "INT64" for QuickSortWorker), causing malfunction.

For example, let's assume qsort is being compiled for a 64-bit machine.
As stated before, the "compare" function will be returning an "int",
and "CompareFunction" will be returning an "INT64". When, for example,
the "compare" function (which was passed as an argument to qsort and,
then, re-passed as an argument to QuickSortWorker) returns -1 (or
0xffffffff, in a 32-bit integer, its original return type) from inside
a call to QuickSortWorker, its return value is interpreted as being an
"INT64" value - which turns out to be 4294967295 (or 0x00000000ffffffff,
in a 64-bit integer) -, making the function QuickSortWorker to behave
unexpectedly.

Note that this unexpected (or incorrect) conversion does not happen when
casting an "INT32" to an "INT64" directly, but does happen when casting
function types.

The issue is fixed by changing the return type of SORT_COMPARE (the type
of "CompareFunction", used by QuickSortWorker) from "INTN" to "int".
This way, both qsort and QuickSortWorker use compatible definitions for
their compare functions.

Contributed-under: TianoCore Contribution Agreement 1.0
Acked-by: Paulo Alcantara Cavalcanti <paulo.alc.cavalcanti@hp.com>
Signed-off-by: Karyne Mayer <kmayer@hp.com>
Signed-off-by: Rodrigo Dias Correa <rodrigo.dia.correa@hp.com>
Signed-off-by: Arthur Crippa Burigo <acb@hp.com>
Reviewed-by: Qin Long <qin.long@intel.com>

git-svn-id: https://svn.code.sf.net/p/edk2/code/trunk/edk2@19748 6f19259b-4bc3-4df7-8a09-765794883524
2016-01-26 08:51:13 +00:00
..
Cipher CryptoPkg: Fix one wrong parameter for weak key checking 2015-09-02 08:44:46 +00:00
Hash CryptoPkg: remove redundant parentheses 2015-07-14 02:57:38 +00:00
Hmac Fix several issues in BaseCryptLib: 2012-08-02 02:49:24 +00:00
Pem Enhance error handling code after calling BIO_new in BaseCryptLib. 2013-08-07 08:11:14 +00:00
Pk [CryptoPkg] Correct one typo in the API comments. 2015-11-25 08:34:57 +00:00
Rand Fixed one typo in MathRShiftU64.S stub implementation; Add one digest algorithm registration to make sure the underlying algorithm is available for the default OpenSSL software PRNG implementation. 2013-11-19 02:57:19 +00:00
SysCall CryptoPkg: Fix function qsort for non 32-bit machines 2016-01-26 08:51:13 +00:00
BaseCryptLib.inf CryptoPkg: fix build support under RVCT 2015-12-17 17:02:06 +00:00
BaseCryptLib.uni CryptoPkg: Convert all .uni files to utf-8 2015-12-15 04:51:26 +00:00
InternalCryptLib.h CryptoPkg: Fix OpenSSL BN wordsize and OPENSSL_SYS_UEFI handling 2015-10-29 14:17:15 +00:00
PeiCryptLib.inf CryptoPkg: fix build support under RVCT 2015-12-17 17:02:06 +00:00
PeiCryptLib.uni CryptoPkg: Convert all .uni files to utf-8 2015-12-15 04:51:26 +00:00
RuntimeCryptLib.inf CryptoPkg: fix build support under RVCT 2015-12-17 17:02:06 +00:00
RuntimeCryptLib.uni CryptoPkg: Convert all .uni files to utf-8 2015-12-15 04:51:26 +00:00
SmmCryptLib.inf CryptoPkg Updates to support RFC3161 timestamp signature verification. 2014-11-12 08:51:45 +00:00
SmmCryptLib.uni CryptoPkg: Convert all .uni files to utf-8 2015-12-15 04:51:26 +00:00