Commit Graph

85 Commits

Author SHA1 Message Date
Liming Gao ce283fd609 BaseTools ConvertFceToStructurePcd: Fix the array value with empty string
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-10-31 22:07:57 +08:00
Liming Gao 7b500c606a BaseTools: Remove the step to freeze python tool
https://bugzilla.tianocore.org/show_bug.cgi?id=1257
Binary python tool is not supported anymore. So, the freeze python tool
step is not required.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-10-19 08:21:47 +08:00
Liming Gao 1ccc4d895d Revert BaseTools: PYTHON3 migration
This reverts commit 6693f359b3c213513c5096a06c6f67244a44dc52..
678f851312.

Python3 migration is the fundamental change. It requires every developer
to install Python3. Before this migration, the well communication and wide
verification must be done. But now, most people is not aware of this change,
and not try it. So, Python3 migration is reverted and be moved to edk2-staging
Python3 branch for the edk2 user evaluation.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Liming Gao <liming.gao@intel.com>
2018-10-15 08:29:14 +08:00
Yunhua Feng 9d7e1e56a7 BaseTools/Scripts: Porting PackageDocumentTools code to use Python3
Porting PackageDocumentTools code to Python3
DoxyGen 1.8.6
wxpython 4.0.3

Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yunhua Feng <yunhuax.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-10-13 09:57:16 +08:00
Dandan Bi 6460513aa6 BaseTools/PatchCheck.py: Fix error when run with Python3
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1145

Currently run PatchCheck.py with Python3 will meet
following error:
.....
File "PatchCheck.py", line 554, in run_git
return Result[0].decode('utf-8', 'ignore') if Result[0] and
Result[0].find("fatal")!=0 else None
TypeError: a bytes-like object is required, not 'str'

This issue was introduce by commit:5ac4548cdf654.

This patch is to convert the str object of "fatal" to
byte object to fix this failure.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Jaben Carsey <jaben.carsey@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Dandan Bi <dandan.bi@intel.com>
Reviewed-by: Jaben Carsey <jaben.carsey@intel.com>
2018-09-06 13:41:35 +08:00
Jaben Carsey 5ac4548cdf PatchCheck - add error message for invalid parameter
Currently if an invalid parameter is passed, it gives a stack trace.
This changes it to an error message.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
Tested-by: Liming Gao <liming.gao@intel.com>
2018-08-08 11:41:28 +08:00
Kinney, Michael D cd3a42640a BaseTools/BinToPcd: Open output file as text file
https://bugzilla.tianocore.org/show_bug.cgi?id=1069

Undo changes from following commit:

83964ebc5e

Change the open mode for the output file from 'wb' to 'w' so the
output file is written as a text file and not a binary file.

This resolves the issue where the text file was not writable from
Python 3.x and also removes b'' from output file when the string
was encoded as a bytearray.

Cc: YanYan Sun <yanyan.sun@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com>
Tested-by:YanYan Sun <yanyan.sun@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-08-06 10:48:35 +08:00
Kinney, Michael D 83964ebc5e BaseTools/BinToPcd: Encode string returned from ByteArray()
https://bugzilla.tianocore.org/show_bug.cgi?id=1069

The ByteArray() method returns a string with the hex bytes of
a PCD value.  Make sure the string is always encoded as a string,
so it can be used to build a complete PCD statement string and be
written out to a file.  This change is required for Python 3.x
compatibility.

Cc: YanYan Sun <yanyan.sun@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-08-02 18:42:27 -07:00
Kinney, Michael D 76c09700ed BaseTools/BinToPcd: Fix Python 2.7.x compatibility issue
https://bugzilla.tianocore.org/show_bug.cgi?id=1042

Convert Buffer to type bytearray before converting to a
string of hex byte values so the type of items in Buffer is
consistent for both Python 2.7.x and Python 3.x.

Cc: YanYan Sun <yanyan.sun@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-08-01 15:27:56 -07:00
Liming Gao ef529e6ab7 BaseTools Script: Add the script to generate Structure PCD setting
Here is this script usage.
1. Build one platform.
2. Use FCE tool to read HII config from the generated FD image.
FCE read -i Platform.fd > Config.txt
3. Call the script to generate StructurePcd setting.
ConvertFceToStructurePcd.py -p Build\PlatformPkg\DEBUG_VS2015x86 \
-c Config.txt -o OutputDir
OutputDir directory has StructurePcd.dec, StructurePcd.dsc, StructurePcd.inf.
4. Refer to wiki https://github.com/lgao4/edk2/wiki/StructurePcd-Enable-Steps
to enable structure pcd in this platform.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-07-25 11:12:18 +08:00
Gary Lin 2add3cffb0 BaseTools: Use absolute import in Scripts
Based on "futurize -f libfuturize.fixes.fix_absolute_import

Contributed-under: TianoCore Contribution Agreement 1.1
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Gary Lin <glin@suse.com>
Reviewed-by: Jaben Carsey <jaben.carsey@intel.com>
2018-07-16 11:22:19 +08:00
Liming Gao f7496d7173 BaseTools: Clean up source files
1. Do not use tab characters
2. No trailing white space in one line
3. All files must end with CRLF

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-07-09 10:25:47 +08:00
Gary Lin 86379ac48b BaseTools: Replace StringIO.StringIO with io.BytesIO
Replace StringIO.StringIO with io.BytesIO to be compatible with python3.
This commit also removes "import StringIO" from those python scripts
that don't really use it.

Contributed-under: TianoCore Contribution Agreement 1.1
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Gary Lin <glin@suse.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-06-27 16:33:28 +08:00
Gary Lin 0d1f5b2b5d BaseTools: Fix old python2 idioms
Based on "futurize -f lib2to3.fixes.fix_idioms"

* Change some type comparisons to isinstance() calls:
    type(x) == T -> isinstance(x, T)
    type(x) is T -> isinstance(x, T)
    type(x) != T -> not isinstance(x, T)
    type(x) is not T -> not isinstance(x, T)

* Change "while 1:" into "while True:".

* Change both

    v = list(EXPR)
    v.sort()
    foo(v)

and the more general

    v = EXPR
    v.sort()
    foo(v)

into

    v = sorted(EXPR)
    foo(v)

Contributed-under: TianoCore Contribution Agreement 1.1
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Gary Lin <glin@suse.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-06-27 16:33:27 +08:00
Gary Lin ccaa7754a2 BaseTools: Adjust the spaces around commas and colons
Based on "futurize -f lib2to3.fixes.fix_ws_comma"

Contributed-under: TianoCore Contribution Agreement 1.1
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Gary Lin <glin@suse.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-06-27 16:33:25 +08:00
Gary Lin 27c4ceb41c BaseTools: Remove the deprecated hash_key()
Replace "has_key()" with "in" to be compatible with python3.
Based on "futurize -f lib2to3.fixes.fix_has_key"

Contributed-under: TianoCore Contribution Agreement 1.1
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Gary Lin <glin@suse.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-06-27 16:33:23 +08:00
Gary Lin 72443dd250 BaseTools: Refactor python print statements
Refactor print statements to be compatible with python 3.
Based on "futurize -f libfuturize.fixes.fix_print_with_import"

Contributed-under: TianoCore Contribution Agreement 1.1
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Gary Lin <glin@suse.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-06-27 16:33:21 +08:00
Gary Lin 5b0671c1e5 BaseTools: Refactor python except statements
Convert "except ... ," to "except ... as" to be compatible with python3.
Based on "futurize -f lib2to3.fixes.fix_except"

Contributed-under: TianoCore Contribution Agreement 1.1
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Gary Lin <glin@suse.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-06-27 16:31:30 +08:00
Gary Lin 00eb12a2c7 BaseTools: Fix a typo in ini.py
"if mis not None:" => "if m is not None:"

Contributed-under: TianoCore Contribution Agreement 1.1
Cc: Jaben Carsey <jaben.carsey@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Gary Lin <glin@suse.com>
Reviewed-by: Jaben Carsey <jaben.carsey@intel.com>
2018-06-27 16:30:31 +08:00
Kinney, Michael D 0c805f4f8f BaseTools/BinToPcd: Follow PEP-8 indent of 4 spaces
https://www.python.org/dev/peps/pep-0008/

Cc: Yanyan Sun <yanyan.sun@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-06-14 15:43:40 -07:00
Kinney, Michael D d0f946f30d BaseTools/BinToPcd: Update for Python 3 compatibility
Update to be compatible with both Python 2.x and Python 3.x.
Also return error code 1 when an error is detected to support
use of this tool in scripts.

Cc: Yanyan Sun <yanyan.sun@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-06-14 15:43:30 -07:00
Kinney, Michael D 8e96580999 BaseTools/BinToPcd: --offset must be 8-byte aligned
https://bugzilla.tianocore.org/show_bug.cgi?id=974
https://bugzilla.tianocore.org/show_bug.cgi?id=965

Update help to state that --offset must be 8-byte aligned.
Verify that --offset is 8-byte aligned and print an error
message if it is not 8-byte aligned.

Cc: Yanyan Sun <yanyan.sun@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-06-14 15:43:22 -07:00
Kinney, Michael D b2e043b689 BaseTools/BinToPcd: Clarify error message for --type HII
https://bugzilla.tianocore.org/show_bug.cgi?id=963

Update error message for --type HII.  If either --variable-guid
or --variable-name is missing, then print an error message that
states that both --variable-guid and --variable-name are required.

Cc: Yanyan Sun <yanyan.sun@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-06-14 15:43:15 -07:00
Kinney, Michael D 1d79b72ee6 BaseTools/BinToPcd: Fix typo in error messages
https://bugzilla.tianocore.org/show_bug.cgi?id=962

Change "PcdToBin" to "BinToPcd"

Cc: Yanyan Sun <yanyan.sun@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-06-14 15:43:06 -07:00
Liming Gao c30084fbac BaseTools Script: Formalize source files to follow DOS format
V3:
support exclude dir and file by name while traversing the directory.
remove close in with statement.

V2:
add version,description,copyright.
add flag -v,-q,--append-extensions,--override-extensions,--debug.
-q will omit default output,-v and --debug are not implemented.
add default file extensions.
support input of file path.
support muliple input path.
simplify comment.
change 'pattern'.encode() to b'pattern',I think this will be better.
change naming of variable and function to keep the same with BinToPcd.py

V1:
FormatDosFiles.py is added to clean up dos source files. It bases on
the rules defined in EDKII C Coding Standards Specification.
5.1.2 Do not use tab characters
5.1.6 Only use CRLF (Carriage Return Line Feed) line endings.
5.1.7 All files must end with CRLF
No trailing white space in one line. (To be added in spec)

The source files in edk2 project with the below postfix are dos format.
.h .c .nasm .nasmb .asm .S .inf .dec .dsc .fdf .uni .asl .aslc .vfr .idf
.txt .bat .py

The package maintainer can use this script to clean up all files in his
package. The prefer way is to create one patch per one package.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Liming Gao <liming.gao@intel.com>
Signed-off-by: Dongao Guo <dongao.guo@intel.com>
2018-06-13 13:03:16 +08:00
Carsey, Jaben 98120f5fe2 BaseTools: refactor and remove out of date use of .keys()
this is no longer required to make dictionary objects iterable.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-04-10 10:05:16 +08:00
Carsey, Jaben 4231a8193e BaseTools: Remove equality operator with None
replace "== None" with "is None" and "!= None" with "is not None"

Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-03-30 08:25:13 +08:00
Kinney, Michael D aedd1559cc BaseTools/BinToPcd: Add support for multiple binary input files
https://bugzilla.tianocore.org/show_bug.cgi?id=890

There are use cases where a VOID * PCD needs to be generated from multiple
binary input files.  This can be in the form of an array of fixed size
elements or a set of variable sized elements.

Update BinToPcd to support multiple one or more -i INPUTFILE arguments.
By default, the contents of each binary input file are concatenated in
the order provided.  This supports generating a PCD that is an array of
fixed size elements

Add -x, --xdr flags to BinToPcd  to encodes the PCD using the
Variable-Length Opaque Data of RFC 4506 External Data Representation
Standard (XDR).

    https://tools.ietf.org/html/rfc4506
    https://tools.ietf.org/html/rfc4506#section-4.10

The data format from RFC 4506 meets the requirements for a PCD that is a
set of variable sized elements in the Variable-Length Opaque Data format.
The overhead of this format is a 32-bit length and 0 to 3 bytes of padding
to align the next element at a 32-bit boundary.

Cc: Sean Brogan <sean.brogan@microsoft.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2018-03-28 08:51:15 -07:00
Yonghong Zhu 7ccc9c954c BaseTools: Add PackageDocumentTools into Scripts folder
This tool is used to generate the document for edk2 packages. The
generated document will be in UDK release. For example, UDK2017
document can be found in:
https://github.com/tianocore/tianocore.github.io/wiki/UDK2017#documentation

Cc: Star Zeng <star.zeng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Star Zeng <star.zeng@intel.com>
2018-03-19 09:19:36 +08:00
Michael D Kinney ef190542b4 BaseTools/Scripts: Add sample makefile for use with RunMakefile.py
https://bugzilla.tianocore.org/show_bug.cgi?id=670

Add sample makefile that can be used to test RunMakefile.py
script and can also be used as a template to start a new
PREBUILD/POSTBUILD makefile.

This makefile contains TAB characters instead of spaces on
purpose to maximize compatibility with make utilities.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.0
Signed-off-by: Michael Kinney <michael.d.kinney@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2017-08-11 11:28:35 -07:00
Michael D Kinney c8d385265f BaseTools/Scripts: Add python script to run a makefile
https://bugzilla.tianocore.org/show_bug.cgi?id=670

Add the python script RunMakefile.py that can be used
in a PREBUILD/POSTBUIILD action to invoke a makefile
passing in context as makefile defines.  The command
line arguments passed into RunMakefile.py are converted
to the following set of defines.

* ACTIVE_PLATFORM
* TARGET_ARCH
* TOOL_CHAIN_TAG
* CONF_DIRECTORY
* TARGET
* EXTRA_FLAGS

In addition, a makefile can access the system environment
variables including WORKSPACE and PACKAGES_PATH.

The makefile target from the following set is also passed
into the makefile.  If no target is passed into build, then
the 'all' target is used.

[all|fds|genc|genmake|clean|cleanall|cleanlib|modules|libraries|run]

A platform DSC file can use a statements in the [Defines]
section of the following form to use this script.  MAKEFILE
is a WORKSPACE or PACKAGES_PATH relative path to the makefile
to run.

[Defines]
  PREBUILD  = python BaseTools/Script/RunMakefile.py --makefile MAKEFILE
  POSTBUILD = python BaseTools/Script/RunMakefile.py --makefile MAKEFILE

Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.0
Signed-off-by: Michael Kinney <michael.d.kinney@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2017-08-11 11:21:04 -07:00
Michael D Kinney f2d1b866df BaseTools/PatchCheck: Support Contribution Agreement 1.1
https://bugzilla.tianocore.org/show_bug.cgi?id=628

Update PatchCheck.py to support either
"Contributed-under: TianoCore Contribution Agreement 1.0"
or "Contributed-under: TianoCore Contribution Agreement 1.1"
in the commit message.

Temporarily continue to allow the TianoCore Contribution
Agreement 1.0 agreement.

Cc: Leif Lindholm <leif.lindholm@linaro.org>
Cc: Andrew Fish <afish@apple.com>
Cc: Jordan Justen <jordan.l.justen@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com>
Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
Reviewed-by: Leif Lindholm <leif.lindholm@linaro.org>
2017-08-03 11:01:46 -07:00
Hao Wu 6a69dd4937 BaseTools/PatchCheck.py: Add warning info for new binary files
The commit adds the detection of adding new binary files in a patch file
or in a commit.

The following warning messages will be appended at the end of the script
output:

WARNING - The following binary files will be added into the repository:
  <BinaryFile1>
  <BinaryFile2>
  ...

Contributed-under: TianoCore Contribution Agreement 1.0
Signed-off-by: Hao Wu <hao.a.wu@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2017-06-26 09:34:11 +08:00
Hao Wu aab57eff5a BaseTools/PatchCheck.py: Fix misreport for binary changes in patch
For a patch file that:
1. Contains a binary change
2. Contains any other changes after the binary change

PatchCheck.py will complains with the following error:
* Patch format error: diff found after end of patch
   Line: literal XXXX

This commit resolves this misreport.

Contributed-under: TianoCore Contribution Agreement 1.0
Signed-off-by: Hao Wu <hao.a.wu@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2017-06-26 09:26:56 +08:00
Ard Biesheuvel f4d3ba87bb BaseTools/Scripts: fix GccBase.lds line endings
Replace a <LF> line ending that was introduced inadvertently by a
recent commit with the correct <CR><LF>

Contributed-under: TianoCore Contribution Agreement 1.0
Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
2017-05-28 12:07:02 +00:00
Ard Biesheuvel 00b00cc57b BaseTools/Scripts: discard .gnu.hash section in GCC builds
Some builds of GCC/binutils will default to using the GNU flavor of
the symbol hash table, and will emit it into a section called .gnu.hash
rather than .hash. We have no use for its contents, and GenFw ignores
it anyway, so it shouldn't really matter what we do with it.

However, due to a workaround for AARCH64 we have in GenFw to deal with
older GCCs that corrupt section-based relocations when merging sections
during the final link, we need the ELF and PE/COFF views of the binary
to be identical. Since we don't place the .gnu.hash section explicitly,
it may end up at the beginning of the ELF binary, causing other sections
to be shifted in the ELF view but not in the PE/COFF view.

So let's add .gnu.hash to the GCC linker script. We don't care about its
contents so add it to the /DISCARD/ section.

Contributed-under: TianoCore Contribution Agreement 1.0
Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Tested-by: Leif Lindholm <leif.lindholm@linaro.org>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2017-05-24 06:09:39 -07:00
Ard Biesheuvel 3cf41b8728 BaseTools: GCC: move most AutoGen.obj contents back to .data section
The generated AutoGen.c files mostly contain read-only data, but due to
lacking annotations, all of it is emitted into the .data section by the
compiler.

Given that GUIDs are UEFI's gaffer tape, having writable GUIDs is a
security hazard, and this was the main rationale for putting AutoGen.obj
in the .text section. However, as it turns out, patchable PCDs are emitted
there as well, which can legally be modified at runtime.

So update the wildcard pattern to only match g...Guid sections, and move
everything else back to .data (Note that this relies on -fdata-sections,
without that option, everything is emitted into .data)

Contributed-under: TianoCore Contribution Agreement 1.0
Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Fixes: 233bd25b00
[lersek@redhat.com: add reference to previous commit being fixed up]
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
2017-02-24 11:27:56 +01:00
Jiewen Yao 88dab294d2 BaseTool/Script: Add SmiHandleProfile OS tool to get symbol.
This tool accepts the input XML file generated by SmiHandlerProfile
application and convert the RVA address to be a user readable
symbol.
It also converts the GUID to be a user readable string.

Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Contributed-under: TianoCore Contribution Agreement 1.0
Signed-off-by: Jiewen Yao <jiewen.yao@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2017-02-22 15:28:22 +08:00
Michael Kinney fd0597aa9c BaseTools/Scripts: Add BinToPcd utility
https://bugzilla.tianocore.org/show_bug.cgi?id=228

Add a utility that converts a binary file into a VOID* PCD value
or a full DSC file VOID* PCD statement with support for all the
DSC supported PCD sections.

usage: BinToPcd [-h] [--version] -i INPUTFILE [-o OUTPUTFILE] [-p PCDNAME]
                [-t {VPD,HII}] [-m MAXSIZE] [-f OFFSET] [-n VARIABLENAME]
                [-g VARIABLEGUID] [-v] [-q] [--debug [0-9]]

Convert a binary file to a VOID* PCD value or DSC file VOID* PCD statement.
Copyright (c) 2016, Intel Corporation. All rights reserved.

optional arguments:
  -h, --help        show this help message and exit
  --version         show program's version number and exit
  -i INPUTFILE, --input INPUTFILE
                    Input binary filename
  -o OUTPUTFILE, --output OUTPUTFILE
                    Output filename for PCD value or PCD statement
  -p PCDNAME, --pcd PCDNAME
                    Name of the PCD in the form
                    <PcdTokenSpaceGuidCName>.<PcdCName>
  -t {VPD,HII}, --type {VPD,HII}
                    PCD statement type (HII or VPD). Default is standard.
  -m MAXSIZE, --max-size MAXSIZE
                    Maximum size of the PCD. Ignored with --type HII.
  -f OFFSET, --offset OFFSET
                    VPD offset if --type is VPD. UEFI Variable offset if
                    --type is HII.
  -n VARIABLENAME, --variable-name VARIABLENAME
                    UEFI variable name. Only used with --type HII.
  -g VARIABLEGUID, --variable-guid VARIABLEGUID
                    UEFI variable GUID C name. Only used with --type HII.
  -v, --verbose     Increase output messages
  -q, --quiet       Reduce output messages
  --debug [0-9]     Set debug level

This utility can be used in PCD value mode to convert a binary
file into a string that can then be copied into the PCD value field
of a VOID* PCD.  The following is an example of PCD value mode on
an 8 byte test.bin file.

  BinToPcd.py -i test.bin

  {0x48, 0x65, 0x6c, 0x6c, 0x6f, 0x20, 0x0d, 0x0a}

The DSC file VOID* PCD statement mode can be used to generate a
complete PCD statement for the PCD section types that a DSC file
supports:

  [PcdsFixedAtBuild]
  [PcdsPatchableInModule]
  [PcdsDynamicDefault]
  [PcdsDynamicExDefault]
  [PcdsDynamicVpd]
  [PcdsDynamicExVpd]
  [PcdsDynamicHii]
  [PcdsDynamicExHii]

The PCD statement mode is useful when combined with a !include
statement in a DSC file.  BinToPcd.py can be used to convert a
binary file to a PCD statement in an output file, and that output
file can be included into a DSC file in the matching PCD section
to set the value of the PCD to the value from the binary file
without having to copy the value into the DSC file.  Updates can be
made to the included file without editing the DSC file.  Some
example use cases are the setting the public key PCDs such as:

  gEfiSecurityPkgTokenSpaceGuid.PcdRsa2048Sha256PublicKeyBuffer
  gEfiSecurityPkgTokenSpaceGuid.PcdPkcs7CertBuffer

The following example converts a public key binary file to a
[PcdsFixedAtBuild] compatible PCD statement:

  BinToPcd.py -i PublicKey.bin -o PublicKey.pcd
    --pcd gEfiSecurityPkgTokenSpaceGuid.PcdPkcs7CertBufferkenSpaceGuid

The PublicKey.pcd output file contains a single line:

  gEfiSecurityPkgTokenSpaceGuid.PcdPkcs7CertBuffer|{0x48, ...}

A DSC file can be updated to include the PublicKey.pcd file:

  [PcdsFixedAtBuild]
  !include PublicKey.pcd

Value examples
===============
  BinToPcd.py -i test.bin
    {0x68, 0x65, 0x6c, 0x6c, 0x6f, 0x20, 0x0d, 0x0a}

Normal examples:
=================
  BinToPcd.py -i test.bin -p Guid.Token
    Guid.Token|{0x68, 0x65, 0x6c, 0x6c, 0x6f, 0x20, 0x0d, 0x0a}

  BinToPcd.py -i test.bin -p Guid.Token -m 20
    Guid.Token|{0x68, 0x65, 0x6c, 0x6c, 0x6f, 0x20, 0x0d, 0x0a}|VOID*|20

VPD examples:
=============
  BinToPcd.py -i test.bin -p Guid.Token -t VPD
    Guid.Name|*|8|{0x48, 0x65, 0x6c, 0x6c, 0x6f, 0x20, 0x0d, 0x0a}

  BinToPcd.py -i test.bin -p Guid.Token -t VPD -f 20
    Guid.Name|20|8|{0x48, 0x65, 0x6c, 0x6c, 0x6f, 0x20, 0x0d, 0x0a}

  BinToPcd.py -i test.bin -p Guid.Token -t VPD -m 10
    Guid.Name|*|10|{0x48, 0x65, 0x6c, 0x6c, 0x6f, 0x20, 0x0d, 0x0a}

  BinToPcd.py -i test.bin -p Guid.Token -t VPD -f 20 -m 10
    Guid.Name|20|10|{0x48, 0x65, 0x6c, 0x6c, 0x6f, 0x20, 0x0d, 0x0a}

HII examples:
=============
  BinToPcd.py -i test.bin -p Guid.Token -t HII -g VarGuid -n VarName
    Guid.Name|L"VarName"|VarGuid|0|{0x48, 0x65, 0x6c, 0x6c}

  BinToPcd.py -i test.bin -p Guid.Token -t HII -g VarGuid -n VarName -f 8
    Guid.Name|L"VarName"|VarGuid|8|{0x48, 0x65, 0x6c, 0x6c}

Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.0
Signed-off-by: Michael Kinney <michael.d.kinney@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2016-11-14 08:40:41 -08:00
Yonghong Zhu c3926cdbbd BaseTools/PatchCheck.py: Update to report error for EFI_D_*
In EDK2, DEBUG_* is recommended to be used instead of EFI_D_*. For new
code, they should use DEBUG_* macro.

Fixes:https://bugzilla.tianocore.org/show_bug.cgi?id=143
Cc: Liming Gao <liming.gao@intel.com>
Cc: Jordan Justen <jordan.l.justen@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.0
Signed-off-by: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
2016-10-21 07:28:01 +08:00
Yonghong Zhu e709bbb1ce BaseTools/PatchCheck.py: Update to handle the two [] as prefix
The bug is that only remove the first [] when it does the char count,
however sometimes we use [edk2][patch] as prefix, this patch fix this bug.

Fixes: https://bugzilla.tianocore.org/show_bug.cgi?id=113
Cc: Liming Gao <liming.gao@intel.com>
Cc: Jordan Justen <jordan.l.justen@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.0
Signed-off-by: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
2016-10-21 07:25:22 +08:00
Yonghong Zhu e61406708c BaseTools/PatchCheck.py: Update for max length of subject and message line
This patch update PatchCheck.py:
1. The subject line of the commit message should be < 72 characters.
2. The other lines of the commit message should be < 76 characters.

Fixes: https://bugzilla.tianocore.org/show_bug.cgi?id=113
Cc: Liming Gao <liming.gao@intel.com>
Cc: Jordan Justen <jordan.l.justen@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.0
Signed-off-by: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
2016-10-21 07:23:57 +08:00
Ard Biesheuvel 669a7562cb BaseTools/GccBase.lds: don't copy RELA section to PE/COFF
The CLANG38 toolchain creates a PIE binary at link time. This is
necessary since the LTO code generation may otherwise result in
code that cannot execute correctly when loaded above 2 GB.

PIE executables contain a RELA section consisting of dynamic
relocation entries that are intended for consumption by the loader
at runtime. For this reason, it has the SHF_ALLOC attribute set by
default, and will be identified by GenFw as a section that needs to
be copied into the PE/COFF binary, resulting in waste of space since
the PE/COFF loader does not use this data at all.

So mark the RELA section as informational: this will prevent the
linker from setting the SHF_ALLOC attribute, causing GenFw to
ignore it.

DxeCore.efi before:

    Detected 'X64' type PE/COFF image consisting of 3 sections
    Section alignment:      0x40
    File alignment:         0x40
    Section '.text' @ 0x00000240
    File offset:            0x240
    Virtual size:           0x21000
    Raw size:               0x21000
    Section '.data' @ 0x00021240
    File offset:            0x21240
    Virtual size:           0x3640
    Raw size:               0x3640
    Section '.reloc' @ 0x00024880
    File offset:            0x24880
    Virtual size:           0x280
    Raw size:               0x280

DxeCore.efi after:

    Detected 'X64' type PE/COFF image consisting of 3 sections
    Section alignment:      0x40
    File alignment:         0x40
    Section '.text' @ 0x00000240
    File offset:            0x240
    Virtual size:           0x1f440
    Raw size:               0x1f440
    Section '.data' @ 0x0001f680
    File offset:            0x1f680
    Virtual size:           0x3640
    Raw size:               0x3640
    Section '.reloc' @ 0x00022cc0
    File offset:            0x22cc0
    Virtual size:           0x280
    Raw size:               0x280

Contributed-under: TianoCore Contribution Agreement 1.0
Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2016-08-22 12:26:42 +02:00
Ard Biesheuvel 7fd5d61980 BaseTools GCC: drop GNU notes section from EFI image
Recent versions of GNU ld automatically emit a .notes section into
the ELF binary containing a build id. Since this is an allocatable
section by default, it will be identified by GenFw as a section
that requires PE/COFF conversion, which may cause sections to be
moved around unexpectedly.

So retain the section, but tag it as INFO, which tells the linker
that it should not be accounted for in the binary's memory layout.

Contributed-under: TianoCore Contribution Agreement 1.0
Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Acked-by: Jordan Justen <jordan.l.justen@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2016-08-02 10:54:11 +02:00
Thomas Palmer 03630a8148 Preserve hii section in GCC binaries
According to UEFI spec:
Once an image is loaded, LoadImage() installs
EFI_HII_PACKAGE_LIST_PROTOCOL on the handle if the image contains a
custom PE/COFF resource with the type 'HII'. The protocol's
interface pointer points to the HII package list which is contained
in the resource's data.

This is controlled by the UEFI_HII_RESOURCE_SECTION define in the INF
file.  When present the HII resource is linked with the module
binary.

Unfortunately GCC-built binaries have been stripping the .hii section
entirely.  See  "[edk2] HII gEfiHiiPackageListProtocolGuid problem
with  GCC48(VS2012x86 works)"
http://thread.gmane.org/gmane.comp.bios.tianocore.devel/13438
http://thread.gmane.org/gmane.comp.bios.tianocore.devel/14899

This patch tells the linker to preserve the .hii sections

Contributed-under: TianoCore Contribution Agreement 1.0
Signed-off-by: Thomas Palmer <thomas.palmer@hpe.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Bruce Cran <bruce.cran@gmail.com>
Reviewed-by: Bruce Cran <bruce.cran@gmail.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2016-07-26 10:21:45 +08:00
Star Zeng d84577e573 BaseTools MemoryProfileSymbolGen.py: Handle 64bits rva from "nm -l xxx.dll"
Current MemoryProfileSymbolGen.py assumes the rva is 32bits,
the patch is to remove the restriction to match any lengths
of rva.

Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.0
Signed-off-by: Star Zeng <star.zeng@intel.com>
Reviewed-by: Jiewen Yao <jiewen.yao@intel.com>
2016-07-05 17:25:38 +08:00
Star Zeng 1d9869f9e9 BaseTools Scripts: Add MemoryProfileSymbolGen.py
This tool depends on DIA2Dump.exe (VS) or nm (gcc) to parse debug entry.

Usage: MemoryProfileSymbolGen.py [--version] [-h] [--help] [-i inputfile
[-o outputfile]]

Copyright (c) 2016, Intel Corporation. All rights reserved.

Options:
  --version             show program's version number and exit
  -h, --help            show this help message and exit
  -i INPUTFILENAME, --inputfile=INPUTFILENAME
                        The input memory profile info file output from
                        MemoryProfileInfo application in MdeModulePkg
  -o OUTPUTFILENAME, --outputfile=OUTPUTFILENAME
                        The output memory profile info file with symbol,
                        MemoryProfileInfoSymbol.txt will be used if it is not

Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.0
Signed-off-by: Star Zeng <star.zeng@intel.com>
Reviewed-by: Jiewen Yao <jiewen.yao@intel.com>
2016-07-01 09:39:59 +08:00
Jordan Justen ad00b0452e BaseTools ConvertMasmToNasm: Don't try to reconvert .nasm files
We now check to see if the destination .nasm file already exists. If
it does, then we don't try to convert the .asm to .nasm.

Contributed-under: TianoCore Contribution Agreement 1.0
Signed-off-by: Jordan Justen <jordan.l.justen@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2016-06-28 13:16:53 -07:00
Jordan Justen 5de927b54b BaseTools ConvertMasmToNasm: Support preserving assembly files
In the first stage of conversion, we need to preserve the AT&T style
.s assembly files for use with OS X toolchains.

This change allows '--keep=s' to be used with the script to preserve
these files.

Contributed-under: TianoCore Contribution Agreement 1.0
Signed-off-by: Jordan Justen <jordan.l.justen@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2016-06-28 13:16:46 -07:00
Jordan Justen 90694f1218 BaseTools ConvertMasmToNasm: put filter/map result in tuple for python3
Python 3's filter and map functions returns an iterator which you
can't call len() on. Since we'll want to use len() later, we put the
filter results into a tuple.

Contributed-under: TianoCore Contribution Agreement 1.0
Signed-off-by: Jordan Justen <jordan.l.justen@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2016-06-28 13:16:39 -07:00