Commit Graph

1272 Commits

Author SHA1 Message Date
Feng, Bob C 7809492c10 BaseTools: Enable --genfds-multi-thread to default build
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1302

This patch enable --genfds-multi-thread to default build.
This patch keep --genfds-multi-thread build option for
compatibility and also add a new build option to disable
genfds-multi-thread as --no-genfds-multi-thread.

Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-09-11 09:43:59 +08:00
Feng, Bob C f23da86415 BaseTools: Fixed the build fail on Linux with --genfds-multi-thread
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1302

If GenSec input file not exist, the related command will fail and make
will stop on Linux. GenSec input file is allow to be non-existent.

This patch is to let "make" continue if gensec input file not exist.

Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-09-11 09:43:59 +08:00
Feng, Bob C e3c8311f8a BaseTools: Fixed the bug of multi-thread genffs for override inf
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1302
If there is a inf override, and multi-thread genffs is enabled, build
will fail.

This patch is going to fix this bug.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Steven Shi <steven.shi@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-09-11 09:43:58 +08:00
Feng, Bob C 000ab98574 BaseTools: Sort Pcd settings to make PcdTokenNumber be fixed
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=2147

This patch is to sort the Pcd settings so that PcdTokenNumber
will not change if the platform's Pcd settings are the same.

Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Steven Shi <steven.shi@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-09-10 08:39:19 +08:00
Feng, Bob C 373298ca0d BaseTools: Fixed issue for IgnoreAutoGen
https://bugzilla.tianocore.org/show_bug.cgi?id=2080

This patch is to improve build -u option to re-use
GlobalVar_<platformguid>_<arch>.bin file which is
introduced by multiple-process-autogen feature.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Steven Shi <steven.shi@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Acked-by: Liming Gao <liming.gao@intel.com>
2019-09-10 08:39:18 +08:00
Feng, Bob C 53e2eaba7a BaseTools: Fixed build clean regression issue
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=2166

file_lock and cache_lock are used as global data,
so move file_lock and cache_lock initialization in Build
object __init__ function.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Steven Shi <steven.shi@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-09-10 08:39:17 +08:00
Bob Feng cbe7543ebb BaseTools: Fix a bug for Hii Pcd override
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=2157

Hii Pcd links to a efi variable. The Variable
default value is evaluated by related Hii Pcds
setting. If multiple Hii Pcds links to one variable,
and the offset overlap, the later Hii Pcds setting
should be effective. There is a tool bug that is if
the Pcds are in different dsc file which are included
into the platform dsc file, build tool does not get
the Pcds relative position correctly. That means
build tool does not know which Pcd is the later one. As
the result, the variable default value will be incorrect.

This patch is to fix this bug.

Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-09-10 08:39:16 +08:00
Fan, ZhijuX d6f5a5050b BaseTools:ECC need to handle lower case 'static'
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=1778

We are going to add keyword (lower case) 'static' for functions and global
variables which are not referenced outside their current C file.

However, the ECC tool only recognizes upper case 'STATIC' at this moment.
This will lead to issue reports for new codes that follow the above coding
style.

This patch is going to handle lower case 'static'

Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-09-09 11:14:42 +08:00
Fan, ZhijuX 03ad3d889c BaseTools:Ecc handle another copyright format
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=2057

Ecc can not handle the copyright format like

(C) Copyright 2015-2016 Hewlett Packard Enterprise Development LP<BR>

This will cause Ecc to report wrong information.

This patch is going to handle this format

Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-09-09 11:13:52 +08:00
Shi, Steven 40db176d9d BaseTools: Support more file types in build cache
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1958

Current build cache does not store and restore all
types file of Hii/vfr, version and dpx. This patch
adds more file types to support them in build cache.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Signed-off-by: Steven Shi <steven.shi@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-09-02 12:17:56 +08:00
Feng, Bob C 37eef91017 BaseTools: Fixed regression issue for building !x86 builds
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=2134

This patch is to fix a build tool regression issue which
was introduced by commit e8449e1d8e.

In commit e8449e1d8e, build tool check the pcd before
filter out the irrelevant library instance.

The logic of evaluating the priority of the library class
resolutions was not changed.

Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Tested-by: Michael D Kinney <michael.d.kinney@intel.com>
Acked-by: Michael D Kinney <michael.d.kinney@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Acked-by: Leif Lindholm <leif.lindholm@linaro.org>
Tested-by: Leif Lindholm <leif.lindholm@linaro.org>
2019-08-29 11:10:29 +08:00
Feng, Bob C df43ea6c08 BaseTools: fix an incremental build issue.
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=2137

This patch is going to fix a regression issue of incremental
build which was introduced by the commit 94459080c.
The changing on INF file will not trigger module rebuild.

Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Steven Shi <steven.shi@intel.com>
Tested-by: Michael D Kinney <michael.d.kinney@intel.com>
Acked-by: Michael D Kinney <michael.d.kinney@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-08-29 11:07:48 +08:00
Feng, Bob C abc0155b03 BaseTools: Incorrect error message for library instance not found
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=2099
This is a regression issue introduced by commit e8449e.

This patch is to fix this issue.

Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-08-23 11:26:24 +08:00
Feng, Bob C 5f7bb39117 BaseTools: Fix incremental build genmake issue
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=2090

This is a regression issue introduced by commit e8449e.
This patch is going to fix this issue.

Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-08-23 11:26:24 +08:00
Shi, Steven 6dd9aa4019 BaseTools: Support long file path in windows for misc functions
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=2103

Current CopyFileOnChange() and SaveFileOnChange() in
BaseTools\Source\Python\Common\Misc.py don't use the dedicated
long file path API to handle the file path strings and cannot
support the long file path copy and save in windows. This patch
enhances them to support the long file path copy and save
correctly.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Signed-off-by: Steven Shi <steven.shi@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-08-23 11:26:23 +08:00
Fan, ZhijuX 0970a80583 BaseTools: Update incorrect variable name 'DataPile'
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=2093

The PlatformAutoGen object has a DataPipe property but no DataPile property
So change the variable name 'DataPile' to 'DataPipe' in BuildReport.py

This patch is going to fix that issue.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-08-20 14:15:47 +08:00
Shi, Steven 94459080c1 BaseTools: Improve the file saving and copying reliability
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=2079

The Basetool CopyFileOnChange() and SaveFileOnChange()
functions might raise the IOError occasionally when build
in Windows with multi-process and build cache enabled.
The CopyFileOnChange() and SaveFileOnChange() might be invoked
in multiple sub-processes simultaneously, and this patch adds
global locks to sync these functions invoking which can
harden their reliability.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Signed-off-by: Steven Shi <steven.shi@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-08-20 14:06:07 +08:00
Shi, Steven d01a998612 BaseTools: Add GenFds multi-thread support in build cache
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1923

Fix the issue that the GenFds multi-thread will build fail
if enable the build cache together.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Signed-off-by: Steven Shi <steven.shi@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-08-20 14:06:07 +08:00
Shi, Steven c340c5bd1b BaseTools: Change the [Arch][Name] module key in Build cache
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1951

Current build cache use the module's [self.Arch][self.Name]
info as the ModuleAutoGen object key in hash list and dictionary.
The [self.Arch][self.Name] is not safe as the module key because
there could be two modules with same module name and arch name in
one platform. E.g. A platform can override a module or library
instance in another different path, the overriding module can has
the same module name and arch name as the original one.
Directly use the ModuleAutoGen obj self as the key, because
the obj __hash__ and __repr__ attributes already contain the
full path and arch name.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Signed-off-by: Steven Shi <steven.shi@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-08-20 14:06:06 +08:00
Shi, Steven 56c786b0a9 BaseTools: Print first cache missing file for build cachle
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1925

When a module build cache miss, add support to print the first
cache missing file path and name.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Signed-off-by: Steven Shi <steven.shi@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-08-20 14:06:06 +08:00
Shi, Steven 0e7e7a264c BaseTools: Improve the cache hit in the edk2 build cache
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1927

Current cache hash algorithm does not parse and generate
the makefile to get the accurate dependency files for a
module. It instead use the platform and package meta files
to get the module depenedency in a quick but over approximate
way. These meta files are monolithic and involve many redundant
dependency for the module, which cause the module build
cache miss easily.
This patch introduces one more cache checkpoint and a new
hash algorithm besides the current quick one. The new hash
algorithm leverages the module makefile to achieve more
accurate and precise dependency info for a module. When
the build cache miss with the first quick hash, the
Basetool will caculate new one after makefile is generated
and then check again.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Signed-off-by: Steven Shi <steven.shi@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-08-20 14:06:05 +08:00
Feng, Bob C 76e12fa334 BaseTools: Fixed issue of incorrect Module Unique Name
https://bugzilla.tianocore.org/show_bug.cgi?id=2088
If there are more than one override instance for a same module,
the Module Unique Name is generated incorrectly.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Steven Shi <steven.shi@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-08-19 09:06:14 +08:00
Jin, Eric 48d8d4d80b BaseTools/Capsule: Tool to generate Windows Firmware Update Driver
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1837

The tool is designed to generate Windows Firmware Update Drivers,
the input is one drivername.cap with related parameters, the output
Windows Driver package are composed by drivername.cap, drivername.inf
and drivername.cat to update the single payload in device.

usage:
GenerateWindowsDriver [-h] [--output-folder OUTPUTFOLDER]
                      [--product-fmp-guid PRODUCTFMPGUID]
                      [--capsuleversion-dotstring CAPSULEVERSION_DOTSTRING]
                      [--capsuleversion-hexstring CAPSULEVERSION_HEXSTRING]
                      [--product-fw-provider PRODUCTFWPROVIDER]
                      [--product-fw-mfg-name PRODUCTFWMFGNAME]
                      [--product-fw-desc PRODUCTFWDESC]
                      [--capsule-file-name CAPSULEFILENAME]
                      [--pfx-file PFXFILE] [--arch ARCH]
                      [--operating-system-string OPERATINGSYSTEMSTRING]

Cc: Sean Brogan <sean.brogan@microsoft.com>
Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Kinney Michael D <michael.d.kinney@intel.com>
Signed-off-by: Eric Jin <eric.jin@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-08-15 20:13:55 +08:00
Sean Brogan e2aacac580 BaseTools/Capsule: Add WindowsCapsuleSupportHelper
Add initial version WindowsCapsuleSupportHelper.py
original source comes from
https://github.com/microsoft/mu_basecore/blob/dev/201905/BaseTools/Plugin/WindowsCapsuleSupportHelper/WindowsCapsuleSupportHelper.py
commit ebd9aad90dee3a3094b575928670cf81b8c685b5
Convert EOL to CRLF

Cc: Sean Brogan <sean.brogan@microsoft.com>
Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Signed-off-by: Eric Jin <eric.jin@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-08-15 20:13:05 +08:00
Bob Feng 08a54c9e0a BaseTools: Move buildoptions.py to Common folder
buildoptions is used by build.py and TargetTxtClassObj.py
Move buildoptions.py to Common folder so that
build tool and ECC tool work fine.

Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-08-14 13:54:49 +08:00
Rodriguez, Christian 82407bd129 BaseTools: Fix checking for Sources section in INF file
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1804

The check to see if [Sources] section lists all the header type
files of a module is missing the exclusion of source files that
fall under the scope of Package includes. This change adds the
exclusions.

Signed-off-by: Christian Rodriguez <christian.rodriguez@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Tested-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-08-14 10:59:08 +08:00
Feng, Bob C 2b4c07bc22 BaseTools: Add "-" in a re to parse line in .map file
https://bugzilla.tianocore.org/show_bug.cgi?id=2058

If there are multiple module instances with different
guid used in .dsc file, the temp path of the module will
contain 8-4-4-4-12 format guid string. So "-" need to be
added into the regular expression for parsing it correctly.

Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-08-14 10:24:36 +08:00
Feng, Bob C a000d5d1ca BaseTools: Fix incremental build bug
https://bugzilla.tianocore.org/show_bug.cgi?id=2078

This is a build tool regression issue that
is introduced by multiple-process-autogen.

The workspace timestamp value is not pass
to module autogen object correctly, so build tool
does not detect the change of .dsc and ignore the autogen.

This patch is to fix this issue.

Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Tested-by: Michael D Kinney <michael.d.kinney@intel.com>
2019-08-14 10:23:25 +08:00
Feng, Bob C a0792697bc BaseTools: Fixed an issue that build raise exception
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=2047

Build raise exception if there is libraryclass
missing in dsc file and --pcd build option is used.

Build should report the missing libraryclass but not
a call stack.

This patch is going to fix it.

Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-08-12 10:40:24 +08:00
Feng, Bob C 1a624dd7cf BaseTools: Enable block queue log agent.
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1875

To support Ctrl+S and Ctrl+Q, we enable block queue
for log.

Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Acked-by: Liming Gao <liming.gao@intel.com>
2019-08-09 23:15:55 +08:00
Feng, Bob C 4acae2b38e BaseTools: Add the support for python 2
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1875

python3 change the module name of Queue to queue.
python3 add a new log handler of QueueHandler.

This patch is to make Multiple process AutoGen
feature work for python2

Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Acked-by: Liming Gao <liming.gao@intel.com>
2019-08-09 23:15:55 +08:00
Feng, Bob C c60fb00f6c BaseTools: Move BuildOption parser out of build.py
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1875

Build tool supports user to specify the conf folder.
To make the build options be evaluated at the beginning
of launching build, extract the buildoption function
from build.py to a new .py file.

Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Acked-by: Liming Gao <liming.gao@intel.com>
2019-08-09 23:15:54 +08:00
Feng, Bob C 636ed13a7f BaseTools: Add LogAgent to support multiple process Autogen
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1875

AutoGen processes race the logfile. To resolve this issue,
this patch create a LogAgent thread in main process to write
the log content to console or file, Other process will send
the log content to the LogAgent.

Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Acked-by: Liming Gao <liming.gao@intel.com>
2019-08-09 23:15:54 +08:00
Feng, Bob C 3285fbda88 BaseTools: Add shared data for processes
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1875

Add shared data for autogen processes.

Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Acked-by: Liming Gao <liming.gao@intel.com>
2019-08-09 23:15:53 +08:00
Feng, Bob C 673d09a2dd BaseTools: Enable Multiple Process AutoGen
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1875

Assign the Module AutoGen tasks into multiple
sub process.

Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Acked-by: Liming Gao <liming.gao@intel.com>
2019-08-09 23:15:53 +08:00
Feng, Bob C e8449e1d8e BaseTools: Decouple AutoGen Objects
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1875

1. Separate the AutoGen.py into 3 small py files.
One is for AutoGen base class, one is for WorkspaceAutoGen class
and PlatformAutoGen class, and the one for ModuleAutoGen class.
2. Create a new class DataPipe to store the Platform scope settings.
Create a new class PlatformInfo to provide the same interface
as PlatformAutoGen. PlatformInfo class is initialized by
DataPipe instance.
Create a new class WorkspaceInfo to provide the same interface
as WorkspaceAutoGen. WorkspaceInfo class is initialized by
DataPipe instance.
3. Change ModuleAutoGen to depends on DataPipe, PlatformInfo and
WorkspaceInfo. Remove the dependency of ModuleAutoGen to PlatformAutoGen.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Steven Shi <steven.shi@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Acked-by: Liming Gao <liming.gao@intel.com>
2019-08-09 23:15:52 +08:00
Feng, Bob C 197ca7febf BaseTools: Add functions to get platform scope build options
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1875

These functions are used for get platform scope
build options. They will be used in later patches.

Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Acked-by: Liming Gao <liming.gao@intel.com>
2019-08-09 23:15:52 +08:00
Feng, Bob C 2927a6fd99 BaseTools: Split WorkspaceAutoGen._InitWorker into multiple functions
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1875

The WorkspaceAutoGen.__InitWorker function is too long, it's hard
to read and understand.
This patch is to separate the __InitWorker into multiple small ones.

Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Acked-by: Liming Gao <liming.gao@intel.com>
2019-08-09 23:15:51 +08:00
Feng, Bob C db01c8e3d8 BaseTools: Singleton the object to handle build conf file
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1875

The build config files are target.txt, build rule, tooldef
During a build, the config is not changed, so the object to
handle them need to be singleton.

Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Acked-by: Liming Gao <liming.gao@intel.com>
2019-08-09 23:15:51 +08:00
Feng, Bob C 83d6207f99 BaseTools: Fixed a typo in Trim.py
This is a regression issue introduced
by commit 307e1650be

This patch is to fix this issue.

Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Philippe Mathieu-Daude <philmd@redhat.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-08-02 00:18:00 +08:00
Matthew Carlson 101dc9e257 BaseTools:Added arch output to build report
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=2016

Added arch output to build report so it's easy to tell
which arch is being generated. Useful when multiple versions
of a single module is being emitted for multiple archs.

This patch is going to Added arch output

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-08-01 15:36:49 +08:00
Bob Feng 8ddec24dea BaseTools:Updata the output encoding of the Popen function
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=2015

Not all output works in utf-8, so change the encoding to
the default

This patch is going to fix that issue.

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-08-01 15:36:48 +08:00
Max Knutsen 307e1650be BaseTools:Add extra debugging message
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=2014

Add extra debugging to improve error identification.
Error while processing file if the file is read incorrectly

This patch is going to fix that issue.

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Philippe Mathieu-Daude <philmd@redhat.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-08-01 15:36:48 +08:00
Sean Brogan 76912197fa BaseTools:replace the chinese quotation mark with unicode "
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=2013

chinese quotation mark are used in the file
This patch is going to fix that issue.

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Philippe Mathieu-Daude <philmd@redhat.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-08-01 15:36:47 +08:00
Feng, Bob C 959791dd6c BaseTools: Sort file list in Makefile
This patch is going to sort the file list in generated
Makefile. This change make the autogen makefile easy to
compare.

Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Philippe Mathieu-Daude <philmd@redhat.com>
2019-07-25 09:00:59 +08:00
Feng, Bob C bb824f685d BaseTools: Fixed the mis-using strip() function issue.
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=2003

lstrip(parameter) do the match based on the char
in parameter but not only the whole parameter string.

In GenMake line 1082,
CmdSign.lstrip('/Fo') will strip the '/' or
'F' or 'o' on the left of CmdSign. This is not expected.

This patch is going to fix such issue.

Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-07-22 12:06:03 +08:00
Feng, Bob C 5f89bcc460 BaseTools: Create ".cache" folder when initialize Build object
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1986

Create "Conf/.cache" folder as early as possible
so that the later code do need to check
if it exits and then create it.

Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Reviewed-by: Michael D Kinney <michael.d.kinney@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-07-22 09:21:14 +08:00
Cole Robinson eebc135ffb BaseTools: Fix python3.8 SyntaxWarning
Building with python3.8 shows a warning like:

SyntaxWarning: invalid escape sequence \(
  GuidName = re.compile("\(GUID=[-a-fA-F0-9]+")

It seems harmless, but it's easy enough to fix: mark the string as
raw with the 'r' prefix like is used elsewhere in the file

Signed-off-by: Cole Robinson <crobinso@redhat.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-07-15 09:09:25 +08:00
Jiewen Yao a79841a024 BaseTools: Add HOST_APPLICATION module type.
It can be used to indicate a module can be build to run
as OS application and run in OS environment.

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Signed-off-by: Xiaoyu Lu <xiaoyux.lu@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-07-12 21:06:57 +08:00
Bob Feng 2d100d1d73 BaseTools: Fixed the issue when ToolDefinitionFile is not generated
ToolDefinitionFile is generated by PlatformAutoGen.ToolDefinition()
Code assume ToolDefinition is always called before using
ToolDefinitionFile, but in some cases, it's not true.

This patch is to fix this issue.

Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-07-12 17:11:58 +08:00
Feng, Bob C 688ec2d006 BaseTools: Detect the change of env variable used in tooldef.txt
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=1921

Add TOOLS_DEF.ARCH file to all workspace meta files list.
TOOLS_DEF.ARCH include the evaluated Tool definition information
which is filtered by current tool_chain.

With this change, when the environment variable which
is used in ToolDef.txt is changed, build tool will
rebuild the platform.

Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Steven Shi <steven.shi@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-07-10 09:29:43 +08:00
Feng, Bob C 1d06b46cf1 BaseTools: Fixed the issue of the CFlag for compile PcdValueInit.c
This issue happens when using Structured PCD.
Build tool use set to store the cflags for compile PcdValueInit.c,
that is the order of cflags is disorder.

This patch make -U, /U flags appear before -D, /D

Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-07-10 09:26:18 +08:00
Antoine Cœur 2d53d54a5b BaseTools: Fix various typos
Fix various typos in BaseTools.

Signed-off-by: Cœur <coeur@gmx.fr>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-07-08 08:59:29 +08:00
Jin, Eric 104a1aa19b BaseTools/Capsule: Supports multiple payloads and drivers in capsule
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1834

1)Add arguments "--embedded-driver" to support embedded driver
in command line.
2)Add arguments "--update-image-index" to identify ImageIndex
within the device in command line.
3)Add arguments "-j JSONFILE" to support multiple payloads and
embedded drivers with JSON file.

The update is in a backwards compatible manner, so all command
line options to support single payload are still supported. But
all the options associated with multiple payloads should be
provided in a JSON file.

Cc: Bob Feng <bob.c.feng@intel.com>
Signed-off-by: Eric Jin <eric.jin@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-06-28 11:39:06 +08:00
Bob Feng b8ac0b7f28 BaseTools: Move Build Cache related function out of CreateAsBuiltInf
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1932
There are two functions in current CreateAsBuiltInf, Copy Binary files
to build cache folder and create asbuild inf file.

This patch is to separate UpdateBuildCache and CreateAsBuiltInf into
two functions.

Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Steven Shi <steven.shi@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Christian Rodriguez <christian.rodriguez@intel.com>
Reviewed-by: Steven Shi <steven.shi@intel.com>
Reviewed-by: Jaben Carsey <jaben.carsey@intel.com>
2019-06-25 09:23:57 +08:00
Bob Feng dc174cd9e5 BaseTools: refine CreateAsBuiltInf function
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1932
ModuleAutogen.CreateAsBuiltInf IsOnlyCopy parameter
and related function can be moved out of CreateAsBuiltInf,
so that CreateAsBuiltInf is more clean.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Steven Shi <steven.shi@intel.com>
Cc: Christian Rodriguez <christian.rodriguez@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Steven Shi <steven.shi@intel.com>
Reviewed-by: Jaben Carsey <jaben.carsey@intel.com>
2019-06-25 09:23:53 +08:00
Fan, ZhijuX 370f16c548 BaseTools:Add import in FvImageSection
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=1872

Since MultipleWorkspace is not imported in FvImageSection,
an error of "global name 'MWS' is not defined" appeared.

This patch is going to fix that issue.

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-06-24 17:02:53 +08:00
Shi, Steven 2378ea5515 BaseTools:Introduce CopyFileOnChange() function to copy cache files
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=1894

Basetool need a CopyFileOnChange() function to avoid cache
file writing race in multi-thread build. Some platforms
build fail with file IO writing race issue when the
build cache is enabled to store cache files in multi-threads.
This is because common same library cache files (e.g. some
libs in MdePkg) can be stored by many different driver modules'
build threads at same time. Current build cache need a function
to check whether the same cache file already exist, and only
copy source file if it is different from the destination file.

This patch introduces an atomic copy function to avoid duplicated
cache files copy.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Christian Rodriguez <christian.rodriguez@intel.com>
Signed-off-by: Steven Shi <steven.shi@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-06-17 17:00:46 +08:00
Shi, Steven 17b082ce67 BaseTools: Cannot store library cache of different arch together
https://bugzilla.tianocore.org/show_bug.cgi?id=1895

Build cache cannot store cache for the same library modules
in different arch together. E.g. Both the below IA32 and X64
arch BaseLib caches should exist after build Ovmf3264, but now
only the one in X64 arch exist.
The reason is the current Basetool use a set() to same all
library AutoGen objects, but the different arch lib AutoGen
objects have same __hash_ value which comes from the lib
MetaFile(The path of module file):
    def __hash__(self):
        return hash(self.MetaFile)

So the different arch lib AutoGen objects are duplicated one
to the set() and only one can exist. This is why the Basetool
can only store one arch cache for library.

This patch adds the arch string into the PlatformAutoGen and
ModuleAutoGen __hash_ definitions and ensure the different
platform and module AutoGen objects have different __hash_ values.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Christian Rodriguez <christian.rodriguez@intel.com>
Signed-off-by: Steven Shi <steven.shi@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-06-17 17:00:45 +08:00
Shi, Steven 04df18535c BaseTools:Build Cache output notification message
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=1868

Build need output the cache miss or hit notification
message when consume the build cache. Current build does not
output any message which is not clear for user to know
whether the module built result is from cache or not.

This patch adds message about the cache miss or hit when
build consumes the cache.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Zhiju.Fan <zhijux.fan@intel.com>
Signed-off-by: Steven Shi <steven.shi@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-06-17 17:00:45 +08:00
Shi, Steven ede54e3588 BaseTools:Build cache cannot store the cache files for library package
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1867

Current build cache cannot store the cache for library package.
build -p MdePkg\MdePkg.dsc -a IA32 -b DEBUG -t VS2015x86 --hash
--binary-destination=BinCache
After build, the expected result is the BinCache folder is generated
and the MdePkg build cache files (e.g. .hash and .lib) are stored in
the BinCache folder. But the BinCache folder is not generated at all.

This patch is going to fix that issue.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Zhiju.Fan <zhijux.fan@intel.com>
Signed-off-by: Steven Shi <steven.shi@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-06-17 17:00:45 +08:00
Rodriguez, Christian 48b0bf6476 BaseTools: Refactor hash tracking after checking for Sources section
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1804

After adding a check to see if [Sources] section lists all the header
type files of a module, track module and library hashes for --hash
feature. If above check is not in compilance for a library or module,
force hash invalidation on that library or module.

Signed-off-by: Christian Rodriguez <christian.rodriguez@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-06-10 19:48:51 +08:00
Rodriguez, Christian 1fa6699e6c BaseTools: Add a checking for Sources section in INF file
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1804

Add a check to see if [Sources] section lists all the header type
files of a module. Performance impact should be minimal with this patch
since information is already being fetched for Makefile purposes. All
other information is already cached in memory. No extra IO time is needed.

Signed-off-by: Christian Rodriguez <christian.rodriguez@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-06-10 19:48:51 +08:00
Fan, ZhijuX 04797875d1 BaseTools:Make BaseTools support new rules to generate RAW FFS FILE
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=1765

If RAW FFS File Rule has no section for its data.For RAW FFS File,
directly call GenFfs tool to generate FFS file.

Ffs Rule:
[Rule.Common.USER_DEFINED.MicroCode]
  FILE RAW = $(NAMED_GUID) {
        $(INF_OUTPUT)/$(MODULE_NAME).bin
  }
[Rule.Common.USER_DEFINED.LOGO]
  FILE RAW = $(NAMED_GUID) {
                       |.bmp
  }

As shown in the rule above,if SectionType and FileType not defined,
FFS files are generated directly, and no other type of file is
generated.

The patch is to make the BaseTools support these two rules

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-06-10 19:48:50 +08:00
Steven Shi b6cfa686e4 BaseTools:Update binary cache restore time to current time
https://bugzilla.tianocore.org/show_bug.cgi?id=1742

Current Binary Cache doesn't update the restored file
creation and modification times to the current time.
Preserve the new restored file creation time as old
cached time might has potential issue to block the make
to build updated files based on the time stamp.
Enhance to update the restored file creation time to
current time.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Zhiju Fan <zhijux.fan@intel.com>
Signed-off-by: Steven Shi <steven.shi@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-05-31 19:46:18 +08:00
Steven Shi 3a0b998bf3 BaseTools:Extend the binary cache to support library cache
https://bugzilla.tianocore.org/show_bug.cgi?id=1797

Current binary cache doesn't support to save and restore
the library module. If a driver module cache miss happen,
all its dependency library modules need rebuild which
is very time-consuming. This patch is to extend the binary
cache to support library.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Christian Rodriguez <christian.rodriguez@intel.com>
Signed-off-by: Steven Shi <steven.shi@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-05-31 19:46:16 +08:00
Fan, ZhijuX c0fd7f734e BaseTools:Change the path of the file that Binary Cache
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=1806

Current Binary Cache use the PlatformInfo.Name to distinguish
the different platform cache paths in the database folder.

But the different platforms might use the same PLATFORM_NAME
in the DSC files. Different platform caches might overlay each
other inside Binary Cache database if they use same platform name

Use the PlatformInfo.Platform.OutputDirectory instead of the
PlatformInfo.Name can solve the problem.

This patch is going to fix that issue.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Steven Shi <steven.shi@intel.com>
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-05-27 08:59:04 +08:00
Bob Feng e812a812c1 BaseTools: Remove './SecMain' from 'run' target
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1561

This patch is related to commit 9ca1f1
It's to remove the second .SecMain call from 'run' target.

Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Michael D Kinney <michael.d.kinney@intel.com>
2019-05-24 15:28:42 +08:00
Feng, Bob C 9ca1f1446a BaseTools: Remove './SecMain' from 'run' target
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1561

When a target of 'run' is passed into build.py,
BaseTools unconditionally attempts to execute the
application called './SecMain' in the build output directory.

This behavior applies to the Nt32Pkg which is being
replaced with features in the EmulatorPkg.

Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Michael D Kinney <michael.d.kinney@intel.com>
2019-05-22 09:15:56 +08:00
Fan, ZhijuX c1387446ed BaseTools:Update mailing list address in BaseTools error messages
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=1815

The edk2 source tree contains four instances of the outdated mailing
list address "edk2-devel@lists.01.org".
I created a new variable, MSG_EDKII_MAIL_ADDR, to receive the
new email address and replaced the old one with this variable

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-05-22 09:15:55 +08:00
Rodriguez, Christian a7ef158b07 BaseTools: Library hashing fix and optimization for --hash feature
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1788

In V3: Must generate hashes before attempting to copy from cache for
hash verifcation
In V2: Build failure caused by passing incorrect boolean parameter to
SaveFileOnChange(). Fixed for patch instances.

Library hashing is now supported by the --hash feature. The --hash
feature implementation assumed that the hashing could be done in
place once per module, but that isn't true for libraries due to the
fact that they are built as dependencies. So on a clean build, we now
generate the .hash after the library dependencies are complete.
Added early escape as optimization, if hash already exists in memory.

Signed-off-by: Christian Rodriguez <christian.rodriguez@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-05-21 13:18:06 +08:00
Michael D Kinney 66b845ae06 BaseTools: Fix private includes for FILE_GUID override
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1648

When a module in a DSC file uses a FILE_GUID override in the
module scoped <Defines> section, a copy of the modified INF file
is placed in the Conf/.cache directory.  The check for private
includes uses the INF path to determine if the module is allowed
to use the private includes.  Since the INF path in this case is
not in any package, this check always fails, and no private
include paths are possible.

The fix is to keep both the OriginalPath and the new Path in
the PathClass object, and always use the OriginalPath to see if
the module INF is in the package with private includes.

Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-05-21 13:14:14 +08:00
Fan, ZhijuX 6b74ccf0f8 BaseTools:Enables FFS to support the GUID value format
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=1768

Per FDF spec, FFS Rule can specify the hard code FILE Guid value.
But,BaseTools reports the build failure.It should be fixed.

This patch is going to fix that issue.

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-05-10 17:07:07 +08:00
Fan, ZhijuX afe8c4110c BaseTools: Correct the value assignment for StructurePcd
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1752

This patch is to fix the code bug in StructurePcd overall
value assignment logic. If a Pcd Array size is fixed but the
size of actual value in Dsc or Dec is bigger than the Pcd
array size, the tool will report error about such setting and
stop build.

The patch is tested minplatform, Ovmf, structure pcd regression test
These tests are build pass. The patch also tested the following cases.
Our cases focused on PcdArraySize. 1.flexiable PcdArraySize.
 2.Fixed PcdArraySize, Pcd overall value exceeds the size of PcdArray.
 3.Fixed PcdArraySize, Pcd overall value Not exceeds the size of PcdArray.

Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-05-10 17:06:44 +08:00
Bob Feng c648e90587 BaseTools: Fix the issue that build report missing Module Size
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1786

commit b1e27d changed GENFW output folder from DEBUG to OUTPUT,
but that commit missed updating the buildreport.py for FwReportFileName.

This patch is going to fix this issue.

Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-05-09 15:03:30 +08:00
Fan, ZhijuX 26e2b295ec BaseTools:Remove unused BaseTools\Source\Python\UPT\Dll
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=1735

BaseTools runs from python source. It doesn't depend on
dll any more. So, DLL can be removed.

This patch is going to fix that issue.

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-05-08 09:41:42 +08:00
Fan, ZhijuX 0cb3f77153 BaseTools:ECC report errors on account of analyze special characters
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=1751

In case that a C function body contains the string of L'', L'\"',
L"\"", L''', L""", L"\"\"", L"\"^", L" \"", L"\" \"", ('L",\\\""')
ECC tool running under python3 interpreter will report error.
The antlr4 module misidentified this character

This patch is going to fix that issue.

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-05-08 09:41:42 +08:00
Fan, ZhijuX 8b4b2fb9a1 BaseTools:corrected error message for the DatumType error
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=1762

For structure Pcd, Its DatumType may be self-defined,
If the PCD DatumType format is incorrect,
this PCD DatumType error message is inappropriate.

This patch corrects the error message.

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-05-08 09:41:41 +08:00
Fan, ZhijuX 273190e189 BaseTools:Fixed an issue where the order of GuidS changed in guid.xref
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=1749

Add content to dsc
[PcdsPatchableInModule.common]
  gEfiMdeModulePkgTokenSpaceGuid.test1|FALSE

The order of file Guid.xref will change after increment build.
The root cause is set() is used in Get all the PCDS
the order of the data may change if set() is used

This patch is going to fix that issue.

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-05-01 10:57:05 +08:00
Bob Feng c9b3fe1586 BaseTools: Support customized compiling command
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1737
User may add other commands before CC command from build_rule.txt
for specific purpose. It worked fine before commit 05217d210e.

This patch is going to fix the bug in commit 05217d210e to
support customized CC command.

Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-04-28 09:11:27 +08:00
Christian Rodriguez 56008eb7f6 BaseTools: Hash false success.. minor change in hash invalidation
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1692

Change part of the hash error handling to invalidate hashes in the
cache destination not the cache source.

Signed-off-by: Christian Rodriguez <christian.rodriguez@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-04-22 16:50:26 +08:00
Fan, ZhijuX 35c2af00d8 BaseTools:fixed the incorrect autogen makefile which cause build failure.
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=1729

On some build environment, build fails but on the other build machines,
build success. This is the regression issue introduced by commit
 05217d210e

As Dict is unordered, an error occurs when extract the index of the Dict
in the order of the keys after the creation of a new item.
Keys are indexed inconsistently before and after adding a new item.
The logic of the program is to store the key's corresponding index as
reference data in the MakeFile and use it as part of the macro.
The data model is: $(LIST_%d) % Dict.keys().index(Key)
So for now, use OrdereDict instead of Dict.

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-04-22 16:50:25 +08:00
Rodriguez, Christian 452b5ad61d BaseTools: Enhance Bin Cache database to support save the cache
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1726

V2: Added the platform name to the path
Add more level sub-directories in the database to support save
the cache for multiple platforms with multiple tool-chains and
targets, just like edk2 Build output.

Signed-off-by: Christian Rodriguez <christian.rodriguez@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-04-22 16:50:24 +08:00
Yunhua Feng 0b836855fb BaseTools: Sometime write file not immediate to disk
BZ: https://bugzilla.tianocore.org/process_bug.cgi

On Windows OS, sometime the generated file is not immediate saved to disk.
When run nmake, prompt AutoGen.h not found, and stop build.
Below blog shows Write-Replace to fix it. This patch uses this way to write
temp file, then rename the temp file to the real file.
https://blog.gocept.com/2013/07/15/reliable-file-updates-with-python/

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Signed-off-by: Yunhua Feng <yunhuax.feng@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2019-04-22 10:05:52 +08:00
Rodriguez, Christian 0eccea3fbe BaseTools: Hash false success with back to back builds
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1692

Add error handling to the --hash feature so that hash files
are invalidated when a build error occurs.

Signed-off-by: Christian Rodriguez <christian.rodriguez@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-04-16 13:14:15 +08:00
Fan, ZhijuX 87bfb9bcb7 BaseTools:Coding problems cause can not encode FMP Auth Header
GenerateCapsule: error: can not encode FMP Auth Header
The system cannot find the file specified.

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-04-16 13:14:14 +08:00
Fan, ZhijuX 05217d210e BaseTools:Enable the /MP option of MSVC compiler
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=1672
The /MP option of MSVC compiler can reduce the total time to compile the
source files on the command line.

This patch is going to enable this MSVC option in BaseTools.

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-04-16 13:14:13 +08:00
Christian Rodriguez db4d47fd3a BaseTools: Fix corner-cases of --hash feature
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1680

Re-use libraries, since they have already been hashed.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Christian Rodriguez <christian.rodriguez@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-04-10 13:32:14 +08:00
Rodriguez, Christian f2b5e04aca BaseTools: Fix corner-cases of --hash feature
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1680

Respect artifact location within directory structure.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Christian Rodriguez <christian.rodriguez@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-04-10 13:32:13 +08:00
Rodriguez, Christian 1b8caf0d87 BaseTools: Fix corner-cases of --hash feature
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1680

Re-order hashing operations so we don't do redundant hashes.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Christian Rodriguez <christian.rodriguez@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-04-10 13:32:12 +08:00
Rodriguez, Christian 2914e8153d BaseTools: Fix corner-cases of --hash feature
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1680

Consider modules with .inc source files as Binary Modules
and do not Skip by hash.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Christian Rodriguez <christian.rodriguez@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-04-10 13:32:11 +08:00
Feng, Bob C b1e27d175a BaseTools: Fixed issue in MultiThread Genfds function
https://bugzilla.tianocore.org/show_bug.cgi?id=1450
In the Multiple thread Genfds feature, build tool generates
GenSec, GenFFS command in Makefile.

The Non-Hii Driver does not generate .offset file for uni string offset,
but the build tool has not knowledge about this in autogen phase. So
in this patch, I add a check in Makefile for GenSec command. If the GenSec
input file does not exist, the GenSec will not be called. And if GenSec
command is not called, its output file, which is also the input file of
GenFfs command, will also not exist.So for GenFfs command,
I add a new command parameter -oi which means
the input file is an optional input file which would not exist. so
that I can generate GenFfs command with "-oi" parameter in Makefile.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-04-10 13:32:10 +08:00
Michael D Kinney 2e351cbe8e BaseTools: Replace BSD License with BSD+Patent License
https://bugzilla.tianocore.org/show_bug.cgi?id=1373

Replace BSD 2-Clause License with BSD+Patent License.  This change is
based on the following emails:

  https://lists.01.org/pipermail/edk2-devel/2019-February/036260.html
  https://lists.01.org/pipermail/edk2-devel/2018-October/030385.html

RFCs with detailed process for the license change:

  V3: https://lists.01.org/pipermail/edk2-devel/2019-March/038116.html
  V2: https://lists.01.org/pipermail/edk2-devel/2019-March/037669.html
  V1: https://lists.01.org/pipermail/edk2-devel/2019-March/037500.html

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-04-09 09:10:20 -07:00
Fan, ZhijuX 1c27ec4236 BaseTools:Coding problems caused by special characters
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=1670

During BaseTools compiling under Chinese or Japanese
language Windows, python exception occurring.
UnicodeDecodeError: 'ascii' codec can't decode byte 0xbd
in position 3528: ordinal not in range(128)

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-04-01 13:37:19 +08:00
Fan, ZhijuX 58742d7945 BaseTools:Trim will trig exception when input asl UTF8 format file
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=1641

The command trim --asl-file -o test.i UTF8.asl will trig the exception.
Trim tool should report error message for unsupported UTF8 file instead
of the exception.

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-04-01 13:37:17 +08:00
Feng, Bob C a004d17d01 BaseTool: Fixed an issue of Structure PCD
https://bugzilla.tianocore.org/show_bug.cgi?id=1665

Build fail when the structure pcd has
member of flexible size array

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-03-27 10:18:43 +08:00
Feng, Bob C 2f2c51acfb BaseTools: Remove the logic SourceOverridePath
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1350
SOURCE_OVERRIDE_PATH is for EDK component INF files.
The corresponding logic should be removed.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-03-24 16:12:38 +08:00
Feng, Bob C f293841f93 BaseTools: Remove EDKI related logic from Trim tool
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1350

Remove EDKI related logic from Trim tool.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-03-24 16:12:37 +08:00
mjohn4 b0189eac00 BaseTools: Explicitly close files after readlines
Rework some file open().readlines to open, readlines, close.
This prevents excessive file handles being open at the same time,
which may be a problem with alternative python environments.

Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Michael Johnson <michael.johnson@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-03-15 15:38:59 +08:00
Jiewen Yao d868846a4e BaseTool/Build: Add --disable-include-path-check.
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1620

This option is added to disable the include path check
for outside of package.
The original purpose of thie check is to make sure EDK II
modules must not reference header files outside of the packages
they depend on or within the module's directory tree.

However, we do see the usage to build EDKII as executable running
in the operating system which requires include path to outside.
For example, EmulatorPkg. The current solution (soft link) is
weird hack - EmulatorPkg\Unix\Host\X11IncludeHack.

With this solution, this can be supported easily.

The patch is validated with and without --disable-include-path-check.
If user does not use --disable-include-path-check, the build will fail
with outside path in the include path.
If user uses --disable-include-path-check, the build will pass
with outside path in the include path.

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jiewen Yao <jiewen.yao@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-03-15 13:29:29 +08:00
Liming Gao 3a4698202c Revert "BaseTools:BaseTools supports to the driver combination."
This reverts commit 838bc257ba.
After further evaluation, there are the unclear behavior in for the
driver combination feature. To not impact Q1 stable tag, remove it first.
1. If the drivers to be combined have the different PCD or library instance
   setting, build should not combine them and report build break. But this
   commit doesn't consider this case.
2. When start the sub driver fail, continue to start other sub driver. This
   behavior is required to be clarifed in build spec.
3. Unload the sub driver when the combined driver start fail. This case need
   to call the sub driver unload function for the driver start fail only.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-02-28 10:01:53 +08:00
Fan, ZhijuX de72c9d1da BaseTools:The BOM character is processed when python reads a file
When python3 reads an XML file it will parse the file in error
if the file has a BOM

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-02-27 09:39:09 +08:00
Feng, Bob C dc082d5029 BaseTools: Fix a Eot issue.
FirmwareVolume.UnDispatchedFfsDict is mutated during iteration,
convert the FirmwareVolume.UnDispatchedFfsDict.keys() to a new list
 to resolve this problem.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-02-26 09:10:23 +08:00
Feng, Bob C dde29b9ca5 BaseTools: Eot failed when enable python3
The Eot will report error when python3 enabled.
We replaced sdict with collections.OrderedDict in python3
patch set, but the sdict implement "append" method which is not
implemented in collections.OrderedDict.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-02-26 09:10:23 +08:00
Feng, Bob C 2cfb9d0f1a BaseTools: Fix a bug about Structure PCD
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1535
If there is Hii Structure Pcd, build will fail, root cause is that
there is an incorrect variable access method used in code.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-02-22 15:47:15 +08:00
Fan, ZhijuX 29c2de1272 BaseTools:Build fail if define [DEPEX] in library inf
When define [DEPEX] in lib inf to build, it will fail and report"
gUefiOvmfPkgTokenSpaceGuid.test1 used in [Depex] section should be
used as FixedAtBuild type and VOID* datum type in the module."
But we define this PCD to FixedAtBuild type and VOID* datum type indeed.

DEC:
[PcdsFixedAtBuild]
gUefiOvmfPkgTokenSpaceGuid.test1 |
{GUID("4096267b-da0a-42eb-b5eb-fef31d207cb4")}|VOID*|0x3c

DSC:
add pcd under lib inf as below:

NULL|TestPkg/TestLib/TestLib.inf
 <PcdsFixedAtBuild>
    gUefiOvmfPkgTokenSpaceGuid.test1 | {GUID(gUefiOvmfPkgTokenSpaceGuid)}

Lib inf:(TestPkg/TestLib/TestLib.inf)
[Depex]
gUefiOvmfPkgTokenSpaceGuid.test1
[FixedPcd]
gUefiOvmfPkgTokenSpaceGuid.test1

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-02-22 15:47:14 +08:00
Feng, Bob C f747640bf3 BaseTool: Fixed incremental rebuild issue.
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1540

This issue in introduced by commit
d943b0c339

To convert bytes to string, we need to use bytes.decode()
instead of using str(bytes).

If the source file is not a txt file, ignore that file.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-02-22 15:47:14 +08:00
Jiewen Yao 4247c24fba BaseTool/GenC: Fix build error when type is BASE or USER_DEFINED.
https://bugzilla.tianocore.org/show_bug.cgi?id=1544

1) The GenC tool does not include the constructor/destructor for
USER_DEFINED module. It should be included.
Only constructor/destructor with BASE type is included here.
The constructor/destructor with PEI/DXE/SMM type is still excluded
to keep original behavior.

2) The GenC tool includes the UnloadImage code for BASE module.
It should NOT be included.

3) The GenC tool uses EFI_STATUS and ASSERT_EFI_ERROR for BASE type.
It should use RETURN_STATUS and ASSERT_RETURN_ERROR.

4) The GenC tool miss DebugLib.h for BASE or USER_DEFINED module
AutoGen.c. Only Base.h is there. It should add Library/DebugLib.h.

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jiewen Yao <jiewen.yao@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-02-22 12:20:36 +08:00
Fan, ZhijuX 8a64c7ea00 BaseTools:PCD value error in structure pcd sku case.
Defined 2 PCDs(Test4 & Test401) and 2 SKUs(DEFAULT & _),
then set "SKUID_Defines" to ALL, for FixedAtBuild
gEfiStructuredPcdPkgTokenSpaceGuid. Test401 in this case,
its value should get from "Default" SKU, not from "_" SKU,
but we does not set value in SKU "_" in dsc, so Test401
should only display the value get from dec.

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-02-20 13:16:58 +08:00
Fan, ZhijuX e8d13fef90 BaseTools:Fixed build failure when specifying multiple BUILDTARGET
With Python3, the dict.value() method returns an iterator.
If a dictionary is updated while an iterator on its keys is used,
a RuntimeError is generated.
Converting the iterator to a list() forces a copy of the mutable
keys in an immutable list which can be safely iterated.

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-02-20 13:16:57 +08:00
Feng, Bob C 7da3ed89d4 BaseTools: replace Sdict with OrderedDict in UPT
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1504

V2: import OrderredDict as Sdict in Misc.py instead
updating all the py files.

Sdict class is removed on the commit
174a9d3cc8
but there are still some py files import it.

This patch is to use collections.OrderedDict to replace Sdict.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Jaben Carsey <jaben.carsey@intel.com>
2019-02-20 13:16:56 +08:00
Fan, ZhijuX 838bc257ba BaseTools:BaseTools supports to the driver combination.
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=1520

To save the image size without the compression, more than
one drivers can be combined into single one. When more than
one drivers are combined, their depex will be AND together.
Below is the example to combine BootManagerPolicyDxe into
DriverHealthManagerDxe.

Besides this patch, BaseTools also needs to check the module
type and make sure all module type are same. Otherwise,
BaseTools will report the error.
DRIVER INF has the parameter ENTRY_POINT
LIBRARY INF has the parameter LIBRARY_CLASS

Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-02-20 13:16:56 +08:00
Feng, Bob C 68c67d3a2a BaseTools: Fixed a code bug for Pcd Array.
For example, PCD gUefiOvmfPkgTokenSpaceGuid.Test001 datatype is Array:
TEST1[2]
and the filed TEST1UINT64ARRAY in TEST1 is also an array:
UINT64 TEST1UINT64ARRAY[2];

Then the following filed assignment in DSC will cause build failure.
gUefiOvmfPkgTokenSpaceGuid.Test001[0].TEST1UINT64ARRAY|{'A','B'}

The root cause is build tool generate incorrect PcdValueInit.c File.

This patch is going to fix this issue.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-02-19 20:32:44 +08:00
Feng, Bob C c0ebd9d30f BaseTools: Fixed a bug in Vpd handling
If there are multiple sku used in a platform and
gEfiMdeModulePkgTokenSpaceGuid.PcdNvStoreDefaultValueBuffer PCD
is used, build will fail.

This is a regression issue introduced by the commit:
5695877ec8 which only update the
Vpd offset for default SKU but not other SKUs.

This patch is going to fix this issue.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-02-19 20:32:43 +08:00
Feng, Bob C 97f003236f BaseTools: Fix a ParseDevPathValue function issue.
In ParseDevPathValue, the binary data need to convert to string
for python3 and python2 compatiblity.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-02-19 20:32:43 +08:00
Fan, ZhijuX 7b6a8c9cf2 BaseTools:Fix a ECC issue
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=1522

A property error occurred because the property of the
function was not defined. a property is now redefined.

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-02-19 20:32:42 +08:00
Feng, Bob C 9108039558 BaseTools: Add MaxSizeUserSet to Pcd deepcopy function
MaxSizeUserSet is missing in Pcd deepcopy function.
This patch is to add it back to Pcd deepcopy function.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-02-19 20:32:42 +08:00
Feng, Bob C c417c1b33d BaseTools: Fixed an issue about StructurePcd
If use a structure pcd in fdf, build tool crash
This is a regression issue introduced by py3 patch set.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-02-18 09:11:48 +08:00
Feng, Bob C 5f094268e0 BaseTools: Fix the build report issue about Structure PCD
https://bugzilla.tianocore.org/show_bug.cgi?id=1472
build report use incorrect method to parse DynamicDefault/DynamicExDefault
and DynamicVpd/DynamicExVpd structure Pcd value.

This patch is to fix this issue.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-02-18 09:11:48 +08:00
Feng, Bob C c196d1d129 BaseTools: Fixed a build report issue.
Generate report fail when -Y EXECUTION_ORDER in build command.
This patch is going to fix this issue.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-02-18 09:11:47 +08:00
Feng, Bob C 8059cd2483 BaseTools: Correct the error message for UPT
This patch is going to correct the error message
for UPT.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Hess Chen <hess.chen@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-02-18 09:11:46 +08:00
Antoine Coeur fb0b35e05f BaseTools: Various typo
Various typo in BaseTools.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Coeur <coeur@gmx.fr>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-02-14 10:02:28 +08:00
Philippe Mathieu-Daud? bac50dfbcd BaseTools: Fix build failure when specifying multiple BUILDTARGET
With Python3, the dict.value() method returns an iterator.
If a dictionary is updated while an iterator on its keys is used,
a RuntimeError is generated.
Converting the iterator to a list() forces a copy of the mutable
keys in an immutable list which can be safely iterated.

Commit f8d11e5a4a converted various uses but missed one:
When specifying multiple BUILDTARGET, the first target builds
successfully, but then the PGen.BuildDatabase._CACHE_ dictionary is
updated, and accessing the next target triggers a RuntimeError.

Convert this iterator to an immutable list, to solve this build error:

    $ build -a IA32 -t GCC5 -b RELEASE -b NOOPT -p OvmfPkg/OvmfPkgIa32.dsc
    [...]
    Processing meta-data ...
    build.py...
     : error C0DE: Unknown fatal error when processing [OvmfPkg/OvmfPkgIa32.dsc]

    (Please send email to edk2-devel@lists.01.org for help, attaching following call stack trace!)

    (Python 3.5.3 on linux) Traceback (most recent call last):
      File "BaseTools/BinWrappers/PosixLike/../../Source/Python/build/build.py", line 2387, in Main
        MyBuild.Launch()
      File "BaseTools/BinWrappers/PosixLike/../../Source/Python/build/build.py", line 2141, in Launch
        self._MultiThreadBuildPlatform()
      File "BaseTools/BinWrappers/PosixLike/../../Source/Python/build/build.py", line 1921, in _MultiThreadBuildPlatform
        self.Progress
      File "BaseTools/Source/Python/AutoGen/AutoGen.py", line 304, in __init__
        self._InitWorker(Workspace, MetaFile, Target, Toolchain, Arch, *args, **kwargs)
      File "BaseTools/Source/Python/AutoGen/AutoGen.py", line 477, in _InitWorker
        for BuildData in PGen.BuildDatabase._CACHE_.values():
    RuntimeError: dictionary changed size during iteration

Note: The culprit commit (f8d11e5a4a) can not be found with bisection.
In 9c2d68c0a2 the build tools default to the python version provided
by the ${PYTHON} environment variable, however the Python3 transition is
not functional before d943b0c339. f8d11e5a4a falls between the
previous two.

Reported-by: Leif Lindholm <leif.lindholm@linaro.org>
Fixes: f8d11e5a4a
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Philippe Mathieu-Daude <philmd@redhat.com>
Tested-by: Leif Lindholm <leif.lindholm@linaro.org>
Acked-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Jaben Carsey <jaben.carsey@intel.com>
2019-02-06 13:08:16 -08:00
Laszlo Ersek 963517211c BaseTools/BuildReport: fix report for platforms/arches without struct PCDs
The goal of commit 97c8f5b9e7 ("BaseTools:StructurePCD value display
incorrect in "Not used" section.", 2019-02-02) was to display the full
contents of such structure PCDs in the build report that were set in the
platform DSC or the FDF, but not used in any module INFs. The listings
would appear in the

  PCDs not used by modules or in conditional directives

section of the build report.

Commit 97c8f5b9e7 assumed that any (platform, architecture) combination
would have a (possibly empty) set of structure PCD (and so the set of the
structure PCDs could be filtered for set-but-unused ones).

This is not the case: in "DscBuildData.py", in method
UpdateStructuredPcds(), if "S_pcd_set" remains an empty OrderedDict(),
then it is not added to "GlobalData.gStructurePcd" *at all*, for the
current (platform, architecture) combination.

As a result, when the PCD report tries to fetch the set of structure PCDs
for the current (platform, architecture), "GlobalData.gStructurePcd" does
not return an empty OrderedDict(); instead, it raises a KeyError. Fix it
by defaulting to an empty OrderedDict(), with the get() method.

Reported-by: Vladimir Olovyannikov <vladimir.olovyannikov@broadcom.com>
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=1513
Fixes: 97c8f5b9e7
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Vladimir Olovyannikov <vladimir.olovyannikov@broadcom.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Vladimir Olovyannikov <vladimir.olovyannikov@broadcom.com>
Reviewed-by: Jaben Carsey <jaben.carsey@intel.com>
2019-02-06 13:07:21 -08:00
Fan, ZhijuX 97c8f5b9e7 BaseTools:StructurePCD value display incorrect in "Not used" section.
StructurePCD value display incorrect in "Not used" section,
that the value defined in structure does not show.

StructurePCD will not display in "Not Used" section if
main structure not define in dsc and pcd not in inf

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-02-02 10:28:56 +08:00
Feng, Bob C 4a6b179f4f BaseTools: Fixed incorrect line number in PcdValueInit.c
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1435
This patch is going to fix the issue of incorrect line
number in PcdValueInit.c

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-02-02 10:28:56 +08:00
Feng, Bob C 9f30e40128 BaseTools: Enable CODE format in DEC file
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1292.
Enable CODE format in DEC File.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-02-02 10:28:55 +08:00
Hess Chen ce3bfbec71 BaseTools/ECC: Add a new type of exception
Add a new type of exception for function body checkpoint.

Contributed-under: TianoCore Contribution Agreement 1.0
Signed-off-by: Hess Chen <hess.chen@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-02-02 10:28:54 +08:00
Fan, ZhijuX 5895f7f6d4 BaseTools: PCD value incorrect in structure pcd sku case.
Defined 2 PCDs(Test4 & Test401) and 2 SKUs(DEFAULT & _),
then set "SKUID_Defines" to ALL, for FixedAtBuild
gEfiStructuredPcdPkgTokenSpaceGuid. Test401 in this case,
its value should get from "Default" SKU, not from "_" SKU,
but we does not set value in SKU "_" in dsc, so Test401
should only display the value get from dec.

Missing the map() function causes SKU specific items to
not be deleted

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-02-02 10:28:54 +08:00
Feng, Bob C 8189be6fd7 BaseTools: Eot tool Python3 adaption
v2:
The python files under CParser4 are generated by antlr4 and for
python3 usage. They have python3 specific syntax, for example
the data type declaration for the arguments of a function. That
is not compitable with python2. this patch is to remove these syntax.

Eot tool Python3 adaption.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-02-01 11:09:25 +08:00
Feng, Bob C c60377d7f9 BaseTools: ECC tool Python3 adaption
v2:
The python files under CParser4 are generated by antlr4 and for
python3 usage. They have python3 specific syntax, for example
the data type declaration for the arguments of a function. That
is not compitable with python2. this patch is to remove these syntax.

ECC tool Python3 adaption.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-02-01 11:09:25 +08:00
Feng, Bob C d943b0c339 BaseTools: Handle the bytes and str difference
Deal with bytes and str is different, remove the unicode(),
correct open file parameter.
Using utcfromtimestamp instead of fromtimestamp.

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-02-01 11:09:24 +08:00
Feng, Bob C f8d11e5a4a BaseTools: the list and iterator translation
In python3,The keys of the dictionary not a list,It needs to be converted

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-02-01 11:09:24 +08:00
Feng, Bob C 7fa0e68afd BaseTools:ord() don't match in py2 and py3
In python2, the FvHeaderBuffer Type is a str
In python3, the FvHeaderBuffer Type is a bytes

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-02-01 11:09:24 +08:00
Feng, Bob C 4a3773e578 BaseTools:Fixed Rsa issue and a set define issue.
ValueError: non-hexadecimal number found in
 fromhex() arg at position 0

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-02-01 11:09:23 +08:00
Liming Gao 7aef7b7cbf BaseTools: Update PYTHON env to PYTHON_COMMAND
Update PYTHON env to PYTHON_COMMAND.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Liming Gao <liming.gao@intel.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-02-01 11:09:23 +08:00
Zhijux Fan fd2d74007b BaseTools:Update build tool to print python version information
print PYTHON3_ENABLE and PYTHON_COMMAND in build tool

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-02-01 11:09:22 +08:00
Zhijux Fan 94c912950c BaseTools: Similar to octal data rectification
In python3, if Value is octal data, the int(Value, 0) report an error

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-02-01 11:09:22 +08:00
Zhiju Fan 30e65c4e19 BaseTools:There is extra blank line in datalog
There should be no blank line across every
line in datalog if open it with Notepad++.

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-02-01 11:09:22 +08:00
Zhijux Fan b3e94a0617 BaseTools: change the Division Operator
PEP 238 -- Changing the Division Operator
x/y to return a reasonable approximation of the mathematical result
    of the division ("true division")
x//y to return the floor ("floor division")

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-02-01 11:09:21 +08:00
Feng, Bob C f935bdd270 BaseTools:File open failed for VPD MapFile
correct open MapFile support python2 and python3

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-02-01 11:09:21 +08:00
Zhiju Fan 62cb98c254 BaseTools:Double carriage return inserted from Trim.py on Python3
https://bugzilla.tianocore.org/show_bug.cgi?id=1379

Line 208 of BaseTools/Source/Python/Trim/Trim.py uses
'NewLines.append(os.linesep)' to insert a new line into
the list that will be written to the output file.
This causes the '\r\n' inserted with os.linesep to be
written as '\r\r\n', causing some assemblers to error.

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-02-01 11:09:21 +08:00
Feng, Bob C 3f7cb70c5a BaseTools: Make sure AllPcdList valid.
This patch is to make sure the AllPcdList is always evaluated.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-02-01 11:09:20 +08:00
Zhijux Fan d7f40203dc BaseTools:Fv BaseAddress must set If it not set
If ForceRebase is not set, and FV is specified in FD region,
 it should have FvBaseAddress

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-02-01 11:09:20 +08:00
Zhijux Fan 2c2bb053a3 Basetools: It went wrong when use os.linesep
in python2 and python3,use of line breaks

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-02-01 11:09:20 +08:00
Zhijux Fan 174a9d3cc8 BaseTools/UPT:merge UPT Tool use Python2 and Python3
In UPT Tool,merge python2 and python3

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-02-01 11:09:19 +08:00
Zhijux Fan 1590d12315 BaseTools:Similar to octal data rectification
In python3, if Value is octal data, the int(Value, 0) report an error

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-02-01 11:09:18 +08:00
Zhijux Fan 56ad03a5dc BaseTools: Update argparse arguments since it not have version now
argparse.ArgumentParser not have version parameter

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-02-01 11:09:18 +08:00
Yunhua Feng af881abc65 BaseTools: replace long by int
replace long by int
Because the long() was not exist in Python3

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yunhua Feng <yunhuax.feng@intel.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-02-01 11:09:18 +08:00
Yunhua Feng 1d2aa01cdf BaseTools: Remove unnecessary super function
Remove unnecessary super function

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yunhua Feng <yunhuax.feng@intel.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-02-01 11:09:17 +08:00
Yunhua Feng b56abd91bc BaseTools: nametuple not have verbose parameter in python3
nametuple not have verbose parameter in python3

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yunhua Feng <yunhuax.feng@intel.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-02-01 11:09:17 +08:00
Feng, Bob C f56c83f815 BaseTools: use OrderedDict instead of sdict
use OrderedDict instead of sdict, and delete sdict

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yunhua Feng <yunhuax.feng@intel.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-02-01 11:09:17 +08:00
Zhijux Fan 00f86d89ab BaseTools: replace get_bytes_le() to bytes_le
UUID does not have the get_bytes_le() in python3

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-02-01 11:09:17 +08:00
Zhijux Fan 3a041437c9 BaseTools: Rename iteritems to items
replace the list iteritems by items in Python3.

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-02-01 11:09:16 +08:00
Zhijux Fan 2f28dca16f BaseTools:use iterate list to replace the itertools
itertools.imap() replace map(), itertools.ifilter() replace filter

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-02-01 11:09:16 +08:00
Zhijux Fan e77e59c9bd BaseTool:Rename xrange() to range()
Because the xrange() was not exist in Python3

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-02-01 11:09:16 +08:00
Carsey, Jaben f30e4aed99 BaseTools/build/build: delete variable
delete the shared global variable from Common.Misc
delete the uncalled users of the variable from build.build

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-01-28 11:44:45 +08:00
Carsey, Jaben 3aaacb2daf BaseTools/Common/Misc: Cleanup the imports
Refactor to 'dict' from 'IterableUserDict' which was only required for old
    python interpreter.
Sort imports according to PEP8
Remove those we dont need.

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-01-28 11:44:44 +08:00
Carsey, Jaben fe5ff16128 BaseTools/Common/Misc: remove uncalled code
no use for this code content.

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-01-28 11:44:43 +08:00
Fan, ZhijuX 51f5233385 BaseTools:Define Macro in Inf file, the pcdvalue Can't convert
Define Macro in Inf file, the pcdvalue Can't convert

in dec:
gUefiOvmfPkgGuid.test1|"ddd"|VOID*|0x3c
in inf:
DEFINE ABC=PATH
gUefiOvmfPkgTokenSpaceGuid.test1|$(ABC)

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-01-28 11:44:42 +08:00
Carsey, Jaben 42bd17508e BaseTools/Common/Misc: move private functions
These 2 functions are only used internally.  move them to the smallest
scope and use them.

Cc: Bob Feng <Bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-01-28 11:44:42 +08:00
Fan, ZhijuX e3ef8f0c52 BaseTools: Fix Pcd Array changes build report issue.
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1472

The Pcd Array feature changes the Pcd Default value
data structure which is used by build report. This
patch is going to update build report to adapt that change.

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-01-28 11:44:41 +08:00
Carsey, Jaben 5c5ca9f1fb BaseTools/GenFds/Capsule: move function logic
Move PackRegistryFormatGuid logic from Common.Misc to this file.
There were no other consumers of the function.
As it is one line, just replace the logic without the separate function.

v2 - change to fix the import statement

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-01-21 18:03:33 +08:00
Carsey, Jaben 1bcc2cd63d BaseTools/AutoGen: move functions
Move SplitOption and ConvertStringToByteArray from
Common.Misc to this file.
There were no other consumers of the functions.

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-01-21 18:03:33 +08:00
Carsey, Jaben 2b95556c06 BaseTools/DscBuildData: move function
Move IsFieldValuieAnArray from Common.Misc to this file.
There were no other consumers of the function.

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-01-21 18:03:32 +08:00
Carsey, Jaben 12ba816e61 BaseTools/Workspace/InfBuildData: move functions
Move ProtocolValue and PpiValue from Common.Misc to this file.
There were no other consumers of these 2 functions.

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-01-21 18:03:32 +08:00
Fan, ZhijuX 5f7650e497 BaseTools:The BuildOptionPcd value is wrong
In GenFds.py, Due to the second assignment, the value is wrong
Its value should a list, not a string.and this line is not required

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-01-21 18:03:31 +08:00
Feng, Bob C 350c9ae081 BaseTools: Fix build report issue.
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1472
The Pcd Array feature changes the Pcd Default value
data structure which is used by build report. This
patch is going to update build report to adapt that change.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-01-21 18:03:31 +08:00
Carsey, Jaben b298567268 BaseTools/build/build: refactor and move functions
Move DataDump and DataRestore from Common.Misc to this file.
There were no other consumers of these 2 functions.

Import threading since that module is used in build.

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-01-21 18:03:30 +08:00
Fan, ZhijuX dd1988cf01 BaseTools:build break if the Path contains SingleFile.Ext
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=1325

if SingleFile.Path = "/foo/bar.Sap/yada/source.S" and
SingleFile.Ext = ".S". Then key would end up "/foo/bar"
instead of "/foo/bar.Sap/yada/source" as intended.

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-01-21 18:03:30 +08:00
Felix Polyudov 842695d2ff BaseTools: Fix incorrect formatting of GenFds command dictionary
GenFdsCommand returned dictionary with elements that
are not compatible with GenFdsApi.
As a result the following options were not processed by GenFdsApi:
-v, -q, -d, --genfds-multi-thread, --ignore-sources
The issue is introduced by commit b3497bad12.

V2: Remove EFI_SOURCE references

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Felix Polyudov <felixp@ami.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-01-21 18:03:29 +08:00
Fan, ZhijuX 3e1ab49481 BaseTools:Build fail when PCD use in the [DEPEX] section of INF files
Update _FixedPcdVoidTypeDict to FixedVoidTypePcds
'_FixedPcdVoidTypeDict' no longer exists because edk2 version (b23414f6).
'ModuleAutoGen' object has no attribute '_FixedPcdVoidTypeDict'.
Build fail when PCD use in the [DEPEX] section of INF files

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-01-21 18:03:28 +08:00
Feng, Bob C cce9d76358 BaseTools: Allow empty value for HiiPcd in Dsc
https://bugzilla.tianocore.org/show_bug.cgi?id=1466

DEC file defines PCD default value and PCD supported type.
DSC can configure PCD type and value.
If the value is same to default value in DEC file,
DSC can only configure PCD type and leave empty for value.
This usage supports all type PCD except for DynamicHii type.
So, DynamicHii PCD should support this usage. Below is one example in DSC.

for example,
[PcdsDynamicHii.common.DEFAULT]
PcdPkgTokenSpaceGuid.PcdCName|L"VarName"|gVarGuid|0x00||NV,BS

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-01-18 10:20:01 +08:00
Feng, Bob C c68b564900 BaseTools: Remove EDK_SOURCE keyword from Inf Parser.
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1350
Remove unused EDK_SOURCE keyword from Inf Parser.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-01-17 19:49:20 +08:00
Feng, Bob C 7f98ef37f6 BaseTools: Remove EDK_SOURCE keyword from GenFds tool.
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1350
Remove unused EDK_SOURCE keyword from GenFds tool.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-01-17 19:49:20 +08:00
Feng, Bob C 0006d3fd0f BaseTools: Remove EDK_SOURCE keyword from ECC Tool
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1350
Remove EDK_SOURCE keyword from ECC Tool.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-01-17 19:49:20 +08:00
Feng, Bob C 4523bc82ae BaseTools: Remove unused logic for EDKI
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1350
Remove gEcpSource global variable in python tool.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
2019-01-14 15:24:22 +08:00
Feng, Bob C 5bc96263ea BaseTools: Enable component override functionality
https://bugzilla.tianocore.org/show_bug.cgi?id=1449
This patch enable build tools to recognize that
when two given files have the same GUID, file path and ARCH in Dsc,
The later one's definition will be used.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Carsey Jaben <jaben.carsey@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-01-14 11:14:24 +08:00
Feng, Bob C 39879ef267 BaseTools: Remove unused logic for IPF
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1350
Remove IPF support from BaseTools C code.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-01-14 09:34:41 +08:00
Feng, Bob C 8229250132 BaseTools: Remove unused logic for EDKI
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1350
Remove EDK module type support from BaseTools python code.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-01-14 09:34:23 +08:00
Feng, Bob C b8920ccc3c BaseTools: Report Error if use SET in Dsc
Build tool do not support SET syntax in DSC.
If the SET statement is used in DSC, build tool just ignore it.
That behavior confused some users that
they think SET statement works in DSC like in FDF.
To avoid such confusion, build tool report ERROR
if there is "SET" statement in Dsc file.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Reviewed-by: Cc: Liming Gao <liming.gao@intel.com>
2019-01-11 10:51:19 +08:00
Carsey, Jaben 938cf4c33a BaseTools: fix imports
1 - Some of these imports are cascaded from another file. Import them locally.
2 - Some of these imports are not used. Remove them.
3 - Some of these were missing the namespace used to import them.

These changes facilitate optimization of BaseTools:
https://bugzilla.tianocore.org/show_bug.cgi?id=42

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-01-10 22:01:11 +08:00
Ard Biesheuvel 8ef653aa5a BaseTools/GenFds: permit stripped MM_CORE_STANDALONE binaries
The standalone MM core is executed in place, and resides in a
separate execution context which may be space constrained.
Since code and data may be mapped with different attributes for
security reasons, the PE/COFF binary could have a section
alignment of 4 KB.

This means that any relocation data is not only useless, but it
will also take up 4 KB of valuable space.

So add support for the RELOCS_STRIPPED attribute on FFS files of
this type, so that we can get rid of the .reloc section altogether.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Reviewed-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-01-04 09:42:17 +01:00
Feng, Bob C 672601cfcc BaseTools: Correct PcdArray value assigment statement
https://bugzilla.tianocore.org/show_bug.cgi?id=1410
BaseTools should not generate C structure array initial value
if the value is not specified with CODE style.

This patch is going to remove the incorrect initial value statement
and correct the Pcd Array value assignment statement.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-01-04 15:18:18 +08:00
Feng, Bob C 82bfbd392a BaseTools: Fixed build report issue.
This patch is going to fix the regression issue
by 72a1d77694

After enable PCD array, the Pcd.OverrideStruct has a new
key, array index, but the build report is not changed correspondingly.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-01-04 15:11:03 +08:00
Feng, Bob C abc4c3386a BaseTools: Reset FdsGlobalVariable
https://bugzilla.tianocore.org/show_bug.cgi?id=1418
This patch is going to fix a regression issue that is introduced
by commit b3497bad12.

Before commit b3497b, build launched a external GenFds.py to generate
Fd, so the global variable in GenFds.py was reset in each execution.

After commit b3497b, each GenFds run in the same python interpeter, so
we need to explicitly reset global variable in each GenFdsApi call.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
2018-12-25 10:40:10 +08:00
Derek Lin 370544d116 BaseTools: Fix GenFds error doesn't break build.
Fix a bug because of b3497bad12.
Before the patch, when GenFds fail, the build continue and return success.

Contributed-under: TianoCore Contribution Agreement 1.0
Signed-off-by: Derek Lin <derek.lin2@hpe.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2018-12-19 13:24:18 +08:00
Zhiju.Fan 073a76e662 BaseTools: Add $(INC)-like support when compiling .nasm files
current edk2\BaseTools\Conf\build_rule.template, the compile of nasm
source files does not have the $(INC) support.

The '-I' option only includes the directory of the nasm source file
(${s_path}(+)). Hence, it will be impossible for nasm files to include
files outside of the nasm source file directory.

As a comparison, the compile of both .s and .asm have $(INC) support
in their compile commands.

Fixes: https://bugzilla.tianocore.org/show_bug.cgi?id=1085
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2018-12-19 08:42:14 +08:00
Feng, Bob C 643556fc48 BaseTools: Fixed metafile parser issues
https://bugzilla.tianocore.org/show_bug.cgi?id=1406
This patch is going to fix the regressions that
is introduced by commit 2f818ed0fb

The internal array for storing the metadata info should be cached
so that the meta file is parsed only once in one build.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-12-18 10:45:31 +08:00
Feng, Bob C 4c6e6f9f75 BaseTools: Fix PcdArray issue
https://bugzilla.tianocore.org/show_bug.cgi?id=1390

1. support hex number for array index
2. support Non-Dynamic Pcd for array data type
3. support {} and {CODE()} for array data type
4. Change GetStructurePcdMaxSize to be a static function since it need to
be called in another static function. And this function does not depend on
it's class instance.
5. Add unittest for RemoveCComments function and
ArrayIndex regular expression.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Cc: Philippe Mathieu-Daud? <philmd@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-12-18 10:15:50 +08:00
Feng, Bob C 48e890457b BaseTools: Fixed the build fail issue for cases
https://bugzilla.tianocore.org/show_bug.cgi?id=1386
This patch is going to fix the regression issue that is
introduced by commit 72a1d77694

The issue will happen in the following cases:
1. There is no Pcd value assignment in Dsc file
2. There are duplicate Pcd filed assignment

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Leif Lindholm <leif.lindholm@linaro.org>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-12-15 20:16:46 +08:00
Feng, Bob C 9bf86b12ce BaseTools: Fixed bugs in CopyDict function
https://bugzilla.tianocore.org/show_bug.cgi?id=1387

This patch is going to fix the regression issue which is
introduced by commit bf9e636605.

This patch Remove the CopyDict incorrect usage for non-dict
input data. Add a check for CopyDict input.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Leif Lindholm <leif.lindholm@linaro.org>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-12-15 20:13:32 +08:00
Feng, Bob C 5695877ec8 BaseTools: Fix PcdNvStoreDefaultValueBuffer Value.
https://bugzilla.tianocore.org/show_bug.cgi?id=1385
This patch is going to fix the regression issue that is
introduced by commit e6eae3b4c7
and commit 0b6c5954e1.

PcdNvStoreDefaultValueBuffer value is update to Vpd Info File,
but it is not update into a internal cache. This patch will
fix this incorrect value in that internal cache.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Leif Lindholm <leif.lindholm@linaro.org>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-12-15 15:45:07 +08:00
BobCF 9601046bf4 BaseTools: Correct CCFLAG for PcdValueInit
https://bugzilla.tianocore.org/show_bug.cgi?id=1361
This patch is going to correct the CCFlag
for building PcdValueInit

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: BobCF <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-12-07 10:39:52 +08:00
BobCF bf9e636605 BaseTools: Customize deepcopy function.
https://bugzilla.tianocore.org/show_bug.cgi?id=1288

This patch is one of build tool performance improvement
series patches.

This patch is going to customize the deepcopy function for
SkuClass, PcdClassObject and python dictionary.

python deepcopy copy everything of a object, but for our current
usage we just need to copy the data we care about recursively.

By implementing __deepcopy__ for SkuClass, PcdClassObject, we can customize
deepcopy function for them.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: BobCF <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-12-07 10:31:04 +08:00
BobCF 4e37570739 BaseTools: Optimize string concatenation
https://bugzilla.tianocore.org/show_bug.cgi?id=1288

This patch is one of build tool performance improvement
series patches.

This patch is going to use join function instead of
string += string2 statement.

Current code use string += string2 in a loop to combine
a string. while creating a string list in a loop and using
"".join(stringlist) after the loop will be much faster.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: BobCF <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Jaben Carsey <jaben.carsey@intel.com>
2018-12-07 10:30:21 +08:00
BobCF 2f818ed0fb BaseTools: Replace the sqlite database with list
https://bugzilla.tianocore.org/show_bug.cgi?id=1288

[V2]
Optimize this patch so that it can be easy to review.
This patch is just apply the change to original files while
not create new similar files.

[V1]
This patch is one of build tool performance improvement
series patches.

This patch is going to use python list to store the parser data
instead of using sqlite database.

The replacement solution is as below:

SQL insert: list.append()
SQL select: list comprehension. for example:
Select * from table where field = “something”
->
[ item for item in table if item[3] == “something”]

SQL update: python map function. for example:
Update table set field1=newvalue where filed2 = “something”.
-> map(lambda x: x[1] = newvalue,
   [item for item in table if item[2] == “something”])

SQL delete: list comprehension.

With this change, We can save the time of interpreting SQL statement
and the time of write database to file system

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: BobCF <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Jaben Carsey <jaben.carsey@intel.com>
2018-12-07 10:23:07 +08:00
Zhao, ZhiqiangX b3497bad12 BaseTools: AutoGen and GenFds share the parser data.
V2:
Extract the common part of new API and the original main() function
into one function.

V1:
https://bugzilla.tianocore.org/show_bug.cgi?id=1288

Currently, AutoGen and GenFds run in different python interpreters. The
parser are duplicated. This patch is going to create new API for GenFds
and have the build to call that API instead of executing GenFds.py. As
such, the GenFds and build can share the parser data.

This patch is expected to save the time of GenFds about 2~3 seconds.
More details will be logged in BZ.

This is the summary measure data generated from python cProfile for
building Ovmf.

Currently:
8379147 function calls (8135450 primitive calls) in 12.580 seconds

After applying this patch:
3428712 function calls (3418881 primitive calls) in 8.944 seconds

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: ZhiqiangX Zhao <zhiqiangx.zhao@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Carsey Jaben <jaben.carsey@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2018-12-07 10:12:05 +08:00
Feng, Bob C 34e733f200 BaseTool: Filter out unused structure pcds
V2:
Fixed the issue that V1 adds new check
to the Pcds in the platform unused library INF files.
It breaks the existing platform.

V1?
The current code handle all the structure pcds
even if there is no module or library use them.
This patch is going to filter out the unused structure pcds.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-12-07 10:03:02 +08:00
bob.c.feng@intel.com 72a1d77694 BaseTools: Enable Pcd Array support.
https://bugzilla.tianocore.org/show_bug.cgi?id=1292

This patch is going to enable Array data type for PCD.

1. Support Pcd ARRAY as Structure PCD type
   including basic datatype array and structure array.
   For example:
   gStructuredPcdPkgTokenSpaceGuid.PcdTest|{0x0}|TEST[10]|0x00010080
   gStructuredPcdPkgTokenSpaceGuid.PcdTest2|{0x0}|UINT8[10]|0x00010081
2. Support C CODE style value initialization in DEC/DSC.
   For example:
gStructuredPcdPkgTokenSpaceGuid.PcdTest|{CODE({
  {0, {0, 0, 0, 0,  0, 0, 0}},
  {0, {0, 0, 0, 0,  0, 0, 0}},
  {0, {0, 0, 0, 0,  0, 0, 0}},
  {0, {0, 0, 0, 0,  0, 0, 0}},
  {0, {0, 0, 0, 0,  0, 0, 0}},
  {0, {0, 0, 0, 0,  0, 0, 0}},
  {0, {0, 0, 0, 0,  0, 0, 0}},
  {0, {0, 0, 0, 0,  0, 0, 0}},
  {0, {0, 0, 0, 0,  0, 0, 0}},
  {0, {0, 0, 0, 0,  0, 0, 0}},
})}

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2018-12-07 09:59:36 +08:00