BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1788
In V3: Must generate hashes before attempting to copy from cache for
hash verifcation
In V2: Build failure caused by passing incorrect boolean parameter to
SaveFileOnChange(). Fixed for patch instances.
Library hashing is now supported by the --hash feature. The --hash
feature implementation assumed that the hashing could be done in
place once per module, but that isn't true for libraries due to the
fact that they are built as dependencies. So on a clean build, we now
generate the .hash after the library dependencies are complete.
Added early escape as optimization, if hash already exists in memory.
Signed-off-by: Christian Rodriguez <christian.rodriguez@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1692
Change part of the hash error handling to invalidate hashes in the
cache destination not the cache source.
Signed-off-by: Christian Rodriguez <christian.rodriguez@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1692
Add error handling to the --hash feature so that hash files
are invalidated when a build error occurs.
Signed-off-by: Christian Rodriguez <christian.rodriguez@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1680
Re-use libraries, since they have already been hashed.
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Christian Rodriguez <christian.rodriguez@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
https://bugzilla.tianocore.org/show_bug.cgi?id=1450
In the Multiple thread Genfds feature, build tool generates
GenSec, GenFFS command in Makefile.
The Non-Hii Driver does not generate .offset file for uni string offset,
but the build tool has not knowledge about this in autogen phase. So
in this patch, I add a check in Makefile for GenSec command. If the GenSec
input file does not exist, the GenSec will not be called. And if GenSec
command is not called, its output file, which is also the input file of
GenFfs command, will also not exist.So for GenFfs command,
I add a new command parameter -oi which means
the input file is an optional input file which would not exist. so
that I can generate GenFfs command with "-oi" parameter in Makefile.
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1620
This option is added to disable the include path check
for outside of package.
The original purpose of thie check is to make sure EDK II
modules must not reference header files outside of the packages
they depend on or within the module's directory tree.
However, we do see the usage to build EDKII as executable running
in the operating system which requires include path to outside.
For example, EmulatorPkg. The current solution (soft link) is
weird hack - EmulatorPkg\Unix\Host\X11IncludeHack.
With this solution, this can be supported easily.
The patch is validated with and without --disable-include-path-check.
If user does not use --disable-include-path-check, the build will fail
with outside path in the include path.
If user uses --disable-include-path-check, the build will pass
with outside path in the include path.
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jiewen Yao <jiewen.yao@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
PEP 238 -- Changing the Division Operator
x/y to return a reasonable approximation of the mathematical result
of the division ("true division")
x//y to return the floor ("floor division")
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
delete the shared global variable from Common.Misc
delete the uncalled users of the variable from build.build
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
Move DataDump and DataRestore from Common.Misc to this file.
There were no other consumers of these 2 functions.
Import threading since that module is used in build.
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
1 - Some of these imports are cascaded from another file. Import them locally.
2 - Some of these imports are not used. Remove them.
3 - Some of these were missing the namespace used to import them.
These changes facilitate optimization of BaseTools:
https://bugzilla.tianocore.org/show_bug.cgi?id=42
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
Fix a bug because of b3497bad12.
Before the patch, when GenFds fail, the build continue and return success.
Contributed-under: TianoCore Contribution Agreement 1.0
Signed-off-by: Derek Lin <derek.lin2@hpe.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
https://bugzilla.tianocore.org/show_bug.cgi?id=1288
[V2]
Optimize this patch so that it can be easy to review.
This patch is just apply the change to original files while
not create new similar files.
[V1]
This patch is one of build tool performance improvement
series patches.
This patch is going to use python list to store the parser data
instead of using sqlite database.
The replacement solution is as below:
SQL insert: list.append()
SQL select: list comprehension. for example:
Select * from table where field = “something”
->
[ item for item in table if item[3] == “something”]
SQL update: python map function. for example:
Update table set field1=newvalue where filed2 = “something”.
-> map(lambda x: x[1] = newvalue,
[item for item in table if item[2] == “something”])
SQL delete: list comprehension.
With this change, We can save the time of interpreting SQL statement
and the time of write database to file system
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: BobCF <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Jaben Carsey <jaben.carsey@intel.com>
V2:
Extract the common part of new API and the original main() function
into one function.
V1:
https://bugzilla.tianocore.org/show_bug.cgi?id=1288
Currently, AutoGen and GenFds run in different python interpreters. The
parser are duplicated. This patch is going to create new API for GenFds
and have the build to call that API instead of executing GenFds.py. As
such, the GenFds and build can share the parser data.
This patch is expected to save the time of GenFds about 2~3 seconds.
More details will be logged in BZ.
This is the summary measure data generated from python cProfile for
building Ovmf.
Currently:
8379147 function calls (8135450 primitive calls) in 12.580 seconds
After applying this patch:
3428712 function calls (3418881 primitive calls) in 8.944 seconds
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: ZhiqiangX Zhao <zhiqiangx.zhao@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Carsey Jaben <jaben.carsey@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
1) remove an identical function and import it from Common.LongFilePathSupport
2) remove an import that is not needed/used.
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by : Bob Feng <bob.c.feng@intel.com>
This reverts commit 6693f359b3c213513c5096a06c6f67244a44dc52..
678f851312.
Python3 migration is the fundamental change. It requires every developer
to install Python3. Before this migration, the well communication and wide
verification must be done. But now, most people is not aware of this change,
and not try it. So, Python3 migration is reverted and be moved to edk2-staging
Python3 branch for the edk2 user evaluation.
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Liming Gao <liming.gao@intel.com>
PEP 238 -- Changing the Division Operator
x/y to return a reasonable approximation of the mathematical result
of the division ("true division")
x//y to return the floor ("floor division")
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yunhua Feng <yunhuax.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
1) use decorators
2) also change some private functions to public when all callers are
external
3) change external callers to use functions instead of directly
accessing private data.
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
I see lots of 'MSFT' throughout code and this can reduce them.
Cc: Bob Feng <Bob.c.Feng@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
1. Do not use tab characters
2. No trailing white space in one line
3. All files must end with CRLF
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
Replace StringIO.StringIO with io.BytesIO to be compatible with python3.
This commit also removes "import StringIO" from those python scripts
that don't really use it.
Contributed-under: TianoCore Contribution Agreement 1.1
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Gary Lin <glin@suse.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
Based on "futurize -f lib2to3.fixes.fix_idioms"
* Change some type comparisons to isinstance() calls:
type(x) == T -> isinstance(x, T)
type(x) is T -> isinstance(x, T)
type(x) != T -> not isinstance(x, T)
type(x) is not T -> not isinstance(x, T)
* Change "while 1:" into "while True:".
* Change both
v = list(EXPR)
v.sort()
foo(v)
and the more general
v = EXPR
v.sort()
foo(v)
into
v = sorted(EXPR)
foo(v)
Contributed-under: TianoCore Contribution Agreement 1.1
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Gary Lin <glin@suse.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
Replace "has_key()" with "in" to be compatible with python3.
Based on "futurize -f lib2to3.fixes.fix_has_key"
Contributed-under: TianoCore Contribution Agreement 1.1
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Gary Lin <glin@suse.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
Refactor print statements to be compatible with python 3.
Based on "futurize -f libfuturize.fixes.fix_print_with_import"
Contributed-under: TianoCore Contribution Agreement 1.1
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Gary Lin <glin@suse.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
Convert "except ... ," to "except ... as" to be compatible with python3.
Based on "futurize -f lib2to3.fixes.fix_except"
Contributed-under: TianoCore Contribution Agreement 1.1
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Gary Lin <glin@suse.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
EDK_COMPONENT_TYPE_LIBRARY was 'LIBRARY'
EDK_COMPONENT_TYPE_SECURITY_CORE was 'SECURITY_CORE'
EDK_COMPONENT_TYPE_COMBINED_PEIM_DRIVER was 'COMBINED_PEIM_DRIVER'
EDK_COMPONENT_TYPE_PIC_PEIM was 'PIC_PEIM'
EDK_COMPONENT_TYPE_RELOCATABLE_PEIM was 'RELOCATABLE_PEIM'
EDK_COMPONENT_TYPE_BS_DRIVER was 'BS_DRIVER'
EDK_COMPONENT_TYPE_RT_DRIVER was 'RT_DRIVER'
EDK_COMPONENT_TYPE_SAL_RT_DRIVER was 'SAL_RT_DRIVER'
EDK_COMPONENT_TYPE_APPLICATION was 'APPLICATION'
v2 - revert 2 files. will update later in own patches.
v3 - fix v2
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
replace raw strings in the code (note: except UPT) with constants.
SUP_MODULE_BASE was 'BASE'
SUP_MODULE_SEC was 'SEC'
SUP_MODULE_PEI_CORE was 'PEI_CORE'
SUP_MODULE_PEIM was 'PEIM'
SUP_MODULE_DXE_CORE was 'DXE_CORE'
SUP_MODULE_DXE_DRIVER was 'DXE_DRIVER'
SUP_MODULE_DXE_RUNTIME_DRIVER was 'DXE_RUNTIME_DRIVER'
SUP_MODULE_DXE_SAL_DRIVER was 'DXE_SAL_DRIVER'
SUP_MODULE_DXE_SMM_DRIVER was 'DXE_SMM_DRIVER'
SUP_MODULE_UEFI_DRIVER was 'UEFI_DRIVER'
SUP_MODULE_UEFI_APPLICATION was 'UEFI_APPLICATION'
SUP_MODULE_USER_DEFINED was 'USER_DEFINED'
SUP_MODULE_SMM_CORE was 'SMM_CORE'
SUP_MODULE_MM_STANDALONE was 'MM_STANDALONE'
SUP_MODULE_MM_CORE_STANDALONE was 'MM_CORE_STANDALONE'
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
Rodriguez, Christian
Feng, Bob C
Michael D Kinney
Jiewen Yao
Antoine Coeur
Liming Gao
Zhijux Fan
Carsey, Jaben
Derek Lin
Zhao, ZhiqiangX
Yunhua Feng
Gary Lin
Carsey, Jaben