https://bugzilla.tianocore.org/show_bug.cgi?id=1288
[V2]
Optimize this patch so that it can be easy to review.
This patch is just apply the change to original files while
not create new similar files.
[V1]
This patch is one of build tool performance improvement
series patches.
This patch is going to use python list to store the parser data
instead of using sqlite database.
The replacement solution is as below:
SQL insert: list.append()
SQL select: list comprehension. for example:
Select * from table where field = “something”
->
[ item for item in table if item[3] == “something”]
SQL update: python map function. for example:
Update table set field1=newvalue where filed2 = “something”.
-> map(lambda x: x[1] = newvalue,
[item for item in table if item[2] == “something”])
SQL delete: list comprehension.
With this change, We can save the time of interpreting SQL statement
and the time of write database to file system
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: BobCF <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Jaben Carsey <jaben.carsey@intel.com>
V2:
Extract the common part of new API and the original main() function
into one function.
V1:
https://bugzilla.tianocore.org/show_bug.cgi?id=1288
Currently, AutoGen and GenFds run in different python interpreters. The
parser are duplicated. This patch is going to create new API for GenFds
and have the build to call that API instead of executing GenFds.py. As
such, the GenFds and build can share the parser data.
This patch is expected to save the time of GenFds about 2~3 seconds.
More details will be logged in BZ.
This is the summary measure data generated from python cProfile for
building Ovmf.
Currently:
8379147 function calls (8135450 primitive calls) in 12.580 seconds
After applying this patch:
3428712 function calls (3418881 primitive calls) in 8.944 seconds
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: ZhiqiangX Zhao <zhiqiangx.zhao@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Carsey Jaben <jaben.carsey@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
1) remove an identical function and import it from Common.LongFilePathSupport
2) remove an import that is not needed/used.
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by : Bob Feng <bob.c.feng@intel.com>
This reverts commit 6693f359b3c213513c5096a06c6f67244a44dc52..
678f851312.
Python3 migration is the fundamental change. It requires every developer
to install Python3. Before this migration, the well communication and wide
verification must be done. But now, most people is not aware of this change,
and not try it. So, Python3 migration is reverted and be moved to edk2-staging
Python3 branch for the edk2 user evaluation.
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Liming Gao <liming.gao@intel.com>
PEP 238 -- Changing the Division Operator
x/y to return a reasonable approximation of the mathematical result
of the division ("true division")
x//y to return the floor ("floor division")
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yunhua Feng <yunhuax.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
1) use decorators
2) also change some private functions to public when all callers are
external
3) change external callers to use functions instead of directly
accessing private data.
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
I see lots of 'MSFT' throughout code and this can reduce them.
Cc: Bob Feng <Bob.c.Feng@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
1. Do not use tab characters
2. No trailing white space in one line
3. All files must end with CRLF
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
Replace StringIO.StringIO with io.BytesIO to be compatible with python3.
This commit also removes "import StringIO" from those python scripts
that don't really use it.
Contributed-under: TianoCore Contribution Agreement 1.1
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Gary Lin <glin@suse.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
Based on "futurize -f lib2to3.fixes.fix_idioms"
* Change some type comparisons to isinstance() calls:
type(x) == T -> isinstance(x, T)
type(x) is T -> isinstance(x, T)
type(x) != T -> not isinstance(x, T)
type(x) is not T -> not isinstance(x, T)
* Change "while 1:" into "while True:".
* Change both
v = list(EXPR)
v.sort()
foo(v)
and the more general
v = EXPR
v.sort()
foo(v)
into
v = sorted(EXPR)
foo(v)
Contributed-under: TianoCore Contribution Agreement 1.1
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Gary Lin <glin@suse.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
Replace "has_key()" with "in" to be compatible with python3.
Based on "futurize -f lib2to3.fixes.fix_has_key"
Contributed-under: TianoCore Contribution Agreement 1.1
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Gary Lin <glin@suse.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
Refactor print statements to be compatible with python 3.
Based on "futurize -f libfuturize.fixes.fix_print_with_import"
Contributed-under: TianoCore Contribution Agreement 1.1
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Gary Lin <glin@suse.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
Convert "except ... ," to "except ... as" to be compatible with python3.
Based on "futurize -f lib2to3.fixes.fix_except"
Contributed-under: TianoCore Contribution Agreement 1.1
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Gary Lin <glin@suse.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
EDK_COMPONENT_TYPE_LIBRARY was 'LIBRARY'
EDK_COMPONENT_TYPE_SECURITY_CORE was 'SECURITY_CORE'
EDK_COMPONENT_TYPE_COMBINED_PEIM_DRIVER was 'COMBINED_PEIM_DRIVER'
EDK_COMPONENT_TYPE_PIC_PEIM was 'PIC_PEIM'
EDK_COMPONENT_TYPE_RELOCATABLE_PEIM was 'RELOCATABLE_PEIM'
EDK_COMPONENT_TYPE_BS_DRIVER was 'BS_DRIVER'
EDK_COMPONENT_TYPE_RT_DRIVER was 'RT_DRIVER'
EDK_COMPONENT_TYPE_SAL_RT_DRIVER was 'SAL_RT_DRIVER'
EDK_COMPONENT_TYPE_APPLICATION was 'APPLICATION'
v2 - revert 2 files. will update later in own patches.
v3 - fix v2
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
replace raw strings in the code (note: except UPT) with constants.
SUP_MODULE_BASE was 'BASE'
SUP_MODULE_SEC was 'SEC'
SUP_MODULE_PEI_CORE was 'PEI_CORE'
SUP_MODULE_PEIM was 'PEIM'
SUP_MODULE_DXE_CORE was 'DXE_CORE'
SUP_MODULE_DXE_DRIVER was 'DXE_DRIVER'
SUP_MODULE_DXE_RUNTIME_DRIVER was 'DXE_RUNTIME_DRIVER'
SUP_MODULE_DXE_SAL_DRIVER was 'DXE_SAL_DRIVER'
SUP_MODULE_DXE_SMM_DRIVER was 'DXE_SMM_DRIVER'
SUP_MODULE_UEFI_DRIVER was 'UEFI_DRIVER'
SUP_MODULE_UEFI_APPLICATION was 'UEFI_APPLICATION'
SUP_MODULE_USER_DEFINED was 'USER_DEFINED'
SUP_MODULE_SMM_CORE was 'SMM_CORE'
SUP_MODULE_MM_STANDALONE was 'MM_STANDALONE'
SUP_MODULE_MM_CORE_STANDALONE was 'MM_CORE_STANDALONE'
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
1. Handle the Pcd maxsize issue for the case
that the length of Pcd value from CommandLine
bigger that its maxsize
2. The Pcd value override in commandline.
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
when MAX_CONCURRENT_THREAD_NUMBER is not specified, tool will
automatically detect number of processor threads.
Fixes: https://bugzilla.tianocore.org/show_bug.cgi?id=775
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
https://bugzilla.tianocore.org/show_bug.cgi?id=706
Add Structure PCD support for Build report. Structure PCD field value described
in DEC/DSC will be display in build report. And, PCD value for each SKU and
Default store will also be shown in build report.
Contributed-under: TianoCore Contribution Agreement 1.1
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
This patch enables hash value check for single module build to decide
whether we can skip to build this module.
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
V2: change to use InfBuildData but not ModuleAutoGen
We meet the case that first build with --hash option, then build it
again with --hash and --binary-destination option, since the hash value
is same, tool will not build the driver again, it cause the binary
files are not backed up.
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
when build a single module with GenC/GenMake option, currently it will
direct return after create Autogen code files, then it cause MaList is
empty, which cause an incorrect error message is reported.
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Fix a bug to use module's Name attribute as compare for single module
build. ModuleFile.File can't be used to compare INF file, because it
is the relative path.
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
The bug is introduced by 1b8eca to collect single module's build time.
Now the fix solution is copied from Platform build.
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Add three new options:
--hash enables hash-based caching during build process. when --hash is
enabled, build tool will base on the module hash value to do the
incremental build, without --hash, build tool will base on the
timestamp to do the incremental build. --hash option use md5 method to
get every hash value, DSC/FDF, tools_def.txt, build_rule.txt and build
command are calculated as global hash value, Package DEC and its
include header files are calculated as package hash value, Module
source files and its INF file are calculated as module hash value.
Library hash value will combine the global hash value and its dependent
package hash value. Driver hash value will combine the global hash
value, its dependent package hash value and its linked library hash
value.
When --hash and --binary-destination are specified, build tool will
copy generated binary files for each module into the directory specified
by binary-destination at the build phase. Binary-destination directory
caches all generated binary files.
When --hash and --binary-source are specified, build tool will try to
get the binary files from the binary source directory at the build
phase.If the cached binary has the same hash value, it will be directly
used. Otherwise, build tool will compile the source files and generate
the binary files.
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
In the build report, we add AutoGen Phase, Make Phase and GenFds Phase
time duration in the Platform Summary section, and we also add a item
in Module section to display module and library's build time.
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Per build spec,the default set of flags for -Y option include 'HASH',
so this patch to update the help info.
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
https://bugzilla.tianocore.org/show_bug.cgi?id=670
* Extend PREBUILD/POSTBUILD define values to support more than
one argument.
* Delay normalization of PREBUILD/POSTBUILD define values
until all arguments in the define values can be processed.
* Convert PREBUILD/POSTBUILD build define value arguments
that are WORKSPACE or PACKAGES_PATH relative paths to
absolute paths.
* Append -p PlatformFile, --conf=ConfDirectory, and build target
flags to command line used to execute PREBUILD/POSTBUILD
actions.
* Remove PrebuildScript and PostbuildScript fields from the
Build class and use Prebuild and Postbuild fields instead.
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.0
Signed-off-by: Michael Kinney <michael.d.kinney@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
In the definition, the warn() function takes at least 2 arguments.
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.0
Signed-off-by: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
This patch add some comments to explain why we use .PrebuildEnv file to
save environment variable settings set by the prebuild script.
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.0
Signed-off-by: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
BobCF
Zhao, ZhiqiangX
Jaben Carsey
Liming Gao
Yunhua Feng
Gary Lin
Carsey, Jaben
Yonghong Zhu
Michael D Kinney