BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1875
AutoGen processes race the logfile. To resolve this issue,
this patch create a LogAgent thread in main process to write
the log content to console or file, Other process will send
the log content to the LogAgent.
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Acked-by: Liming Gao <liming.gao@intel.com>
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1875
1. Separate the AutoGen.py into 3 small py files.
One is for AutoGen base class, one is for WorkspaceAutoGen class
and PlatformAutoGen class, and the one for ModuleAutoGen class.
2. Create a new class DataPipe to store the Platform scope settings.
Create a new class PlatformInfo to provide the same interface
as PlatformAutoGen. PlatformInfo class is initialized by
DataPipe instance.
Create a new class WorkspaceInfo to provide the same interface
as WorkspaceAutoGen. WorkspaceInfo class is initialized by
DataPipe instance.
3. Change ModuleAutoGen to depends on DataPipe, PlatformInfo and
WorkspaceInfo. Remove the dependency of ModuleAutoGen to PlatformAutoGen.
Cc: Liming Gao <liming.gao@intel.com>
Cc: Steven Shi <steven.shi@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Acked-by: Liming Gao <liming.gao@intel.com>
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1875
The build config files are target.txt, build rule, tooldef
During a build, the config is not changed, so the object to
handle them need to be singleton.
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Acked-by: Liming Gao <liming.gao@intel.com>
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=2015
Not all output works in utf-8, so change the encoding to
the default
This patch is going to fix that issue.
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1986
Create "Conf/.cache" folder as early as possible
so that the later code do need to check
if it exits and then create it.
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Reviewed-by: Michael D Kinney <michael.d.kinney@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
It can be used to indicate a module can be build to run
as OS application and run in OS environment.
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Signed-off-by: Xiaoyu Lu <xiaoyux.lu@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1834
1)Add arguments "--embedded-driver" to support embedded driver
in command line.
2)Add arguments "--update-image-index" to identify ImageIndex
within the device in command line.
3)Add arguments "-j JSONFILE" to support multiple payloads and
embedded drivers with JSON file.
The update is in a backwards compatible manner, so all command
line options to support single payload are still supported. But
all the options associated with multiple payloads should be
provided in a JSON file.
Cc: Bob Feng <bob.c.feng@intel.com>
Signed-off-by: Eric Jin <eric.jin@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=1894
Basetool need a CopyFileOnChange() function to avoid cache
file writing race in multi-thread build. Some platforms
build fail with file IO writing race issue when the
build cache is enabled to store cache files in multi-threads.
This is because common same library cache files (e.g. some
libs in MdePkg) can be stored by many different driver modules'
build threads at same time. Current build cache need a function
to check whether the same cache file already exist, and only
copy source file if it is different from the destination file.
This patch introduces an atomic copy function to avoid duplicated
cache files copy.
Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Christian Rodriguez <christian.rodriguez@intel.com>
Signed-off-by: Steven Shi <steven.shi@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1804
After adding a check to see if [Sources] section lists all the header
type files of a module, track module and library hashes for --hash
feature. If above check is not in compilance for a library or module,
force hash invalidation on that library or module.
Signed-off-by: Christian Rodriguez <christian.rodriguez@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=1765
If RAW FFS File Rule has no section for its data.For RAW FFS File,
directly call GenFfs tool to generate FFS file.
Ffs Rule:
[Rule.Common.USER_DEFINED.MicroCode]
FILE RAW = $(NAMED_GUID) {
$(INF_OUTPUT)/$(MODULE_NAME).bin
}
[Rule.Common.USER_DEFINED.LOGO]
FILE RAW = $(NAMED_GUID) {
|.bmp
}
As shown in the rule above,if SectionType and FileType not defined,
FFS files are generated directly, and no other type of file is
generated.
The patch is to make the BaseTools support these two rules
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=1815
The edk2 source tree contains four instances of the outdated mailing
list address "edk2-devel@lists.01.org".
I created a new variable, MSG_EDKII_MAIL_ADDR, to receive the
new email address and replaced the old one with this variable
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1788
In V3: Must generate hashes before attempting to copy from cache for
hash verifcation
In V2: Build failure caused by passing incorrect boolean parameter to
SaveFileOnChange(). Fixed for patch instances.
Library hashing is now supported by the --hash feature. The --hash
feature implementation assumed that the hashing could be done in
place once per module, but that isn't true for libraries due to the
fact that they are built as dependencies. So on a clean build, we now
generate the .hash after the library dependencies are complete.
Added early escape as optimization, if hash already exists in memory.
Signed-off-by: Christian Rodriguez <christian.rodriguez@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1648
When a module in a DSC file uses a FILE_GUID override in the
module scoped <Defines> section, a copy of the modified INF file
is placed in the Conf/.cache directory. The check for private
includes uses the INF path to determine if the module is allowed
to use the private includes. Since the INF path in this case is
not in any package, this check always fails, and no private
include paths are possible.
The fix is to keep both the OriginalPath and the new Path in
the PathClass object, and always use the OriginalPath to see if
the module INF is in the package with private includes.
Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1752
This patch is to fix the code bug in StructurePcd overall
value assignment logic. If a Pcd Array size is fixed but the
size of actual value in Dsc or Dec is bigger than the Pcd
array size, the tool will report error about such setting and
stop build.
The patch is tested minplatform, Ovmf, structure pcd regression test
These tests are build pass. The patch also tested the following cases.
Our cases focused on PcdArraySize. 1.flexiable PcdArraySize.
2.Fixed PcdArraySize, Pcd overall value exceeds the size of PcdArray.
3.Fixed PcdArraySize, Pcd overall value Not exceeds the size of PcdArray.
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
BZ: https://bugzilla.tianocore.org/process_bug.cgi
On Windows OS, sometime the generated file is not immediate saved to disk.
When run nmake, prompt AutoGen.h not found, and stop build.
Below blog shows Write-Replace to fix it. This patch uses this way to write
temp file, then rename the temp file to the real file.
https://blog.gocept.com/2013/07/15/reliable-file-updates-with-python/
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Signed-off-by: Yunhua Feng <yunhuax.feng@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1692
Add error handling to the --hash feature so that hash files
are invalidated when a build error occurs.
Signed-off-by: Christian Rodriguez <christian.rodriguez@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=1670
During BaseTools compiling under Chinese or Japanese
language Windows, python exception occurring.
UnicodeDecodeError: 'ascii' codec can't decode byte 0xbd
in position 3528: ordinal not in range(128)
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
Defined 2 PCDs(Test4 & Test401) and 2 SKUs(DEFAULT & _),
then set "SKUID_Defines" to ALL, for FixedAtBuild
gEfiStructuredPcdPkgTokenSpaceGuid. Test401 in this case,
its value should get from "Default" SKU, not from "_" SKU,
but we does not set value in SKU "_" in dsc, so Test401
should only display the value get from dec.
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
In ParseDevPathValue, the binary data need to convert to string
for python3 and python2 compatiblity.
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
PEP 238 -- Changing the Division Operator
x/y to return a reasonable approximation of the mathematical result
of the division ("true division")
x//y to return the floor ("floor division")
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
replace long by int
Because the long() was not exist in Python3
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yunhua Feng <yunhuax.feng@intel.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
delete the shared global variable from Common.Misc
delete the uncalled users of the variable from build.build
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
Refactor to 'dict' from 'IterableUserDict' which was only required for old
python interpreter.
Sort imports according to PEP8
Remove those we dont need.
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
no use for this code content.
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
These 2 functions are only used internally. move them to the smallest
scope and use them.
Cc: Bob Feng <Bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
Move PackRegistryFormatGuid logic from Common.Misc to this file.
There were no other consumers of the function.
As it is one line, just replace the logic without the separate function.
v2 - change to fix the import statement
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
Move SplitOption and ConvertStringToByteArray from
Common.Misc to this file.
There were no other consumers of the functions.
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
Move IsFieldValuieAnArray from Common.Misc to this file.
There were no other consumers of the function.
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
Move ProtocolValue and PpiValue from Common.Misc to this file.
There were no other consumers of these 2 functions.
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
Move DataDump and DataRestore from Common.Misc to this file.
There were no other consumers of these 2 functions.
Import threading since that module is used in build.
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
https://bugzilla.tianocore.org/show_bug.cgi?id=1466
DEC file defines PCD default value and PCD supported type.
DSC can configure PCD type and value.
If the value is same to default value in DEC file,
DSC can only configure PCD type and leave empty for value.
This usage supports all type PCD except for DynamicHii type.
So, DynamicHii PCD should support this usage. Below is one example in DSC.
for example,
[PcdsDynamicHii.common.DEFAULT]
PcdPkgTokenSpaceGuid.PcdCName|L"VarName"|gVarGuid|0x00||NV,BS
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
1 - Some of these imports are cascaded from another file. Import them locally.
2 - Some of these imports are not used. Remove them.
3 - Some of these were missing the namespace used to import them.
These changes facilitate optimization of BaseTools:
https://bugzilla.tianocore.org/show_bug.cgi?id=42
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
https://bugzilla.tianocore.org/show_bug.cgi?id=1390
1. support hex number for array index
2. support Non-Dynamic Pcd for array data type
3. support {} and {CODE()} for array data type
4. Change GetStructurePcdMaxSize to be a static function since it need to
be called in another static function. And this function does not depend on
it's class instance.
5. Add unittest for RemoveCComments function and
ArrayIndex regular expression.
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Cc: Philippe Mathieu-Daud? <philmd@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
https://bugzilla.tianocore.org/show_bug.cgi?id=1387
This patch is going to fix the regression issue which is
introduced by commit bf9e636605.
This patch Remove the CopyDict incorrect usage for non-dict
input data. Add a check for CopyDict input.
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Leif Lindholm <leif.lindholm@linaro.org>
Reviewed-by: Liming Gao <liming.gao@intel.com>
https://bugzilla.tianocore.org/show_bug.cgi?id=1385
This patch is going to fix the regression issue that is
introduced by commit e6eae3b4c7
and commit 0b6c5954e1.
PcdNvStoreDefaultValueBuffer value is update to Vpd Info File,
but it is not update into a internal cache. This patch will
fix this incorrect value in that internal cache.
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Leif Lindholm <leif.lindholm@linaro.org>
Reviewed-by: Liming Gao <liming.gao@intel.com>
https://bugzilla.tianocore.org/show_bug.cgi?id=1288
This patch is one of build tool performance improvement
series patches.
This patch is going to customize the deepcopy function for
SkuClass, PcdClassObject and python dictionary.
python deepcopy copy everything of a object, but for our current
usage we just need to copy the data we care about recursively.
By implementing __deepcopy__ for SkuClass, PcdClassObject, we can customize
deepcopy function for them.
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: BobCF <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
https://bugzilla.tianocore.org/show_bug.cgi?id=1288
This patch is one of build tool performance improvement
series patches.
This patch is going to use join function instead of
string += string2 statement.
Current code use string += string2 in a loop to combine
a string. while creating a string list in a loop and using
"".join(stringlist) after the loop will be much faster.
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: BobCF <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Jaben Carsey <jaben.carsey@intel.com>
Feng, Bob C
Jiewen Yao
Antoine Cœur
Jin, Eric
Shi, Steven
Rodriguez, Christian
Fan, ZhijuX
Michael D Kinney
Yunhua Feng
Carsey, Jaben