Commit Graph

394 Commits

Author SHA1 Message Date
Bob Feng b8ac0b7f28 BaseTools: Move Build Cache related function out of CreateAsBuiltInf
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1932
There are two functions in current CreateAsBuiltInf, Copy Binary files
to build cache folder and create asbuild inf file.

This patch is to separate UpdateBuildCache and CreateAsBuiltInf into
two functions.

Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Steven Shi <steven.shi@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Christian Rodriguez <christian.rodriguez@intel.com>
Reviewed-by: Steven Shi <steven.shi@intel.com>
Reviewed-by: Jaben Carsey <jaben.carsey@intel.com>
2019-06-25 09:23:57 +08:00
Bob Feng dc174cd9e5 BaseTools: refine CreateAsBuiltInf function
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1932
ModuleAutogen.CreateAsBuiltInf IsOnlyCopy parameter
and related function can be moved out of CreateAsBuiltInf,
so that CreateAsBuiltInf is more clean.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Steven Shi <steven.shi@intel.com>
Cc: Christian Rodriguez <christian.rodriguez@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Steven Shi <steven.shi@intel.com>
Reviewed-by: Jaben Carsey <jaben.carsey@intel.com>
2019-06-25 09:23:53 +08:00
Shi, Steven 2378ea5515 BaseTools:Introduce CopyFileOnChange() function to copy cache files
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=1894

Basetool need a CopyFileOnChange() function to avoid cache
file writing race in multi-thread build. Some platforms
build fail with file IO writing race issue when the
build cache is enabled to store cache files in multi-threads.
This is because common same library cache files (e.g. some
libs in MdePkg) can be stored by many different driver modules'
build threads at same time. Current build cache need a function
to check whether the same cache file already exist, and only
copy source file if it is different from the destination file.

This patch introduces an atomic copy function to avoid duplicated
cache files copy.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Christian Rodriguez <christian.rodriguez@intel.com>
Signed-off-by: Steven Shi <steven.shi@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-06-17 17:00:46 +08:00
Shi, Steven 17b082ce67 BaseTools: Cannot store library cache of different arch together
https://bugzilla.tianocore.org/show_bug.cgi?id=1895

Build cache cannot store cache for the same library modules
in different arch together. E.g. Both the below IA32 and X64
arch BaseLib caches should exist after build Ovmf3264, but now
only the one in X64 arch exist.
The reason is the current Basetool use a set() to same all
library AutoGen objects, but the different arch lib AutoGen
objects have same __hash_ value which comes from the lib
MetaFile(The path of module file):
    def __hash__(self):
        return hash(self.MetaFile)

So the different arch lib AutoGen objects are duplicated one
to the set() and only one can exist. This is why the Basetool
can only store one arch cache for library.

This patch adds the arch string into the PlatformAutoGen and
ModuleAutoGen __hash_ definitions and ensure the different
platform and module AutoGen objects have different __hash_ values.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Christian Rodriguez <christian.rodriguez@intel.com>
Signed-off-by: Steven Shi <steven.shi@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-06-17 17:00:45 +08:00
Shi, Steven ede54e3588 BaseTools:Build cache cannot store the cache files for library package
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1867

Current build cache cannot store the cache for library package.
build -p MdePkg\MdePkg.dsc -a IA32 -b DEBUG -t VS2015x86 --hash
--binary-destination=BinCache
After build, the expected result is the BinCache folder is generated
and the MdePkg build cache files (e.g. .hash and .lib) are stored in
the BinCache folder. But the BinCache folder is not generated at all.

This patch is going to fix that issue.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Zhiju.Fan <zhijux.fan@intel.com>
Signed-off-by: Steven Shi <steven.shi@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-06-17 17:00:45 +08:00
Rodriguez, Christian 48b0bf6476 BaseTools: Refactor hash tracking after checking for Sources section
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1804

After adding a check to see if [Sources] section lists all the header
type files of a module, track module and library hashes for --hash
feature. If above check is not in compilance for a library or module,
force hash invalidation on that library or module.

Signed-off-by: Christian Rodriguez <christian.rodriguez@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-06-10 19:48:51 +08:00
Rodriguez, Christian 1fa6699e6c BaseTools: Add a checking for Sources section in INF file
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1804

Add a check to see if [Sources] section lists all the header type
files of a module. Performance impact should be minimal with this patch
since information is already being fetched for Makefile purposes. All
other information is already cached in memory. No extra IO time is needed.

Signed-off-by: Christian Rodriguez <christian.rodriguez@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-06-10 19:48:51 +08:00
Steven Shi b6cfa686e4 BaseTools:Update binary cache restore time to current time
https://bugzilla.tianocore.org/show_bug.cgi?id=1742

Current Binary Cache doesn't update the restored file
creation and modification times to the current time.
Preserve the new restored file creation time as old
cached time might has potential issue to block the make
to build updated files based on the time stamp.
Enhance to update the restored file creation time to
current time.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Zhiju Fan <zhijux.fan@intel.com>
Signed-off-by: Steven Shi <steven.shi@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-05-31 19:46:18 +08:00
Steven Shi 3a0b998bf3 BaseTools:Extend the binary cache to support library cache
https://bugzilla.tianocore.org/show_bug.cgi?id=1797

Current binary cache doesn't support to save and restore
the library module. If a driver module cache miss happen,
all its dependency library modules need rebuild which
is very time-consuming. This patch is to extend the binary
cache to support library.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Christian Rodriguez <christian.rodriguez@intel.com>
Signed-off-by: Steven Shi <steven.shi@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-05-31 19:46:16 +08:00
Fan, ZhijuX c0fd7f734e BaseTools:Change the path of the file that Binary Cache
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=1806

Current Binary Cache use the PlatformInfo.Name to distinguish
the different platform cache paths in the database folder.

But the different platforms might use the same PLATFORM_NAME
in the DSC files. Different platform caches might overlay each
other inside Binary Cache database if they use same platform name

Use the PlatformInfo.Platform.OutputDirectory instead of the
PlatformInfo.Name can solve the problem.

This patch is going to fix that issue.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Steven Shi <steven.shi@intel.com>
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-05-27 08:59:04 +08:00
Rodriguez, Christian a7ef158b07 BaseTools: Library hashing fix and optimization for --hash feature
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1788

In V3: Must generate hashes before attempting to copy from cache for
hash verifcation
In V2: Build failure caused by passing incorrect boolean parameter to
SaveFileOnChange(). Fixed for patch instances.

Library hashing is now supported by the --hash feature. The --hash
feature implementation assumed that the hashing could be done in
place once per module, but that isn't true for libraries due to the
fact that they are built as dependencies. So on a clean build, we now
generate the .hash after the library dependencies are complete.
Added early escape as optimization, if hash already exists in memory.

Signed-off-by: Christian Rodriguez <christian.rodriguez@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-05-21 13:18:06 +08:00
Michael D Kinney 66b845ae06 BaseTools: Fix private includes for FILE_GUID override
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1648

When a module in a DSC file uses a FILE_GUID override in the
module scoped <Defines> section, a copy of the modified INF file
is placed in the Conf/.cache directory.  The check for private
includes uses the INF path to determine if the module is allowed
to use the private includes.  Since the INF path in this case is
not in any package, this check always fails, and no private
include paths are possible.

The fix is to keep both the OriginalPath and the new Path in
the PathClass object, and always use the OriginalPath to see if
the module INF is in the package with private includes.

Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-05-21 13:14:14 +08:00
Bob Feng c9b3fe1586 BaseTools: Support customized compiling command
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1737
User may add other commands before CC command from build_rule.txt
for specific purpose. It worked fine before commit 05217d210e.

This patch is going to fix the bug in commit 05217d210e to
support customized CC command.

Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-04-28 09:11:27 +08:00
Fan, ZhijuX 35c2af00d8 BaseTools:fixed the incorrect autogen makefile which cause build failure.
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=1729

On some build environment, build fails but on the other build machines,
build success. This is the regression issue introduced by commit
 05217d210e

As Dict is unordered, an error occurs when extract the index of the Dict
in the order of the keys after the creation of a new item.
Keys are indexed inconsistently before and after adding a new item.
The logic of the program is to store the key's corresponding index as
reference data in the MakeFile and use it as part of the macro.
The data model is: $(LIST_%d) % Dict.keys().index(Key)
So for now, use OrdereDict instead of Dict.

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-04-22 16:50:25 +08:00
Rodriguez, Christian 452b5ad61d BaseTools: Enhance Bin Cache database to support save the cache
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1726

V2: Added the platform name to the path
Add more level sub-directories in the database to support save
the cache for multiple platforms with multiple tool-chains and
targets, just like edk2 Build output.

Signed-off-by: Christian Rodriguez <christian.rodriguez@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-04-22 16:50:24 +08:00
Fan, ZhijuX 05217d210e BaseTools:Enable the /MP option of MSVC compiler
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=1672
The /MP option of MSVC compiler can reduce the total time to compile the
source files on the command line.

This patch is going to enable this MSVC option in BaseTools.

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-04-16 13:14:13 +08:00
Christian Rodriguez db4d47fd3a BaseTools: Fix corner-cases of --hash feature
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1680

Re-use libraries, since they have already been hashed.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Christian Rodriguez <christian.rodriguez@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-04-10 13:32:14 +08:00
Rodriguez, Christian f2b5e04aca BaseTools: Fix corner-cases of --hash feature
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1680

Respect artifact location within directory structure.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Christian Rodriguez <christian.rodriguez@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-04-10 13:32:13 +08:00
Rodriguez, Christian 1b8caf0d87 BaseTools: Fix corner-cases of --hash feature
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1680

Re-order hashing operations so we don't do redundant hashes.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Christian Rodriguez <christian.rodriguez@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-04-10 13:32:12 +08:00
Rodriguez, Christian 2914e8153d BaseTools: Fix corner-cases of --hash feature
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1680

Consider modules with .inc source files as Binary Modules
and do not Skip by hash.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Christian Rodriguez <christian.rodriguez@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-04-10 13:32:11 +08:00
Feng, Bob C b1e27d175a BaseTools: Fixed issue in MultiThread Genfds function
https://bugzilla.tianocore.org/show_bug.cgi?id=1450
In the Multiple thread Genfds feature, build tool generates
GenSec, GenFFS command in Makefile.

The Non-Hii Driver does not generate .offset file for uni string offset,
but the build tool has not knowledge about this in autogen phase. So
in this patch, I add a check in Makefile for GenSec command. If the GenSec
input file does not exist, the GenSec will not be called. And if GenSec
command is not called, its output file, which is also the input file of
GenFfs command, will also not exist.So for GenFfs command,
I add a new command parameter -oi which means
the input file is an optional input file which would not exist. so
that I can generate GenFfs command with "-oi" parameter in Makefile.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-04-10 13:32:10 +08:00
Michael D Kinney 2e351cbe8e BaseTools: Replace BSD License with BSD+Patent License
https://bugzilla.tianocore.org/show_bug.cgi?id=1373

Replace BSD 2-Clause License with BSD+Patent License.  This change is
based on the following emails:

  https://lists.01.org/pipermail/edk2-devel/2019-February/036260.html
  https://lists.01.org/pipermail/edk2-devel/2018-October/030385.html

RFCs with detailed process for the license change:

  V3: https://lists.01.org/pipermail/edk2-devel/2019-March/038116.html
  V2: https://lists.01.org/pipermail/edk2-devel/2019-March/037669.html
  V1: https://lists.01.org/pipermail/edk2-devel/2019-March/037500.html

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-04-09 09:10:20 -07:00
Feng, Bob C 2f2c51acfb BaseTools: Remove the logic SourceOverridePath
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1350
SOURCE_OVERRIDE_PATH is for EDK component INF files.
The corresponding logic should be removed.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-03-24 16:12:38 +08:00
mjohn4 b0189eac00 BaseTools: Explicitly close files after readlines
Rework some file open().readlines to open, readlines, close.
This prevents excessive file handles being open at the same time,
which may be a problem with alternative python environments.

Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Michael Johnson <michael.johnson@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-03-15 15:38:59 +08:00
Jiewen Yao d868846a4e BaseTool/Build: Add --disable-include-path-check.
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1620

This option is added to disable the include path check
for outside of package.
The original purpose of thie check is to make sure EDK II
modules must not reference header files outside of the packages
they depend on or within the module's directory tree.

However, we do see the usage to build EDKII as executable running
in the operating system which requires include path to outside.
For example, EmulatorPkg. The current solution (soft link) is
weird hack - EmulatorPkg\Unix\Host\X11IncludeHack.

With this solution, this can be supported easily.

The patch is validated with and without --disable-include-path-check.
If user does not use --disable-include-path-check, the build will fail
with outside path in the include path.
If user uses --disable-include-path-check, the build will pass
with outside path in the include path.

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jiewen Yao <jiewen.yao@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-03-15 13:29:29 +08:00
Liming Gao 3a4698202c Revert "BaseTools:BaseTools supports to the driver combination."
This reverts commit 838bc257ba.
After further evaluation, there are the unclear behavior in for the
driver combination feature. To not impact Q1 stable tag, remove it first.
1. If the drivers to be combined have the different PCD or library instance
   setting, build should not combine them and report build break. But this
   commit doesn't consider this case.
2. When start the sub driver fail, continue to start other sub driver. This
   behavior is required to be clarifed in build spec.
3. Unload the sub driver when the combined driver start fail. This case need
   to call the sub driver unload function for the driver start fail only.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-02-28 10:01:53 +08:00
Fan, ZhijuX 29c2de1272 BaseTools:Build fail if define [DEPEX] in library inf
When define [DEPEX] in lib inf to build, it will fail and report"
gUefiOvmfPkgTokenSpaceGuid.test1 used in [Depex] section should be
used as FixedAtBuild type and VOID* datum type in the module."
But we define this PCD to FixedAtBuild type and VOID* datum type indeed.

DEC:
[PcdsFixedAtBuild]
gUefiOvmfPkgTokenSpaceGuid.test1 |
{GUID("4096267b-da0a-42eb-b5eb-fef31d207cb4")}|VOID*|0x3c

DSC:
add pcd under lib inf as below:

NULL|TestPkg/TestLib/TestLib.inf
 <PcdsFixedAtBuild>
    gUefiOvmfPkgTokenSpaceGuid.test1 | {GUID(gUefiOvmfPkgTokenSpaceGuid)}

Lib inf:(TestPkg/TestLib/TestLib.inf)
[Depex]
gUefiOvmfPkgTokenSpaceGuid.test1
[FixedPcd]
gUefiOvmfPkgTokenSpaceGuid.test1

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-02-22 15:47:14 +08:00
Feng, Bob C f747640bf3 BaseTool: Fixed incremental rebuild issue.
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1540

This issue in introduced by commit
d943b0c339

To convert bytes to string, we need to use bytes.decode()
instead of using str(bytes).

If the source file is not a txt file, ignore that file.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-02-22 15:47:14 +08:00
Jiewen Yao 4247c24fba BaseTool/GenC: Fix build error when type is BASE or USER_DEFINED.
https://bugzilla.tianocore.org/show_bug.cgi?id=1544

1) The GenC tool does not include the constructor/destructor for
USER_DEFINED module. It should be included.
Only constructor/destructor with BASE type is included here.
The constructor/destructor with PEI/DXE/SMM type is still excluded
to keep original behavior.

2) The GenC tool includes the UnloadImage code for BASE module.
It should NOT be included.

3) The GenC tool uses EFI_STATUS and ASSERT_EFI_ERROR for BASE type.
It should use RETURN_STATUS and ASSERT_RETURN_ERROR.

4) The GenC tool miss DebugLib.h for BASE or USER_DEFINED module
AutoGen.c. Only Base.h is there. It should add Library/DebugLib.h.

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jiewen Yao <jiewen.yao@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-02-22 12:20:36 +08:00
Fan, ZhijuX e8d13fef90 BaseTools:Fixed build failure when specifying multiple BUILDTARGET
With Python3, the dict.value() method returns an iterator.
If a dictionary is updated while an iterator on its keys is used,
a RuntimeError is generated.
Converting the iterator to a list() forces a copy of the mutable
keys in an immutable list which can be safely iterated.

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-02-20 13:16:57 +08:00
Fan, ZhijuX 838bc257ba BaseTools:BaseTools supports to the driver combination.
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=1520

To save the image size without the compression, more than
one drivers can be combined into single one. When more than
one drivers are combined, their depex will be AND together.
Below is the example to combine BootManagerPolicyDxe into
DriverHealthManagerDxe.

Besides this patch, BaseTools also needs to check the module
type and make sure all module type are same. Otherwise,
BaseTools will report the error.
DRIVER INF has the parameter ENTRY_POINT
LIBRARY INF has the parameter LIBRARY_CLASS

Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-02-20 13:16:56 +08:00
Feng, Bob C c0ebd9d30f BaseTools: Fixed a bug in Vpd handling
If there are multiple sku used in a platform and
gEfiMdeModulePkgTokenSpaceGuid.PcdNvStoreDefaultValueBuffer PCD
is used, build will fail.

This is a regression issue introduced by the commit:
5695877ec8 which only update the
Vpd offset for default SKU but not other SKUs.

This patch is going to fix this issue.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-02-19 20:32:43 +08:00
Antoine Coeur fb0b35e05f BaseTools: Various typo
Various typo in BaseTools.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Coeur <coeur@gmx.fr>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-02-14 10:02:28 +08:00
Philippe Mathieu-Daud? bac50dfbcd BaseTools: Fix build failure when specifying multiple BUILDTARGET
With Python3, the dict.value() method returns an iterator.
If a dictionary is updated while an iterator on its keys is used,
a RuntimeError is generated.
Converting the iterator to a list() forces a copy of the mutable
keys in an immutable list which can be safely iterated.

Commit f8d11e5a4a converted various uses but missed one:
When specifying multiple BUILDTARGET, the first target builds
successfully, but then the PGen.BuildDatabase._CACHE_ dictionary is
updated, and accessing the next target triggers a RuntimeError.

Convert this iterator to an immutable list, to solve this build error:

    $ build -a IA32 -t GCC5 -b RELEASE -b NOOPT -p OvmfPkg/OvmfPkgIa32.dsc
    [...]
    Processing meta-data ...
    build.py...
     : error C0DE: Unknown fatal error when processing [OvmfPkg/OvmfPkgIa32.dsc]

    (Please send email to edk2-devel@lists.01.org for help, attaching following call stack trace!)

    (Python 3.5.3 on linux) Traceback (most recent call last):
      File "BaseTools/BinWrappers/PosixLike/../../Source/Python/build/build.py", line 2387, in Main
        MyBuild.Launch()
      File "BaseTools/BinWrappers/PosixLike/../../Source/Python/build/build.py", line 2141, in Launch
        self._MultiThreadBuildPlatform()
      File "BaseTools/BinWrappers/PosixLike/../../Source/Python/build/build.py", line 1921, in _MultiThreadBuildPlatform
        self.Progress
      File "BaseTools/Source/Python/AutoGen/AutoGen.py", line 304, in __init__
        self._InitWorker(Workspace, MetaFile, Target, Toolchain, Arch, *args, **kwargs)
      File "BaseTools/Source/Python/AutoGen/AutoGen.py", line 477, in _InitWorker
        for BuildData in PGen.BuildDatabase._CACHE_.values():
    RuntimeError: dictionary changed size during iteration

Note: The culprit commit (f8d11e5a4a) can not be found with bisection.
In 9c2d68c0a2 the build tools default to the python version provided
by the ${PYTHON} environment variable, however the Python3 transition is
not functional before d943b0c339. f8d11e5a4a falls between the
previous two.

Reported-by: Leif Lindholm <leif.lindholm@linaro.org>
Fixes: f8d11e5a4a
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Philippe Mathieu-Daude <philmd@redhat.com>
Tested-by: Leif Lindholm <leif.lindholm@linaro.org>
Acked-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Jaben Carsey <jaben.carsey@intel.com>
2019-02-06 13:08:16 -08:00
Feng, Bob C d943b0c339 BaseTools: Handle the bytes and str difference
Deal with bytes and str is different, remove the unicode(),
correct open file parameter.
Using utcfromtimestamp instead of fromtimestamp.

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-02-01 11:09:24 +08:00
Feng, Bob C f8d11e5a4a BaseTools: the list and iterator translation
In python3,The keys of the dictionary not a list,It needs to be converted

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-02-01 11:09:24 +08:00
Zhijux Fan 94c912950c BaseTools: Similar to octal data rectification
In python3, if Value is octal data, the int(Value, 0) report an error

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-02-01 11:09:22 +08:00
Zhijux Fan b3e94a0617 BaseTools: change the Division Operator
PEP 238 -- Changing the Division Operator
x/y to return a reasonable approximation of the mathematical result
    of the division ("true division")
x//y to return the floor ("floor division")

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-02-01 11:09:21 +08:00
Feng, Bob C 3f7cb70c5a BaseTools: Make sure AllPcdList valid.
This patch is to make sure the AllPcdList is always evaluated.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-02-01 11:09:20 +08:00
Zhijux Fan 2c2bb053a3 Basetools: It went wrong when use os.linesep
in python2 and python3,use of line breaks

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-02-01 11:09:20 +08:00
Zhijux Fan 1590d12315 BaseTools:Similar to octal data rectification
In python3, if Value is octal data, the int(Value, 0) report an error

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-02-01 11:09:18 +08:00
Yunhua Feng af881abc65 BaseTools: replace long by int
replace long by int
Because the long() was not exist in Python3

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yunhua Feng <yunhuax.feng@intel.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-02-01 11:09:18 +08:00
Yunhua Feng 1d2aa01cdf BaseTools: Remove unnecessary super function
Remove unnecessary super function

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Yunhua Feng <yunhuax.feng@intel.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-02-01 11:09:17 +08:00
Zhijux Fan 3a041437c9 BaseTools: Rename iteritems to items
replace the list iteritems by items in Python3.

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-02-01 11:09:16 +08:00
Zhijux Fan e77e59c9bd BaseTool:Rename xrange() to range()
Because the xrange() was not exist in Python3

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-02-01 11:09:16 +08:00
Carsey, Jaben 1bcc2cd63d BaseTools/AutoGen: move functions
Move SplitOption and ConvertStringToByteArray from
Common.Misc to this file.
There were no other consumers of the functions.

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-01-21 18:03:33 +08:00
Fan, ZhijuX dd1988cf01 BaseTools:build break if the Path contains SingleFile.Ext
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=1325

if SingleFile.Path = "/foo/bar.Sap/yada/source.S" and
SingleFile.Ext = ".S". Then key would end up "/foo/bar"
instead of "/foo/bar.Sap/yada/source" as intended.

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-01-21 18:03:30 +08:00
Felix Polyudov 842695d2ff BaseTools: Fix incorrect formatting of GenFds command dictionary
GenFdsCommand returned dictionary with elements that
are not compatible with GenFdsApi.
As a result the following options were not processed by GenFdsApi:
-v, -q, -d, --genfds-multi-thread, --ignore-sources
The issue is introduced by commit b3497bad12.

V2: Remove EFI_SOURCE references

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Felix Polyudov <felixp@ami.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-01-21 18:03:29 +08:00
Fan, ZhijuX 3e1ab49481 BaseTools:Build fail when PCD use in the [DEPEX] section of INF files
Update _FixedPcdVoidTypeDict to FixedVoidTypePcds
'_FixedPcdVoidTypeDict' no longer exists because edk2 version (b23414f6).
'ModuleAutoGen' object has no attribute '_FixedPcdVoidTypeDict'.
Build fail when PCD use in the [DEPEX] section of INF files

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-01-21 18:03:28 +08:00
Feng, Bob C 8229250132 BaseTools: Remove unused logic for EDKI
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1350
Remove EDK module type support from BaseTools python code.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-01-14 09:34:23 +08:00