250 Commits

Author SHA1 Message Date
Bob Feng
536a3e6726 BaseTools: Fix the build report crash issue
In the following corner case, the build report
will crash. This patch is to fix this problem.

Case:
Multiple SKU are used and 2 more DynamicHii structure Pcds
are set in dsc file under different SKU. And 1 more of those
Pcds are not used in any INF file.

Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <gaoliming@byosoft.com.cn>
Cc: Yuwei Chen <yuwei.chen@intel.com>
Reviewed-by: Liming Gao <gaoliming@byosoft.com.cn>
Reviewed-by: Yuwei Chen<yuwei.chen@intel.com>
2021-01-14 04:12:09 +00:00
Yunhua Feng
8577d63cd8 BaseTools: Update report map file format
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=2977

For a source-level BIOS debugger the .map files are quite useful with one
major shortcoming: the debugger cannot know, solely from the .map file,
the format (PE/COFF vs. TE) of the image included in the final BIOS ROM

Update report map file format

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <gaoliming@byosoft.com.cn>
Cc: Yuwei Chen <yuwei.chen@intel.com>
Signed-off-by: Yunhua Feng <fengyunhua@byosoft.com.cn>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2020-11-10 04:22:27 +00:00
Bob Feng
09af9bd9be BaseTools: Enable Module Scope Structure Pcd
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=2648

This patch is to enable the Module scoped Structure Pcd usage.
User can set structure pcd field value in module scope. For example,
under the [components] section of a dsc file, user can override some
field value for a specific module.

  Package/Module.inf{
      <PcdsFixedAtBuild>
      gUefiTokenSpaceGuid.StructurePcdModule.FieldName | 5
  }

Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <gaoliming@byosoft.com.cn>
Cc: Yuwei Chen <yuwei.chen@intel.com>

Tested-by: Liming Gao <gaoliming@byosoft.com.cn>
Acked-by: Liming Gao <gaoliming@byosoft.com.cn>
2020-11-05 04:27:28 +00:00
Bob Feng
5dc2699d10 BaseTools: Move CreateAsBuiltInf into AutoGenWorker for parallel
AsBuiltInf can be created during AutoGen phase. Move CreateAsBuiltInf
into AutoGenWorker to make this step run in parallel.

Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yuwei Chen <yuwei.chen@intel.com>
Reviewed-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2020-08-14 01:43:32 +00:00
Feng, Bob C
bfe36cb4ef BaseTools: Improve the method of checking queue empty
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=2807

The Queue.empty() method is not reliable in the multiple
process runtime environment. This patch uses a new method
to check if all modules are processed and workers need
to be stopped. That is to add a None item at the bottom
of the queue. Worker check if it gets that None item to
know if all the module is processed.

Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yuwei Chen <yuwei.chen@intel.com>
Cc: Lucy Yan <lucyyan@google.com>

Reviewed-by: Liming Gao <liming.gao@intel.com>
2020-08-07 01:44:36 +00:00
Irene Park
28dd887d68 BaseTools/build.py: Exit with 1 when AutoGen error occurred
AutoGen manager/workers halt the progress when an error occurs but
doesn't propagate the error code to main and allows main exit with 0
and gets the build system unable to catch the occurrence of an error.
This change informs main with an error when a progress is halted and
helps main exit with 1.

Signed-off-by: Irene Park <ipark@nvidia.com>
Reviewed-by: Bob Feng<bob.c.feng@intel.com>
2020-06-07 12:51:25 +00:00
Abner Chang
f60d5ca97f BaseTools: BaseTools changes for RISC-V platform.
Python code changes for building EDK2 RISC-V platform.

Signed-off-by: Abner Chang <abner.chang@hpe.com>
Co-authored-by: Gilbert Chen <gilbert.chen@hpe.com>
Co-authored-by: Daniel Helmut Schaefer <daniel.schaefer@hpe.com>
Reviewed-by: Leif Lindholm <leif.lindholm@linaro.org>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Leif Lindholm <leif.lindholm@linaro.org>
Cc: Gilbert Chen <gilbert.chen@hpe.com>
2020-04-29 02:52:08 +00:00
Fan, Zhiju
3a3a3af4a2 BaseTools:Add the spare space FV image size checker
REF:https://bugzilla.tianocore.org/show_bug.cgi?id=2654

If FV is placed in FD region, its FV image size is fixed.
When FV image size exceeds it, it will trig the build break.
To alert the developer to adjust FV image size earlier,
I request to add new checker for the the spare FV space.
When the spare FV space is less than the specified threshold,
build tool will report the error.

This checker is the optional.
It can be enabled by -D FV_SPARE_SPACE_THRESHOLD=10000.
Macro is the value of the spare space threshold size.
It can be decimal or hex format. If it is enabled,
BaseTools will check every FV with the fixed size.
If FV doesn't meet with the size requirement,
Build tool will report error message to say there is no
enough spare space.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2020-04-23 01:29:34 +00:00
Fan, ZhijuX
8c944c9383 BaseTools:GuidedSectionTools.txt is not generated correctly
REF:https://bugzilla.tianocore.org/show_bug.cgi?id=2538

For LzmaCompress or BrotliCompress, the platform may use the different
options and add their batch file, such as LzmaCompressPlatform.
Then, specify it in platform.dsc [BuildOptions] to override the default
one in tools_def.txt.

*_*_*_LZMA_PATH = LzmaCompressPlatform

This override tool will be used. But, its name is not specified in the
generated GuidedSectionTools.txt.

Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2020-03-31 04:09:43 +00:00
Fan, ZhijuX
ec97412b7c BaseTools: Fixed build failure when using python38
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=2304

SyntaxWarning: "is" with a literal. Did you mean "=="?
Using "is" instead of "==" is an irregular syntax

Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Cc: Bob C Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>

Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2020-02-12 14:54:03 +00:00
Pierre Gondois
818283de3f BaseTools: Rationalise makefile generation
The GenMake.py script tests the platform environment
to determine the type of makefile that needs to be
generated. If a Windows build host is detected, the
makefile generated is of Nmake type. Otherwise a
GNUmake type is generated.

Furthermore, the <TARGET>_<TAGNAME>_<ARCH>_MAKE_PATH
option in tools_def.template defines the make tool
to use.
E.g.: for VS2017 this is configured to use Nmake, cf.
*_VS2017_*_MAKE_PATH = DEF(VS2017_BIN_HOST)\nmake.exe
while for GCC5 it is setup to use GNU make.
*_GCC5_*_MAKE_PATH = DEF(GCC_HOST_PREFIX)make

This prevents using the GCC compiler toolchain on a
Windows build host.

To address this issue this patch introduces 2 factors
to determine the generated makefile output.
  1. Platform -> to determine shell commands used
                 in makefile.
  2. MakeTool -> to determine the type of makefile
                 that needs to be generated.

Signed-off-by: Pierre Gondois <pierre.gondois@arm.com>
Signed-off-by: Sami Mujawar <sami.mujawar@arm.com>

Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2020-02-12 02:34:44 +00:00
Bob Feng
5cd3d4bc43 BaseTools: Fixed a incremental build bug
REF:https://bugzilla.tianocore.org/show_bug.cgi?id=2451

If removing a header file from source code and file
system, the incremental build will fail.

This patch is to fix this issue by setting each header file
as a target without any actions in makefile.

Signed-off-by: Bob Feng <bob.c.feng@intel.com>

Cc: Liming Gao <liming.gao@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2020-01-19 07:46:18 +00:00
Fan, ZhijuX
4465cd124f BaseTools:Fix GenFds issue for BuildOption replace GenFdsOption
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=2455

BuildOption is used by TargetTxtClassObj.py
GenFdsOption is used by GenFds.py
When the GenFds tool is used alone (e.g. python3 -m GenFds.GenFds -h)
With the OptionParser function, the first detected function
prints the help message

import TargetTxtClassObj to GenFds,
The BuildOption will be executed and replace GenFdsOption

We removed all objects associated with this problem that
were created directly during the import process
(e.g. BuildOption, BuildTarget = MyOptionParser(),
 TargetTxt = TargetTxtDict())

The Patch is going to fix this issue

Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>

Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2020-01-13 02:08:46 +00:00
Bob Feng
01b6090b75 BaseTools: Resolve a issue of Incremental build
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=2311

In patch set 13c5e34a - 0c3e8e99, we implemented incremental build with
using compiler/pre-processor generate dependent header file function.

A issue is found for MSVC compiler, that the cl.exe /showIncludes
build option generate header file list to either stdout or stderr.
For .c file, the header file list is print out to stdout while for
.vfr, .aslc and .nasm file, the file list is print out to stderr.

The build tool use two threads to process the message from stdout and
stderr, but to generate correct *.deps file, build tool need to
combine the header file list from stderr and other messages from stdout
together with correct time sequence order.

So this patch is trying to combine the stdout and stderr together for
the process which is for calling make program.

The impact of this patch is that the output message of build with -q
will be changed. The compiler error message will not print out.
The build behavior of other log level setting will not be impacted.

Signed-off-by: Bob Feng <bob.c.feng@intel.com>

Cc: Liming Gao <liming.gao@intel.com>
Cc: Steven Shi <steven.shi@intel.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-12-18 07:24:45 +00:00
Bob Feng
78fb6b0e02 BaseTools: Fix build failure when multiple build targets given
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=2371

This patch is to fix a regression issue that build fails
if multiple build targets given.

Two changes cause this regression issue.
One is AutoGen object __hash__ function only
hash file path and arch, missing ToolChain and build target.

The other is changing the multiple-thread-genfds function as default
build behavior. To generate the genffs command to Makefile, there
is a global data set is used, GenFdsGlobalVariable, which cause build
tool use the data of first build-target build in
the second build-target build.

Signed-off-by: Bob Feng <bob.c.feng@intel.com>

Cc: Liming Gao <liming.gao@intel.com>
Cc: Steven Shi <steven.shi@intel.com>
Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-12-18 01:01:57 +00:00
Steven Shi
fc8b8deac2 BaseTools: Leverage compiler output to optimize binary cache
Redesign the binary cache and bases on the compiler to
output the dependency header files info for every module.
The binary cache will directly consume the dependency header
files info and doesn't parse the C source code by iteself.
Also redesign the dependency files list format for module
and try to share the common lib hash result as more as
possible in local process. Remove the unnecessary share data
access across multiprocessing.

Signed-off-by: Steven Shi <steven.shi@intel.com>

Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-12-10 02:07:12 +00:00
Bob Feng
0c3e8e9947 BaseTools: Enhance Basetool for incremental build
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=2311

Include dependency file in Makefile to enhance
incremental build

Signed-off-by: Bob Feng <bob.c.feng@intel.com>

Cc: Liming Gao <liming.gao@intel.com>
Cc: Steven Shi <steven.shi@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-12-10 01:31:55 +00:00
Fan, ZhijuX
e0f8261ad0 BaseTools:fix regression issue for platform .map file
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=2363

This patch is to fix a build tool regression issue which was introduced
by commit b8ac0b7f28.This issue caused map file lost the line of IMAGE=***.
For example,in Ovmf.map, there is no line of (IMAGE=<path to efi> ) under
each of modules item.

The path to the efi file generated by each module is written on this line
The purpose of this line is add the debug image full path.
there is no information about the module in the map file other than FVName,
it allows us to quickly know which module this part corresponds to.

In commit b8ac0b7f28,add a line ("self.BuildModules = []") in function,
but it's used to calculate the variable ModuleList in the following code.

Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>

Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-11-25 06:01:10 +00:00
Ni, Ray
49fb9f7e06 BaseTools: Fix build failure when using python38
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=2304

Signed-off-by: Ray Ni <ray.ni@intel.com>
Cc: Bob C Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-11-13 03:47:48 +00:00
Fan, ZhijuX
85ccbee2ab BaseTools:Fix the issue that build report failed
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=2201

An error occurs using special VpdPcd that is not used in the Inf file

In dsc:
[PcdsDynamicExVpd.common.DEFAULT]
gBoardModuleTokenSpaceGuid.test1|*|{CODE({
  {0x0}  // terminator
})}

In dec:
[PcdsDynamicEx]
# Vpd GPIO table
  gBoardModuleTokenSpaceGuid.test1|{0}|GPIO_INIT_CONFIG[]|0x50000018 {
  <HeaderFiles>
    Library/GpioLib.h
  <Packages>
    MdePkg/MdePkg.dec
}
ValueError: invalid literal for int() with base 0: '*'

This Patch is going to fix issue

Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-09-19 17:11:57 +08:00
Fan, ZhijuX
b67735a7e8 BaseTools:Replace PlatformInfo with PlatformAutoGen for Moudle
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=2188

build -p MdeModulePkg\MdeModulePkg.dsc -a IA32 -m
MdeModulePkg\Universal\PCD\Pei\Pcd.inf

Error:
AttributeError: 'PlatformInfo' object has no attribute
'DynamicPcdList'

The DSC data object used to build a separate module today
is PlatformInfo rather than PlatformAutoGen
'PlatformAutoGen' object has attribute 'DynamicPcdList'

This patch is going to fixed this issue

Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-09-17 10:18:51 +08:00
Feng, Bob C
5a9db85880 BaseTools: Fixed a bug of IgnoreAutoGen
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=2080
After checking that if the build can't
ignore Autogen due to there is no compelet autogen files,
the build tool need to do a completely Autogen.

This patch is to fix a bug that if AutoGen
can't be skiped, the SkipAutoGen flag
need to set to False

Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-09-12 14:08:15 +08:00
Feng, Bob C
7809492c10 BaseTools: Enable --genfds-multi-thread to default build
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1302

This patch enable --genfds-multi-thread to default build.
This patch keep --genfds-multi-thread build option for
compatibility and also add a new build option to disable
genfds-multi-thread as --no-genfds-multi-thread.

Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-09-11 09:43:59 +08:00
Feng, Bob C
e3c8311f8a BaseTools: Fixed the bug of multi-thread genffs for override inf
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1302
If there is a inf override, and multi-thread genffs is enabled, build
will fail.

This patch is going to fix this bug.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Steven Shi <steven.shi@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-09-11 09:43:58 +08:00
Feng, Bob C
373298ca0d BaseTools: Fixed issue for IgnoreAutoGen
https://bugzilla.tianocore.org/show_bug.cgi?id=2080

This patch is to improve build -u option to re-use
GlobalVar_<platformguid>_<arch>.bin file which is
introduced by multiple-process-autogen feature.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Steven Shi <steven.shi@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Acked-by: Liming Gao <liming.gao@intel.com>
2019-09-10 08:39:18 +08:00
Feng, Bob C
53e2eaba7a BaseTools: Fixed build clean regression issue
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=2166

file_lock and cache_lock are used as global data,
so move file_lock and cache_lock initialization in Build
object __init__ function.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Steven Shi <steven.shi@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-09-10 08:39:17 +08:00
Feng, Bob C
5f7bb39117 BaseTools: Fix incremental build genmake issue
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=2090

This is a regression issue introduced by commit e8449e.
This patch is going to fix this issue.

Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-08-23 11:26:24 +08:00
Fan, ZhijuX
0970a80583 BaseTools: Update incorrect variable name 'DataPile'
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=2093

The PlatformAutoGen object has a DataPipe property but no DataPile property
So change the variable name 'DataPile' to 'DataPipe' in BuildReport.py

This patch is going to fix that issue.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-08-20 14:15:47 +08:00
Shi, Steven
94459080c1 BaseTools: Improve the file saving and copying reliability
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=2079

The Basetool CopyFileOnChange() and SaveFileOnChange()
functions might raise the IOError occasionally when build
in Windows with multi-process and build cache enabled.
The CopyFileOnChange() and SaveFileOnChange() might be invoked
in multiple sub-processes simultaneously, and this patch adds
global locks to sync these functions invoking which can
harden their reliability.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Signed-off-by: Steven Shi <steven.shi@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-08-20 14:06:07 +08:00
Shi, Steven
c340c5bd1b BaseTools: Change the [Arch][Name] module key in Build cache
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1951

Current build cache use the module's [self.Arch][self.Name]
info as the ModuleAutoGen object key in hash list and dictionary.
The [self.Arch][self.Name] is not safe as the module key because
there could be two modules with same module name and arch name in
one platform. E.g. A platform can override a module or library
instance in another different path, the overriding module can has
the same module name and arch name as the original one.
Directly use the ModuleAutoGen obj self as the key, because
the obj __hash__ and __repr__ attributes already contain the
full path and arch name.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Signed-off-by: Steven Shi <steven.shi@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-08-20 14:06:06 +08:00
Shi, Steven
0e7e7a264c BaseTools: Improve the cache hit in the edk2 build cache
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1927

Current cache hash algorithm does not parse and generate
the makefile to get the accurate dependency files for a
module. It instead use the platform and package meta files
to get the module depenedency in a quick but over approximate
way. These meta files are monolithic and involve many redundant
dependency for the module, which cause the module build
cache miss easily.
This patch introduces one more cache checkpoint and a new
hash algorithm besides the current quick one. The new hash
algorithm leverages the module makefile to achieve more
accurate and precise dependency info for a module. When
the build cache miss with the first quick hash, the
Basetool will caculate new one after makefile is generated
and then check again.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Signed-off-by: Steven Shi <steven.shi@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-08-20 14:06:05 +08:00
Bob Feng
08a54c9e0a BaseTools: Move buildoptions.py to Common folder
buildoptions is used by build.py and TargetTxtClassObj.py
Move buildoptions.py to Common folder so that
build tool and ECC tool work fine.

Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-08-14 13:54:49 +08:00
Feng, Bob C
1a624dd7cf BaseTools: Enable block queue log agent.
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1875

To support Ctrl+S and Ctrl+Q, we enable block queue
for log.

Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Acked-by: Liming Gao <liming.gao@intel.com>
2019-08-09 23:15:55 +08:00
Feng, Bob C
c60fb00f6c BaseTools: Move BuildOption parser out of build.py
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1875

Build tool supports user to specify the conf folder.
To make the build options be evaluated at the beginning
of launching build, extract the buildoption function
from build.py to a new .py file.

Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Acked-by: Liming Gao <liming.gao@intel.com>
2019-08-09 23:15:54 +08:00
Feng, Bob C
636ed13a7f BaseTools: Add LogAgent to support multiple process Autogen
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1875

AutoGen processes race the logfile. To resolve this issue,
this patch create a LogAgent thread in main process to write
the log content to console or file, Other process will send
the log content to the LogAgent.

Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Acked-by: Liming Gao <liming.gao@intel.com>
2019-08-09 23:15:54 +08:00
Feng, Bob C
3285fbda88 BaseTools: Add shared data for processes
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1875

Add shared data for autogen processes.

Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Acked-by: Liming Gao <liming.gao@intel.com>
2019-08-09 23:15:53 +08:00
Feng, Bob C
673d09a2dd BaseTools: Enable Multiple Process AutoGen
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1875

Assign the Module AutoGen tasks into multiple
sub process.

Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Acked-by: Liming Gao <liming.gao@intel.com>
2019-08-09 23:15:53 +08:00
Feng, Bob C
e8449e1d8e BaseTools: Decouple AutoGen Objects
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1875

1. Separate the AutoGen.py into 3 small py files.
One is for AutoGen base class, one is for WorkspaceAutoGen class
and PlatformAutoGen class, and the one for ModuleAutoGen class.
2. Create a new class DataPipe to store the Platform scope settings.
Create a new class PlatformInfo to provide the same interface
as PlatformAutoGen. PlatformInfo class is initialized by
DataPipe instance.
Create a new class WorkspaceInfo to provide the same interface
as WorkspaceAutoGen. WorkspaceInfo class is initialized by
DataPipe instance.
3. Change ModuleAutoGen to depends on DataPipe, PlatformInfo and
WorkspaceInfo. Remove the dependency of ModuleAutoGen to PlatformAutoGen.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Steven Shi <steven.shi@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Acked-by: Liming Gao <liming.gao@intel.com>
2019-08-09 23:15:52 +08:00
Feng, Bob C
db01c8e3d8 BaseTools: Singleton the object to handle build conf file
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1875

The build config files are target.txt, build rule, tooldef
During a build, the config is not changed, so the object to
handle them need to be singleton.

Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Acked-by: Liming Gao <liming.gao@intel.com>
2019-08-09 23:15:51 +08:00
Matthew Carlson
101dc9e257 BaseTools:Added arch output to build report
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=2016

Added arch output to build report so it's easy to tell
which arch is being generated. Useful when multiple versions
of a single module is being emitted for multiple archs.

This patch is going to Added arch output

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-08-01 15:36:49 +08:00
Feng, Bob C
5f89bcc460 BaseTools: Create ".cache" folder when initialize Build object
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1986

Create "Conf/.cache" folder as early as possible
so that the later code do need to check
if it exits and then create it.

Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Reviewed-by: Michael D Kinney <michael.d.kinney@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-07-22 09:21:14 +08:00
Cole Robinson
eebc135ffb BaseTools: Fix python3.8 SyntaxWarning
Building with python3.8 shows a warning like:

SyntaxWarning: invalid escape sequence \(
  GuidName = re.compile("\(GUID=[-a-fA-F0-9]+")

It seems harmless, but it's easy enough to fix: mark the string as
raw with the 'r' prefix like is used elsewhere in the file

Signed-off-by: Cole Robinson <crobinso@redhat.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-07-15 09:09:25 +08:00
Bob Feng
b8ac0b7f28 BaseTools: Move Build Cache related function out of CreateAsBuiltInf
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1932
There are two functions in current CreateAsBuiltInf, Copy Binary files
to build cache folder and create asbuild inf file.

This patch is to separate UpdateBuildCache and CreateAsBuiltInf into
two functions.

Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Steven Shi <steven.shi@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Christian Rodriguez <christian.rodriguez@intel.com>
Reviewed-by: Steven Shi <steven.shi@intel.com>
Reviewed-by: Jaben Carsey <jaben.carsey@intel.com>
2019-06-25 09:23:57 +08:00
Bob Feng
dc174cd9e5 BaseTools: refine CreateAsBuiltInf function
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1932
ModuleAutogen.CreateAsBuiltInf IsOnlyCopy parameter
and related function can be moved out of CreateAsBuiltInf,
so that CreateAsBuiltInf is more clean.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Steven Shi <steven.shi@intel.com>
Cc: Christian Rodriguez <christian.rodriguez@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Steven Shi <steven.shi@intel.com>
Reviewed-by: Jaben Carsey <jaben.carsey@intel.com>
2019-06-25 09:23:53 +08:00
Shi, Steven
04df18535c BaseTools:Build Cache output notification message
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=1868

Build need output the cache miss or hit notification
message when consume the build cache. Current build does not
output any message which is not clear for user to know
whether the module built result is from cache or not.

This patch adds message about the cache miss or hit when
build consumes the cache.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Zhiju.Fan <zhijux.fan@intel.com>
Signed-off-by: Steven Shi <steven.shi@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-06-17 17:00:45 +08:00
Rodriguez, Christian
48b0bf6476 BaseTools: Refactor hash tracking after checking for Sources section
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1804

After adding a check to see if [Sources] section lists all the header
type files of a module, track module and library hashes for --hash
feature. If above check is not in compilance for a library or module,
force hash invalidation on that library or module.

Signed-off-by: Christian Rodriguez <christian.rodriguez@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-06-10 19:48:51 +08:00
Bob Feng
e812a812c1 BaseTools: Remove './SecMain' from 'run' target
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1561

This patch is related to commit 9ca1f1
It's to remove the second .SecMain call from 'run' target.

Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Michael D Kinney <michael.d.kinney@intel.com>
2019-05-24 15:28:42 +08:00
Feng, Bob C
9ca1f1446a BaseTools: Remove './SecMain' from 'run' target
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1561

When a target of 'run' is passed into build.py,
BaseTools unconditionally attempts to execute the
application called './SecMain' in the build output directory.

This behavior applies to the Nt32Pkg which is being
replaced with features in the EmulatorPkg.

Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Michael D Kinney <michael.d.kinney@intel.com>
2019-05-22 09:15:56 +08:00
Fan, ZhijuX
c1387446ed BaseTools:Update mailing list address in BaseTools error messages
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=1815

The edk2 source tree contains four instances of the outdated mailing
list address "edk2-devel@lists.01.org".
I created a new variable, MSG_EDKII_MAIL_ADDR, to receive the
new email address and replaced the old one with this variable

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-05-22 09:15:55 +08:00
Rodriguez, Christian
a7ef158b07 BaseTools: Library hashing fix and optimization for --hash feature
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1788

In V3: Must generate hashes before attempting to copy from cache for
hash verifcation
In V2: Build failure caused by passing incorrect boolean parameter to
SaveFileOnChange(). Fixed for patch instances.

Library hashing is now supported by the --hash feature. The --hash
feature implementation assumed that the hashing could be done in
place once per module, but that isn't true for libraries due to the
fact that they are built as dependencies. So on a clean build, we now
generate the .hash after the library dependencies are complete.
Added early escape as optimization, if hash already exists in memory.

Signed-off-by: Christian Rodriguez <christian.rodriguez@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-05-21 13:18:06 +08:00