Commit Graph

320 Commits

Author SHA1 Message Date
Konstantin Aladyshev 0f6eccdbf7 BaseTools: Add missing spaces for PCD expression values in AutoGenC
Currently the PCD values calculated from the expressions have different
formating from the simple byte arrays in AutoGenC.

Example:
The following definition in DEC:
gTokenSpaceGuid.PcdArray|{0x44, 0x33, 0x22, 0x11}|VOID*|0x55555555
gTokenSpaceGuid.PcdArrayByExpression|{UINT32(0x11223344)}|VOID*|0x66666666

Produces these strings in AutoGenC:
<...> _gPcd_<...>_PcdArray[4] = {0x44, 0x33, 0x22, 0x11};
<...> _gPcd_<...>_PcdArrayByExpression[4] = {0x44,0x33,0x22,0x11};

Add missing space character between the array elements to unify PCD value
formatting.

Signed-off-by: Konstantin Aladyshev <aladyshev22@gmail.com>
Reviewed-by: Liming Gao <gaoliming@byosoft.com.cn>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2022-10-19 06:32:07 +00:00
Konstantin Aladyshev 81aeb94648 BaseTools: Correct BPDG tool error prints
Popen communication returns bytestrings. It is necessary to perform
decode on these strings before passing them to the EdkLogger that
works with ordinary strings.

Signed-off-by: Konstantin Aladyshev <aladyshev22@gmail.com>
Reviewed-by: Liming Gao <gaoliming@byosoft.com.cn>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2022-10-19 06:32:07 +00:00
Chao Li c53807cb7b BaseTools: Updated build tools to support new LoongArch.
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=4053

Python code changes for building EDK2 LoongArch platform.

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <gaoliming@byosoft.com.cn>
Cc: Yuwei Chen <yuwei.chen@intel.com>

Signed-off-by: Chao Li <lichao@loongson.cn>
Co-authored-by: Baoqi Zhang <zhangbaoqi@loongson.cn>
Reviewed-by: Liming Gao <gaoliming@byosoft.com.cn>
2022-10-14 02:16:33 +00:00
Yi Li bf9230a9f3 BaseTools: Add the FeatureFlagExpression usage to the Source Section
REF:https://bugzilla.tianocore.org/show_bug.cgi?id=3828

FeatureFlagExpression Support in Source section of INF file. The Pcd
value in the expression is from INF or DEC.
When a FeatureFlagExpression is present,if the expression evaluates
to TRUE,then the entry is valid. If the expression evaluates to FALSE,
then the EDK II build tools must ignore the entry.

This patch is going to add this feature.

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <gaoliming@byosoft.com.cn>
Cc: Heng Luo <heng.luo@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
Signed-off-by: Yi Li <yi1.li@intel.com>
2022-03-26 00:59:18 +00:00
Chen, Lin Z 22c3b5a865 BaseTools: Add authenticated variable store support
In order to support secure boot with authenticated type variable store
and non secure boot with normal type variable store, add one flag to
switch them.

User can append '-D VPD_AUTHENTICATED_VARIABLE_STORE' to build command
to enable authenticated type varaible store.

Also, user can add 'VPD_AUTHENTICATED_VARIABLE_STORE = TRUE/FALSE' to the
defines section of Dsc file to switch authenticated/normal type variable
store.

VPD_AUTHENTICATED_VARIABLE_STORE is a new reserved key word for this function.

Signed-off-by: Chen Lin Z <lin.z.chen@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <gaoliming@byosoft.com.cn>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2021-11-11 08:12:19 +00:00
Bob Feng 40a9066439 BaseTools: Enable the flag to treat dynamic pcd as dynamicEx
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=1688

In order to support binary build, build tool add a
flag to convert type of Dynamic Pcd to DynamicEx Pcd

User can append -D PCD_DYNAMIC_AS_DYNAMICEX to build command
to enable this function.
Also, user can add "PCD_DYNAMIC_AS_DYNAMICEX = TRUE/FALSE" to the
defines section of Dsc file to enable this function.
PCD_DYNAMIC_AS_DYNAMICEX is a new reserved key word for this function.

Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yuwei Chen <yuwei.chen@intel.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Nate DeSimone <nathaniel.l.desimone@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2021-07-12 07:40:07 +00:00
Sughosh Ganu 5531fd48de BaseTools: Add support for version 3 of FMP Image Header structure
Add support for the ImageCapsuleSupport field, introduced in version 3
of the EFI_FIRMWARE_MANAGEMENT_CAPSULE_IMAGE_HEADER structure. This
structure member is used to indicate if the corresponding payload has
support for authentication and dependency.

Signed-off-by: Sughosh Ganu <sughosh.ganu@linaro.org>
Reviewed-by: Michael D Kinney <michael.d.kinney@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2021-05-12 01:03:53 +00:00
Michael D Kinney ab957f036f BaseTools/Source/Python: New Target/ToolChain/Arch in DSC [BuildOptions]
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=3359

Update BaseTools to support new build targets, new tool chains,
and new architectures declared in DSC file [BuildOptions] sections.

* Do not expand * when tools_def.txt is parsed.  Only expand when
  both tools_def.txt and DSC [BuilsOptions] sections have been parsed.
  This also requires more flexible matching of tool keys that contain *
  in tool key fields.

* Pre-scan the platform DSC file for FAMILY and TOOLCHAIN declarations
  DSC in [BuildOptions] sections before the FAMILY and TOOLCHAIN need
  to be known.

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <gaoliming@byosoft.com.cn>
Cc: Yuwei Chen <yuwei.chen@intel.com>
Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2021-04-29 07:29:17 +00:00
Cole Robinson 43bec9ea3d BaseTools: Work around array.array.tostring() removal in python 3.9
In python3, array.array.tostring() was a compat alias for tobytes().
tostring() was removed in python 3.9.

Convert this to use tolist() which should be valid for all python
versions.

This fixes this build error on python3.9:

(Python 3.9.0b5 on linux) Traceback (most recent call last):
  File "/root/edk2/edk2-edk2-stable202002/BaseTools/BinWrappers/PosixLike/../../Source/Python/Trim/Trim.py", line 593, in Main
    GenerateVfrBinSec(CommandOptions.ModuleName, CommandOptions.DebugDir, CommandOptions.OutputFile)
  File "/root/edk2/edk2-edk2-stable202002/BaseTools/BinWrappers/PosixLike/../../Source/Python/Trim/Trim.py", line 449, in GenerateVfrBinSec
    VfrUniOffsetList = GetVariableOffset(MapFileName, EfiFileName, VfrNameList)
  File "/root/edk2/edk2-edk2-stable202002/BaseTools/Source/Python/Common/Misc.py", line 88, in GetVariableOffset
    return _parseForGCC(lines, efifilepath, varnames)
  File "/root/edk2/edk2-edk2-stable202002/BaseTools/Source/Python/Common/Misc.py", line 151, in _parseForGCC
    efisecs = PeImageClass(efifilepath).SectionHeaderList
  File "/root/edk2/edk2-edk2-stable202002/BaseTools/Source/Python/Common/Misc.py", line 1638, in __init__
    if ByteArray.tostring() != b'PE\0\0':
AttributeError: 'array.array' object has no attribute 'tostring'

Signed-off-by: Cole Robinson <crobinso@redhat.com>
Reviewed-by: Yuwei Chen <yuwei.chen@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2020-08-14 02:21:18 +00:00
Yuwei Chen d35773d5c0 BaseTools: Add Guid name support in GenFfs.
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=2762

The Fv Section in the FDF files use hard coding Guid values
which is inconvenient to manage. This patch adds Guid name
support in GenFfs to solve this problem.

Signed-off-by: Yuwei Chen <yuwei.chen@intel.com>
Reviewed-by: Bob Feng<bob.c.feng@intel.com>
2020-07-17 05:25:41 +00:00
Feng, YunhuaX ceacd9e992 BaseTools: Fix parse PCD GUID expression issue
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=2686

The build tool will give an incorrect GUID value if the GUID includes character ' or " ASCII value.
This patch is going to fix this issue.

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Yunhua Feng <yunhuax.feng@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2020-05-13 06:37:18 +00:00
Abner Chang f60d5ca97f BaseTools: BaseTools changes for RISC-V platform.
Python code changes for building EDK2 RISC-V platform.

Signed-off-by: Abner Chang <abner.chang@hpe.com>
Co-authored-by: Gilbert Chen <gilbert.chen@hpe.com>
Co-authored-by: Daniel Helmut Schaefer <daniel.schaefer@hpe.com>
Reviewed-by: Leif Lindholm <leif.lindholm@linaro.org>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Leif Lindholm <leif.lindholm@linaro.org>
Cc: Gilbert Chen <gilbert.chen@hpe.com>
2020-04-29 02:52:08 +00:00
Fan, Zhiju 3a3a3af4a2 BaseTools:Add the spare space FV image size checker
REF:https://bugzilla.tianocore.org/show_bug.cgi?id=2654

If FV is placed in FD region, its FV image size is fixed.
When FV image size exceeds it, it will trig the build break.
To alert the developer to adjust FV image size earlier,
I request to add new checker for the the spare FV space.
When the spare FV space is less than the specified threshold,
build tool will report the error.

This checker is the optional.
It can be enabled by -D FV_SPARE_SPACE_THRESHOLD=10000.
Macro is the value of the spare space threshold size.
It can be decimal or hex format. If it is enabled,
BaseTools will check every FV with the fixed size.
If FV doesn't meet with the size requirement,
Build tool will report error message to say there is no
enough spare space.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2020-04-23 01:29:34 +00:00
Li, Aaron f6f66e0c30 BaseTools/Capsule: Add capsule dependency support
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=2412

Capsule generate tool support encode capsule dependencies through '-j'
command with a JSON file. To enable dependency feature, "Dependencies"
field for each payload in JSON file is required.
The value of "Dependencies" field is C style infix notation expression.
For example:
  "Dependencies":"72E2945A-00DA-448E-9AA7-075AD840F9D4 > 0x00000001"

The relation of Dependency Expression Opcode in UEFI2.8 chap 23.2 and
infix notation expression value is as follows:
+-----------------------------+--------------------------+
| OPCODE                      | INFIX EXPRESSION VALUE   |
+-----------------------------+--------------------------+
| 0x00 (PUSH_GUID)            | {GUID}                   |
| 0x01 (PUSH_VERSION)         | {UINT32}                 |
| 0x02 (DECLEAR_VERSION_NAME} | DECLEAR "{VERSION_NAME}" |
| 0x03 (AND)                  | &&                       |
| 0x04 (OR)                   | ||                       |
| 0x05 (NOT)                  | ~                        |
| 0x06 (TRUE)                 | TRUE                     |
| 0x07 (FALSE)                | FALSE                    |
| 0x08 (EQ)                   | ==                       |
| 0x09 (GT)                   | >                        |
| 0x0A (GTE)                  | >=                       |
| 0x0B (LT)                   | <                        |
| 0x0C (LTE)                  | <=                       |
+-----------------------------+--------------------------+

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>

Signed-off-by: Aaron Li <aaron.li@intel.com>

Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2020-01-15 03:16:46 +00:00
Fan, ZhijuX 4465cd124f BaseTools:Fix GenFds issue for BuildOption replace GenFdsOption
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=2455

BuildOption is used by TargetTxtClassObj.py
GenFdsOption is used by GenFds.py
When the GenFds tool is used alone (e.g. python3 -m GenFds.GenFds -h)
With the OptionParser function, the first detected function
prints the help message

import TargetTxtClassObj to GenFds,
The BuildOption will be executed and replace GenFdsOption

We removed all objects associated with this problem that
were created directly during the import process
(e.g. BuildOption, BuildTarget = MyOptionParser(),
 TargetTxt = TargetTxtDict())

The Patch is going to fix this issue

Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>

Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2020-01-13 02:08:46 +00:00
Steven Shi fc8b8deac2 BaseTools: Leverage compiler output to optimize binary cache
Redesign the binary cache and bases on the compiler to
output the dependency header files info for every module.
The binary cache will directly consume the dependency header
files info and doesn't parse the C source code by iteself.
Also redesign the dependency files list format for module
and try to share the common lib hash result as more as
possible in local process. Remove the unnecessary share data
access across multiprocessing.

Signed-off-by: Steven Shi <steven.shi@intel.com>

Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-12-10 02:07:12 +00:00
Fan, ZhijuX bf1ea933ec BaseTools:Add [packages] section in dsc file
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=2270

Currently a PCD (e.g. FeaturePCD) cannot be used in a conditional
statement in a DSC/FDF file without a module in the build referencing
the PCD package DEC file.

An example implementation that to support this is to allow a [Packages]
section in the DSC file to list additional package dependencies for PCD
references in the package DSC/FDF files.

this patch is going to  add the ability to have the [packages] section
defined in the DSC file

Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Acked-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-11-20 07:46:42 +00:00
Zhiguang Liu 5cef92771f BaseTools: Add map file parsing support for CLANG9
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>

Signed-off-by: Zhiguang Liu <zhiguang.liu@intel.com>
2019-11-08 08:29:36 +08:00
Leif Lindholm c1bcb2258c BaseTools: add missing newlines at end of files
Some scripts in Source/Python were missing newlines at end of files,
so add them.

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Leif Lindholm <leif.lindholm@linaro.org>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Philippe Mathieu-Daude <philmd@redhat.com>
2019-10-04 11:18:22 +01:00
Steven Shi b0c15fb128 BaseTools: Add more parameter checking for CopyFileOnChange()
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=2193

The current CopyFileOnChange() method in Misc.py does not
accept the input SrcFile parameter as a dir, but the method
does not check the SrcFile is dir or not. This patch is to
add more input parameter type checking and error message output
for method CopyFileOnChange.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Signed-off-by: Steven Shi <steven.shi@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-09-19 08:48:31 +08:00
Feng, Bob C 22e75231ae BaseTools:Remove the unnecessary operation of renaming a file.
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=2183

This patch is going to remove rename call to reduce unnecessary
io operation so that saving build time.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-09-17 10:19:05 +08:00
Fan, ZhijuX e32f7bc96d BaseTools:change some incorrect parameter defaults
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=1858

for Dict={},There are pitfalls in the way this default parameter is set
and Dict is not used in functions, other functions have these two cases,
I will change some incorrect parameter defaults

This patch is going to fix this issue

Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-09-17 10:18:51 +08:00
Feng, Bob C 7809492c10 BaseTools: Enable --genfds-multi-thread to default build
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1302

This patch enable --genfds-multi-thread to default build.
This patch keep --genfds-multi-thread build option for
compatibility and also add a new build option to disable
genfds-multi-thread as --no-genfds-multi-thread.

Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-09-11 09:43:59 +08:00
Shi, Steven 6dd9aa4019 BaseTools: Support long file path in windows for misc functions
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=2103

Current CopyFileOnChange() and SaveFileOnChange() in
BaseTools\Source\Python\Common\Misc.py don't use the dedicated
long file path API to handle the file path strings and cannot
support the long file path copy and save in windows. This patch
enhances them to support the long file path copy and save
correctly.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Signed-off-by: Steven Shi <steven.shi@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-08-23 11:26:23 +08:00
Shi, Steven 94459080c1 BaseTools: Improve the file saving and copying reliability
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=2079

The Basetool CopyFileOnChange() and SaveFileOnChange()
functions might raise the IOError occasionally when build
in Windows with multi-process and build cache enabled.
The CopyFileOnChange() and SaveFileOnChange() might be invoked
in multiple sub-processes simultaneously, and this patch adds
global locks to sync these functions invoking which can
harden their reliability.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Signed-off-by: Steven Shi <steven.shi@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-08-20 14:06:07 +08:00
Shi, Steven 0e7e7a264c BaseTools: Improve the cache hit in the edk2 build cache
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1927

Current cache hash algorithm does not parse and generate
the makefile to get the accurate dependency files for a
module. It instead use the platform and package meta files
to get the module depenedency in a quick but over approximate
way. These meta files are monolithic and involve many redundant
dependency for the module, which cause the module build
cache miss easily.
This patch introduces one more cache checkpoint and a new
hash algorithm besides the current quick one. The new hash
algorithm leverages the module makefile to achieve more
accurate and precise dependency info for a module. When
the build cache miss with the first quick hash, the
Basetool will caculate new one after makefile is generated
and then check again.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Signed-off-by: Steven Shi <steven.shi@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-08-20 14:06:05 +08:00
Bob Feng 08a54c9e0a BaseTools: Move buildoptions.py to Common folder
buildoptions is used by build.py and TargetTxtClassObj.py
Move buildoptions.py to Common folder so that
build tool and ECC tool work fine.

Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-08-14 13:54:49 +08:00
Feng, Bob C 2b4c07bc22 BaseTools: Add "-" in a re to parse line in .map file
https://bugzilla.tianocore.org/show_bug.cgi?id=2058

If there are multiple module instances with different
guid used in .dsc file, the temp path of the module will
contain 8-4-4-4-12 format guid string. So "-" need to be
added into the regular expression for parsing it correctly.

Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-08-14 10:24:36 +08:00
Feng, Bob C 1a624dd7cf BaseTools: Enable block queue log agent.
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1875

To support Ctrl+S and Ctrl+Q, we enable block queue
for log.

Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Acked-by: Liming Gao <liming.gao@intel.com>
2019-08-09 23:15:55 +08:00
Feng, Bob C 4acae2b38e BaseTools: Add the support for python 2
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1875

python3 change the module name of Queue to queue.
python3 add a new log handler of QueueHandler.

This patch is to make Multiple process AutoGen
feature work for python2

Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Acked-by: Liming Gao <liming.gao@intel.com>
2019-08-09 23:15:55 +08:00
Feng, Bob C c60fb00f6c BaseTools: Move BuildOption parser out of build.py
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1875

Build tool supports user to specify the conf folder.
To make the build options be evaluated at the beginning
of launching build, extract the buildoption function
from build.py to a new .py file.

Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Acked-by: Liming Gao <liming.gao@intel.com>
2019-08-09 23:15:54 +08:00
Feng, Bob C 636ed13a7f BaseTools: Add LogAgent to support multiple process Autogen
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1875

AutoGen processes race the logfile. To resolve this issue,
this patch create a LogAgent thread in main process to write
the log content to console or file, Other process will send
the log content to the LogAgent.

Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Acked-by: Liming Gao <liming.gao@intel.com>
2019-08-09 23:15:54 +08:00
Feng, Bob C e8449e1d8e BaseTools: Decouple AutoGen Objects
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1875

1. Separate the AutoGen.py into 3 small py files.
One is for AutoGen base class, one is for WorkspaceAutoGen class
and PlatformAutoGen class, and the one for ModuleAutoGen class.
2. Create a new class DataPipe to store the Platform scope settings.
Create a new class PlatformInfo to provide the same interface
as PlatformAutoGen. PlatformInfo class is initialized by
DataPipe instance.
Create a new class WorkspaceInfo to provide the same interface
as WorkspaceAutoGen. WorkspaceInfo class is initialized by
DataPipe instance.
3. Change ModuleAutoGen to depends on DataPipe, PlatformInfo and
WorkspaceInfo. Remove the dependency of ModuleAutoGen to PlatformAutoGen.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Steven Shi <steven.shi@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Acked-by: Liming Gao <liming.gao@intel.com>
2019-08-09 23:15:52 +08:00
Feng, Bob C db01c8e3d8 BaseTools: Singleton the object to handle build conf file
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1875

The build config files are target.txt, build rule, tooldef
During a build, the config is not changed, so the object to
handle them need to be singleton.

Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Acked-by: Liming Gao <liming.gao@intel.com>
2019-08-09 23:15:51 +08:00
Bob Feng 8ddec24dea BaseTools:Updata the output encoding of the Popen function
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=2015

Not all output works in utf-8, so change the encoding to
the default

This patch is going to fix that issue.

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-08-01 15:36:48 +08:00
Feng, Bob C 5f89bcc460 BaseTools: Create ".cache" folder when initialize Build object
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1986

Create "Conf/.cache" folder as early as possible
so that the later code do need to check
if it exits and then create it.

Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Reviewed-by: Michael D Kinney <michael.d.kinney@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-07-22 09:21:14 +08:00
Jiewen Yao a79841a024 BaseTools: Add HOST_APPLICATION module type.
It can be used to indicate a module can be build to run
as OS application and run in OS environment.

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Signed-off-by: Xiaoyu Lu <xiaoyux.lu@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-07-12 21:06:57 +08:00
Antoine Cœur 2d53d54a5b BaseTools: Fix various typos
Fix various typos in BaseTools.

Signed-off-by: Cœur <coeur@gmx.fr>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-07-08 08:59:29 +08:00
Jin, Eric 104a1aa19b BaseTools/Capsule: Supports multiple payloads and drivers in capsule
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1834

1)Add arguments "--embedded-driver" to support embedded driver
in command line.
2)Add arguments "--update-image-index" to identify ImageIndex
within the device in command line.
3)Add arguments "-j JSONFILE" to support multiple payloads and
embedded drivers with JSON file.

The update is in a backwards compatible manner, so all command
line options to support single payload are still supported. But
all the options associated with multiple payloads should be
provided in a JSON file.

Cc: Bob Feng <bob.c.feng@intel.com>
Signed-off-by: Eric Jin <eric.jin@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-06-28 11:39:06 +08:00
Shi, Steven 2378ea5515 BaseTools:Introduce CopyFileOnChange() function to copy cache files
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=1894

Basetool need a CopyFileOnChange() function to avoid cache
file writing race in multi-thread build. Some platforms
build fail with file IO writing race issue when the
build cache is enabled to store cache files in multi-threads.
This is because common same library cache files (e.g. some
libs in MdePkg) can be stored by many different driver modules'
build threads at same time. Current build cache need a function
to check whether the same cache file already exist, and only
copy source file if it is different from the destination file.

This patch introduces an atomic copy function to avoid duplicated
cache files copy.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Christian Rodriguez <christian.rodriguez@intel.com>
Signed-off-by: Steven Shi <steven.shi@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-06-17 17:00:46 +08:00
Rodriguez, Christian 48b0bf6476 BaseTools: Refactor hash tracking after checking for Sources section
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1804

After adding a check to see if [Sources] section lists all the header
type files of a module, track module and library hashes for --hash
feature. If above check is not in compilance for a library or module,
force hash invalidation on that library or module.

Signed-off-by: Christian Rodriguez <christian.rodriguez@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-06-10 19:48:51 +08:00
Fan, ZhijuX 04797875d1 BaseTools:Make BaseTools support new rules to generate RAW FFS FILE
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=1765

If RAW FFS File Rule has no section for its data.For RAW FFS File,
directly call GenFfs tool to generate FFS file.

Ffs Rule:
[Rule.Common.USER_DEFINED.MicroCode]
  FILE RAW = $(NAMED_GUID) {
        $(INF_OUTPUT)/$(MODULE_NAME).bin
  }
[Rule.Common.USER_DEFINED.LOGO]
  FILE RAW = $(NAMED_GUID) {
                       |.bmp
  }

As shown in the rule above,if SectionType and FileType not defined,
FFS files are generated directly, and no other type of file is
generated.

The patch is to make the BaseTools support these two rules

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-06-10 19:48:50 +08:00
Fan, ZhijuX c1387446ed BaseTools:Update mailing list address in BaseTools error messages
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=1815

The edk2 source tree contains four instances of the outdated mailing
list address "edk2-devel@lists.01.org".
I created a new variable, MSG_EDKII_MAIL_ADDR, to receive the
new email address and replaced the old one with this variable

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-05-22 09:15:55 +08:00
Rodriguez, Christian a7ef158b07 BaseTools: Library hashing fix and optimization for --hash feature
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1788

In V3: Must generate hashes before attempting to copy from cache for
hash verifcation
In V2: Build failure caused by passing incorrect boolean parameter to
SaveFileOnChange(). Fixed for patch instances.

Library hashing is now supported by the --hash feature. The --hash
feature implementation assumed that the hashing could be done in
place once per module, but that isn't true for libraries due to the
fact that they are built as dependencies. So on a clean build, we now
generate the .hash after the library dependencies are complete.
Added early escape as optimization, if hash already exists in memory.

Signed-off-by: Christian Rodriguez <christian.rodriguez@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-05-21 13:18:06 +08:00
Michael D Kinney 66b845ae06 BaseTools: Fix private includes for FILE_GUID override
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1648

When a module in a DSC file uses a FILE_GUID override in the
module scoped <Defines> section, a copy of the modified INF file
is placed in the Conf/.cache directory.  The check for private
includes uses the INF path to determine if the module is allowed
to use the private includes.  Since the INF path in this case is
not in any package, this check always fails, and no private
include paths are possible.

The fix is to keep both the OriginalPath and the new Path in
the PathClass object, and always use the OriginalPath to see if
the module INF is in the package with private includes.

Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-05-21 13:14:14 +08:00
Fan, ZhijuX afe8c4110c BaseTools: Correct the value assignment for StructurePcd
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1752

This patch is to fix the code bug in StructurePcd overall
value assignment logic. If a Pcd Array size is fixed but the
size of actual value in Dsc or Dec is bigger than the Pcd
array size, the tool will report error about such setting and
stop build.

The patch is tested minplatform, Ovmf, structure pcd regression test
These tests are build pass. The patch also tested the following cases.
Our cases focused on PcdArraySize. 1.flexiable PcdArraySize.
 2.Fixed PcdArraySize, Pcd overall value exceeds the size of PcdArray.
 3.Fixed PcdArraySize, Pcd overall value Not exceeds the size of PcdArray.

Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-05-10 17:06:44 +08:00
Yunhua Feng 0b836855fb BaseTools: Sometime write file not immediate to disk
BZ: https://bugzilla.tianocore.org/process_bug.cgi

On Windows OS, sometime the generated file is not immediate saved to disk.
When run nmake, prompt AutoGen.h not found, and stop build.
Below blog shows Write-Replace to fix it. This patch uses this way to write
temp file, then rename the temp file to the real file.
https://blog.gocept.com/2013/07/15/reliable-file-updates-with-python/

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Signed-off-by: Yunhua Feng <yunhuax.feng@intel.com>
Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com>
2019-04-22 10:05:52 +08:00
Rodriguez, Christian 0eccea3fbe BaseTools: Hash false success with back to back builds
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1692

Add error handling to the --hash feature so that hash files
are invalidated when a build error occurs.

Signed-off-by: Christian Rodriguez <christian.rodriguez@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-04-16 13:14:15 +08:00
Michael D Kinney 2e351cbe8e BaseTools: Replace BSD License with BSD+Patent License
https://bugzilla.tianocore.org/show_bug.cgi?id=1373

Replace BSD 2-Clause License with BSD+Patent License.  This change is
based on the following emails:

  https://lists.01.org/pipermail/edk2-devel/2019-February/036260.html
  https://lists.01.org/pipermail/edk2-devel/2018-October/030385.html

RFCs with detailed process for the license change:

  V3: https://lists.01.org/pipermail/edk2-devel/2019-March/038116.html
  V2: https://lists.01.org/pipermail/edk2-devel/2019-March/037669.html
  V1: https://lists.01.org/pipermail/edk2-devel/2019-March/037500.html

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-04-09 09:10:20 -07:00
Fan, ZhijuX 1c27ec4236 BaseTools:Coding problems caused by special characters
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=1670

During BaseTools compiling under Chinese or Japanese
language Windows, python exception occurring.
UnicodeDecodeError: 'ascii' codec can't decode byte 0xbd
in position 3528: ordinal not in range(128)

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-04-01 13:37:19 +08:00