Commit Graph

1208 Commits

Author SHA1 Message Date
Li, Aaron f6f66e0c30 BaseTools/Capsule: Add capsule dependency support
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=2412

Capsule generate tool support encode capsule dependencies through '-j'
command with a JSON file. To enable dependency feature, "Dependencies"
field for each payload in JSON file is required.
The value of "Dependencies" field is C style infix notation expression.
For example:
  "Dependencies":"72E2945A-00DA-448E-9AA7-075AD840F9D4 > 0x00000001"

The relation of Dependency Expression Opcode in UEFI2.8 chap 23.2 and
infix notation expression value is as follows:
+-----------------------------+--------------------------+
| OPCODE                      | INFIX EXPRESSION VALUE   |
+-----------------------------+--------------------------+
| 0x00 (PUSH_GUID)            | {GUID}                   |
| 0x01 (PUSH_VERSION)         | {UINT32}                 |
| 0x02 (DECLEAR_VERSION_NAME} | DECLEAR "{VERSION_NAME}" |
| 0x03 (AND)                  | &&                       |
| 0x04 (OR)                   | ||                       |
| 0x05 (NOT)                  | ~                        |
| 0x06 (TRUE)                 | TRUE                     |
| 0x07 (FALSE)                | FALSE                    |
| 0x08 (EQ)                   | ==                       |
| 0x09 (GT)                   | >                        |
| 0x0A (GTE)                  | >=                       |
| 0x0B (LT)                   | <                        |
| 0x0C (LTE)                  | <=                       |
+-----------------------------+--------------------------+

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>

Signed-off-by: Aaron Li <aaron.li@intel.com>

Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2020-01-15 03:16:46 +00:00
Fan, ZhijuX 4465cd124f BaseTools:Fix GenFds issue for BuildOption replace GenFdsOption
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=2455

BuildOption is used by TargetTxtClassObj.py
GenFdsOption is used by GenFds.py
When the GenFds tool is used alone (e.g. python3 -m GenFds.GenFds -h)
With the OptionParser function, the first detected function
prints the help message

import TargetTxtClassObj to GenFds,
The BuildOption will be executed and replace GenFdsOption

We removed all objects associated with this problem that
were created directly during the import process
(e.g. BuildOption, BuildTarget = MyOptionParser(),
 TargetTxt = TargetTxtDict())

The Patch is going to fix this issue

Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>

Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2020-01-13 02:08:46 +00:00
Fan, ZhijuX 072b9c2839 BaseTools:Change the case rules for ECC check pointer names
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=2087

In CryptHkdf.c  line 42

  EVP_PKEY_CTX *pHkdfCtx;

Variable pHkdfCtx begins with lower case 'p',
which should be acceptable because it it is a pointer.
(Refer to CCS_2_1_Draft, 4.3.3.3)

So ECC tool should be improved to handle issues like this.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>

Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
2020-01-13 02:08:46 +00:00
Bob Feng 01b6090b75 BaseTools: Resolve a issue of Incremental build
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=2311

In patch set 13c5e34a - 0c3e8e99, we implemented incremental build with
using compiler/pre-processor generate dependent header file function.

A issue is found for MSVC compiler, that the cl.exe /showIncludes
build option generate header file list to either stdout or stderr.
For .c file, the header file list is print out to stdout while for
.vfr, .aslc and .nasm file, the file list is print out to stderr.

The build tool use two threads to process the message from stdout and
stderr, but to generate correct *.deps file, build tool need to
combine the header file list from stderr and other messages from stdout
together with correct time sequence order.

So this patch is trying to combine the stdout and stderr together for
the process which is for calling make program.

The impact of this patch is that the output message of build with -q
will be changed. The compiler error message will not print out.
The build behavior of other log level setting will not be impacted.

Signed-off-by: Bob Feng <bob.c.feng@intel.com>

Cc: Liming Gao <liming.gao@intel.com>
Cc: Steven Shi <steven.shi@intel.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-12-18 07:24:45 +00:00
Bob Feng 78fb6b0e02 BaseTools: Fix build failure when multiple build targets given
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=2371

This patch is to fix a regression issue that build fails
if multiple build targets given.

Two changes cause this regression issue.
One is AutoGen object __hash__ function only
hash file path and arch, missing ToolChain and build target.

The other is changing the multiple-thread-genfds function as default
build behavior. To generate the genffs command to Makefile, there
is a global data set is used, GenFdsGlobalVariable, which cause build
tool use the data of first build-target build in
the second build-target build.

Signed-off-by: Bob Feng <bob.c.feng@intel.com>

Cc: Liming Gao <liming.gao@intel.com>
Cc: Steven Shi <steven.shi@intel.com>
Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-12-18 01:01:57 +00:00
Steven Shi a80032dc44 BaseTools: Remove redundant binary cache file
Redesign the binary cache and not need to save the
cache intermediate result and state in memory as a
ModuleBuildCacheIR class instance. So remove the
CacheIR.py which define the ModuleBuildCacheIR class.

Signed-off-by: Steven Shi <steven.shi@intel.com>

Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-12-10 02:07:12 +00:00
Steven Shi fc8b8deac2 BaseTools: Leverage compiler output to optimize binary cache
Redesign the binary cache and bases on the compiler to
output the dependency header files info for every module.
The binary cache will directly consume the dependency header
files info and doesn't parse the C source code by iteself.
Also redesign the dependency files list format for module
and try to share the common lib hash result as more as
possible in local process. Remove the unnecessary share data
access across multiprocessing.

Signed-off-by: Steven Shi <steven.shi@intel.com>

Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-12-10 02:07:12 +00:00
Steven Shi 3bfbc91507 BaseTools: enhance the CacheCopyFile method arg names
Enhance the CacheCopyFile method arg names to be more
clear and readable

Signed-off-by: Steven Shi <steven.shi@intel.com>

Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-12-10 02:07:12 +00:00
Steven Shi 91f6c533f8 BaseTools: store more complete output files in binary cache
Binary cache use the OutputFile method to return the module
built output files needed to store in cache, but current
OutputFile implementation doesn't return complete output files.
Enhance the OutputFile method to return more complete output files.

Signed-off-by: Steven Shi <steven.shi@intel.com>

Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-12-10 02:07:12 +00:00
Bob Feng 0c3e8e9947 BaseTools: Enhance Basetool for incremental build
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=2311

Include dependency file in Makefile to enhance
incremental build

Signed-off-by: Bob Feng <bob.c.feng@intel.com>

Cc: Liming Gao <liming.gao@intel.com>
Cc: Steven Shi <steven.shi@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-12-10 01:31:55 +00:00
Bob Feng e6edbe315f BaseTools: Generate dependent files for ASL and ASM files
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=2311

Implement the function in Trim tool to get the included
file list for ASL and ASM file.

Signed-off-by: Bob Feng <bob.c.feng@intel.com>

Cc: Liming Gao <liming.gao@intel.com>
Cc: Steven Shi <steven.shi@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-12-10 01:31:55 +00:00
Philippe Mathieu-Daude 490a62beb7 BaseTools: Avoid "is" with a literal Python 3.8 warnings
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=2304

The following statement produces a SyntaxWarning with Python 3.8:

  if str(FdRegion.RegionType) is 'FILE' and self.Platform.VpdToolGuid in \
    str(FdRegion.RegionDataList):
  BaseTools/Source/Python/AutoGen/WorkspaceAutoGen.py:168: SyntaxWarning: \
    "is" with a literal. Did you mean "=="?

Change the 'is' operator by the conventional '==' comparator.

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
Signed-off-by: Philippe Mathieu-Daude <philmd@redhat.com>
2019-12-06 03:07:37 +00:00
Fan, ZhijuX c8ff8e05af BaseTools:Enhance the way to handling included dsc file
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=2400

In Dsc Parser, included dsc file is parsed always no matter
if its condition is False

  gUefiOvmfPkgTokenSpaceGuid.test1|FALSE
!if gUefiOvmfPkgTokenSpaceGuid.test1 == FALSE
  !include OvmfPkg/test1.dsc
!else
  !include OvmfPkg/test2.dsc
!endif

The patch avoids processing redundant dsc files and improves
the way Tool handles them.

In the above case, since the conditional result is FALSE,
"test2.dsc" is not parsed.

Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>

Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-12-04 07:56:55 +00:00
Fan, ZhijuX e0f8261ad0 BaseTools:fix regression issue for platform .map file
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=2363

This patch is to fix a build tool regression issue which was introduced
by commit b8ac0b7f28.This issue caused map file lost the line of IMAGE=***.
For example,in Ovmf.map, there is no line of (IMAGE=<path to efi> ) under
each of modules item.

The path to the efi file generated by each module is written on this line
The purpose of this line is add the debug image full path.
there is no information about the module in the map file other than FVName,
it allows us to quickly know which module this part corresponds to.

In commit b8ac0b7f28,add a line ("self.BuildModules = []") in function,
but it's used to calculate the variable ModuleList in the following code.

Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>

Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-11-25 06:01:10 +00:00
Fan, ZhijuX bf1ea933ec BaseTools:Add [packages] section in dsc file
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=2270

Currently a PCD (e.g. FeaturePCD) cannot be used in a conditional
statement in a DSC/FDF file without a module in the build referencing
the PCD package DEC file.

An example implementation that to support this is to allow a [Packages]
section in the DSC file to list additional package dependencies for PCD
references in the package DSC/FDF files.

this patch is going to  add the ability to have the [packages] section
defined in the DSC file

Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Acked-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-11-20 07:46:42 +00:00
Ni, Ray 49fb9f7e06 BaseTools: Fix build failure when using python38
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=2304

Signed-off-by: Ray Ni <ray.ni@intel.com>
Cc: Bob C Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-11-13 03:47:48 +00:00
Feng, Bob C d847ac1f27 BaseTools: Enable MACRO for DSC Components section tag
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=2308

This patch is to enable MACRO for Components section architecture
modifier.

Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-11-13 03:22:39 +00:00
Sean Brogan f7978bb258 BaseTools: Add YAML files with path env and tool extdeps
https://bugzilla.tianocore.org/show_bug.cgi?id=2315

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-11-11 13:01:53 -08:00
Zhiguang Liu 5cef92771f BaseTools: Add map file parsing support for CLANG9
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>

Signed-off-by: Zhiguang Liu <zhiguang.liu@intel.com>
2019-11-08 08:29:36 +08:00
Vitaly Cheptsov via Groups.Io 5c7006c9de BaseTools: Do not call sys.setdefaultencoding with python 3
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=2301

This interface was a originally a no-op in python 3, and now
is fully removed causing a build warning on macOS (Darwin).

Signed-off-by: Vitaly Cheptsov <vit9696@protonmail.com>>
Acked-by: Bob Feng <bob.c.feng@intel.com>
2019-10-23 14:25:23 +08:00
Lin, Derek (HPS SW) b1c6e9f55e BaseTools: Fix an incremental build issue caused by macro in #include
When c/h file use macro after #include, for example,
In this case, GenMake is not able to create a healthy dependency for the c
file. GenMake used to add $(FORCE_REBUILD) dependency in the c file, this
guarantee the c file is always compiled in incremental build. But, this
function is broken since 05217d210e which
enable /MP for MSVC compiler, in order to compile multiple c files in one
command multi-processing. The fix here is adding '$(FORCE_REBUILD)' back to
retain the original function.

Line number 1728 and 978 are the code pieces which handle this logic.

Signed-off-by: Derek Lin <derek.lin2@hpe.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-10-18 08:30:38 +08:00
Feng, Bob C a1f94045ff BaseTools: Fix a bug of genffs command generation
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=2221

The command used by multiple thread genffs feature in makefile
for testing if file exist is generated based on the toolchain family.
It should be based on the OS type.

Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-10-14 09:00:10 +08:00
Leif Lindholm 0deca401f0 BaseTools: strip trailing whitespace
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Leif Lindholm <leif.lindholm@linaro.org>
Reviewed-by: Michael D Kinney <michael.d.kinney@intel.com>
2019-10-04 11:18:22 +01:00
Leif Lindholm c1bcb2258c BaseTools: add missing newlines at end of files
Some scripts in Source/Python were missing newlines at end of files,
so add them.

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Leif Lindholm <leif.lindholm@linaro.org>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Philippe Mathieu-Daude <philmd@redhat.com>
2019-10-04 11:18:22 +01:00
Kubacki, Michael A 61af5f2494 BaseTools: Fix GenMake multi-workspace failure
REF:https://bugzilla.tianocore.org/show_bug.cgi?id=2232

Commit 0075ab2cec introduced an issue that causes an exception
when multiple workspace packages paths are specified. For example,
if edk2-platforms is used, the root directory will contain an edk
and edk2-platforms directory representing the respective
repositories.

In GenMake, the path to the package DEC file for a module is
discovered by getting the relative path of the INF to the
workspace root directory. Each directory in the relative path
is incrementally joined to the WORKSPACE directory. The file
list in the joined path is searched for a DEC file.

As an example, if the build command is used on a package outside
the edk2 repository, the INF file path is relative to the
edk2-platforms directory not edk2. This causes directory paths
to be built that do not exist. Commit 0075ab2cec replaced the
os.path.exists() call with a try except block that always fails
when os.listdir() is invoked to enumerate the list of files in
the built directory path on packages outside edk2.

This commit restores the original conditional statement which
avoids calling os.listdir() with an invalid directory path.

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Signed-off-by: Michael Kubacki <michael.a.kubacki@intel.com>
Reviewed-by: Michael D Kinney <michael.d.kinney@intel.com>
Reviewed-by:  Bob Feng <bob.c.feng@intel.com>
2019-10-04 11:13:37 +08:00
Feng, Bob C 1f5e4d9128 BaseTools: Fix the lib order in static_library_files.lst
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=2216

This patch is going to fix the lib order in static_library_files.lst.
This issue is introduced by commit 673d09a2dd.
Before 673d09a2dd, build tool apply build rule for the module's library
firstly and then apply build rule for module itself. Now, build tool
apply build rule for module self and then for its library. That behavior
impact the lib order in static_library_files.lst.

This patch is to call module's LibraryAutoGenList function
explicitly, where the applying build rule action for
library is triggered.

Cc: Liming Gao <liming.gao@intel.com>
Cc: dann frazier <dann.frazier@canonical.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Tested-by: dann frazier <dann.frazier@canonical.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-09-29 11:59:31 +08:00
Feng, Bob C 4747b92cab BaseTools: init DependencyHeaderFileSet for each MakeFile class
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=2203

This patch is going to fix a regression issue that
build breaks when the custom makefile exist.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Steven Shi <steven.shi@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-09-24 10:29:09 +08:00
Liming Gao fcdedafd97 Revert "BaseTools: Improve GetDependencyList function"
This reverts commit bc9e4194cf.
This change causes the dependent header files are missing in Makefile.
It makes the incremental build not work. So, revert this change.

Cc: Bob Feng<bob.c.feng@Intel.com>
Signed-off-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Bob Feng<bob.c.feng@Intel.com>
2019-09-21 00:25:48 +08:00
Fan, ZhijuX 85ccbee2ab BaseTools:Fix the issue that build report failed
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=2201

An error occurs using special VpdPcd that is not used in the Inf file

In dsc:
[PcdsDynamicExVpd.common.DEFAULT]
gBoardModuleTokenSpaceGuid.test1|*|{CODE({
  {0x0}  // terminator
})}

In dec:
[PcdsDynamicEx]
# Vpd GPIO table
  gBoardModuleTokenSpaceGuid.test1|{0}|GPIO_INIT_CONFIG[]|0x50000018 {
  <HeaderFiles>
    Library/GpioLib.h
  <Packages>
    MdePkg/MdePkg.dec
}
ValueError: invalid literal for int() with base 0: '*'

This Patch is going to fix issue

Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-09-19 17:11:57 +08:00
Steven Shi b0c15fb128 BaseTools: Add more parameter checking for CopyFileOnChange()
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=2193

The current CopyFileOnChange() method in Misc.py does not
accept the input SrcFile parameter as a dir, but the method
does not check the SrcFile is dir or not. This patch is to
add more input parameter type checking and error message output
for method CopyFileOnChange.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Signed-off-by: Steven Shi <steven.shi@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-09-19 08:48:31 +08:00
Feng, Bob C 22e75231ae BaseTools:Remove the unnecessary operation of renaming a file.
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=2183

This patch is going to remove rename call to reduce unnecessary
io operation so that saving build time.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-09-17 10:19:05 +08:00
Feng, Bob C 0075ab2cec BaseTools: remove unnecessary calls of os.exist
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=2101

This patch is going to remove unnecessary calls
of os.exist()

Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-09-17 10:19:05 +08:00
Feng, Bob C bc9e4194cf BaseTools: Improve GetDependencyList function
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=2102

GetDependencyList get the header file via
re.findall in the whole header file.

This patch is to pre-process the header file and
to feed the shorter string to re.findall.

This patch is to improve GetDependencyList() efficiency

Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-09-17 10:19:05 +08:00
Fan, ZhijuX e32f7bc96d BaseTools:change some incorrect parameter defaults
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=1858

for Dict={},There are pitfalls in the way this default parameter is set
and Dict is not used in functions, other functions have these two cases,
I will change some incorrect parameter defaults

This patch is going to fix this issue

Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-09-17 10:18:51 +08:00
Fan, ZhijuX b67735a7e8 BaseTools:Replace PlatformInfo with PlatformAutoGen for Moudle
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=2188

build -p MdeModulePkg\MdeModulePkg.dsc -a IA32 -m
MdeModulePkg\Universal\PCD\Pei\Pcd.inf

Error:
AttributeError: 'PlatformInfo' object has no attribute
'DynamicPcdList'

The DSC data object used to build a separate module today
is PlatformInfo rather than PlatformAutoGen
'PlatformAutoGen' object has attribute 'DynamicPcdList'

This patch is going to fixed this issue

Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-09-17 10:18:51 +08:00
Feng, Bob C 5a9db85880 BaseTools: Fixed a bug of IgnoreAutoGen
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=2080
After checking that if the build can't
ignore Autogen due to there is no compelet autogen files,
the build tool need to do a completely Autogen.

This patch is to fix a bug that if AutoGen
can't be skiped, the SkipAutoGen flag
need to set to False

Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-09-12 14:08:15 +08:00
Feng, Bob C 7809492c10 BaseTools: Enable --genfds-multi-thread to default build
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1302

This patch enable --genfds-multi-thread to default build.
This patch keep --genfds-multi-thread build option for
compatibility and also add a new build option to disable
genfds-multi-thread as --no-genfds-multi-thread.

Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-09-11 09:43:59 +08:00
Feng, Bob C f23da86415 BaseTools: Fixed the build fail on Linux with --genfds-multi-thread
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1302

If GenSec input file not exist, the related command will fail and make
will stop on Linux. GenSec input file is allow to be non-existent.

This patch is to let "make" continue if gensec input file not exist.

Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-09-11 09:43:59 +08:00
Feng, Bob C e3c8311f8a BaseTools: Fixed the bug of multi-thread genffs for override inf
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1302
If there is a inf override, and multi-thread genffs is enabled, build
will fail.

This patch is going to fix this bug.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Steven Shi <steven.shi@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-09-11 09:43:58 +08:00
Feng, Bob C 000ab98574 BaseTools: Sort Pcd settings to make PcdTokenNumber be fixed
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=2147

This patch is to sort the Pcd settings so that PcdTokenNumber
will not change if the platform's Pcd settings are the same.

Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Steven Shi <steven.shi@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-09-10 08:39:19 +08:00
Feng, Bob C 373298ca0d BaseTools: Fixed issue for IgnoreAutoGen
https://bugzilla.tianocore.org/show_bug.cgi?id=2080

This patch is to improve build -u option to re-use
GlobalVar_<platformguid>_<arch>.bin file which is
introduced by multiple-process-autogen feature.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Steven Shi <steven.shi@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Acked-by: Liming Gao <liming.gao@intel.com>
2019-09-10 08:39:18 +08:00
Feng, Bob C 53e2eaba7a BaseTools: Fixed build clean regression issue
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=2166

file_lock and cache_lock are used as global data,
so move file_lock and cache_lock initialization in Build
object __init__ function.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Steven Shi <steven.shi@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-09-10 08:39:17 +08:00
Bob Feng cbe7543ebb BaseTools: Fix a bug for Hii Pcd override
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=2157

Hii Pcd links to a efi variable. The Variable
default value is evaluated by related Hii Pcds
setting. If multiple Hii Pcds links to one variable,
and the offset overlap, the later Hii Pcds setting
should be effective. There is a tool bug that is if
the Pcds are in different dsc file which are included
into the platform dsc file, build tool does not get
the Pcds relative position correctly. That means
build tool does not know which Pcd is the later one. As
the result, the variable default value will be incorrect.

This patch is to fix this bug.

Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-09-10 08:39:16 +08:00
Fan, ZhijuX d6f5a5050b BaseTools:ECC need to handle lower case 'static'
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=1778

We are going to add keyword (lower case) 'static' for functions and global
variables which are not referenced outside their current C file.

However, the ECC tool only recognizes upper case 'STATIC' at this moment.
This will lead to issue reports for new codes that follow the above coding
style.

This patch is going to handle lower case 'static'

Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-09-09 11:14:42 +08:00
Fan, ZhijuX 03ad3d889c BaseTools:Ecc handle another copyright format
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=2057

Ecc can not handle the copyright format like

(C) Copyright 2015-2016 Hewlett Packard Enterprise Development LP<BR>

This will cause Ecc to report wrong information.

This patch is going to handle this format

Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-09-09 11:13:52 +08:00
Shi, Steven 40db176d9d BaseTools: Support more file types in build cache
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1958

Current build cache does not store and restore all
types file of Hii/vfr, version and dpx. This patch
adds more file types to support them in build cache.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Signed-off-by: Steven Shi <steven.shi@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-09-02 12:17:56 +08:00
Feng, Bob C 37eef91017 BaseTools: Fixed regression issue for building !x86 builds
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=2134

This patch is to fix a build tool regression issue which
was introduced by commit e8449e1d8e.

In commit e8449e1d8e, build tool check the pcd before
filter out the irrelevant library instance.

The logic of evaluating the priority of the library class
resolutions was not changed.

Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Tested-by: Michael D Kinney <michael.d.kinney@intel.com>
Acked-by: Michael D Kinney <michael.d.kinney@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Acked-by: Leif Lindholm <leif.lindholm@linaro.org>
Tested-by: Leif Lindholm <leif.lindholm@linaro.org>
2019-08-29 11:10:29 +08:00
Feng, Bob C df43ea6c08 BaseTools: fix an incremental build issue.
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=2137

This patch is going to fix a regression issue of incremental
build which was introduced by the commit 94459080c.
The changing on INF file will not trigger module rebuild.

Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Steven Shi <steven.shi@intel.com>
Tested-by: Michael D Kinney <michael.d.kinney@intel.com>
Acked-by: Michael D Kinney <michael.d.kinney@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-08-29 11:07:48 +08:00
Feng, Bob C abc0155b03 BaseTools: Incorrect error message for library instance not found
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=2099
This is a regression issue introduced by commit e8449e.

This patch is to fix this issue.

Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-08-23 11:26:24 +08:00
Feng, Bob C 5f7bb39117 BaseTools: Fix incremental build genmake issue
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=2090

This is a regression issue introduced by commit e8449e.
This patch is going to fix this issue.

Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-08-23 11:26:24 +08:00
Shi, Steven 6dd9aa4019 BaseTools: Support long file path in windows for misc functions
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=2103

Current CopyFileOnChange() and SaveFileOnChange() in
BaseTools\Source\Python\Common\Misc.py don't use the dedicated
long file path API to handle the file path strings and cannot
support the long file path copy and save in windows. This patch
enhances them to support the long file path copy and save
correctly.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Signed-off-by: Steven Shi <steven.shi@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-08-23 11:26:23 +08:00
Fan, ZhijuX 0970a80583 BaseTools: Update incorrect variable name 'DataPile'
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=2093

The PlatformAutoGen object has a DataPipe property but no DataPile property
So change the variable name 'DataPile' to 'DataPipe' in BuildReport.py

This patch is going to fix that issue.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-08-20 14:15:47 +08:00
Shi, Steven 94459080c1 BaseTools: Improve the file saving and copying reliability
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=2079

The Basetool CopyFileOnChange() and SaveFileOnChange()
functions might raise the IOError occasionally when build
in Windows with multi-process and build cache enabled.
The CopyFileOnChange() and SaveFileOnChange() might be invoked
in multiple sub-processes simultaneously, and this patch adds
global locks to sync these functions invoking which can
harden their reliability.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Signed-off-by: Steven Shi <steven.shi@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-08-20 14:06:07 +08:00
Shi, Steven d01a998612 BaseTools: Add GenFds multi-thread support in build cache
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1923

Fix the issue that the GenFds multi-thread will build fail
if enable the build cache together.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Signed-off-by: Steven Shi <steven.shi@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-08-20 14:06:07 +08:00
Shi, Steven c340c5bd1b BaseTools: Change the [Arch][Name] module key in Build cache
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1951

Current build cache use the module's [self.Arch][self.Name]
info as the ModuleAutoGen object key in hash list and dictionary.
The [self.Arch][self.Name] is not safe as the module key because
there could be two modules with same module name and arch name in
one platform. E.g. A platform can override a module or library
instance in another different path, the overriding module can has
the same module name and arch name as the original one.
Directly use the ModuleAutoGen obj self as the key, because
the obj __hash__ and __repr__ attributes already contain the
full path and arch name.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Signed-off-by: Steven Shi <steven.shi@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-08-20 14:06:06 +08:00
Shi, Steven 56c786b0a9 BaseTools: Print first cache missing file for build cachle
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1925

When a module build cache miss, add support to print the first
cache missing file path and name.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Signed-off-by: Steven Shi <steven.shi@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-08-20 14:06:06 +08:00
Shi, Steven 0e7e7a264c BaseTools: Improve the cache hit in the edk2 build cache
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1927

Current cache hash algorithm does not parse and generate
the makefile to get the accurate dependency files for a
module. It instead use the platform and package meta files
to get the module depenedency in a quick but over approximate
way. These meta files are monolithic and involve many redundant
dependency for the module, which cause the module build
cache miss easily.
This patch introduces one more cache checkpoint and a new
hash algorithm besides the current quick one. The new hash
algorithm leverages the module makefile to achieve more
accurate and precise dependency info for a module. When
the build cache miss with the first quick hash, the
Basetool will caculate new one after makefile is generated
and then check again.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Signed-off-by: Steven Shi <steven.shi@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-08-20 14:06:05 +08:00
Feng, Bob C 76e12fa334 BaseTools: Fixed issue of incorrect Module Unique Name
https://bugzilla.tianocore.org/show_bug.cgi?id=2088
If there are more than one override instance for a same module,
the Module Unique Name is generated incorrectly.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Steven Shi <steven.shi@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-08-19 09:06:14 +08:00
Jin, Eric 48d8d4d80b BaseTools/Capsule: Tool to generate Windows Firmware Update Driver
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1837

The tool is designed to generate Windows Firmware Update Drivers,
the input is one drivername.cap with related parameters, the output
Windows Driver package are composed by drivername.cap, drivername.inf
and drivername.cat to update the single payload in device.

usage:
GenerateWindowsDriver [-h] [--output-folder OUTPUTFOLDER]
                      [--product-fmp-guid PRODUCTFMPGUID]
                      [--capsuleversion-dotstring CAPSULEVERSION_DOTSTRING]
                      [--capsuleversion-hexstring CAPSULEVERSION_HEXSTRING]
                      [--product-fw-provider PRODUCTFWPROVIDER]
                      [--product-fw-mfg-name PRODUCTFWMFGNAME]
                      [--product-fw-desc PRODUCTFWDESC]
                      [--capsule-file-name CAPSULEFILENAME]
                      [--pfx-file PFXFILE] [--arch ARCH]
                      [--operating-system-string OPERATINGSYSTEMSTRING]

Cc: Sean Brogan <sean.brogan@microsoft.com>
Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Kinney Michael D <michael.d.kinney@intel.com>
Signed-off-by: Eric Jin <eric.jin@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-08-15 20:13:55 +08:00
Sean Brogan e2aacac580 BaseTools/Capsule: Add WindowsCapsuleSupportHelper
Add initial version WindowsCapsuleSupportHelper.py
original source comes from
https://github.com/microsoft/mu_basecore/blob/dev/201905/BaseTools/Plugin/WindowsCapsuleSupportHelper/WindowsCapsuleSupportHelper.py
commit ebd9aad90dee3a3094b575928670cf81b8c685b5
Convert EOL to CRLF

Cc: Sean Brogan <sean.brogan@microsoft.com>
Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Signed-off-by: Eric Jin <eric.jin@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-08-15 20:13:05 +08:00
Bob Feng 08a54c9e0a BaseTools: Move buildoptions.py to Common folder
buildoptions is used by build.py and TargetTxtClassObj.py
Move buildoptions.py to Common folder so that
build tool and ECC tool work fine.

Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-08-14 13:54:49 +08:00
Rodriguez, Christian 82407bd129 BaseTools: Fix checking for Sources section in INF file
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1804

The check to see if [Sources] section lists all the header type
files of a module is missing the exclusion of source files that
fall under the scope of Package includes. This change adds the
exclusions.

Signed-off-by: Christian Rodriguez <christian.rodriguez@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Tested-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-08-14 10:59:08 +08:00
Feng, Bob C 2b4c07bc22 BaseTools: Add "-" in a re to parse line in .map file
https://bugzilla.tianocore.org/show_bug.cgi?id=2058

If there are multiple module instances with different
guid used in .dsc file, the temp path of the module will
contain 8-4-4-4-12 format guid string. So "-" need to be
added into the regular expression for parsing it correctly.

Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-08-14 10:24:36 +08:00
Feng, Bob C a000d5d1ca BaseTools: Fix incremental build bug
https://bugzilla.tianocore.org/show_bug.cgi?id=2078

This is a build tool regression issue that
is introduced by multiple-process-autogen.

The workspace timestamp value is not pass
to module autogen object correctly, so build tool
does not detect the change of .dsc and ignore the autogen.

This patch is to fix this issue.

Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Tested-by: Michael D Kinney <michael.d.kinney@intel.com>
2019-08-14 10:23:25 +08:00
Feng, Bob C a0792697bc BaseTools: Fixed an issue that build raise exception
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=2047

Build raise exception if there is libraryclass
missing in dsc file and --pcd build option is used.

Build should report the missing libraryclass but not
a call stack.

This patch is going to fix it.

Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-08-12 10:40:24 +08:00
Feng, Bob C 1a624dd7cf BaseTools: Enable block queue log agent.
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1875

To support Ctrl+S and Ctrl+Q, we enable block queue
for log.

Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Acked-by: Liming Gao <liming.gao@intel.com>
2019-08-09 23:15:55 +08:00
Feng, Bob C 4acae2b38e BaseTools: Add the support for python 2
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1875

python3 change the module name of Queue to queue.
python3 add a new log handler of QueueHandler.

This patch is to make Multiple process AutoGen
feature work for python2

Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Acked-by: Liming Gao <liming.gao@intel.com>
2019-08-09 23:15:55 +08:00
Feng, Bob C c60fb00f6c BaseTools: Move BuildOption parser out of build.py
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1875

Build tool supports user to specify the conf folder.
To make the build options be evaluated at the beginning
of launching build, extract the buildoption function
from build.py to a new .py file.

Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Acked-by: Liming Gao <liming.gao@intel.com>
2019-08-09 23:15:54 +08:00
Feng, Bob C 636ed13a7f BaseTools: Add LogAgent to support multiple process Autogen
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1875

AutoGen processes race the logfile. To resolve this issue,
this patch create a LogAgent thread in main process to write
the log content to console or file, Other process will send
the log content to the LogAgent.

Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Acked-by: Liming Gao <liming.gao@intel.com>
2019-08-09 23:15:54 +08:00
Feng, Bob C 3285fbda88 BaseTools: Add shared data for processes
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1875

Add shared data for autogen processes.

Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Acked-by: Liming Gao <liming.gao@intel.com>
2019-08-09 23:15:53 +08:00
Feng, Bob C 673d09a2dd BaseTools: Enable Multiple Process AutoGen
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1875

Assign the Module AutoGen tasks into multiple
sub process.

Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Acked-by: Liming Gao <liming.gao@intel.com>
2019-08-09 23:15:53 +08:00
Feng, Bob C e8449e1d8e BaseTools: Decouple AutoGen Objects
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1875

1. Separate the AutoGen.py into 3 small py files.
One is for AutoGen base class, one is for WorkspaceAutoGen class
and PlatformAutoGen class, and the one for ModuleAutoGen class.
2. Create a new class DataPipe to store the Platform scope settings.
Create a new class PlatformInfo to provide the same interface
as PlatformAutoGen. PlatformInfo class is initialized by
DataPipe instance.
Create a new class WorkspaceInfo to provide the same interface
as WorkspaceAutoGen. WorkspaceInfo class is initialized by
DataPipe instance.
3. Change ModuleAutoGen to depends on DataPipe, PlatformInfo and
WorkspaceInfo. Remove the dependency of ModuleAutoGen to PlatformAutoGen.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Steven Shi <steven.shi@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Acked-by: Liming Gao <liming.gao@intel.com>
2019-08-09 23:15:52 +08:00
Feng, Bob C 197ca7febf BaseTools: Add functions to get platform scope build options
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1875

These functions are used for get platform scope
build options. They will be used in later patches.

Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Acked-by: Liming Gao <liming.gao@intel.com>
2019-08-09 23:15:52 +08:00
Feng, Bob C 2927a6fd99 BaseTools: Split WorkspaceAutoGen._InitWorker into multiple functions
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1875

The WorkspaceAutoGen.__InitWorker function is too long, it's hard
to read and understand.
This patch is to separate the __InitWorker into multiple small ones.

Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Acked-by: Liming Gao <liming.gao@intel.com>
2019-08-09 23:15:51 +08:00
Feng, Bob C db01c8e3d8 BaseTools: Singleton the object to handle build conf file
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1875

The build config files are target.txt, build rule, tooldef
During a build, the config is not changed, so the object to
handle them need to be singleton.

Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Acked-by: Liming Gao <liming.gao@intel.com>
2019-08-09 23:15:51 +08:00
Feng, Bob C 83d6207f99 BaseTools: Fixed a typo in Trim.py
This is a regression issue introduced
by commit 307e1650be

This patch is to fix this issue.

Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Philippe Mathieu-Daude <philmd@redhat.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-08-02 00:18:00 +08:00
Matthew Carlson 101dc9e257 BaseTools:Added arch output to build report
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=2016

Added arch output to build report so it's easy to tell
which arch is being generated. Useful when multiple versions
of a single module is being emitted for multiple archs.

This patch is going to Added arch output

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-08-01 15:36:49 +08:00
Bob Feng 8ddec24dea BaseTools:Updata the output encoding of the Popen function
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=2015

Not all output works in utf-8, so change the encoding to
the default

This patch is going to fix that issue.

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-08-01 15:36:48 +08:00
Max Knutsen 307e1650be BaseTools:Add extra debugging message
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=2014

Add extra debugging to improve error identification.
Error while processing file if the file is read incorrectly

This patch is going to fix that issue.

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Philippe Mathieu-Daude <philmd@redhat.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-08-01 15:36:48 +08:00
Sean Brogan 76912197fa BaseTools:replace the chinese quotation mark with unicode "
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=2013

chinese quotation mark are used in the file
This patch is going to fix that issue.

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Philippe Mathieu-Daude <philmd@redhat.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-08-01 15:36:47 +08:00
Feng, Bob C 959791dd6c BaseTools: Sort file list in Makefile
This patch is going to sort the file list in generated
Makefile. This change make the autogen makefile easy to
compare.

Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Philippe Mathieu-Daude <philmd@redhat.com>
2019-07-25 09:00:59 +08:00
Feng, Bob C bb824f685d BaseTools: Fixed the mis-using strip() function issue.
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=2003

lstrip(parameter) do the match based on the char
in parameter but not only the whole parameter string.

In GenMake line 1082,
CmdSign.lstrip('/Fo') will strip the '/' or
'F' or 'o' on the left of CmdSign. This is not expected.

This patch is going to fix such issue.

Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-07-22 12:06:03 +08:00
Feng, Bob C 5f89bcc460 BaseTools: Create ".cache" folder when initialize Build object
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1986

Create "Conf/.cache" folder as early as possible
so that the later code do need to check
if it exits and then create it.

Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Reviewed-by: Michael D Kinney <michael.d.kinney@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-07-22 09:21:14 +08:00
Cole Robinson eebc135ffb BaseTools: Fix python3.8 SyntaxWarning
Building with python3.8 shows a warning like:

SyntaxWarning: invalid escape sequence \(
  GuidName = re.compile("\(GUID=[-a-fA-F0-9]+")

It seems harmless, but it's easy enough to fix: mark the string as
raw with the 'r' prefix like is used elsewhere in the file

Signed-off-by: Cole Robinson <crobinso@redhat.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-07-15 09:09:25 +08:00
Jiewen Yao a79841a024 BaseTools: Add HOST_APPLICATION module type.
It can be used to indicate a module can be build to run
as OS application and run in OS environment.

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Signed-off-by: Xiaoyu Lu <xiaoyux.lu@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-07-12 21:06:57 +08:00
Bob Feng 2d100d1d73 BaseTools: Fixed the issue when ToolDefinitionFile is not generated
ToolDefinitionFile is generated by PlatformAutoGen.ToolDefinition()
Code assume ToolDefinition is always called before using
ToolDefinitionFile, but in some cases, it's not true.

This patch is to fix this issue.

Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-07-12 17:11:58 +08:00
Feng, Bob C 688ec2d006 BaseTools: Detect the change of env variable used in tooldef.txt
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=1921

Add TOOLS_DEF.ARCH file to all workspace meta files list.
TOOLS_DEF.ARCH include the evaluated Tool definition information
which is filtered by current tool_chain.

With this change, when the environment variable which
is used in ToolDef.txt is changed, build tool will
rebuild the platform.

Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Steven Shi <steven.shi@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-07-10 09:29:43 +08:00
Feng, Bob C 1d06b46cf1 BaseTools: Fixed the issue of the CFlag for compile PcdValueInit.c
This issue happens when using Structured PCD.
Build tool use set to store the cflags for compile PcdValueInit.c,
that is the order of cflags is disorder.

This patch make -U, /U flags appear before -D, /D

Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
2019-07-10 09:26:18 +08:00
Antoine Cœur 2d53d54a5b BaseTools: Fix various typos
Fix various typos in BaseTools.

Signed-off-by: Cœur <coeur@gmx.fr>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-07-08 08:59:29 +08:00
Jin, Eric 104a1aa19b BaseTools/Capsule: Supports multiple payloads and drivers in capsule
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1834

1)Add arguments "--embedded-driver" to support embedded driver
in command line.
2)Add arguments "--update-image-index" to identify ImageIndex
within the device in command line.
3)Add arguments "-j JSONFILE" to support multiple payloads and
embedded drivers with JSON file.

The update is in a backwards compatible manner, so all command
line options to support single payload are still supported. But
all the options associated with multiple payloads should be
provided in a JSON file.

Cc: Bob Feng <bob.c.feng@intel.com>
Signed-off-by: Eric Jin <eric.jin@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-06-28 11:39:06 +08:00
Bob Feng b8ac0b7f28 BaseTools: Move Build Cache related function out of CreateAsBuiltInf
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1932
There are two functions in current CreateAsBuiltInf, Copy Binary files
to build cache folder and create asbuild inf file.

This patch is to separate UpdateBuildCache and CreateAsBuiltInf into
two functions.

Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Steven Shi <steven.shi@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Christian Rodriguez <christian.rodriguez@intel.com>
Reviewed-by: Steven Shi <steven.shi@intel.com>
Reviewed-by: Jaben Carsey <jaben.carsey@intel.com>
2019-06-25 09:23:57 +08:00
Bob Feng dc174cd9e5 BaseTools: refine CreateAsBuiltInf function
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1932
ModuleAutogen.CreateAsBuiltInf IsOnlyCopy parameter
and related function can be moved out of CreateAsBuiltInf,
so that CreateAsBuiltInf is more clean.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Steven Shi <steven.shi@intel.com>
Cc: Christian Rodriguez <christian.rodriguez@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Steven Shi <steven.shi@intel.com>
Reviewed-by: Jaben Carsey <jaben.carsey@intel.com>
2019-06-25 09:23:53 +08:00
Fan, ZhijuX 370f16c548 BaseTools:Add import in FvImageSection
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=1872

Since MultipleWorkspace is not imported in FvImageSection,
an error of "global name 'MWS' is not defined" appeared.

This patch is going to fix that issue.

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-06-24 17:02:53 +08:00
Shi, Steven 2378ea5515 BaseTools:Introduce CopyFileOnChange() function to copy cache files
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=1894

Basetool need a CopyFileOnChange() function to avoid cache
file writing race in multi-thread build. Some platforms
build fail with file IO writing race issue when the
build cache is enabled to store cache files in multi-threads.
This is because common same library cache files (e.g. some
libs in MdePkg) can be stored by many different driver modules'
build threads at same time. Current build cache need a function
to check whether the same cache file already exist, and only
copy source file if it is different from the destination file.

This patch introduces an atomic copy function to avoid duplicated
cache files copy.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Christian Rodriguez <christian.rodriguez@intel.com>
Signed-off-by: Steven Shi <steven.shi@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-06-17 17:00:46 +08:00
Shi, Steven 17b082ce67 BaseTools: Cannot store library cache of different arch together
https://bugzilla.tianocore.org/show_bug.cgi?id=1895

Build cache cannot store cache for the same library modules
in different arch together. E.g. Both the below IA32 and X64
arch BaseLib caches should exist after build Ovmf3264, but now
only the one in X64 arch exist.
The reason is the current Basetool use a set() to same all
library AutoGen objects, but the different arch lib AutoGen
objects have same __hash_ value which comes from the lib
MetaFile(The path of module file):
    def __hash__(self):
        return hash(self.MetaFile)

So the different arch lib AutoGen objects are duplicated one
to the set() and only one can exist. This is why the Basetool
can only store one arch cache for library.

This patch adds the arch string into the PlatformAutoGen and
ModuleAutoGen __hash_ definitions and ensure the different
platform and module AutoGen objects have different __hash_ values.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Christian Rodriguez <christian.rodriguez@intel.com>
Signed-off-by: Steven Shi <steven.shi@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-06-17 17:00:45 +08:00
Shi, Steven 04df18535c BaseTools:Build Cache output notification message
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=1868

Build need output the cache miss or hit notification
message when consume the build cache. Current build does not
output any message which is not clear for user to know
whether the module built result is from cache or not.

This patch adds message about the cache miss or hit when
build consumes the cache.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Zhiju.Fan <zhijux.fan@intel.com>
Signed-off-by: Steven Shi <steven.shi@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-06-17 17:00:45 +08:00
Shi, Steven ede54e3588 BaseTools:Build cache cannot store the cache files for library package
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1867

Current build cache cannot store the cache for library package.
build -p MdePkg\MdePkg.dsc -a IA32 -b DEBUG -t VS2015x86 --hash
--binary-destination=BinCache
After build, the expected result is the BinCache folder is generated
and the MdePkg build cache files (e.g. .hash and .lib) are stored in
the BinCache folder. But the BinCache folder is not generated at all.

This patch is going to fix that issue.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Zhiju.Fan <zhijux.fan@intel.com>
Signed-off-by: Steven Shi <steven.shi@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-06-17 17:00:45 +08:00
Rodriguez, Christian 48b0bf6476 BaseTools: Refactor hash tracking after checking for Sources section
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1804

After adding a check to see if [Sources] section lists all the header
type files of a module, track module and library hashes for --hash
feature. If above check is not in compilance for a library or module,
force hash invalidation on that library or module.

Signed-off-by: Christian Rodriguez <christian.rodriguez@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-06-10 19:48:51 +08:00
Rodriguez, Christian 1fa6699e6c BaseTools: Add a checking for Sources section in INF file
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1804

Add a check to see if [Sources] section lists all the header type
files of a module. Performance impact should be minimal with this patch
since information is already being fetched for Makefile purposes. All
other information is already cached in memory. No extra IO time is needed.

Signed-off-by: Christian Rodriguez <christian.rodriguez@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-06-10 19:48:51 +08:00
Fan, ZhijuX 04797875d1 BaseTools:Make BaseTools support new rules to generate RAW FFS FILE
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=1765

If RAW FFS File Rule has no section for its data.For RAW FFS File,
directly call GenFfs tool to generate FFS file.

Ffs Rule:
[Rule.Common.USER_DEFINED.MicroCode]
  FILE RAW = $(NAMED_GUID) {
        $(INF_OUTPUT)/$(MODULE_NAME).bin
  }
[Rule.Common.USER_DEFINED.LOGO]
  FILE RAW = $(NAMED_GUID) {
                       |.bmp
  }

As shown in the rule above,if SectionType and FileType not defined,
FFS files are generated directly, and no other type of file is
generated.

The patch is to make the BaseTools support these two rules

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
2019-06-10 19:48:50 +08:00