This patch updates the GenC logic to generate a random stack cookie value
for the stack check libraries. These random values improve security
for modules which cannot update the global intrinsics.
If the stack cookie value is randomized in the AutoGen.h file each
build, the build system will determine the module/library must be
rebuilt causing effectively a clean build every time. This also makes
binary reproducibility impossible.
This patch updates the early build scripts to create 32 and 64-bit JSON
files in the build output directory which each contain 100 randomized
stack cookie values for each bitwidth. If the JSON files are already
present, then they are not recreated which allows them to be stored and
moved to other builds for binary reproducibility. Because they are in
the build directory, a clean build will cause the values to be
regenerated.
The logic which creates AutoGen.h will read these JSON files and use a
hash of the module GUID (the hash seed is fixed in Basetools) to index
into the array of stack cookie values for the module bitwidth. This
model is necessary because there isn't thread-consistent data so we
cannot use a locking mechanism to ensure only one thread is writing to
the stack cookie files at a time. With this model, the build threads
only need to read from the files.
Signed-off-by: Oliver Smith-Denny <osde@linux.microsoft.com>
EdkLogger logs were not showing up as part of the build log output.
Adding the EdkLogger import to GenMake.py fixes the missing log prints.
Signed-off-by: Kenneth Lautner <kenlautner3@gmail.com>
Most module types have standardized entry point function prototypes. They
are declared in headers like
- MdePkg/Include/Library/PeiCoreEntryPoint.h
- MdePkg/Include/Library/PeimEntryPoint.h
- MdePkg/Include/Library/DxeCoreEntryPoint.h
- MdePkg/Include/Library/UefiDriverEntryPoint.h
- MdePkg/Include/Library/UefiApplicationEntryPoint.h
These header files also declare matching ProcessLibraryConstructorList()
prototypes.
The SEC module type does not have a standardized entry point prototype
(aka parameter list), therefore no header file like the above ones exists
for SEC. Consequently, no header file *declares*
ProcessLibraryConstructorList() for SEC modules, even though AutoGen
always *defines* ProcessLibraryConstructorList() with the same, empty,
parameter list (i.e., just (VOID)).
The lack of a central declaration is a problem because in SEC code,
ProcessLibraryConstructorList() needs to be called manually, and those
calls need a prototype. Most SEC modules in edk2 get around this by
declaring ProcessLibraryConstructorList() manually, while some others use
an incorrect (PEIM) prototype.
Liming suggested in
<https://bugzilla.tianocore.org/show_bug.cgi?id=991#c2> that AutoGen
provide the declaration as well; implement that in this patch.
Mike suggested that the feature be gated with INF_VERSION, for
compatibility reasons. (INF_VERSION >= 1.30) reflects that the latest
(draft) version of the INF specification, as of this writing, is commit
a31e3c842bee / version 1.29.
For example, if we modify "OvmfPkg/Sec/SecMain.inf" as follows:
> diff --git a/OvmfPkg/Sec/SecMain.inf b/OvmfPkg/Sec/SecMain.inf
> index 3c47a664a95d..dca932a474ee 100644
> --- a/OvmfPkg/Sec/SecMain.inf
> +++ b/OvmfPkg/Sec/SecMain.inf
> @@ -8,7 +8,7 @@
> ##
>
> [Defines]
> - INF_VERSION = 0x00010005
> + INF_VERSION = 1.30
> BASE_NAME = SecMain
> FILE_GUID = df1ccef6-f301-4a63-9661-fc6030dcc880
> MODULE_TYPE = SEC
then the patch produces the following difference in
"Build/OvmfX64/NOOPT_GCC5/X64/OvmfPkg/Sec/SecMain/DEBUG/AutoGen.h":
> --- AutoGen.h.orig 2024-02-06 23:10:23.469535345 +0100
> +++ AutoGen.h 2024-02-07 00:00:57.361294055 +0100
> @@ -220,6 +220,13 @@
>
> // Definition of PCDs used in libraries is in AutoGen.c
>
> +// ProcessLibraryConstructorList() declared here because SEC has no standard entry point.
> +VOID
> +EFIAPI
> +ProcessLibraryConstructorList (
> + VOID
> + );
> +
>
> #ifdef __cplusplus
> }
which presently (as of edk2 commit edc6681206) triggers the following
build error:
> In file included from OvmfPkg/Sec/SecMain.c:14:
> MdePkg/Include/Library/PeimEntryPoint.h:74:1: error: conflicting types for
> ‘ProcessLibraryConstructorList’; have ‘void(void *, const
> EFI_PEI_SERVICES **)’ {aka ‘void(void *, const struct _EFI_PEI_SERVICES
> **)’}
> 74 | ProcessLibraryConstructorList (
> | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> In file included from <command-line>:
> Build/OvmfX64/NOOPT_GCC5/X64/OvmfPkg/Sec/SecMain/DEBUG/AutoGen.h:226:1: note:
> previous declaration of ‘ProcessLibraryConstructorList’ with type
> ‘void(void)’
> 226 | ProcessLibraryConstructorList (
> | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
That's a genuine bug in OvmfPkg that needs to be fixed, but we keep
compatibility with existent SEC modules until/unless they upgrade
INF_VERSION to 1.30+.
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <gaoliming@byosoft.com.cn>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Rebecca Cran <rebecca@bsdio.com>
Cc: Yuwei Chen <yuwei.chen@intel.com>
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=991
Suggested-by: Liming Gao <gaoliming@byosoft.com.cn>
Suggested-by: Michael D Kinney <michael.d.kinney@intel.com>
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Message-Id: <20240224210504.41873-1-lersek@redhat.com>
Reviewed-by: Liming Gao <gaoliming@byosoft.com.cn>
Currently when the platform has many SKUs then allskuset will be having
so many duplicate. and while parsing the allskuset will take longer
time while assigning Pcd.SkuInfoList.
This patch is to eliminate those duplicate entries to reduce the
build time
Cc: Yuwei Chen <yuwei.chen@intel.com>
Cc: Rebecca Cran <rebecca@bsdio.com>
Cc: Liming Gao <gaoliming@byosoft.com.cn>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Amy Chan <amy.chan@intel.com>
Cc: Sai Chaganty <rangasai.v.chaganty@intel.com>
Signed-off-by: Ashraf Ali S <ashraf.ali.s@intel.com>
Reviewed-by: Yuwei Chen <yuwei.chen@intel.com>
Reviewed-by: Amy Chan <amy.chan@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
During the Incremental build GenerateByteArrayValue used to generate the
ByteArrayValue even when there is no change in the PCD/VPDs. which is
time consuming API based on the number of PCD/VPDs and SKU IDs.
The optimization is that GenerateByteArrayValue is used to store the
StructuredPcdsData in a JSON file for each of the arch. and during the
Incremental build this API will check, if there is any change in the
Structured PCD/VPDs then rest of the flow remains the same.
if there is no change then it will return the provious build data.
Flow:
during the 1st build StructuredPcdsData.json is not exists,
StructuredPcdsData will be dumped to json file. and it will copy the
output.txt as well.
Note: as the output.txt are different for different Arch, so it will be
stored in the Arch folder.
During the Incremental build check if there is any change in Structured
PCD/VPD. if there is a change in Structured VPD/PCD then recreate the
StructuredPcdsData.json, and rest of the flow remains same.
if there is no change in VPD/PCD read the output.txt and return the data
Unit Test:
Test1: Modified the Structured Pcds default from DEC file. current flow
is executing.
Test2: Override the default value of the PCD from DEC file. current flow
is executing.
Test3: Modified/Override the PCD from DSC file. current flow executing
Test4: Modified/Override the FDF from DSC file. current flow executing
Test5: update the default value from Command Line.current flow executing
Test6: Build without change in PCD in DSC, FDF, DEC and Command Line the
proposed changes will be executing, and the return data remains the same
with and without the changes.
Test7: Build with and without modified the include headers of Structured
PCDs. if there is any change in those Structured PCD header then
current flow will be executed.
With these changes it's helping to save around ~2.5min to ~3.5min of
Incremental build time in my build environment.
Sample PR: https://github.com/tianocore/edk2-basetools/pull/113
Cc: Yuwei Chen <yuwei.chen@intel.com>
Cc: Rebecca Cran <rebecca@bsdio.com>
Cc: Liming Gao <gaoliming@byosoft.com.cn>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Amy Chan <amy.chan@intel.com>
Cc: Sai Chaganty <rangasai.v.chaganty@intel.com>
Cc: Digant H Solanki <digant.h.solanki@intel.com>
Signed-off-by: Ashraf Ali S <ashraf.ali.s@intel.com>
Reviewed-by: Yuwei Chen <yuwei.chen@intel.com>
The patch "[PATCH v3 1/2] StandaloneMmPkg: Make StandaloneMmCpu driver
architecture independent" (https://edk2.groups.io/g/devel/message/109178)
removed ArmPkg/ArmPkg.dec from the Packages section in the
INF file: StandaloneMmPkg/Drivers/StandaloneMmCpu/StandaloneMmCpu.inf
This change was done as part of making the StandaloneMmCpu driver
architecture independent.
Although this change is correct, it results in a side effect
here some platforms that utilise PCDs declared in ArmPkg.dec are
no longer declared.
An example of this issue can be seen when building
edk2-platforms/Platform/ARM/SgiPkg/PlatformStandaloneMm.dsc
$ build -a AARCH64 -t GCC -p Platform/ARM/SgiPkg/PlatformStandaloneMm.dsc
build.py...
/mnt/source/edk2-platforms/Platform/ARM/SgiPkg/PlatformStandaloneMm.fdf(23):
error F001: PCD (gArmTokenSpaceGuid.PcdFdBaseAddress) used in
FDF is not declared in DEC files.
As seen above, removing ArmPkg.dec from the Packages section in the
StandAloneMmCpu Driver Inf file triggers build failure.
Although, ArmPkg.dec is included in other Library Instances,
the build system does not include the declarations from
.dec files defined in Library instances.
The build system only includes the PCD declarations from DEC files
that are specified in INF files for Modules (components).
Therefore, extend the build system to include the Packages from
Library Instances so that the PCD declarations from the respective package
DEC files are included.
This patch can be seen on
https://github.com/LeviYeoReum/edk2/tree/levi/2848_dec_check_on_library
Signed-off-by: levi.yun <yeoreum.yun@arm.com>
Tested-by: Pierre Gondois <pierre.gondois@arm.com>
Reviewed-by: Rebecca Cran <rebecca@bsdio.com>
Reviewed-by: Sami Mujawar <sami.mujawar@arm.com>
Fixes raw regex strings that contain valid (and purposeful) escape
characters as they are being treated as individual characters rather
than the single escaped character they represent (i.e. '\t' is being
treated as a '\' and a 't' rather than a single tab character).
Signed-off-by: Joey Vagedes <joey.vagedes@gmail.com>
Cc: Rebecca Cran <rebecca@bsdio.com>
Cc: Liming Gao <gaoliming@byosoft.com.cn>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Yuwei Chen <yuwei.chen@intel.com>
Reviewed-by: Yuwei Chen <yuwei.chen@intel.com>
Reviewed-by: Liming Gao <gaoliming@byosoft.com.cn>
Switches regex patterns to raw text to resolve python 3.12 syntax
warnings in regards to invalid escape sequences, as is suggested by the
re (regex) module in python.
Cc: Rebecca Cran <rebecca@bsdio.com>
Cc: Liming Gao <gaoliming@byosoft.com.cn>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Yuwei Chen <yuwei.chen@intel.com>
Signed-off-by: Joey Vagedes <joey.vagedes@gmail.com>
Reviewed-by: Rebecca Cran <rebecca@bsdio.com>
Currently BaseTools use 'cp' command for PcdValueInit and GenMake
process, as the command can not keep the time info of the source
file, which will cause incremental build issue in Linux system,
thus the '-p' need be added to keep the source file's attributes
in copy process.
This patch fixes this issue.
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <gaoliming@byosoft.com.cn>
Signed-off-by: Yuwei Chen <yuwei.chen@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
Build-rules.txt lists .cc and .cpp as supported file extensions.
BaseTools commit 05217d210e introduce a regression issue that
ignore the .cc and .cpp file type.
This patch is to fix this bug.
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <gaoliming@byosoft.com.cn>
Cc: Yuwei Chen <yuwei.chen@intel.com>
Reviewed-by: Yuwei Chen<yuwei.chen@intel.com>
RVCT is obsolete and no longer used.
Remove support for it.
Signed-off-by: Rebecca Cran <quic_rcran@quicinc.com>
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=3828
This is a bugfix of
bf9230a9f3.
1.In the current code, gPlatformFinalPcd will save all PCDs used at
whole compile process, which wastes runtime memory and is unnecessary.
This patch makes gPlatformFinalPcd save only the PCDes which are
assigned in the DSC file, and the PCD that has not been assigned will
use the default value in DEC.
2.During the compilation process, gPlatformFinalPcd may be lost, and
the current code cannot selectively assign PCD in DSC by specifying ARCH.
This patch moves gPlatformFinalPcd into datapipe and modifies the
assignment logicto fix this.
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <gaoliming@byosoft.com.cn>
Signed-off-by: yi1 li <yi1.li@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
There are two type variable header and their size are different,
need to use matched size when calculating offset info, otherwise
it'll destroy other variables content when patching.
Signed-off-by: Chen, Lin Z <lin.z.chen@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
currently the --cmd-len build option does not work.
This patch is going to fix this bug.
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <gaoliming@byosoft.com.cn>
Cc: Yuwei Chen <yuwei.chen@intel.com>
Reviewed-by: Yuwei Chen <yuwei.chen@intel.com>
In order to support secure boot with authenticated type variable store
and non secure boot with normal type variable store, add one flag to
switch them.
User can append '-D VPD_AUTHENTICATED_VARIABLE_STORE' to build command
to enable authenticated type varaible store.
Also, user can add 'VPD_AUTHENTICATED_VARIABLE_STORE = TRUE/FALSE' to the
defines section of Dsc file to switch authenticated/normal type variable
store.
VPD_AUTHENTICATED_VARIABLE_STORE is a new reserved key word for this function.
Signed-off-by: Chen Lin Z <lin.z.chen@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <gaoliming@byosoft.com.cn>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=3653
The Makefile and MakefilName fields are never set/used. Remove them.
To check this, the following commands can be used:
- grep -rIn "\.Makefile"
- grep -rIn "\.MakefileName"
Signed-off-by: Pierre Gondois <Pierre.Gondois@arm.com>
Reviewed-by: Chris Jones <christopher.jones@arm.com>
Reviewed-by: Liming Gao <gaoliming@byosoft.com.cn>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=3653
Use the value set in tools_def.txt when the makefile type is
not explicitly set via BuildOption. This allows to have a
valid default makefile name instead of an empty string.
Also use GMAKE_FILETYPE instead of hard-coded "gmake".
Signed-off-by: Pierre Gondois <Pierre.Gondois@arm.com>
Reviewed-by: Chris Jones <christopher.jones@arm.com>
Reviewed-by: Liming Gao <gaoliming@byosoft.com.cn>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=3359
If the DSC file provides an override of a Guided tool path
and/or Guided tool GUID value, then make sure the one from the
DSC file is used if it is higher priority than the Guided tool
in the tools_def.txt file. This makes the Guided tool used by
GenFds match the tool listed GuidedSectionTools.txt.
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <gaoliming@byosoft.com.cn>
Cc: Yuwei Chen <yuwei.chen@intel.com>
Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=3359
* Save/Restore global state in GetToolChainAndFamilyFromDsc()
This resolves an issue where the multi-arch build for
UefiPayloadPkg would skip the autogen and build of IA32
components.
* Expand tools wildcard.
This resolves the issue where autogen makefile contents
would have a BUIDLRULEFAMILY tools definitions with an
'*' in the tool field that breaks the build from invalid
makefile syntax.
* Build rule family higher priority than Family.
This resolves the issue where flags were appended from
both the BUILDRULEFAMILY and FAMILY when only
BUILDRULEFAMILY should be appended when present.
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <gaoliming@byosoft.com.cn>
Cc: Yuwei Chen <yuwei.chen@intel.com>
Cc: Ray Ni <ray.ni@intel.com>
Cc: Steven Shi <steven.shi@intel.com>
Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com>
Acked-by: Rebecca Cran <rebecca@bsdio.com>
Reviewed-by: Andrew Fish <afish@apple.com>
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=3359
Update BaseTools to support new build targets, new tool chains,
and new architectures declared in DSC file [BuildOptions] sections.
* Do not expand * when tools_def.txt is parsed. Only expand when
both tools_def.txt and DSC [BuilsOptions] sections have been parsed.
This also requires more flexible matching of tool keys that contain *
in tool key fields.
* Pre-scan the platform DSC file for FAMILY and TOOLCHAIN declarations
DSC in [BuildOptions] sections before the FAMILY and TOOLCHAIN need
to be known.
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <gaoliming@byosoft.com.cn>
Cc: Yuwei Chen <yuwei.chen@intel.com>
Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=3353
Sort the tool definition content of generated makefiles to help
verify that makefile contents have not changed after BaseTools
code changes.
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <gaoliming@byosoft.com.cn>
Cc: Yuwei Chen <yuwei.chen@intel.com>
Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Yuwei Chen <yuwei.chen@intel.com>
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=3344
Veify that TAB_TOD_DEFINES_FAMILY key is present in ToolDef
dictionary before access.
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <gaoliming@byosoft.com.cn>
Cc: Yuwei Chen <yuwei.chen@intel.com>
Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=3312
Update parsing of MAKE_FLAGS in DSC [BuildOptions] sections
to split the flags into a list to be compatible with
running the make command using Popen(). Parsing MAKE_FLAGS
from tools_def.txt already uses _SplitOption(). This change
uses the same _SplitOption() method for MAKE_FLAGS from a
DSC [BuildOptions] section.
Also update the parsing of MAKE_PATH to support MAKE_PATH
from tools_def.txt or the DSC [BuildOptions] section. MAKE_PATH
in DSC [BuildOptions] section is higher priority than MAKE_PATH
in tools_def.txt.
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <gaoliming@byosoft.com.cn>
Cc: Yuwei Chen <yuwei.chen@intel.com>
Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com>
Reviewed-by: Yuwei Chen <yuwei.chen@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
EdkLogger.ERROR() was replaced with EdkLogger.error() to deliver the
expected error message when an error occurs.
Signed-off-by: Irene Park <ipark@nvidia.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Liming Gao <gaoliming@byosoft.com.cn>
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=3120
FixedAtBuild, PatchableInModule and FeatureFlag PCD don't use PCD TOKEN.
Their PCD TOKEN value can always be zero.
AutoGen.h will not be changed when static PCD is added or removed.
Dynamic PCD add or remove will still cause dynamic PCD token value be
changed.
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <gaoliming@byosoft.com.cn>
Cc: Yuwei Chen <yuwei.chen@intel.com>
Signed-off-by: Yunhua Feng <fengyunhua@byosoft.com.cn>
Reviewed-by: Liming Gao <gaoliming@byosoft.com.cn>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
This problem is generated by solving bz2972's
patch, and the commit ID is
0af7f8e6a9.
This is a problem when updating the DEPs file.
The code does not consider that there is only
one line of content in the file, so the filter
condition is added to prevent the index from
exceeding the range.
Signed-off-by: Mingyue Liang <mingyuex.liang@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <gaoliming@byosoft.com.cn>
Cc: Yuwei Chen <yuwei.chen@intel.com>
Reviewed-by: Yuwei Chen <yuwei.chen@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Liming Gao <gaoliming@byosoft.com.cn>
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=2528
Currently, CL command contains multiple C files will be compiled,
and that caused command line too long, which may trigger build error.
In order to solve this issue, the following rules is used in this scene:
If the number of C files is greater than one, a txt file will be used
to record these C files, and replaces the corresponding content in
command line with the file name.
Else (only one C file listed in the command line), the length of the
whole CL command line will determine whether use a file to record. If
the length exceeds the limited max length, use the recording file; else
C file name directly listed in the command line
Signed-off-by: Mingyue Liang <mingyuex.liang@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <gaoliming@byosoft.com.cn>
Cc: Yuwei Chen <yuwei.chen@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=2648
This patch is to enable the Module scoped Structure Pcd usage.
User can set structure pcd field value in module scope. For example,
under the [components] section of a dsc file, user can override some
field value for a specific module.
Package/Module.inf{
<PcdsFixedAtBuild>
gUefiTokenSpaceGuid.StructurePcdModule.FieldName | 5
}
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <gaoliming@byosoft.com.cn>
Cc: Yuwei Chen <yuwei.chen@intel.com>
Tested-by: Liming Gao <gaoliming@byosoft.com.cn>
Acked-by: Liming Gao <gaoliming@byosoft.com.cn>
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=2978
If a module add a new PCD, the pcd token number will be
reassigned. The new Pcd token number should be updated
to all module's autogen files. CanSkip can only detect a
single module's change but not others. CanSkip block the
pcd token number update in incremental build, so this
patch is going to remove this call.
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <gaoliming@byosoft.com.cn>
Cc: Yuwei Chen <yuwei.chen@intel.com>
Reviewed-by: Yuwei Chen<yuwei.chen@intel.com>
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=2880
Currently, When doing the Incremental build, the directory
macros extended to absolute path in output Makefile, which
is inconsistent with the output of Clean build.
When we do macro replacement, we can't replace macro due to
inconsistent path case, which results in inconsistent display
of incremental build and clean build in makefile.Therefore,
the path is converted to achieve the correct macro replacement.
Signed-off-by: Mingyue Liang <mingyuex.liang@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <gaoliming@byosoft.com.cn>
Cc: Yuwei Chen <yuwei.chen@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Yuwei Chen <yuwei.chen@intel.com>
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=2882
After changing the name of the include source file,
when doing incremental build, the previous source file
is not covered in the. DEPs file, and a build error occurs.
The root cause is that the build tools filter out some dependency
files, which are listed in inf source section, from the deps_target file.
Add those files back to deps_target file to resolve the above problem.
Signed-off-by: Mingyue Liang <mingyuex.liang@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <gaoliming@byosoft.com.cn>
Cc: Yuwei Chen <yuwei.chen@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Yuwei Chen <yuwei.chen@intel.com>
If VPD PcdNvStoreDefaultValueBuffer is used, all DynamicHii and
DynamicExHii PCD value will be generated into that VPD.
In order to generate the same VPD binary file in every build,
sort the Pcd set when generating VPD.
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <gaoliming@byosoft.com.cn>
Cc: Yuwei Chen <yuwei.chen@intel.com>
Reviewed-by: Liming Gao <gaoliming@byosoft.com.cn>
Reviewed-by: Philippe Mathieu-Daude <philmd@redhat.com>
python3.9 changed/fixed codec.register behavior to always replace
hyphen with underscore for passed in codec names:
https://bugs.python.org/issue37751
So the custom Ucs2Search needs to be adapted to handle 'ucs_2' in
addition to existing 'ucs-2' for back compat.
This fixes test failures on python3.9, example:
======================================================================
FAIL: testUtf16InUniFile (CheckUnicodeSourceFiles.Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/builddir/build/BUILD/edk2-edk2-stable202002/BaseTools/Source/Python/AutoGen/UniClassObject.py", line 375, in PreProcess
FileIn = UniFileClassObject.OpenUniFile(LongFilePath(File.Path))
File "/builddir/build/BUILD/edk2-edk2-stable202002/BaseTools/Source/Python/AutoGen/UniClassObject.py", line 303, in OpenUniFile
UniFileClassObject.VerifyUcs2Data(FileIn, FileName, Encoding)
File "/builddir/build/BUILD/edk2-edk2-stable202002/BaseTools/Source/Python/AutoGen/UniClassObject.py", line 312, in VerifyUcs2Data
Ucs2Info = codecs.lookup('ucs-2')
LookupError: unknown encoding: ucs-2
Signed-off-by: Cole Robinson <crobinso@redhat.com>
Reviewed-by: Yuwei Chen <yuwei.chen@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
AsBuiltInf can be created during AutoGen phase. Move CreateAsBuiltInf
into AutoGenWorker to make this step run in parallel.
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yuwei Chen <yuwei.chen@intel.com>
Reviewed-by: Jaben Carsey <jaben.carsey@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=2807
The Queue.empty() method is not reliable in the multiple
process runtime environment. This patch uses a new method
to check if all modules are processed and workers need
to be stopped. That is to add a None item at the bottom
of the queue. Worker check if it gets that None item to
know if all the module is processed.
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yuwei Chen <yuwei.chen@intel.com>
Cc: Lucy Yan <lucyyan@google.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=2762
The Fv Section in the FDF files use hard coding Guid values
which is inconvenient to manage. This patch adds Guid name
support in GenFfs to solve this problem.
Signed-off-by: Yuwei Chen <yuwei.chen@intel.com>
Reviewed-by: Bob Feng<bob.c.feng@intel.com>
Using Python 3.7.2 on win32, when printing a FileBuildRule
instance, the following error occurs:
File "edk2\BaseTools\Source\Python\AutoGen\BuildEngine.py",
line 177, in __str__
DestString = ", ".join(self.DestFileList)
TypeError: sequence item 0: expected str instance, PathClass found
This patch converts each PathClass element of the list to a string
instance before concatenating them.
Signed-off-by: Pierre Gondois <pierre.gondois@arm.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
This patch modifies the Makefile generation not to stop
adding Makfile rules when the first final target is found.
E.g.:
If the following rules are described in build_rule.txt:
-[Rule1]: .X files generate .Y and .Z files;
-[Rule2]: .Z files generate .Z1 files.
Currently, if a File1.X file was part of the sources of a
module, only [Rule1] would be generated in the Makefile.
Indeed, there are no rules to apply to .Y files: .Y files
are a final target. However, there is still [Rule2] to
apply to .Z files.
This patch also adds a dependency between the first
ouput file of a rule and the other output files.
For instance, with the same example as above, File1.Y
and File1.Z are generated by the following rule:
File1.Y: File1.X
<Generate File1.Y>
<Generate File1.Z>
and the new dependency is:
File1.Z: File1.Y
This is necessary to keep a dependency order during the
execution of the Makefile. Indeed, .Y and .Z files are
generated by the execution of a common set of commands,
and without this rule, there is no explicit dependency
relation between them.
Signed-off-by: Pierre Gondois <Pierre.Gondois@arm.com>
Suggested-by: Tomas Pilar <Tomas.Pilar@arm.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
REF:https://bugzilla.tianocore.org/show_bug.cgi?id=2538
For LzmaCompress or BrotliCompress, the platform may use the different
options and add their batch file, such as LzmaCompressPlatform.
Then, specify it in platform.dsc [BuildOptions] to override the default
one in tools_def.txt.
*_*_*_LZMA_PATH = LzmaCompressPlatform
This override tool will be used. But, its name is not specified in the
generated GuidedSectionTools.txt.
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=2610
We meet a case that the DEC file declaring the PCD isn't
included in the INF.it cause build tools report Traceback error.
Remove raise statements that generate Tracebacks that were only
intended for development/debug. With the raise statements removed
proper error messages are shown.
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=2563
This patch is to fix a incremental build regression bug
which happen when using nmake. That's introduced by 818283de3f.
If there is white space before !INCLUDE instruction, nmake will not
process it. Source code's dependent header files are listed in
${deps_file} file, if it's not included successfully, nmake will
not detect the change of those header file.
This patch has been verified in Windows with VS2015 and Linux with GCC5.
The header file add/modify/delete can trig the incremental build with this fix.
There is no impact on the clean build.
Cc: Andrew Fish <afish@apple.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Cc: Leif Lindholm <leif@nuviainc.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Pierre Gondois <pierre.gondois@arm.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Tested-by: Liming Gao <liming.gao@intel.com>
REF:https://bugzilla.tianocore.org/show_bug.cgi?id=2520
Incremental build failure in CLANGPDB tool chain on Windows host
The build failure is like below when do incremental build.
The root cause is in generated deps_target file. It has one line ":".
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
NASM_INC contains the list of directory to include when using
the nasm assembler.
In nmake makefiles, a trailing backslash escapes the newline char
and replaces it with a space ' '. To have a literal trailing
backslash, it must be escaped with a caret '^'. This is not
necessary for GNU makefiles.
On windows platforms, for the NASM_INC macro, a caret escaping
a trailing a backslash was appended to the last included
folder regardless of the makefile type.
For instance, "/Include/" was replaced by "/Include/^\".
This is causing a build failure on windows platforms using
GNU makefiles since the caret '^' doesn't escape any chars in
GNU makefiles and is thus conserved.
"/Include^\" was replaced by "/Include\/" in nmake makefiles,
but remained "/Include/^\" in GNU makefiles.
This patch removes the caret '^' on the build using GNU makefiles.
Signed-off-by: Pierre Gondois <pierre.gondois@arm.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
The GenMake.py script tests the platform environment
to determine the type of makefile that needs to be
generated. If a Windows build host is detected, the
makefile generated is of Nmake type. Otherwise a
GNUmake type is generated.
Furthermore, the <TARGET>_<TAGNAME>_<ARCH>_MAKE_PATH
option in tools_def.template defines the make tool
to use.
E.g.: for VS2017 this is configured to use Nmake, cf.
*_VS2017_*_MAKE_PATH = DEF(VS2017_BIN_HOST)\nmake.exe
while for GCC5 it is setup to use GNU make.
*_GCC5_*_MAKE_PATH = DEF(GCC_HOST_PREFIX)make
This prevents using the GCC compiler toolchain on a
Windows build host.
To address this issue this patch introduces 2 factors
to determine the generated makefile output.
1. Platform -> to determine shell commands used
in makefile.
2. MakeTool -> to determine the type of makefile
that needs to be generated.
Signed-off-by: Pierre Gondois <pierre.gondois@arm.com>
Signed-off-by: Sami Mujawar <sami.mujawar@arm.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
The .map file is not update to FFS_OUTPUT_DIR folder
in the incremental build.
Signed-off-by: Guo Dong <guo.dong@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
REF:https://bugzilla.tianocore.org/show_bug.cgi?id=2451
If removing a header file from source code and file
system, the incremental build will fail.
This patch is to fix this issue by setting each header file
as a target without any actions in makefile.
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=2455
BuildOption is used by TargetTxtClassObj.py
GenFdsOption is used by GenFds.py
When the GenFds tool is used alone (e.g. python3 -m GenFds.GenFds -h)
With the OptionParser function, the first detected function
prints the help message
import TargetTxtClassObj to GenFds,
The BuildOption will be executed and replace GenFdsOption
We removed all objects associated with this problem that
were created directly during the import process
(e.g. BuildOption, BuildTarget = MyOptionParser(),
TargetTxt = TargetTxtDict())
The Patch is going to fix this issue
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=2371
This patch is to fix a regression issue that build fails
if multiple build targets given.
Two changes cause this regression issue.
One is AutoGen object __hash__ function only
hash file path and arch, missing ToolChain and build target.
The other is changing the multiple-thread-genfds function as default
build behavior. To generate the genffs command to Makefile, there
is a global data set is used, GenFdsGlobalVariable, which cause build
tool use the data of first build-target build in
the second build-target build.
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Steven Shi <steven.shi@intel.com>
Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Taylor Beebe
Ken Lautner
Laszlo Ersek
Ashraf Ali S
devel@edk2.groups.io
levi.yun
Joey Vagedes
Joey Vagedes via groups.io
Chen, Christine
Feng, Bob C
Rebecca Cran
Li, Yi1
Chen, Lin Z
Pierre Gondois
Cole
Michael D Kinney
Irene Park
Yunhua Feng
Mingyue Liang
Fan, ZhijuX