Building and Iterating

[Sean Bruno]

Before I dive into the mk files of a buildworld, I'd like to describe
"what I want" so as to start a discussion of my goal.

1. If I select no toolchain (WITHOUT_TOOLCHAIN), but clang needs to be
built, only build a toolchain that targets the ARCH being requested.

2. If I select no toolchain (WITHOUT_TOOLCHAIN), but clang needs to be
built, give me a knob to turn that aborts the build with a meaningful
message that lets me know I need to update the toolchain on my buildbox.

3. If the boostrap toolchain needs to be built in the normal case, only
target the ARCH being requested. I understand that we "want" a CC
installed that targets all architectures and this is something I agree with.

sean

[Ed Maste]

Post by Sean Bruno
Before I dive into the mk files of a buildworld, I'd like to describe
"what I want" so as to start a discussion of my goal.
1. If I select no toolchain (WITHOUT_TOOLCHAIN), but clang needs to be
built, only build a toolchain that targets the ARCH being requested.

Note that WITHOUT_TOOLCHAIN controls only what tool chain is built for
your target -- i.e., whether the installed world has a /usr/bin/clang,
/usr/bin/ld, etc. It has no effect on whether or not
Clang/LLVM/LLD/ELF Tool Chain is built as a build tool.

[Ryan Stone]

Post by Sean Bruno
3. If the boostrap toolchain needs to be built in the normal case, only
target the ARCH being requested. I understand that we "want" a CC
installed that targets all architectures and this is something I agree with.

Has anybody instrumented the build to determine how much time this
would actually save? Before trying to optimize the build we need to
be sure that we're actually targetting the right optimizations.

[Lev Serebryakov]

Post by Ryan Stone

Post by Sean Bruno
3. If the boostrap toolchain needs to be built in the normal case, only
target the ARCH being requested. I understand that we "want" a CC
installed that targets all architectures and this is something I agree with.

Has anybody instrumented the build to determine how much time this
would actually save? Before trying to optimize the build we need to
be sure that we're actually targetting the right optimizations.

LLVM build could easily take one hour to be built on rather modern system.

My VM to build NanoBSD images has 8GiB of RAM (1/4 of total physical
RAM), 4 cores of i7 (out of 4 cores / 8 threads) and resides on SSD
(SATA one). Build WITH cross-toolchain AND world toolchain is about 2.5
hours and build WITHOUT ANY toolchain (using host's one as
cross-compiler) is less than a hour.

[Conrad Meyer]

Post by Lev Serebryakov

Post by Ryan Stone

Post by Sean Bruno
3. If the boostrap toolchain needs to be built in the normal case, only
target the ARCH being requested. I understand that we "want" a CC
installed that targets all architectures and this is something I agree with.

Has anybody instrumented the build to determine how much time this
would actually save? Before trying to optimize the build we need to
be sure that we're actually targetting the right optimizations.

LLVM build could easily take one hour to be built on rather modern system.
My VM to build NanoBSD images has 8GiB of RAM (1/4 of total physical
RAM), 4 cores of i7 (out of 4 cores / 8 threads) and resides on SSD
(SATA one). Build WITH cross-toolchain AND world toolchain is about 2.5
hours and build WITHOUT ANY toolchain (using host's one as
cross-compiler) is less than a hour.

Unfortunately, you aren't actually measuring the impact of Sean's #3 —
which proposes eliminating cross-compilers from Clang *when building
Clang is actually desired or required.* It isn't clear that the
additional cross-compiler backends add significant overhead to the
total Clang build time, which includes a lot of shared code, like C
and C++ parsing, generic optimization code, etc.

Ryan's just asking for someone to measure that to justify the change.

Best,
Conrad

[Brooks Davis]

Post by Sean Bruno
3. If the boostrap toolchain needs to be built in the normal case, only
target the ARCH being requested. I understand that we "want" a CC
installed that targets all architectures and this is something I agree with.

The LLVM backends are a tiny part of the LLVM build both in terms
of number of files and compile complexity. Removing them would
require quite a bit of work (and ongoing maintenance) for a negliable
improvement.

--- Brooks

[Sean Bruno]

Post by Brooks Davis

Post by Sean Bruno
3. If the boostrap toolchain needs to be built in the normal case, only
target the ARCH being requested. I understand that we "want" a CC
installed that targets all architectures and this is something I agree with.

The LLVM backends are a tiny part of the LLVM build both in terms
of number of files and compile complexity. Removing them would
require quite a bit of work (and ongoing maintenance) for a negliable
improvement.
--- Brooks

Brooks:

Can you educate me on why its so hard to maintain this part of our
tools? I'm ignorant here and haven't looked to deeply into the abyss
whereas you have been swimming in the darkness.

sean

[Brooks Davis]

Post by Sean Bruno

Post by Brooks Davis

Post by Sean Bruno
3. If the boostrap toolchain needs to be built in the normal case, only
target the ARCH being requested. I understand that we "want" a CC
installed that targets all architectures and this is something I agree with.

The LLVM backends are a tiny part of the LLVM build both in terms
of number of files and compile complexity. Removing them would
require quite a bit of work (and ongoing maintenance) for a negliable
improvement.

Can you educate me on why its so hard to maintain this part of our
tools? I'm ignorant here and haven't looked to deeply into the abyss
whereas you have been swimming in the darkness.

Because upstream makes absolutely no provision for this. In our case we
do maintain the build infrastructure which would help a bit (since we
wouldn't be maintaining diffs to CMakeFiles), but it won't help at all
with the fact that any code can assume that all backends are there and
the constants associated with there are defined. I'm not sure how big
that part is, but we'd certainly have some divergence to maintain. IIRC
the backends are <5% of LLVM compile time.

IMO, the best way to avoid building LLVM as a bootstrap tool is to use
xtoolchain ports. For i386 and amd64 I mostly use
CROSS_TOOLCHAIN=llvm60 (having installed xtoolchain-llvm60). I think
there is still work to be done to make all of this more friendly (e.g.
I'd love an xtoolchain-universe12 metaport and a simple way to use it.)

-- Brooks

[Alexander Richardson]

Post by Brooks Davis

Post by Sean Bruno

Post by Brooks Davis

Post by Sean Bruno
3. If the boostrap toolchain needs to be built in the normal case, only
target the ARCH being requested. I understand that we "want" a CC
installed that targets all architectures and this is something I agree with.

The LLVM backends are a tiny part of the LLVM build both in terms
of number of files and compile complexity. Removing them would
require quite a bit of work (and ongoing maintenance) for a negliable
improvement.

Can you educate me on why its so hard to maintain this part of our
tools? I'm ignorant here and haven't looked to deeply into the abyss
whereas you have been swimming in the darkness.

Because upstream makes absolutely no provision for this. In our case we
do maintain the build infrastructure which would help a bit (since we
wouldn't be maintaining diffs to CMakeFiles), but it won't help at all
with the fact that any code can assume that all backends are there and
the constants associated with there are defined. I'm not sure how big
that part is, but we'd certainly have some divergence to maintain. IIRC
the backends are <5% of LLVM compile time.

If you build from the upstream CMakeLists you can set
-DLLVM_TARGETS_TO_BUILD=X86 (I believe =host should also work) and
then compare that to the time it takes when building with
-DLLVM_TARGETS_TO_BUILD=all. I don't think it will save very much time
compared to the total build duration since you will still need to
build quite a few files from lib/Target (especially for x86).

~/cheri/llvm(master * u=)> find lib/Target -name "*.cpp" | wc -l
723
~/cheri/llvm(master * u=)> find . -type d -name "test" -prune -o -name
"*.cpp" | wc -l
3147

Just based on this it would seem like in the best case you *might* be
able to reduce LLVM compile time by < 20%. However, depending on the
target you will have to build about 200+ files in lib/Target as well
and at least to me it seems like the .cpp in clang take a lot longer
to build than in LLVM.

My guess is that omitting the cross toolchain could give you maybe
5-10% reduction in LLVM compile time but I haven't measured it.

Alex

[Bryan Drewery]

Post by Sean Bruno
Before I dive into the mk files of a buildworld, I'd like to describe
"what I want" so as to start a discussion of my goal.
1. If I select no toolchain (WITHOUT_TOOLCHAIN), but clang needs to be
built, only build a toolchain that targets the ARCH being requested.
2. If I select no toolchain (WITHOUT_TOOLCHAIN), but clang needs to be
built, give me a knob to turn that aborts the build with a meaningful
message that lets me know I need to update the toolchain on my buildbox.

https://reviews.freebsd.org/D11077 has this but I suspect it's wildly
stale already.

Also having a knob like this would ultimately lead to someone making
their /usr/bin/cc less useful for optimizations like WITH_SYSTEM_COMPILER.

Post by Sean Bruno
3. If the boostrap toolchain needs to be built in the normal case, only
target the ARCH being requested. I understand that we "want" a CC
installed that targets all architectures and this is something I agree with.

Hm yes there is no real reason to have multi-arch support in the
WORLDTMP compiler. Though I do have a pending patch to build clang
*once* for universe that relies on this full-arch-support behavior but
I'm sure it's trivial to continue using it for that piece.

[Bryan Drewery]

Post by Bryan Drewery

Post by Sean Bruno
Before I dive into the mk files of a buildworld, I'd like to describe
"what I want" so as to start a discussion of my goal.
1. If I select no toolchain (WITHOUT_TOOLCHAIN), but clang needs to be
built, only build a toolchain that targets the ARCH being requested.
2. If I select no toolchain (WITHOUT_TOOLCHAIN), but clang needs to be
built, give me a knob to turn that aborts the build with a meaningful
message that lets me know I need to update the toolchain on my buildbox.

https://reviews.freebsd.org/D11077 has this but I suspect it's wildly
stale already.
Also having a knob like this would ultimately lead to someone making
their /usr/bin/cc less useful for optimizations like WITH_SYSTEM_COMPILER.

Post by Sean Bruno
3. If the boostrap toolchain needs to be built in the normal case, only
target the ARCH being requested. I understand that we "want" a CC
installed that targets all architectures and this is something I agree with.

Hm yes there is no real reason to have multi-arch support in the
WORLDTMP compiler. Though I do have a pending patch to build clang
*once* for universe that relies on this full-arch-support behavior but
I'm sure it's trivial to continue using it for that piece.

For the archives, a lot of this is effectively done now. See
https://lists.freebsd.org/pipermail/freebsd-current/2018-June/069994.html

[Sean Bruno]

Post by Bryan Drewery

Post by Bryan Drewery

Post by Sean Bruno
Before I dive into the mk files of a buildworld, I'd like to describe
"what I want" so as to start a discussion of my goal.
1. If I select no toolchain (WITHOUT_TOOLCHAIN), but clang needs to be
built, only build a toolchain that targets the ARCH being requested.
2. If I select no toolchain (WITHOUT_TOOLCHAIN), but clang needs to be
built, give me a knob to turn that aborts the build with a meaningful
message that lets me know I need to update the toolchain on my buildbox.

https://reviews.freebsd.org/D11077 has this but I suspect it's wildly
stale already.
Also having a knob like this would ultimately lead to someone making
their /usr/bin/cc less useful for optimizations like WITH_SYSTEM_COMPILER.

Post by Sean Bruno
3. If the boostrap toolchain needs to be built in the normal case, only
target the ARCH being requested. I understand that we "want" a CC
installed that targets all architectures and this is something I agree with.

Hm yes there is no real reason to have multi-arch support in the
WORLDTMP compiler. Though I do have a pending patch to build clang
*once* for universe that relies on this full-arch-support behavior but
I'm sure it's trivial to continue using it for that piece.

For the archives, a lot of this is effectively done now. See
https://lists.freebsd.org/pipermail/freebsd-current/2018-June/069994.html

Good lord Bryan. Thank you.

sean