Discussion:
Making C++11 a hard requirement for FreeBSD
Warner Losh
2017-10-05 23:28:44 UTC
Permalink
I'd like to start a conversation about the viability of making C++11 a hard
requirement for bootstrapping FreeBSD and setting a specific deadline for
doing so.

This discussion is motivated by an ask from the jemalloc folks to use a
limited subset of C++11 inside of malloc in such a way that is C safe (so
the C programs wouldn't bloat with a C++ runtime). That's an ongoing
discussion in another forum, and isn't appropriate for this thread because
this has become a frequent request (but if you have opinions, please find
the thread in current@ about it). I don't know the timeline of their plans
to do this.

I'd like to take the rather vague plans we've had "before 12" and create a
timeline for removal of gcc 4.2 coupled with a requirement for support in
clang or a working external toolchain. This requirement would be coupled
with the requirement that the external toolchain support C++11 constructs.

I'd like to propose we do this 12/31/17. Any architectures that can't meet
this timeline will be disconnected from universe at that time and deleted
3/31/18.

It's my belief that i386, amd64, arm, aarch64, powerpc and powerpc64 are
ready for this change and mips* would be ready for this change with an
external clang toolchain. I'm unsure of riscv and sparc64, but suspect that
a newer version of gcc as an external toolchain could work.

Comments?

Warner
Brooks Davis
2017-10-05 23:41:49 UTC
Permalink
Post by Warner Losh
I'd like to start a conversation about the viability of making C++11 a hard
requirement for bootstrapping FreeBSD and setting a specific deadline for
doing so.
This discussion is motivated by an ask from the jemalloc folks to use a
limited subset of C++11 inside of malloc in such a way that is C safe (so
the C programs wouldn't bloat with a C++ runtime). That's an ongoing
discussion in another forum, and isn't appropriate for this thread because
this has become a frequent request (but if you have opinions, please find
to do this.
I'd like to take the rather vague plans we've had "before 12" and create a
timeline for removal of gcc 4.2 coupled with a requirement for support in
clang or a working external toolchain. This requirement would be coupled
with the requirement that the external toolchain support C++11 constructs.
I'd like to propose we do this 12/31/17. Any architectures that can't meet
this timeline will be disconnected from universe at that time and deleted
3/31/18.
This deadline seems viable to me.
Post by Warner Losh
It's my belief that i386, amd64, arm, aarch64, powerpc and powerpc64 are
ready for this change and mips* would be ready for this change with an
external clang toolchain. I'm unsure of riscv and sparc64, but suspect that
a newer version of gcc as an external toolchain could work.
mips64 should be good to go with external clang and lld in LLVM 6.0
and the llvm-devel port should be ready before then (We need to get
multi-got support landed upstream and then Alex has a fix for the
remaining lld issues). As it is, I run clang built mips64 binaries
daily.

I'm certain the riscv supporting GCC supports C++11, but we might need
to do some testing and tweak some make bits.

If someone wants to test sparc64, that's fine, but with Oracle laying
off the SPARC it's arguably more dead than IA64 was when we removed it
(Intel shipped the last design *this* May).

-- Brooks
A. Wilcox
2017-10-06 00:12:28 UTC
Permalink
Post by Brooks Davis
Post by Warner Losh
It's my belief that i386, amd64, arm, aarch64, powerpc and powerpc64 are
ready for this change and mips* would be ready for this change with an
external clang toolchain. I'm unsure of riscv and sparc64, but suspect that
a newer version of gcc as an external toolchain could work.
If someone wants to test sparc64, that's fine, but with Oracle laying
off the SPARC it's arguably more dead than IA64 was when we removed it
(Intel shipped the last design *this* May).
-- Brooks
Thinking out loud:

That doesn't change the fact that sparc64 still exists, and with Oracle
laying off Solaris as well, FreeBSD becomes a "way out" for people
heavily invested (DC full of sparc64 gear, or such). That could make
the sparc64 port not only more widely used, but more widely tested, with
more potential developers.

Or maybe not.

I can't see the future :)

--arw
--
A. Wilcox (awilfox)
Open-source programmer (C, C++, Python)
https://code.foxkit.us/u/awilfox/
Mark Linimon
2017-10-07 17:41:24 UTC
Permalink
Post by A. Wilcox
That doesn't change the fact that sparc64 still exists, and with Oracle
laying off Solaris as well, FreeBSD becomes a "way out" for people
heavily invested (DC full of sparc64 gear, or such).
I have thought for some time that we've been a "way out" for Solaris
sites wanting to keep ZFS and not deal with licensing issues, and have
worked to keep sparc64 alive. (AFAIK FreeBSD is the only open source
sparc64/zfs solution?)

But here's the current problem.

All gccs > 4.9 fail to build. Looking at the logs AFAICT the failure
is a floating-point exception as soon as the first built binary is run
during the internal testing.

Neither Marcel nor Gerald nor I have any insight on how to fix this.
Gerald does state that those gccs build on other OSes, so this is almost
certainly a FreBSD problem.

The default ports compiler has recently moved to gcc5 and then again
to gcc6. The only reason gcc49 still exists in the Ports Collection is
specifically for sparc64 ports.

Recent llvms do not build. I have no insight into that failure, either.

So, the long and short is, even with using gcc4.2.1 as an external
compiler, over time, fewer and fewer ports build as they adapt to the
newer compilers.

This is something I don't have the cycles to fix. Unless someone else
can step up and fix the compilers, we're close to the end of feasibility.

In the meantime, I'll keep running package builds with gcc4.9 as long as
it produces some kind of useful results.

I'll be happy to discuss the build status of individual ports, but let's
have that on sparc64@ rather than arch@, please.

mcl
K. Macy
2017-10-07 19:06:29 UTC
Permalink
Post by Mark Linimon
Post by A. Wilcox
That doesn't change the fact that sparc64 still exists, and with Oracle
laying off Solaris as well, FreeBSD becomes a "way out" for people
heavily invested (DC full of sparc64 gear, or such).
I have thought for some time that we've been a "way out" for Solaris
sites wanting to keep ZFS and not deal with licensing issues, and have
worked to keep sparc64 alive. (AFAIK FreeBSD is the only open source
sparc64/zfs solution?)
But here's the current problem.
All gccs > 4.9 fail to build. Looking at the logs AFAICT the failure
is a floating-point exception as soon as the first built binary is run
during the internal testing.
Neither Marcel nor Gerald nor I have any insight on how to fix this.
Gerald does state that those gccs build on other OSes, so this is almost
certainly a FreBSD problem.
The default ports compiler has recently moved to gcc5 and then again
to gcc6. The only reason gcc49 still exists in the Ports Collection is
specifically for sparc64 ports.
Recent llvms do not build. I have no insight into that failure, either.
So, the long and short is, even with using gcc4.2.1 as an external
compiler, over time, fewer and fewer ports build as they adapt to the
newer compilers.
This is something I don't have the cycles to fix. Unless someone else
can step up and fix the compilers, we're close to the end of feasibility.
In the meantime, I'll keep running package builds with gcc4.9 as long as
it produces some kind of useful results.
My recollection of sparc64 from sun4v work was that unsupported operations
would trap in to the kernel which would in turn trap in to a user space
handler for floating point emulation. If someone wants to fix it that’s
where to look. I think that FreeBSD needs to always have one big-endian
arch and one arch that requires IOMMU. Bonus points if it fulfills both.
For a time that was sparc64. These days other arches meet that need. And at
this point the most recent hardware supported by the sparc64 port shipped
in ~2003. One could amortize the cost of a low end 2017 server in just the
power bill within a year. I don’t know how much work continuing to maintain
sparc64 really adds to non sparc64 enthusiasts. Nonetheless, it is non-zero.



-M
Mark Linimon
2017-10-07 19:20:24 UTC
Permalink
I think that FreeBSD needs to always have one big-endian arch
IMHO it keeps things honest. fwiw, if you fix a port on sparc64 it
will usually fix it on powerpc64 and vice versa (~80% correlation).

But powerpc64 has a (hardware) future and sparc64 doesn't. I run both
at home, but not the powerpc64 continuously. (Actually first typed
"4U" it as "$U". Same idea.)

So, I'm willing to help keep it going (and even loan a machine to the
effort), but I am overcommitted in other areas already.

mcl
Peter Jeremy
2017-10-09 06:49:31 UTC
Permalink
Post by K. Macy
Post by Mark Linimon
Post by A. Wilcox
That doesn't change the fact that sparc64 still exists, and with Oracle
laying off Solaris as well, FreeBSD becomes a "way out" for people
heavily invested (DC full of sparc64 gear, or such).
I have thought for some time that we've been a "way out" for Solaris
sites wanting to keep ZFS and not deal with licensing issues, and have
worked to keep sparc64 alive. (AFAIK FreeBSD is the only open source
sparc64/zfs solution?)
AFAIK Illumos still supports sparc64 and is probably an easier
migration for Solaris sites so I don't think that argument holds.

Also, we run into the same situation we had with Alpha - a basically dead
architecture that only runs on old equipment. Unless there's a critical
mass of FreeBSD developers that are willing to keep it running, we're better
off killing it quickly, rather than letting it soak up developer effort.
Post by K. Macy
My recollection of sparc64 from sun4v work was that unsupported operations
would trap in to the kernel which would in turn trap in to a user space
handler for floating point emulation.
Yes. I did some poking at that some time ago. The userland package is
basically a complete single/double/quad precision IEEE FP implementation
(see /usr/src/lib/libc/sparc64/fpu). I have a test suite for it but it
hasn't been committed and I'd need to check if it's developed any bitrot.
--
Peter Jeremy
Bruce Evans
2017-10-09 07:57:26 UTC
Permalink
Post by Peter Jeremy
Post by K. Macy
My recollection of sparc64 from sun4v work was that unsupported operations
would trap in to the kernel which would in turn trap in to a user space
handler for floating point emulation.
Yes. I did some poking at that some time ago. The userland package is
basically a complete single/double/quad precision IEEE FP implementation
(see /usr/src/lib/libc/sparc64/fpu). I have a test suite for it but it
hasn't been committed and I'd need to check if it's developed any bitrot.
No, the trap method is almost never used by default, and should never be
used since it is so slow.

sparc64 on the 1 sparc64 system that used to be in the FreeBSD cluster
uses soft-float for long doubles by default (this is the gcc default)
since "hardware" (actually usuallly or always emulated by trap handlers)
for long doubles is so slow. Something like 6000 times slower for
"hard" (trapping) long doubles on old sparc64 vs not so old x86 (with
x86 clock speed about 6 times higher, and better pipelining).
Soft-float for long doubles is only about 4000 times slower. Real
hard-float for doubles and floats is only about 20 times slower (much
the same as for integers).

The only advantage of "hard" float on sparc64 is that it is easier to
debug, provided the bug is not in the trap handlers when it is harder
to debug.

Soft-float has more chance of working on sparc64 since it is needed in
some cases. The flag for the case where it is needed is -msoft-quad-float.
On x86, clang is too broken to even refuse to support -msoft-float -- clang
silently ignores this flag, so this flag non longer works in kern.mk where
it is supposed to stop the compiler using an FPU in the kernel. gcc-4.2.1
only has this bug on amd64. The i387 happens not to be used anyway in code
without float variables. SSE is more generally useful so the -mno-sse*
flags to prevent using it are more needed.

Bruce
Marius Strobl
2017-10-10 21:14:28 UTC
Permalink
Post by Mark Linimon
All gccs > 4.9 fail to build. Looking at the logs AFAICT the failure
is a floating-point exception as soon as the first built binary is run
during the internal testing.
The most plausible cause for that is executables and/or dynamic libraries
not installing the user trap handlers as specified by the libc 64 psABI,
i. e. not call __sparc_utrap_setup(). Do the ports GCCs use their own CRT
nowadays? Do they no longer link libc last? Please provide their linker
invocation. Also, please provide the backtrace of a minimal program
exhibiting that problem.

Marius
Ngie Cooper (yaneurabeya)
2017-10-23 05:47:03 UTC
Permalink
Post by Marius Strobl
Post by Mark Linimon
All gccs > 4.9 fail to build. Looking at the logs AFAICT the failure
is a floating-point exception as soon as the first built binary is run
during the internal testing.
The most plausible cause for that is executables and/or dynamic libraries
not installing the user trap handlers as specified by the libc 64 psABI,
i. e. not call __sparc_utrap_setup(). Do the ports GCCs use their own CRT
nowadays? Do they no longer link libc last? Please provide their linker
invocation. Also, please provide the backtrace of a minimal program
exhibiting that problem.
An idea occurred to me (after having dealt with building things over, and over, and over, this weekend): since we can’t rely on the ABI on ^/head to be stable, why don’t we produce working dynamic/static toolchains on HEAD-1 in ports, then require them for the areas that can’t bootstrap (yet, or at all?) with clang? We’ve already done that with some of our code that’s been deorbited from base (like rsh, etc). I don’t see why making a toolchain based on a stable ABI for architectures that will migrate or will be killed off needs to be a huge undertaking (politically), and needs to hold us back from making progress using a compiler that implements an almost 7 year old C++ spec.
Thanks,
-Ngie
Ngie Cooper (yaneurabeya)
2017-10-23 05:48:28 UTC
Permalink
Post by Ngie Cooper (yaneurabeya)
Post by Marius Strobl
Post by Mark Linimon
All gccs > 4.9 fail to build. Looking at the logs AFAICT the failure
is a floating-point exception as soon as the first built binary is run
during the internal testing.
The most plausible cause for that is executables and/or dynamic libraries
not installing the user trap handlers as specified by the libc 64 psABI,
i. e. not call __sparc_utrap_setup(). Do the ports GCCs use their own CRT
nowadays? Do they no longer link libc last? Please provide their linker
invocation. Also, please provide the backtrace of a minimal program
exhibiting that problem.
An idea occurred to me (after having dealt with building things over, and over, and over, this weekend): since we can’t rely on the ABI on ^/head to be stable, why don’t we produce working dynamic/static toolchains on HEAD-1 in ports, then require them for the areas that can’t bootstrap (yet, or at all?) with clang? We’ve already done that with some of our code that’s been deorbited from base (like rsh, etc). I don’t see why making a toolchain based on a stable ABI for architectures that will migrate or will be killed off needs to be a huge undertaking (politically), and needs to hold us back from making progress using a compiler that implements an almost 7 year old C++ spec.

 and yes, this can be interpreted as “I will do it as long as people don’t bikeshed me to death on the idea”.
-Ngie
Nathan Whitehorn
2017-10-23 15:55:54 UTC
Permalink
Post by Marius Strobl
Post by Mark Linimon
All gccs > 4.9 fail to build. Looking at the logs AFAICT the failure
is a floating-point exception as soon as the first built binary is run
during the internal testing.
The most plausible cause for that is executables and/or dynamic libraries
not installing the user trap handlers as specified by the libc 64 psABI,
i. e. not call __sparc_utrap_setup(). Do the ports GCCs use their own CRT
nowadays? Do they no longer link libc last? Please provide their linker
invocation. Also, please provide the backtrace of a minimal program
exhibiting that problem.
An idea occurred to me (after having dealt with building things over, and over, and over, this weekend): since we can’t rely on the ABI on ^/head to be stable, why don’t we produce working dynamic/static toolchains on HEAD-1 in ports, then require them for the areas that can’t bootstrap (yet, or at all?) with clang? We’ve already done that with some of our code that’s been deorbited from base (like rsh, etc). I don’t see why making a toolchain based on a stable ABI for architectures that will migrate or will be killed off needs to be a huge undertaking (politically), and needs to hold us back from making progress using a compiler that implements an almost 7 year old C++ spec.
… and yes, this can be interpreted as “I will do it as long as people don’t bikeshed me to death on the idea”.
-Ngie
I'm not quite sure what is being suggested, but the core problem with
this particular thing is that you can't build ports without a toolchain,
so there's a catch-22. The only way to break that involves packages --
which we don't tend to provide on the architectures that have the problem!

bapt has done some work on getting this to happen (the base/ ports), but
the base/gcc port is both out-of-date and doesn't seem to build anymore,
which is where I am currently stuck. Any help with that would be really
appreciated...
-Nathan
Marius Strobl
2017-10-23 20:39:52 UTC
Permalink
Post by Marius Strobl
Post by Mark Linimon
All gccs > 4.9 fail to build. Looking at the logs AFAICT the failure
is a floating-point exception as soon as the first built binary is run
during the internal testing.
The most plausible cause for that is executables and/or dynamic libraries
not installing the user trap handlers as specified by the libc 64 psABI,
i. e. not call __sparc_utrap_setup(). Do the ports GCCs use their own CRT
nowadays? Do they no longer link libc last? Please provide their linker
invocation. Also, please provide the backtrace of a minimal program
exhibiting that problem.
An idea occurred to me (after having dealt with building things over, and over, and over, this weekend): since we can?t rely on the ABI on ^/head to be stable, why don?t we produce working dynamic/static toolchains on HEAD-1 in ports, then require them for the areas that can?t bootstrap (yet, or at all?) with clang? We?ve already done that with some of our code that?s been deorbited from base (like rsh, etc). I don?t see why making a toolchain based on a stable ABI for architectures that will migrate or will be killed off needs to be a huge undertaking (politically), and needs to hold us back from making progress using a compiler that implements an almost 7 year old C++ spec.
To be honest, I've no idea what your proposal has to do with the above,
what it actually is about or why rsh(1) would be a viable example in
this case given that rsh(1) hardly is a bootstrapping tool. However,
ABI changes (the above wasn't about a change in ABI, btw.) are just
one good example why an external toolchain would be a PITA as system
compiler. Think of when we e. g. turned on TLS in the base compiler
configurations after having added TLS support to rtld(1). The next
buildworld ensured that not only the compiler used TLS, but also that
an rtld(1) capable of TLS will be in place. Now with a similar future
change and an external toolchain built on HEAD - 1, some additional
magic would be needed to ensure that binaries built for HEAD use the
expected ABI and all HEAD components are in sync.

Generally, I'm fine with moving to an external toolchain for the
system compiler, but only if it will also be the default for x86
so the life of tier-2 architectures won't become even harder - both
politically and technically - as it is now.

Marius
Ngie Cooper (yaneurabeya)
2017-10-30 08:38:15 UTC
Permalink
Post by Marius Strobl
Post by Marius Strobl
Post by Mark Linimon
All gccs > 4.9 fail to build. Looking at the logs AFAICT the failure
is a floating-point exception as soon as the first built binary is run
during the internal testing.
The most plausible cause for that is executables and/or dynamic libraries
not installing the user trap handlers as specified by the libc 64 psABI,
i. e. not call __sparc_utrap_setup(). Do the ports GCCs use their own CRT
nowadays? Do they no longer link libc last? Please provide their linker
invocation. Also, please provide the backtrace of a minimal program
exhibiting that problem.
An idea occurred to me (after having dealt with building things over, and over, and over, this weekend): since we can?t rely on the ABI on ^/head to be stable, why don?t we produce working dynamic/static toolchains on HEAD-1 in ports, then require them for the areas that can?t bootstrap (yet, or at all?) with clang? We?ve already done that with some of our code that?s been deorbited from base (like rsh, etc). I don?t see why making a toolchain based on a stable ABI for architectures that will migrate or will be killed off needs to be a huge undertaking (politically), and needs to hold us back from making progress using a compiler that implements an almost 7 year old C++ spec.
To be honest, I've no idea what your proposal has to do with the above,
what it actually is about or why rsh(1) would be a viable example in
this case given that rsh(1) hardly is a bootstrapping tool. However,
ABI changes (the above wasn't about a change in ABI, btw.) are just
one good example why an external toolchain would be a PITA as system
compiler. Think of when we e. g. turned on TLS in the base compiler
configurations after having added TLS support to rtld(1). The next
buildworld ensured that not only the compiler used TLS, but also that
an rtld(1) capable of TLS will be in place. Now with a similar future
change and an external toolchain built on HEAD - 1, some additional
magic would be needed to ensure that binaries built for HEAD use the
expected ABI and all HEAD components are in sync.
Generally, I'm fine with moving to an external toolchain for the
system compiler, but only if it will also be the default for x86
so the life of tier-2 architectures won't become even harder - both
politically and technically - as it is now.
(Replying to single-message in thread, since the rest seem to follow the same flow of expressed confusion)

Distilling down my scatterbrained message before: I was suggesting building a cross-compiler with an appropriate target/host tuple as a package, then allowing users to install that package, instead of building the toolchain from scratch every time. This is what you suggested is being worked on by ***@.

Someone should have realized that gcc 4.2.1 is a dead end at the point anyhow, and clang will need to be bootstrapped from a c++11 capable compiler (which gcc 4.2.1 is not by any means), right? Which means that making binary packages work for the toolchain is a must for 2nd tier architectures if clang is the target compiler (MK_CLANG_BOOTSTRAP==yes), which is the eventual end-goal.

Cheers,
-Ngie
John Baldwin
2017-10-05 23:47:57 UTC
Permalink
Post by Warner Losh
I'd like to start a conversation about the viability of making C++11 a hard
requirement for bootstrapping FreeBSD and setting a specific deadline for
doing so.
This discussion is motivated by an ask from the jemalloc folks to use a
limited subset of C++11 inside of malloc in such a way that is C safe (so
the C programs wouldn't bloat with a C++ runtime). That's an ongoing
discussion in another forum, and isn't appropriate for this thread because
this has become a frequent request (but if you have opinions, please find
to do this.
I'd like to take the rather vague plans we've had "before 12" and create a
timeline for removal of gcc 4.2 coupled with a requirement for support in
clang or a working external toolchain. This requirement would be coupled
with the requirement that the external toolchain support C++11 constructs.
I'd like to propose we do this 12/31/17. Any architectures that can't meet
this timeline will be disconnected from universe at that time and deleted
3/31/18.
It's my belief that i386, amd64, arm, aarch64, powerpc and powerpc64 are
ready for this change and mips* would be ready for this change with an
external clang toolchain. I'm unsure of riscv and sparc64, but suspect that
a newer version of gcc as an external toolchain could work.
In-tree clang 5.0 for MIPS mostly works (modulo some small patches). However,
it requires external ld.bfd. I know that there ld.lld can link a working
mips64 world with some patches (notably the multigot patch). mips works fine
with external GCC. riscv is already using external GCC that is C++11-capable.

The problem with external GCC is that you can cross-build a world + kernel
just fine and get a working system via CROSS_TOOLCHAIN=foo-gcc. However,
that system has no viable '/usr/bin/cc' once GCC 4.2 is removed. bapt@
started on ports to cross-build binutils and gcc packages via the base/*
ports, but those are not yet finished / fully tested. I don't think anyone
has thought about how release builds will work either with only an external
toolchain available. (I'm pretty sure sparc64 has booted fine with
external GCC, it's just in the same boat as mips with regard to /usr/bin/cc.)

Also, if you svn rm contrib/gcc you will nuke all of our systems because we
still use 'crtstuff.c' from there on all architectures for part of the C
startup code. emaste@ has looked at a replacement for that from NetBSD in
the past but I'm not sure what state that is in currently.

Another concern is fully replacing the compiler support libraries (libgcc and
friends). Some of those also come from contrib/gcc. For mips I have some
patches in review upstream to add mips to LLVM's libunwind (which allows mips
to use that for libgcc unwinding). I think sparc64 might be the only other
architecture not using llvm libunwind. (Fixing that is a much smaller lift
than fixing clang on sparc64 btw, and I've successfully used llvm libunwind
on mips worlds that are fully compiled with external GCC.)

That said, I definitely support the goal of requiring C++11. I will happily
start using it myself in some userland bits (truss for example could benefit
from std::unordered_map<>) once it is available across the board.

12/31/17 might be a bit aggressive given the holidays at the end of the
quarter, but we can start with that and revisit if need be.
--
John Baldwin
Baptiste Daroussin
2017-10-06 07:20:10 UTC
Permalink
Post by John Baldwin
Post by Warner Losh
I'd like to start a conversation about the viability of making C++11 a hard
requirement for bootstrapping FreeBSD and setting a specific deadline for
doing so.
This discussion is motivated by an ask from the jemalloc folks to use a
limited subset of C++11 inside of malloc in such a way that is C safe (so
the C programs wouldn't bloat with a C++ runtime). That's an ongoing
discussion in another forum, and isn't appropriate for this thread because
this has become a frequent request (but if you have opinions, please find
to do this.
I'd like to take the rather vague plans we've had "before 12" and create a
timeline for removal of gcc 4.2 coupled with a requirement for support in
clang or a working external toolchain. This requirement would be coupled
with the requirement that the external toolchain support C++11 constructs.
I'd like to propose we do this 12/31/17. Any architectures that can't meet
this timeline will be disconnected from universe at that time and deleted
3/31/18.
It's my belief that i386, amd64, arm, aarch64, powerpc and powerpc64 are
ready for this change and mips* would be ready for this change with an
external clang toolchain. I'm unsure of riscv and sparc64, but suspect that
a newer version of gcc as an external toolchain could work.
In-tree clang 5.0 for MIPS mostly works (modulo some small patches). However,
it requires external ld.bfd. I know that there ld.lld can link a working
mips64 world with some patches (notably the multigot patch). mips works fine
with external GCC. riscv is already using external GCC that is C++11-capable.
The problem with external GCC is that you can cross-build a world + kernel
just fine and get a working system via CROSS_TOOLCHAIN=foo-gcc. However,
started on ports to cross-build binutils and gcc packages via the base/*
ports, but those are not yet finished / fully tested. I don't think anyone
has thought about how release builds will work either with only an external
toolchain available. (I'm pretty sure sparc64 has booted fine with
external GCC, it's just in the same boat as mips with regard to /usr/bin/cc.)
Actually I did test those and they were working (tested in qemu) they were
working fine. I have given up working on them due to the lack of interested by
the community (by interest I mean people really testing, working on it, not just
saying "hey nice sounds cool").

As for the boot when I initially worked on external toolchain sparc64 was my
guinea pig and so yes it worked an booted just fine.
Post by John Baldwin
Also, if you svn rm contrib/gcc you will nuke all of our systems because we
still use 'crtstuff.c' from there on all architectures for part of the C
the past but I'm not sure what state that is in currently.
Another concern is fully replacing the compiler support libraries (libgcc and
friends). Some of those also come from contrib/gcc. For mips I have some
patches in review upstream to add mips to LLVM's libunwind (which allows mips
to use that for libgcc unwinding). I think sparc64 might be the only other
architecture not using llvm libunwind. (Fixing that is a much smaller lift
than fixing clang on sparc64 btw, and I've successfully used llvm libunwind
on mips worlds that are fully compiled with external GCC.)
That said, I definitely support the goal of requiring C++11. I will happily
start using it myself in some userland bits (truss for example could benefit
from std::unordered_map<>) once it is available across the board.
12/31/17 might be a bit aggressive given the holidays at the end of the
quarter, but we can start with that and revisit if need be.
I'm fine with that date.

Best regards,
Bapt
Poul-Henning Kamp
2017-10-06 16:47:48 UTC
Permalink
If we allow C++ in libc, it should not merely be for the convenience
of a few programmers, but because we have a vision for how it that
makes the world, or at least FreeBSD, a better place.

Having C++ in libc is no trivial detail, there is a number of areas
where this causes bootstrapping issues and conflicts.

We can solve those issues with unsightly local hacks, most
notably a bogo-malloc to malloc while C++ constructs jemalloc.

But hand on heart, we all know that is a bad idea, all of us have
been down that road before, and we also know that there is no way
to be a little bit pregnant.

The other way, the right way, to accomodate the jemalloc request
is to go all in.

Nothing in the ISO verbiage says that you cannot have C and C++
runtimes in the same library, as long as your linker knows the zip
code of it.

Libc as a combined C and C++ runtime can be implemented a lot cleaner
than a libc which hides C++ components in the closet.

So that is my input to this question:

Either we tell the jemalloc people "sorry, it's called libc for a
reason" or we decide to make our libc a native C *and* C++ runtime.

I see no sane or even possible "middle ground" or compromise position.

Poul-Henning
--
Poul-Henning Kamp | UNIX since Zilog Zeus 3.20
***@FreeBSD.ORG | TCP/IP since RFC 956
FreeBSD committer | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
Adrian Chadd
2017-10-06 18:40:59 UTC
Permalink
Post by Poul-Henning Kamp
If we allow C++ in libc, it should not merely be for the convenience
of a few programmers, but because we have a vision for how it that
makes the world, or at least FreeBSD, a better place.
Having C++ in libc is no trivial detail, there is a number of areas
where this causes bootstrapping issues and conflicts.
We can solve those issues with unsightly local hacks, most
notably a bogo-malloc to malloc while C++ constructs jemalloc.
But hand on heart, we all know that is a bad idea, all of us have
been down that road before, and we also know that there is no way
to be a little bit pregnant.
The other way, the right way, to accomodate the jemalloc request
is to go all in.
Nothing in the ISO verbiage says that you cannot have C and C++
runtimes in the same library, as long as your linker knows the zip
code of it.
Libc as a combined C and C++ runtime can be implemented a lot cleaner
than a libc which hides C++ components in the closet.
Either we tell the jemalloc people "sorry, it's called libc for a
reason" or we decide to make our libc a native C *and* C++ runtime.
I see no sane or even possible "middle ground" or compromise position.
I don't mind it as long as it's "no C++ runtime bloat please". But
yes, I also feel the pain of where you start that path and then
suddenly you find you're al in that path.

(I face this at work right now on linux platforms because a "little
C++" becomes .. not little.)




-adrian
John Baldwin
2017-10-06 23:21:05 UTC
Permalink
Post by Poul-Henning Kamp
If we allow C++ in libc, it should not merely be for the convenience
of a few programmers, but because we have a vision for how it that
makes the world, or at least FreeBSD, a better place.
Having C++ in libc is no trivial detail, there is a number of areas
where this causes bootstrapping issues and conflicts.
We can solve those issues with unsightly local hacks, most
notably a bogo-malloc to malloc while C++ constructs jemalloc.
But hand on heart, we all know that is a bad idea, all of us have
been down that road before, and we also know that there is no way
to be a little bit pregnant.
The other way, the right way, to accomodate the jemalloc request
is to go all in.
Nothing in the ISO verbiage says that you cannot have C and C++
runtimes in the same library, as long as your linker knows the zip
code of it.
Libc as a combined C and C++ runtime can be implemented a lot cleaner
than a libc which hides C++ components in the closet.
Either we tell the jemalloc people "sorry, it's called libc for a
reason" or we decide to make our libc a native C *and* C++ runtime.
I see no sane or even possible "middle ground" or compromise position.
Hmm, I don't quite agree. I think it's possible to use a restricted C++
(no rtti, no exceptions, no STL) such that you are only using language
features like templates or 'auto' without requiring runtime support. I
think that is the requirement we would place on the jemalloc implementation
for it to remain in libc. Right now the C++ runtime is split into a
couple of different pieces: libc++ (STL bits, roughly), libcxxrt (rtti
/ exception support), libgcc_s (either llvm libunwind or gcc for _Unwind_*
along with intrinsics from compiler-rt). All of these are variable in
some sense (if you wanted to build a GCC-based system you might want to
use libstdc++ instead of libc++, libgcc_s already varies by platform,
and upstream in LLVM there is already a libcxxabi alternative to libcxxrt
plus the GNU libsupc++).

I think bundling any of those pieces into libc makes our system less
flexible and different from all the other UNIXy systems currently in
vogue.
--
John Baldwin
Poul-Henning Kamp
2017-10-07 00:04:38 UTC
Permalink
--------
Post by John Baldwin
Hmm, I don't quite agree. I think it's possible to use a restricted C++
(no rtti, no exceptions, no STL) such that you are only using language
features like templates or 'auto' without requiring runtime support.
That's what Bjarne used to call "C++ as a better C compiler".

If the jemalloc crew can stay inside that dotted line _and_ the C++
compilers still allow you to do so, then that could be an "not quite
pregnant yet" option.
Post by John Baldwin
[...]
Right now the C++ runtime is split into a
couple of different pieces: libc++ (STL bits, roughly), libcxxrt (rtti
/ exception support), libgcc_s (either llvm libunwind or gcc for _Unwind_*
along with intrinsics from compiler-rt).
[...]
I think bundling any of those pieces into libc makes our system less
flexible and different from all the other UNIXy systems currently in
vogue.
That goes to my point about ld: The standard doesn't say which
library file which bits of the C++ runtime have to go into, we
get to decide that if we want to, as long as we provide a ld(1)
which knows where to find things.
--
Poul-Henning Kamp | UNIX since Zilog Zeus 3.20
***@FreeBSD.ORG | TCP/IP since RFC 956
FreeBSD committer | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
Konstantin Belousov
2017-10-07 08:32:12 UTC
Permalink
Post by Poul-Henning Kamp
--------
Post by John Baldwin
Hmm, I don't quite agree. I think it's possible to use a restricted C++
(no rtti, no exceptions, no STL) such that you are only using language
features like templates or 'auto' without requiring runtime support.
That's what Bjarne used to call "C++ as a better C compiler".
If the jemalloc crew can stay inside that dotted line _and_ the C++
compilers still allow you to do so, then that could be an "not quite
pregnant yet" option.
Post by John Baldwin
[...]
Right now the C++ runtime is split into a
couple of different pieces: libc++ (STL bits, roughly), libcxxrt (rtti
/ exception support), libgcc_s (either llvm libunwind or gcc for _Unwind_*
along with intrinsics from compiler-rt).
[...]
I think bundling any of those pieces into libc makes our system less
flexible and different from all the other UNIXy systems currently in
vogue.
That goes to my point about ld: The standard doesn't say which
library file which bits of the C++ runtime have to go into, we
get to decide that if we want to, as long as we provide a ld(1)
which knows where to find things.
This is not how linkers work.

The language standard does not say that, but the platform standards do:
system ABI very much put the symbol origin in a stone. Dynamic libraries
expose symbols under versions, if we ever expose some symbol under the
specific version, we promise to do so to the end of times (or we break
the ABI promise of stability).

C++ runtime external symbols are already exported from the specific set
of libraries. There are some tricks like filter objects which could
somewhat alleviate the move, but maintaining them for third-party libs
is too much efforts to ask, for almost no visible effect until things
break.

And of course there are parts of the C++ ABI which are underspecified by
the Itanium ABI, which means that the ABI is not yet stable. Also we get
the tie for specific compiler implementation, which runtime is imported.
Since other compiler ABI is partially incompatible, but only partially,
the other compiler would be impossible to use: some symbols resolution
would come from our libc, some from the compiler libraries.
Konstantin Belousov
2017-10-07 08:24:11 UTC
Permalink
Post by John Baldwin
Post by Poul-Henning Kamp
If we allow C++ in libc, it should not merely be for the convenience
of a few programmers, but because we have a vision for how it that
makes the world, or at least FreeBSD, a better place.
Having C++ in libc is no trivial detail, there is a number of areas
where this causes bootstrapping issues and conflicts.
We can solve those issues with unsightly local hacks, most
notably a bogo-malloc to malloc while C++ constructs jemalloc.
But hand on heart, we all know that is a bad idea, all of us have
been down that road before, and we also know that there is no way
to be a little bit pregnant.
The other way, the right way, to accomodate the jemalloc request
is to go all in.
Nothing in the ISO verbiage says that you cannot have C and C++
runtimes in the same library, as long as your linker knows the zip
code of it.
Libc as a combined C and C++ runtime can be implemented a lot cleaner
than a libc which hides C++ components in the closet.
Either we tell the jemalloc people "sorry, it's called libc for a
reason" or we decide to make our libc a native C *and* C++ runtime.
I see no sane or even possible "middle ground" or compromise position.
Hmm, I don't quite agree. I think it's possible to use a restricted C++
(no rtti, no exceptions, no STL) such that you are only using language
features like templates or 'auto' without requiring runtime support. I
think that is the requirement we would place on the jemalloc implementation
for it to remain in libc.
This is a requirement not only on jemalloc, but also on the compilers.
I am much more worried about C++ compiler's runtime requirements than
the ability of the jemalloc developers to restrict used language features
to the subset which does not need some external support _at the current
compiler version_.

Seeing the route that clang took making C compiler unusable for normal
work, I am just sure that clang++ would cause a lot of troubles if we
ever try to rely on the undocumented and unpromised detail of the
current implementation.
Post by John Baldwin
Right now the C++ runtime is split into a
couple of different pieces: libc++ (STL bits, roughly), libcxxrt (rtti
/ exception support), libgcc_s (either llvm libunwind or gcc for _Unwind_*
along with intrinsics from compiler-rt). All of these are variable in
some sense (if you wanted to build a GCC-based system you might want to
use libstdc++ instead of libc++, libgcc_s already varies by platform,
and upstream in LLVM there is already a libcxxabi alternative to libcxxrt
plus the GNU libsupc++).
I think bundling any of those pieces into libc makes our system less
flexible and different from all the other UNIXy systems currently in
vogue.
This also hits the ABI stability hard, see my other reply.
Nathan Whitehorn
2017-10-09 05:01:13 UTC
Permalink
Post by Baptiste Daroussin
Post by John Baldwin
Post by Warner Losh
I'd like to start a conversation about the viability of making C++11 a hard
requirement for bootstrapping FreeBSD and setting a specific deadline for
doing so.
This discussion is motivated by an ask from the jemalloc folks to use a
limited subset of C++11 inside of malloc in such a way that is C safe (so
the C programs wouldn't bloat with a C++ runtime). That's an ongoing
discussion in another forum, and isn't appropriate for this thread because
this has become a frequent request (but if you have opinions, please find
to do this.
I'd like to take the rather vague plans we've had "before 12" and create a
timeline for removal of gcc 4.2 coupled with a requirement for support in
clang or a working external toolchain. This requirement would be coupled
with the requirement that the external toolchain support C++11 constructs.
I'd like to propose we do this 12/31/17. Any architectures that can't meet
this timeline will be disconnected from universe at that time and deleted
3/31/18.
It's my belief that i386, amd64, arm, aarch64, powerpc and powerpc64 are
ready for this change and mips* would be ready for this change with an
external clang toolchain. I'm unsure of riscv and sparc64, but suspect that
a newer version of gcc as an external toolchain could work.
In-tree clang 5.0 for MIPS mostly works (modulo some small patches). However,
it requires external ld.bfd. I know that there ld.lld can link a working
mips64 world with some patches (notably the multigot patch). mips works fine
with external GCC. riscv is already using external GCC that is C++11-capable.
The problem with external GCC is that you can cross-build a world + kernel
just fine and get a working system via CROSS_TOOLCHAIN=foo-gcc. However,
started on ports to cross-build binutils and gcc packages via the base/*
ports, but those are not yet finished / fully tested. I don't think anyone
has thought about how release builds will work either with only an external
toolchain available. (I'm pretty sure sparc64 has booted fine with
external GCC, it's just in the same boat as mips with regard to /usr/bin/cc.)
Actually I did test those and they were working (tested in qemu) they were
working fine. I have given up working on them due to the lack of interested by
the community (by interest I mean people really testing, working on it, not just
saying "hey nice sounds cool").
As for the boot when I initially worked on external toolchain sparc64 was my
guinea pig and so yes it worked an booted just fine.
So far as I know, we never solved any of the infrastructural problems
associated with this concept:
1. Providing built releases with a /usr/bin/cc
2. Coversioning base and in-ports toolchain, including ensuring commit
atomicity between toolchains and libc
3. Adding a dependency on ports for src, including out-of-tree code that
has to be fetched from external servers
4. Getting make universe to do the right thing

We really need to solve those. If we go the external toolchain route,
which it is not clear to me is the best option, #2 and #1 are quite
complex problems. I think that, frankly, a deadline in two months to
solve this set of political problems we have had for two years is
probably crazy, but maybe making the status quo unsustainable will
finally force progress.

Also, are there any directions for how to build a toolchain from the
base ports? I haven't been able to figure it out -- the README starts
with "pkg install", but that doesn't get me very far on platforms
without binary packages, which are the ones this is meant to target.
-Nathan
Warner Losh
2017-10-09 05:26:53 UTC
Permalink
Post by Nathan Whitehorn
Post by Baptiste Daroussin
Post by John Baldwin
Post by Warner Losh
I'd like to start a conversation about the viability of making C++11 a hard
requirement for bootstrapping FreeBSD and setting a specific deadline for
doing so.
This discussion is motivated by an ask from the jemalloc folks to use a
limited subset of C++11 inside of malloc in such a way that is C safe (so
the C programs wouldn't bloat with a C++ runtime). That's an ongoing
discussion in another forum, and isn't appropriate for this thread because
this has become a frequent request (but if you have opinions, please find
to do this.
I'd like to take the rather vague plans we've had "before 12" and create a
timeline for removal of gcc 4.2 coupled with a requirement for support in
clang or a working external toolchain. This requirement would be coupled
with the requirement that the external toolchain support C++11 constructs.
I'd like to propose we do this 12/31/17. Any architectures that can't meet
this timeline will be disconnected from universe at that time and deleted
3/31/18.
It's my belief that i386, amd64, arm, aarch64, powerpc and powerpc64 are
ready for this change and mips* would be ready for this change with an
external clang toolchain. I'm unsure of riscv and sparc64, but suspect that
a newer version of gcc as an external toolchain could work.
In-tree clang 5.0 for MIPS mostly works (modulo some small patches).
However,
it requires external ld.bfd. I know that there ld.lld can link a working
mips64 world with some patches (notably the multigot patch). mips works fine
with external GCC. riscv is already using external GCC that is C++11-capable.
The problem with external GCC is that you can cross-build a world + kernel
just fine and get a working system via CROSS_TOOLCHAIN=foo-gcc. However,
started on ports to cross-build binutils and gcc packages via the base/*
ports, but those are not yet finished / fully tested. I don't think anyone
has thought about how release builds will work either with only an external
toolchain available. (I'm pretty sure sparc64 has booted fine with
external GCC, it's just in the same boat as mips with regard to /usr/bin/cc.)
Actually I did test those and they were working (tested in qemu) they were
working fine. I have given up working on them due to the lack of interested by
the community (by interest I mean people really testing, working on it, not just
saying "hey nice sounds cool").
As for the boot when I initially worked on external toolchain sparc64 was my
guinea pig and so yes it worked an booted just fine.
So far as I know, we never solved any of the infrastructural problems
1. Providing built releases with a /usr/bin/cc
2. Coversioning base and in-ports toolchain, including ensuring commit
atomicity between toolchains and libc
3. Adding a dependency on ports for src, including out-of-tree code that
has to be fetched from external servers
4. Getting make universe to do the right thing
We really need to solve those. If we go the external toolchain route,
which it is not clear to me is the best option, #2 and #1 are quite complex
problems. I think that, frankly, a deadline in two months to solve this set
of political problems we have had for two years is probably crazy, but
maybe making the status quo unsustainable will finally force progress.
External toolchains have been in flight for 5 or more years now. It's time
they land. Though the requirements for them have never included
cross-threading between /usr/src and /usr/ports like you suggest above, and
those sorts of things won't be sorted by my deadlines (which are closer to
3 months). Nor, imho, should they.
Post by Nathan Whitehorn
Also, are there any directions for how to build a toolchain from the base
ports? I haven't been able to figure it out -- the README starts with "pkg
install", but that doesn't get me very far on platforms without binary
packages, which are the ones this is meant to target.
External toolchains are by definition external. Not part of the tree nor
integrated into the tree. How you get them depends on the external
toolchain and is unspecified. You get a lower level of integration than you
do with in-tree toolchains, not the same level with reach-overs into some
external repo.

Warner
Nathan Whitehorn
2017-10-09 17:47:13 UTC
Permalink
On Sun, Oct 8, 2017 at 11:01 PM, Nathan Whitehorn
I'd like to start a conversation about the viability
of making C++11 a hard
requirement for bootstrapping FreeBSD and setting a
specific deadline for
doing so.
This discussion is motivated by an ask from the
jemalloc folks to use a
limited subset of C++11 inside of malloc in such a way
that is C safe (so
the C programs wouldn't bloat with a C++ runtime).
That's an ongoing
discussion in another forum, and isn't appropriate for
this thread because
this has become a frequent request (but if you have
opinions, please find
timeline of their plans
to do this.
I'd like to take the rather vague plans we've had
"before 12" and create a
timeline for removal of gcc 4.2 coupled with a
requirement for support in
clang or a working external toolchain. This
requirement would be coupled
with the requirement that the external toolchain
support C++11 constructs.
I'd like to propose we do this 12/31/17. Any
architectures that can't meet
this timeline will be disconnected from universe at
that time and deleted
3/31/18.
It's my belief that i386, amd64, arm, aarch64, powerpc
and powerpc64 are
ready for this change and mips* would be ready for
this change with an
external clang toolchain. I'm unsure of riscv and
sparc64, but suspect that
a newer version of gcc as an external toolchain could work.
In-tree clang 5.0 for MIPS mostly works (modulo some small
patches). However,
it requires external ld.bfd. I know that there ld.lld can
link a working
mips64 world with some patches (notably the multigot
patch). mips works fine
with external GCC. riscv is already using external GCC
that is C++11-capable.
The problem with external GCC is that you can cross-build
a world + kernel
just fine and get a working system via
CROSS_TOOLCHAIN=foo-gcc. However,
that system has no viable '/usr/bin/cc' once GCC 4.2 is
started on ports to cross-build binutils and gcc packages
via the base/*
ports, but those are not yet finished / fully tested. I
don't think anyone
has thought about how release builds will work either with
only an external
toolchain available. (I'm pretty sure sparc64 has booted fine with
external GCC, it's just in the same boat as mips with
regard to /usr/bin/cc.)
Actually I did test those and they were working (tested in qemu) they were
working fine. I have given up working on them due to the lack
of interested by
the community (by interest I mean people really testing,
working on it, not just
saying "hey nice sounds cool").
As for the boot when I initially worked on external toolchain
sparc64 was my
guinea pig and so yes it worked an booted just fine.
So far as I know, we never solved any of the infrastructural
1. Providing built releases with a /usr/bin/cc
2. Coversioning base and in-ports toolchain, including ensuring
commit atomicity between toolchains and libc
3. Adding a dependency on ports for src, including out-of-tree
code that has to be fetched from external servers
4. Getting make universe to do the right thing
We really need to solve those. If we go the external toolchain
route, which it is not clear to me is the best option, #2 and #1
are quite complex problems. I think that, frankly, a deadline in
two months to solve this set of political problems we have had for
two years is probably crazy, but maybe making the status quo
unsustainable will finally force progress.
External toolchains have been in flight for 5 or more years now. It's
time they land. Though the requirements for them have never included
cross-threading between /usr/src and /usr/ports like you suggest
above, and those sorts of things won't be sorted by my deadlines
(which are closer to 3 months). Nor, imho, should they.
Well, sure. But the fact remains that we cannot build usable systems
with external toolchains right now. Those are real problems that need to
be solved somehow.

Let's focus on #1, the largest if not the only major problem. If I
build, say, a ppc64 system with an external toolchain right now, it
boots and runs fine. But the system is completely unusable since:
- There are no binary packages built for PPC64, because of project
policy preventing the use of native build systems
- You cannot cross-compile packages for PPC64, because of limitations in
QEMU
- There is no compiler installed in the base system, so you cannot
install any software from source code
- You cannot build the compiler from source, because you don't have one
to bootstrap from and can't get one pre-built because of the no-packages
problem.

We can't ship systems like this -- how is anyone expected to be able to
use them?

These are easy to fix -- for example, here are three possibilities --
but solutions been held up for *years* by project policy:
1. Allow Tier-2 packages to be built on native hardware without as much
centralized control, letting pkg install of the toolchain work. This
fixes ppc64, but maybe not embedded systems.
2. Have external-toolchain builds cross-build (or native-build) and
pre-install a standard compiler package from ports during make
buildworld, which keeps the user experience of current tier-2 and later
tier-1 systems.
3. Include a newer compiler fully in the tree or in some nearby
repository, which does the same.

If we break any of these policy impasses by the deadline, I am 100% for
dropping GCC 4.2. But we *have* to do something. Just axing code while
preventing a meaningful replacement isn't a solution.
Also, are there any directions for how to build a toolchain from
the base ports? I haven't been able to figure it out -- the README
starts with "pkg install", but that doesn't get me very far on
platforms without binary packages, which are the ones this is
meant to target.
External toolchains are by definition external. Not part of the tree
nor integrated into the tree. How you get them depends on the external
toolchain and is unspecified. You get a lower level of integration
than you do with in-tree toolchains, not the same level with
reach-overs into some external repo.
I think you missed the point. I was asking how to build the port to test
it -- it doesn't build at present and I can't figure out how to make an
external toolchain via the "base" ports (the other toolchains in ports
work fine).
-Nathan
Warner
Warner Losh
2017-10-09 18:32:17 UTC
Permalink
Post by Nathan Whitehorn
Post by Baptiste Daroussin
Post by John Baldwin
Post by Warner Losh
I'd like to start a conversation about the viability of making C++11 a hard
requirement for bootstrapping FreeBSD and setting a specific deadline for
doing so.
This discussion is motivated by an ask from the jemalloc folks to use a
limited subset of C++11 inside of malloc in such a way that is C safe (so
the C programs wouldn't bloat with a C++ runtime). That's an ongoing
discussion in another forum, and isn't appropriate for this thread because
this has become a frequent request (but if you have opinions, please find
to do this.
I'd like to take the rather vague plans we've had "before 12" and create a
timeline for removal of gcc 4.2 coupled with a requirement for support in
clang or a working external toolchain. This requirement would be coupled
with the requirement that the external toolchain support C++11 constructs.
I'd like to propose we do this 12/31/17. Any architectures that can't meet
this timeline will be disconnected from universe at that time and deleted
3/31/18.
It's my belief that i386, amd64, arm, aarch64, powerpc and powerpc64 are
ready for this change and mips* would be ready for this change with an
external clang toolchain. I'm unsure of riscv and sparc64, but suspect that
a newer version of gcc as an external toolchain could work.
In-tree clang 5.0 for MIPS mostly works (modulo some small patches).
However,
it requires external ld.bfd. I know that there ld.lld can link a working
mips64 world with some patches (notably the multigot patch). mips works fine
with external GCC. riscv is already using external GCC that is C++11-capable.
The problem with external GCC is that you can cross-build a world + kernel
just fine and get a working system via CROSS_TOOLCHAIN=foo-gcc. However,
started on ports to cross-build binutils and gcc packages via the base/*
ports, but those are not yet finished / fully tested. I don't think anyone
has thought about how release builds will work either with only an external
toolchain available. (I'm pretty sure sparc64 has booted fine with
external GCC, it's just in the same boat as mips with regard to /usr/bin/cc.)
Actually I did test those and they were working (tested in qemu) they were
working fine. I have given up working on them due to the lack of interested by
the community (by interest I mean people really testing, working on it, not just
saying "hey nice sounds cool").
As for the boot when I initially worked on external toolchain sparc64 was my
guinea pig and so yes it worked an booted just fine.
So far as I know, we never solved any of the infrastructural problems
1. Providing built releases with a /usr/bin/cc
2. Coversioning base and in-ports toolchain, including ensuring commit
atomicity between toolchains and libc
3. Adding a dependency on ports for src, including out-of-tree code that
has to be fetched from external servers
4. Getting make universe to do the right thing
We really need to solve those. If we go the external toolchain route,
which it is not clear to me is the best option, #2 and #1 are quite complex
problems. I think that, frankly, a deadline in two months to solve this set
of political problems we have had for two years is probably crazy, but
maybe making the status quo unsustainable will finally force progress.
External toolchains have been in flight for 5 or more years now. It's time
they land. Though the requirements for them have never included
cross-threading between /usr/src and /usr/ports like you suggest above, and
those sorts of things won't be sorted by my deadlines (which are closer to
3 months). Nor, imho, should they.


Well, sure. But the fact remains that we cannot build usable systems with
external toolchains right now. Those are real problems that need to be
solved somehow.


Sure we can. I've built a bootable i386 system with gcc 6. It is a solved
problem.

Let's focus on #1, the largest if not the only major problem. If I build,
say, a ppc64 system with an external toolchain right now, it boots and runs
fine. But the system is completely unusable since:
- There are no binary packages built for PPC64, because of project policy
preventing the use of native build systems


System is still usable w/o packages. People can still fire up custom
poudrier repos. We could also change project policy. This is is a specific
wrinkle for powerpc64 it seems.

- You cannot cross-compile packages for PPC64, because of limitations in
QEMU


Then we should fix those, like we did for arm and MIPS.

- There is no compiler installed in the base system, so you cannot install
any software from source code


You can install it as a package.

- You cannot build the compiler from source, because you don't have one to
bootstrap from and can't get one pre-built because of the no-packages
problem.


If you fix the other problems, this is solved.

We can't ship systems like this -- how is anyone expected to be able to use
them?


Most systems don't build software, so there are plenty of uses.

These are easy to fix -- for example, here are three possibilities -- but
solutions been held up for *years* by project policy:
1. Allow Tier-2 packages to be built on native hardware without as much
centralized control, letting pkg install of the toolchain work. This fixes
ppc64, but maybe not embedded systems.


If we can't build in QEMU then sure. It doesn't matter where the binaries
come from, so long as they work.

2. Have external-toolchain builds cross-build (or native-build) and
pre-install a standard compiler package from ports during make buildworld,
which keeps the user experience of current tier-2 and later tier-1 systems.


No need for this to be during buildworld. I see no benefit from that. You
can install it after installworld either native or to a destdir.

3. Include a newer compiler fully in the tree or in some nearby repository,
which does the same.


Yea, that's also possible. But then it isn't an external toolchain.

If we break any of these policy impasses by the deadline, I am 100% for
dropping GCC 4.2. But we *have* to do something. Just axing code while
preventing a meaningful replacement isn't a solution.



We have sometthing that works today with a few warts. We shouldn't let the
warts get in the way.

I'm happy to delay if there are specific items that have a definitive
timeline, but wanting 100% full integration from external toolchain with
full packages isn't a gating factor here.

Besides, I thought powerpc64 just worked with clang...

Warner

Also, are there any directions for how to build a toolchain from the base
Post by Nathan Whitehorn
ports? I haven't been able to figure it out -- the README starts with "pkg
install", but that doesn't get me very far on platforms without binary
packages, which are the ones this is meant to target.
External toolchains are by definition external. Not part of the tree nor
integrated into the tree. How you get them depends on the external
toolchain and is unspecified. You get a lower level of integration than you
do with in-tree toolchains, not the same level with reach-overs into some
external repo.


I think you missed the point. I was asking how to build the port to test it
-- it doesn't build at present and I can't figure out how to make an
external toolchain via the "base" ports (the other toolchains in ports work
fine).
-Nathan

Warner
Nathan Whitehorn
2017-10-10 04:57:04 UTC
Permalink
Post by Warner Losh
Post by Nathan Whitehorn
Post by Baptiste Daroussin
Post by John Baldwin
Post by Warner Losh
I'd like to start a conversation about the viability of making C++11 a hard
requirement for bootstrapping FreeBSD and setting a specific deadline for
doing so.
This discussion is motivated by an ask from the jemalloc folks to use a
limited subset of C++11 inside of malloc in such a way that is C safe (so
the C programs wouldn't bloat with a C++ runtime). That's an ongoing
discussion in another forum, and isn't appropriate for this thread because
this has become a frequent request (but if you have opinions, please find
to do this.
I'd like to take the rather vague plans we've had "before 12" and create a
timeline for removal of gcc 4.2 coupled with a requirement for support in
clang or a working external toolchain. This requirement would be coupled
with the requirement that the external toolchain support C++11 constructs.
I'd like to propose we do this 12/31/17. Any architectures that can't meet
this timeline will be disconnected from universe at that time and deleted
3/31/18.
It's my belief that i386, amd64, arm, aarch64, powerpc and powerpc64 are
ready for this change and mips* would be ready for this change with an
external clang toolchain. I'm unsure of riscv and sparc64, but suspect that
a newer version of gcc as an external toolchain could work.
In-tree clang 5.0 for MIPS mostly works (modulo some small patches).
However,
it requires external ld.bfd. I know that there ld.lld can link a working
mips64 world with some patches (notably the multigot patch). mips works fine
with external GCC. riscv is already using external GCC that is C++11-capable.
The problem with external GCC is that you can cross-build a world + kernel
just fine and get a working system via CROSS_TOOLCHAIN=foo-gcc. However,
started on ports to cross-build binutils and gcc packages via the base/*
ports, but those are not yet finished / fully tested. I don't think anyone
has thought about how release builds will work either with only an external
toolchain available. (I'm pretty sure sparc64 has booted fine with
external GCC, it's just in the same boat as mips with regard to /usr/bin/cc.)
Actually I did test those and they were working (tested in qemu) they were
working fine. I have given up working on them due to the lack of interested by
the community (by interest I mean people really testing, working on it, not just
saying "hey nice sounds cool").
As for the boot when I initially worked on external toolchain sparc64 was my
guinea pig and so yes it worked an booted just fine.
So far as I know, we never solved any of the infrastructural problems
1. Providing built releases with a /usr/bin/cc
2. Coversioning base and in-ports toolchain, including ensuring commit
atomicity between toolchains and libc
3. Adding a dependency on ports for src, including out-of-tree code that
has to be fetched from external servers
4. Getting make universe to do the right thing
We really need to solve those. If we go the external toolchain route,
which it is not clear to me is the best option, #2 and #1 are quite complex
problems. I think that, frankly, a deadline in two months to solve this set
of political problems we have had for two years is probably crazy, but
maybe making the status quo unsustainable will finally force progress.
External toolchains have been in flight for 5 or more years now. It's time
they land. Though the requirements for them have never included
cross-threading between /usr/src and /usr/ports like you suggest above, and
those sorts of things won't be sorted by my deadlines (which are closer to
3 months). Nor, imho, should they.
Well, sure. But the fact remains that we cannot build usable systems with
external toolchains right now. Those are real problems that need to be
solved somehow.
Sure we can. I've built a bootable i386 system with gcc 6. It is a solved
problem.
"Bootable" is not the same as "usable" here. External toolchain
powerpc64 *boots* fine -- and has for years -- but there isn't much you
can actual *do* with the system except admire dmesg without the ability
to install ports.
Post by Warner Losh
Let's focus on #1, the largest if not the only major problem. If I build,
say, a ppc64 system with an external toolchain right now, it boots and runs
- There are no binary packages built for PPC64, because of project policy
preventing the use of native build systems
System is still usable w/o packages. People can still fire up custom
poudrier repos. We could also change project policy. This is is a specific
wrinkle for powerpc64 it seems.
How would I do that? We can't run these natively, because the system
ships without a compiler. And we can't cross-compile, because the ports
tree does not support that.
Post by Warner Losh
- You cannot cross-compile packages for PPC64, because of limitations in
QEMU
Then we should fix those, like we did for arm and MIPS.
That's a tremendous amount of work on a non-FreeBSD codebase. Are we now
going to require QEMU patches from platform maintainers?
Post by Warner Losh
- There is no compiler installed in the base system, so you cannot install
any software from source code
You can install it as a package.
Where do you get the package?
Post by Warner Losh
- You cannot build the compiler from source, because you don't have one to
bootstrap from and can't get one pre-built because of the no-packages
problem.
If you fix the other problems, this is solved.
Yes, absolutely. But how do we solve the others?
Post by Warner Losh
We can't ship systems like this -- how is anyone expected to be able to use
them?
Most systems don't build software, so there are plenty of uses.
I think all systems *install* software. If we don't have packages, then
it has to be buildable from source and we need a compiler. We just can't
close *both* paths.
Post by Warner Losh
These are easy to fix -- for example, here are three possibilities -- but
1. Allow Tier-2 packages to be built on native hardware without as much
centralized control, letting pkg install of the toolchain work. This fixes
ppc64, but maybe not embedded systems.
If we can't build in QEMU then sure. It doesn't matter where the binaries
come from, so long as they work.
Yes, agreed. This has not been project policy thus far, however.
Post by Warner Losh
2. Have external-toolchain builds cross-build (or native-build) and
pre-install a standard compiler package from ports during make buildworld,
which keeps the user experience of current tier-2 and later tier-1 systems.
No need for this to be during buildworld. I see no benefit from that. You
can install it after installworld either native or to a destdir.
Fair enough. Read this as "included as part of the release media and,
ideally, extracted and installed by default". I do not care -- at all --
how it comes to be part of the release media/hosted by freebsd.org, so
long as it is and is updated at least as often as releases, ideally as
part of the release process.
Post by Warner Losh
3. Include a newer compiler fully in the tree or in some nearby repository,
which does the same.
Yea, that's also possible. But then it isn't an external toolchain.
Absolutely -- this is included for completeness in the more general
conversation of (finally) removing GCC 4.2.
Post by Warner Losh
If we break any of these policy impasses by the deadline, I am 100% for
dropping GCC 4.2. But we *have* to do something. Just axing code while
preventing a meaningful replacement isn't a solution.
We have sometthing that works today with a few warts. We shouldn't let the
warts get in the way.
The situation right now is pretty wart-y: there is no way to install
software, of any kind, on a number of platforms with external
toolchains, powerpc64 among them.
Post by Warner Losh
I'm happy to delay if there are specific items that have a definitive
timeline, but wanting 100% full integration from external toolchain with
full packages isn't a gating factor here.
Absolutely! I don't want to make the perfect the enemy of the good,
especially when the good is finally freeing ourselves from GCC 4.2. But
we have been blocked for years by the fact that we have not been able,
as a result of basically purely policy issues, to find a way to
bootstrap a GCC 4.2-less system so that ports can be installed. And, as
much as I like running fortune, I think not being able to install
3rd-party software is a blocker.
Post by Warner Losh
Besides, I thought powerpc64 just worked with clang...
Unfortunately, no. There are a wide variety of subtle issues. You can
boot a kernel built with -O0, but higher optimization levels or userland
code (especially C++) break.
-Nathan
Post by Warner Losh
Warner
Also, are there any directions for how to build a toolchain from the base
Post by Nathan Whitehorn
ports? I haven't been able to figure it out -- the README starts with "pkg
install", but that doesn't get me very far on platforms without binary
packages, which are the ones this is meant to target.
External toolchains are by definition external. Not part of the tree nor
integrated into the tree. How you get them depends on the external
toolchain and is unspecified. You get a lower level of integration than you
do with in-tree toolchains, not the same level with reach-overs into some
external repo.
I think you missed the point. I was asking how to build the port to test it
-- it doesn't build at present and I can't figure out how to make an
external toolchain via the "base" ports (the other toolchains in ports work
fine).
-Nathan
Warner
_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-arch
Warner Losh
2017-10-10 05:56:07 UTC
Permalink
Post by Warner Losh
Post by Nathan Whitehorn
Post by Baptiste Daroussin
Post by Warner Losh
I'd like to start a conversation about the viability of making C++11 a
Post by Warner Losh
hard
requirement for bootstrapping FreeBSD and setting a specific deadline for
doing so.
This discussion is motivated by an ask from the jemalloc folks to use a
limited subset of C++11 inside of malloc in such a way that is C safe (so
the C programs wouldn't bloat with a C++ runtime). That's an ongoing
discussion in another forum, and isn't appropriate for this thread because
this has become a frequent request (but if you have opinions, please find
to do this.
I'd like to take the rather vague plans we've had "before 12" and create a
timeline for removal of gcc 4.2 coupled with a requirement for support in
clang or a working external toolchain. This requirement would be coupled
with the requirement that the external toolchain support C++11 constructs.
I'd like to propose we do this 12/31/17. Any architectures that can't meet
this timeline will be disconnected from universe at that time and deleted
3/31/18.
It's my belief that i386, amd64, arm, aarch64, powerpc and powerpc64 are
ready for this change and mips* would be ready for this change with an
external clang toolchain. I'm unsure of riscv and sparc64, but suspect that
a newer version of gcc as an external toolchain could work.
In-tree clang 5.0 for MIPS mostly works (modulo some small patches).
However,
it requires external ld.bfd. I know that there ld.lld can link a working
mips64 world with some patches (notably the multigot patch). mips
works
fine
with external GCC. riscv is already using external GCC that is C++11-capable.
The problem with external GCC is that you can cross-build a world + kernel
just fine and get a working system via CROSS_TOOLCHAIN=foo-gcc.
However,
started on ports to cross-build binutils and gcc packages via the base/*
ports, but those are not yet finished / fully tested. I don't think anyone
has thought about how release builds will work either with only an external
toolchain available. (I'm pretty sure sparc64 has booted fine with
external GCC, it's just in the same boat as mips with regard to /usr/bin/cc.)
Actually I did test those and they were working (tested in qemu) they
were
working fine. I have given up working on them due to the lack of interested by
the community (by interest I mean people really testing, working on it, not just
saying "hey nice sounds cool").
As for the boot when I initially worked on external toolchain sparc64
was
my
guinea pig and so yes it worked an booted just fine.
So far as I know, we never solved any of the infrastructural problems
1. Providing built releases with a /usr/bin/cc
2. Coversioning base and in-ports toolchain, including ensuring commit
atomicity between toolchains and libc
3. Adding a dependency on ports for src, including out-of-tree code that
has to be fetched from external servers
4. Getting make universe to do the right thing
We really need to solve those. If we go the external toolchain route,
which it is not clear to me is the best option, #2 and #1 are quite complex
problems. I think that, frankly, a deadline in two months to solve this set
of political problems we have had for two years is probably crazy, but
maybe making the status quo unsustainable will finally force progress.
External toolchains have been in flight for 5 or more years now. It's
time
they land. Though the requirements for them have never included
cross-threading between /usr/src and /usr/ports like you suggest above, and
those sorts of things won't be sorted by my deadlines (which are closer to
3 months). Nor, imho, should they.
Well, sure. But the fact remains that we cannot build usable systems with
external toolchains right now. Those are real problems that need to be
solved somehow.
Sure we can. I've built a bootable i386 system with gcc 6. It is a solved
problem.
"Bootable" is not the same as "usable" here. External toolchain powerpc64
*boots* fine -- and has for years -- but there isn't much you can actual
*do* with the system except admire dmesg without the ability to install
ports.
Right, and at least some of the issues with powerpc64 are non-technical. If
you have binary packages, none of the things that you point out matter.
There's an external compiler that can be installed for cross building a
system, or you have a running system that you were able to build, so
necessarily have a compiler for further bootstrapping.

But doesn't clang work well enough for this on powerpc? If not, then
wouldn't we be in the same position with or without external toolchains?

Let's focus on #1, the largest if not the only major problem. If I build,
Post by Warner Losh
say, a ppc64 system with an external toolchain right now, it boots and runs
- There are no binary packages built for PPC64, because of project policy
preventing the use of native build systems
System is still usable w/o packages. People can still fire up custom
poudrier repos. We could also change project policy. This is is a specific
wrinkle for powerpc64 it seems.
How would I do that? We can't run these natively, because the system ships
without a compiler. And we can't cross-compile, because the ports tree does
not support that.
Fair point about ports not supporting cross compile. I thought it
supported, though setting CC. If you can set CC, you can natively build the
packages. If not, then we should fix that. Most ports, however, just
require 'cc' to be in the path and don't much care if it is /usr/bin/cc or
/usr/local/bin/cc.

- You cannot cross-compile packages for PPC64, because of limitations in
Post by Warner Losh
QEMU
Then we should fix those, like we did for arm and MIPS.
That's a tremendous amount of work on a non-FreeBSD codebase. Are we now
going to require QEMU patches from platform maintainers?
Require? No. But other platform maintainers have fixed qemu when it was
broken for arm and mips. Sometimes Sean Bruno fixed it, other times he got
help from the port maintainer. But he really wanted packages.

- There is no compiler installed in the base system, so you cannot install
Post by Warner Losh
any software from source code
You can install it as a package.
Where do you get the package
Well, if you have a native system, you needed one to build the system.
Cross building is more tricky, I'll grant. A carefully curated set of
natively built compilers would suffice.
- You cannot build the compiler from source, because you don't have one to
Post by Warner Losh
bootstrap from and can't get one pre-built because of the no-packages
problem.
If you fix the other problems, this is solved.
Yes, absolutely. But how do we solve the others?
We can't ship systems like this -- how is anyone expected to be able to use
Post by Warner Losh
them?
Most systems don't build software, so there are plenty of uses.
I think all systems *install* software. If we don't have packages, then it
has to be buildable from source and we need a compiler. We just can't close
*both* paths.
And the paths to get packages are solvable problems.
These are easy to fix -- for example, here are three possibilities -- but
Post by Warner Losh
1. Allow Tier-2 packages to be built on native hardware without as much
centralized control, letting pkg install of the toolchain work. This fixes
ppc64, but maybe not embedded systems.
If we can't build in QEMU then sure. It doesn't matter where the binaries
come from, so long as they work.
Yes, agreed. This has not been project policy thus far, however.
Policy changes all the time. As pointed out in other email, though, it's
more complicated and at least partially political.

2. Have external-toolchain builds cross-build (or native-build) and
Post by Warner Losh
pre-install a standard compiler package from ports during make buildworld,
which keeps the user experience of current tier-2 and later tier-1 systems.
No need for this to be during buildworld. I see no benefit from that. You
can install it after installworld either native or to a destdir.
Fair enough. Read this as "included as part of the release media and,
ideally, extracted and installed by default". I do not care -- at all --
how it comes to be part of the release media/hosted by freebsd.org, so
long as it is and is updated at least as often as releases, ideally as part
of the release process.
3. Include a newer compiler fully in the tree or in some nearby repository,
Post by Warner Losh
which does the same.
Yea, that's also possible. But then it isn't an external toolchain.
Absolutely -- this is included for completeness in the more general
conversation of (finally) removing GCC 4.2.
If we break any of these policy impasses by the deadline, I am 100% for
Post by Warner Losh
dropping GCC 4.2. But we *have* to do something. Just axing code while
preventing a meaningful replacement isn't a solution.
We have sometthing that works today with a few warts. We shouldn't let the
warts get in the way.
The situation right now is pretty wart-y: there is no way to install
software, of any kind, on a number of platforms with external toolchains,
powerpc64 among them.
I'm happy to delay if there are specific items that have a definitive
Post by Warner Losh
timeline, but wanting 100% full integration from external toolchain with
full packages isn't a gating factor here.
Absolutely! I don't want to make the perfect the enemy of the good,
especially when the good is finally freeing ourselves from GCC 4.2. But we
have been blocked for years by the fact that we have not been able, as a
result of basically purely policy issues, to find a way to bootstrap a GCC
4.2-less system so that ports can be installed. And, as much as I like
running fortune, I think not being able to install 3rd-party software is a
blocker.
Policy can be changed.

Besides, I thought powerpc64 just worked with clang...
Unfortunately, no. There are a wide variety of subtle issues. You can boot
a kernel built with -O0, but higher optimization levels or userland code
(especially C++) break.
C++ isn't needed to boostrap gcc.
Warner
Post by Warner Losh
Also, are there any directions for how to build a toolchain from the base
Post by Nathan Whitehorn
ports? I haven't been able to figure it out -- the README starts with "pkg
install", but that doesn't get me very far on platforms without binary
packages, which are the ones this is meant to target.
External toolchains are by definition external. Not part of the tree nor
integrated into the tree. How you get them depends on the external
toolchain and is unspecified. You get a lower level of integration than you
do with in-tree toolchains, not the same level with reach-overs into some
external repo.
I think you missed the point. I was asking how to build the port to test it
-- it doesn't build at present and I can't figure out how to make an
external toolchain via the "base" ports (the other toolchains in ports work
fine).
-Nathan
Warner
I guess I'm looking for an actionable set of blockers with a timetable for
resolution. These all sound a lot like inconvenient problems stemming from
some bad choices of project controlled equipment. External toolchains could
be supplied by third parties as packages. That would solve much of the
bootstrapping problem. Some time spent fixing the bugs in qemu that keep it
from being a viable option would also open up a flood of packages (we have
good infrastructure for creating mips and arm packages today I'm told would
be relatively easy to expand to new architectures).

However, absent a clear plan, as well as resourced dedicated to realize it,
I'm not sure that we should block things further based strictly on a list
of things that are broken. If there's a plan to fix, I'm willing to defer
to give that plan a chance to happen. In the absence of a specific plan
with timelines, I'm reluctant to indefinitely delay the negotiated plan for
12 which has taken years to pull together. Otherwise, is 3 more months
enough? Is 6? are 12? Do we wait for FreeBSD 13?

Warner
A. Wilcox
2017-10-10 06:13:05 UTC
Permalink
Post by Warner Losh
(we have
good infrastructure for creating mips and arm packages today I'm told would
be relatively easy to expand to new architectures).
Does this include the elusive sparc64 (if qemu were fixed, and the LLVM
issues resolved)?

Just curious.

--arw
--
A. Wilcox (awilfox)
Open-source programmer (C, C++, Python)
https://code.foxkit.us/u/awilfox/
Warner Losh
2017-10-10 13:20:25 UTC
Permalink
Post by A. Wilcox
Post by Warner Losh
(we have
good infrastructure for creating mips and arm packages today I'm told
would
Post by Warner Losh
be relatively easy to expand to new architectures).
Does this include the elusive sparc64 (if qemu were fixed, and the LLVM
issues resolved)?
If someone fixes these things, I suppose. I'm not optimistic though.

Warner
Nathan Whitehorn
2017-10-10 14:36:25 UTC
Permalink
On 10/09/17 22:56, Warner Losh wrote:

[...]
Post by Warner Losh
"Bootable" is not the same as "usable" here. External toolchain powerpc64
*boots* fine -- and has for years -- but there isn't much you can actual
*do* with the system except admire dmesg without the ability to install
ports.
Right, and at least some of the issues with powerpc64 are non-technical. If
you have binary packages, none of the things that you point out matter.
There's an external compiler that can be installed for cross building a
system, or you have a running system that you were able to build, so
necessarily have a compiler for further bootstrapping.
I think they are *all* non-technical. We just need to have some
official, blessed place to get that package from. Cross-building src/ is
quite a lot easier than cross-building packages, and people do generally
install some of our platforms (PPC, SPARC, etc.) from install media, so
it would need to be there.
Post by Warner Losh
But doesn't clang work well enough for this on powerpc? If not, then
wouldn't we be in the same position with or without external toolchains?
No. GCC works great, however, 4.2 and higher.
Post by Warner Losh
Let's focus on #1, the largest if not the only major problem. If I build,
Post by Warner Losh
say, a ppc64 system with an external toolchain right now, it boots and runs
- There are no binary packages built for PPC64, because of project policy
preventing the use of native build systems
System is still usable w/o packages. People can still fire up custom
poudrier repos. We could also change project policy. This is is a specific
wrinkle for powerpc64 it seems.
How would I do that? We can't run these natively, because the system ships
without a compiler. And we can't cross-compile, because the ports tree does
not support that.
Fair point about ports not supporting cross compile. I thought it
supported, though setting CC. If you can set CC, you can natively build the
packages. If not, then we should fix that. Most ports, however, just
require 'cc' to be in the path and don't much care if it is /usr/bin/cc or
/usr/local/bin/cc.
Not generally. A few very specific packages work this way, but not all.
You also need to bootstrap pkg somehow, which is itself a port, and
binutils, and a few other things. It's almost certainly not a
significant technical problem.
Post by Warner Losh
- You cannot cross-compile packages for PPC64, because of limitations in
Post by Warner Losh
QEMU
Then we should fix those, like we did for arm and MIPS.
That's a tremendous amount of work on a non-FreeBSD codebase. Are we now
going to require QEMU patches from platform maintainers?
Require? No. But other platform maintainers have fixed qemu when it was
broken for arm and mips. Sometimes Sean Bruno fixed it, other times he got
help from the port maintainer. But he really wanted packages.
Yes, and that turned out to be incredibly complicated for PPC. I think
there had been more heavy lifting ahead of time on Linux for ARM and
MIPS, which made it easier.
Post by Warner Losh
- There is no compiler installed in the base system, so you cannot install
Post by Warner Losh
any software from source code
You can install it as a package.
Where do you get the package
Well, if you have a native system, you needed one to build the system.
Or maybe you downloaded the install media? All I want is for the install
media to have a compiler on them or a way to get one.
Post by Warner Losh
Cross building is more tricky, I'll grant. A carefully curated set of
natively built compilers would suffice.
- You cannot build the compiler from source, because you don't have one to
Post by Warner Losh
bootstrap from and can't get one pre-built because of the no-packages
problem.
If you fix the other problems, this is solved.
Yes, absolutely. But how do we solve the others?
We can't ship systems like this -- how is anyone expected to be able to use
Post by Warner Losh
them?
Most systems don't build software, so there are plenty of uses.
I think all systems *install* software. If we don't have packages, then it
has to be buildable from source and we need a compiler. We just can't close
*both* paths.
And the paths to get packages are solvable problems.
They *should* be. There has been a policy brick wall on this for the
last five years, which is why we *still* don't have any of this.
Post by Warner Losh
These are easy to fix -- for example, here are three possibilities -- but
Post by Warner Losh
1. Allow Tier-2 packages to be built on native hardware without as much
centralized control, letting pkg install of the toolchain work. This fixes
ppc64, but maybe not embedded systems.
If we can't build in QEMU then sure. It doesn't matter where the binaries
come from, so long as they work.
Yes, agreed. This has not been project policy thus far, however.
Policy changes all the time. As pointed out in other email, though, it's
more complicated and at least partially political.
Yes. But this one, now, needs to be solved. If we can get a commitment
from core@ to allow at least one of the options I outlined above by the
deadline, I will be ecstatic. I do not think we can move forward otherwise.


[...]
Post by Warner Losh
Post by Warner Losh
timeline, but wanting 100% full integration from external toolchain with
full packages isn't a gating factor here.
Absolutely! I don't want to make the perfect the enemy of the good,
especially when the good is finally freeing ourselves from GCC 4.2. But we
have been blocked for years by the fact that we have not been able, as a
result of basically purely policy issues, to find a way to bootstrap a GCC
4.2-less system so that ports can be installed. And, as much as I like
running fortune, I think not being able to install 3rd-party software is a
blocker.
Policy can be changed.
Besides, I thought powerpc64 just worked with clang...
It does not. It has been 90% of the way there for 6 years, but still is
not there.
Post by Warner Losh
Unfortunately, no. There are a wide variety of subtle issues. You can boot
a kernel built with -O0, but higher optimization levels or userland code
(especially C++) break.
C++ isn't needed to boostrap gcc.
No, but it is needed to build clang, which it can't currently do. You
also can't build a working kernel or libc. I haven't tried to build GCC
with it lately, but the last time I tried, that also didn't work.
Post by Warner Losh
Warner
Post by Warner Losh
Also, are there any directions for how to build a toolchain from the base
Post by Nathan Whitehorn
ports? I haven't been able to figure it out -- the README starts with "pkg
install", but that doesn't get me very far on platforms without binary
packages, which are the ones this is meant to target.
External toolchains are by definition external. Not part of the tree nor
integrated into the tree. How you get them depends on the external
toolchain and is unspecified. You get a lower level of integration than you
do with in-tree toolchains, not the same level with reach-overs into some
external repo.
I think you missed the point. I was asking how to build the port to test it
-- it doesn't build at present and I can't figure out how to make an
external toolchain via the "base" ports (the other toolchains in ports work
fine).
-Nathan
Warner
I guess I'm looking for an actionable set of blockers with a timetable for
resolution. These all sound a lot like inconvenient problems stemming from
some bad choices of project controlled equipment. External toolchains could
be supplied by third parties as packages. That would solve much of the
bootstrapping problem. Some time spent fixing the bugs in qemu that keep it
from being a viable option would also open up a flood of packages (we have
good infrastructure for creating mips and arm packages today I'm told would
be relatively easy to expand to new architectures).
However, absent a clear plan, as well as resourced dedicated to realize it,
I'm not sure that we should block things further based strictly on a list
of things that are broken. If there's a plan to fix, I'm willing to defer
to give that plan a chance to happen. In the absence of a specific plan
with timelines, I'm reluctant to indefinitely delay the negotiated plan for
12 which has taken years to pull together. Otherwise, is 3 more months
enough? Is 6? are 12? Do we wait for FreeBSD 13?
I think we can fix the problem, which is severe, in a week if we can get
a commitment from core@ to allow at least one of the following things:

1. Package sets (potentially only very minimal ones) uploaded for tier-2
systems from machines not controlled and hosted by portmaster, but
otherwise project-affiliated (like our PPC build systems).
2. The "base" ports built as part of the release/snapshot process and
either included on the media (better) or, at the least, available on the
official package repositories.
3. Allow inclusion of the compiler either in src or in another
project-hosted repository connected to src (I realize this isn't
external toolchain, but it does still allow us to remove GCC 4.2 without
making the system unusable)

I am quite happy to do the technical work personally to enable one or
all of these, but at present cannot do so because project policy
prevents it and has done for a very, very long time.
-Nathan
John Baldwin
2017-10-10 23:26:10 UTC
Permalink
Post by Nathan Whitehorn
I think we can fix the problem, which is severe, in a week if we can get
1. Package sets (potentially only very minimal ones) uploaded for tier-2
systems from machines not controlled and hosted by portmaster, but
otherwise project-affiliated (like our PPC build systems).
2. The "base" ports built as part of the release/snapshot process and
either included on the media (better) or, at the least, available on the
official package repositories.
3. Allow inclusion of the compiler either in src or in another
project-hosted repository connected to src (I realize this isn't
external toolchain, but it does still allow us to remove GCC 4.2 without
making the system unusable)
I would prefer 2) as I think that is most consistent with what other folks
have been pushing towards. I would be fine if we just hosted them in
repositories so that 'pkg install' worked out of the box. At least if we
start building them and publishing the repositories I think it is much
easier to do more testing of them. Eventually we may also want them on
the install media along with the base system dists, but just having them
regularly built and published would be a good first step. I would propose
that we build the sysroot required using the foo-xtoolchain-gcc toolchains.

We don't yet produce any install images for MIPS. If we did I would
advocate that we start providing release snapshots built using
mips-xtoolchain-gcc (make CROSS_TOOLCHAIN=blah) and use the base dist from
those releases as the sysroot to also build the base/ packages and provide
those in a repository.

I think that same approach could be perhaps more readily adopted for
powerpc. That is, cross-building releases using powerpc-xtoolchain-gcc and
then using the resulting base dist as a sysroot to build the base/ packages
and publishing both of those as our snapshots for current.

I suspect our release generation bits don't yet understand CROSS_TOOLCHAIN
and probably will need some changes to handle that.
--
John Baldwin
Warner Losh
2017-10-11 02:36:50 UTC
Permalink
Post by John Baldwin
Post by Nathan Whitehorn
I think we can fix the problem, which is severe, in a week if we can get
1. Package sets (potentially only very minimal ones) uploaded for tier-2
systems from machines not controlled and hosted by portmaster, but
otherwise project-affiliated (like our PPC build systems).
You should take that up with ***@. Individuals can provide these packages,
of course, but part of the project's providing packages is ensuring that
the packages have a clear provenance. That's tricky to do on machines the
project provides.
Post by John Baldwin
Post by Nathan Whitehorn
2. The "base" ports built as part of the release/snapshot process and
either included on the media (better) or, at the least, available on the
official package repositories.
That's possible, but there's no release integration for external
toolchains. And for tier 2 platforms, there's no requirement to do so. And
nobody's stepped up to do this work.
Post by John Baldwin
Post by Nathan Whitehorn
3. Allow inclusion of the compiler either in src or in another
project-hosted repository connected to src (I realize this isn't
external toolchain, but it does still allow us to remove GCC 4.2 without
making the system unusable)
This has its own issues. I'm not sure how this would play out. It seems
like a lot of work, and nobody's name is by it.

There's also:

4. Fix QEMU's issues with powerpc user mode emulation and get slotted into
the same cluster we have building arm and soon mips packages. It seems to
me to be less work than #3 or even #2, and doesn't have the clear
provenance issues #1 has. We have good infrastructure here, though some of
it depends currently on setting up native binaries that produce target
output to improve the speed. That's not a requirement, though, and 100%
emulation is possible, but slow. Still, it beats the alternative we have
today: which is no way to produce anything the project can release.
Post by John Baldwin
I would prefer 2) as I think that is most consistent with what other folks
have been pushing towards. I would be fine if we just hosted them in
repositories so that 'pkg install' worked out of the box. At least if we
start building them and publishing the repositories I think it is much
easier to do more testing of them. Eventually we may also want them on
the install media along with the base system dists, but just having them
regularly built and published would be a good first step. I would propose
that we build the sysroot required using the foo-xtoolchain-gcc toolchains.
If this can be cross built, then that's a good first step. We'd need some
way to produce those packages that the ports that do cross build work on.
The lack of native machines under project control, or of qemu (either
userland or full system) limit our options here.
Post by John Baldwin
We don't yet produce any install images for MIPS. If we did I would
advocate that we start providing release snapshots built using
mips-xtoolchain-gcc (make CROSS_TOOLCHAIN=blah) and use the base dist from
those releases as the sysroot to also build the base/ packages and provide
those in a repository.
I think that same approach could be perhaps more readily adopted for
powerpc. That is, cross-building releases using powerpc-xtoolchain-gcc and
then using the resulting base dist as a sysroot to build the base/ packages
and publishing both of those as our snapshots for current.
That's another path forward, but as you point out would require some more
integration work. We can build the raw images today, but the snapshot
integration to generate installable images aren't there.
Post by John Baldwin
I suspect our release generation bits don't yet understand CROSS_TOOLCHAIN
and probably will need some changes to handle that.
That would be nice, but integration into the release process never was part
of the discussion on external toolchains. It was envisioned for a tier-2 or
3 architecture where release engineering or security officer integration
isn't guaranteed. Both powerpc and mips are tier 2 platforms, which make it
more of a challenge to use them.

But I'm puzzled about something. We don't provide packages today, why
should that become a new requirement to remove gcc from the tree? If we
look at why, we see that we don't have PowerPC hardware in the project that
can build packages. We don't have an emulation alternative like we do for
arm and mips to fall back on. If the cross building works for a small
number of packages, we could create a pkg release repo for that. So long as
we have a complete chain of custody, there should be no barrier to doing
this, even if we don't release release images. This wouldn't be hard to do,
but someone needs to do the work and integrate into the tree and
src/release.

But all these things are stuff we could do. I'm happy to delay to give
people time to do them. However, I'd like to know who is doing the stuff
and on what timeframe. If someone can come up with that, then I can push
back the proposed timeline. I don't oppose any of this. I think it's all
great. I just oppose delaying for vague plans.

Warner
Nathan Whitehorn
2017-10-11 03:02:30 UTC
Permalink
Post by Nathan Whitehorn
Post by Nathan Whitehorn
I think we can fix the problem, which is severe, in a week if we
can get
Post by Nathan Whitehorn
1. Package sets (potentially only very minimal ones) uploaded
for tier-2
Post by Nathan Whitehorn
systems from machines not controlled and hosted by portmaster, but
otherwise project-affiliated (like our PPC build systems).
packages, of course, but part of the project's providing packages is
ensuring that the packages have a clear provenance. That's tricky to
do on machines the project provides.
Post by Nathan Whitehorn
2. The "base" ports built as part of the release/snapshot
process and
Post by Nathan Whitehorn
either included on the media (better) or, at the least,
available on the
Post by Nathan Whitehorn
official package repositories.
That's possible, but there's no release integration for external
toolchains. And for tier 2 platforms, there's no requirement to do so.
And nobody's stepped up to do this work.
Post by Nathan Whitehorn
3. Allow inclusion of the compiler either in src or in another
project-hosted repository connected to src (I realize this isn't
external toolchain, but it does still allow us to remove GCC 4.2
without
Post by Nathan Whitehorn
making the system unusable)
This has its own issues. I'm not sure how this would play out. It
seems like a lot of work, and nobody's name is by it.
4. Fix QEMU's issues with powerpc user mode emulation and get slotted
into the same cluster we have building arm and soon mips packages. It
seems to me to be less work than #3 or even #2, and doesn't have the
clear provenance issues #1 has. We have good infrastructure here,
though some of it depends currently on setting up native binaries that
produce target output to improve the speed. That's not a requirement,
though, and 100% emulation is possible, but slow. Still, it beats the
alternative we have today: which is no way to produce anything the
project can release.
This has been in progress for years and is very hard. #2 or #3 are super
easy, except from an administrative standpoint, where, like #1, they
have proved impossible. I am stepping up to do the work right now, and
have for a long time, but the work is entirely administrative and has
been blocked.
Post by Nathan Whitehorn
I would prefer 2) as I think that is most consistent with what other folks
have been pushing towards. I would be fine if we just hosted them in
repositories so that 'pkg install' worked out of the box. At least
if we
start building them and publishing the repositories I think it is much
easier to do more testing of them. Eventually we may also want them on
the install media along with the base system dists, but just having them
regularly built and published would be a good first step. I would
propose
that we build the sysroot required using the foo-xtoolchain-gcc toolchains.
If this can be cross built, then that's a good first step. We'd need
some way to produce those packages that the ports that do cross build
work on. The lack of native machines under project control, or of qemu
(either userland or full system) limit our options here.
We don't yet produce any install images for MIPS. If we did I would
advocate that we start providing release snapshots built using
mips-xtoolchain-gcc (make CROSS_TOOLCHAIN=blah) and use the base dist from
those releases as the sysroot to also build the base/ packages and provide
those in a repository.
I think that same approach could be perhaps more readily adopted for
powerpc. That is, cross-building releases using
powerpc-xtoolchain-gcc and
then using the resulting base dist as a sysroot to build the base/ packages
and publishing both of those as our snapshots for current.
That's another path forward, but as you point out would require some
more integration work. We can build the raw images today, but the
snapshot integration to generate installable images aren't there.
I suspect our release generation bits don't yet understand
CROSS_TOOLCHAIN
and probably will need some changes to handle that.
That would be nice, but integration into the release process never was
part of the discussion on external toolchains. It was envisioned for a
tier-2 or 3 architecture where release engineering or security officer
integration isn't guaranteed. Both powerpc and mips are tier 2
platforms, which make it more of a challenge to use them.
And yet, we *do* provide releases for them consistently.
Post by Nathan Whitehorn
But I'm puzzled about something. We don't provide packages today, why
should that become a new requirement to remove gcc from the tree?
Because we currently have users build from ports with the compiler
included in the system. If we take that compiler away, we provide only
the option to install a compiler from packages. If we also don't provide
a compiler package, the system is a brick and we might as well remove
the platform from the tree.
Post by Nathan Whitehorn
If we look at why, we see that we don't have PowerPC hardware in the
project that can build packages.
We do have PowerPC hardware on which we can build packages, but we do
not actually do that for a variety of largely administrative reasons
that are out of scope for this thread.
Post by Nathan Whitehorn
We don't have an emulation alternative like we do for arm and mips to
fall back on. If the cross building works for a small number of
packages, we could create a pkg release repo for that. So long as we
have a complete chain of custody, there should be no barrier to doing
this, even if we don't release release images. This wouldn't be hard
to do, but someone needs to do the work and integrate into the tree
and src/release.
If this is the plan, great. I'm happy to do whatever technical work is
required for this to meet your end-of-year deadline so long as the
project is actually willing to support this.
Post by Nathan Whitehorn
But all these things are stuff we could do. I'm happy to delay to give
people time to do them. However, I'd like to know who is doing the
stuff and on what timeframe. If someone can come up with that, then I
can push back the proposed timeline. I don't oppose any of this. I
think it's all great. I just oppose delaying for vague plans.
It's not vague: If we have a commitment to cross-build the toolchain
packages and upload them with releases, then we're all done here. If we
don't have that, then removing GCC 4 makes all tier-2 platforms except
ARM totally unusable. The two need to be coupled in the proposal.
-Nathan
Post by Nathan Whitehorn
Warner
Mark Linimon
2017-10-11 22:32:12 UTC
Permalink
Post by Nathan Whitehorn
1. Package sets (potentially only very minimal ones) uploaded for tier-2
systems from machines not controlled and hosted by portmaster
nit: s/portmaster/portmgr/

Kind of a sore point with me, sorry.

mcl
Mark Linimon
2017-10-11 22:44:48 UTC
Permalink
Post by Warner Losh
However, absent a clear plan, as well as resourced dedicated to realize it,
I'm not sure that we should block things further based strictly on a list
of things that are broken. If there's a plan to fix, I'm willing to defer
to give that plan a chance to happen. In the absence of a specific plan
with timelines, I'm reluctant to indefinitely delay the negotiated plan
for 12 which has taken years to pull together. Otherwise, is 3 more months
enough? Is 6? are 12? Do we wait for FreeBSD 13?
Even though I work on both affected archs, I think gcc4.2.1 in base has to
be _gone_ for 12. It's time.

Here's my compromise suggestion.

We give nathan (and myself and whoever else wants to help) until the BSDCan
Devsummit to fix this. But at that point it's the Danish Axe(*) no matter
what state it's in. Period.

That gives us an absolute deadline.

For reference, gjb has guesstimated that the 12.0 release cycle will begin
around February 2019. (good grief I nearly typed 2109. stahp.)

In any case, axing things during the Devsummit is a Grand Tradition at this
point. Why not continue it? :-)

mcl

* or Wemm axe, or Losh axe, or whatever
John Baldwin
2017-10-10 13:49:44 UTC
Permalink
Post by Nathan Whitehorn
Post by Warner Losh
Post by Nathan Whitehorn
Post by Baptiste Daroussin
Post by John Baldwin
Post by Warner Losh
I'd like to start a conversation about the viability of making C++11 a hard
requirement for bootstrapping FreeBSD and setting a specific deadline for
doing so.
This discussion is motivated by an ask from the jemalloc folks to use a
limited subset of C++11 inside of malloc in such a way that is C safe (so
the C programs wouldn't bloat with a C++ runtime). That's an ongoing
discussion in another forum, and isn't appropriate for this thread because
this has become a frequent request (but if you have opinions, please find
to do this.
I'd like to take the rather vague plans we've had "before 12" and create a
timeline for removal of gcc 4.2 coupled with a requirement for support in
clang or a working external toolchain. This requirement would be coupled
with the requirement that the external toolchain support C++11 constructs.
I'd like to propose we do this 12/31/17. Any architectures that can't meet
this timeline will be disconnected from universe at that time and deleted
3/31/18.
It's my belief that i386, amd64, arm, aarch64, powerpc and powerpc64 are
ready for this change and mips* would be ready for this change with an
external clang toolchain. I'm unsure of riscv and sparc64, but suspect that
a newer version of gcc as an external toolchain could work.
In-tree clang 5.0 for MIPS mostly works (modulo some small patches).
However,
it requires external ld.bfd. I know that there ld.lld can link a working
mips64 world with some patches (notably the multigot patch). mips works fine
with external GCC. riscv is already using external GCC that is C++11-capable.
The problem with external GCC is that you can cross-build a world + kernel
just fine and get a working system via CROSS_TOOLCHAIN=foo-gcc. However,
started on ports to cross-build binutils and gcc packages via the base/*
ports, but those are not yet finished / fully tested. I don't think anyone
has thought about how release builds will work either with only an external
toolchain available. (I'm pretty sure sparc64 has booted fine with
external GCC, it's just in the same boat as mips with regard to /usr/bin/cc.)
Actually I did test those and they were working (tested in qemu) they were
working fine. I have given up working on them due to the lack of interested by
the community (by interest I mean people really testing, working on it, not just
saying "hey nice sounds cool").
As for the boot when I initially worked on external toolchain sparc64 was my
guinea pig and so yes it worked an booted just fine.
So far as I know, we never solved any of the infrastructural problems
1. Providing built releases with a /usr/bin/cc
2. Coversioning base and in-ports toolchain, including ensuring commit
atomicity between toolchains and libc
3. Adding a dependency on ports for src, including out-of-tree code that
has to be fetched from external servers
4. Getting make universe to do the right thing
We really need to solve those. If we go the external toolchain route,
which it is not clear to me is the best option, #2 and #1 are quite complex
problems. I think that, frankly, a deadline in two months to solve this set
of political problems we have had for two years is probably crazy, but
maybe making the status quo unsustainable will finally force progress.
External toolchains have been in flight for 5 or more years now. It's time
they land. Though the requirements for them have never included
cross-threading between /usr/src and /usr/ports like you suggest above, and
those sorts of things won't be sorted by my deadlines (which are closer to
3 months). Nor, imho, should they.
Well, sure. But the fact remains that we cannot build usable systems with
external toolchains right now. Those are real problems that need to be
solved somehow.
Sure we can. I've built a bootable i386 system with gcc 6. It is a solved
problem.
"Bootable" is not the same as "usable" here. External toolchain
powerpc64 *boots* fine -- and has for years -- but there isn't much you
can actual *do* with the system except admire dmesg without the ability
to install ports.
Post by Warner Losh
Let's focus on #1, the largest if not the only major problem. If I build,
say, a ppc64 system with an external toolchain right now, it boots and runs
- There are no binary packages built for PPC64, because of project policy
preventing the use of native build systems
System is still usable w/o packages. People can still fire up custom
poudrier repos. We could also change project policy. This is is a specific
wrinkle for powerpc64 it seems.
How would I do that? We can't run these natively, because the system
ships without a compiler. And we can't cross-compile, because the ports
tree does not support that.
The base/ ports _do_ cross-compile. See /usr/ports/base/README. You have to
build a world image using the external toolchain that can then be used as a
--sysroot with the external toolchain to build binutils and gcc packages.
These packages should then be able to be installed. To be clear, you would
build a world on an amd64 host with the foo-xtoolchain-gcc toolchain, install
it to some 'rootfs' directory, then build the base/ packages on the same
amd64 host but the generated packages can be installed on the target via
pkg install. In particular, we could automate building of the base/ packages
in the cluster and publish repositories that contain just binutils and gcc.
--
John Baldwin
Brooks Davis
2017-10-10 18:16:39 UTC
Permalink
Post by John Baldwin
Post by Nathan Whitehorn
Post by Warner Losh
Post by Nathan Whitehorn
Post by Baptiste Daroussin
Post by John Baldwin
Post by Warner Losh
I'd like to start a conversation about the viability of making C++11 a hard
requirement for bootstrapping FreeBSD and setting a specific deadline for
doing so.
This discussion is motivated by an ask from the jemalloc folks to use a
limited subset of C++11 inside of malloc in such a way that is C safe (so
the C programs wouldn't bloat with a C++ runtime). That's an ongoing
discussion in another forum, and isn't appropriate for this thread because
this has become a frequent request (but if you have opinions, please find
to do this.
I'd like to take the rather vague plans we've had "before 12" and create a
timeline for removal of gcc 4.2 coupled with a requirement for support in
clang or a working external toolchain. This requirement would be coupled
with the requirement that the external toolchain support C++11 constructs.
I'd like to propose we do this 12/31/17. Any architectures that can't meet
this timeline will be disconnected from universe at that time and deleted
3/31/18.
It's my belief that i386, amd64, arm, aarch64, powerpc and powerpc64 are
ready for this change and mips* would be ready for this change with an
external clang toolchain. I'm unsure of riscv and sparc64, but suspect that
a newer version of gcc as an external toolchain could work.
In-tree clang 5.0 for MIPS mostly works (modulo some small patches).
However,
it requires external ld.bfd. I know that there ld.lld can link a working
mips64 world with some patches (notably the multigot patch). mips works fine
with external GCC. riscv is already using external GCC that is
C++11-capable.
The problem with external GCC is that you can cross-build a world + kernel
just fine and get a working system via CROSS_TOOLCHAIN=foo-gcc. However,
started on ports to cross-build binutils and gcc packages via the base/*
ports, but those are not yet finished / fully tested. I don't think anyone
has thought about how release builds will work either with only an external
toolchain available. (I'm pretty sure sparc64 has booted fine with
external GCC, it's just in the same boat as mips with regard to /usr/bin/cc.)
Actually I did test those and they were working (tested in qemu) they were
working fine. I have given up working on them due to the lack of interested by
the community (by interest I mean people really testing, working on it,
not just
saying "hey nice sounds cool").
As for the boot when I initially worked on external toolchain sparc64 was my
guinea pig and so yes it worked an booted just fine.
So far as I know, we never solved any of the infrastructural problems
1. Providing built releases with a /usr/bin/cc
2. Coversioning base and in-ports toolchain, including ensuring commit
atomicity between toolchains and libc
3. Adding a dependency on ports for src, including out-of-tree code that
has to be fetched from external servers
4. Getting make universe to do the right thing
We really need to solve those. If we go the external toolchain route,
which it is not clear to me is the best option, #2 and #1 are quite complex
problems. I think that, frankly, a deadline in two months to solve this set
of political problems we have had for two years is probably crazy, but
maybe making the status quo unsustainable will finally force progress.
External toolchains have been in flight for 5 or more years now. It's time
they land. Though the requirements for them have never included
cross-threading between /usr/src and /usr/ports like you suggest above, and
those sorts of things won't be sorted by my deadlines (which are closer to
3 months). Nor, imho, should they.
Well, sure. But the fact remains that we cannot build usable systems with
external toolchains right now. Those are real problems that need to be
solved somehow.
Sure we can. I've built a bootable i386 system with gcc 6. It is a solved
problem.
"Bootable" is not the same as "usable" here. External toolchain
powerpc64 *boots* fine -- and has for years -- but there isn't much you
can actual *do* with the system except admire dmesg without the ability
to install ports.
Post by Warner Losh
Let's focus on #1, the largest if not the only major problem. If I build,
say, a ppc64 system with an external toolchain right now, it boots and runs
- There are no binary packages built for PPC64, because of project policy
preventing the use of native build systems
System is still usable w/o packages. People can still fire up custom
poudrier repos. We could also change project policy. This is is a specific
wrinkle for powerpc64 it seems.
How would I do that? We can't run these natively, because the system
ships without a compiler. And we can't cross-compile, because the ports
tree does not support that.
The base/ ports _do_ cross-compile. See /usr/ports/base/README. You have to
build a world image using the external toolchain that can then be used as a
--sysroot with the external toolchain to build binutils and gcc packages.
These packages should then be able to be installed. To be clear, you would
build a world on an amd64 host with the foo-xtoolchain-gcc toolchain, install
it to some 'rootfs' directory, then build the base/ packages on the same
amd64 host but the generated packages can be installed on the target via
pkg install. In particular, we could automate building of the base/ packages
in the cluster and publish repositories that contain just binutils and gcc.
I think building and publishing mini-repositories of pkg and "the things
required to cross build a release" is a must. We should include the
repo along side the releasese in the FTP[0] hierarchy. It's fairly
straightforward and we should do it.

-- Brooks

[0] Can we please stop using this ancient protocol before we are the
last signficant project using it and it is blocked by every enterprise
firewall.
Warner Losh
2017-10-10 19:27:23 UTC
Permalink
Post by Warner Losh
Post by John Baldwin
Post by Nathan Whitehorn
On Sun, Oct 8, 2017 at 11:01 PM, Nathan Whitehorn <
Post by Nathan Whitehorn
Post by Baptiste Daroussin
Post by John Baldwin
Post by Warner Losh
I'd like to start a conversation about the viability of making
C++11 a
Post by John Baldwin
Post by Nathan Whitehorn
Post by Nathan Whitehorn
Post by Baptiste Daroussin
Post by John Baldwin
Post by Warner Losh
hard
requirement for bootstrapping FreeBSD and setting a specific
deadline
Post by John Baldwin
Post by Nathan Whitehorn
Post by Nathan Whitehorn
Post by Baptiste Daroussin
Post by John Baldwin
Post by Warner Losh
for
doing so.
This discussion is motivated by an ask from the jemalloc folks
to use a
Post by John Baldwin
Post by Nathan Whitehorn
Post by Nathan Whitehorn
Post by Baptiste Daroussin
Post by John Baldwin
Post by Warner Losh
limited subset of C++11 inside of malloc in such a way that is C
safe
Post by John Baldwin
Post by Nathan Whitehorn
Post by Nathan Whitehorn
Post by Baptiste Daroussin
Post by John Baldwin
Post by Warner Losh
(so
the C programs wouldn't bloat with a C++ runtime). That's an
ongoing
Post by John Baldwin
Post by Nathan Whitehorn
Post by Nathan Whitehorn
Post by Baptiste Daroussin
Post by John Baldwin
Post by Warner Losh
discussion in another forum, and isn't appropriate for this
thread
Post by John Baldwin
Post by Nathan Whitehorn
Post by Nathan Whitehorn
Post by Baptiste Daroussin
Post by John Baldwin
Post by Warner Losh
because
this has become a frequent request (but if you have opinions,
please
Post by John Baldwin
Post by Nathan Whitehorn
Post by Nathan Whitehorn
Post by Baptiste Daroussin
Post by John Baldwin
Post by Warner Losh
find
their
Post by John Baldwin
Post by Nathan Whitehorn
Post by Nathan Whitehorn
Post by Baptiste Daroussin
Post by John Baldwin
Post by Warner Losh
plans
to do this.
I'd like to take the rather vague plans we've had "before 12" and
create a
timeline for removal of gcc 4.2 coupled with a requirement for
support
Post by John Baldwin
Post by Nathan Whitehorn
Post by Nathan Whitehorn
Post by Baptiste Daroussin
Post by John Baldwin
Post by Warner Losh
in
clang or a working external toolchain. This requirement would be
coupled
Post by John Baldwin
Post by Nathan Whitehorn
Post by Nathan Whitehorn
Post by Baptiste Daroussin
Post by John Baldwin
Post by Warner Losh
with the requirement that the external toolchain support C++11
constructs.
I'd like to propose we do this 12/31/17. Any architectures that
can't
Post by John Baldwin
Post by Nathan Whitehorn
Post by Nathan Whitehorn
Post by Baptiste Daroussin
Post by John Baldwin
Post by Warner Losh
meet
this timeline will be disconnected from universe at that time and
deleted
3/31/18.
It's my belief that i386, amd64, arm, aarch64, powerpc and
powerpc64 are
Post by John Baldwin
Post by Nathan Whitehorn
Post by Nathan Whitehorn
Post by Baptiste Daroussin
Post by John Baldwin
Post by Warner Losh
ready for this change and mips* would be ready for this change
with an
Post by John Baldwin
Post by Nathan Whitehorn
Post by Nathan Whitehorn
Post by Baptiste Daroussin
Post by John Baldwin
Post by Warner Losh
external clang toolchain. I'm unsure of riscv and sparc64, but
suspect
Post by John Baldwin
Post by Nathan Whitehorn
Post by Nathan Whitehorn
Post by Baptiste Daroussin
Post by John Baldwin
Post by Warner Losh
that
a newer version of gcc as an external toolchain could work.
In-tree clang 5.0 for MIPS mostly works (modulo some small
patches).
Post by John Baldwin
Post by Nathan Whitehorn
Post by Nathan Whitehorn
Post by Baptiste Daroussin
Post by John Baldwin
However,
it requires external ld.bfd. I know that there ld.lld can link a
working
Post by John Baldwin
Post by Nathan Whitehorn
Post by Nathan Whitehorn
Post by Baptiste Daroussin
Post by John Baldwin
mips64 world with some patches (notably the multigot patch).
mips works
Post by John Baldwin
Post by Nathan Whitehorn
Post by Nathan Whitehorn
Post by Baptiste Daroussin
Post by John Baldwin
fine
with external GCC. riscv is already using external GCC that is
C++11-capable.
The problem with external GCC is that you can cross-build a world
+
Post by John Baldwin
Post by Nathan Whitehorn
Post by Nathan Whitehorn
Post by Baptiste Daroussin
Post by John Baldwin
kernel
just fine and get a working system via CROSS_TOOLCHAIN=foo-gcc.
However,
Post by John Baldwin
Post by Nathan Whitehorn
Post by Nathan Whitehorn
Post by Baptiste Daroussin
Post by John Baldwin
that system has no viable '/usr/bin/cc' once GCC 4.2 is removed.
started on ports to cross-build binutils and gcc packages via the
base/*
Post by John Baldwin
Post by Nathan Whitehorn
Post by Nathan Whitehorn
Post by Baptiste Daroussin
Post by John Baldwin
ports, but those are not yet finished / fully tested. I don't
think
Post by John Baldwin
Post by Nathan Whitehorn
Post by Nathan Whitehorn
Post by Baptiste Daroussin
Post by John Baldwin
anyone
has thought about how release builds will work either with only an
external
toolchain available. (I'm pretty sure sparc64 has booted fine
with
Post by John Baldwin
Post by Nathan Whitehorn
Post by Nathan Whitehorn
Post by Baptiste Daroussin
Post by John Baldwin
external GCC, it's just in the same boat as mips with regard to
/usr/bin/cc.)
Actually I did test those and they were working (tested in qemu)
they were
Post by John Baldwin
Post by Nathan Whitehorn
Post by Nathan Whitehorn
Post by Baptiste Daroussin
working fine. I have given up working on them due to the lack of
interested by
the community (by interest I mean people really testing, working
on it,
Post by John Baldwin
Post by Nathan Whitehorn
Post by Nathan Whitehorn
Post by Baptiste Daroussin
not just
saying "hey nice sounds cool").
As for the boot when I initially worked on external toolchain
sparc64 was
Post by John Baldwin
Post by Nathan Whitehorn
Post by Nathan Whitehorn
Post by Baptiste Daroussin
my
guinea pig and so yes it worked an booted just fine.
So far as I know, we never solved any of the infrastructural
problems
Post by John Baldwin
Post by Nathan Whitehorn
Post by Nathan Whitehorn
1. Providing built releases with a /usr/bin/cc
2. Coversioning base and in-ports toolchain, including ensuring
commit
Post by John Baldwin
Post by Nathan Whitehorn
Post by Nathan Whitehorn
atomicity between toolchains and libc
3. Adding a dependency on ports for src, including out-of-tree code
that
Post by John Baldwin
Post by Nathan Whitehorn
Post by Nathan Whitehorn
has to be fetched from external servers
4. Getting make universe to do the right thing
We really need to solve those. If we go the external toolchain
route,
Post by John Baldwin
Post by Nathan Whitehorn
Post by Nathan Whitehorn
which it is not clear to me is the best option, #2 and #1 are quite
complex
Post by John Baldwin
Post by Nathan Whitehorn
Post by Nathan Whitehorn
problems. I think that, frankly, a deadline in two months to solve
this set
Post by John Baldwin
Post by Nathan Whitehorn
Post by Nathan Whitehorn
of political problems we have had for two years is probably crazy,
but
Post by John Baldwin
Post by Nathan Whitehorn
Post by Nathan Whitehorn
maybe making the status quo unsustainable will finally force
progress.
Post by John Baldwin
Post by Nathan Whitehorn
External toolchains have been in flight for 5 or more years now.
It's time
Post by John Baldwin
Post by Nathan Whitehorn
they land. Though the requirements for them have never included
cross-threading between /usr/src and /usr/ports like you suggest
above, and
Post by John Baldwin
Post by Nathan Whitehorn
those sorts of things won't be sorted by my deadlines (which are
closer to
Post by John Baldwin
Post by Nathan Whitehorn
3 months). Nor, imho, should they.
Well, sure. But the fact remains that we cannot build usable systems
with
Post by John Baldwin
Post by Nathan Whitehorn
external toolchains right now. Those are real problems that need to
be
Post by John Baldwin
Post by Nathan Whitehorn
solved somehow.
Sure we can. I've built a bootable i386 system with gcc 6. It is a
solved
Post by John Baldwin
Post by Nathan Whitehorn
problem.
"Bootable" is not the same as "usable" here. External toolchain
powerpc64 *boots* fine -- and has for years -- but there isn't much you
can actual *do* with the system except admire dmesg without the ability
to install ports.
Let's focus on #1, the largest if not the only major problem. If I
build,
Post by John Baldwin
Post by Nathan Whitehorn
say, a ppc64 system with an external toolchain right now, it boots
and runs
Post by John Baldwin
Post by Nathan Whitehorn
- There are no binary packages built for PPC64, because of project
policy
Post by John Baldwin
Post by Nathan Whitehorn
preventing the use of native build systems
System is still usable w/o packages. People can still fire up custom
poudrier repos. We could also change project policy. This is is a
specific
Post by John Baldwin
Post by Nathan Whitehorn
wrinkle for powerpc64 it seems.
How would I do that? We can't run these natively, because the system
ships without a compiler. And we can't cross-compile, because the ports
tree does not support that.
The base/ ports _do_ cross-compile. See /usr/ports/base/README. You
have to
Post by John Baldwin
build a world image using the external toolchain that can then be used
as a
Post by John Baldwin
--sysroot with the external toolchain to build binutils and gcc packages.
These packages should then be able to be installed. To be clear, you
would
Post by John Baldwin
build a world on an amd64 host with the foo-xtoolchain-gcc toolchain,
install
Post by John Baldwin
it to some 'rootfs' directory, then build the base/ packages on the same
amd64 host but the generated packages can be installed on the target via
pkg install. In particular, we could automate building of the base/
packages
Post by John Baldwin
in the cluster and publish repositories that contain just binutils and
gcc.
I think building and publishing mini-repositories of pkg and "the things
required to cross build a release" is a must. We should include the
repo along side the releasese in the FTP[0] hierarchy. It's fairly
straightforward and we should do it.
-- Brooks
[0] Can we please stop using this ancient protocol before we are the
last signficant project using it and it is blocked by every enterprise
firewall.
Given that PowerPC is a tier 2 architecture, I'm not sure that I'd call it
a MUST. It's nice to have, and really useful if we have it. But Tier 2 is
best effort to support, so it isn't a "must" in the sense it is a hard
blocker for the removal of gcc.

However, if there's plans in flight to do this, I'll let the complete (or
timeout) before moving forward with any deorbiting. So long as there's a
concrete timeline of reasonable things, I'm happy to defer to allow that to
proceed. What I don't want to do is defer to some vague plans by some
unnamed individuals that might get around to it, but we're blocked on
that...

Warner
Nathan Whitehorn
2017-10-10 22:30:27 UTC
Permalink
On 10/10/17 12:27, Warner Losh wrote:
[...]
Post by Warner Losh
Post by Nathan Whitehorn
Post by Warner Losh
Sure we can. I've built a bootable i386 system with gcc 6.
It is a solved
Post by Nathan Whitehorn
Post by Warner Losh
problem.
"Bootable" is not the same as "usable" here. External toolchain
powerpc64 *boots* fine -- and has for years -- but there isn't
much you
Post by Nathan Whitehorn
can actually *do* with the system except admire dmesg without
the ability
Post by Nathan Whitehorn
to install ports.
Post by Warner Losh
Let's focus on #1, the largest if not the only major
problem. If I build,
Post by Nathan Whitehorn
Post by Warner Losh
say, a ppc64 system with an external toolchain right now, it
boots and runs
Post by Nathan Whitehorn
Post by Warner Losh
- There are no binary packages built for PPC64, because of
project policy
Post by Nathan Whitehorn
Post by Warner Losh
preventing the use of native build systems
System is still usable w/o packages. People can still fire
up custom
Post by Nathan Whitehorn
Post by Warner Losh
poudrier repos. We could also change project policy. This is
is a specific
Post by Nathan Whitehorn
Post by Warner Losh
wrinkle for powerpc64 it seems.
How would I do that? We can't run these natively, because the
system
Post by Nathan Whitehorn
ships without a compiler. And we can't cross-compile, because
the ports
Post by Nathan Whitehorn
tree does not support that.
The base/ ports _do_ cross-compile.  See
/usr/ports/base/README.  You have to
build a world image using the external toolchain that can then
be used as a
--sysroot with the external toolchain to build binutils and gcc
packages.
These packages should then be able to be installed.  To be
clear, you would
build a world on an amd64 host with the foo-xtoolchain-gcc
toolchain, install
it to some 'rootfs' directory, then build the base/ packages on
the same
amd64 host but the generated packages can be installed on the
target via
pkg install.  In particular, we could automate building of the
base/ packages
in the cluster and publish repositories that contain just
binutils and gcc.
I think building and publishing mini-repositories of pkg and "the things
required to cross build a release" is a must.  We should include the
repo along side the releasese in the FTP[0] hierarchy. It's fairly
straightforward and we should do it.
-- Brooks
[0] Can we please stop using this ancient protocol before we are the
last signficant project using it and it is blocked by every enterprise
firewall.
Given that PowerPC is a tier 2 architecture, I'm not sure that I'd
call it a MUST. It's nice to have, and really useful if we have it.
But Tier 2 is best effort to support, so it isn't a "must" in the
sense it is a hard blocker for the removal of gcc.
However, if there's plans in flight to do this, I'll let the complete
(or timeout) before moving forward with any deorbiting. So long as
there's a concrete timeline of reasonable things, I'm happy to defer
to allow that to proceed. What I don't want to do is defer to some
vague plans by some unnamed individuals that might get around to it,
but we're blocked on that...
Warner
This problem applies to all architectures except ARM and x86 and removal
of the compiler from the base system without doing something like this
will make any installed system unusable. It should be a blocker.

We have multiple volunteers to do the work, which, as Brooks notes, is
super easy, but need a change in project policy, which has prevented the
work in question from being done for many years. If you include that
policy change as part of your proposal, I am very happy to support it.
-Nathan
Nathan Whitehorn
2017-10-21 04:38:25 UTC
Permalink
On 10/10/17 06:49, John Baldwin wrote:
[...]
Post by John Baldwin
Post by Nathan Whitehorn
Post by Warner Losh
Let's focus on #1, the largest if not the only major problem. If I build,
say, a ppc64 system with an external toolchain right now, it boots and runs
- There are no binary packages built for PPC64, because of project policy
preventing the use of native build systems
System is still usable w/o packages. People can still fire up custom
poudrier repos. We could also change project policy. This is is a specific
wrinkle for powerpc64 it seems.
How would I do that? We can't run these natively, because the system
ships without a compiler. And we can't cross-compile, because the ports
tree does not support that.
The base/ ports _do_ cross-compile. See /usr/ports/base/README. You have to
build a world image using the external toolchain that can then be used as a
--sysroot with the external toolchain to build binutils and gcc packages.
These packages should then be able to be installed. To be clear, you would
build a world on an amd64 host with the foo-xtoolchain-gcc toolchain, install
it to some 'rootfs' directory, then build the base/ packages on the same
amd64 host but the generated packages can be installed on the target via
pkg install. In particular, we could automate building of the base/ packages
in the cluster and publish repositories that contain just binutils and gcc.
Has anyone managed to have this work recently? GCC seems to play quite
poorly with our libc++ and I haven't managed to get base/gcc to build at
all despite trying for a few days. I did get base/binutils and pkg to
cross-compile after some tweaks, but base/gcc just won't go.
-Nathan
Nathan Whitehorn
2017-12-18 23:15:11 UTC
Permalink
Post by Nathan Whitehorn
[...]
Post by Nathan Whitehorn
Post by Warner Losh
Let's focus on #1, the largest if not the only major problem. If I build,
say, a ppc64 system with an external toolchain right now, it boots and runs
- There are no binary packages built for PPC64, because of project policy
preventing the use of native build systems
System is still usable w/o packages. People can still fire up custom
poudrier repos. We could also change project policy. This is is a specific
wrinkle for powerpc64 it seems.
How would I do that? We can't run these natively, because the system
ships without a compiler. And we can't cross-compile, because the ports
tree does not support that.
The base/ ports _do_ cross-compile.  See /usr/ports/base/README.  You
have to
build a world image using the external toolchain that can then be used as a
--sysroot with the external toolchain to build binutils and gcc packages.
These packages should then be able to be installed.  To be clear, you
would
build a world on an amd64 host with the foo-xtoolchain-gcc toolchain, install
it to some 'rootfs' directory, then build the base/ packages on the same
amd64 host but the generated packages can be installed on the target via
pkg install.  In particular, we could automate building of the base/
packages
in the cluster and publish repositories that contain just binutils and gcc.
Has anyone managed to have this work recently? GCC seems to play quite
poorly with our libc++ and I haven't managed to get base/gcc to build
at all despite trying for a few days. I did get base/binutils and pkg
to cross-compile after some tweaks, but base/gcc just won't go.
-Nathan
_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-arch
I eventually managed to get the ports to build. It would be great if a
ports committer could look through this PR, which is enough to get
base/binutils, base/gcc, and ports-mgmt/pkg to cross-compile:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=224217
-Nathan

Mark Linimon
2017-10-09 19:56:52 UTC
Permalink
Post by Warner Losh
- There are no binary packages built for PPC64, because of project policy
preventing the use of native build systems
I don't think this is 100% correct.

Although I am no longer on portmgr and thus don't work on the details,
my understanding is that:

- "official" FreeBSD packages can only be issued from machines that
are under the exclusive control of FreeBSD.org
- the powerpc64 machine that falls under that category is not yet
reliable enough

The first is a security concern. The odds of that policy changing are
about the same as Elvis doing another concert.

IIUC the second is because we run package builds under virtualization
on freebsd.org's powerpc64 machine, and hit memory contstraints often
enough to make it "not quite ready for production". You would have to
ask swills@ whether the latter is still true.

My own powerpc64 builds are constrained for other reasons[*]. When I
get back from my current road trip I am willing to build and make
available a subset of powerpc64 packages on the same basis I currently
do for sparc64, if that will help the situation.
Post by Warner Losh
- You cannot cross-compile packages for PPC64, because of limitations
in QEMU
s/limitations/a bug/

qemu/powerpc64 simply hangs when you start it up (as does qemu/sparc64).
Otherwise qemu is usable for cross-builds and is, in fact, how armv6,
mips, and mips64 package builds are currently done.

I know less about your other points, but, there needs to be some kind
of script or wrapper around the "do a base cross-build". I had had one
set up for amd64 to sparc64 as a test but got distracted before I finished.
I recall it being non-trivial.

mcl

[*] a couple of kernel bugs which I think I know how to fix; not being
able to run diskful, and power consumption
Peter Jeremy
2017-10-08 18:48:00 UTC
Permalink
Post by Warner Losh
I'd like to start a conversation about the viability of making C++11 a hard
requirement for bootstrapping FreeBSD and setting a specific deadline for
doing so.
I don't have any specific objection and the suggested timeframe sounds
reasonable to me.
Post by Warner Losh
I'd like to propose we do this 12/31/17. Any architectures that can't meet
this timeline will be disconnected from universe at that time and deleted
3/31/18.
Note that FreeBSD is an international project. Using abbreviated US-format
dates can be ambiguous. Please either use ISO standard dates (2017-12-31)
or a variant with the abbreviated month name (2017-Dec-31) to reduce the
scope for confusion.
--
Peter Jeremy
Loading...