Discussion:
deorbiting /usr/lib/libstand.a, moving to sysboot
Warner Losh
2017-10-09 05:45:37 UTC
Permalink
I'd like to deorbit /usr/lib/stand.a and /usr/include/stand.h. These are
really parts of the boot loader with an unstable API and shouldn't be
installed into the system. It's really a private library to the boot loader.

Further, I'd like to move the whole shooting match into sys/boot/libsa (the
name change is on purpose to break anything that used libstand).

With these changes, one can cd sys/boot and do a make clean; make all and
have it not depend (too much) on the host computer. It will also let us
clean up a lot of the cut and paste and only half working implementations
of this idea we have now in a clean way.

We can also move the files in sys/boot/common that aren't actually part of
/boot/loader into libsa since there's would be no good reason to keep them
separate (though ultra-tiny loaders can cherry pick individual files, or
compile things in a custom way if that's really needed).

I've been cleaning up the sys/boot Makefile technical debt and this will go
a long way towards disentangling things (though there's a lot to do besides
these fixes).

History is against FreeBSD's practice, btw. We're the only BSD that
cross-threads like this (dragonfly may also, I haven't checked, but they
got it from us if so). NetBSD/OpenBSD have sys/lib/libsa for this stuff.
4.4bsd had each arch build its own libsa.a from files living in
sys/ARCH/stand. 4.2BSD did install things as /usr/lib/libsa.a, but it was
from inside of sys/stand and friends. v7 had a similar structure to 4.2BSD,
but with files living in different directories (they installed libsa.a into
/usr/lib, but again from sys directories). Since we've ditched the 4.4BSD
structure that NetBSD/OpenBSD evolved, I don't think it makes sense to go
the sys/lib route (since this is only used in the boot loader) Since
sys/boot programs know its internals so much that claims of independence
would be hard to sustain. Having it in sys/boot would also help the POLA
this change might cause by fixing a POLA breakage now (it fixes when
libstand API changes break the naive build, as well as ensuring hacks to
libstand will get built when a new boot loader is built). So we invented
the current situation, we can change it since nothing outside the boot
loader uses it (and if there are out-of-tree examples that use it, they
could be converted to the new structure).

Any objections to my moving forward with this (assume I'll help people fix
any in-flight work this disrupts)? I know the plans are a little vague, but
I was hoping to get rough consensus while I came up with woking code and go
all IETF old-school on the problem.

Warner
Poul-Henning Kamp
2017-10-09 06:50:08 UTC
Permalink
--------
Post by Warner Losh
I'd like to deorbit /usr/lib/stand.a and /usr/include/stand.h. These are
really parts of the boot loader with an unstable API and shouldn't be
installed into the system. It's really a private library to the boot loader.
Long overdue!
--
Poul-Henning Kamp | UNIX since Zilog Zeus 3.20
***@FreeBSD.ORG | TCP/IP since RFC 956
FreeBSD committer | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
Doug Rabson
2017-10-09 07:00:21 UTC
Permalink
I've been wanting to do that for years - go for it.
Post by Poul-Henning Kamp
--------
Post by Warner Losh
I'd like to deorbit /usr/lib/stand.a and /usr/include/stand.h. These are
really parts of the boot loader with an unstable API and shouldn't be
installed into the system. It's really a private library to the boot
loader.
Long overdue!
--
Poul-Henning Kamp | UNIX since Zilog Zeus 3.20
FreeBSD committer | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-arch
Ed Maste
2017-10-09 10:54:14 UTC
Permalink
Any objections to my moving forward with this [moving libstand to sys/boot/libsa]
Go ahead, IMO. Note that we already have sys/boot/libstand32 that
we'll want to include in the cleanup.
Warner Losh
2017-10-09 15:36:30 UTC
Permalink
Any objections to my moving forward with this [moving libstand to
sys/boot/libsa]
Go ahead, IMO. Note that we already have sys/boot/libstand32 that
we'll want to include in the cleanup.
And don't forget sys/boot/userboot/libstand too :)

Warner
Dimitry Andric
2017-10-09 15:57:10 UTC
Permalink
Post by Warner Losh
I'd like to deorbit /usr/lib/stand.a and /usr/include/stand.h. These are
really parts of the boot loader with an unstable API and shouldn't be
installed into the system. It's really a private library to the boot loader.
Though I completely agree with this, I am still interested in the
historical reasons for separating out this library for general userland
consumption. Were there any other parts of world that happened to use
libstand?

-Dimitry
Ian Lepore
2017-10-09 16:04:45 UTC
Permalink
Post by Dimitry Andric
Post by Warner Losh
I'd like to deorbit /usr/lib/stand.a and /usr/include/stand.h. These are
really parts of the boot loader with an unstable API and shouldn't be
installed into the system. It's really a private library to the boot loader.
Though I completely agree with this, I am still interested in the
historical reasons for separating out this library for general userland
consumption.  Were there any other parts of world that happened to use
libstand?
-Dimitry
There are out-of-tree users of libstand.  Perhaps not many, but a
couple times after doing something to libstand I've received emails
from people that thanked me for the enhancement and mentioned some non-
loader(8) use of the lib in passing.  (Unfortunately, I can't find any
of those mails now, they were from 2-3 years ago.)

-- Ian
Warner Losh
2017-10-09 16:09:41 UTC
Permalink
Post by Warner Losh
Post by Dimitry Andric
Post by Warner Losh
I'd like to deorbit /usr/lib/stand.a and /usr/include/stand.h. These
are
Post by Dimitry Andric
Post by Warner Losh
really parts of the boot loader with an unstable API and shouldn't be
installed into the system. It's really a private library to the boot
loader.
Post by Dimitry Andric
Though I completely agree with this, I am still interested in the
historical reasons for separating out this library for general userland
consumption. Were there any other parts of world that happened to use
libstand?
-Dimitry
There are out-of-tree users of libstand. Perhaps not many, but a
couple times after doing something to libstand I've received emails
from people that thanked me for the enhancement and mentioned some non-
loader(8) use of the lib in passing. (Unfortunately, I can't find any
of those mails now, they were from 2-3 years ago.)
They can email me and I'll help them convert over... :)

Warner
Ian Lepore
2017-10-09 16:15:59 UTC
Permalink
Post by Warner Losh
Post by Warner Losh
Post by Dimitry Andric
Post by Warner Losh
I'd like to deorbit /usr/lib/stand.a and /usr/include/stand.h. These
are
Post by Dimitry Andric
Post by Warner Losh
really parts of the boot loader with an unstable API and shouldn't be
installed into the system. It's really a private library to the boot
loader.
Post by Dimitry Andric
Though I completely agree with this, I am still interested in the
historical reasons for separating out this library for general userland
consumption.  Were there any other parts of world that happened to use
libstand?
-Dimitry
There are out-of-tree users of libstand.  Perhaps not many, but a
couple times after doing something to libstand I've received emails
from people that thanked me for the enhancement and mentioned some non-
loader(8) use of the lib in passing.  (Unfortunately, I can't find any
of those mails now, they were from 2-3 years ago.)
They can email me and I'll help them convert over... :)
Warner
Actually, I got distracted, then came back and hit Send too soon.  I
meant to ask "Will the library still be accessible to out of tree
users?", so that adjusting to the change will amount to fixing some
build breakage to adjust to a new location?

-- Ian
Warner Losh
2017-10-09 16:21:53 UTC
Permalink
Post by Dimitry Andric
Post by Warner Losh
Post by Warner Losh
Post by Dimitry Andric
Post by Warner Losh
I'd like to deorbit /usr/lib/stand.a and /usr/include/stand.h.
These
Post by Warner Losh
Post by Warner Losh
are
Post by Dimitry Andric
Post by Warner Losh
really parts of the boot loader with an unstable API and shouldn't
be
Post by Warner Losh
Post by Warner Losh
Post by Dimitry Andric
Post by Warner Losh
installed into the system. It's really a private library to the
boot
Post by Warner Losh
Post by Warner Losh
loader.
Post by Dimitry Andric
Though I completely agree with this, I am still interested in the
historical reasons for separating out this library for general
userland
Post by Warner Losh
Post by Warner Losh
Post by Dimitry Andric
consumption. Were there any other parts of world that happened to
use
Post by Warner Losh
Post by Warner Losh
Post by Dimitry Andric
libstand?
-Dimitry
There are out-of-tree users of libstand. Perhaps not many, but a
couple times after doing something to libstand I've received emails
from people that thanked me for the enhancement and mentioned some non-
loader(8) use of the lib in passing. (Unfortunately, I can't find any
of those mails now, they were from 2-3 years ago.)
They can email me and I'll help them convert over... :)
Warner
Actually, I got distracted, then came back and hit Send too soon. I
meant to ask "Will the library still be accessible to out of tree
users?", so that adjusting to the change will amount to fixing some
build breakage to adjust to a new location?
The proposal is to take it 100% internal and officially not support its use
outside the loader.

It's open source, so if you really want to use it, you can with some
effort. If there's one or two people, they can adjust. If there's lots,
then I can revisit the proposal. It would help to know who they were and
what, exactly, they used it for.

Warner
John Baldwin
2017-10-09 23:19:51 UTC
Permalink
Post by Warner Losh
Post by Dimitry Andric
Post by Warner Losh
Post by Warner Losh
Post by Dimitry Andric
Post by Warner Losh
I'd like to deorbit /usr/lib/stand.a and /usr/include/stand.h.
These
Post by Warner Losh
Post by Warner Losh
are
Post by Dimitry Andric
Post by Warner Losh
really parts of the boot loader with an unstable API and shouldn't
be
Post by Warner Losh
Post by Warner Losh
Post by Dimitry Andric
Post by Warner Losh
installed into the system. It's really a private library to the
boot
Post by Warner Losh
Post by Warner Losh
loader.
Post by Dimitry Andric
Though I completely agree with this, I am still interested in the
historical reasons for separating out this library for general
userland
Post by Warner Losh
Post by Warner Losh
Post by Dimitry Andric
consumption. Were there any other parts of world that happened to
use
Post by Warner Losh
Post by Warner Losh
Post by Dimitry Andric
libstand?
-Dimitry
There are out-of-tree users of libstand. Perhaps not many, but a
couple times after doing something to libstand I've received emails
from people that thanked me for the enhancement and mentioned some non-
loader(8) use of the lib in passing. (Unfortunately, I can't find any
of those mails now, they were from 2-3 years ago.)
They can email me and I'll help them convert over... :)
Warner
Actually, I got distracted, then came back and hit Send too soon. I
meant to ask "Will the library still be accessible to out of tree
users?", so that adjusting to the change will amount to fixing some
build breakage to adjust to a new location?
The proposal is to take it 100% internal and officially not support its use
outside the loader.
It's open source, so if you really want to use it, you can with some
effort. If there's one or two people, they can adjust. If there's lots,
then I can revisit the proposal. It would help to know who they were and
what, exactly, they used it for.
Note that one wrinkle is that libstand pulls in source bits from libc.
Historically it's been possible to checkout just sys/ and build sys/boot if
you had a matching world because libstand would be installed. This will
now require a full world checkout. That's probably not the end of the world
(and it is debatable if 'boot' even belongs in sys/ vs being a separate
top-level dir in the source tree). In general we try to make sys/ (for
the kernel) be self-contained and this would kind of break that, though I
think I'd be happy moving boot out of sys/ altogether to restore that
convention.
--
John Baldwin
Warner Losh
2017-10-09 23:37:05 UTC
Permalink
Post by John Baldwin
Post by Warner Losh
Post by Dimitry Andric
Post by Warner Losh
Post by Warner Losh
Post by Dimitry Andric
Post by Warner Losh
I'd like to deorbit /usr/lib/stand.a and /usr/include/stand.h.
These
Post by Warner Losh
Post by Warner Losh
are
Post by Dimitry Andric
Post by Warner Losh
really parts of the boot loader with an unstable API and shouldn't
be
Post by Warner Losh
Post by Warner Losh
Post by Dimitry Andric
Post by Warner Losh
installed into the system. It's really a private library to the
boot
Post by Warner Losh
Post by Warner Losh
loader.
Post by Dimitry Andric
Though I completely agree with this, I am still interested in the
historical reasons for separating out this library for general
userland
Post by Warner Losh
Post by Warner Losh
Post by Dimitry Andric
consumption. Were there any other parts of world that happened to
use
Post by Warner Losh
Post by Warner Losh
Post by Dimitry Andric
libstand?
-Dimitry
There are out-of-tree users of libstand. Perhaps not many, but a
couple times after doing something to libstand I've received emails
from people that thanked me for the enhancement and mentioned some non-
loader(8) use of the lib in passing. (Unfortunately, I can't find any
of those mails now, they were from 2-3 years ago.)
They can email me and I'll help them convert over... :)
Warner
Actually, I got distracted, then came back and hit Send too soon. I
meant to ask "Will the library still be accessible to out of tree
users?", so that adjusting to the change will amount to fixing some
build breakage to adjust to a new location?
The proposal is to take it 100% internal and officially not support its use
outside the loader.
It's open source, so if you really want to use it, you can with some
effort. If there's one or two people, they can adjust. If there's lots,
then I can revisit the proposal. It would help to know who they were and
what, exactly, they used it for.
Note that one wrinkle is that libstand pulls in source bits from libc.
Historically it's been possible to checkout just sys/ and build sys/boot if
you had a matching world because libstand would be installed. This will
now require a full world checkout. That's probably not the end of the world
(and it is debatable if 'boot' even belongs in sys/ vs being a separate
top-level dir in the source tree). In general we try to make sys/ (for
the kernel) be self-contained and this would kind of break that, though I
think I'd be happy moving boot out of sys/ altogether to restore that
convention.
Except that’s not been true for a long time
.

libstand32 pulls from lib/libstand, or did until recently.
userboot/libstand likewise.

sys/boot needs both to build. It’s historically been under sys since around V32 if my sleuthing is right, though there’s been expcetions (cf src/mdec in BSDs as late as 4.4BSD). v7 had them in /usr/mdec as well as src/cmd/standalone, so hoisting them a level would represent a return to the past. Though we’ve hopelessly mixed up the clean separation of low-level ‘boot1’ like loaders that knew just enough to find boot2 on the disk programs with our all singing, all dancing /boot/loader (which is more or less analogous to src/cmd/standalone, though at the time things weren’t packages together for size reasons). Not sure that it’s worth sorting that last stuff out at this late date though


That’s a long way of saying “I’d support moving it to src/$(result-of-bikeshed sys/boot)” but that’s not what I’m doing today.

Warner

P.S. Since there was no pushback, and evidence of a ton of pent up demand, I’ve gone ahead and pulled the trigger on the move.
Warner Losh
2017-10-09 16:06:24 UTC
Permalink
Post by Warner Losh
Post by Warner Losh
I'd like to deorbit /usr/lib/stand.a and /usr/include/stand.h. These are
really parts of the boot loader with an unstable API and shouldn't be
installed into the system. It's really a private library to the boot
loader.
Though I completely agree with this, I am still interested in the
historical reasons for separating out this library for general userland
consumption. Were there any other parts of world that happened to use
libstand?
Nope. As far as I can tell, it's never been used in FreeBSD.

Historically, it was use for the stand-alone programs used to bootstrap the
system (eg to copy the miniroot file from tape to disk swap area for the
install and disk formatting and partitioning).

Warner
Brooks Davis
2017-10-12 16:56:55 UTC
Permalink
Post by Warner Losh
I'd like to deorbit /usr/lib/stand.a and /usr/include/stand.h. These are
really parts of the boot loader with an unstable API and shouldn't be
installed into the system. It's really a private library to the boot loader.
Kicking it out of src/lib will be a good thing. It doesn't make sense
to build and install as part of the world and, for good reason, doesn't
follow normal rules. It was a pain to deal with for CHERI and I think
we've disabled it entierly.

-- Brooks
Warner Losh
2017-10-12 17:37:41 UTC
Permalink
Post by Warner Losh
Post by Warner Losh
I'd like to deorbit /usr/lib/stand.a and /usr/include/stand.h. These are
really parts of the boot loader with an unstable API and shouldn't be
installed into the system. It's really a private library to the boot
loader.
Kicking it out of src/lib will be a good thing. It doesn't make sense
to build and install as part of the world and, for good reason, doesn't
follow normal rules. It was a pain to deal with for CHERI and I think
we've disabled it entierly.
Yes. I've moved it into sys/boot. So now it's possible to hack on it w/o
crazy gymnastics. The BERI boot loaders do interesting things in the tree,
so I'm not surprised.

I'm contemplating moving src/sys/boot up to just src/boot (this mirrors in
some ways what old-school Unix did with src/cmd/standalone and src/mdec).
It would build, but not install, as part of buildworld. All libraries would
be internal / private to the build.

Warner
Lev Serebryakov
2017-10-18 14:48:47 UTC
Permalink
Post by Warner Losh
Post by Warner Losh
Post by Warner Losh
I'd like to deorbit /usr/lib/stand.a and /usr/include/stand.h. These are
really parts of the boot loader with an unstable API and shouldn't be
installed into the system. It's really a private library to the boot
loader.
Kicking it out of src/lib will be a good thing. It doesn't make sense
to build and install as part of the world and, for good reason, doesn't
follow normal rules. It was a pain to deal with for CHERI and I think
we've disabled it entierly.
Yes. I've moved it into sys/boot. So now it's possible to hack on it w/o
crazy gymnastics. The BERI boot loaders do interesting things in the tree,
so I'm not surprised.
I'm contemplating moving src/sys/boot up to just src/boot (this mirrors in
some ways what old-school Unix did with src/cmd/standalone and src/mdec).
It would build, but not install, as part of buildworld. All libraries would
be internal / private to the build.
New world can not be build with WITHOUT_CDDL / WITHOUT_ZFS. It fails in
"boot1".
--
// Lev Serebryakov
Loading...