Feedback on proposed loader changes

--------

Post by Warner Losh
So I'd like feedback on two questions: Should I kill the forth features I
oulined above? And should I make the build system build multiple loaders
with a link controlling the default?

I think you should just move forward and go for the end-stage
without too many temporary bandaids.

The loader is pretty decoupled from everything, so in case anybody
needs any of these Forth cornercases, they can use 11.X loader with
very little, if any, trouble.

--
Poul-Henning Kamp | UNIX since Zilog Zeus 3.20
***@FreeBSD.ORG | TCP/IP since RFC 956
FreeBSD committer | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.

Devin Teske

2018-02-01 16:58:24 UTC

Post by Poul-Henning Kamp
--------

I think you should just move forward and go for the end-stage
without too many temporary bandaids.
The loader is pretty decoupled from everything, so in case anybody
needs any of these Forth cornercases, they can use 11.X loader with
very little, if any, trouble.

As a person that both reviewed the GSoC code you are working with (in-depth; including a list of short-comings) and the most likely person to bring it up-to-par after it is committed, I have 2 opinions:

1. Please allow both boot systems for a while so that the lua-based menu can be made as feature full as the Forth menu. Example: submenus were added in Forth long after the GSoC lua project had ended

2. Please don’t force us to run Lua until I can code the new features

And as the principal author of the Forth menu since 9.0:

3. Please give me a way to run my code (at the very least until I can bring the Lua up to snuff; and if I can’t just let me run Forth in-perpetuity).

Interrupting boot1 so I can drive the system in the pre-boot Execution env is very important to me.
—
Devin

Warner Losh

2018-02-01 17:59:14 UTC

Post by Poul-Henning Kamp
--------
In message <CANCZdfoF4M1k=wOzueg0KQk9tRoQT-hO0SrB51wxv=-

Post by Warner Losh
So I'd like feedback on two questions: Should I kill the forth features

Post by Poul-Henning Kamp

Post by Warner Losh
oulined above? And should I make the build system build multiple loaders
with a link controlling the default?

I think you should just move forward and go for the end-stage
without too many temporary bandaids.
The loader is pretty decoupled from everything, so in case anybody
needs any of these Forth cornercases, they can use 11.X loader with
very little, if any, trouble.

As a person that both reviewed the GSoC code you are working with
(in-depth; including a list of short-comings) and the most likely person to
1. Please allow both boot systems for a while so that the lua-based menu
can be made as feature full as the Forth menu. Example: submenus were added
in Forth long after the GSoC lua project had ended

OK. The plan outlined does that. The lua code will be installed into /boot.
But it will be .lua, so no conflicts with .4th. And we start from
loader.lua not loader.rc.

Post by Devin Teske
2. Please don’t force us to run Lua until I can code the new features

Post by Devin Teske
3. Please give me a way to run my code (at the very least until I can
bring the Lua up to snuff; and if I can’t just let me run Forth
in-perpetuity).
Interrupting boot1 so I can drive the system in the pre-boot Execution env
is very important to me.

For !EFI, this is relatively easy. boot1 you can type /boot/loader_forth
instead of the default /boot/loader if the symlink changes and you want to
go back.

For EFI the answer is more complicated. boot1.efi is going away, so
loader.efi will move to the ESP in \efi\freebsd\loader.efi, but it's easy
enough to have multiple versions there (loader_lua.efi and
loader_forth.efi) and select via EFI Shell or EFI Env variables which one
you want should you need to fall back.

Does that answer your concerns?

Warner

John Baldwin

2018-02-06 00:38:31 UTC

For !EFI, this is relatively easy. boot1 you can type /boot/loader_forth
instead of the default /boot/loader if the symlink changes and you want to
go back.
For EFI the answer is more complicated. boot1.efi is going away, so
loader.efi will move to the ESP in \efi\freebsd\loader.efi, but it's easy
enough to have multiple versions there (loader_lua.efi and
loader_forth.efi) and select via EFI Shell or EFI Env variables which one
you want should you need to fall back.
Does that answer your concerns?

I think there is an even easier route for doing scripted loader development
now: Use bhyve with bhyveload. You get to run the loader as a plain userland
program (so you can use normal debuggers on it while it is running) while
being able to test all the menu features. You can even make it use the
builtin host: filesystem without requiring a disk image so you can modify the
lua files directly on the host that are getting pulled in for a fast
edit-debug cycle.

--
John Baldwin

Julian Elischer

2018-02-07 16:22:23 UTC

Post by Poul-Henning Kamp
--------
In message <CANCZdfoF4M1k=wOzueg0KQk9tRoQT-hO0SrB51wxv=-

Post by Warner Losh
So I'd like feedback on two questions: Should I kill the forth features

Post by Poul-Henning Kamp

Post by Warner Losh
oulined above? And should I make the build system build multiple loaders
with a link controlling the default?

I think you should just move forward and go for the end-stage
without too many temporary bandaids.
The loader is pretty decoupled from everything, so in case anybody
needs any of these Forth cornercases, they can use 11.X loader with
very little, if any, trouble.

OK. The plan outlined does that. The lua code will be installed into /boot.
But it will be .lua, so no conflicts with .4th. And we start from
loader.lua not loader.rc.

Post by Devin Teske
2. Please don’t force us to run Lua until I can code the new features

For !EFI, this is relatively easy. boot1 you can type /boot/loader_forth
instead of the default /boot/loader if the symlink changes and you want to
go back.
For EFI the answer is more complicated. boot1.efi is going away, so
loader.efi will move to the ESP in \efi\freebsd\loader.efi, but it's easy
enough to have multiple versions there (loader_lua.efi and
loader_forth.efi) and select via EFI Shell or EFI Env variables which one
you want should you need to fall back.

so, there are multiple loaders. zfsloader and loader for example.
how does this fit into the picture you are drawing? a symlink for each?

Post by Warner Losh
Does that answer your concerns?
Warner
_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-arch

Warner Losh

2018-02-07 16:40:01 UTC

Post by Poul-Henning Kamp
--------
In message <CANCZdfoF4M1k=wOzueg0KQk9tRoQT-hO0SrB51wxv=-
So I'd like feedback on two questions: Should I kill the forth features
I
oulined above? And should I make the build system build multiple loaders

Post by Warner Losh
with a link controlling the default?

I think you should just move forward and go for the end-stage
without too many temporary bandaids.
The loader is pretty decoupled from everything, so in case anybody
needs any of these Forth cornercases, they can use 11.X loader with
very little, if any, trouble.
As a person that both reviewed the GSoC code you are working with

(in-depth; including a list of short-comings) and the most likely person to
1. Please allow both boot systems for a while so that the lua-based menu
can be made as feature full as the Forth menu. Example: submenus were added
in Forth long after the GSoC lua project had ended
OK. The plan outlined does that. The lua code will be installed into

/boot.
But it will be .lua, so no conflicts with .4th. And we start from
loader.lua not loader.rc.
2. Please don’t force us to run Lua until I can code the new features

Post by Devin Teske
OK
3. Please give me a way to run my code (at the very least until I can
bring the Lua up to snuff; and if I can’t just let me run Forth
in-perpetuity).
Interrupting boot1 so I can drive the system in the pre-boot Execution env
is very important to me.

For !EFI, this is relatively easy. boot1 you can type /boot/loader_forth
instead of the default /boot/loader if the symlink changes and you want to
go back.
For EFI the answer is more complicated. boot1.efi is going away, so
loader.efi will move to the ESP in \efi\freebsd\loader.efi, but it's easy
enough to have multiple versions there (loader_lua.efi and
loader_forth.efi) and select via EFI Shell or EFI Env variables which one
you want should you need to fall back.

so, there are multiple loaders. zfsloader and loader for example.
how does this fit into the picture you are drawing? a symlink for each?

That's my plan. Of course, we shouldn't have a separate zfsloader and
loader, but due to other limitaitons we do. At least we don't have a
zfsloader.efi.

Warner

Ed Maste

2018-02-08 02:04:21 UTC

Post by Devin Teske
1. Please allow both boot systems for a while so that the lua-based menu can be made as feature full as the Forth menu.

Zak, my co-op student from last term did a bunch of this work; it's in
the lua-bootloader branch of https://github.com/ZakSN/freebsd. The
work term ended before Warner's rework was ready, but it should be
relatively straightforward to rebase it and begin review / refinement.

I agree we should support both variants for at least some time.

Warner Losh

2018-02-08 05:42:45 UTC

Post by Devin Teske
1. Please allow both boot systems for a while so that the lua-based menu

can be made as feature full as the Forth menu.
Zak, my co-op student from last term did a bunch of this work; it's in
the lua-bootloader branch of https://github.com/ZakSN/freebsd. The
work term ended before Warner's rework was ready, but it should be
relatively straightforward to rebase it and begin review / refinement.

I haven't changed the .lua code at all yet from what was in the github
branch that Rui did to forward port the GSoC svn branch. I have plans for a
lot of changes (since it's lame to call perform to call setenv when we can
just directly do a setenv function call, for example).

This code looks to be mergeable, so I'll do that as a separate commit. The
non-lua bits I have to pick and choose, since there's some conflicts
though...

Post by Devin Teske
I agree we should support both variants for at least some time.

Forth will be in 12, but not 13 is my plan. Lua will be in 12 and the
default. And thanks to Kyle Evans' heroic MFCs, there's a fighting chance
lua will be an off-by-default option in 11.2.

Warner

Baptiste Daroussin

2018-02-08 08:57:41 UTC

Post by Devin Teske
1. Please allow both boot systems for a while so that the lua-based menu

I haven't changed the .lua code at all yet from what was in the github
branch that Rui did to forward port the GSoC svn branch. I have plans for a
lot of changes (since it's lame to call perform to call setenv when we can
just directly do a setenv function call, for example).
This code looks to be mergeable, so I'll do that as a separate commit. The
non-lua bits I have to pick and choose, since there's some conflicts
though...

Post by Devin Teske
I agree we should support both variants for at least some time.

Forth will be in 12, but not 13 is my plan. Lua will be in 12 and the
default. And thanks to Kyle Evans' heroic MFCs, there's a fighting chance
lua will be an off-by-default option in 11.2.

I just want to say thank you to both of you!!! Can wait to be able to use lua
there!

Best regards,
Bapt

Eric McCorkle

2018-02-07 12:41:15 UTC

Might I suggest we integrate the GELI work before this goes in? It can
be added to loader as-is, and it works fine if you apply my standalone
loader patch (I have it deployed on a personal laptop).

I'm on the third major revision of this work (with countless smaller
rebases), and I'd like to not have to redo it a fourth time.

Basically, you'd need to commit some fixes to the efipart driver, the
UEFI KMS driver, the keybuf integration, and finally, the GELI driver
itself. I doubt this would interfere with 4th replacement, but I'd like
to not have this stuff get hit by stray changes.

Post by Warner Losh
Greetings,
As you may have read or guessed, I'm nearing the end game on integrating
lua into the boot loader from the GSoC a few years ago. I've tried to
resolve all the issues it brought up in libsa and other structural changes.
This has allowed lua to be imported unmodified, for example.
I've been trying to figure out how to handle the transition from forth to
lua and find myself with a few decisions that I should seek feedback on
since I'm at a crossroads.
The first one is that we have two sets of 4th words, both of which I wrote,
that don't fit neatly into the current build system. We have a bit of a
hack in place for both the pcibios-* and efi-* functions in 4th. The former
was something I did as a hack for Netflix that I judged at the time to be
more useful than it turned out to be (as far as I've been able to tell).
The latter turns out to be a road not taken (I'd planned originally on
implementing UEFI boot manager with 4th, but that turns out to be not
desirable even if 4th might be out the door). My plan is to simply retire
this stuff, along with pnp.4th which we've never installed. If I do this,
then I can build everything in the tree w/o regard to whether FORTH is on
or off, which dove tails nicely to my next question...
If no .o depends on the interpreter we're using (other than the ones that
implement the interpreter), then there'd be no technical barrier to
building multiple interpreters. So, I'd like to change to building both
the loader with forth and the one without, as well as installing both (as
loader_simple and loader_forth) with a symlink to the default. This would
allow people to switch, as well as provide a fallback for most systems
(uboot on FAT would be trickier, but we don't directly install those from
installworld, EFI on FAT would be as well, but there it will matter much
less shortly). This would allow me to roll out loader_lua when it's ready
and have it installed everywhere for people that want to take the plunge
and switch it when the time is ripe. This path would also leave the old
boot loaders around for people to interrupt boot1 with (EFI is another
matter, but I'm hoping efibootmgr wills solve that ball of wax).
So I'd like feedback on two questions: Should I kill the forth features I
oulined above? And should I make the build system build multiple loaders
with a link controlling the default?
Comments?
Warner
_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-arch

Warner Losh

2018-02-07 15:54:05 UTC

I'm afraid not. There's still unresolved issues in the efipart driver
changes in the reviews that tsoome raised, so it isn't ready. Lua is 3
commits away and is to the point where all the refinement of those three
changes is in code that's not in the tree yet. It goes in first, hopefully
this week. I doubt there will be any affect on your ongoing work.

Warner

Post by Eric McCorkle
Might I suggest we integrate the GELI work before this goes in? It can
be added to loader as-is, and it works fine if you apply my standalone
loader patch (I have it deployed on a personal laptop).
I'm on the third major revision of this work (with countless smaller
rebases), and I'd like to not have to redo it a fourth time.
Basically, you'd need to commit some fixes to the efipart driver, the
UEFI KMS driver, the keybuf integration, and finally, the GELI driver
itself. I doubt this would interfere with 4th replacement, but I'd like
to not have this stuff get hit by stray changes.

changes.

Post by Warner Losh
This has allowed lua to be imported unmodified, for example.
I've been trying to figure out how to handle the transition from forth to
lua and find myself with a few decisions that I should seek feedback on
since I'm at a crossroads.
The first one is that we have two sets of 4th words, both of which I

wrote,

Post by Warner Losh
that don't fit neatly into the current build system. We have a bit of a
hack in place for both the pcibios-* and efi-* functions in 4th. The

former

Post by Warner Losh
was something I did as a hack for Netflix that I judged at the time to be
more useful than it turned out to be (as far as I've been able to tell).
The latter turns out to be a road not taken (I'd planned originally on
implementing UEFI boot manager with 4th, but that turns out to be not
desirable even if 4th might be out the door). My plan is to simply retire
this stuff, along with pnp.4th which we've never installed. If I do

this,

Post by Warner Losh
then I can build everything in the tree w/o regard to whether FORTH is on
or off, which dove tails nicely to my next question...
If no .o depends on the interpreter we're using (other than the ones that
implement the interpreter), then there'd be no technical barrier to
building multiple interpreters. So, I'd like to change to building both
the loader with forth and the one without, as well as installing both (as
loader_simple and loader_forth) with a symlink to the default. This would
allow people to switch, as well as provide a fallback for most systems
(uboot on FAT would be trickier, but we don't directly install those from
installworld, EFI on FAT would be as well, but there it will matter much
less shortly). This would allow me to roll out loader_lua when it's ready
and have it installed everywhere for people that want to take the plunge
and switch it when the time is ripe. This path would also leave the old
boot loaders around for people to interrupt boot1 with (EFI is another
matter, but I'm hoping efibootmgr wills solve that ball of wax).
So I'd like feedback on two questions: Should I kill the forth features I
oulined above? And should I make the build system build multiple loaders
with a link controlling the default?
Comments?
Warner
_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-arch

_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-arch

Warner Losh

2018-02-07 16:04:12 UTC

Just re-read what I posted, and I need to followup. I should add at least
some of your other changes are somewhat close. I'll do those before my
pending efi boot loader changes. Any rework I have on them would be on me.
I'm not sure how to address tsoome's concerns, though, since they are
rather fundamental to the other work, so I can't fix that on the way in.
So it isn't quite as harsh as I think my original message sounds.

As for redoing things, I've just finished redoing ~15-years of sys/boot
neglect for stuff that wasn't done right the first time, so please be
patient with my pickiness.

Warner

Post by Warner Losh
I'm afraid not. There's still unresolved issues in the efipart driver
changes in the reviews that tsoome raised, so it isn't ready. Lua is 3
commits away and is to the point where all the refinement of those three
changes is in code that's not in the tree yet. It goes in first, hopefully
this week. I doubt there will be any affect on your ongoing work.
Warner

changes.

Post by Warner Losh
This has allowed lua to be imported unmodified, for example.
I've been trying to figure out how to handle the transition from forth

Post by Warner Losh
lua and find myself with a few decisions that I should seek feedback on
since I'm at a crossroads.
The first one is that we have two sets of 4th words, both of which I

wrote,

Post by Warner Losh
that don't fit neatly into the current build system. We have a bit of a
hack in place for both the pcibios-* and efi-* functions in 4th. The

former

Post by Warner Losh
was something I did as a hack for Netflix that I judged at the time to

Post by Warner Losh
more useful than it turned out to be (as far as I've been able to tell).
The latter turns out to be a road not taken (I'd planned originally on
implementing UEFI boot manager with 4th, but that turns out to be not
desirable even if 4th might be out the door). My plan is to simply

retire

Post by Warner Losh
this stuff, along with pnp.4th which we've never installed. If I do

this,

Post by Warner Losh
then I can build everything in the tree w/o regard to whether FORTH is

Post by Warner Losh
or off, which dove tails nicely to my next question...
If no .o depends on the interpreter we're using (other than the ones

that

Post by Warner Losh
implement the interpreter), then there'd be no technical barrier to
building multiple interpreters. So, I'd like to change to building both
the loader with forth and the one without, as well as installing both

(as

Post by Warner Losh
loader_simple and loader_forth) with a symlink to the default. This

would

Post by Warner Losh
allow people to switch, as well as provide a fallback for most systems
(uboot on FAT would be trickier, but we don't directly install those

from

Post by Warner Losh
installworld, EFI on FAT would be as well, but there it will matter much
less shortly). This would allow me to roll out loader_lua when it's

ready

Post by Warner Losh
and have it installed everywhere for people that want to take the plunge
and switch it when the time is ripe. This path would also leave the old
boot loaders around for people to interrupt boot1 with (EFI is another
matter, but I'm hoping efibootmgr wills solve that ball of wax).
So I'd like feedback on two questions: Should I kill the forth features

Post by Warner Losh
oulined above? And should I make the build system build multiple loaders
with a link controlling the default?
Comments?
Warner
_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-arch

_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-arch

Julian Elischer

2018-02-07 16:30:53 UTC

Post by Warner Losh
As for redoing things, I've just finished redoing ~15-years of sys/boot
neglect for stuff that wasn't done right the first time, so please be
patient with my pickiness.
Warner

the boot code in all its varying forms needs a full (re) writeup.
the boot(8) man page and the handbook description (chapter 12 I think)
are nowhere near complete.
If you are doing this work I'd ask if you can spend a few hours on
that too?
That way we can follow what the new code is doing better.
(and don't forget examples)
Especially concentrating on changes..

Eric McCorkle

2018-02-07 16:36:10 UTC

The keybuf system also needs a man page (that's on me as it was my invention).

Post by Warner Losh
As for redoing things, I've just finished redoing ~15-years of

sys/boot

Post by Warner Losh
neglect for stuff that wasn't done right the first time, so please be
patient with my pickiness.
Warner

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Eric McCorkle

2018-02-07 16:24:10 UTC

The issues tsoome raised with efipart have been dealt with in more recent updates. He just hasn't been able to respond yet.

Post by Eric McCorkle
Might I suggest we integrate the GELI work before this goes in? It

can

Post by Eric McCorkle
be added to loader as-is, and it works fine if you apply my

standalone

Post by Eric McCorkle
loader patch (I have it deployed on a personal laptop).
I'm on the third major revision of this work (with countless smaller
rebases), and I'd like to not have to redo it a fourth time.
Basically, you'd need to commit some fixes to the efipart driver, the
UEFI KMS driver, the keybuf integration, and finally, the GELI driver
itself. I doubt this would interfere with 4th replacement, but I'd

Post by Eric McCorkle
to not have this stuff get hit by stray changes.

Post by Warner Losh
Greetings,
As you may have read or guessed, I'm nearing the end game on

integrating

Post by Warner Losh
lua into the boot loader from the GSoC a few years ago. I've tried

Post by Warner Losh
resolve all the issues it brought up in libsa and other structural

changes.

Post by Warner Losh
This has allowed lua to be imported unmodified, for example.
I've been trying to figure out how to handle the transition from

forth to

Post by Warner Losh
lua and find myself with a few decisions that I should seek

feedback on

Post by Warner Losh
since I'm at a crossroads.
The first one is that we have two sets of 4th words, both of which

Post by Eric McCorkle
wrote,

Post by Warner Losh
that don't fit neatly into the current build system. We have a bit

of a

Post by Warner Losh
hack in place for both the pcibios-* and efi-* functions in 4th.

The

Post by Eric McCorkle
former

Post by Warner Losh
was something I did as a hack for Netflix that I judged at the time

to be

Post by Warner Losh
more useful than it turned out to be (as far as I've been able to

tell).

Post by Warner Losh
The latter turns out to be a road not taken (I'd planned originally

Post by Warner Losh
implementing UEFI boot manager with 4th, but that turns out to be

not

Post by Warner Losh
desirable even if 4th might be out the door). My plan is to simply

retire

Post by Warner Losh
this stuff, along with pnp.4th which we've never installed. If I

Post by Eric McCorkle
this,

Post by Warner Losh
then I can build everything in the tree w/o regard to whether FORTH

is on

Post by Warner Losh
or off, which dove tails nicely to my next question...
If no .o depends on the interpreter we're using (other than the

ones that

Post by Warner Losh
implement the interpreter), then there'd be no technical barrier to
building multiple interpreters. So, I'd like to change to building

both

Post by Warner Losh
the loader with forth and the one without, as well as installing

both (as

Post by Warner Losh
loader_simple and loader_forth) with a symlink to the default. This

would

Post by Warner Losh
allow people to switch, as well as provide a fallback for most

systems

Post by Warner Losh
(uboot on FAT would be trickier, but we don't directly install

those from

Post by Warner Losh
installworld, EFI on FAT would be as well, but there it will matter

much

Post by Warner Losh
less shortly). This would allow me to roll out loader_lua when it's

ready

Post by Warner Losh
and have it installed everywhere for people that want to take the

plunge

Post by Warner Losh
and switch it when the time is ripe. This path would also leave the

old

Post by Warner Losh
boot loaders around for people to interrupt boot1 with (EFI is

another

Post by Warner Losh
matter, but I'm hoping efibootmgr wills solve that ball of wax).
So I'd like feedback on two questions: Should I kill the forth

features I

Post by Warner Losh
oulined above? And should I make the build system build multiple

loaders