Discussion:
lua in zfs
Andriy Gapon
2017-09-28 11:40:29 UTC
Permalink
I am going to import the upstream ZFS Channel Programs change that adds Lua
interpreter to ZFS kernel code. In addition to pure Lua (no libraries) it also
has a number of functions to interact with ZFS. No general purpose kernel access.
--
Andriy Gapon
Warner Losh
2017-09-28 13:13:19 UTC
Permalink
Which version? Where in the tree? Can this be disabled?

Warner
Post by Andriy Gapon
I am going to import the upstream ZFS Channel Programs change that adds Lua
interpreter to ZFS kernel code. In addition to pure Lua (no libraries) it also
has a number of functions to interact with ZFS. No general purpose kernel access.
--
Andriy Gapon
_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-arch
Andriy Gapon
2017-09-28 15:16:16 UTC
Permalink
Post by Warner Losh
Which version? Where in the tree? Can this be disabled?
It's lua 5.2.4 at the moment. Its source code will reside in
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/lua. At run-time it will be in
zfs.ko or in kernel if it's built with ZFS option. For time being, it's not
possible to turn it off individually, but if you don't include ZFS or any CDDL
licensed code at all, then you won't have Lua too. But I think that it should
be possible to add a new knob specifically to compile ZFS without Lua, if
anybody wants it.
Post by Warner Losh
Post by Andriy Gapon
I am going to import the upstream ZFS Channel Programs change that adds Lua
interpreter to ZFS kernel code. In addition to pure Lua (no libraries) it also
has a number of functions to interact with ZFS. No general purpose kernel access.
--
Andriy Gapon
Warner Losh
2017-09-28 15:40:56 UTC
Permalink
Post by Andriy Gapon
Post by Warner Losh
Which version? Where in the tree? Can this be disabled?
It's lua 5.2.4 at the moment. Its source code will reside in
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/lua. At run-time it will be in
zfs.ko or in kernel if it's built with ZFS option. For time being, it's not
possible to turn it off individually, but if you don't include ZFS or any CDDL
licensed code at all, then you won't have Lua too. But I think that it should
be possible to add a new knob specifically to compile ZFS without Lua, if
anybody wants it.
OK. Any chance you could put it in sys/contrib/lua instead? The same place
that the boot loader lua project has it? The lua boot loader project is
using 5.3.0 it seems, is there any reason you couldn't use that instead?

I've been working on this in the background and was hoping to get it in
before 12.

Warner
Post by Andriy Gapon
Post by Warner Losh
Post by Andriy Gapon
I am going to import the upstream ZFS Channel Programs change that adds
Lua
Post by Warner Losh
Post by Andriy Gapon
interpreter to ZFS kernel code. In addition to pure Lua (no libraries)
it
Post by Warner Losh
Post by Andriy Gapon
also
has a number of functions to interact with ZFS. No general purpose
kernel
Post by Warner Losh
Post by Andriy Gapon
access.
--
Andriy Gapon
Warner Losh
2017-09-28 15:46:29 UTC
Permalink
Post by Warner Losh
Post by Andriy Gapon
Post by Warner Losh
Which version? Where in the tree? Can this be disabled?
It's lua 5.2.4 at the moment. Its source code will reside in
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/lua. At run-time it will be in
zfs.ko or in kernel if it's built with ZFS option. For time being, it's not
possible to turn it off individually, but if you don't include ZFS or any CDDL
licensed code at all, then you won't have Lua too. But I think that it should
be possible to add a new knob specifically to compile ZFS without Lua, if
anybody wants it.
OK. Any chance you could put it in sys/contrib/lua instead? The same place
that the boot loader lua project has it? The lua boot loader project is
using 5.3.0 it seems, is there any reason you couldn't use that instead?
I've been working on this in the background and was hoping to get it in
before 12.
Oh, and I forgot to add that I wasn't requesting a new knob at this time,
per se. Just wanted to know if it was there.

Warner
Post by Warner Losh
Warner
Post by Andriy Gapon
Post by Warner Losh
Post by Andriy Gapon
I am going to import the upstream ZFS Channel Programs change that
adds Lua
Post by Warner Losh
Post by Andriy Gapon
interpreter to ZFS kernel code. In addition to pure Lua (no
libraries) it
Post by Warner Losh
Post by Andriy Gapon
also
has a number of functions to interact with ZFS. No general purpose
kernel
Post by Warner Losh
Post by Andriy Gapon
access.
--
Andriy Gapon
Ian Lepore
2017-09-28 15:55:09 UTC
Permalink
Post by Warner Losh
Post by Warner Losh
Which version? Where in the tree? Can this be disabled?
It's lua 5.2.4 at the moment.  Its source code will reside in
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/lua.  At run-time it
will
be in
zfs.ko or in kernel if it's built with ZFS option.  For time being,
it's
not
possible to turn it off individually, but if you don't include ZFS
or any
CDDL
licensed code at all, then you won't have Lua too.  But I think
that it
should
be possible to add a new knob specifically to compile ZFS without Lua, if
anybody wants it.
OK. Any chance you could put it in sys/contrib/lua instead? The same place
that the boot loader lua project has it? The lua boot loader project is
using 5.3.0 it seems, is there any reason you couldn't use that instead?
I've been working on this in the background and was hoping to get it in
before 12.
Warner
Iirc, the big difference between 5.2.x and 5.3 is that the latter added
support for integers.  It seems like that would be a good thing, in the
kernel.

-- Ian
Warner Losh
2017-09-28 15:59:27 UTC
Permalink
Post by Ian Lepore
Post by Warner Losh
Post by Andriy Gapon
Post by Warner Losh
Which version? Where in the tree? Can this be disabled?
It's lua 5.2.4 at the moment. Its source code will reside in
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/lua. At run-time it
will
be in
zfs.ko or in kernel if it's built with ZFS option. For time being,
it's
not
possible to turn it off individually, but if you don't include ZFS
or any
CDDL
licensed code at all, then you won't have Lua too. But I think
that it
should
be possible to add a new knob specifically to compile ZFS without Lua, if
anybody wants it.
OK. Any chance you could put it in sys/contrib/lua instead? The same place
that the boot loader lua project has it? The lua boot loader project is
using 5.3.0 it seems, is there any reason you couldn't use that instead?
I've been working on this in the background and was hoping to get it in
before 12.
Warner
Iirc, the big difference between 5.2.x and 5.3 is that the latter added
support for integers. It seems like that would be a good thing, in the
kernel.
I have a dim memory that this is why the boot loader project used it, but I
may be misremembering.

In any event, so people can follow along at home, there's a freshly rebased
version of the lua-bootloader branch in
https://github.com/bsdimp/freebsd.git I just pushed.

Warner
Andriy Gapon
2017-09-28 16:44:16 UTC
Permalink
Post by Ian Lepore
Iirc, the big difference between 5.2.x and 5.3 is that the latter added
support for integers.  It seems like that would be a good thing, in the
kernel.
I am sure that the ZFS Lua represents numbers as integers and has no floating
point support at all.
--
Andriy Gapon
Andriy Gapon
2017-09-28 16:50:04 UTC
Permalink
Post by Andriy Gapon
Post by Ian Lepore
Iirc, the big difference between 5.2.x and 5.3 is that the latter added
support for integers.  It seems like that would be a good thing, in the
kernel.
I am sure that the ZFS Lua represents numbers as integers and has no floating
point support at all.
Some more info:
https://github.com/illumos/illumos-gate/blob/master/usr/src/uts/common/fs/zfs/lua/README.zfs
--
Andriy Gapon
Warner Losh
2017-09-28 17:07:35 UTC
Permalink
Post by Andriy Gapon
Post by Andriy Gapon
Post by Ian Lepore
Iirc, the big difference between 5.2.x and 5.3 is that the latter added
support for integers. It seems like that would be a good thing, in the
kernel.
I am sure that the ZFS Lua represents numbers as integers and has no
floating
Post by Andriy Gapon
point support at all.
https://github.com/illumos/illumos-gate/blob/master/usr/
src/uts/common/fs/zfs/lua/README.zfs
Reading this strongly suggests to me that we're going to have two versions
of lua in the tree then:

1) This version is a modified version of 5.2.4
2) Significant changes have been made to harden this interpreter, which
won't be in 5.3.0
3) A number of APIs have been removed, which makes it difficult to use as a
general purpose thing.

So I think that you should go ahead and import it. The level of work for my
request is very high, and significantly diverges us from upstream. It's
unfortunate that we'll have two versions. But this isn't really pure lua
but a variant maybe better described as lua-zfs.

Warner
John Baldwin
2017-09-28 17:52:44 UTC
Permalink
Post by Warner Losh
Post by Andriy Gapon
Post by Andriy Gapon
Post by Ian Lepore
Iirc, the big difference between 5.2.x and 5.3 is that the latter added
support for integers. It seems like that would be a good thing, in the
kernel.
I am sure that the ZFS Lua represents numbers as integers and has no
floating
Post by Andriy Gapon
point support at all.
https://github.com/illumos/illumos-gate/blob/master/usr/
src/uts/common/fs/zfs/lua/README.zfs
Reading this strongly suggests to me that we're going to have two versions
1) This version is a modified version of 5.2.4
2) Significant changes have been made to harden this interpreter, which
won't be in 5.3.0
3) A number of APIs have been removed, which makes it difficult to use as a
general purpose thing.
So I think that you should go ahead and import it. The level of work for my
request is very high, and significantly diverges us from upstream. It's
unfortunate that we'll have two versions. But this isn't really pure lua
but a variant maybe better described as lua-zfs.
Yes, that was my guess from Andriy's original e-mail. ZFS tends to be a bit
self-contained and includes its own version of zlib for example. In general
it seems to use its own version of 3rd party utils rather than sharing those
with other parts of the system.
--
John Baldwin
Warner Losh
2017-09-28 18:47:46 UTC
Permalink
Post by Warner Losh
Post by Warner Losh
Post by Andriy Gapon
Post by Andriy Gapon
Post by Ian Lepore
Iirc, the big difference between 5.2.x and 5.3 is that the latter
added
Post by Warner Losh
Post by Andriy Gapon
Post by Andriy Gapon
Post by Ian Lepore
support for integers. It seems like that would be a good thing, in
the
Post by Warner Losh
Post by Andriy Gapon
Post by Andriy Gapon
Post by Ian Lepore
kernel.
I am sure that the ZFS Lua represents numbers as integers and has no
floating
Post by Andriy Gapon
point support at all.
https://github.com/illumos/illumos-gate/blob/master/usr/
src/uts/common/fs/zfs/lua/README.zfs
Reading this strongly suggests to me that we're going to have two
versions
Post by Warner Losh
1) This version is a modified version of 5.2.4
2) Significant changes have been made to harden this interpreter, which
won't be in 5.3.0
3) A number of APIs have been removed, which makes it difficult to use
as a
Post by Warner Losh
general purpose thing.
So I think that you should go ahead and import it. The level of work for
my
Post by Warner Losh
request is very high, and significantly diverges us from upstream. It's
unfortunate that we'll have two versions. But this isn't really pure lua
but a variant maybe better described as lua-zfs.
Yes, that was my guess from Andriy's original e-mail. ZFS tends to be a bit
self-contained and includes its own version of zlib for example. In general
it seems to use its own version of 3rd party utils rather than sharing those
with other parts of the system.
Sure. I was vaguely aware of that... But I had to ask to make sure that we
couldn't use it, and also provide a 'paper trail' to those grepping the
tree / mailing lists for why...

Warner
Xin LI
2017-09-28 19:06:15 UTC
Permalink
Post by John Baldwin
Post by Warner Losh
Post by Andriy Gapon
Post by Andriy Gapon
Post by Ian Lepore
Iirc, the big difference between 5.2.x and 5.3 is that the latter added
support for integers. It seems like that would be a good thing, in the
kernel.
I am sure that the ZFS Lua represents numbers as integers and has no
floating
Post by Andriy Gapon
point support at all.
https://github.com/illumos/illumos-gate/blob/master/usr/
src/uts/common/fs/zfs/lua/README.zfs
Reading this strongly suggests to me that we're going to have two versions
1) This version is a modified version of 5.2.4
2) Significant changes have been made to harden this interpreter, which
won't be in 5.3.0
3) A number of APIs have been removed, which makes it difficult to use as a
general purpose thing.
So I think that you should go ahead and import it. The level of work for my
request is very high, and significantly diverges us from upstream. It's
unfortunate that we'll have two versions. But this isn't really pure lua
but a variant maybe better described as lua-zfs.
Yes, that was my guess from Andriy's original e-mail. ZFS tends to be a bit
self-contained and includes its own version of zlib for example. In general
it seems to use its own version of 3rd party utils rather than sharing those
with other parts of the system.
The zlib situation with ZFS is somewhat different, I think the goal
was to share one copy of zlib in the whole Solaris kernel, and as such
it was implemented as wrapper around regular zlib rather than a fork
like in ZFS. The only reason it was imported this way, if I
understood correctly, is that the zlib in FreeBSD kernel at the time
was too old.

Actually there was an attempt to unify zlib across the kernel a few
years ago, by the way:

https://svnweb.freebsd.org/base?view=revision&revision=199954

I'd be *really* happy to see we unify all zlib users into one,
hopefully with the base userland contrib/zlib. Maybe a good hackathon
goal?

Similar situation applies to the LZ4 compression too, but the change
in LZ4 is more intrusive than zlib: at the time it was added, there is
no other consumers of LZ4, and there was no way to change memory
allocator without making changes to LZ4 code itself (which was fixed
by more recent LZ4 versions). If in the future we decided to have LZ4
in other FreeBSD kernel components, it would be beneficial to unify
them too.

The lua-zfs is different as many pointed out, and I think the benefit
of importing the functionality is great enough to warrant having two
copies of code at the same time, at least as an interim solution.

Cheers,
Julian Elischer
2017-10-09 05:08:47 UTC
Permalink
Post by Andriy Gapon
Post by Andriy Gapon
Post by Ian Lepore
Iirc, the big difference between 5.2.x and 5.3 is that the latter added
support for integers.  It seems like that would be a good thing, in the
kernel.
I am sure that the ZFS Lua represents numbers as integers and has no floating
point support at all.
https://github.com/illumos/illumos-gate/blob/master/usr/src/uts/common/fs/zfs/lua/README.zfs
I suggest as a point to feed back to the originators that a name
change may be in order.. klua (kernel lua) perhaps.

Andriy Gapon
2017-09-28 16:42:57 UTC
Permalink
Post by Warner Losh
Which version? Where in the tree? Can this be disabled?
It's lua 5.2.4 at the moment.  Its source code will reside in
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/lua.  At run-time it will be in
zfs.ko or in kernel if it's built with ZFS option.  For time being, it's not
possible to turn it off individually, but if you don't include ZFS or any CDDL
licensed code at all, then you won't have Lua too.  But I think that it should
be possible to add a new knob specifically to compile ZFS without Lua, if
anybody wants it.
OK. Any chance you could put it in sys/contrib/lua instead? The same place that
the boot loader lua project has it? The lua boot loader project is using 5.3.0
it seems, is there any reason you couldn't use that instead?
I don't really know. I am just importing an upstream ZFS commit, because
without it we are getting more and more out of sync. I don't know if the ZFS
Lua has any special customizations, etc. I don't want to spend a lot of time on
investigating that and fighting with any possible quirks. If the consolidation
of Lua-s is possible and practical, then I hope that someone will do it.

Of course, there is another option which is not to import any Lua at all right
now and to stub out all calls into Lua-dependent routines with ENOTSUP. In that
case we will have plenty of time to work on importing a perfect Lua. On the
other hand, switching from one Lua to another and doing svn rm is not hard either.
I've been working on this in the background and was hoping to get it in before 12.
Post by Warner Losh
Post by Andriy Gapon
I am going to import the upstream ZFS Channel Programs change that adds Lua
interpreter to ZFS kernel code.  In addition to pure Lua (no libraries) it
also
has a number of functions to interact with ZFS.  No general purpose kernel
access.
--
Andriy Gapon
Warner Losh
2017-09-28 16:49:11 UTC
Permalink
Post by Andriy Gapon
Post by Andriy Gapon
Post by Warner Losh
Which version? Where in the tree? Can this be disabled?
It's lua 5.2.4 at the moment. Its source code will reside in
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/lua. At run-time it
will be in
Post by Andriy Gapon
zfs.ko or in kernel if it's built with ZFS option. For time being,
it's not
Post by Andriy Gapon
possible to turn it off individually, but if you don't include ZFS
or any CDDL
Post by Andriy Gapon
licensed code at all, then you won't have Lua too. But I think that
it should
Post by Andriy Gapon
be possible to add a new knob specifically to compile ZFS without
Lua, if
Post by Andriy Gapon
anybody wants it.
OK. Any chance you could put it in sys/contrib/lua instead? The same
place that
Post by Andriy Gapon
the boot loader lua project has it? The lua boot loader project is using
5.3.0
Post by Andriy Gapon
it seems, is there any reason you couldn't use that instead?
I don't really know. I am just importing an upstream ZFS commit, because
without it we are getting more and more out of sync. I don't know if the ZFS
Lua has any special customizations, etc. I don't want to spend a lot of time on
investigating that and fighting with any possible quirks. If the consolidation
of Lua-s is possible and practical, then I hope that someone will do it.
Of course, there is another option which is not to import any Lua at all right
now and to stub out all calls into Lua-dependent routines with ENOTSUP.
In that
case we will have plenty of time to work on importing a perfect Lua. On the
other hand, switching from one Lua to another and doing svn rm is not hard either.
I'm happy to work with you on this. I'd like to get lua bootloader into the
tree, and this gives me motivation to finish it up, even if it is a bit
rough around the edges... What's your timeline for bringing this stuff in?

Warner
Post by Andriy Gapon
Post by Andriy Gapon
I've been working on this in the background and was hoping to get it in
before 12.
Post by Andriy Gapon
Post by Warner Losh
Post by Andriy Gapon
I am going to import the upstream ZFS Channel Programs change
that adds Lua
Post by Andriy Gapon
Post by Warner Losh
Post by Andriy Gapon
interpreter to ZFS kernel code. In addition to pure Lua (no
libraries) it
Post by Andriy Gapon
Post by Warner Losh
Post by Andriy Gapon
also
has a number of functions to interact with ZFS. No general
purpose kernel
Post by Andriy Gapon
Post by Warner Losh
Post by Andriy Gapon
access.
--
Andriy Gapon
Andriy Gapon
2017-09-28 16:50:42 UTC
Permalink
Post by Warner Losh
I'm happy to work with you on this. I'd like to get lua bootloader into the
tree, and this gives me motivation to finish it up, even if it is a bit rough
around the edges... What's your timeline for bringing this stuff in?
I was aiming for this weekend.
--
Andriy Gapon
Loading...