[HEADS UP] - OFED/RDMA stack update

[Navdeep Parhar]

+freebsd-arch@

Hi Meny,

Can you please post the KPI/KBI analysis that you generated to some
public location and provide a link here? A straight MFC would be a
major break of KPI/KBI in -STABLE and the options we're looking at are:

a) Ignore the breakage and let downstream consumers deal with the
fallout. This obviously isn't ideal in a -STABLE branch.

b) Provide compat shims to at least preserve the KPI. One challenge is
that the changes include functions with the same name but different
signature/behavior. See, for example, ib_create_cq in Meny's list once
he publishes it.

c) Have two versions of the OFED interfaces in 11-STABLE and not break
existing downstream consumers at all.

I've reached out to users that I know of and know will be affected.
If you use OFED and FreeBSD 11 this would be a good time to weigh
in with your thoughts, ideas, concerns etc..

Regards,
Navdeep

Hi,
We are currently working on MFC'ing the OFED/RDMA stack update
mentioned below to FreeBSD11-STABLE.
We already have working version in 'projects/bsd_rdma_4_9_stable_11'
(pending iWARP updates) and currently working on ULP integration.
Again, as always, for any concern/comments you might have, please
don't hesitate contacting us.
Regards,
Meny Yossefi,
Mellanox technologies
-----Original Message-----
From: Meny Yossefi
Sent: Monday, November 13, 2017 11:09 AM
Subject: [HEADS UP] - OFED/RDMA stack update
Hi,
This is to inform you that by end of this week we plan to merge the
OFED/RDMA stack update from the project - 'bsd_rdma_4_9' into 12-
CURRENT.
The update aligns the OFED common code and RDMA vendor drivers with
Linux v4.9.
We are still working on final modifications and build testing it prior
to submission.
For any concern/comments you might have, please don't hesitate
contacting us.
Regards,
Meny Yossefi,
Mellanox technologies
_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-infiniband
sd.org"

[Konstantin Belousov]

Post by Navdeep Parhar
Hi Meny,
Can you please post the KPI/KBI analysis that you generated to some
public location and provide a link here? A straight MFC would be a

I put the report at
https://kib.kiev.ua/kib/ibcore_11_to_11_merged_compat_report.html

Post by Navdeep Parhar
a) Ignore the breakage and let downstream consumers deal with the
fallout. This obviously isn't ideal in a -STABLE branch.
b) Provide compat shims to at least preserve the KPI. One challenge is
that the changes include functions with the same name but different
signature/behavior. See, for example, ib_create_cq in Meny's list once
he publishes it.

Project did handled similar issues already. One of the approaches is to
renname the ib_create_cq with the new signature to ib_create_cq_n12 and
check for (e.g.) _WANT_NEW_OFED symbol and to select one or another:
#ifdef _WANT_NEW_OFED
#define ib_create_cq(new args there) ib_create_cq_n21(new args there)
#else
#define ib_create_cq (ib_create_cq)
#endif

Then ULP that wants new KPI defines _WANT_NEW_OFED.

Post by Navdeep Parhar
c) Have two versions of the OFED interfaces in 11-STABLE and not break
existing downstream consumers at all.

It is possible to make them loadable simultaneously as modules, but it
is quite confusing to users, because Mellanox clearly wants mlx5_ib and
mlx4_ib to work only with new OFED, while cxgbe would use old OFED ?

Also, either we would need to mess with the ibcore.ko module name, or
with version. I am not sure that our module handling is robust enough
to make the version trick possible.

Post by Navdeep Parhar
I've reached out to users that I know of and know will be affected.
If you use OFED and FreeBSD 11 this would be a good time to weigh
in with your thoughts, ideas, concerns etc..

[Hans Petter Selasky]

Hi,

The bsd_rdma_4_9_stable_11 projects branch is close to being merged into
FreeBSD 11-stable. Mellanox plans to merge no later than 12:00 CEST TUE
20th of March 2018, unless objections are received.

A compatibility header file has been created, ib_verbs_compat.h, which
offers full source compatibility to existing OFED kernel applications,
as a response to the raised conserns. User-space compatibility is
maintained through library symbol versioning.

https://svnweb.freebsd.org/base/projects/bsd_rdma_4_9_stable_11/sys/ofed/include/rdma/ib_verbs_compat.h

An example client for this header file can be found here:

https://svnweb.freebsd.org/base/projects/bsd_rdma_4_9_stable_11/sys/contrib/rdma/krping_compat/

Currently the cxgb and cxgbe i-Warp drivers are not building, and will
be stubbed from the kernel build before the branch is merged, unless
Chelsio can add patches for these.

Here is a quick and dirty patch to make the bsd_rdma_4_9_stable_11

diff --git a/sys/modules/Makefile b/sys/modules/Makefile
index 6b005c854d7..b918a208f21 100644
--- a/sys/modules/Makefile
+++ b/sys/modules/Makefile
@@ -530,7 +530,7 @@ _txp= txp
.if ${MK_SOURCELESS_UCODE} != "no" && ${MACHINE_CPUARCH} != "arm" && \
${MACHINE_ARCH:C/mips(el)?/mips/} != "mips" && \
${MACHINE_ARCH} != "powerpc" && ${MACHINE_CPUARCH} != "riscv"
-_cxgbe= cxgbe
+#_cxgbe= cxgbe
.endif
.if ${MK_TESTS} != "no" || defined(ALL_MODULES)
@@ -554,7 +554,7 @@ _vpo= vpo
_sym= sym
# intr_disable() is a macro, causes problems
.if ${MK_SOURCELESS_UCODE} != "no"
-_cxgb= cxgb
+#_cxgb= cxgb
.endif
.endif

--HPS

[Navdeep Parhar]

Hold your horses. Do you have confirmation from the affected party that
the shims are adequate for them? I have been waiting for that before
looking at this branch.

Is the iw_cxgbe breakage a simple merge conflict as previously discussed
or do the shims require driver changes? If they don't then it should be
possible to drop the iw_cxgbe from head into this branch and it should
just work, is that correct?

Regards,
Navdeep

Post by Hans Petter Selasky
Hi,
The bsd_rdma_4_9_stable_11 projects branch is close to being merged into
FreeBSD 11-stable. Mellanox plans to merge no later than 12:00 CEST TUE 20th
of March 2018, unless objections are received.
A compatibility header file has been created, ib_verbs_compat.h, which
offers full source compatibility to existing OFED kernel applications, as a
response to the raised conserns. User-space compatibility is maintained
through library symbol versioning.
https://svnweb.freebsd.org/base/projects/bsd_rdma_4_9_stable_11/sys/ofed/include/rdma/ib_verbs_compat.h
https://svnweb.freebsd.org/base/projects/bsd_rdma_4_9_stable_11/sys/contrib/rdma/krping_compat/
Currently the cxgb and cxgbe i-Warp drivers are not building, and will be
stubbed from the kernel build before the branch is merged, unless Chelsio
can add patches for these.
Here is a quick and dirty patch to make the bsd_rdma_4_9_stable_11 branch

diff --git a/sys/modules/Makefile b/sys/modules/Makefile
index 6b005c854d7..b918a208f21 100644
--- a/sys/modules/Makefile
+++ b/sys/modules/Makefile
@@ -530,7 +530,7 @@ _txp= txp
.if ${MK_SOURCELESS_UCODE} != "no" && ${MACHINE_CPUARCH} != "arm" && \
${MACHINE_ARCH:C/mips(el)?/mips/} != "mips" && \
${MACHINE_ARCH} != "powerpc" && ${MACHINE_CPUARCH} != "riscv"
-_cxgbe= cxgbe
+#_cxgbe= cxgbe
.endif
.if ${MK_TESTS} != "no" || defined(ALL_MODULES)
@@ -554,7 +554,7 @@ _vpo= vpo
_sym= sym
# intr_disable() is a macro, causes problems
.if ${MK_SOURCELESS_UCODE} != "no"
-_cxgb= cxgb
+#_cxgb= cxgb
.endif
.endif

--HPS

[Hans Petter Selasky]

Post by Navdeep Parhar
Hold your horses. Do you have confirmation from the affected party that
the shims are adequate for them? I have been waiting for that before
looking at this branch.

Hi Navdeep,

Mellanox has received an API list from at least one party, and has taken
the action to support all the required APIs.

Post by Navdeep Parhar
Is the iw_cxgbe breakage a simple merge conflict as previously discussed
or do the shims require driver changes?

It is a merge conflict. The code already compiles in 12-current.

Post by Navdeep Parhar
If they don't then it should be
possible to drop the iw_cxgbe from head into this branch and it should
just work, is that correct?

Yes, it shouldn't be too hard to do and I would appreciate if Chelsio
could throw some resources at this ASAP. I believe the new code will
benefit everyone using Infiniband/RoCE and iWarp with FreeBSD.

--HPS

[Navdeep Parhar]

Post by Hans Petter Selasky

Post by Navdeep Parhar
Hold your horses. Do you have confirmation from the affected party that
the shims are adequate for them? I have been waiting for that before
looking at this branch.

Hi Navdeep,
Mellanox has received an API list from at least one party, and has taken the
action to support all the required APIs.

Post by Navdeep Parhar
Is the iw_cxgbe breakage a simple merge conflict as previously discussed
or do the shims require driver changes?

It is a merge conflict. The code already compiles in 12-current.

Ok that doesn't sound that bad. I'll take a look early this week.

Regards,
Navdeep

Post by Hans Petter Selasky

Post by Navdeep Parhar
If they don't then it should be
possible to drop the iw_cxgbe from head into this branch and it should
just work, is that correct?

Yes, it shouldn't be too hard to do and I would appreciate if Chelsio could
throw some resources at this ASAP. I believe the new code will benefit
everyone using Infiniband/RoCE and iWarp with FreeBSD.
--HPS
_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-arch

[Navdeep Parhar]

Post by Hans Petter Selasky

Hold your horses.  Do you have confirmation from the affected party that
the shims are adequate for them?  I have been waiting for that before
looking at this branch.

Hi Navdeep,
Mellanox has received an API list from at least one party, and has taken
the action to support all the required APIs.

Is the iw_cxgbe breakage a simple merge conflict as previously discussed
or do the shims require driver changes? 

It is a merge conflict. The code already compiles in 12-current.

I tried backing out r329391 and r329017 in a local copy and then was
able to merge r320418, r323082, and r326169 in that order without any
conflicts. But iw_cxgbe/cm.c still doesn't compile in the projects
branch because it has some socket code that relies on some of glebius's
changes available only in head (I checked with him and they aren't
MFC'able). I'm trying to figure out what to do about those.

And what about the cxgb breakage? Is there any simple way to make an
old style driver work with the new stack? T3 iw_cxgb in head was
retired before the ofed overhaul.

Regards,
Navdeep

[Hans Petter Selasky]

Post by Navdeep Parhar
And what about the cxgb breakage? Is there any simple way to make an
old style driver work with the new stack? T3 iw_cxgb in head was
retired before the ofed overhaul.

Hi,

The most simple way is to pull this driver from Linux 4.9, including
libcxgb for user-space, if possible, though I'm not sure about the
FreeBSD specific areas of iWarp and socket handling. I think you know
that part better than I do.

After pulling from Linux 4.9 you need to extend this change to cover the
iw_cxgb:

https://svnweb.freebsd.org/changeset/base/330508

Or can you retire iw_cxgb?

--HPS

[Meny Yossefi]

Hi Navdeep,

What's the current iWARP status on the project branch?
If it's ready we'd like to merge it to FreeBSD11-STABLE immediately.

-Meny


-----Original Message-----
From: Navdeep Parhar [mailto:***@gmail.com] On Behalf Of Navdeep Parhar
Sent: Tuesday, March 20, 2018 10:08 PM
To: Hans Petter Selasky <***@selasky.org>; Konstantin Belousov <***@freebsd.org>; 'freebsd-***@freebsd.org' <freebsd-***@freebsd.org>; freebsd-drivers <freebsd-***@mellanox.com>; Meny Yossefi <***@mellanox.com>; 'FreeBSD-***@FreeBSD.org' <FreeBSD-***@FreeBSD.org>; freebsd-arch <freebsd-***@freebsd.org>
Subject: Re: [HEADS UP] - OFED/RDMA stack update

Post by Hans Petter Selasky

Hold your horses.  Do you have confirmation from the affected party
that the shims are adequate for them?  I have been waiting for that
before looking at this branch.

Hi Navdeep,
Mellanox has received an API list from at least one party, and has
taken the action to support all the required APIs.

Is the iw_cxgbe breakage a simple merge conflict as previously
discussed or do the shims require driver changes?

It is a merge conflict. The code already compiles in 12-current.

I tried backing out r329391 and r329017 in a local copy and then was able to merge r320418, r323082, and r326169 in that order without any conflicts. But iw_cxgbe/cm.c still doesn't compile in the projects branch because it has some socket code that relies on some of glebius's changes available only in head (I checked with him and they aren't MFC'able). I'm trying to figure out what to do about those.

And what about the cxgb breakage? Is there any simple way to make an old style driver work with the new stack? T3 iw_cxgb in head was retired before the ofed overhaul.

Regards,
Navdeep

[Navdeep Parhar]

iw_cxgbe still doesn't build in the branch because of some socket
changes in head that are not in 11. I need to add the 11 equivalent of
the dequeue_socket code to iw_cxgbe to get it to compile, and then try
it to see if it actually works.

Regards,
Navdeep

Post by Hans Petter Selasky
Hi Navdeep,
What's the current iWARP status on the project branch?
If it's ready we'd like to merge it to FreeBSD11-STABLE immediately.
-Meny
-----Original Message-----
Sent: Tuesday, March 20, 2018 10:08 PM
Subject: Re: [HEADS UP] - OFED/RDMA stack update

Post by Hans Petter Selasky

Hold your horses.  Do you have confirmation from the affected party
that the shims are adequate for them?  I have been waiting for that
before looking at this branch.

Hi Navdeep,
Mellanox has received an API list from at least one party, and has
taken the action to support all the required APIs.

Is the iw_cxgbe breakage a simple merge conflict as previously
discussed or do the shims require driver changes?

It is a merge conflict. The code already compiles in 12-current.

I tried backing out r329391 and r329017 in a local copy and then was able to merge r320418, r323082, and r326169 in that order without any conflicts. But iw_cxgbe/cm.c still doesn't compile in the projects branch because it has some socket code that relies on some of glebius's changes available only in head (I checked with him and they aren't MFC'able). I'm trying to figure out what to do about those.
And what about the cxgb breakage? Is there any simple way to make an old style driver work with the new stack? T3 iw_cxgb in head was retired before the ofed overhaul.
Regards,
Navdeep