callout_stop() return value

It seems to me a better solution would be to track the state / lifetime
of the resource separately rather than trying to infer the state of the
resource from the state of the callout as viewed through a semi-opaque
interface.

If the original thread that schedules the callout keeps enough
information about the resource to free it after cancelling, then it is
already maintaining some kind of sideband info about the resource, even
if it's just a pointer to be freed. Couldn't the callout routine
manipulate this resource tracking info (null out the pointer or set a
bool or whatever), then after cancelling you don't really care whether
the callout ran or not, you just examine the pointer/bool/whatever and
free or not based on that.

-- Ian

Post by Mark Johnston
Before r302350, our code happened to work; it could use the return value
of callout_stop() to correctly determine whether to release the
resource. Now, however, it cannot: the return value does not contain
sufficient information. In particular, if the return value is 0, we do
not know whether a future invocation of the callout was cancelled.
The resource's lifetime is effectively tied to the scheduling of the
callout, so I think it's preferable to address this by extending the
callout API rather than by adding some extra state to the structure
containing the callout. Does anyone have any opinions on adding a new
flag to _callout_stop_safe(), somewhat akin to CS_EXECUTING? When this
flag is specified, _callout_stop_safe() will return 1 if a future
invocation was cancelled, even if the callout was running at the time of
the call.
The patch below implements this suggestion, but I haven't yet tested
it and was wondering if anyone had opinions on how to handle this
scenario. If I end up going ahead with this approach, I'll be sure to
update the callout man page with a description of CS_EXECUTING and the
new flag. Thanks in advance.
diff --git a/sys/kern/kern_timeout.c b/sys/kern/kern_timeout.c
index 81b4a14ecf08..23c8c7dc3675 100644
--- a/sys/kern/kern_timeout.c
+++ b/sys/kern/kern_timeout.c
@@ -1183,6 +1183,8 @@ _callout_stop_safe(struct callout *c, int
flags, void (*drain)(void *))
int direct, sq_locked, use_lock;
int cancelled, not_on_a_list;

+ KASSERT((flags & (CS_EXECUTING | CS_DESCHEDULED)) !=
+     (CS_EXECUTING | CS_DESCHEDULED), ("invalid flags %#x",
flags));
if ((flags & CS_DRAIN) != 0)
WITNESS_WARN(WARN_GIANTOK | WARN_SLEEPOK, c->c_lock,
    "calling %s", __func__);
@@ -1391,7 +1393,7 @@ _callout_stop_safe(struct callout *c, int
flags, void (*drain)(void *))
KASSERT(!sq_locked, ("sleepqueue chain still
locked"));
cancelled = ((flags & CS_EXECUTING) != 0);
} else
- cancelled = 1;
+ cancelled = ((flags & CS_DESCHEDULED) == 0);

if (sq_locked)
sleepq_release(&cc_exec_waiting(cc, direct));
@@ -1422,6 +1424,8 @@ _callout_stop_safe(struct callout *c, int
flags, void (*drain)(void *))
} else {
TAILQ_REMOVE(&cc->cc_expireq, c,
c_links.tqe);
}
+ if ((flags & CS_DESCHEDULED) != 0)
+ cancelled = 1;
}
callout_cc_del(c, cc);
CC_UNLOCK(cc);
diff --git a/sys/sys/callout.h b/sys/sys/callout.h
index e4d91c69da3f..5142b3255122 100644
--- a/sys/sys/callout.h
+++ b/sys/sys/callout.h
@@ -70,6 +70,9 @@ struct callout_handle {
#define CS_DRAIN 0x0001 /* callout_drain(),
wait allowed */
#define CS_EXECUTING 0x0002 /* Positive return
value indicates that
  the callout was executing
*/
+#define CS_DESCHEDULED 0x0004 /* Positive
return value indicates that
+   a future invocation of the
callout was
+   cancelled. */

#ifdef _KERNEL
/*
_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-arch
g"

Mark Johnston

2018-05-02 15:42:37 UTC

It seems to me a better solution would be to track the state / lifetime
of the resource separately rather than trying to infer the state of the
resource from the state of the callout as viewed through a semi-opaque
interface.
If the original thread that schedules the callout keeps enough
information about the resource to free it after cancelling, then it is
already maintaining some kind of sideband info about the resource, even
if it's just a pointer to be freed. Couldn't the callout routine
manipulate this resource tracking info (null out the pointer or set a
bool or whatever), then after cancelling you don't really care whether
the callout ran or not, you just examine the pointer/bool/whatever and
free or not based on that.

I'd considered that. It's not quite as elegant a solution as you
suggest, since in my case the resource is embedded in an opaque
structure, so I need to add an extra field beside the callout to track
state that's already tracked by the callout subsystem. That plus the
fact that we have multiple instances of this bug make me want to fix it
in a general way, though I recognize that the callout API is already
overly complicated.

Konstantin Belousov

2018-05-02 16:24:12 UTC

It seems to me a better solution would be to track the state / lifetime
of the resource separately rather than trying to infer the state of the
resource from the state of the callout as viewed through a semi-opaque
interface.
If the original thread that schedules the callout keeps enough
information about the resource to free it after cancelling, then it is
already maintaining some kind of sideband info about the resource, even
if it's just a pointer to be freed. Couldn't the callout routine
manipulate this resource tracking info (null out the pointer or set a
bool or whatever), then after cancelling you don't really care whether
the callout ran or not, you just examine the pointer/bool/whatever and
free or not based on that.

I gave up on trying to get this fixed. r302350 was not the first time
the return value was broken.

I had to do r303426 exactly for this reason.

Mark Johnston

2018-05-02 16:40:02 UTC

Post by Konstantin Belousov

It seems to me a better solution would be to track the state / lifetime
of the resource separately rather than trying to infer the state of the
resource from the state of the callout as viewed through a semi-opaque
interface.
If the original thread that schedules the callout keeps enough
information about the resource to free it after cancelling, then it is
already maintaining some kind of sideband info about the resource, even
if it's just a pointer to be freed. Couldn't the callout routine
manipulate this resource tracking info (null out the pointer or set a
bool or whatever), then after cancelling you don't really care whether
the callout ran or not, you just examine the pointer/bool/whatever and
free or not based on that.

I forgot to add that in some cases, extra synchronization would be
needed to maintain this hypothetical flag. Specifically, some of these
callouts do not have an associated lock, so the callout handler doesn't
have an easy way to mark the resource as consumed.

Post by Konstantin Belousov
I gave up on trying to get this fixed. r302350 was not the first time
the return value was broken.
I had to do r303426 exactly for this reason.

What kind of fix did you envision? The problem seems to be that
callout_stop()'s return value simply does not contain enough information
for certain use cases.

Konstantin Belousov

2018-05-02 17:00:39 UTC

Post by Konstantin Belousov

It seems to me a better solution would be to track the state / lifetime
of the resource separately rather than trying to infer the state of the
resource from the state of the callout as viewed through a semi-opaque
interface.
If the original thread that schedules the callout keeps enough
information about the resource to free it after cancelling, then it is
already maintaining some kind of sideband info about the resource, even
if it's just a pointer to be freed. Couldn't the callout routine
manipulate this resource tracking info (null out the pointer or set a
bool or whatever), then after cancelling you don't really care whether
the callout ran or not, you just examine the pointer/bool/whatever and
free or not based on that.

Post by Konstantin Belousov
I gave up on trying to get this fixed. r302350 was not the first time
the return value was broken.
I had to do r303426 exactly for this reason.

What kind of fix did you envision? The problem seems to be that
callout_stop()'s return value simply does not contain enough information
for certain use cases.

The fix is really social, to make people who change the callout_stop() KPI
to care about non-tcp stack uses.

I tried to add a flag to _callout_stop_safe() to make return values useful
for sleepqueues, see r296320. But as I said, it was broken again and I
gave up.

Ian Lepore

2018-05-02 17:02:09 UTC

It seems to me a better solution would be to track the state / lifetime
of the resource separately rather than trying to infer the state of the
resource from the state of the callout as viewed through a semi-opaque
interface.
If the original thread that schedules the callout keeps enough
information about the resource to free it after cancelling, then it is
already maintaining some kind of sideband info about the resource, even
if it's just a pointer to be freed. Couldn't the callout routine
manipulate this resource tracking info (null out the pointer or set a
bool or whatever), then after cancelling you don't really care whether
the callout ran or not, you just examine the pointer/bool/whatever and
free or not based on that.

Wouldn't there be implied temporal synchronization? The callout will
free the resource and manipulate the sideband info to say so, knowing
that nothing else will modify it or even examine it at the same time
because the callout would be cancelled and drained before the resource
is manipulated by someone else. Conversely, the someone-else doesn't
examine or modify the resource or sideband info until the callout is
drained (whether cancelled or it just runs to normal completion).

Only if it were impossible to reliably cancel the callout and reach a
point where you know it's not running would there be any chance of a
race that needs explicit locking.

-- Ian

I gave up on trying to get this fixed. r302350 was not the first time
the return value was broken.
I had to do r303426 exactly for this reason.

What kind of fix did you envision? The problem seems to be that
callout_stop()'s return value simply does not contain enough information
for certain use cases.

Mark Johnston

2018-05-02 21:43:03 UTC

It seems to me a better solution would be to track the state / lifetime
of the resource separately rather than trying to infer the state of the
resource from the state of the callout as viewed through a semi-opaque
interface.
If the original thread that schedules the callout keeps enough
information about the resource to free it after cancelling, then it is
already maintaining some kind of sideband info about the resource, even
if it's just a pointer to be freed. Couldn't the callout routine
manipulate this resource tracking info (null out the pointer or set a
bool or whatever), then after cancelling you don't really care whether
the callout ran or not, you just examine the pointer/bool/whatever and
free or not based on that.

In one case, the callout isn't drained but simply stopped. Only if
callout_stop() returns 1 (i.e., we cancelled a future invocation AND the
callout wasn't running at the time of the call) do we release the
resource. However, if the call returns 0 we don't know whether to
release the resource.

In this case though, I think it's sufficient to check whether
callout_pending() is true before calling callout_stop(). I will spend
some time trying to fix these issues locally before trying to push this
topic further.

Post by Ian Lepore
Only if it were impossible to reliably cancel the callout and reach a
point where you know it's not running would there be any chance of a
race that needs explicit locking.

Hans Petter Selasky

2018-05-03 07:24:06 UTC