RFC: Automatically Reloading /etc/resolv.conf

[Eric van Gyzen]

I would like to change the libc resolver to automatically reload
/etc/resolv.conf when the latter changes. I would like to hear opinions
about the implementation. Broadly, I see two approaches.

== "stat" ==

When loading the file, record the mod time. Before each query, stat()
the file to see if it has changed.

Advantage: It uses no extra persistently allocated objects.

Disadvantage: It incurs a stat() on every query. I don't see this as a
major disadvantage, since the resolver already does a lot of work on
every query. (For example, it creates and destroys a kqueue and a socket.)

OpenBSD uses this approach. It also uses clock_gettime(CLOCK_MONOTONIC)
to rate-limit the stat() calls to one per several seconds.

== "kqueue" ==

When loading the file, open a kqueue and register for the appropriate
events. Before each query, check for kevents.

Advantage: The per-query overhead is fairly small.

Disadvantage: This would persistently allocate an open file and a
kqueue for every thread that ever uses the resolver, for the life of the
thread. This seems fairly expensive.

NetBSD uses this approach. It mitigates most of the space-cost by using
a shared pool of res_state objects, instead of one per thread [that uses
the resolver]. On each query, a thread allocates/borrows a res_state
from the pool, uses it, and returns it. So, the number of objects is
only the high water mark of the number of threads _concurrently_ issuing
resolver queries.

There are probably several variations on each theme, of course. I would
appreciate your thoughts on these approaches and others I missed, as
well as variations and details.

FYI, I'm leaning toward the "stat" approach.

Cheers,

Eric

[Jilles Tjoelker]

Post by Eric van Gyzen
I would like to change the libc resolver to automatically reload
/etc/resolv.conf when the latter changes. I would like to hear opinions
about the implementation. Broadly, I see two approaches.
== "stat" ==
When loading the file, record the mod time. Before each query, stat()
the file to see if it has changed.
Advantage: It uses no extra persistently allocated objects.
Disadvantage: It incurs a stat() on every query. I don't see this as a
major disadvantage, since the resolver already does a lot of work on
every query. (For example, it creates and destroys a kqueue and a socket.)
OpenBSD uses this approach. It also uses clock_gettime(CLOCK_MONOTONIC)
to rate-limit the stat() calls to one per several seconds.

This looks reasonable.

Nitpick: the resolver has used poll instead of kqueue for a while, but
that does not fundamentally change your argument.

Some glibc people think the extra stat may be too slow, though:
https://sourceware.org/bugzilla/show_bug.cgi?id=984

Post by Eric van Gyzen
== "kqueue" ==
When loading the file, open a kqueue and register for the appropriate
events. Before each query, check for kevents.
Advantage: The per-query overhead is fairly small.
Disadvantage: This would persistently allocate an open file and a
kqueue for every thread that ever uses the resolver, for the life of the
thread. This seems fairly expensive.

This sounds a bit scary in conjunction with code that bluntly closes
file descriptors it does not know about.

Also, kqueues do not inherit across fork, so the resolver needs some
sort of atfork handler.

Post by Eric van Gyzen
NetBSD uses this approach. It mitigates most of the space-cost by using
a shared pool of res_state objects, instead of one per thread [that uses
the resolver]. On each query, a thread allocates/borrows a res_state
from the pool, uses it, and returns it. So, the number of objects is
only the high water mark of the number of threads _concurrently_ issuing
resolver queries.

Is there code that depends on implicit per-thread res_state objects? If
so, this will break it.

Post by Eric van Gyzen
There are probably several variations on each theme, of course. I would
appreciate your thoughts on these approaches and others I missed, as
well as variations and details.
FYI, I'm leaning toward the "stat" approach.

This seems safe, and being able to rely on automatic /etc/resolv.conf
reloading can simplify application code considerably.

The NextBSD people have a more efficient alternative notify(3). Using
that, resolvconf(8) can cause a counter in shared memory to be
incremented, which can be detected efficiently by the resolver in each
process. The stat() approach need not wait for this, though.

[Eric van Gyzen]

Post by Jilles Tjoelker
Nitpick: the resolver has used poll instead of kqueue for a while, but
that does not fundamentally change your argument.

I was looking at a 10.2-RELEASE system. Indeed, I see it using poll on
head. Thanks.

Post by Jilles Tjoelker
https://sourceware.org/bugzilla/show_bug.cgi?id=984

I added simple code to call stat() on every query and ran a benchmark.
It reduces queries-per-second by 15.87%, so I definitely won't take that
approach.

Post by Jilles Tjoelker

Post by Eric van Gyzen
When loading the file, open a kqueue and register for the appropriate
events. Before each query, check for kevents.

This sounds a bit scary in conjunction with code that bluntly closes
file descriptors it does not know about.

That's true, but such code would break a variety of other facilities,
such as syslog. I'll keep this in mind.

Post by Jilles Tjoelker
Also, kqueues do not inherit across fork, so the resolver needs some
sort of atfork handler.

I hadn't thought of that. Thanks.

Post by Jilles Tjoelker

Post by Eric van Gyzen
NetBSD uses this approach. It mitigates most of the space-cost by using
a shared pool of res_state objects, instead of one per thread [that uses
the resolver]. On each query, a thread allocates/borrows a res_state
from the pool, uses it, and returns it. So, the number of objects is
only the high water mark of the number of threads _concurrently_ issuing
resolver queries.

Is there code that depends on implicit per-thread res_state objects? If
so, this will break it.

I'm just describing NetBSD's current approach. I don't plan to adopt
this in FreeBSD. To answer your question, though, I don't know if such
code exists.

Post by Jilles Tjoelker
The NextBSD people have a more efficient alternative notify(3). Using
that, resolvconf(8) can cause a counter in shared memory to be
incremented, which can be detected efficiently by the resolver in each
process. The stat() approach need not wait for this, though.

I didn't know about this. It sounds rather useful. Thanks!

Eric

[Warner Losh]

Post by Eric van Gyzen
I would like to change the libc resolver to automatically reload
/etc/resolv.conf when the latter changes. I would like to hear opinions
about the implementation. Broadly, I see two approaches.
== "stat" ==
When loading the file, record the mod time. Before each query, stat()
the file to see if it has changed.
Advantage: It uses no extra persistently allocated objects.
Disadvantage: It incurs a stat() on every query. I don't see this as a
major disadvantage, since the resolver already does a lot of work on
every query. (For example, it creates and destroys a kqueue and a socket.)
OpenBSD uses this approach. It also uses clock_gettime(CLOCK_MONOTONIC)
to rate-limit the stat() calls to one per several seconds.

Make sure you rate limit it. Calling stat at a high rate from multiple CPUs
exposes poor locking
scaleability and eats a lot of CPU. We had some setting wrong on NGINX at
work that lead to
large NGINX latency spikes because this contention was blocking the worker
threads. Correcting
the settings to cache the results of stat made this problem disappear.

Post by Eric van Gyzen
== "kqueue" ==
When loading the file, open a kqueue and register for the appropriate
events. Before each query, check for kevents.
Advantage: The per-query overhead is fairly small.
Disadvantage: This would persistently allocate an open file and a
kqueue for every thread that ever uses the resolver, for the life of the
thread. This seems fairly expensive.

Why does this follow? Can't you have a global one for the process?

Post by Eric van Gyzen
NetBSD uses this approach. It mitigates most of the space-cost by using
a shared pool of res_state objects, instead of one per thread [that uses
the resolver]. On each query, a thread allocates/borrows a res_state
from the pool, uses it, and returns it. So, the number of objects is
only the high water mark of the number of threads _concurrently_ issuing
resolver queries.n

What's the locking scaleability here when resolv.conf changes?

There are probably several variations on each theme, of course. I would

Post by Eric van Gyzen
appreciate your thoughts on these approaches and others I missed, as
well as variations and details.
FYI, I'm leaning toward the "stat" approach.

It sounds simpler to implement, but likely a higher overhead than the
kqueue approach. resolv.conf changes on an time scale measured in minutes
or hours for the mobile user, and on the scale of years for servers.
Polling on the scale of seconds (at least two orders of magnitude faster
than the rate of change) seems like a lot of extra work over the life of
resolv.conf.

Warner

[Eric van Gyzen]

Post by Warner Losh
Make sure you rate limit it.

Agreed. As I just wrote in my other reply, calling stat() on every
query reduces queries-per-second by 15.87%. Rate-limiting to one stat()
every ten seconds fixes it. That is, by manually running three or four
benchmarks with and without my changes, I see no real difference. This
was on amd64 where the cost of clock_gettime() is trivialized by vdso.

Post by Warner Losh

Post by Eric van Gyzen
Disadvantage: This would persistently allocate an open file and a
kqueue for every thread that ever uses the resolver, for the life of the
thread. This seems fairly expensive.

Why does this follow? Can't you have a global one for the process?

Well, gee, that seems like a good compromise, now that you mention it.
;-) Seriously, I simply didn't consider that.

Given that the rate-limited stat() approach is so cheap, I wonder if
it's worth the trouble to try the global kqueue approach.

Post by Warner Losh

Post by Eric van Gyzen
NetBSD uses this approach. It mitigates most of the space-cost by using
a shared pool of res_state objects, instead of one per thread [that uses
the resolver]. On each query, a thread allocates/borrows a res_state
from the pool, uses it, and returns it. So, the number of objects is
only the high water mark of the number of threads _concurrently_ issuing
resolver queries.n

What's the locking scaleability here when resolv.conf changes?

The mutex is only held during one or two SLIST operations, so the
locking is independent of resolv.conf changing. Maybe I misunderstood
your question?

Post by Warner Losh

Post by Eric van Gyzen
FYI, I'm leaning toward the "stat" approach.

It sounds simpler to implement, but likely a higher overhead than the
kqueue approach. resolv.conf changes on an time scale measured in minutes
or hours for the mobile user, and on the scale of years for servers.
Polling on the scale of seconds (at least two orders of magnitude faster
than the rate of change) seems like a lot of extra work over the life of
resolv.conf.

On the other hand, when changes _are_ needed, you don't want to restart
critical services in order to make them effective. (This is my
motivation, in fact.) It would be trivial to add a "no-reload" option
so performance-sensitive users can turn it off. In fact, I half
expected this request from someone focused on embedded work, such as
yourself. ;-)

Eric

[Hajimu UMEMOTO]

Hi,

On Wed, 7 Oct 2015 16:02:39 -0500

vangyzen> I would like to change the libc resolver to automatically reload
vangyzen> /etc/resolv.conf when the latter changes. I would like to hear opinions
vangyzen> about the implementation.

When I proposed it in the past, it ended up with dns/nss_resinit port.
Please take a look at it for reference.

Sincerely,

--
Hajimu UMEMOTO
***@mahoroba.org ***@FreeBSD.org
http://www.mahoroba.org/~ume/

[Eric van Gyzen]

Post by Hajimu UMEMOTO
vangyzen> I would like to change the libc resolver to automatically reload
vangyzen> /etc/resolv.conf when the latter changes. I would like to hear opinions
vangyzen> about the implementation.
When I proposed it in the past, it ended up with dns/nss_resinit port.
Please take a look at it for reference.

Nice. That's exactly what I want to do in libc. Doing it in nss is a
clever way to avoid changing libc. Thanks for the pointer.

Eric

[Eric van Gyzen]

Post by Eric van Gyzen
I would like to change the libc resolver to automatically reload
/etc/resolv.conf when the latter changes.

I implemented and tested the rate-limited stat(2) approach.
I would be grateful for a review:

https://reviews.freebsd.org/D3867

Eric