Posix shared memory problem

[Lothar Scholz]

Hello,

Thanks for solving the posix semaphore problem. But with shared memory
there comes the next issue:

int main() {
int m;
shm_unlink("/barfoo");
m = shm_open("/barfoo", O_RDWR|O_CREAT|O_EXCL, S_IRWXU);
if (m == 1) perror("shm_open error");
}

i always get permission denied error, and i tried many values
for flags and mode? I can only get this working as root but not
as a normal user.

[Jilles Tjoelker]

Post by Lothar Scholz
Thanks for solving the posix semaphore problem. But with shared memory
int main() {
int m;
shm_unlink("/barfoo");
m = shm_open("/barfoo", O_RDWR|O_CREAT|O_EXCL, S_IRWXU);
if (m == 1) perror("shm_open error");
}
i always get permission denied error, and i tried many values
for flags and mode? I can only get this working as root but not
as a normal user.

shm_open/shm_unlink refer to the filesystem; they are fairly direct
wrappers around open and unlink.

POSIX suggests making the pathname a configuration option;
alternatively, using a directory for temporary files such as /tmp could
work.

[Ed Schouten]

Hi Jilles,

Post by Jilles Tjoelker
shm_open/shm_unlink refer to the filesystem; they are fairly direct
wrappers around open and unlink.

Achtung: this is no longer the case in CURRENT if I remember correctly.

[Lothar Scholz]

Hello Jilles,

Post by Lothar Scholz
Thanks for solving the posix semaphore problem. But with shared memory
int main() {
int m;
shm_unlink("/barfoo");
m = shm_open("/barfoo", O_RDWR|O_CREAT|O_EXCL, S_IRWXU);
if (m == 1) perror("shm_open error");
}
i always get permission denied error, and i tried many values
for flags and mode? I can only get this working as root but not
as a normal user.

JT> shm_open/shm_unlink refer to the filesystem; they are fairly direct
JT> wrappers around open and unlink.

Question is where are they stored?
In Linux it is "/dev/shm".

In my case it looks like the directory for shm_open files has some
wrong access rights so that a normal user can't generate it.

JT> POSIX suggests making the pathname a configuration option;
JT> alternatively, using a directory for temporary files such as /tmp could
JT> work.

I will try this hack soon if nobody comes up with a solution.

[Garrett Wollman]

Post by Lothar Scholz
JT> shm_open/shm_unlink refer to the filesystem; they are fairly direct
JT> wrappers around open and unlink.
Question is where are they stored?

In the fileststem, in the path that you specify. They are just
ordinary files.

There was some thought that this was a bad (or at least
not-like-Linux) way of implementing this feature, so I believe
more-recent versions of FreeBSD do it differently. When I wrote this
code, I could not see any reason for the "path" argument to be
interpreted differently from any other path.

-GAWollman

[Lothar Scholz]

Hello Garrett,

Sunday, May 10, 2009, 7:00:16 AM, you wrote:

GW> In

Post by Lothar Scholz
JT> shm_open/shm_unlink refer to the filesystem; they are fairly direct
JT> wrappers around open and unlink.
Question is where are they stored?

GW> In the fileststem, in the path that you specify. They are just
GW> ordinary files.

GW> There was some thought that this was a bad (or at least
GW> not-like-Linux) way of implementing this feature, so I believe
GW> more-recent versions of FreeBSD do it differently. When I wrote this
GW> code, I could not see any reason for the "path" argument to be
GW> interpreted differently from any other path.

Oh thats a very very bad idea.

First of all you can't use '/' if you want stay portable.
It is also just a maximum of 13 char long (says the FreeBSD 6.X man page)
and usually you now pass names like
"com.mycompany.myproduct.mypurpose" as names to prevent namespace
collisons.

The path has nothing to do with the filesystem, it's a separate
namespace. Let alone that semaphores and shared memory already use the
same namespace is something i didn't expect on Linux.

Now it is clear where my problem is and i go to a mmap to a $HOME/.
file. Not nice but if anybody gives a shit about compatibility
(backward and to other systems before implementing stuff) it is
the only way.

[Sujit K M]

Post by Lothar Scholz
Now it is clear where my problem is and i go to a mmap to a $HOME/.
file. Not nice but if anybody gives a shit about compatibility
(backward and to other systems before implementing stuff) it is
the only way.

i donot understand why this is an compatility issue. Just use /path/to/file.
say /proc/shm/shm[0-9]+[a-z]+[0-9]+

[Garrett Wollman]

Post by Lothar Scholz
First of all you can't use '/' if you want stay portable.

The Standard says otherwise.

Post by Lothar Scholz
It is also just a maximum of 13 char long (says the FreeBSD 6.X man page)

Not in the manual page I have, and the Standard says otherwise.

Post by Lothar Scholz
The path has nothing to do with the filesystem, it's a separate
namespace.

Again, the Standard says otherwise (or rather, it says that this is an
implementation option). To quote the 2001 edition of the standard
(XSH6 page 1313):

It is unspecified whether the name appears in the file system
and is visible to other functions that take pathnames as
arguments. The name argument conforms to the construction
rules for a pathname.

-GAWollman

[Lothar Scholz]

Hello Garrett,

Sunday, May 10, 2009, 5:54:31 PM, you wrote:

GW> <<On Sun, 10 May 2009 07:57:06 +0200, Lothar Scholz

Post by Lothar Scholz
First of all you can't use '/' if you want stay portable.

GW> The Standard says otherwise.

It's not a standard think. Read about the real world programming
hints. You see recommendations to only use a starting '/'

Post by Lothar Scholz
It is also just a maximum of 13 char long (says the FreeBSD 6.X man page)

GW> Not in the manual page I have, and the Standard says otherwise.

This time you are right. It was about named semaphores and there
the limit seems to be removed - it was ridiculous low anyway.

GW> Again, the Standard says otherwise (or rather, it says that this is an
GW> implementation option). To quote the 2001 edition of the standard
GW> (XSH6 page 1313):

GW> It is unspecified whether the name appears in the file system
GW> and is visible to other functions that take pathnames as
GW> arguments. The name argument conforms to the construction
GW> rules for a pathname.

Thats why the man page calls this parameter 'name' and not 'path'.

Some idiots started to think about this as a file path. But it isn't
and it shouldn't. Thats what this spec is saying in the typical commitee
polite form when some members made a mistake but are to important to
be blamed in public.

So this needs to be fixed.

If i have to find a useable filefile location anyway the whole function does
not make any sense, then i can directly use mmap. The purpose is to
have a unique name (and in 2009 it is an URI not a file path). Thats
how serious non kiddy operating systems are doing like
Linux/Solaris/MacOSX-Darwin/HP-UX.

And i guess also the accounting functions are wrong then. shm_open
does not open a file so the (internal used) file should not add to the
file space quota but to the memory allocation quota.

[Dag-Erling Smørgrav]

Post by Lothar Scholz
Some idiots started to think about this as a file path. But it isn't
and it shouldn't. Thats what this spec is saying in the typical commitee
polite form when some members made a mistake but are to important to
be blamed in public.

You are wrong. The standard says what it says specifically because it
makes it possible to implement semaphores and shared memory in terms of
file operations. I've been there and done that.

Post by Lothar Scholz
So this needs to be fixed.

You've already been told that it *has* been fixed (or rather changed,
since it was not broken to begin with) in head.

Post by Lothar Scholz
If i have to find a useable filefile location anyway the whole function does
not make any sense, then i can directly use mmap. The purpose is to
have a unique name (and in 2009 it is an URI not a file path). Thats
how serious non kiddy operating systems are doing like
Linux/Solaris/MacOSX-Darwin/HP-UX.

Insulting the developers will get you nowhere.

DES

[Sujit K M]

Any sort of frustration here?

Post by Lothar Scholz
Some idiots started to think about this as a file path. But it isn't
and it shouldn't. Thats what this spec is saying in the typical commitee
polite form when some members made a mistake but are to important to
be blamed in public.

What ever the Idiots are saying is correct. Read up some decent Unix manual.

Post by Lothar Scholz
So this needs to be fixed.

What needs to be fixed? Could you be more specific?

Post by Lothar Scholz
If i have to find a useable filefile location anyway the whole function does
not make any sense, then i can directly use mmap. The purpose is to
have a unique name (and in 2009 it is an URI not a file path). Thats
how serious non kiddy operating systems are doing like
Linux/Solaris/MacOSX-Darwin/HP-UX.

Try using these giant, great, long lasting things.

Post by Lothar Scholz
And i guess also the accounting functions are wrong then. shm_open
does not open a file so the (internal used) file should not add to the
file space quota but to the memory allocation quota.

I think you need a check up. You seem to be contradicting what ever you said
before.

[Lothar Scholz]

Hello Sujit,

Monday, May 11, 2009, 1:10:27 PM, you wrote:
SKM> What ever the Idiots are saying is correct. Read up some decent Unix manual.
I read and even better i ported and finally yes i thought about the
function, why it is there and why it is better then System V IPC.

SKM> What needs to be fixed? Could you be more specific?

That the name argument is just that "a name" (in its own
name space) not a path.

[Dag-Erling Smørgrav]

Post by Sujit K M
What needs to be fixed? Could you be more specific?

That the name argument is just that "a name" (in its own name space)
not a path.

Allow me to quote from the SUSv3 again:

DESCRIPTION

The shm_open() function shall establish a connection between a shared
memory object and a file descriptor. [...] The name argument points
to a string naming a shared memory object. It is unspecified whether
the name appears in the file system and is visible to other functions
that take pathnames as arguments. The name argument conforms to the
construction rules for a pathname. [...]

RATIONALE

[...]

Note that such shared memory objects can actually be implemented as
mapped files. In both cases, the size can be set after the open using
ftruncate(). The shm_open() function itself does not create a shared
object of a specified size because this would duplicate an extant
function that set the size of an object referenced by a file
descriptor.

On implementations where memory objects are implemented using the
existing file system, the shm_open() function may be implemented using
a macro that invokes open(), and the shm_unlink() function may be
implemented using a macro that invokes unlink().

[...]

DES

[Garrett Wollman]

Post by Lothar Scholz
Some idiots started to think about this as a file path. But it isn't
and it shouldn't.

Actually, it really should be. Ask a security person or a
virtualization person to explain why an unnecessary multiplicity of
namespaces is a bad idea.

Post by Lothar Scholz
If i have to find a useable filefile location anyway the whole
function does not make any sense, then i can directly use mmap.

But of course you won't get the same behavior, because open()/mmap()
guarantees that the backing store will get updated. That's why
there's a separate interface.

-GAWollman

[Robert Watson]

Post by Lothar Scholz
Some idiots started to think about this as a file path. But it isn't
and it shouldn't.

Actually, it really should be. Ask a security person or a virtualization
person to explain why an unnecessary multiplicity of namespaces is a bad
idea.

Despite having been partly responsible for the new POSIX shm code in 8.x that
removes file system namespace use for POSIX shm, I strongly agree with your
statement.

The hierarchal and access-controlled structure of the file system namespace is
a key feature that makes it preferable to the plethora of other weird global
namespaces arriving with various new IPC models. A hierarchal namespace with
access control allows reliable delegation of portions of the namespace -- for
example, administrators can authorize a user to use any name in
"/home/username" without worrying that users will spoof each others services
based on application start order, crashes, etc. The existence of additional
flat namespaces, such as used by System V IPC, POSIX shm, POSIX sem, etc, is
quite problematic from this perspective, and significantly increases the risk
of vulnerability.

Robert N M Watson
Computer Laboratory
University of Cambridge

[John Baldwin]

Post by Lothar Scholz
Some idiots started to think about this as a file path. But it isn't
and it shouldn't. Thats what this spec is saying in the typical commitee
polite form when some members made a mistake but are to important to
be blamed in public.

Hmm, why don't you head down to your local bookstore and buy a copy
of "Solaris Internals" and then come back and explain to us all how the
developers of Solaris are idiots.

[Nikos Ntarmos]

Post by Garrett Wollman

Post by Lothar Scholz
JT> shm_open/shm_unlink refer to the filesystem; they are fairly direct
JT> wrappers around open and unlink.
Question is where are they stored?

In the fileststem, in the path that you specify. They are just
ordinary files.
There was some thought that this was a bad (or at least
not-like-Linux) way of implementing this feature, so I believe
more-recent versions of FreeBSD do it differently. When I wrote this
code, I could not see any reason for the "path" argument to be
interpreted differently from any other path.

FWIW the test code in the original email still fails even if an absolute
path is used as a sem name, ie:
sem_t *s = sem_open("/path/to/foobar", O_CREAT | O_EXCL, S_IWUSR, 0);
with /path/to/foobar pointing to a user writable directory, segfaults
with "invalid system call". Note that the error is not printed by
perror(3) but by the system itself. A backtrace of the resulting core
shows that the problem is burried deep in ksem_open():

***@ace:~% ./ts
zsh: invalid system call (core dumped) ./ts
***@ace:~% gdb -q ./ts ts.core
Core was generated by `ts'.
Program terminated with signal 12, Bad system call.
Reading symbols from /lib/libc.so.7...done.
Loaded symbols for /lib/libc.so.7
Reading symbols from /libexec/ld-elf.so.1...done.
Loaded symbols for /libexec/ld-elf.so.1
#0 0x280c214b in ksem_open () from /lib/libc.so.7
(gdb) bt
#0 0x280c214b in ksem_open () from /lib/libc.so.7
#1 0x280b78fc in sem_open () from /lib/libc.so.7
#2 0x080484e5 in main () at test-sem.c:7
(gdb)

This is on i386/7.2-RELEASE.

Cheers.

\n\n

[Dag-Erling Smørgrav]

Post by Nikos Ntarmos
FWIW the test code in the original email still fails even if an absolute
sem_t *s = sem_open("/path/to/foobar", O_CREAT | O_EXCL, S_IWUSR, 0);
with /path/to/foobar pointing to a user writable directory, segfaults
with "invalid system call".

As previously mentioned, 'kldload sem'.

To forestall any further gripes about the POSIX IPC system calls not
being compiled in by default: they are very rarely used, because the
SysV IPC API is almost universally available and is generally considered
superior to the POSIX API, which it predates by more than ten years.
The SysV IPC system calls are in GENERIC, and are used by e.g. Sendmail,
X.org and PostgreSQL.

DES