Thread View: comp.unix.programmer
5 messages
5 total messages
Started by Ryan Underwood
Mon, 13 Feb 2006 22:30
SysV shared memory coherency issues
Author: Ryan Underwood
Date: Mon, 13 Feb 2006 22:30
Date: Mon, 13 Feb 2006 22:30
102 lines
7115 bytes
7115 bytes
[This is crossposted to comp.lang.asm.x86 due to the instruction level issue
below]
Hi,
I'm getting really puzzled here. The basic story is that we have a 4 way Xeon
machine with hyperthreading enabled (8 logical processors). The application is
a fairly straightforward SysV (not POSIX) shared memory application. The
shared memory region is declared as a volatile double *. We are using GCC
4.0.2 with -O0 and a 2.6.6 Linux kernel with RTLinux extensions.
One of the processes sets up essentially a RPC call by placing several double
arguments in the shared memory region along with a function ID, and then
setting a flag to notify the other process that a call is waiting. The other
process is doing soft-realtime computational work as fast as it can with no
system calls, and occasionally checks the flag mailbox to see if a RPC is
waiting. If it is, then it reads the function ID and arguments and goes to
work on that task.
This seemed to work fine in my test cases, but eventually broke down for a
different user. The shared memory is written to and verified by the writer
process, but the computation process _never_ "sees" the flag even if allowed to
run forever. I can get the computation process to "see" the flag in several
ways:
1) Insert a sleep of between 1 and 10ms before the flag is polled (less than
1ms is not sufficient)
2) Print the contents of the mailbox on the computation process side before
polling the flag
Running under gdb or valgrind does not affect the bug.
Based on the above, my hypothesis is that in the cases where the condition
improves, it is because a system call has been taken, and thus the registers
clobbered. In the case where the sleep is insufficient, I think it is because
short sleep durations are executed as busy waits instead of forfeiting the
processor.
However, as a matter of experimentation, I changed the memory access in the
computation process from:
if (m_Mailbox[STALE_MBOX] != 1)
{ /* these actions are never taken */ }
which generated the following assembly:
movl 8(%ebp), %eax # this, this
movl 32(%eax), %eax # <variable>.m_Mail
addl $288, %eax #, D.40882
fldl (%eax) #* D.40882
fld1
to: [with knowledge that sizeof(double)==2*sizeof(uint32_t)]
if (((uint32_t*)m_Mailbox)[STALE_MBOX*2] != 1)
{ /* these actions are taken when mbox value is set to 0 */ }
which generated the following assembly:
movl 8(%ebp), %eax # this, this
movl 32(%eax), %eax # <variable>.m_Mail
addl $288, %eax #, D.40882
movl (%eax), %eax #* D.40882, D.40883
cmpl $1, %eax #, D.40883
As you can see, the load changes from a floating point to an integer load. To
my utter astonishment, this corrected the symptom that I was observing.
Apparently the memory access semantics of integer and FP loads differ in some
way on these processors. But is this actually the root of the problem, or is
there a more fundamental issue I am failing to understand? And is the issue
specific to the SysV shared page implementation (meaning I should look into
POSIX shared memory, pthreads, or mmap instead), or is it a processor-level
issue, perhaps with hyperthreading?
I was advised that I need memory barriers in this application for two reasons:
1) to keep the cache consistent across processors
2) because the mailbox writes may be reordered, meaning that even if the other
process saw the flag, there would be no guarantee that the function argument
writes had been performed yet.
If memory barrier functionality even exists on the IA-32 platform I don't seem
to be finding a way to access it from userspace. I am under the impression
that all operations on IA-32 are cache-coherent, and did not believe
hyperthreading changed that. Also, the wmb() macros in the Linux kernel seem
to evaluate to void for IA-32, supporting my position that memory barriers are
unnecessary in this case (that the writes, even if reordered when issued, are
committed in program order):
* For now, "wmb()" doesn't actually do anything, as all
* Intel CPU's follow what Intel calls a *Processor Order*,
* in which all writes are seen in the program order even
* outside the CPU.
#define wmb() __asm__ __volatile__ ("": : :"memory")
I'm having a difficult time seeing how this application differs from a typical
pthreads application, except in this case we only share specific preallocated
physical pages, instead of the entire page table of the parent process.
Looking forward to your thoughts,
Ryan Underwood
Re: SysV shared memory coherency issues
Author: "Nils O. Selåsdal"
Date: Tue, 14 Feb 2006 07:42
Date: Tue, 14 Feb 2006 07:42
64 lines
4868 bytes
4868 bytes
Ryan Underwood wrote:
> [This is crossposted to comp.lang.asm.x86 due to the instruction level issue
> below]
>
> Hi,
> I'm getting really puzzled here. The basic story is that we have a 4 way Xeon
> machine with hyperthreading enabled (8 logical processors). The application is
> a fairly straightforward SysV (not POSIX) shared memory application. The
> shared memory region is declared as a volatile double *. We are using GCC
> 4.0.2 with -O0 and a 2.6.6 Linux kernel with RTLinux extensions.
>
> One of the processes sets up essentially a RPC call by placing several double
> arguments in the shared memory region along with a function ID, and then
> setting a flag to notify the other process that a call is waiting. The other
> process is doing soft-realtime computational work as fast as it can with no
> system calls, and occasionally checks the flag mailbox to see if a RPC is
> waiting. If it is, then it reads the function ID and arguments and goes to
> work on that task.
>
> This seemed to work fine in my test cases, but eventually broke down for a
> different user. The shared memory is written to and verified by the writer
> process, but the computation process _never_ "sees" the flag even if allowed to
> run forever. I can get the computation process to "see" the flag in several
> ways:
>
> 1) Insert a sleep of between 1 and 10ms before the flag is polled (less than
> 1ms is not sufficient)
>
> 2) Print the contents of the mailbox on the computation process side before
> polling the flag
>
> Running under gdb or valgrind does not affect the bug.
>
> Based on the above, my hypothesis is that in the cases where the condition
> improves, it is because a system call has been taken, and thus the registers
> clobbered. In the case where the sleep is insufficient, I think it is because
> short sleep durations are executed as busy waits instead of forfeiting the
> processor.
>
> However, as a matter of experimentation, I changed the memory access in the
> computation process from:
>
> if (m_Mailbox[STALE_MBOX] != 1)
> { /* these actions are never taken */ }
>
> which generated the following assembly:
>
> movl 8(%ebp), %eax # this, this
> movl 32(%eax), %eax # <variable>.m_Mail
> addl $288, %eax #, D.40882
> fldl (%eax) #* D.40882
> fld1
>
> to: [with knowledge that sizeof(double)==2*sizeof(uint32_t)]
>
> if (((uint32_t*)m_Mailbox)[STALE_MBOX*2] != 1)
> { /* these actions are taken when mbox value is set to 0 */ }
Why all this casting ?
It'd be interresting to see what differences there were if you made
the flag a simple int, with the volatile keyword too.
(But really - you should be protecting this using semaphores)
Re: SysV shared memory coherency issues
Author: Robert Redelmeier
Date: Tue, 14 Feb 2006 14:30
Date: Tue, 14 Feb 2006 14:30
45 lines
4250 bytes
4250 bytes
In comp.lang.asm.x86 Ryan Underwood <spamtrap@crayne.org> wrote in part:
> if (m_Mailbox[STALE_MBOX] != 1)
> { /* these actions are never taken */ }
>
> which generated the following assembly:
>
> movl 8(%ebp), %eax # this, this
> movl 32(%eax), %eax # <variable>.m_Mail
> addl $288, %eax #, D.40882
> fldl (%eax) #* D.40882
> fld1
Egads! You're not testing FP numbers for _equality_, are you?
This is a cardinal sin! FP numbers are like sandpiles.
Whenever you move them, you lose some sand, and mix some dirt in.
When comparing floats/doubles, subtract&abs and test if the
difference is small enough.
In theory, if you start with clean FP registers (load an integer)
and never do anything nasty with them (like divide by non-2powers
or multiply by non-integers) then maybe they'll stay clean enough
for an equality compare. But I wouldn't count on it!
>
> to: [with knowledge that sizeof(double)==2*sizeof(uint32_t)]
>
> if (((uint32_t*)m_Mailbox)[STALE_MBOX*2] != 1)
> { /* these actions are taken when mbox value is set to 0 */ }
>
> which generated the following assembly:
>
> movl 8(%ebp), %eax # this, this
> movl 32(%eax), %eax # <variable>.m_Mail
> addl $288, %eax #, D.40882
> movl (%eax), %eax #* D.40882, D.40883
> cmpl $1, %eax #, D.40883
>
> As you can see, the load changes from a floating point to
> an integer load. To my utter astonishment, this corrected
> the symptom that I was observing.
Doh!
-- Robert
Re: SysV shared memory coherency issues
Author: Ryan Underwood
Date: Tue, 14 Feb 2006 20:35
Date: Tue, 14 Feb 2006 20:35
19 lines
3166 bytes
3166 bytes
=?ISO-8859-1?Q?=22Nils_O=2E_Sel=E5sdal=22?= <spamtrap@crayne.org> writes:
>> if (((uint32_t*)m_Mailbox)[STALE_MBOX*2] != 1)
>> { /* these actions are taken when mbox value is set to 0 */ }
>Why all this casting ?
The memory region is a double *. The cast is so that the compiler would
generate an integer load instead of a FP load.
>It'd be interresting to see what differences there were if you made
>the flag a simple int, with the volatile keyword too.
I imagine it would work since the above cast works. My question is why loading
it as a double does not work.
>(But really - you should be protecting this using semaphores)
Why? I don't see a mutual exclusion problem.
Re: SysV shared memory coherency issues
Author: Ryan Underwood
Date: Tue, 14 Feb 2006 20:37
Date: Tue, 14 Feb 2006 20:37
29 lines
3485 bytes
3485 bytes
Robert Redelmeier <redelm@ev1.net.invalid> writes:
>In comp.lang.asm.x86 Ryan Underwood <spamtrap@crayne.org> wrote in part:
>> if (m_Mailbox[STALE_MBOX] != 1)
>> { /* these actions are never taken */ }
>>
>> which generated the following assembly:
>>
>> movl 8(%ebp), %eax # this, this
>> movl 32(%eax), %eax # <variable>.m_Mail
>> addl $288, %eax #, D.40882
>> fldl (%eax) #* D.40882
>> fld1
>Egads! You're not testing FP numbers for _equality_, are you?
>This is a cardinal sin! FP numbers are like sandpiles.
>Whenever you move them, you lose some sand, and mix some dirt in.
No, no. This person is testing it for inequality. I understand what you're
saying, but in that case shouldn't that code block ALWAYS be executed rather
than NEVER? :) I agree that it is bad style, but check out the generated asm
in the correction post.
>In theory, if you start with clean FP registers (load an integer)
>and never do anything nasty with them (like divide by non-2powers
>or multiply by non-integers) then maybe they'll stay clean enough
>for an equality compare. But I wouldn't count on it!
Yes, excellent point.
Thread Navigation
This is a paginated view of messages in the thread with full content displayed inline.
Messages are displayed in chronological order, with the original post highlighted in green.
Use pagination controls to navigate through all messages in large threads.
Back to All Threads