🚀 go-pugleaf

[This is crossposted to comp.lang.asm.x86 due to the instruction level issue
below]

Hi,
I'm getting really puzzled here.  The basic story is that we have a 4 way Xeon
machine with hyperthreading enabled (8 logical processors).  The application is
a fairly straightforward SysV (not POSIX) shared memory application.  The
shared memory region is declared as a volatile double *.  We are using GCC
4.0.2 with -O0 and a 2.6.6 Linux kernel with RTLinux extensions.

One of the processes sets up essentially a RPC call by placing several double
arguments in the shared memory region along with a function ID, and then
setting a flag to notify the other process that a call is waiting.  The other
process is doing soft-realtime computational work as fast as it can with no
system calls, and occasionally checks the flag mailbox to see if a RPC is
waiting.  If it is, then it reads the function ID and arguments and goes to
work on that task.

This seemed to work fine in my test cases, but eventually broke down for a
different user.  The shared memory is written to and verified by the writer
process, but the computation process _never_ "sees" the flag even if allowed to
run forever.  I can get the computation process to "see" the flag in several
ways:

1) Insert a sleep of between 1 and 10ms before the flag is polled (less than
1ms is not sufficient)

2) Print the contents of the mailbox on the computation process side before
polling the flag

Running under gdb or valgrind does not affect the bug.

Based on the above, my hypothesis is that in the cases where the condition
improves, it is because a system call has been taken, and thus the registers
clobbered.  In the case where the sleep is insufficient, I think it is because
short sleep durations are executed as busy waits instead of forfeiting the
processor.

However, as a matter of experimentation, I changed the memory access in the
computation process from:

	if (m_Mailbox[STALE_MBOX] != 1)
		{ /* these actions are never taken */ }

which generated the following assembly:

	movl    8(%ebp), %eax   # this, this
	movl    32(%eax), %eax  # <variable>.m_Mail
	addl    $288, %eax      #, D.40882
	fldl    (%eax)  #* D.40882
	fld1

to: [with knowledge that sizeof(double)==2*sizeof(uint32_t)]

	if (((uint32_t*)m_Mailbox)[STALE_MBOX*2] != 1)
		{ /* these actions are taken when mbox value is set to 0 */ }

which generated the following assembly:

	movl    8(%ebp), %eax   # this, this
	movl    32(%eax), %eax  # <variable>.m_Mail
	addl    $288, %eax      #, D.40882
	movl    (%eax), %eax    #* D.40882, D.40883
	cmpl    $1, %eax        #, D.40883

As you can see, the load changes from a floating point to an integer load.  To
my utter astonishment, this corrected the symptom that I was observing.
Apparently the memory access semantics of integer and FP loads differ in some
way on these processors.  But is this actually the root of the problem, or is
there a more fundamental issue I am failing to understand?  And is the issue
specific to the SysV shared page implementation (meaning I should look into
POSIX shared memory, pthreads, or mmap instead), or is it a processor-level
issue, perhaps with hyperthreading?

I was advised that I need memory barriers in this application for two reasons:

1) to keep the cache consistent across processors
2) because the mailbox writes may be reordered, meaning that even if the other
process saw the flag, there would be no guarantee that the function argument
writes had been performed yet.

If memory barrier functionality even exists on the IA-32 platform I don't seem
to be finding a way to access it from userspace.  I am under the impression
that all operations on IA-32 are cache-coherent, and did not believe
hyperthreading changed that.  Also, the wmb() macros in the Linux kernel seem
to evaluate to void for IA-32, supporting my position that memory barriers are
unnecessary in this case (that the writes, even if reordered when issued, are
committed in program order):

 * For now, "wmb()" doesn't actually do anything, as all
 * Intel CPU's follow what Intel calls a *Processor Order*,
 * in which all writes are seen in the program order even
 * outside the CPU.
#define wmb()   __asm__ __volatile__ ("": : :"memory")

I'm having a difficult time seeing how this application differs from a typical
pthreads application, except in this case we only share specific preallocated
physical pages, instead of the entire page table of the parent process.

Looking forward to your thoughts,

Ryan Underwood