🚀 go-pugleaf

In article <1992Aug24.190727.19006@shograf.com> jimb@shograf.UUCP (Jim battle) writes:
| Allen Akin has seemed to make a big deal out of the data reformatting
| issue.  He seems to think it is a decisive factor in the debate of whether
| PEX or OpenGL does/will provide a superior interface for 3D graphics.
| ...

Overall bus traffic for graphics (of which data reformatting is one
component) is a major concern for systems designers.  It proved to be
the bottleneck for a couple of important applications on the last
PEX-based graphics workstation with which I was involved, and therefore
became a competitive issue.  It's an important factor for designs at
SGI.  Published descriptions of machines from Hewlett-Packard have
discussed it.  During the PEX/OpenGL Debate at SIGGRAPH, Marty Hess of
Sun mentioned it twice as a more likely bottleneck than network
throughput or graphics engine performance.

I believe there's a significant architectural difference between PEX
and OpenGL in this regard, and therefore I've argued that OpenGL is
preferable to PEX with respect to data reformatting costs.

Certainly there are other factors.  In postings here I've concentrated
on data reformatting just because people were interested in discussing
it.


| As I understand it from the descriptions of the SGI hardware given
| in this thread, combined with technical briefs out of SGI, the
| actual mechanism to perform 'zero' overhead reformatting of data
| is twofold:
| 
|     1) vertex data type is passed along with the data and converted
|        to the appropriate floating-point format on the way to the
|        geometry engine.
| 
|     2) rather than having the CPU send each word of the vertex data
|        to the geometry engine, instead a pointer to the vertex data
|        is sent to a DMA engine, which takes care of sending all the
|        data items for the vertex to the geometry pipe.
| 
| If this is the case (please correct me if I'm wrong), ...

I think this description is accurate for many cases.  The exact
mechanism varies somewhat from machine to machine, even in SGI's
product line.  Of course, other implementations are possible.

Also, the data-type conversion mentioned in point (1) isn't what
I mean by the term "data reformatting."  From Jim's questions below
I believe he understand this; I just want to clarify the issue for
other readers.


|                                                   ... the memory must
| still be read by something (DMA engine, in this case), which will block
| CPU accesses to memory.  I don't know exactly what you mean by
| 'system bus', but I'd suppose that this DMA operation does interfere
| with the CPU's gross memory bandwidth.  ...

Sure.  I hope I didn't imply that this mechanism was completely free;
it's just cheaper (requires less bus bandwidth) than the alternatives.
For the simplified example I gave in an earlier posting, the CPU was
still busy controlling data transfers, but had only half as many to
control.


|                                      ...And, of course, there is
| the problem in CPU's that have write-back cache; how does the DMA
| engine fetch data from main memory with confidence that it has the
| most recent copy?

Well, if you want to use a DMA mechanism for data transfer then you
have to solve this problem somehow.  The solution you choose is very
dependent on the price and performance goals for your system.  Maybe
you would choose to have the cache snoop DMA transactions, and override
the data provided by main memory whenever there's a hit.  Maybe you
would require the CPU to read the data into the cache with ordinary
memory transactions, then build your graphics DMA engine so that it
only reads data from the cache.  Maybe your system already guarantees
cache coherency because it supports multiple processors with shared
memory, and you can use that feature in some way.  Maybe you'd take
advantage of user-mode cache flushing instructions if your CPU supports
them.  Or maybe you'd just punt the concept of DMA altogether, and rely
on some other form of three-way transfer.  There are lots of options.


| The way I see it, Allen Akin is correct is saying that the OpenGL
| approach to this particular issue offers more flexibility than
| the PEX way of doing things.  I only disagree that it is '*very*'
| important difference.
| 
| It is a meaningless comparison to look at only part of the problem
| and decide what the overall outcome is.  For the cited example of
| a tetrahedral mesh simulation, I would ask how many operations that
| program does compared to performing one extra write?  If you spend
| 2us per vertex (to make up a juicy number) calculating forces, spending
| another 25ns to perform an extra write to memory to build the data
| structure that PEX wants and another 25ns to read the data item,
| is insignificant (4%).

I think you're on the right track when you try to quantify these
issues.  However, it's very difficult to do so because there are so
many dependencies on application behavior and so many interactions with
other aspects of system design.  It's usually necessary to present
simplified cases for illustration, and the simplifications can always
be challenged.

Your question implies that you want to move to a more realistic level
of detail for the tetrahedral mesh example.  To do so, I'd want to know
a few more pieces of information.  25ns for a write all the way through
to main memory seems low to me; are you assuming RAMBUS or synchronous
DRAM technology?  What are their access cycles like for random and
sequential addresses?  How about overhead cycles to acquire and release
the main memory bus, and to fill the other words in a cache line?  What
computations will the application need to perform for each vertex?
What is the CPU cycle time?  Does the CPU instruction set have any
special features for overlapping computation with memory access?  How
much of the data structure would typically be in-cache?  On PEX-based
systems, will the application be making PEXlib or PHIGS calls?  If
PHIGS, is the PHIGS implementation layered on PEXlib?  How much input
buffering capacity does the graphics engine have (because that might
affect the buffering strategy for graphics requests)?  And so on.

Even after we settle on these details, it may still turn out that, for
the given example on a given machine, data reformatting is a small
percentage of total cycles.  That's true for some cases, and past
experience indicates that it has been false in others.  (See comments
above.)  Since this uncertainty exists, why not choose a 3D graphics
interface that allows you to reduce bus bandwidth requirements in the
cases where that results in performance or price improvement?


| Looking at the wide variety of prices, performances, and
| price/performances on the market, I don't think that this issue
| will affect things enough to make or break anyone.

I like to look at the problem from a slightly different point of view.
As a graphics hardware vendor, there's not a lot I can do to optimize
applications.  However, I control my system implementation almost
completely; I don't have much excuse for building a suboptimal
product.  What's the best performance I can achieve in a given price
band?  How do I go about reaching that performance level?

Obviously bus bandwidth is one of the factors I need to consider.
Since most databases are larger than the size of the system cache,
reading a database and writing out graphics commands can often involve
(or be approximated by) memory-to-memory data transfer.  Here's a
quick-and-dirty program to test memory-to-memory copy rates, assuming
your local implementation of memcpy() is fully optimized:

	#include <stdio.h>
	#include <stdlib.h>
	#include <sys/types.h>
	#include <sys/times.h>
	#include <sys/param.h>
	
	#define MEG		(1024 * 1024)
	
	/*
	 * NBYTES must be greater than the machine's cache size, and less than
	 * half the amount of physical memory available to the process.  This
	 * ensures we measure memory-to-memory copy speed rather than cache-
	 * to-cache or disk-to-memory speed.
	 */
	#define NBYTES		(2 * MEG)
	
	/*
	 * NITERATIONS must be large enough to amortize error due to timer
	 * granularity and overhead.
	 */
	#define NITERATIONS	100
	
	
	
	main()
		{
		char* source;
		char* destination;
		struct tms start;
		struct tms finish;
		register int i;
	
		source = malloc(NBYTES);
		destination = malloc(NBYTES);
	
		memcpy(destination, source, NBYTES);	/* touch all pages */
	
		times(&start);
		for (i = 0; i < NITERATIONS; ++i)
			memcpy(destination, source, NBYTES);
		times(&finish);
	
		printf("%.1fM B/s\n",
		    ((double) NBYTES * NITERATIONS / 1000000.0)
		    / ((double) (finish.tms_utime - start.tms_utime) / HZ));
	
		exit(0);
		}

When I run this on my R3000 Indigo, it reports "14.0M B/s".

Suppose I want to know the best performance I could possibly achieve
for drawing independent quadrilaterals.  Each quad might consist of
four vertices, and each vertex might have three coordinates, three
color values, and three surface-normal values.  If I use
single-precision floating-point throughout, then each quad is 4 * (3*4
+ 3*4 + 3*4), or 144, bytes long.

If my graphics implementation uses the CPU to copy the database just
once (e.g. from a display list to a memory-mapped graphics engine),
then the best performance I can manage is about (14.0M B/s) / (144
B/quad) = 97K quads/s.  If my graphics implementation copies the
database twice (e.g. from application database to PEX protocol packet
to graphics engine), then the best I can manage is 49K quads/s.  If my
graphics implementation copies the database three times (e.g. from
application database to PHIGS API format to PEX protocol packet to
graphics engine), then the best I can manage is 32K quads/s.  This
drastic decrease in performance is what encourages system designers to
use alternative implementation techniques, like compilable display
lists, DMA, and special programmed-I/O paths for graphics.

To put these numbers in perspective, the Elan graphics subsystem on my
Indigo is rated at 110K quads/s.  To a first (very, very rough)
approximation, you can see that on an R3000 Indigo I can't afford to
copy the data even once if I'm to keep up with the maximum performance
of this graphics subsystem.  It's in this context that I say data
reformatting is a "*very*" important issue.

Now I've played fast and loose with a large number of design
considerations here.  My purpose wasn't to propose and critique a
design in realistic detail, but to give everyone a feel for just one of
the issues involved.  If you're interested, you might want to contact
your salesperson for technical papers describing whatever graphics
system you use.  Those papers will probably give you some good clues
about the factors your system designers felt were most important, and
how they resolved conflicting requirements.


| Just my 0x00000002 cents worth.

Thanks!  Very much worthwhile.

Allen