🚀 go-pugleaf

In article <1990Sep5.194621.11656@athena.mit.edu>, pae@athena.mit.edu (Philip Earnhardt) writes:
>The alternative to this is to impose the semantic limitation that the user's
>appliction code must not run for "too long". This would qualify as a pretty
>arbitrary limitation to put on a distributed applications programmer.

I agree.  That's why I am (and OSF is) an advocate for a standard threading
interface with a portable implementation.

I might say:

    Of course, Sun RPC has gotten by (or maybe not gotten by :-) for
    years with limitations of this sort (and more).  E.g., if you use
    Sun RPC over UDP/IP, you're limited to 8K worth of data and the call
    semantics are unpredicatble.  "Use Sun RPC over TCP/IP", you might
    say.  Oh well, then the number of concurrent calls I'm allowed to
    have is limited by the number of TCP connection I can (effectively)
    have open at a time.  Seems like a pretty arbitrary limitation to
    put on a distributed applications programmer.  The load of having,
    oh, 100 clients doing lightweight things may be low.  It's just the
    poorly chosen requirement the RPC package made on the system that's
    preventing me from doing what I want.

But I wouldn't want to compare NCS to Sun RPC :-)

--
                    -- Nat Mishkin
                       Cooperative Object Computing Operation
                       Hewlett-Packard Company
                       mishkin@apollo.hp.com

In comp.protocols.misc mishkin@apollo.HP.COM writes:

>    Of course, Sun RPC has gotten by (or maybe not gotten by :-) for
>    years with limitations of this sort (and more).  E.g., if you use
>    Sun RPC over UDP/IP, you're limited to 8K worth of data and the call
>    semantics are unpredicatble.  "Use Sun RPC over TCP/IP", you might
>    say.  Oh well, then the number of concurrent calls I'm allowed to
>    have is limited by the number of TCP connection I can (effectively)
>    have open at a time.  Seems like a pretty arbitrary limitation to
>    put on a distributed applications programmer.  The load of having,
>    oh, 100 clients doing lightweight things may be low.  It's just the
>    poorly chosen requirement the RPC package made on the system that's
>    preventing me from doing what I want.


I have been reading with interest your discussions on RPCs.  I do not
have much documentation on NCS2.0 and hence have some difficulty in
understanding lots of NCS issues and policies.  For example,
 - how does the NCS library decide which transport to use.  Say, I have
   both TCP and UDP on both client and the server machine.  Are there any
   heuristics in place upon which you decide which transport has to be used?
   Do you prefer one over the other?
 - Using UDP for hundreds of connections is a good idea, but I wonder whether
   to support streaming on top of that is  a good idea because it can lead
   to all sorts of rate-flow problems.  TCP is great at backing off in
   stormy and overloaded networks, while UDP just exacerbates the problem.

Any comments?

Vipin Samar.

Disclaimer: I speak for none but myself.

vipin@Eng.sun.com || sun!Eng!vipin

In article <142133@sun.Eng.Sun.COM>, vipin@samsun.Sun.COM (Vipin Samar) writes:
> - how does the NCS library decide which transport to use.  Say, I have
>   both TCP and UDP on both client and the server machine.  Are there any
>   heuristics in place upon which you decide which transport has to be used?
>   Do you prefer one over the other?

The model (which is more or less the same for all versions of NCS) is
that when servers start up, they create some set of communication endpoints
with certain characteristics (in BSD-speak:  sockets of some socket type
and protocol in some protocol family with either a well-known or
dynamically assigned port).  The choice of characteristics can be
controlled by the server writer, or (at NCS 2.0) the server can say (in
effect) "create sockets of all protocol families and socket types that
both the RPC library and the underlying system support.  (You could
actually do this yourself in NCS 1.5, but we've made a simple function
to do it for you in 2.0.)

Once the server has created the communication endpoints, it advertises
the locations (essentially sockaddrs) of these endpoints in some network
accessible database.  NCS 1.5 supports the Global Location Broker for
doing this.  NCS 2.0 (OSF DCE RPC) works in conjunction with a distributed,
hierarchical name service.  Both the GLB and the name service can store
locations.

Now for the client side of things.  Presumably, a client has some abstract
identification for the server it's trying to reach.  E.g., it may have
an "interface ID" (what Sun RPC calls a program number, I think), or
it may have a pathname in the distributed name service, or it might have
some sort of "object ID" for the object (e.g., database, printer, file)
that it wants to operate on.  Whatever this abstract ID is, the client
contacts some 3rd party service (like the GLB or the name service) (which
typically it uses broadcast to locate) to translate the abstract ID into
a set of locations of servers.

Once the client has the locations, it picks out the ones that it can
use.  (Like sockaddrs, the stored locations hold information about the
network protocols you need to use to talk to the server at that location.)
This "picking out" step happens transparently to the client.  (More
transparently in 2.0 than in 1.5.)  The client application (or client
stub, in case you're using one of the fancier automatic binding features)
then presents a location to the RPC runtime which returns a "binding
handle" to the client.  The client then makes remote calls using the
binding handle.

(BTW, NCS currently has no way to pick among an apparently equivalent
set of locations.  This is an area for future work.  E.g., you might
want to favor TCP over X.25, if both are available; you might favor servers
on the same network as you, if they exist.  Some of these schemes are
hard to formalize and/or implement given existing underlying technology.)

At no point does the client express anything like "use TCP", or "use
connection-based RPC", or the like.  All that information is encapsulated
in the location structure and used transparently by the RPC runtime to
dispatch to the appropriate RPC runtime code to run the appropriate network
protocols (from RPC down through transport).

> - Using UDP for hundreds of connections is a good idea, but I wonder whether
>   to support streaming on top of that is  a good idea because it can lead
>   to all sorts of rate-flow problems.  TCP is great at backing off in
>   stormy and overloaded networks, while UDP just exacerbates the problem.

I guess I'm not sure what your point is.  NCS uses a flow control protocol
on top of UDP.  I.e., NCS uses UDP in much the way that TCP uses IP.
Thus, virtually all the tricks that TCP uses are candidates for being
used (and many are used) by NCS.

--
                    -- Nat Mishkin
                       Cooperative Object Computing Operation
                       Hewlett-Packard Company
                       mishkin@apollo.hp.com

mishkin@apollo.HP.COM (Nathaniel Mishkin) writes:


>Once the client has the locations, it picks out the ones that it can
>use.  (Like sockaddrs, the stored locations hold information about the
>network protocols you need to use to talk to the server at that location.)
>This "picking out" step happens transparently to the client.  (More
			         ^^^^^^^^^^^^^
>transparently in 2.0 than in 1.5.)

Yes, this is precisely what I am interested in knowing.  Does transparently
mean "random" here because any of the available protocols
give the same functionality or is there some implicit ordering? 

>(BTW, NCS currently has no way to pick among an apparently equivalent
>set of locations.  This is an area for future work.  E.g., you might
>want to favor TCP over X.25, if both are available; you might favor servers
>on the same network as you, if they exist.  Some of these schemes are
>hard to formalize and/or implement given existing underlying technology.)

Yes, I agree this is a tough problem on formalizing these
heuristics - perhaps after all it is not such a bad idea to give control to
users in such cases.  One can never really have all the bases covered.

>> - Using UDP for hundreds of connections is a good idea, but I wonder whether
>>   to support streaming on top of that is  a good idea because it can lead
>>   to all sorts of rate-flow problems.  TCP is great at backing off in
>>   stormy and overloaded networks, while UDP just exacerbates the problem.

>I guess I'm not sure what your point is.  NCS uses a flow control protocol
>on top of UDP.  I.e., NCS uses UDP in much the way that TCP uses IP.
>Thus, virtually all the tricks that TCP uses are candidates for being
>used (and many are used) by NCS.

O.K., so you did build the entire TCP layer on top of UDP and
you added all the code for that in the NCS library.  Has that step really
solved all the problems or perhaps has just moved the problems from one
place to another.

If one cannot support huge (say 100) TCP connections because of resource
problem, then you have said that NCS/UDP will be able to
overcome the resource problem.  I wonder if that is true, because
building up of entire TCP functionality on top of UDP is going to require
system resources as well.  Multiplexing the file descriptors may
solve this problem to some extent, but not completely, because one would
still need to keep all those buffers hanging around for any decent
congestion control and rate-flow problems.

Also, retransmissions would be of the entire UDP packet size and not just
of the size of the MTU of the network.  Or have you worked around this
problem by reducing the UDP packet size to the MTU size?  I hope not.
This retransmission problem will happen only during stormy weather, but
whenever that happens, it sure will let hell break loose.
TCP backs off very graciously in such circumstances. How does your protocol
behave in these cases?

Are all the TCP tricks on top of UDP in the kernel or in the user land?
If it is in user land, I am not sure how gratifying performance you are going
to get out of it.  Atleast for +8K data, the performance will be abysmal.  Most
of the TCP magic is normally in the kernel (which provides for higher
thru'put).  Would you like to share some performance numbers for NCS/UDP
and NCS/TCP? or perhaps just their ratio [if perf numbers are confidential].

Another problem with NCS approach is that the user cannot take advantages
out of any of the new performance improvements of any of the transports.
They will have to wait for the NCS team to hack those same fixes into their
software.  In ONC, they need not do anything.

Any comments ?
vipin
Disclaimer: I speak for no one but myself.
vipin@Eng.sun.com || sun!Eng!vipin

In article <142319@sun.Eng.Sun.COM> vipin@samsun.Sun.COM (Vipin Samar) writes:
>
>O.K., so you did build the entire TCP layer on top of UDP and
>you added all the code for that in the NCS library.  Has that step really
>solved all the problems or perhaps has just moved the problems from one
>place to another.
>
>Another problem with NCS approach is that the user cannot take advantages
>out of any of the new performance improvements of any of the transports.
>They will have to wait for the NCS team to hack those same fixes into their
>software.  In ONC, they need not do anything.
>

I am just joining this discussion so please excuse my innocence on some issues
that may have be resolved before.  It seems to me that implementing TCP 
functions in a layer 7 application or ROSE or RPC function is a duplication
of effort.  TCP (and TP4) has been in developement for years and should be
fairly stable from one implementation to another.  With this stability
people should be able to replace existing software based TCP (or TP4)
implementations with either better software solutions or hopefully silicon
assisted solutions.

In other words why recreate the wheel?  Reuse what already is in place and 
works.
--------------------------------------------------------------------------------
-- 
Ed Korsberg             E-mail: korsberg@aa.ab.com
Allen Bradley Inc.      phone:  313-998-2470
555 Briarwood Circle
Ann Arbor, Mich 48104

In article <142319@sun.Eng.Sun.COM>, vipin@samsun.Sun.COM (Vipin Samar) writes:
>Yes, this is precisely what I am interested in knowing.  Does transparently
>mean "random" here because any of the available protocols
>give the same functionality or is there some implicit ordering? 

It means "random because any of the available protocols give the same
functionality".  Like I said, define an scheme for making a more
intelligent choice among equivalent servers is an area for future work.

>Yes, I agree this is a tough problem on formalizing these
>heuristics - perhaps after all it is not such a bad idea to give control to
>users in such cases.  One can never really have all the bases covered.

I definitely agree that there should be at least one mode that lets
user policy choices come into play.

>O.K., so you did build the entire TCP layer on top of UDP and
>you added all the code for that in the NCS library.  Has that step really
>solved all the problems or perhaps has just moved the problems from one
>place to another.

Clearly there are SOME problems that just don't go away.  We believe that
many do.

>If one cannot support huge (say 100) TCP connections because of resource
>problem, then you have said that NCS/UDP will be able to
>overcome the resource problem.  I wonder if that is true, because
>building up of entire TCP functionality on top of UDP is going to require
>system resources as well.  Multiplexing the file descriptors may
>solve this problem to some extent, but not completely, because one would
>still need to keep all those buffers hanging around for any decent
>congestion control and rate-flow problems.

Of course.  However it is also of course true that NCS RPC's buffers
(unlike those of the kernel implementation of TCP) aren't consuming wired
physical memory, thus giving NCS some more room to play.  And then there's
the wired space in the kernel for TCP connection blocks and any other
miscellaneous TCP overhead.  Further, the nature of the NCS RPC protocol
lets the server simply blow off its pseudo-connections to clients at
any point that a call is not in progress; the pseudo-connection
reestablishment happens automatically and transparently to the application
(and the stubs).

It is also true that even an RPC running on top of a COTP has buffer
management problems and that they're arguably HARDER than RPC's over
CLTP's.  Note that when I say RPC/COTP I mean one that has somewhat more
complicated problems than say, SunRPC/TCP.  The problematic scenario
is the case where for some reason, the stub is not consuming its input
data stream for some period of time.  In our model, we have a thread
(or set of threads), distinct from the thread running the stub, that's
responsible for actually reading from the connection.  One reason for
this is so that client-generated "cancels" (i.e., requests to abort calls)
can be processed even if the stub/server thread isn't currently in the
RPC runtime.  Thus, there is a layer of buffering between the connection
and the stub.  There is finite space for this buffering.  Since once
you've read data from a connection, you can't discard the data (unless
you want to oblige the client stubs to be smart enough to restart the
call), you have the problem that you might have to be smart enough to
figure that you have to stop reading from some of your open connections.
But then you don't see the "cancel" message.  Without boring people any
more, you can more or less work around these problems, but it's by no
means simple.

>Also, retransmissions would be of the entire UDP packet size and not just
>of the size of the MTU of the network.  Or have you worked around this
>problem by reducing the UDP packet size to the MTU size?  I hope not.
>This retransmission problem will happen only during stormy weather, but
>whenever that happens, it sure will let hell break loose.
>TCP backs off very graciously in such circumstances. How does your protocol
>behave in these cases?

We endeavor to send MTU-sized UDP packets and I don't know why you "hope
not".  The general wisdom as I see it seems to be that depending on
IP-level fragmentation is a bad idea.  Of course, it's currently hard
to know the (what's called) "path MTU".  (I refer people to the IETF
"Path MTU Discovery" document for more details.)  So we guess and try
to be conservative and we eagerly await the widespread implementation
of the protocol modifications required to do path MTU discovery.  I think
the IETF has now endorsed them.

>Are all the TCP tricks on top of UDP in the kernel or in the user land?
>If it is in user land, I am not sure how gratifying performance you are going
>to get out of it.

We have no special kernel support.  (Note that there's a version of NCS
that runs in the kernel, but it's only for the benefit of NCS applications
that live in the kernel.)  I can't give you numbers right now.  (I don't
have them and I don't know that I could make them public right now anyway.)
(BTW, I would not be surprised to hear that NCS 1.5's bulk data throughput
is lower than TCP's on some systems.)

However, we believe that doing (what amounts to) a COTP in user space
on top of UDP will be sufficiently rewarding.  Even if it is only as
fast or slight slower than TCP it is rewarding in that it has better
scaling properties, which we care a lot about.  Further, we're basing
much our opinions on experiments that we've done and on the real-life
experience the CMU ITC people with their Rx RPC system, which essentially
does the same thing that NCS RPC does.  In any case I take seriously
my responsibility to produce a real evaluation of the wisdom of this
approach once I'm in a position to do the evaluation.

>Another problem with NCS approach is that the user cannot take advantages
>out of any of the new performance improvements of any of the transports.
>They will have to wait for the NCS team to hack those same fixes into their
>software.  In ONC, they need not do anything.

Fair enough.  On the other hand, to get the benefits of performance
improvements in NCS, all you need to do is get a new NCS library and
not get a whole new kernel from your vendor (as would likely be required
to get any improved TCP).  My experience (as both vendor and consumer)
has been that getting updated user-mode software to customers is typically
much easier than getting new kernels to them.

--
                    -- Nat Mishkin
                       Cooperative Object Computing Operation
                       Hewlett-Packard Company
                       mishkin@apollo.hp.com

> Vipin Samar writes:
>>Yes, I agree this is a tough problem on formalizing these
>>heuristics - perhaps after all it is not such a bad idea to give control to
>>users in such cases.  One can never really have all the bases covered.
>
>I definitely agree that there should be at least one mode that lets
>user policy choices come into play.

So, we have agreed on one thing that it is a good idea to have an
escape hatch for programmers/users to be able to choose a
particular transport.  Any ideas as to how will you extend NCS to do it?
ONC (tirpc) allows it today by using NETPATH environment variable.
One can either set it or use one of the default transports of the required
type available on the system.

>>If one cannot support huge (say 100) TCP connections because of resource
>>problem, then you have said that NCS/UDP will be able to
>>overcome the resource problem.  I wonder if that is true, because
>>building up of entire TCP functionality on top of UDP is going to require
>>system resources as well.

>Of course.  However it is also of course true that NCS RPC's buffers
>(unlike those of the kernel implementation of TCP) aren't consuming wired
>physical memory, thus giving NCS some more room to play.  And then there's
>the wired space in the kernel for TCP connection blocks and any other
>miscellaneous TCP overhead.  Further, the nature of the NCS RPC protocol
>lets the server simply blow off its pseudo-connections to clients at
>any point that a call is not in progress; the pseudo-connection
>reestablishment happens automatically and transparently to the application
>(and the stubs).

I am assuming that you blow away the connection state only when you have the
guarantee that the data has reached the other side.  TCP also frees
up all the associated buffer space in such cases.  The only difference
would be in the number of mbuf's (2?)reqd to keep the state/sequencing.
Streaming/UDP becomes a win, only if one knows beforehand that there will be
hundreds of connections for that particular RPC service.
	Even when one is using TCP, the server can blow away the
connection (if the call is not in progress) and the client will simply
have to rebind (at some extra cost).  The client RPC library should
provide the required transparency.  The question is which is better -
pay huge cost at each call (discussed in earlier messages) OR pay
rebinding cost only when it is required?  In my opinion, second approach
is better.

I am sure that there are many TCP/IP experts on this newsgroup.  Does any
one want to share his/her opinion on building the TCP functionality
on top of UDP?

>It is also true that even an RPC running on top of a COTP has buffer
>management problems and that they're arguably HARDER than RPC's over
>CLTP's.  Note that when I say RPC/COTP I mean one that has somewhat more
>complicated problems than say, SunRPC/TCP.  The problematic scenario
>is the case where for some reason, the stub is not consuming its input
>data stream for some period of time.  In our model, we have a thread
>(or set of threads), distinct from the thread running the stub, that's
>responsible for actually reading from the connection.  One reason for
>this is so that client-generated "cancels" (i.e., requests to abort calls)
>can be processed even if the stub/server thread isn't currently in the
>RPC runtime.  Thus, there is a layer of buffering between the connection
>and the stub.  There is finite space for this buffering. 

If I assume that both the threads are share the buffer space, then what do
you mean by "finite space for this buffering"?  Also, the problem
described above seems more like a problem created by your implementation.
Also, this problem should exist for CLTP too.  How did you handle that case?

>We endeavor to send MTU-sized UDP packets and I don't know why you "hope
>not".

Because I want good performance and that can happen only by minimizing
transitions between kernel and user land.  Choosing a large number is
bad for stormy cases & retransmissions and leads to overrun for machines
with fewer mbuf's or less memory.  Choosing a small number for UDP packet
size is bad for performance.  Sun decided to go with a compromise - 8K
UDP packet size.  In NCS, is there a mechanism for size negotiation
for faster/bigger machines (lots of memory) which can easily handle
bigger size UDP packets?  Just curious - what do you guys do for NFS?
Do you use 1K packets only?

Actually, you can easily rest all of my fears by posting some performance
ratios for UDP/TCP (not numbers - I know they are confidential)


> In any case I take seriously
>my responsibility to produce a real evaluation of the wisdom of this
>approach once I'm in a position to do the evaluation.

I hope that performance will be one of the criterian.

>--
>                    -- Nat Mishkin

vipin
Disclaimer: I speak for no one but myself.

vipin@Eng.sun.com || sun!Eng!vipin

In article <1990Sep14.093420@apollo.HP.COM>, mishkin@apollo.HP.COM (Nathaniel Mishkin) writes:
> 
> >Another problem with NCS approach is that the user cannot take advantages
> >out of any of the new performance improvements of any of the transports.
> >They will have to wait for the NCS team to hack those same fixes into their
> >software.  In ONC, they need not do anything.
> 
> Fair enough.  On the other hand, to get the benefits of performance
> improvements in NCS, all you need to do is get a new NCS library and
> not get a whole new kernel from your vendor (as would likely be required
> to get any improved TCP).  My experience (as both vendor and consumer)
> has been that getting updated user-mode software to customers is typically
> much easier than getting new kernels to them.
> 
> --
>                     -- Nat Mishkin


Kernel would normally come with the machine.  However, the RPC library may
or may not come with the machine.  So, to order NCS library, I will
need to know which version/release of transports I am running.
And the vendors (you as the technology supplier, others as distributors
for different platforms) will have to keep different versions of the
library depending upon the transport available on my machine.  Unless
this chain is short and things happen smoothly, the time lag between
the day I get my machine and the day I have compatible/tuned NCS software could
be substantial - perhaps time to buy a new machine or install the new
OS and then the cycle would start all over again.
	And you as a vendor would certainly have a fun time
maintaining all different versions of software for various transports!

vipin
Disclaimer: I speak for no one but myself.

vipin@Eng.sun.com || sun!Eng!vipin

In article <142630@sun.Eng.Sun.COM>, vipin@samsun.Sun.COM (Vipin Samar) writes:
>So, we have agreed on one thing that it is a good idea to have an
>escape hatch for programmers/users to be able to choose a
>particular transport.  Any ideas as to how will you extend NCS to do it?

I don't have any concrete proposal.  However, let me point out some things.

There are two kinds of choices one might want to make:  (1) choose which
protocol stack (by which I mean a generalization of your scheme that
lets you choose which transport you want) you want to use, and (2) choose
which SERVER you want, among a set of equivalent servers.  The first
kind of choice is addressed in part by your environment variable.  (I
think it's more likely what you'd want to specify is a ranking of stacks;
i.e., "I want to use RPC/UDP/IP if the server has it, otherwise I'll
accept RPC/ROSE/.../TP4, otherwise ...".)  The second choice is a little
trickier.  You might want to choose based on the number of hops to the
server or the class of machine it is or how busy it is.

Basically, NCS 2.0 has three functions:  "import", "lookup", and "select".
(These aren't the exact names of course.) "import" basically just calls
"lookup" followed by "select".  "lookup" returns a set of "compatible
servers" -- i.e., ones that you have a prayer of being able to talk with
(i.e., they've registered support for the interface you're interested
in and they share all the necessary communications protocols with you).
"select" picks one server out of a set returned by "lookup", currently
at random.  The idea is that most clients will just use "import" and
trust that "select" does an OK thing and that in the future maybe it
will do even more OK things.  However, sophisticated clients are free
to call "lookup" and select a server based on whatever criteria the
application writer feels like applying, including looking at environment
variables and reading configuration files.

>I am assuming that you blow away the connection state only when you have the
>guarantee that the data has reached the other side.  

Yes.

>TCP also frees
>up all the associated buffer space in such cases.  The only difference
>would be in the number of mbuf's (2?)reqd to keep the state/sequencing.
>Streaming/UDP becomes a win, only if one knows beforehand that there will be
>hundreds of connections for that particular RPC service.
>	Even when one is using TCP, the server can blow away the
>connection (if the call is not in progress) and the client will simply
>have to rebind (at some extra cost).  The client RPC library should
>provide the required transparency.  

This is in essence what NCS 2.0 does when running over a COTP.  Note
that there are some slightly tricky details having to do with what
it looks like to the client in the case where it's launching a "request"
over the connection at exactly the same time the server is closing
the connection.  The client might have a bit of a hard time distinguishing
the network close from a server crash and can't know whether the call
might have been executed and as a result not know whether it's safe
-- i.e., meets the "at most once" rule -- to try to execute the call
again at the same or another server.

>The question is which is better -
>pay huge cost at each call (discussed in earlier messages) OR pay
>rebinding cost only when it is required?  In my opinion, second approach
>is better.

What's the huge cost?  Anyway, I wouldn't write off the rebinding cost.
The TCP disconnect and connect are going to cost around 6 network messages.
A busy server is going to have to close connections fairly regularly.
(Of course NCS has some analogous overhead, but (a) it's lower because
there are no close messages, and (b) the authentication setup is
piggy-backed on the connection setup messages.  Any authentication
messages [challenge/response] would be yet more overhead messages in a
TCP-based scheme.)

Note also that I don't know how small you can reduce the "state/sequencing"
info in TCP and I don't know how big your mbufs are, but in NCS/RPC we
can keep literally just (the equivalent of) the state value, connection
ID, and sequence number and we're using the generalized user space heap
storage package that has few restrictions (e.g., in the granularity of
the allocation size).  We can keep a LOT of connections' state.

>If I assume that both the threads are share the buffer space, then what do
>you mean by "finite space for this buffering"?  Also, the problem
>described above seems more like a problem created by your implementation.

It may seem that way, but I think it's not.  It all falls out from the
requirement of being able to cancel an in-progress call.  I believe this
makes it problematic to let the thread that's doing the unmarshalling
actually be the thread that's reading from the connection.  Thus, a layer
of buffering is required.  There is some fixed number of buffers you
will be willing to assign to this task.  Once this space is exhausted
you have a problem.

>Also, this problem should exist for CLTP too.  How did you handle that case?

RPC/CLTP can always discard data that it's read from the network because
the sender never discards the data until the data has been ack'd by the
receiver.  When the receiver discards data, it doesn't ack it, inducing
the sender to retransmit the data.  (The details include the fact that
the receiver will set his "offered window" value in the ack's it sends
to zero.)

>>We endeavor to send MTU-sized UDP packets and I don't know why you "hope
>>not".
>
>Because I want good performance and that can happen only by minimizing
>transitions between kernel and user land.  Choosing a large number is
>bad for stormy cases & retransmissions and leads to overrun for machines
>with fewer mbuf's or less memory.  Choosing a small number for UDP packet
>size is bad for performance.  Sun decided to go with a compromise - 8K
>UDP packet size.  

I understand the penalties of kernel/user transitions, but I also
understand the sending of 8K UDP datagrams to be a dangerous proposition
since if any one IP fragment is lost, you are obliged to retransmit
ALL the fragments since there's no mechanism for indicating which
fragment was lost.

>In NCS, is there a mechanism for size negotiation
>for faster/bigger machines (lots of memory) which can easily handle
>bigger size UDP packets?  

This is not a machine size/speed issue.  It has to do with the network.
If there are 5 gateways between the sender and the receiver and they're
busy, the odds that one of the fragments will be lost could well be
unacceptably high (esp. considering the cost of a loss will be the load
of sending ALL the fragments again).

I'll leave it for the Internet experts to make some "official" statement
(if such a thing is possible), but I've gotten the strong feeling that
depending on IP fragmentation is a bad idea.

>Just curious - what do you guys do for NFS?  Do you use 1K packets only?

I'm the wrong person to ask about NFS.  I don't have anything to do with
it.

>Actually, you can easily rest all of my fears by posting some performance
>ratios for UDP/TCP (not numbers - I know they are confidential)

I don't have numbers right now.

>> In any case I take seriously
>>my responsibility to produce a real evaluation of the wisdom of this
>>approach once I'm in a position to do the evaluation.
>
>I hope that performance will be one of the criterian.

Yes, but it's important to remember that it's only one of the criteria

--
                    -- Nat Mishkin
                       Cooperative Object Computing Operation
                       Hewlett-Packard Company
                       mishkin@apollo.hp.com