🚀 go-pugleaf

The only part I do not like is you base your PING tests, which are highly
useless from a single source.  Pings are NOT designed to test performance
in any way shape or form.  Also you tested the sites from 1 source,
therefore it is only usefull if you are providing this for someone who is
on the same source.  Ping testing is the most useless test for performance
that exsists.  Also it just adds to already congested backbone structures.

Frank

mreid@rogers.wave.ca wrote in article <871994997.26550@dejanews.com>...
> Hello,...
> 
> I've just finished phase 1 of my National Capital Region ISPs info
> webpages. I have data on about 50 ISPs in common usage in the Ottawa/Hull
> area.
> 
> Includes:
>  Live current & historical (last 6 days) ping performance graphs.
>  Past Ping and HTTP performance data.
>  Survey & Test results of pre-sales support.
>  Network maps to show who connects to whom.
> 
>  The pages are currently at:
> http://rpc-747.on.rogers.wave.ca/ottisp/index.html
>  also known as:
> http://24.112.92.228/ottisp/index.html
>  and occasionally at:
> http://home.on.rogers.wave.ca/mreid/ottisp/index.html
> 
> 
>  I also have some Rogers Wave Info pages at:
> http://home.on.rogers.wave.ca/mreid/rogwave/rogwave.html
> 
>  and a less than 100% reliable webcam pointed at the Ex today at:
> http://rpc-747.on.rogers.wave.ca/webcam.html
> 
> 
> Hopefully this will be a useful resource. Please feel free to email or
> post comments or whatever. Especially looking for comments about which
> local ISPs have the best, most reliable and fast news servers...
> 
> Mike Reid.
> 
> -------------------==== Posted via Deja News ====-----------------------
>       http://www.dejanews.com/     Search, Read, Post to Usenet
> 

Hi Frank,

it's nice to see ISP employees or prinicipals such as yourself
participating in online discussion...

In article <01bcad60$702e9d20$921bf8ce@frankisdn.ott.igs.net>,
  "Frank Kahle" <frank@igs.net> wrote:
>
> The only part I do not like is you base your PING tests, which are highly
> useless from a single source.

I agree that the single source for my current tests is a major
limitation. I hope I've made that clear on my pages. It would be more
useful to perform testing from many locations and networks and I hope to
get involved in such distributed testing. A cross-network system of
communicating monitor programs might help achieve this goal, albeit at
the price of increasing Internet traffic.

> Pings are NOT designed to test performance in any way shape or form.

I contend that they help to determine the latencies experienced by
packets travelling from one site to another. They also help to determine
the internet availability of hosts. Latency and availability to various
sites are both important (but not the only) performance factors to
consider when choosing an ISP or backcone network provider. If ping
performance numbers meant nothing, I would think that more ping utilities
would report success or failure only, rather than ping times in
milliseconds.


>Also it just adds to already congested backbone structures.

I was concerned about this issue in my testing and made some observations
and calculations. I had Pinger pinging about 70 systems, with two pings
of about 64 bytes every 10 minutes at one point.

Including DNS lookups this was sending about 29 KBytes in 282 packets
every 10 minutes, including traffic to my DNS server on Rogers Wave. (I'm
currently running my own cacheing DNS server which reduces my own load on
Rogers network, at least.)

Data received was about 91 KBytes and 300-340 packets every 10 minutes.
The total both ways was about 1 Million bits (120 KBytes) and 600 packets
every 600 seconds. On average this is equivalent to a constant flow of
about 1.8 Kbits/second and 1 packet/second for all of the sites I've been
pinging.


If the Internet can't handle this flow in 1997 from my Rogers Wave
connection then it is in a sorry state and unready for the future. Each
ISP was taking only about 1.5 % of this minimal load. Furthermore, I
beleive that by making the information I collect publicly available and
in a "live" fashion, that I am offering a valuable public service easily
justifying the minimal extra load. We're not talking here about the
Gigabytes of copyright violations, pornography and spam currently
clogging the Internet.


I AM more concerned about the HTTP tests I was running and which I intend
to run again at some point, with my own code. These were causing 5-10
times as much traffic in byte terms. I'm quite concerned about causing
any undue load. I'm open to further comments or suggestions about how to
minimize this load and provide the most useful performance data...


Regards,
Mike Reid.

-------------------==== Posted via Deja News ====-----------------------
      http://www.dejanews.com/     Search, Read, Post to Usenet

In article <872108165.15684@dejanews.com>,  <mreid@rogers.wave.ca> wrote:
>and calculations. I had Pinger pinging about 70 systems, with two pings
>of about 64 bytes every 10 minutes at one point.

While ping is not the greatest of methods for determining a meaningful
metric, it does give a pretty good idea on a comparative basis when
viewed with a large body of data.

Two pings is not, however, very statictically meanginfull and
it more "rough" than "metric"  as a "rough metric". 10 or 20
would be better; better yet would be to sample once an hour.

-- 
 
          ... so I lost them in Mexico City.

Neither Dejanews nor Rogers news server has the post Vic sent (to me
perhaps?)
and that Michael is responding to here. If anyone could email that
post to
mreid@rogers.wave.ca it would be appreciated...

Thanks for the URLs, I had just asked for such references in another
post.

In article <slrn5vpu82.5n.mcr@amaterasu.sandelman.ottawa.on.ca>,
  mcr@sandelman.ottawa.on.ca wrote:

>Vik Wrote:
> >You cannot expect to provide real-world performance figures from
> >one host.  Accordingly, the best you can do is give a time-averaged
> >estimate by RTT method of the effective throughput for
unidirectional
> >communcation to an ISP along the link from ISP to network provider.
> >Of course, this is of only limited value for obvious reasons.
>
>   Mike, please listen to Vik, he knows what he is talking about.

I'm happy to hear anyone's views on the subject. I hope I made it
agonizingly
clear that I have posted multiple disclaimers to that effect and that
I have
intentions of implementing distributed performance measurement. I
don't
relish the alternative of removing my web pages just because some
people may
not read the disclaimers and misinterpret the very limited data I have
at
this time.

> >And I don't mean to seem antagonistic to the idea, because
> >I am not.  I just maintain that the problem is far more complicated
> >than has been suggested, and there is very little information
> >that can be gained from a single host, regardless of method.

Here's a quote from my performance testing page at
http://home.on.rogers.wave.ca/mreid/ottisp/perftest.html :

 .Comprehensive testing of ISP performance levels is a daunting task.
There are
 .many aspects to performance and no single test tells a complete
story.
 .Furthermore,
 .there are many uncontrollable aspects of the test environment. Test
results
 .will often be different depending on when and from where they are
done. More
 .complete
 .and comprehensive testing would require the use of multiple test
locations and
 .continuous testing over a period of time to ensure the statistical
validity of
 .the tests.


If I've suggested anywhere on my pages, the newsgroups or elsewhere
that
performance testing is not fraught with difficulty, please let me know
and
I'll be more than happy to retract the statement. If my original
posting
suggested this to you or anybody because of it's brevity or lack of
disclaimers
in the original posting itself, then I humbly apologize.

Regards,
Mike Reid.

-------------------==== Posted via Deja News ====-----------------------
      http://www.dejanews.com/     Search, Read, Post to Usenet

Vik,
neither of your posts to me made it to Dejanews. Someone was gracious
enough to send me a copy of your post and I've reponded here.

Vik wrote:
>mreid@rogers.wave.ca wrote:
>
>: I find that ICMP echoes CAN be used to show various trends. These
>: graphs show
>: what I would expect: Latency increases in such busy periods such as
>: from 10:00 -
>: 12:00, from 14:00 to 16:00 and from 18:00 to 01:00 .
>
>Yes, but these trends are valid only in one direction and for
>you.  Real throughput is dependent on router conditions, route loss
>patterns (which are asymmetric), and various other things
>which cannot effectively be measured from your site, including
>end-user stack configuration.  This is a very complicated issue
>and I do not feel that you should extrapolate anything
>from ping reports.  In essence, ping reports from one
>single site to another are of absolutely no value to anyone
>else.


In an academic and theoretical sense this may be true. Nothing can
be extrapolated from anything with any certainty. We can't extrapolate
tomorrows network performance from todays with any certainty for example.

In practice, when I see the latency time graphs for two ISPs
connected to the same network backbone by the same routers, and one
is very flat and low while the other is much higher and spiked,
I conclude  that the ISP with huge latency spikes PROBABLY has
a much lower general performance. I'm well aware that technically
this might be untrue, but I find it telling that the now deceased
Ottawa Public Net fared the worst in my tests, when they should
have had an advantage by being on iStar.

Infinite ping times for an extended period are also quite interesting.
When I see this happening, I make efforts to verify from various remote
systems that the ISP or site in  question is indeed down. Note that
this is not a single site observation.

>What statistical models and time-resolved samples of intermediate
>networks have you used to make adjustments to ping-times
>such that they reflect the reality of anyone's situation?

None.

>If you mean that you've just smoothed the figures by
>subtratcting intermediate delays and taking the time-averaged
>value, this hasn't really done anything.

I haven't done any such thing. These are just some initial and
apparently incorrect thoughts I've had. These points are moot
in any case as I acknowledge that distributed testing is much
more meaningful. Fine. I have no problem admitting when I am wrong.


>: intend to add such logic to my software although I acknowledge it's
>: not a simple
>: task and could require the dynamic traceroutes I've mentioned.
>
>It's not a simple task;  people at the DSL at my University
>routinely do theses on related topics.

Granted, nothing is simple. Have they written any code that is
publicly available ?


>: There are many different tools for many different purposes. I'm
>: familiar with
>: Bing and aware it does little more than ping two routers with
>: different size
>: ICMP packets. This provides a sometimes accurate bandwidth estimate,
>: but of
>: course there are many ifs, ands and buts... I've done the same thing
>: manually
>: with ping and some calculations. The T1 line to Rogers I use appeared
>: to vary
>: from 1.6 - 1.8 Mbits/s according to these calcs, when it's actually
>: 1.544
>: Mbits/s minus overhead.
>
>You are confusing bandwidth and single-user throughput.

I don't believe I'm the one who is confused about the differences
between bandwidth and throughput. I don't want to get picky about
terminology as we all use technically incorrect terminology sometimes.
As an example, it's been argued that the term multimedia is incorrect
and that multimedium would have been more proper. When confusion may or
does arise, I define my terms.

I define bandwidth as the total amount of data (bits, bytes, cycles
or whatever) which  can be sent and/or received over a given link in
a given period of time (by all users and processes). For example,
the Rogers Wave T1 line from Ottawa to Toronto  provides a
bi-directional bandwidth of 1.544 Mbps .

Throughput I define as the actual amount of data any given user,
process, session or whatever can send or receive in a given period
of time. For example, my cable modem is limited to 512 Kbps or so
and the maximum throughput I can achieve through this modem is in
theory 512 Kbps. In practice the maximum I see is about 480 Kbps.


>You cannot expect to provide real-world performance figures from
>one host.  Accordingly, the best you can do is give a time-averaged
>estimate by RTT method of the effective throughput for unidirectional
>communcation to an ISP along the link from ISP to network provider.
>Of course, this is of only limited value for obvious reasons.

You may call it anything you wish and you may be more technically
accurate. The term "Ping latency test" suits me just fine.


>: speeds. Bing
>: results purport to show actual total line speeds rather than speed
>: available to
>: your host.
>
>Actually, no, it displays a weighted average throughput and
>packet loss, which is really the only useful information
>you can gain from single-host examination.


Perhaps you are referring to some other Bing. The Bing I refer
to is based on Bing 1.0.4 and available at
http://www.mygale.org/~fgouget/apps/bing_src-1.0.4.tar.gz .

I'll quote the readme file. Note that the writer(s) of this document
appear to use bandwidth and throughput interchangably.


*Bing is a point-to-point bandwidth measurement tool (hence the
*'b'), based on ping.
*
*Bing determines the real (raw, as opposed to available or average)
*throughput on a link by measuring ICMP echo requests roundtrip times
*for different packet sizes for each end of the link.

<snip>

*If we do that for two different packet sizes, we can compute the
*raw capacity (bps) of the link.



The readme.txt file from the "Unofficial Release" of Bing 1.1.3
available at http://www.mygale.org/~fgouget/apps/bing_src-1.1.3.tar.gz
has the same information

The bing.txt file from that Bing 1.1.3 has this:

*NAME
*       bing  -  compute point to point throughput using two sizes
*       of ICMP ECHO_REQUEST packets to pairs of remote hosts.
*
*DESCRIPTION
*       Bing determines bandwidth  on  a  point-to-point  link  by
*       sending  ICMP  ECHO_REQUEST  packets  and  measuring their
*       roundtrip times for different packet sizes on each end  of
*       the link.


I suggest that the writers of these documents and source code have
used the terms bandwidth and throughput interchangeably and incorrectly
and that you may have become similarly confused.

The readme.txt file from both Bings also contains:

*5. Measurement problems
*-----------------------
*
*There are many cases in which the measurements may not be accurate
*(read:  "plain wrong") :
*
*	- saturated links. bing works by measuring the minimal rtts.
*	  The more saturated links there are in the measure path,
*	  the more time it takes to get a packet through all of
*	  them with minimal delay.


I would suggest that attempting to use Bing to estimate throughput
from the testing host to some other host is invalid and merely
demonstrates this problem.

Running Bing on my own computer to estimate the bandwidth of Rogers
T1 link from Ottawa to Toronto produces figures from about 1.2 to 1.9
Mbps . My cable modem is limited to about 512 Kbps. If Bing was
measuring my throughput it would show figures much closer to 500 Kbps.


>I don't understand these http tests;  the tests would be so
>algorithm-dependent that I would advise a high degree
>of caution in presenting the results.

Perhaps that would be appropriate. OTOH, I beleive that as long as
one is reasonably aware of the limitations of the data, having
some data, even raw data, is better than having no data at all.
We could argue all day long about how useful it is or isn't but I
really don't wish to argue, beyond defending myself from misconceptions
and the like.


Mike.

-------------------==== Posted via Deja News ====-----------------------
      http://www.dejanews.com/     Search, Read, Post to Usenet

On 21 Aug 1997 20:49:42 GMT, mcr@sandelman.ocunix.on.ca (Michael
Richardson) wrote:

>  Most of the backbone ISP's have monitoring already. The IETF ISP
>metrics group will be standardizing the collection and reporting of
>these numbers.

I'm aware that they monitor and generally don't disclose this info.
Will they make the results of this monitoring publicly available ?


>  Default on SunOS is to do exactly that.

News to me, but interesting.

>  Ping transit times and packet lossage rates *are* of interest to
>technicians (and yes, you count),

You may call me whatever you wish. I enjoy academic debate from time to
time but find it gets little real work accomplished. I prefer results to
debate.


>but are not very interesting to end
>users.

Joe User cares most about how long it takes to load a web page after
he presses a button. Anything else we use as a tool or a benchmark
can only simulate this imperfectly. It's arguable that nothing else
but the results seen by 10 million Joe Users are important. We could
also argue endlessly about the existence of gods or whether bees can
fly or not...


What is it that you are suggesting I should do ? Are you suggesting I
should remove the ping data from my pages ? Use the gaudy BLINK attribute
for my disclaimers ? There are many other sites out there showing ping
times to hosts from a single source. Would you suggest they shut down ?
Or is all this just because my webpages are intended to assist people in
the selection of an ISP ?


Mike.

-------------------==== Posted via Deja News ====-----------------------
      http://www.dejanews.com/     Search, Read, Post to Usenet

On Tue, 26 Aug 1997 16:36:07 -0600, mreid@rogers.wave.ca wrote:

>
>Joe User cares most about how long it takes to load a web page after
>he presses a button. 

Well, that's one thing Jane User cares about along with, not in any
specific order, how long it takes to connect to a website, reliable
email, reliable newsfeed, few or no line drops, good polite
explanations of any problems, reasonable cost for service provided,
reasonable billing options, and few or no busy signals when dialing
in.

As far as I can tell, no ISP has figured out how to do all of these.

sph
sandrap@flora.ottawa.on.ca  http://www.flora.org/sandra/
Ottawa Area Homeschooling   http://www.flora.org/hbln/
Nature, praise be, neither talks nor is rational, and 
therein is comfort. -John A Livingston