Thread View: comp.windows.x
1 messages
1 total messages
Started by Art Mulder
Tue, 15 Jun 1993 15:55
comp.windows.x: Getting more performance out of X. FAQ
Author: Art Mulder
Date: Tue, 15 Jun 1993 15:55
Date: Tue, 15 Jun 1993 15:55
677 lines
30668 bytes
30668 bytes
Archive-name: x-faq/speedups
Last-modified: 1993/4/20
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
HOW TO MAXIMIZE THE PERFORMANCE OF X -- monthly posting
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Compiled by Art Mulder (art@cs.ualberta.ca)
More RAM, Faster CPU's, More disk space, Faster Ethernet... These
are the standard responses you hear when you ask how to improve the
performance of your workstation.
Well, more hardware isn't always an option, and I wonder if more
hardware is always even a necessity.
This "FAQ" list is a collection of suggestions and ideas from different
people on the net on how you can the best possible performance from X
Windows on your workstation, WITHOUT PURCHASING MORE HARDWARE.
Performance is a highly subjective issue. The individual user must
balance `speed' versus `features' in order to come to a personal
decision. Therefore this document can be be expected to contain many
subjective opinions in and amongst the objective facts.
This document is specifically concerned with X. There are of course
many other factors that can affect the performance of a workstation.
However, they are outside the scope of this document.
[ People seriously interested in the whole area of system
performance, might want to look at the O'Reilly Nutshell Handbook
"System Performance Tuning" by Mike Loukides. IMHO, it contains a
well-written, comprehensive treatment of system performance. I'm
unaware of any other similar books. --ed.]
-----------------
Table of Contents
-----------------
0. Introduction & Administrivia
1. What about the "Other X FAQ"?
2. Window Managers
3. The X Server
Which Server?
Locking the Server into RAM?
Starting your Server
Fonts
About the Resources File
Define Your Display Properly
4. Clients
A Better Clock for X
A Better Terminal Emulator for X
Tuning your client
5. Miscellaneous Suggestions
Pretty Pictures
A Quicker Mouse
Programming Thoughts
Say What!?
6. Other Sources of Information
7. Author & Notes
! = changed since last issue.
* = new since last issue.
-----------------------------
Introduction & Administrivia
-----------------------------
This document is posted each month, on or around the 15th, to the
Usenet news groups comp.windows.x, news.answers, and comp.answers.
If you are reading a copy of this FAQ which is more than a few
months old (see the "Last-modified" date above) you should probably
locate the latest edition, since the information may be outdated.
If you do not know how to get those newsgroups and/or your site does
not receive them and/or this article has already expired, you can
retrieve this FAQ from an archive site.
There exist several usenet FAQ archive sites. To find out more about
them and how to access them, please see the "Introduction to the
news.answers newsgroup" posting in news.answers.
The main FAQ archive is at rtfm.mit.edu. This document can be found
there in /pub/usenet/news.answers/x-faq/speedups. If you do not have
access to anonymous ftp, you can retrieve it by sending a mail
message to mail-server@rtfm.mit.edu with the command "send
usenet/news.answers/x-faq/speedups" in the message body.
-----------------------------
What about the "Other X FAQ"?
-----------------------------
David B. Lewis (faq%craft@uunet.uu.net) maintains the informative and
well written "comp.windows.x Frequently Asked Questions" document.
Its focus is on general X information, while this FAQ concentrates
on performance.
The comp.windows.x FAQ does address the issue of speed, but only with
regards to the X server. The gist of that topic seems to be:
"Use X11R5, it is faster than R4".
(Please see the X FAQ for complete details).
---------------
Window Managers
---------------
There are a lot of window managers out there, with lots of different
features and abilities. The choice of which to use is by necessity a
balancing act between performance and useful features. At this
point, most respondents have agreed upon "twm" as the best candidate
for a speedy window manager.
A couple of generic tricks you can try to soup up your window manger,
is turning off unnecessary things like "zooming" and "opaque move".
Also, if you lay out your windows in a tiled manner, you reduce the
amount of cpu power spent in raising and lowering overlapping
windows. Joe English (joe@trystero.art.com)
I've found that a good font for tiling is 7x13 (aka:
-misc-fixed-medium-r-normal--13-100-100-100-c-70-iso8859-1 ). It is
the biggest font I know of that I can use on my Sun (1152x900 screen)
and still get two 80 column terminal windows side-by-side on the
display with no overlap. Other font suggestions will be accepted.
------------
The X Server
------------
Which Server?
- - - - - - -
Make sure that your server is a proper match for your hardware.
If you have a monochrome monitor, use a monochrome X11 server.
On my Monochrome Sun, I haven't noticed much difference between
the Xsun (colour) server and XsunMono, however it was pointed out to
me that XsunMono is about 800k smaller and therefore should contribute
to less paging.
[ thanks to: Jonny Farringdon (j.farringdon@psychol.ucl.ac.uk),
Michael Salmon (Michael.Salmon@eos.ericsson.se) ]
How your server was compiled can also make a difference. Jeff Law
(law@schirf.cs.utah.edu) advises us that on a Sun system, X should be
compiled with gcc (version 2.*) or with the unbundled Sun compiler.
You can expect to get "*very* large speedups in the server" by not
using the bundled SunOS compiler. I assume that similar results
would occur if you used one of the other high-quality commercial
compilers on the market.
Locking the Server into RAM?
- - - - - - - - - - - - - - -
Has anyone tried hacking the X server so that it is locked into RAM and
does not get paged? eg: via a call to plock(). Does this help
performance at all? I've had one inquiry on this topic, and a few
pointers to the plock() function call, but no hard evidence from someone
who's tried it. I am not in a position to give it a try.
[thanks to: Eric C Claeys (ecc@eperm.att.com),
Danny Backx (db@sunbim.be),
Juan D. Martin (juando@cnm.us.es) ]
Starting your Server
- - - - - - - - - - -
Joe English (joe@trystero.art.com) :
If you start up a lot of clients in your .xsession or whatever, sleep
for a second or two after launching each one. After I changed my
.xclients script to do this, logging in actually took *less* time...
we have a heavily loaded system without much core, though.
This sounds crazy, but I have confirmed that it works!
Warner Losh (imp@Solbourne.COM) provided me with a good explanation of
why this works, which I have summarized here:
When you start up an X server it takes a huge amount of time to
start accepting connections. A lot of initialization is done by
the server when it starts. This process touches a large number of
pages. Any other process running at the same time would fight the
server for use of the CPU, and more importantly, memory. If you
put a sleep in there, you give the Server a chance to get itself
sorted out before the clients start up.
Similarly, there is also a lot of initialization whenever an X
client program starts: toolkits registering widgets, resources
being fetched, programs initializing state and "databases" and so
forth. All this activity is typically memory intensive. Once this
initialization is done ("The process has reached a steady state"),
the memory usage typically settles down to using only a few pages.
By using sleeps to stagger the launching of your clients in your
.Xinitrc , you avoid them fighting each other for your
workstation's limited resources
This is most definitely a "Your Mileage May Vary" situation, as there
are so many variables to be considered: available RAM, local swap
space, load average, number of users on your system, which clients
you are starting, etc.
Currently in my .xinitrc I have a situation like:
(sleep 1; exec xclock ) &
(sleep 1; exec xbiff ) &
(sleep 1; exec xterm ) &
(sleep 1; exec xterm ) &
I've experimented with:
(sleep 1; exec xclock ) &
(sleep 2; exec xbiff ) &
(sleep 3; exec xterm ) &
(sleep 4; exec xterm ) &
I've even tried:
(sleep 2; exec start_X_clients_script ) &
and then in start_X_clients_script I had:
(sleep 1; exec xclock ) &
(sleep 1; exec xbiff ) &
(sleep 1; exec xterm ) &
(sleep 1; exec xterm ) &
[ The idea with this last one was to make sure that xinit had
completely finished processing my .xinitrc, and had settled down
into a "steady state" before the sleep expired and all my clients
were launched. ]
All of these yielded fairly comparable results, and so I just stuck with
my current setup, for its simplicity. You will probably have to
experiment a bit to find a setup which suits you.
Fonts
- - -
Loading fonts takes time and RAM. If you minimize the number of fonts
your applications use, you'll get speed increases in load-up time.
One simple strategy is to choose a small number of fonts (one small, one
large, one roman, whatever suits you) and configure all your clients -- or
at least all your heavily used clients -- to use only those few fonts.
Client programs should start up quicker if their font is already loaded
into the server. This will also conserve server resources, since fewer
fonts will be loaded by the server.
[ Farrell McKay (fbm@ptcburp.ptcbu.oz.au),
Joe English (joe@trystero.art.com) ]
eg: My main xterm font is 7x13, so I also have twm set up to use 7x13
in all it's menus and icons etc. Twm's default font is 8x13. Since
I don't normally use 8x13, I've eliminated one font from my server.
Oliver Jones (oj@roadrunner.pictel.com):
Keep fonts local to the workstation, rather than loading them over nfs.
If you will make extensive use of R5 scalable fonts, use a font server.
About the Resources File
- - - - - - - - - - - - -
Keep your .Xresources / .Xdefaults file small. Saves RAM and saves
on server startup time. Joe English (joe@trystero.art.com)
One suggestion:
In your .Xdefaults (.Xresources) file, try putting only the minimum
number of resources that you want to have available to all of your
applications. For example: *reverseVideo: true
Then, separate your resources into individual client-specific
resource files. For example: $HOME/lib/app-defaults. In your
.login file set the environment variable XUSERFILESEARCHPATH:
setenv XUSERFILESEARCHPATH $HOME/lib/app-defaults/%N
[ The "comp.windows.x Frequently Asked Questions" FAQ contains
an excellent explanation of how these environment variables work.
--ed.]
So, when xterm launches, it loads its resources from
.../app-defaults/XTerm. Xdvi finds them in .../app-defaults/XDvi,
and so on and so forth. Note that not all clients follow the same
XXxxx resource-file naming pattern. You can check in your system
app-defaults directory (often: /usr/X11R5/lib/X11/app-defaults/) to
find the proper name, and then name your personal resource files
with the same name.
This is all documented in the Xt Specification (pg 125 & 666).
[Thanks to: Kevin Samborn (samborn@mtkgc.com),
Michael Urban (urban@cobra.jpl.nasa.gov),
and Mike Long (mikel@ee.cornell.edu).
Kevin is willing mail his setup files to inquirers.]
This method of organizing your personal resources has the following
benefits:
- Easier to maintain / more usable.
- Fewer resources are stored in the X server in the RESOURCE_MANAGER
property. As a side benefit your server may start fractionally
quicker, since it doesn`t have to load all your resources.
- Applications only process their own resources, never have to sort
through all of your resources to find the ones that affect them.
It also has drawbacks:
- the application that you are interested in has to load an
additional file every time it starts up. This doesn't seem to
make that much of a performance difference, and you might
consider this a huge boon to usability. If you are modifying an
application's resource database, you just need to re-run the
application without having to "xrdb" again.
- xrdb will by default run your .Xdefaults file through cpp. When
your resources are split out into multiple resource files and
then loaded by the individual client programs, they will not.
WATCH OUT FOR THIS!!
I had C style comments in my .Xdefaults file, which cpp stripped
out. When I switched to this method of distributed resource
files I spent several frustrating days trying to figure out why
my clients were not finding their resources. Xt did *NOT*
provide any error message when it encountered the C style
comments in the resource files, it simply, silently, aborted
processing the resource file.
The loss of preprocessing (which can be very handy, e.g. ``#ifdef
COLOR'' ...) is enough to cause some people to dismiss this
method of resource management.
- You may also run into some clients which break the rules. For
example, neither Emacs (18.58.3) nor Xvt (1.0) will find their
resources if they are anywhere other than in .Xdefaults.
- when starting up a client on a machine that does not share files
with the machine where your resources are stored, your client
will not find its resources. Loading all your resources into the
server will guarantee that all of your clients will always find
their resources. Casey Leedom (casey@gauss.llnl.gov)
A possible compromise suggestion that I have (and am planning on trying)
is to put resources for all my heavily used clients (eg: xterm) into my
.Xdefaults file, and to use the "separate resources files" method for
clients that I seldom use.
Define Your Display Properly
- - - - - - - - - - - - - - -
Client programs are often executed on the same machine as the server. In
that situation, rather than setting your DISPLAY environment variable to
"<hostname>:0.0", where <hostname> is the name of your workstation, you
should set your DISPLAY variable to "unix:0.0" or ":0.0". By doing this
you access optimized routines that know that the server is on the same
machine and use a shared memory method of transferring requests.
[thanks to Patrick J Horgan (pjh70@ras.amdahl.com)]
See the _DISPLAY NAMES_ section of the X(1) man page for further
explanation of how to properly set your display name.
"I don't think it's stock MIT, but (at least) Data General and HP have
libraries that are smart enough to use local communication even when
the DISPLAY isn't set specially."
Rob Sartin (88opensi!sartin@uunet.UU.NET)
[Jody Goldberg (jody@algorithmics.com) sent me an Xlib patch to change
stock R5 to use local communication even if DISPLAY is not properly set.
I don't want to get in the business of distributing or trying to juggle
non-MIT patches and so have elected not to include it here. Hopefully MIT
will apply this minor (~8 lines) patch themselves. In the meantime, if
you want to try it yourself, email Jody. --ed.]
-------
Clients
-------
If you only have a few megabytes of Ram then you should think
carefully about the number of programs you are running. Think also
about the _kind_ of programs you are running. For example: Is there
a smaller clock program than xclock?
Unfortunately, I haven't really noticed that programs advertise how large
they are, so the onus is on us to do the research and spread the word.
[ Suggestions on better alternatives to the some of the standard clients
(eg: Xclock, Xterm, Xbiff) are welcome. --ed.]
I've received some contradictory advice from people, on the subject
of X client programs. Some advocate the use of programs that are
strictly Xlib based, since Xt, Xaw and other toolkits are rather
large. Others warn us that other applications which you are using
may have already loaded up one or more of these shared libraries. In
this case, using a non-Xt (for example) client program may actually
_increase_ the amount of RAM consumed.
The upshot of all this seems to be: Don't mix toolkits. That is, try
and use just Athena clients, or just Xview clients (or just Motif
clients, etc). If you use more than one, then you're dragging in
more than one toolkit library.
Know your environment, and think carefully about which client
programs would work best together in that environment.
[Thanks to: Rob Sartin (88opensi!sartin@uunet.UU.NET),
Duncan Sinclair (sinclair@dcs.gla.ac.uk | sinclair@uk.ac.gla.dcs) ]
A Better Clock for X
- - - - - - - - - - -
1) xcuckoo
suggested by: Duncan Sinclair (sinclair@dcs.gla.ac.uk)
available: on export.lcs.mit.edu
Xcuckoo displays a clock in the title bar of *another* program.
Saves screen real estate.
2) mclock
suggested by: der Mouse (mouse@Lightning.McRCIM.McGill.EDU)
available: larry.mcrcim.mcgill.edu (132.206.1.1) in /X/mclock.shar
Non Xt-based. Extensively configurable. it can be made to look
very much like MIT oclock, or mostly like xclock purely by changing
resources.
Of course, the ultimate clock --- one that consumes no resources, and
takes up no screen real estate --- is the one that hangs on your wall.
:-)
A Better Terminal Emulator for X
- - - - - - - - - - - - - - - - -
From the README file distributed with xterm:
+-----
| Abandon All Hope, Ye Who Enter Here
|
| This is undoubtedly the most ugly program in the distribution.
| ...
+-----
Ugly maybe, but at my site it's still the most used. I suspect that
xterm is one of the most used clients at many, if not most sites.
Laziness? Isn't there a better terminal emulator available? See below.
If you must use xterm, you can try reducing the number of saveLines
to reduce memory usage. [ Oliver Jones (oj@roadrunner.pictel.com),
Jonny Farringdon (j.farringdon@psychol.ucl.ac.uk) ]
1) Xvt
suggested by: Richard Hesketh (rlh2@ukc.ac.uk) :
available: export.lcs.mit.edu in /contrib/xvt-1.0.tar.Z
"...if you don't need all the esoteric features of xterm, then get
hold of xvt ... it was written here just to save swap space as
xterm is rather a hog! "
This was written as a partial 'clone' of xterm. You don't have to
rename your resources, as xvt pretends to be XTerm. In it's current
version, you cannot bind keys as you can in xterm. I've heard that
there are versions of xvt with this feature, but I've not found any
yet.
UPDATE (March 1993): I recently had a few email conversations with
Brian Warkentin (brian.warkentine@eng.sun.com) regarding xvt. He
questions whether xvt really is at all faster than xterm. For
instance, xvt may initialize slightly faster, but compare scrolling
speed (try this quickie benchmark: /bin/time dd if=/etc/termcap
bs=40) and see which program can scroll faster. Also, while xterm
may be slightly larger in RAM requirements (We don't have any hard
numbers here, does anyone else?) shared libraries and shared text
segments mean that xterm's paging requirements are not that major.
As an experiment, he ripped out all the tek stuff from xterm, but it
made little difference, since if you never use it, it never gets
brought into memory.
So here we stand with some conflicting reports on the validity of
xvt over xterm. In summary? Caveat Emptor, your mileage may vary.
If you can provide some hard data, I'd like to see it.
Specifically: How much RAM each occupies, how much swap each needs,
relative speed of each
2) mterm
suggested by: der Mouse (mouse@Lightning.McRCIM.McGill.EDU)
available: larry.mcrcim.mcgill.edu (132.206.1.1) in
/X/mterm.src/mterm.ball-o-wax.
"I also have my own terminal emulator. Its major lack is
scrollback, but some people like it anyway."
Tuning your client
- - - - - - - - - -
Suggestions on how you can tune your client programs to work faster.
From Scott Barman (scott@asd.com) comes a suggestion regarding Motif
Text Field Widgets:
I noticed that during data entry into Motif text field widgets, I
was getting a slight lag in response to some keystrokes,
particularly the initial one in the field. Examining the what was
going on with xscope I found it. It seems that when the resource
XmNblinkRate is non-zero and the focus is on a text field widget
(or even just a text widget) the I-beam cursor will blink.
Every time the cursor appears or disappears in those widgets, the
widget code is making a request to the server (CopyArea). The user
can stop this by setting the resource XmNblinkRate to 0. It is not
noticeable on a 40MHz SPARC, but it does make a little difference
on a [slower system].
This specific suggestion can probably be applied in general to lots
of areas. Consider your heavily used clients, are there any minor
embellishments that can be turned off and thereby save on Server
requests?
-------------------------
Miscellaneous Suggestions
-------------------------
Pretty Pictures
- - - - - - - -
Don't use large bitmaps (GIF's, etc) as root window backgrounds.
- The more complicated your root window bitmap, the slower the server
is at redrawing your screen when you reposition windows (or redraw, etc)
- These take up RAM, and CPU power. I work on a Sun SPARC and I'm
conscious of performance issues, I can't comprehend it when I see
people with a 4mb Sun 3/60 running xphoon as their root window.
I'll let someone else figure out how much RAM would be occupied by
having a full screen root image on a colour workstation.
- If you're anything like me, you need all the screen real estate
that you can get for clients, and so rarely see the root window anyway.
[ Thanks to Qiang Alex Zhao (azhao@cs.arizona.edu)
for reminding me of this one. --ed.]
A Quicker Mouse
- - - - - - - -
Using xset, you can adjust how fast your pointer moves on the screen
when you move your mouse. I use "xset m 3 10" in my .xinitrc file,
which lets me send my pointer across the screen with just a flick of
the wrist. See the xset man page for further ideas and information.
Hint: sometimes you may want to *slow down* your mouse tracking for
fine work. To cover my options, I have placed a number of different
mouse setting commands into a menu in my window manager.
e.g. (for twm) :
menu "mouse settings" {
"Mouse Settings:" f.title
" Very Fast" ! "xset m 7 10 &"
" Normal (Fast)" ! "xset m 3 10 &"
" System Default (Un-Accelerated)" ! "xset m default &"
" Glacial" ! "xset m 0 10 &"
}
Programming Thoughts
- - - - - - - - - - -
Joe English (joe@trystero.art.com) :
To speed up applications that you're developing, there are tons of
things you can do. Some that stick out:
- For Motif programs, don't set XmFontList resources for individual
buttons, labels, lists, et. al.; use the defaultFontList or
labelFontList or whatever resource of the highest-level manager
widget. Again, stick to as few fonts as possible.
- Better yet, don't use Motif at all. It's an absolute pig.
- Don't create and destroy widgets on the fly. Try to reuse them.
(This will avoid many problems with buggy toolkits, too.)
- Use a line width of 0 in GCs. On some servers this makes a HUGE
difference.
- Compress and collapse multiple Expose events. This can make the
difference between a fast application and a completely unusable
one.
Francois Staes (frans@kiwi.uia.ac.be) :
Just a small remark: I once heard that using a better malloc
function would greatly increase performance of Xt based
applications since they use malloc heavily. They suggested trying
out the GNUY malloc, but I didn't find the time yet. I did some
tests on small programs just doing malloc and free, and the
differences were indeed very noticeable ( somewhat 5 times faster)
[ Any confirmation on this from anyone? --ed.]
Andre' Beck (Andre_Beck@IRS.Inf.TU-Dresden.de) :
- Unnecessary NoExpose Events.
Most people use XCopyArea/XCopyPlane as fastest blit routines, but
they forget to reset graphics_exposures in the GC used for the
blits. This will cause a NoExpose Event every blit, that, in most
cases, only puts load onto the connection and forces the client to
run through it's event-loop again and again.
- Thousands of XChangeGC requests.
This "Gfx Context Switching" is also seen in most handcoded X-Apps,
where only one or few GCs are created and then heavily changed
again and again. Xt uses a definitely better mechanism, by caching
and sharing a lot of GCs with all needed parameters. This will
remove the load of subsequent XChangeGC requests from the
connection (by moving it toward the client startup phase).
Say What!?
- - - - - -
Some contributors proposed ideas that seem right off the wall at first:
David B. Lewis (by day: dbl@osf.org, by night: david%craft@uunet.uu.net) :
How about this: swap displays with someone else. Run all your programs
on the other machine and display locally; the other user runs off your
machine onto the other display. Goal: reduce context switches in the
same operation between client and server.
I'm not in a situation where I can easily try this, but I have received
the following confirmation...
Michael Salmon (Michael.Salmon@eos.ericsson.se):
I regularly run programs on other machines and I notice a big
difference. I try to run on a machine where I will reduce net usage
and usually with nice to reduce the impact of my intrusion. This
helps a lot on my poor little SS1+ with only 16 MB, it was
essential when I only had 8 MB.
Casey Leedom (casey@gauss.llnl.gov) :
[The X11 Server and the client are] competing for the same CPU as
your server when you run it on the same machine. Not really a
major problem, except that the X11 client and the server are in
absolute synchronicity and are context thrashing.
Timothy H Panton (thp@westhawk.uucp) :
Firstly it relies on the fact that most CPU's are mostly idle, X's
cpu usage is bursty. so the chances of you and your teammate
doing something cpu-intensive at the same time is small. If they
are not then you get twice the cpu+memory available for your
action.
The second factor is that context switches are expensive, using 2
cpu's halves them, you pay a price due to the overhead of going
over the network, but this is offset in most cases by the improved
buffering of a network (typically 20k vs 4k for a pipe), allowing
even fewer context switches.
----------------------------
Other Sources of Information
----------------------------
Volume 8 in O'Reilly's X Window System Series, ``X Window System
Administrator's Guide'' is a book all X administrator's should read.
Adrian Nye (adrian@ora.com):
A lot more tips on performance are in the paper "Improving X
Application Performance" by Chris D. Peterson and Sharon Chang, in
Issue 3 of The X Resource.
An earlier version of this paper appeared in the Xhibition 1992
conference proceedings.
This paper is absolutely essential reading for X programmers.
--------------
Author & Notes
--------------
This list is currently maintained by Art Mulder (art@cs.ualberta.ca)
Suggestions, corrections, or submission for inclusion in this list
are gladly accepted. Layout suggestions and comments (spelling
mistak's too! :-) are also welcome.
Currently I have listed all contributors of the various comments and
suggestions. If you do not want to be credited, please tell me.
speedup-x-faq is copyright (c) 1993 by Arthur E. Mulder
You may copy this document in whole or in part as long as you don't
try to make money off it, or pretend that you wrote it.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
--
...art mulder ( art@cs.ualberta.ca ) | "Do not be conformed to this world,
Department of Computing Science | but be transformed by the renewal
University of Alberta, Edmonton, Canada | of your mind, ..." Romans 12:2
----------
Thread Navigation
This is a paginated view of messages in the thread with full content displayed inline.
Messages are displayed in chronological order, with the original post highlighted in green.
Use pagination controls to navigate through all messages in large threads.
Back to All Threads