🚀 go-pugleaf

tedwards@qa.nad.northrop.com (Tony Edwards) writes:

>In article <1992Jul29.210951.18597@wvus.org> pete@wvus.org (Pete Gregory) writes:
>>Hi - I need to learn sendmail - not just understand what it does, but I 
>>need to become an expert, soon.
>>
>>I'd appreciate any recommendations on good classes and books on sendmail.
>>
>>Thanks...
>>-- 
>*******************************************************************
>I was curious to know if anyone had sent you any information on 
>sendmail, i would appricate it if you could forward anything you
>recieve to me. 
>Thanks TOny

Here is a couple of txt files describing sendmail that I found:

From: Eliot Lear <lear@NET.BIO.NET>

The following was written by Dr. Charles Hedrick of Rutgers University
sometime in 1985.  Please read it with the understanding that rule 
numbers are nothing more than function names.  For further reference,
I suggest the Sun Tutorial on Sendmail in their manuals.
-eliot

>Command: followup
>Newsgroups: net.unix-wizards,net.mail
>To: steve@jplgodo.UUCP
>Subject: a brief tutorial on sendmail rules
>Distribution: 
>References: <902@rlgvax.UUCP> <545@jplgodo.UUCP>

A previous message suggested using "sendmail -bt" to see how sendmail
is going to process an address.  This is indeed a handy command for
testing how an address will be processed.  However the instructions
given were not quite right.  To see how sendmail is going to deliver
mail to a given address, a reasonable thing to type is
	sendmail -bt
	0,4 address
Even this isn't quite right, but with "normal" rule sets it should work.

Because there is so much confusion about sendmail rules, the rest of
this message contains a brief tutorial.  My own opinion of sendmail is
that it is quite a good piece of work.  Many people have complained
about the difficulty of understanding sendmail rule sets.  However I
have also worked with mailers that code address processing directly
into the program.  I much prefer sendmail.  The real problem is not
with sendmail, but with the rules.  The rules normally shipped from
Berkeley have lots of code that does strange Berkeley-specific things,
and they are not commented.  Also, typical complex rule sets are
trying to handle lots of things, forwarding mail among several
different mail systems with incompatible addressing conventions.  A
rule set to handle just old-style (non-domain) UUCP mail would be very
simple and easy to understand.  But real rule sets are not doing
simple things, so they are not simple.

For those not familiar with sendmail, -bt invokes the rule tester.  It
lets you type a set of rule numbers and an address, and then shows you
what the rules will do to that address.  In addition, rule test mode
automatically applies rule 3 before whatever rule you ask it to apply.
As we will see shortly, this is a reasonable thing to do.

Before describing the rule sets, let me define two terms: "header" and
"envelope".  Header refers to the lines at the beginning of the
message, starting with "from:", "to:", "subject:", etc.  Sendmail does
process these lines.  E.g. with uucp mail it will add its own host
name at the beginning of the from line, so that the final recipient
stands some change of replying to the message.  However sendmail
normally does not depend upon the from and to lines to perform its
actual delivery.  It has more direct knowledge, passed on to it from
the program that generated the mail, or if it came from another site,
the mailer at that site.  This information is referred to as the
"envelope", since it is like the addresses on the outside of an
envelope.  For Arpanet mail, the envelope is passed to the next site
by the MAIL FROM: and RCPT TO: commands.  For UUCP mail, it is passed
on as arguments to the remote rmail command.  To see why there have to
be separate addresses "on the envelope", consider what happens when
you send mail to "john@vax, mary@sun".  Two copies of the message will
be dispatched, one to vax and the other to sun.  The "to: " line in
the headers will show both addresses.  However the envelope will show
only the right address that we want this copy to go to.  The copy sent
to vax will show "john@vax" and the copy sent to sun will show
"mary@sun".  If sendmail had to look at the "to: " line, it would
never know which of the addresses shown there it was responsible for
handling.

Anyway, here is what the rules do:

3: always done first.  This turns addresses from their normal textual
form into a form that the rest of the rules understand.  In most
cases, all it does it put < > around the name of the host that is next
in line.  Thus foo@bar turns into foo<@bar>.  However it also does a
few transformations.  E.g. it turns foo!bar!user into
bar!user<@foo.UUCP>.  Since sendmail accepts either ! syntax or
@....UUCP syntax, rule 3 standardizes on @ syntax.  It also does a few
other minor things.  But you won't be far off if you just think of it
as adding < > around the host name.

4: always done last.  This turns addresses from internal form  back
into external form.  It removes the < > around the host name, and
turns foo@bar.UUCP back into bar!foo.  Again, there are one or two
other minor things, but you won't be too far off if you think of 4 as
just removing the < > around the host name.

0: This is the rule that handles the destination address on the
envelope.  It is in some sense the primary rule.  It returns a triple:
protocol, host, user.  The protocol is usually one of local, TCP, or
UUCP.  At the moment, it figures this out syntactically.  In our rule
set, hosts ending in .UUCP are handled by UUCP, the current host is
local, and everything else is TCP.  As domains are integrated into
UUCP, obviously this rule is going to change.  This rule does very
little other than simply look at the format of the host name, though
as usual a few other details are involved (e.g. it removes the local
host.  So myhost!foo!bar will be sent directly to foo).

1 and 2 are protocol-independent transformations used for sender and
recipient lines in the header (i.e. from: and to: lines).  In our
rule sets, they don't do anything.

Each protocol has its own rules to use for sender and recipient lines
in the header.  E.g. UUCP rules might add the local host name to the
beginning of the from line and remove it from the to line.  In our
rule set, the complexities in these rules are primarily caused by
forwarding between UUCP and TCP.  The line that defines the mailer for
a protocol lists the rule to use for source and recipient, in the S=
and R=.

Finally, here is the exact sequence in which these rules are used.
For example, the first line means that the destination specified in
the envelope is processed first by rule 3, then rule 0, then rule 4.

envelope recipient:    3,0,4  [actually rule 4 is applied only to the
				user name portion of what rule 0 returns]
envelope sender:       3,1,4
header recipient:      3,2,xx,4  [xx is the rule number specified in R=]
header sender:         3,1,xx,4  [xx is the rule number specified in S=]

I have the impression that the sender from the envelope (the
return-path) may actually get processed twice, once by 3,1,4 and the
second time by 3,1,xx,4.  However I'm not sure about that.

Now for the format of the rules themselves.  I'm just going to show
some examples, since sendmail comes with a reference manual, which you
can refer to.  However these examples are probably enough to let you
understand any set of rules that makes sense in the first place (which
the normal rules do not).  This example is from our UUCP definition.
It a simplified version of the set of rules used to process the sender
specification.  As such, the major thing it has to do is to add our
host name to the beginning, so that the guy at the end will know that
the mail went through us.

S13
R$+<@$-.UUCP>		$2!$1				u@host.UUCP => host!u
R$=U!$+			$2				strip local name
R$+			$:$U!$1				stick on our host name

Briefly, the first rule turns the address from the form foo<@bar.UUCP>
back into bar!foo.  The second rule removes our local host name, if
it happens to be there already, so we don't get it twice.  The third
rule adds our host name to the beginning.

S13 says that this is the beginning of a new rule set, number 13.

R$+<@$-.UUCP>		$2!$1				u@host.UUCP => host!u

R says that this is a rule.  The thing immediately after it,
$+<@$-.UUCP> is a pattern.  If this pattern matches the address, then
the rule "triggers".  If the rule triggers, the address is replaced
with the "right hand side", i.e. what is after the tab(s).  In this
rule, the right hand sie is $2!$1.  The thing after the next tab(s) is
a comment.  This rule is used in processing UUCP addresses.  As noted
above, by the time we get to it, rule 3 has already been applied.  So
if we had a UUCP address of the form host1!host2!user, it would now be
in the form host2!user<@host1.UUCP>.  This does match the pattern:

	$+	  <@$-   .UUCP>   
        host2!user<@host1.UUCP>

$+ and $- are "wildcards" that match anything.  $- will match exactly
one word, while $+ will match any number.  (By the way, with the
increasing use of domains, this production should probably use
$+.UUCP, not $-.UUCP.)  Since the pattern matches, we replace this
with the "right hand side" of the rule, $2!$1.  $ followed by a digit
means the Nth thing matched by a wildcard.  In this case there were
two wildcards, so
   $1 = host2!user
   $2 = host1
The final result is 
   host1!host2!user
As you can see, we have simply turned UUCP addresses from the format
produced by rule 3 back into normal ! format.

The second rule is

R$=U!$+			$2				strip local name

This is needed because there are situations in which our host name
ends up on the beginning of the recipient address.  Since we are
about to add our host name, we don't want it to be there twice.
So if it was there before, we remove it.  $= is used to see if
something is a member of a specified "class".  U happens to be a list
of our UUCP host name and any nicknames.  So $=U!$+ matches
any address that begins with our host name or nickname, then !, then
anything else.  Suppose we had  topaz!host1!host2!user.  The
match would be

     $=U  !$+
     topaz!host1!host2!user

The result of the match is that

     $1 = topaz
     $2 = host1!host2!user

Since the right hand side of this rule is simply "$2", the result is

     host1!host2!user

I.e. we have removed the topaz from the beginning.  By the way, the
class U used by the rule would have been defined earlier in the file
by the statement

CUtopaz ru-topaz

C defines a class.  U is the name of the class.  The rest of the
line is the list of things that will be in the class.

Finally we have the rule

R$+			$:$U!$1				stick on our host name

The $+ matches anything.  In this case the name is host1!host2!user, so the
result of the match is

	$1 = host1!host2!user

The result looks slightly obscure.  $: is a tag that says to do this
only once.  The problem is that this rule always applies, since the
pattern matches anything.  Normally, rules are applied over and
over, as long as they apply.  In this case, the result would be
an infinite loop.  Putting $: at the beginning says to do it only
once.  $U says to use the value of the macro U.  Earlier in the
file we defined U as our UUCP host name, with a definition

DUtopaz

Note that there can be a class and a macro with the same name.
$=U tests whether something is in the class U.  $U is replaced
by the value of the macro U.

So the final value of this rule, $:$U!$1, is

     topaz!host1!host2!user

So this rule has managed to add our host name to the beginning, as it
was supposed to.  Since there are no further rules in the set (the
next line is the end of file or the beginning of a new rule set),
this value is returned.

There are several more magic things that can appear in a pattern.
The most important are:

$* - this is another wild card.  It is similar to $+, but $+ matches
anything, whereas $* matches both anything and nothing.  I.e. $+
matches 1 or more tokens and $* matches 0 or more tokens.  So here
is a list of the wildcards I have mentioned:

   $*   0 or more
   $+   1 or more
   $-   exactly 1
   $=x  any member of class x

A typical example of $* is a production where we aren't sure whether
the user name is before or after the host name:

R$*<@$+.UUCP>$*		$@$1<@$2.UUCP>$3

This production would test for the host name ending in .UUCP, and
return immediately.  $@ is a flag you haven't seen yet.  It is simply
a return statement.  It causes the right hand side of this rule to be
returned as the final value of this rule set.

The other magic thing I will mention is $>.  This is a subroutine
call.  Here is an example taken from rule set 24, which is used to
process recipients in TCP mail.  Its purpose is to handle the
situation where we might have an address like topaz!user@red.  (Our
host name is topaz.  Red is a local host that we talk to via TCP.)
I.e. someone is asking us to relay mail to red.  Rule 3 will have
turned this into user@red<@topaz.UUCP>.  What we want to do is
get rid of the topaz.UUCP and treat red as the host.  (Rule set 0
would do this for the recipient on the envelope.  This rule is
used for the to: field in the header.)  Here is the rule.

R$+<@$=U.UUCP>		$@$>9$1				in case local!a@b

The pattern matches our example, as follows:

   $+      <@$=U  .UUCP>
   user@red<@topaz.UUCP>

Recall that $+ matches anything and $=U tests whether something is our
UUCP host name or one of our nicknames.  The result of the match is

   $1 = user@red
   $2 = topaz

The right hand side is $@$>9$1.  The $@ is the tag saying to stop the
rule set here and return this value.  $>9 is a subroutine call.  It
says to take the right hand side, pass it to rule set 9, and then
use the value of rule set 9.  The actual right hand side is simply
$1, which in this case is user@red.  Here is rule set 9:

S9
R$*<$*>$*		$1$2$3				defocus
R$+			$:$>3$1				make canonical
R$+			$@$>24$1			and do 24 again

The first rule simply removes < >.  It is sort of a quick and dirty
version of rule 4.  In fact we have no < > left, since we have removed
the <@topaz.UUCP>.  So this rule does not trigger.  (Now that I think
about it, I suspect it is probably never going to trigger, and so is
not needed.)  

The next rule is a simple subroutine call.  It matches anything ($+
matches any 1 or more token).  The right hand side is $:$>3$1 The $:
says to do it only once.  Since the rule matches anything, you need
this, or you will have an infinite loop.  The $>3 says to call rule 3
as a subroutine.  The $1 is the actual right hand side.  Since the
left hand side matched the whole address, what this rule does is
simply call rule set 3 on the whole address.  Recall that rule set 3
basically locates the host name and puts < > around it.  So in this
case the result is user<@red>.  As you can see, it was not enough to
remove <@topaz.UUCP>.  That leaves us with no  host name.  We have to
call rule 3 to find the current host name and put < > around it.

The last rule is really just a goto statement.  The pattern is $+,
which matches anything, so it always triggers.  The right hand side is
$@$>24$1.  The $@ is the return tag.  It says to stop this rule set
and return that value.  $>24 says to call rule set 24.  The actual
right hand side is $1, so we call rule set 24 with the whole address.
If you recall, this ruleset (9) was called from the  middle of 24 when
we found user@red<@topaz.UUCP>.  So what we have done is to change
this into user<@red> and say to start rule set 24 over again.

I hope you have found this exposition useful.  As a final convenience,
here is a "reference card" for reading rule sets.  Note that this
contains only operators used by the rules.  There are plenty of
other facilities used in the configuration section which I am
not documenting here.  (I'd love to see someone produce a complete
reference card.)

wildcards:
   $*   0 or more tokens
   $+   1 or more tokens
   $-   exactly one token
   $=x  member of class x (x must be a letter, lower/upper case distinct)
   $~x  not a member of class x

macro values (usable in pattern or on right hand side)
   $x	value of macro x (x must be a letter, lower/upper case distinct)
	At least on the Pyramid, $x is replaced by the macro's value
	when the sendmail.cf file is being read in.

on the right hand side:
   $n	string matched by the Nth wildcard
   $>n	call rule set N as a subroutine
   $@   return
   $:   only do this rule once

in rule 0, defining the return value
   $#   protocol
   $@   host
   $:   user

Rutgers extensions, usable only on right hand side
   $%n  take the string matched by the Nth wildcard, look it up in
	/etc/hosts, and if found use the primary host name
   $&x	use the current value of macro x.  x must be a letter.
	upper and lower case are treated as distinct.


>(Message inbox:31)
>Return-Path: comp.mail.sendmail
>From: rickert@mp.cs.niu.edu (Neil Rickert)
>Date: Mon, 29 Apr 1991 14:40:19 GMT
>Newsgroup: comp.mail.sendmail/2224
>Message-Id: <1991Apr29.144019.22206@mp.cs.niu.edu>
>Subject: Sendmail address rewrite sequence (was Re: From: , relaying and FAQ)

		 ----------------------------

                  ADDRESS REWRITING IN SENDMAIL.

 Prepared by Neil Rickert, Northern Illinois University, Apr 29, 1991.
 <rickert@cs.niu.edu>

 The author of this document has no relation whatsoever with the developers of 
sendmail at Berkeley.  Consequently any errors are in no way the fault of
Berkeley.  No guarantee is made as to the accuracy of this document.  Those
who wish guaranteed accurate information must read the source code themselves.

 Any comments or corrections are welcome.

                     -------------------------

 This description is intended to be a more thorough description of address 
parsing than is contained in the usual manual.  Some basic familiarity with 
the manual is assumed.  In particular, we do not attempt to explain the 
rewriting which occurs in a single rule, or the process in which an address 
passes from one rule to the next. 

 The description contains a few oversimplifications, such as omitting mention 
of some special cases during internal processing of alias expansions.  These 
simplifications are not directly related to the main issues covered.

 I make no guarantees as to the accuracy of this document.  It is based on 
experience with developing and debugging rulesets, and with reading parts of 
the code.  It mostly reflects my understanding of the status of sendmail based 
on the 5.65+IDA versions but with some cross checks with the Berkeley 5.65 
code. 

1.  A BRIEF OVERVIEW OF THE REWRITING PROCESS.

    Each address is initially rewritten by ruleset 3.  Thereafter the 
    processing depends on whether this is a recipient address or a sender 
    address.  A sender address is processed by ruleset 1, then by the ruleset 
    declared in the mailer, and finally by ruleset 4.  A recipient address, 
    after processing by ruleset 3, is normally processed by ruleset 2, the 
    ruleset declared in the mailer, and by ruleset 4. 

    The above description is, of course, a gross oversimplification.  We shall 
    fill in some of the details below. 

2.  BASIC ADDRESS PARSING.

    In order to send a message, the recipient address must be parsed, so as to 
    determine the transport mechanism and the next step in the chain of mail 
    relays.  This is handled in the function parseaddr(), which is in 
    parseaddr.c 

    The basic parsing strategy is as follows:

    As with all rewriting, the address is first rewritten by ruleset 3.  The 
    result of this rewrite is now rewritten by ruleset 0.  The purpose of 
    ruleset 0 is to resolve the address into a triple: Mailer, Host, 
    User_address.  This resolution occurs when the left hand side matches a 
    suitable rule, and the RHS is rewritten as: 

             $# MAILER_NAME $@ HOST_NAME $: USER_ADDRESS

    Apart from the exceptions noted below, every address must resolve into 
    such a triple.  If not 'sendmail' will complain.  We should clarify, 
    however, that the user address is allowed to contain a host component.  
    Thus a UUCP address of john@uunet.UUCP might resolve to:
                      $# UUCP $@ uunet $: john 
    while uunet!seismo!harry might resolve to
                       $# UUCP $@ uunet $: seismo!harry
    (In all the above examples the spacing is for readability.  We will comment 
    more on spacing when we discuss the function prescan() later in this 
    document). 

    It should also be clarified that the HOST_NAME as returned by ruleset 0 
    does not necessarily have anything to do with the domain component of the 
    original address.  It is the name of the host to which the message will 
    be next sent as known to the transmission software.  For the tcp mailers, 
    this is the fully qualified domain name.  For other mailers it may be 
    different.  In our example above, although the fully qualified uucp name 
    of uunet may be uunet.UUCP, the name known to the uucp software is just 
    uunet, so that must be the host name returned from ruleset 0.

    There are two exceptions to the above description.  The local mailer, and 
    the ERROR mailer. Both of these must resolve to just MAILER_NAME and 
    USER_ADDRESS, since a host does not make sense.  (The IDA versions of 
    'sendmail' permit the use of a triple with the local mailer, but ignore 
    the host).  In the case of the ERROR mailer, the USER_ADDRESS is actually 
    the error message.  Thus one could imagine a rewrite rule in ruleset 0. 

    R$+@annex1.$D           $#ERROR $:terminal servers don't accept mail 

    It is a basic principle of 'sendmail' that ruleset 0 must select a mailer, 
    even if only the ERROR mailer. 

    An address which resolves to the ERROR mailer is said to be 'unparseable' 
    and normally leads to bounced mail. 

3.  DIFFERENT CLASSES OF ADDRESSES: ENVELOPES AND HEADERS.

    'sendmail' must deal with addresses on the message envelope, and addresses 
    on the headers inside the message.  The distinction between envelope and 
    header is quite important for an understanding of how 'sendmail' 
    functions.  However it can be confusing at first. 

    In basic terms, the "header recipients" are the addresses on the "To:" and 
    "Cc:" header lines.  (We could include the "Bcc:" header, but this is 
    usually discarded.)  The "header senders" are the addresses on the "From:" 
    and the "Reply-To:" headers.  Sometimes there may also be "Resent-To:", 
    "Resent-Cc:", "Resent-From:" headers with additional header addresses. 

    The "envelope recipients" are the addresses the messages is really being 
    sent to.  They are often different from the "header recipients".  The 
    "envelope sender" is the real sender of the message, and is occasionally 
    different from the "header senders". 

    To the novice it often not obvious why there need be a distinction between 
    header and envelope addresses.  When a typical message originates the 
    envelope and header addresses are the same.  However much can change 
    during processing. A message might be sent to two recipients A and B on 
    different machines, host-A and host-B.  Initially both addresses are in 
    the envelope.  However when the message is relayed to host-A, only 
    recipient A remains in the envelope.  Likewise the copy of the message 
    sent to host-B will contain only B in the envelope.  Roughly speaking, as 
    a message is delivered to a recipient address, that address is deleted 
    from the envelope.  However all recipient addresses remain in the headers, 
    as these provide documentation to the human reader of the message as to 
    whom the message was sent to. 

    As another example, you may have a '.forward' file in your home directory 
    to forward your mail.  The forwarding action applies only to the envelope 
    recipients of a message, while the addresses on the header are not 
    affected. 

    The distinction between header senders and envelope sender is sometimes 
    harder to explain.  We start with an example unrelated to computing.  
    Suppose as a gift you purchase for a friend a subscription to "Scientific 
    American".  As part of the process you fill out a gift card which the 
    publishers will send.  When the gift card arrives, the card says it is 
    from you.  However the return address on the envelope is that of 
    Scientific American.  Essentially you were the header sender, but 
    Scientific American was the envelope sender.  For a comparable example in 
    computing, suppose you post an article in Usenet news.  This article comes 
    from you, and this is shown in the "From:" header.  However at some other 
    site, a copy of your article is mailed to someone without direct Usenet 
    news access.  The Usenet news system on the mailing host is the real 
    sender, or envelope sender, while as the author of the article you remain 
    the header sender. 

    In simple terms, the addresses in the headers are for humans to read, 
    while the addresses in the envelope are for machines to read. 

4.  WHERE DO ENVELOPES COME FROM AND WHERE DO THEY GO.

    When mail is sent or received over the network with SMTP, the envelope 
    sender is transmitted first in the 'MAIL-From' SMTP command.  Then a 
    sequence of 'RCPT-To' commands transfer the envelope recipients. 

    When you send a normal message on your system, the Mail User Agent (MUA) 
    invokes sendmail as:

    /usr/lib/sendmail recipient1 recipient2 recipient3 ...

    Thus the envelope recipients are passed as parameters on the argument 
    list.  The envelope sender, in that case, is taken from the uid of the 
    person sending the message.  It is also possible to invoke sendmail as 

    /usr/lib/sendmail -fsender recipient1 recipient2 recipient3 ...

    and thereby give a separate envelope sender.  However sendmail normally 
    will ignore the '-fsender' operand unless given by a trusted user such as 
    'root' or 'uucp' or 'daemon'. 

    When 'sendmail' invokes /bin/mail for delivery to your mailbox, it uses a 
    parameter list something like: 

    /bin/mail -r sender -d recipient1 recipient2 recipient3 ...

    Thus the envelope information is passed along to /bin/mail in a manner 
    similar to the way 'sendmail' received it.  If mail is sent out through 
    UUCP, the envelope sender is recorded on the unix "From " line which is 
    the first line of the message, and the envelope recipients become the 
    command operands to 'rmail' on the remote system. 

5.  TOKENIZING AND REBUILDING ADDRESSES.  prescan() AND cataddr().

    Before an address it processed by one of the rewrite rulesets, it must be 
    tokenized.  The rewriting rules only examine complete tokens and strings 
    of tokens during their matching.  After the rewriting is complete the 
    tokenized address must be converted back to a character string.  The 
    function prescan() does the tokenizing, while cataddr() converts a 
    tokenized address back to a character string. 

    The output of prescan() is an array of strings, with one array entry for 
    each token.  The tokenizing is done based on the characters in $o, defined 
    in the line beginning 'Do' in 'sendmail.cf'.  There are, additionally, a 
    few characters whose special handling is built into prescan().

    In an address such as the following (from RFC822):

         Wilt . (the Stilt) Chamberlain@NBA.US

    the parenthesized comments "(the Stilt)" are filtered out by prescan().  
    (However when such an address appears on a header, the comments are saved 
    for reinsertion after the address has been rewritten).  prescan() also 
    insists on properly balanced parentheses and properly balanced <angle 
    brackets>.  A string between "double quotes" becomes a single token.  With 
    one exception, an escaped character (such as \@) loses any special 
    properties.  Thus the address 'user@host' would ordinarily become the 
    three tokens "user", "@", "host", while the address 'user\@host' becomes a 
    single token.  The one exception to the backslash escaping is \!, which is 
    simply converted to !.  The special handling is an attempt to be somewhat 
    forgiving to csh users who sometimes become overly zealous in escaping 
    their bangs. 

    In tokenizing, special characters (those characters defined in $o) become 
    single character special tokens.  A string of ordinary characters becomes 
    a single token.  The space character, however, is never part of a token 
    except when in a quoted string or when backslash escaped.  The space 
    character is a token separator.  Thus the string AB become one token "AB", 
    while the string A B becomes two tokens "A", "B".  A space before or after 
    a special character, however, is completely superfluous.  Thus user @ host 
    is tokenized as "user","@","host", exactly as would be user@host 

    The function cataddr() sounds simple.  Just concatenate all the tokens to 
    form a new string.  This is almost what it does.  But you have to make a 
    special case where there are two ordinary tokens.  If 'A  B' is tokenized 
    as "A","B", then just concatenating the strings would produce the 
    incorrect 'AB'.  Instead, when cataddr() discovers two consecutive 
    ordinary tokens, in inserts between them the space substitute character 
    defined on the 'OB' line of 'sendmail.cf'.  In most versions of 
    'sendmail.cf' this character is the period '.' leading to the effect that 
    the original 'A B' was tokenized to "A","B", then is untokenized to 'A.B' 
    If you don't like this, you can define the space substitute as a blank.  
    If you do so, but forward the message to another 'sendmail' it will 
    probably be converted to a period again.  (One problem with defining the 
    space substitute as a blank is that this can easily become invisible in 
    'sendmail.cf', and if it is accidently deleted you might find you are 
    accidently defining the space substitute to be '\n' or '\0'). 


6.  THE C-FLAG PROCESSING.

    Certain mailers have the C flag set in their mailer definition.  If 
    effects processing of addresses as follows:

    After an address has been rewritten by ruleset 3, a check is done to see 
    if the address now contains an "@".  If it already contains an "@", then 
    C-FLAG processing has no effect.  If there is no "@", the 'receiving 
    mailer' is checked.  (The determination of the receiving mailer is 
    described below).  If the receiving mailer has the C-FLAG defined, and if 
    the sender address in $f contains an '@', everything from the '@' onward 
    in $f is appended to the end of the address as outputted by ruleset 3, and 
    the modified address is now reprocessed by ruleset 3.  The actual check 
    for the '@' in $f is done while the sender address is still tokenized, so 
    there is no additional call to prescan() in the C-FLAG processing.


7.  Mailer-specific rulesets.  A typical mailer definition looks something 
    like the following:

    Mlocal, P=/bin/mail, F=MFlusr, S=10, R=12, A=mail -d $u

    Here the S= and R= operands specify address rewrite rulesets to be used 
    for mail sent by this mailer.  The S= operand is for sender addresses, and 
    the R= is for recipient addresses.  We shall refer to these as the mailer-
    specific rulesets.  In the IDA version of sendmail, these can optionally 
    be specified as say S=13/15, etc.  This would use ruleset 13 for envelope 
    sender addresses and ruleset 15 for header sender addresses.  The simple 
    definition S=10 is equivalent to S=10/10.

    If the mailer specific ruleset is omitted, or is defined as 0, this means 
    that there is no mailer-specific ruleset.  Ruleset 0 itself is never used 
    as a mailer-specific ruleset.

8.  A MORE DETAILED LOOK AT SENDMAIL PROCESSING.

8a. Parsing the sender address.

    One of the first steps is to parse the envelope sender address.  This 
    follows the procedure of address parsing as described above.

    prescan()
    rewrite with ruleset 3.
    rewrite with ruleset 0.
    rewrite the user address portion with rulesets 2, the mailer specific 
    ruleset, and ruleset 4.
    process the output of ruleset 4 with cataddr()

    If the address proves unparseable a message is written to the log, and the 
    address 'Postmaster' is parsed in its place.  One effect is that should 
    the mail be undeliverable, and the sender address is unparseable, any 
    bounced mail will in this case be delivered to Postmaster.

    The mailer returned from ruleset 0 is called the receiving mailer, and is 
    examined for the presence of a C flag as discussed above in our 
    description of C-FLAG processing.

    If the receiving mailer is the local mailer, the sender is assumed local.  
    In that case the user address is looked up in the password file to find 
    the full name (for possible use on the 'From:' header) and the home 
    directory (in case the message should be stored in dead.letter).

8b. Defining $f.

    The sender address is processed with

    prescan()
    rulesets 3,1,4

    Search for an '@' in the address, and save a copy of the portion of the 
    tokenized address starting with the '@', in case needed for C-FLAG 
    processing.
    
    apply cataddr() to the output of ruleset 4, and save as the value of $f

    NOTE:  Even if the original address was determined to be unparseable in 
    step 8a, and replaced by Postmaster, it is still the original address and 
    not 'Postmaster' which is used for defining $f.  However there is one 
    anomoly here.  If the message cannot be delivered immediately but must be 
    queued, and if the original sender address was unparseable, the original 
    sender address is not saved in the queue file, so the sender will become 
    'Postmaster' in that case. 

8c. Building the recipient list.

    Each envelope recipient address is now added to an internal list of 
    recipients.  Before an address is added to the recipient list, it goes 
    through the following procedures: 

    prescan()
    rewrite with ruleset 3
    rewrite with ruleset 0
     The mailer and host returned are saved.  The user portion is further
    processed:
    rewrite with ruleset 2
    rewrite with the mailer specific ruleset (for the mailer returned by 
       ruleset 0)
    rewrite with ruleset 4.
    cataddr()

    Next the recipient list is searched to see if this address already 
    appears.  The search is based on a comparison of the mailer returned by 
    ruleset 0, the host returned by ruleset 0 (except the host is ignored if 
    the 'l' flag is set for the mailer), and the user name as output from 
    cataddr().  If the address is a duplicate it is not actually added to the 
    recipient list.

    Next the mailer is checked to see if the local mailer.  If so, the address 
    is looked up in the aliases database.  If there is an aliases entry the 
    current address is flagged QDONTSEND so that mail will not be sent to it, 
    and each entry in the alias expansion recursively goes through the same 
    process for adding to the recipient list.  An additional flag is set for 
    aliases to indicate they are indeed aliases. 

    Next, if the mailer is the local mailer, and if the QDONTSEND flag is not 
    set, there is a test to see if the user address begins with '|' after 
    removal of quotes.  If so, and if the address is from an alias expansion, 
    the mailer is changed to the 'prog' mailer, and the initial '|' is 
    removed. 

    Next, if the mailer is local, the name is looked up in /etc/passwd, the 
    home directory searched for a '.forward' file which, if found, is treated 
    to similar processing as an aliases entry.

    If at any stage something goes wrong, the address is flagged as bad for a 
    later bounce message.

8d. Beginning the delivery phase.

    Once the recipient list is complete, sendmail is ready to attempt 
    delivery.  This involves running down the recipient list.  As an address 
    is selected for a delivery attempt, it is marked QDONTSEND which is 
    approximately the equivalent of deleting it from the recipient list.  This 
    is to ensure it will not be sent twice.

    Once an address is found for a delivery attempt, a check is made to see if 
    the 'm' flag is set in the mailer.  If so, sendmail will attempt to send 
    to as many recipients with the same mailer/host combination as possible in 
    a single operation.

    If delivery requires sending to several hosts, the next few steps will be 
    repeated several times.

8e. Determining the envelope sender for delivery.

    Start with the expansion of $f
    prescan()
    rewrite with rulesets 3, C-FLAG processing, 1, the mailer specific 
    ruleset, and 4.
    cataddr()
    The result is assigned to $g

    Note that, because of the way $f was originally determined, this means the 
    original incoming address has been processed by 3,1,4,3,1,mailer 
    specific,4 

8f. Determining the command. 

    The command (the P= and A= operands) from the mailer definition are now 
    expanded, with $h evaluating to the host.  If there are multiple 
    recipients, and $u is in the last argument, multiple such arguments are 
    created, one for each recipient for this delivery transaction.  If there 
    is no $u in the argument list, the SMTP protocol is used instead.

    For SMTP use only, the recipient address is now processed by
    prescan(), ruleset 3, C-FLAG processing, 2, mailer specific, 4, cataddr(). 
    The IDA versions, however, do not do this additional rewriting step which 
    seems superfluous, and which could conceivably cause mailing loops because 
    of the C-FLAG processing. 

8g. Header rewriting.

    The headers are now sent to the mailer program as part of the message.  
    Before sending them they are subject to any rewriting.  If the required 
    'From:' header does not exist it is created from the definition in 
    'sendmail.cf'.  If the required 'To:' is missing an 'Apparently-To:' 
    header is created. 

    A special word on the 'From:' header.  If the incoming message has a 
    'From:' header whose contents are identical in all respects (except 
    leading and trailing white space) to the envelope header, that 'From:' 
    header is deleted.  The assumption is that it will be recreated with a 
    chance of adding the full name.

8h. Header sender rewriting.

    The address is extracted from the 'From:' header and other similar headers 
    such as 'Reply-To:' and 'Resent-From:', carefully saving the comments for 
    later use.  The address is processed with
    prescan(), 3, C-FLAG processing, 1, Mailer specific, 4, cataddr().

    The IDA versions use ruleset 5 in place of ruleset 1.  This is part of the 
    IDA strategy of allowing a distinction between the formatting of headers 
    and the formatting of the envelope.

    If there was no "From:", or if it was deleted and must be recreated, the 
    usual definition of "From:" in most version of 'sendmail.cf' begins with 
    $g, or with $q which is defined in terms of $g.  In that case, and 
    remembering how $f and then $g are determined, the address on the 'From:' 
    goes through the following steps: 

    incoming envelope sender
    prescan(), rulesets 3,1,4, cataddr()
    prescan(), rulesets 3, C-FLAG, 1, Mailer specific, 4, cataddr()
    prescan(), rulesets 3, C-FLAG, 1, Mailer specific, 4, cataddr()

    You will note the large amount of redundancy.  If designing rulesets you 
    must keep this in mind.  In particular you should be wary of approaches 
    which give different results depending on how many times the address is 
    passed through rulesets 3,1,Mailer-specific,4.  In the current IDA/NIU 
    rulesets, the $q variable is defined in terms of $f instead of $g, in 
    order to eliminate the most troublesome one of these extra rewrites, in 
    which a header address is rewritten with a mailer specific ruleset 
    intended for envelope addresses only. 

8i. Recipient header rewriting.

    Each recipient address on a "To:" or "Cc:" or "Apparently-To:" or 
    "Resent-To:" or "Resent-Cc:" header goes through the following steps: 

    prescan(), rulesets 3, C-FLAG, 2, Mailer specific, 4, cataddr().

    In the above, the IDA versions use ruleset 6 in place of ruleset 2.  Again 
    this is to allow header addresses to be formatted differently from 
    envelope addresses.

9.  GENERAL COMMENTS ON DESIGNING RULESETS.

    If you plan on designing your own 'sendmail.cf', or modifying an existing 
    one to add more functionality, here are some things to keep in mind:

9a. Remember the C-FLAG.

    Because of the C-FLAG processing, it is desirable that every address which 
    contains a host name should contain an '@' by the end of ruleset 3, and 
    addresses without hostname should not have one added in ruleset 3. 

    This also means that if you want to rewrite 'user' as 
    'user@your.full.domain' you should not do it in ruleset 3, but should do 
    it somewhat later. 

    This means that an address like 'uunet!seismo!harry' needs to be converted 
    to something like 'seismo!harry@uunet.UUCP' in ruleset 3, so as to ensure 
    that there is an '@' in the address and C-FLAG processing doesn't 
    incorrectly add another domain level. 

9b. Make major rewriting steps reversible.

    Ideally any address processed by ruleset 3 followed by ruleset 4 should 
    finish up in its original form.  Anything non-reversible done in ruleset 3 
    can never be fully compensated for later.  This is difficult to do in 
    practice, however.  In many versions of sendmail.cf, both 'uunet!john' and 
    'john@uunet.uucp' will be rewritten as 'john<@uunet.UUCP>' in ruleset 3, 
    so the original form cannot always be recovered by use of ruleset 4.  What 
    is probably more critical is that using rulesets 3,4,3,4 should yield the 
    same result as just using 3,4.  If your rulesets don't manage at least 
    this degree of consistency you are likely to run into major problems.

    The IDA/NIU rulesets are pretty close to the ideal that ruleset 4 
    completely reverse ruleset 3.  But to achieve this they use in internal 
    form which only vaguely looks like the original address.  Thus in these 
    ruleset 3 would rewrite 'uunet!john' as '<@uunet.UUCP>!john' while they 
    would rewrite 'john@uunet.UUCP' as '<@uunet.UUCP>,john'.  And, of course, 
    ruleset 4 is rather more complex also.  (This approach in IDA/NIU is not 
    simply a matter of purism.  It has to do with the need to be able to 
    unambiguously merge two addresses in the pathalias file lookup.) 

9c. Allow for the fact that sender addresses may be rewritten multiple times.

    The habit of sendmail of reprocessing sender addresses can cause some 
    problems and result in incorrect addresses if you do not properly allow 
    for it.

9d. Be particularly cautious with how you handle envelope recipients. 

    Basically if you mess these up, the mail probably won't go anywhere, or 
    won't go where you want it.

    As an example, suppose I have a uucp neighbor 'uuhost'.  Suppose my 
    neighbor wants all mail to leave in the format 
    'uuhost!user@my.full.domain'.  Now it might be that I become a little too 
    vigorous in my changes, and I even rewrite the addresses that way in mail 
    sent to 'uuhost'.  If this happens to a header address the problem is not 
    very serious.  At the worst, when someone on 'uuhost' sends a reply, the 
    reply will be sent first to my system, then back to 'uuhost'.  But if I 
    make the same transformation to the envelope, I will cause a mailing loop.  
    In other words, if mail I see destined for 'uuhost!user' is sent to 
    'uuhost' with the user address of 'uuhost!user@my.full.domain', the mailer 
    on 'uuhost' is likely to just send it back.  Then the same thing happens 
    over and over again.


-- 
=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=
  Neil W. Rickert, Computer Science               <rickert@cs.niu.edu>
  Northern Illinois Univ.
  DeKalb, IL 60115                                   +1-815-753-6940
-- 
Dan Warburton warb@faatcrl.uucp     warb@pilot.njin.net 
TGF (Target Generation Facility, an Air Traffic Control Simulator) 
FAA Technical Center, Atlantic City International Airport, NJ
08405  Mail Stop ACN-400c  Phone: 609-484-4480