🚀 go-pugleaf

                              
                     HPFF Meeting Notes
                              
                              
                      July 23-24, 1992
             Washington, DC - Hyatt Crystal City
                Notes taken by Chuck Koelbel,
                 with help from Bob Knighten

Executive Summary

  This was the fourth meeting of the High Performance
Fortran Forum working group. It was held immediately after
the International Conference on Supercomputing in
Washington, DC, where a Birds-of-a-Feather session
introducing the language was also held. The effort is mostly
on schedule for presenting a document at the Supercomputing
'92 conference.
  Some of the more important decisions made at this meeting
included:
     Adoption of the proposal for REALIGN and
       REDISTRIBUTE made at the last meeting, except
       for the sections regarding subroutine linkage,
       COMMON blocks, and ALLOCATABLE arrays. This
       proposal contained the basic explicit remapping
       capabilities within a single subprogram.
     Adoption of the proposal for sequence association
       in the presence of distributed arrays made at
       the last meeting. The proposal essentially
       disallows sequence association for distributed
       actual arguments.
     Guy Steele and Rich Shapiro's proposal for
       managing distributed dummy arguments in
       subprograms was accepted as a first reading.
       The major point of the proposal was to disallow
       the redistribution of an actual argument
       through a subroutine call, although the
       argument could be temporarily remapped within
       the subroutine.
     Chuck Koelbel and David Loveman's proposal for a
       FORALL statement and construct was accepted
       with minor revisions on first reading. These
       statements are not intended as a general
       "parallel loop" or as a general parallel
       construct; instead, they allow assignments to
       multiple array elements, in somewhat the same
       way as array assignments and WHERE constructs
       in Fortran 90.
     Guy Steele's proposal for an INDEPENDENT assertion
       applied to DO loops was accepted as a first
       reading. The directive is an assertion
       providing more information to the compiler; if
       properly used, it does not change the semantics
       of the program where it appears. Work is
       proceeding to apply the assertion to other
       language constructs.
     A large number of new standard library routines
       and intrinsics was approved as a first reading.
       These functions perform tasks useful for data-
       parallel computing, including prefix
       accumulations, new reduction functions, and
       system inquiry functions. Discussion continues
       on whether these functions are sufficiently
       common to merit their inclusion or simply as
       standard library functions.
  Dates and agendas for future HPFF meetings are as
follows:
     September 9-11 Storage Association, Subroutine
                    Linkage for Distributions,
                    Pointers and ALLOCATABLE Array
                    Distributions, FORALL, DO
                    INDEPENDENT (all second
                    readings);
                    Local Subroutines, Intrinsic
                    Functions, Fortran 90 Subset,
                    FORALL INDEPENDENT, Parallel I/O
                    (all first readings)
     October 21-23  Local Subroutines, Intrinsic
                    Functions, Fortran 90 Subset,
                    FORALL INDEPENDENT, Parallel I/O
                    (all second readings); Tentative
                    approval of draft for
                    Supercomputing '92 Tutorial
     December 9-11  Approval of Final Draft


HPFF Documents

  The following documents related to HPFF are available by
anonymous FTP from titan.cs.rice.edu in the public/HPFF
directory. Additions since the June meeting are marked with
a plus sign (+).
     announce.*  HPFF Announcement and Call For
       Participation
     archives    Directory containing the HPFF mailing
       group archives. (All files are updated
       nightly.)
       + hpff           All messages sent to
          hpff@rice.edu
       + hpff-core      All messages sent to hpff-
          core@rice.edu
       + hpff-distributeAll messages sent to distribution
          subgroup
       + hpff-f90       All messages sent to
          Fortran 90 subgroup
       + hpff-forall      All messages sent to
          FORALL subgroup
       + hpff-format    All messages sent to
          editorial subgroup
       + hpff-intrinsicsAll messages sent to
          intrinsics subgroup
       + hpff-io        All messages sent to I/O
          subgroup
     database    The HPFF mailing list
     handouts    Directory containing handouts from the
       HPFF meetings
       apr    Subdirectory containing April meeting
          handouts
          ALL       Concatenation of the other files in this
            subdirectory
          Distribute  Proposal for data distribution
            model by the distribution subgroup
          FORALL    Proposals and discussion from
            the FORALL and local subroutines
            subgroup
          Fortran90 Storage association proposal by
            the Fortran 90 subgroup
          Pointer     Discussion of Fortran 90
            pointers by the F90 subgroup
          Subroutine  Discussion by the Subroutine
            interfaces subgroup
       june   Subdirectory containing all June
          meeting handouts
          ALL            Concatenation of the other
            files in this subdirectory
          Association      Storage and Sequence-
            Association Proposal
          COMMON-Kennedy Distributing COMMON Blocks
            Proposal (Ken Kennedy)
          COMMON-Meltzer Distributing COMMON Blocks
            Proposal (Andy Meltzer)
          Distribute.tex   Data Distribution Proposal
            (Guy Steele)
          FORALL-Loveman.tex  FORALL construct
            proposal (David Loveman)
          FORALL-Wu      FORALL construct proposal
            (Min-You Wu)
          Intrinsics          Intrinsics Proposal
            (Rob Schreiber)
          Subset         Fortran 90 Subset Proposal
       + july   Subdirectory containing all July
          meeting handouts
          + ALL.shar       Shar file containing all
            other files
          + Anti-FORALL.tex   Arguments against the
            FORALL construct (Alan Karp)
          + Pro-FORALL     Arguments in favor of the
            FORALL construct (Rich Shapiro)
          + Critique.tex   Critique of HPF
            distributions (Barbara Chapman and Hans
            Zima)
          + Distribute.tex    Data distribution
            proposal (Guy Steele)
          + FORALL.tex        FORALL proposal (David
            Loveman and Chuck Koelbel)
          + Original-Storage  Sequence and Storage
            Association proposal, beginning of
            meeting
          + Final-Storage     Sequence and Storage
            Association proposal, as finally
            presented (F90 group)
          + IO.tex         Parallel I/O proposal
            (Marc Snir)
          + Independent.tex     INDEPENDENT proposal
            (Min-You Wu)
          + Intrinsics        Intrinsics proposal
            (Rob Schreiber)
          + Local-Subr.tex    Local Subroutine
            proposal (Marc Snir)
          + MIMD         MIMD support proposal
            (Clemens-August Thole)
     minutes     Directory containing minutes of past
       HPFF meetings
       jan-minutes   Summary of the first HPFF
          meeting
       jan-proposals Point-by-point comparison of
          language proposals at first HPFF meeting
       mar-minutes   Summary of HPFF meeting, March
          9-10, Dallas
       apr-minutes   Summary of HPFF meeting, April
          23-24, Dallas
       europe-minutesSummary of European Workshop
          on High Level Programming Models for
          Parallel Architectures
       june-minutesSummary of HPFF meeting, June 9-
          10, Dallas
       + july-minutesSummary of HPFF meeting, July
          23-24, Washington
     papers    Directory containing papers related to
       HPFF. (Note that these are not intended as
       formal drafts, but rather as supporting
       documents.)
       convex.*    Slides for Convex presentation
       fd.*     Fortran D language specification
       fd-over.*   Fortran D project overview
       fd-sc91.*Fortran D compilation paper
          (presented at Supercomputing '91)
       hpf.ps      DEC HPF language specification
       + hpff-europe.ps "High Performance Fortran:
          A Perspective" by Brian Wylie, Michael
          Norman, and Lyndon Clarke (University of
          Edinburgh)
       + merlin.ps "Techniques for the Automatic
          Parallelisation of 'Distributed Fortran
          90'" by John Merlin (University of
          Southhampton)
       mpp.ps   Cray MPP Fortran language
          specification
       tmc      Position paper by Thinking Machines
          on HPFF
       vf.*     Vienna Fortran language
          specification
       vf-over.*   Vienna Fortran language overview
     welcome     "Welcome to HPFF" message, including
       instructions for using anonymous FTP and
       joining mail lists.
See the README file in the main directory for information on
file extensions and compressed files.
  Public comment on any of the proposals is welcome. Please
address your remarks to the appropriate email list (see the
"welcome" file for a list of these groups).

Detailed Meeting Notes

Attendees

Robbie Babb (Oregon Graduate Institute), Barbara Chapman
(University of Vienna), Marina Chen (Yale University), Alok
Choudhary (Syracuse University), James Cownie (Meiko), Rich
Fuhler (Lahey), Peter Highnam (Schlumberger), Maureen
Hoffert (Hewlett Packard), Hidetoshi Iwashita (Fujitsu
Labs), Ken Kennedy (Rice University), Bob Knighten (Intel),
Chuck Koelbel (Rice), Dave Loveman (DEC), Piyush Mehrotra
(ICASE), Andy Meltzer (Cray Research), Ken Miura (Fujitsu
America), Prakash Narayan (SunPro, Sun Microsystems), Tin-
Fook Ngai (HP), Edwin Paalvast (Technical University,
Delft), Rex Page (Amoco), Jean-Laurent Philippe (Archipel),
David Presberg (Cornell), J. Ramanujam (Louisiana State
University), P. Sadayappan (Ohio State University), Rony
Sawdayi (Applied Parallel Research), Randy Scarborough
(IBM), Rich Shapiro (Thinking Machines Corporation),
Margaret Simmons (Los Alamos National Laboratory), Marc Snir
(IBM), Guy Steele (TMC), Kate Stewart (IBM Canada), Richard
Swift (MasPar), Clemens-August Thole (GMD, St. Augustin),
Joel Williamson (Convex), Min-You Wu (SUNY Buffalo), Mary
Zosel (Lawrence Livermore National Laboratory)

Memorable self-introductions

  Margaret Simmons, substituting for the honeymooning Ralph
Brickner, signed the attendance sheet "virtual Ralph
Brickner".
  Dave Loveman identified himself as being "from the newly-
CEO-ed DEC".

HPF Europe Report

  Clemens-August Thole gave a brief report on developments
in Europe related to HPFF. The HPF Europe group held a
meeting June 23 in Brussels which included attendees from
many major companies and universities on the continent. The
topics of discussion included
     Discussion of data distribution at subroutine
       boundaries: Most of the points made there are
       covered in the current proposal, thanks to
       strong European contributions to the mailing
       list discussion.
     Parallel I/O: Peter Brezany from the University of
       Vienna presented the results of experiments in
       Vienna Fortran showing that special treatment
       of distributed arrays in file I/O could have
       significant advantages. James Cownie presented
       a programmer interface approach to the same
       problem.
     MIMD support and message passing: Clemens-August
       Thole and Delces Watson presented their ideas
       on adding this support to HPF. The goal is to
       allow concurrent execution of everything in a
       program without resort to local subroutines.
       Such support would require messages as a part
       of the language, not restricted to local
       subroutines. The group plans to propose these
       features for the second round of HPFF after the
       core language discussed in the US has been
       adopted.
     Indirect mapping arrays: Presented by Hans Zima,
       these were seen as an important feature for
       supporting irregular computations. They will
       also be proposed for later versions of HPF, but
       not for the current draft.
The next European meeting will be held September 21.
Interested parties should contact Clemens-August Thole for
details.
  Another European meeting  may be possible after
Supercomputing '92 to disseminate information about HPF.
This remark started a short discussion of ways that High
Performance Fortran could be publicized. Ken Kennedy wanted
to disseminate as much information as possible at SC '92,
thus getting as much input as possible for the final draft
approval. Some felt that SC 92 would be too late for this (a
topic that would recur later). It was generally agreed that
a full draft would be needed in October, well in advance of
Supercomputing '92. Ken asked if HPFF could arrange for
European feedback. Clemens Thole suggested either a special
event or a presentation at a planned Esprit meeting at
Brussels; although he did not know of any conferences in the
near future, he did promise to check on that as a third
possibility. He noted that US participation at such an event
would be important for image.

Data Distribution: REALIGN and REDISTRIBUTE

  Guy Steele next led a short discussion of the REALIGN and
REDISTRIBUTE statements. The proposal had not changed since
the last meeting, and there had been little objection to it
in the electronic discussions. The major objection raised
was that the declaration (without the !HPF$ directive)
          REALIGNABLE A
could also be parsed as
          REAL IGNABLEA
This is not a problem in the current language, because
REALIGNABLE is a structured comment; it does, however,
conflict with the goal of eventually adding the HPF
extensions to Fortran simply by removing the comment
headings. Three fixes were proposed:
     1. Always take the longest possible keyword when
       there is an ambiguity.
     2. Change the spelling of the HPF attributes to
       RE_ALIGN and RE_ALIGNABLE.
     3. Add significant blanks to Fortran, at least in
       this case.
The group had a slight preference to the first alternative,
and the proposal was amended appropriately. In a vote taken
just after dinner, the forum formally accepted the REALIGN
and REDISTRIBUTE features, with possible changes in syntax
to remove the parsing ambiguity allowed if a new proposal
emerged. The final vote was 15 in favor, 3 against, and 3
abstentions.
  Discussion of data distribution in subroutines and
ALLOCATABLE arrays was deferred until later, to give Guy a
chance to make up overhead slides.

Official HPF Subset

  (This section is mostly taken from Bob Knighten's notes,
as mine were wiped out by an ill-timed battery failure.)
  Mary Zosel next took a few straw polls regarding proposed
features for the High Performance Fortran subset.
  The first question was 'Should procedure-less modules be
a part of the HPF official subset?" The full group vote was
15 yes, 9 no, and 13 abstain. Vendors, in a separate count,
voted 4 yes, 4 no, and 7 abstain. The intent was to limit
the form and content of INTERFACE blocks to provide explicit
interfaces and satisfy requirements for explicitly
interfaces (useful for handling some distribution scoping
issues) but not user-defined operators, overloaded operators
or generic interfaces. Some vendors claimed that the most
difficult part of modules was the naming features, so
restricting them to remove procedures was not an advantage.
  The next question was "Should `entity-oriented' (in
previous meeting notes, `object-oriented') declarators using
attributes in type declaration statements be part of the HPF
official subset?" The vote was 19 yes, 8 no, and 10 abstain.
Mary noted that this form was already defined in the
distribution directives.
  Taking the negative, the next question was "Should HPF
not include free-form source in the official subset?" The
vote was 23 yes, 0 no, and 10 Abstain. The group agreed that
long variable names and a few other features of the free-
form source were needed for HPF, but these had been approved
separately already.
  Mary noted that the first reading of the subset
definition will be at the next meeting, and the only
features under consideration for inclusion were those she
had just asked for votes on. She asked anybody with any
other items from Fortran 90 that should be added to the
subset to contact the F90 subgroup soon.
  After the first set of polls was taken, a short
discussion of the form of the subset started. Rex Page
raised the issue of whether the HPF subset should include
all new HPF features. The arguments in favor claimed that
any subsetting diminished the impact of HPF. Arguments
opposed to this stressed the importance of quick
implementations which might be delayed by overly-ambitious
HPF features. Mixed into this discussion was a suggestion of
an HPF "superset" that might include proposed HPF features
that were not adopted; another approach suggested was to
have a "Journal of Fortran Development" that could document
these suggestions. A very informal poll found the majority
in favor of including all of the new features in the subset
at this point in the meeting. (See the Intrinsics section
for later views on this matter.) Consideration of an HPF
superset was deferred until December, when it would be
clearer what features were accepted and what (if anything)
was not.

Data Distribution: Subroutine Interfaces

  After a one-hour break to produce slides and otherwise
handle logistics, Guy Steele presented the subroutine
interface proposal. This was the first reading for the
proposal, and thus it will be formally voted on at the next
meeting. The main goal of the document was to define the
behavior of distributed arrays when used as actual arguments
to subprograms, and to define the meaning of dummy arguments
which have explicit data distributions.
  The presentation started with a definition and two design
decisions:
     *  Two identically declared processor arrangements
       are "similar". (Here, "identically declared"
       refers only to the arrangements' shape, not the
       names used. Some question remains about whether
       arrangements with differing lower bounds are
       similar, which will be cleared up in the final
       draft.)
     *  Two identically declared and implicitly
       distributed TEMPLATEs will be similar, and
       distributed in the same way onto similar
       processor arrangements.
     *  Two TEMPLATEs will also be similar if explicitly
       distributed onto similar processor arrangements
       in the same way.
The purpose of these definitions was to support later
proposal features. The objection was raised that an
application may want two same-sized arrays to be dissimilar,
to which the response was that the proposal only applied to
TEMPLATE constructs. Chuck Koelbel  noted that this did not
solve the problem, since implicitly declared TEMPLATEs from
equal-sized (and not explicitly distributed) arrays would be
similar. Rich Shapiro got the proposal out of that corner by
suggesting that the distribution restrictions only applies
to explicitly declared TEMPLATES. In any case, Guy pointed
out that the intent was to make sure identically declared
TEMPLATEs are aligned.
  Guy advanced to his next slide.
     *  TEMPLATES  declared in different scoping units
       are distinct (but may be similar).
     *  Note: TEMPLATES cannot appear in COMMON blocks,
       so the only way to share a template among
       several program units is to use Fortran 90
       modules.
     *  Returning from a subprogram causes locally
       declared TEMPLATES to become undefined. (That
       is, there is no SAVE for TEMPLATES.)
To clarify the difference between TEMPLATES that are similar
and those that are equivalent, Guy used the following
analogy: "Any two pennies are similar in the sense that you
can spend the in the same way. They are equivalent you put
one on railroad track and the other one gets squished." The
intent of these rules is to define a lifetime of TEMPLATES.
Guy pointed out that this was not the only possibility, but
was a convenient one. It may, however, have some problems
for pointers.
  Guy then moved on to his next slide, demonstrating
several options for subroutine linkage.
     Declarations:
          SUBROUTINE FOO(A)
          TEMPLATE T
          REAL A(:)
     Option 1:
          !HPF$ ALIGN WITH T :: A
       The actual argument (or a copy) is forcibly
       realigned with T on entry to the routine.
     Option 2:
          !HPF$ ALIGN WITH * :: A
       The dummy argument is aligned with a TEMPLATE
       which is similar to (a copy of) the actual's.
     Option 3:
          !HPF$ ALIGN WITH *T :: A
       The dummy is aligned with T; program is
       nonconforming if this is not true of the
       actual.
     Option 4:
       If a dummy has no explicit align attribute the
       compiler chooses a default of the form
          ALIGN with T
       for some T, or
          ALIGN with *
       but not
          ALIGN with *T
  The full set of rules is laid out in the draft proposal,
available by FTP. The limitation on option 4 derives from
the fact that the *T form disallows certain actual arguments
to be passed to the corresponding parameter. Much discussion
followed. Rich Fuller objected to certain forms of T,
particularly optional arguments. Guy amended the proposal to
disallow both optional and keyword arguments. The key
requirement intended was that the resulting mapping (i.e.
both alignment and distribution) be expressible in HPF.
Clemens-August Thole and Marc Snir argued this was not true
for array arguments with inherited mappings (option 2);
Chuck Koelbel and Rich Shapiro answered that specific case,
and Guy Steele requested that any counterexamples be
presented off-line.
  The theme of the next slide was that "Nasty aliasing is
forbidden".
          MODULE FOO
          REAL A(10,10)
          REALIGNABLE, REDISTRIBUTABLE A
          END
          PROGRAM MAIN
          <INTERFACE BLOCK FOR SUB>
          CALL SUB(A(1:5,3:9))
          END
          SUBROUTINE SUB(B)
          USE FOO
          REAL B(:,:)
          REDISTRIBUTE A ! ILLEGAL
          REALIGN B   ! ILLEGAL
          END
In general, remapping is not allowed in HPF whenever
assignment is not allowed (i.e. when it reveals
aliases).James Cownie claimed that the example breaks this
rule because B is redistributed on entry to SUB. Guy Steele
responded that explicit and implicit remappings are
different, and the above statement only applies to explicit
operations. The intent is to steer as far as possible from
aliasing problems. A final restriction along these lines was
that COMMON arrays may not be REALIGNABLE, since that would
allow remapping by aliasing again.
  After a short break, Rex Page raised the question "What
are we trying to do in HPF? Support linear memory
programming styles or evolve Fortran forward?" His point was
that many of the points raised in the distribution work, and
which would be raised in the storage association proposal,
were attempts to retain compatibility with a memory model
that does not fit modern machines. Ken Kennedy responded
that this point -- compatibility versus flexibility -- keeps
recurring, and the forum has generally sided with as much
compatibility as possible. Randy Scarborough noted that
Fortran 90 MODULES cause the same or similar problems, so we
have to address them anyway. Regardless, a straw poll was
taken for this case over whether the subcommittee should
keep trying to solve the non-modules problem. By a 16-3
margin, the group voted to continue with the work.

Data Distribution: ALLOCATABLE Arrays

  After a short break, Guy continued with his presentation.
The next part addressed distributing dynamically allocated
arrays; as before, this was a first reading for this
proposal.
     *  A POINTER or ALLOCATABLE variable may not have
       the ALIGN attribute.
     *  Instead, alignment is specified at ALLOCATE time
       (This also goes for the DISTRIBUTION).
     *  Three syntax variants are proposed:
       Syntax 1:
          allocate( p(100) )
          !HPF$ distribute a(block)          ! attached
          declarations
       Syntax 2:
          allocate( p(100) )
          !HPF$ redistribute a(block)   ! even if a is not
          distributable
       Syntax 3:
          allocate( p(100) )
          !HPF$ allocate( a(100), distribute(BLOCK) )
  The following discussion started by noting that all three
proposals have problems, in that they each break at least
one rule followed elsewhere in the language. David Presberg
noted that the proposal broadened the distributions possible
in any case (because the programmer can use general
expressions, not just specification expressions, as
arguments to BLOCK and CYCLIC).
  Piyush Mehrotra proposed another option: Make the
distribution an attribute of the variable, declared when the
variable is declared. This disallows allocating arrays with
different distributions to the same variable, but that can
be fixed by making those arrays REDISTRIBUTABLE or
REALIGNABLE.
  Rich Fuhler started a tangent discussion by asking "In
general, are programmers forced to keep an ALLOCATE
statement on one line?" in the current syntax proposals. Guy
Steele replied that the directives applied to statements,
not proceeding lines, and "Yes, you can write visually
confusing things this way." Rich Fuhler then proposed
allocating a POINTER WITH a TEMPLATE, or aligning with
another POINTER. Guy pointed out (sorry, no pun intended)
that POINTERS as such don't have TEMPLATES. Rich Shapiro
noted that of all the United Technologies codes he checked,
only 2 or 3 need ALIGNS, but lots need DISTRIBUTE
directives. Guy asked if Rich was saying we don't need
TEMPLATES? Rich emphatically answered "No"; TEMPLATES are
really needed in certain fine grain routines taking 99% of
the time in some programs. He argued instead for concise
syntax for the most common case, but retaining capabilities
for other uses.
  Returning to the ALLOCATE syntax, Peter Highnam remarked
that syntax 1 and 2 are hacks, but 3 is the future because
it brings distribution into the right place. Joel Williamson
worried that the maintenance problem is a headache if there
are separate comments. James Cownie commented that Piyush
Mehrotra wants one declaration for the whole routine, and
thought Rich Shapiro was missing this. Joel Williamson
remarked that he couldn't see Piyush's proposal, but thought
it had the least astonishment value. Clemens-August Thole
modified the unseen proposal to make HPF directives
additions at the end of a statement, and several other
variations were suggested. Richard Swift wondered what
happened if ALLOCATE fails, and claimed this was a mark
against option 2 and in favor of syntax 3 or Piyush's
proposal. Guy Steele thought it was strange to distribute
something that doesn't' exist yet, but others noted that the
system needs the distribution to do ALLOCATE, else it
doesn't know the memory requirements. David Presberg
concluded that we need to bite the bullet and invent some
new keywords; this met with the usual resistance to language
additions.
  Ken Kennedy called for a point of order "This is the
first reading of this proposal, so it is not essential to
get the spelling details right today." The key, he claimed,
was whether the group liked the idea of attaching align to
allocate, or favored another approach. Maureen Hoffert
pointed out that the arguments so far pointed toward adding
to the ALLOCATABLE attribute. This effectively supported
Piyush Mehrotra's proposal. Clemens-August Thole proposed
leaving the issue for next working group meeting, and was
greeted by general applause. Ken, however, wanted to at
least take a straw poll to guide the subgroup.
  After some further discussion, it was agreed that the
basic issue was whether to allow "static" distribution of
ALLOCATABLE objects at all, or to always attach the mapping
to the ALLOCATE statement. The first question was whether
multiple mappings of ALLOCATABLE should be treated as a
special case of REALIGN; it passed 17 to 6 with 10 abstains.
The next question was whether ALLOCATABLE arrays could be
distributed once statically (19 in favor) versus being
defined at the point of allocation (7 in favor, with 7 more
abstains).
  After that series of votes, the group broke for a well-
deserved "banquet" of soup and sandwiches.

FORALL and INDEPENDENT

  After dinner, Chuck Koelbel presented the FORALL subgroup
proposal. This was a first reading; the majority of it will
be voted on at the next meeting. The proposal had three
parts: the FORALL statement, the INDEPENDENT assertion, and
LOCAL subroutines; Chuck presented the first two, and Marc
Snir was to present the last one.
  The first slide introduced the single-statement FORALL.
Chuck prefaced the discussion by clarifying that FORALL is
not to be considered as a general-purpose "parallel loop,"
but rather as a construct for making simultaneous
assignments to array elements. In this respect, FORALL is
very similar to Fortran 90 array assignments, rather than
(for example) the PCF DOALL. This position was reached in
email discussions after it became clear that a deterministic
parallel construct was desired, and there was great
disagreement on any functionality more elaborate than this.
FORALL is a new statement to be added to the language; it
cannot be treated as a directive, because it changes the
semantics of the statements within it.
     *  Syntax
       FORALL (forall-triplet-spec-list [, scalar-
          mask] ) forall-assignment
       forall-triplet-spec as in CM Fortran (i.e. a
          subscript triplet assigned to an index
          variable)
       forall-assignment must assign to an array
          element or section
     *  Constraints
       Triplet bounds and strides cannot depend on
          other indices
       Mask can depend on the indices
       Array elements cannot be assigned more than
          once
       Side effects in forall-assignment cannot
          affect other instantiations (i.e.
          computations with other index values)
       Side effects in left-hand side and right-hand
          side cannot interfere in the same
          instantiation
There was relatively little discussion, most of it asking
about details of the reasoning about constraints. The side
effect constraints are identical in spirit to the
constraints on Fortran 90 array assignments; compilers need
not check them (as they are undecidable properties, this
seems for the best). The independence of triplet bounds was
originally in the Fortran 8X proposal; it allows the bounds
to be computed in any order, and also makes the task of
scalarization somewhat easier. No statements are allowed in
the FORALL body except assignments. Andy Meltzer proposed
allowing CALL statements as well; this was debated later.
The slide on the interpretation of the single-statement
FORALL produced more comment
     1. Evaluate all bounds and strides in the
       triplets, in any order. (This produces the
       valid set of index values.)
     2. Evaluate the mask for each combination of
       values in the valid set. (This produces the
       active set of index values.) These evaluations
       may be done in any order
     3. Evaluate the right-hand side for each element
       of the active set. These evaluations may be
       done in any order.
     4. Assign the computed values to their
       corresponding right-hand sides. These
       assignments can be performed in any order.
Rich Fuhler questioned whether the definition of active set
was consistent with Fortran 90 array expressions,
particularly in the case of WHERE masks applied to intrinsic
function evaluations. Fortran 90 experts in the audience
thought that it was, but asked him to prepare a
counterexample off-line if he had doubts. There was some
discussion whether the evaluation of functions in right-hand
sides following these rules would be generally identical to
Fortran 90 array assignments. A consensus eventually
developed that there were two appropriate analogies: user
functions returning arrays, which are executed once, and
intrinsic elemental functions, which are executed once for
each element. The semantics given were consistent with
elemental functions, satisfying the group. In general, it
was agreed by the group that wherever possible the FORALL
interpretation should match the array assignment
interpretation.
  The next two slides proposed the block FORALL (also known
as the multi-statement FORALL). This was again proposed as a
complex set of assignments rather than a general-purpose
parallel loop.
     Syntax
       FORALL (forall-triplet-spec-list [, scalar-
       mask] )
         forall-body-list
       END FORALL
       forall-body must be an assignment, array
          assignment, WHERE, or nested FORALL
     Constraints
       Individual assignment statements (always the
          bottom-level statement) have the same
          restrictions as in single-statement
          FORALLs
       Masks, bounds, and stride expressions in
          nested statements cannot have inter-
          instantiation side effects
       Inner FORALLs cannot redefine outer FORALL
          index names
Again, the syntax produced little argument. In general,
answers were the same as for the single-statement FORALL.
The inter-instantiation side effects on masks, bounds, and
strides were meant to refer to interference with the same
(lexical) statement, not with other statements. The
interpretation slide drew more discussion.
     1. Compute outer FORALL bounds and strides. (This
       produces the outer valid set.)
     2. Compute the outer FORALL mask. (This produces
       the outer active set.)
     3. Execute each body statement in turn for all
       elements of the outer active set.
       *  Assignments and array assignments are
          interpreted as for single-statement
          FORALLs
       *  WHERE statements compute masks for all outer
          active set elements, then execute masked
          array assignments for each assignment
          nested within them.
       *  FORALL generates an inner valid set (i.e.
          bounds and strides) from the outer active
          set, then generates an inner active set
          (i.e. mask) from the inner valid set.
          Given this set, apply these rules
          recursively to evaluate the statement(s)
          nested in the inner FORALL; that is,
          evaluate one statement at a time using the
          (inner) active set.
The intent was to make the interpretation of a block FORALL
equivalent to a series of single-statement FORALLs. The
major exceptions to this intent were somewhat more general
indexing spaces (because inner FORALL bounds can refer to
outer indices, triangular and ragged arrays are possible).
Evaluation of nested masks is also more fully defined, since
Fortran does not have a short-circuit evaluation of logical
expressions. All references to nested WHERE statements
implicitly include the WHERE-ELSEWHERE construct. WHERE
within a FORALL has the same restrictions as in ordinary
code, in particular its mask must be an array mask, and the
nested assignments must be array assignments rather than
ordinary assignments. (Extending WHERE to allow scalar masks
and assignments was discussed in the mailing list and
rejected as changing the language syntax and semantics.)
Several "are these equivalent" examples of block FORALL were
given, which didn't necessarily clarify the issues due to
possible error conditions in the mask expressions and
similar nastiness. Some ambiguities in the draft,
particularly in regards to whether function evaluations (and
their side effects) were atomic or not, were raised and will
be fixed in the next draft.
  The final FORALL slide concerned function calls in FORALL
statements.
     Function calls are allowed in FORALL subject to
       the following conditions:
       *  Side effects cannot affect values computed
          by the same statement in other
          instantiations
       *  Side effects cannot affect other
          subexpressions in the same statement on
          the same instantiation
     John Merlin proposed user-defined ELEMENTAL
       functions; these were not adopted
John Merlin's ELEMENTAL functions were designed to constrain
what was allowed in functions to prevent side effects and
avoid random access to distributed memory structures. They
were not adopted because there was still intense discussion
over particular features allowed in them, not because there
was a fundamental resistance to the idea. David Loveman
pointed out that these constraints were consistent with both
the earlier FORALL constraints and with the Fortran 90
constraints on array assignments. (This had indeed been one
of the primary considerations in developing them.) ELEMENTAL
functions had been dropped from Fortran 90 for requiring too
much support from an already large language. Andy Meltzer
took this opportunity to again propose CALL statements
within FORALL, arguing that certain subroutines acted in
effect as very complex series of assignments. For example,
values might be computed for three related quantities
simultaneously. Rex Page argued that such subroutines were
better handled by returning a structure than using multiple
OUT arguments. Others felt that CALL was against the spirit
of elemental assignment as FORALL was currently defined. Guy
Steele, however, pointed out that MVBITS (a Fortran 90
intrinsic) operated precisely as an assignment, and was
extremely important for certain applications run by secret
government agencies. He felt that allowing at least that one
case in FORALL would be well worth the trouble.
  Chuck's final slide concerned the INDEPENDENT directive.
This was intended as an assertion to give the compiler more
information, not as a new language statement.
     Syntax
          !HPF$ INDEPENDENT
          !HPF$ INDEPENDENT(i,j,k)
     Interpretation
       A user assertion that no iteration (of a DO)
          or combination of index values (of a
          FORALL) assigns to a location read or
          written by another iteration / combination
          of values
       Does not change the semantics of the
          construct if the assertion is true
       Not standard conforming if the assertion is
          false. The compiler can take any action it
          deems necessary, including executing the
          programmer.
This is a strong assertion, but equivalent to the PCF DOALL
(without private variables or explicit synchronization).
After some discussion, it was decided to drop INDEPENDENT
applied to FORALL from immediate consideration, as there was
concern that its meaning was different from the INDEPENDENT
DO case. Clemens-August Thole spoke in favor of introducing
an interpretation of INDEPENDENT applied to other statement
types, and this suggestion was taken under consideration by
the subgroup. Limitations of the current INDEPENDENT were
pointed out, including that it disallowed reductions and
scalars used as temporaries in the loop. PRIVATE variables
were suggested, much to the dismay of those who remembered
the PCF meetings. The question was raised whether DO
INDEPENDENT was a general parallel loop; it was decided that
in its present form it was not, because no synchronization
or communication was possible.
  At this point, Chuck Koelbel was ready to turn the floor
over to Marc Snir's LOCAL subroutine proposal. In the
interest of continuity, it was decided to finish discussion
of FORALL and INDEPENDENT first. Marc agreed after being
promised a chance to present his proposal before the end of
the meeting; Ken Kennedy said he would be the next topic
that night.
  Marina Chen began the discussion by asking, "What about
Alan Karp's comments?" This was in reference to a recent --
and very vocal -- discussion on the mailing list, in which
Karp claimed that FORALL was unnecessary because it could
always be simulated by DO loops and temporary arrays. (His
comments, and some of the responses to them, were
distributed as handouts, and can be found in the FTP
archives.) Rich Shapiro, although he was one of the
staunchest defenders of FORALL, tried to present Karp's
views fairly based on an extended discussion they had had.
Karp was correct that DO INDEPENDENT and temporary arrays
suffice to provide the functionality. It could similarly be
argued that DO is unnecessary because GOTO can be used to
construct loops, although the analogy is not really fair to
the anti-FORALL camp. Karp's next criticism was that FORALL
is dangerous because statements in it have different
semantics. For example, the statement
          A(I) = A(I) + A(I-1)
has a markedly different affect inside DO and FORALL loops;
one is a prefix sum operation, while the other is a shift
and add. Rich admitted this was a problem, particularly in
block FORALLs, but was willing to accept it for the other
benefits of FORALL. Finally, Karp gave an example of a
computation that was much clearer when written without
FORALL than with it; Rich countered with a different example
that was much longer without FORALL. As Rich put it,
"Readability is in the eye of the beholder". Having both
examples suggested to several people that both constructs
(FORALL and DO INDEPENDENT) were needed.
  Clemens-August Thole argued that FORALL is not intuitive,
since the HPFF group itself took so long to agree on its
semantics. Rich Shapiro noted that Fortran 90 has other
nonintuitive features, such as SUM under a WHERE (see
previous discussion of FORALL); he reiterated that
programmers will make mistakes, but the benefits when FORALL
is used correctly outweigh the bugs. Dave Loveman opined
that the real danger in this discussion was that we knew too
much about FORALL; some of the group expected a "parallel
loop" and were trying to force FORALL into that mold. Ken
Kennedy asked which is the greater oxymoron: "Trust the
user" or "Trust the compiler"? Marc Snir emphasized the main
point regarding FORALL, that HPF was introducing a new
construct not related to the data mapping it was supposed to
do, and opposed FORALL on those grounds. Peter Highnam
responded that FORALL was correcting brain-dead compilers.
Others mentioned that it corrected an omission of Fortran
90. A short digression reviewed of why FORALL had been
dropped from F90 (to simplify the language), and noted the
usefulness of the Fortran Journal of Development (an F90
appendix for a time) in documenting these features. Marina
Chen noted that if FORALL was not included, then
implementations would start to diverge, and Richard Swift
chimed in that users want this functionality.
  Joel Williamson brought up the question of whether FORALL
must be part of the HPF subset, or as he put it "Is this
decision all or nothing?" Ken Kennedy recommended voting to
accept it first, and later move to put it in the subset if
the group felt that was appropriate. He referred to the
earlier straw poll that restricted the subset to a Fortran
90 subset. There was general agreement to resolve whether to
put FORALL in the subset next time.
  A series of straw polls was then started.
     Should HPF have at least the single-statement
       FORALL? Yes 28, No 4, Abstain 4
       Should HPF leave FORALL  out if it could be
          in a superset? Yes 2
       Should FORALL be in HPF but not in the
          official subset? 15
       Should FORALL be in the official subset? 16
       Abstain from the subset/superset votes 5
     Should HPF have a block FORALL with at least the
       following features (with the currently proposed
       semantics  in each case)?
     Only assign & unmasked array assign?  18 yes 7 no
       9 abstain
       WHERE? Yes 16, No 6, Abstain 13
       FORALL and WHERE? Yes 13,  No 8, Abstain 15
       (Jokes regarding voting for "none of the
          above" and/or Ross Perot had been building
          for some time; this vote brought them out
          in force, too many to record.)
     Should CALL be allowed in FORALL (under the same
       restrictions as functions regarding side
       effects)? This vote was delayed pending the
       next, more fundamental, issue.
     Should function calls in FORALL be required to
       have no side effects except to return a single
       value? Yes 15, No 12, Abstain 8
       That made subroutine calls much less
          interesting by themselves.
     Should HPF allow CALL in FORALL if side effects
       are later allowed? Yes 7, No 14, Abstain 12
     Should HPF include the INDEPENDENT assertion for
       DO loops? Yes 27, No 2, Abstain 5
       Ken Kennedy's immediate reaction was "We
          should run this for President!"
An attempt was made to poll on INDEPENDENT for FORALL, with
the intended meaning that no location written for one
combination of index values is read or written for another
combination. Other meanings, including assertions that
synchronization conditions between statements could be
relaxed, were also proposed. The group finally postponed
discussion of the issue, since it was well after 10:00pm.
They agreed to resume at 9:00 the next morning.
  In the morning, Min-You Wu presented another example of a
possible use of INDEPENDENT in FORALL.
          a = 0
          forall ( i=2:n, a(i-1)=0 )
            a(i) = 1
          end forall
His question was, "Does INDEPENDENT cover the mask
expression of a FORALL?" Using this and examples from a
proposal he had e-mailed earlier, he argued for INDEPENDENT
blocks to control synchronization operations in FORALL.
Piyush Mehrotra and Ken Kennedy pointed out that INDEPENDENT
for FORALL was not defined yet, and recommended considering
these issues in the FORALL subgroup. Clemens-August Thole
agreed, noting again that there is no reason to restrict
INDEPENDENT to FORALL and DO statements.

Local Subroutines

  Marc Snir then began his presentation of LOCAL
subroutines (as Ken Kennedy said, "only 12 hours after