🚀 go-pugleaf

In my previous article [IPC (1): Ports & Messages] I was arguing that we
need a properly defined protocol for passing messages between programs,
and I suggested that we might need something more general than
individually created links between specific programs.  In this article I
want to look at that idea in more detail.

Ron Minnich's "tools-based hypercard" concept, which started this whole
discussion off, is just a nebulous glimmer at this stage, but if you
envisage it as I do it will be as a suite of small programs, any one of
which can call on any other to perform the task it is designed for, using
information provided by -- or generating information for -- the invoking
program.  As the whole point is to keep the system modular and adaptable,
it would be a horrendous task to keep track of a network of ports belonging
to the individual programs.

It will be much cleaner if a module that wants something done can "say to
the air", so to speak, "I want information on this; does anyone have it?",
or "I have this new information: does anyone want it?".  In other words, a
Broadcast message.  Once a broadcast request has been answered, the two
programs can always set up private communication ports of course, but this
might not always be the way to go either.  You can broadcast specific
information as well as requests, and the advantage of this is that you
don't have to be concerned how many others are listening: any program that
wants the information can pick it up and use it; if no one wants it, it
will just drift off into the ozone.

Another example: we have a "tools-oriented" publisher package (I wish!),
together with our favorite editor and paint package, which also have such
facilities [Stop dreaming, Peter...].  The publisher has no edit or paint
facilities of its own: it can read existing files, but all manipulation is
done by the other programs.  Suppose we start up the publisher and ask it
to format a particular text file; it reads the file as it is at that
moment, but at the same time sets up to receive any new information that
might be generated about that file.  We look at the results previewed on
the screen, and decide that we want to re-edit a paragraph, so we click on
the editor icon (on the workbench -- the publisher doesn't know about it
specifically).  The editor first off sends out a message "What text file?",
which the publisher --knowing which file we are working on -- will respond
to.  Note that in the general case SEVERAL programs might respond, and the
editor would then pop up a requester asking the user to select one.  The
editor then reads the original file (it won't try to work on the
publisher's data image -- no telling what format it is in...), and you can
start editing.  Every significant change you make to the text (with some
appropriate atomicity) is put into a message which the editor tosses out.
The publisher is looking for these, so each time it updates its own preview
image suitably.

That's a sketch which might give you some idea of my dreams -- I've left
out most of the details, and ALL the snags.  This sort of scenario could
also be handled by straight message passing between the two programs, but I
like the general idea of the editor, for instance, not having to know or
care what use is going to be made of the information it tosses out. The
program receiving it might be an incremental compiler rather than a
publishing package, or it could be just a communication link, sending it
(via DNET, naturally...) to some remote installation.

In this situation, there probably wouldn't be more than one other program
interested in a particular message, but how about a scientific data
acquisition module, continuously collecting data?  You would want a
permanent record of it all on disk, perhaps compressed by a second module,
and you might also want to do some analysis (or a couple of kinds of
analysis) on it in real time as it arrived.  Here there are several
modules, all working on the same stream of messages from the first module.

Closer to the needs of most people, there's been some desire expressed on
the net recently for VT100 to have "plug-in modules" for specific purposes.
The "Broadcast message" mechanism would be a natural for this.  In fact I'd
go one further and just have a fairly dumb Communications Module (without
screen display or keyboard input) that simply sends text block messages it
receives out the serial port, and broadcasts incoming characters to whoever
wants them.  If all messages ended up being one character long, we could
hit severe speed restrictions, but I think there may be ways around this.
I'll leave that for further thought, though.


Alright, let's get to some specifics.  What I'm proposing is a "message
broker", which is built from a single public port with a process attached
[or a process with a port attached; whatever].  It could also be
constructed as a resident library, but for now let's go with it as a
process.  Other programs would pass it messages regarding their interest in
a particular "topic" (using the standard "IPC" protocol defined in the
previous article).  These messages would initially be one of: "I want to
know about this topic", or "I want to give information on this topic".

The topic is a text string -- whatever is suitable --, but this only has to
be used for the initial contact.  In the ID word at the head of the message
data area the broker will return a handle value, to be used when actual
information is passed on that topic.  Topics are kept (with their handles)
in a lookup list, which is added to as necessary. The requesting program
supplies a reply port as usual, and this is filed away by the broker under
that topic.

When the broker receives a data message on a topic, it looks up -- using
the handle -- which reply ports should be informed, makes duplicates of the
message as necessary, and dispatches these.  Normally the original message
will only be replied when all the spawned copies have been replied; the
sender could probably also specify that it wanted an immediate reply if the
message didn't contain any pointers to other memory.  You won't be able to
pass information back in replied messages, though, because they're copies.

If we keep the handle less than 32 bits it can be one of the "private" IDs
discussed in the last article, and can go in the ipc_ID slot.  The message
itself also needs an type ID, so that the receiving program will know how
to handle it, and this would presumably go into the next word. The data
area proper would follow this, so the message structure would be -- aside
from the extra word -- just the same as a standard IPC message.

        struct BrokerMsg {
            struct Message  brkr_Msg;
            ULONG           brkr_Topic;
            ULONG           brkr_ID;
            /* Data follows...*/
        };

With a scheme like this, a receiving program gets only the messages it is
interested in without having to know who or where the originator is. To me
this looks like a good foundation for a flexible network of programs. The
disadvantages as usual are in extra overhead, both in message structure and
the time taken in duplication and so on.  So now tell me all the other
objections...

                                        -- Pete --