🚀 go-pugleaf

I have two problems with Expect.

I'm trying to use libexpect to write an application which manages
a dialogue with two pairs (4 total) of children concurrently.
I've discovered that Expect's use of global buffers makes it
impossible to leave anything in one child's output buffer while
doing an exp_expectl() on another child's output.

Unfortunately, this limits the utility of Expect for my purposes.
I've had to restructure my program to work around it and I've
sacrificed a fair bit of simplicity and elegance in order to do
so.  

When I read the documentation, I realized that exp_buffer and the
rest were globals.  However, I (incorrectly) assumed that
you used per-process buffers internally and artificially set these
globals after a call to expect to point to the appropriate child's
buffer.  That is, that the global would only be useful until the next
expect call was made.  Would it be feasible to modify the
implementation to behave this way?  The way it is now, I could
have almost as easily used strtok() to do my parsing.

I've worked around this particular problem but have now encountered a
much more serious challenge. One of the things my program does is issue
a command to a utility which produces a list. The output looks like
this:
NUMIDS  29
IDNAME  S1       A
CLASS              variable : 4 1
IDNAME  S1       AREA1
CLASS              type : 0 1
IDNAME  S1       AREA2
CLASS              type : 0 1
IDNAME  S1       INT
CLASS              type : 0 1
...
end-of-data [SIC]

For each IDNAME, I need to get the symbol (A, AREA1, ...), the
class (variable, type, sym range elem, ...), and something called
the def_index (the first digit in the pair).

I use a front-end to exp_expectl() to generalize my parsing. It looks
like this:

=============================================================================
  /* -----------------------------------------------------------*/
  /* Get symbol name, class, and def_index                      */
  /*                                                            */
  /* NAMEID <Section> <Symbol>                                  */
  /* CLASS  <class> : <def_index> <num_references>              */
  /* e.g.                                                       */
  /* NAMEID MYSECT1 RED                                         */
  /* CLASS         sym range elem : 4 1                         */
  /*                                                            */
  Read_To_Pattern(xref, "IDNAME +([A-Z0-9]+) +([A-Z0-9]+)\r\n", 2);
  symbol->id     = strdup(Last_Buffer_Read(xref, 2));
    
  Read_To_Pattern(xref, "CLASS +([a-z ]+) : ([0-9]+) +([0-9]+)\r\n",  3);
  symbol->sclass    = strdup(Last_Buffer_Read(xref, 1)); /* sym range elem */
  symbol->def_index = strdup(Last_Buffer_Read(xref, 2)); /* 4              */
=============================================================================

Read_To_Pattern() does the following:

=============================================================================
  regexp *re; /* compiled regular expression */
  re = TclRegComp(pattern);
  
  state = exp_expectl(child->in_fd,
                      exp_compiled, pattern, re, ES_MATCH,
                      exp_end);
  switch (state)
  {
  case ES_MATCH:
      {
        int sub_pattern_index;
        
        for (sub_pattern_index=1;
             sub_pattern_index <= match_num;
             sub_pattern_index++)
        {
          re->endp[sub_pattern_index][0]='\0';
          child->last_response[sub_pattern_index-1] 
             = strdup(re->startp[sub_pattern_index]);
        }
      }
      break;
  case ES_APPL_ERROR: ...
  case EXP_TIMEOUT: ...
  case EXP_FULLBUFFER: ...
  case EXP_EOF ...
  default
  }
=============================================================================

Now this all works.  Except that after approximately 20 iterations
(i.e.  it varies from one run to the next), it skips part of the
buffer and prematurely gets to the end of the output.  It then
times out.

Here's a fragment of my debugging output (with exp_loguser on) to
illustrate:

=========================START SAMPLE OUTPUT===============================
...
IDNAME  S1       SET2
CLASS              type : 0 1
get_next_global_symbol: ID:        SET2
get_next_global_symbol: CLASS:     type
get_next_global_symbol: DEF_INDEX: 0
IDNAME  S1       SET3
CLASS              type : 0 1
get_next_global_symbol: ID:        SET3
get_next_global_symbol: CLASS:     type
get_next_global_symbol: DEF_INDEX: 0
IDNAME  S1       SET4
CLASS              type : 0 1
IDNAME  S1       T12
CLASS              type : 0 1
IDNAME  S1       TABLE1
CLASS              type : 0 1
IDNAME  S1       TABLE3
CLASS              type : 0 1
IDNAME  S1       TABLE4
CLASS              get_next_global_symbol: ID:        SET4
get_next_global_symbol: CLASS:     type
get_next_global_symbol: DEF_INDEX: 0
type : 0 1
IDNAME  S1       TABLE5
CLASS              type : 0 1
IDNAME  S1       TS1
CLASS              type : 0 1
IDNAME  S1       TS2
CLASS              type : 0 1
IDNAME  S1       TS3
CLASS              type : 0 1
IDNAME  S1       U
CLASS              variable : 0 1
IDNAME  S1       X
CLASS              variable : 1 1
IDNAME  S1       Y
CLASS              variable : 1 1
IDNAME  S1       Z
CLASS              variable : 1 1
end-of-data
get_next_global_symbol: ID:        T12
get_next_global_symbol: CLASS:     type
get_next_global_symbol: DEF_INDEX: 0

ERROR: Read_To_Pattern(4): TIMEOUT trying to match 'IDNAME +([A-Z0-9]+) +([A-Z0-9]+)
'
keithh%bcarh635 [98]: 
==========================END SAMPLE OUTPUT===============================

Everything works for a while and then all of a sudden, Expect lets
more than the match get consumed.  Notice in my code fragment
above how I've constructed the two regular expressions explicitly
to avoid this.  There is no wild-card character in either
expression.  In theory, only a single line of output can match.

No matter what I do, the problem persists. I've tried to speed the
processing up by turning off all my debugging output. I've turned off
exp_loguser. I've set exp_fullbuffer to True (and it never happens).
I've increased exp_match_max to 5000 and decreased it to 1000. Nothing
helped.

I'm in rather desperate straights and would appreciate your help.

Please reply directly to me as well as to the newsgroups because
my newsfeed is currently in a state of flux and is unreliable.

Thank you very much,
Keith Hanlan KeithH@bnr.ca  Bell-Northern Research, Ottawa, Canada 613-765-4645