🚀 go-pugleaf

I've defended lazy* evaluation in a previous message, now I have
question that may give some ammunition to the strict* camp.

Consider the following Haskell fragment which reads lines data from a
file into the "store" :-

  read_ASCII_Store st file fail succ = readFile file fail (succ . store)
    where
      store = (foldl (flip process_line) st) . (map words) . lines

where "process_line" has the form :-

  process_line :: [String] -> Store -> Store

This works fine.  It is nice and lazy i.e. if you invoke it, it does
not do anything until you actually try to access some of the values of
"store".  However, there is a price to pay for this.  When the input
file is reasonably large, this creates a large amounts of "thunks"
when reading in the data, in fact the "thunks" dominate the
proceedings and cause much more heap to be used than should be
necessary.  (It runs out of space in a 5Mb for an "interesting" input
file)

This appears to be well known problem (a number of people here have
had similar problems anyway), and the suggested solution is to "force"
the store at regular intervals so that it fully evaluates itself.
This results in the following code, where 100 is the number of lines
to read between "forcing" :-

    store xs = store_loop 0 st (lines xs)
    store_loop n st []    =
      if (st == st)           -- force the store
      then st
      else error "the FORCE is not with us"
    store_loop n st stuff =
      let st' = process_line (words (head stuff)) st
      in if (n `mod` 100 == 0)
         then if (st' == st') -- force the store
              then store_loop (n+1) st' (tail stuff)
              else error "the FORCE was not with us"
         else store_loop (n+1) st' (tail stuff)

This works well, and keeps the usage down to under 2Mb for the input
that previously ran out of heap.  However, there are a couple of
things that bother me about this :-

1. The extra amount of work that has to be done in "forcing" the store.
   This bumps up the runtime.

2. A simple, one-liner using higher order functions has turned into a
   10 line tail recursive loop.  Now being a Schemer at heart, I have
   no particular problem with this, but it does seem to go against the
   whole idea of "higher order functions are good" which most fp books
   espouse.

So, I'm asking the net at large for advice.  Does anybody know of any
other methods of keeping the heap usage low and that do better in
answering 1&2?  Something using continuations or monads maybe?

bevan

* apologies for the imprecise terminology, I hope you all know what I
  mean anyway.