🚀 go-pugleaf

Mark Sherman ran into this problem when he was doing the original MacIP
multitasking, and I am currently doing a UNIX-like multitasking system for
the Mac and working on the same problem.  I have not yet actually tried to
disable the stack sniffer, but here are some approaches:

(1) Patch SysError to return when it is called with the relevant error.
This would likely not be an RTS, but a trickier stack pop.  Problem: future
ROM versions.

(2) Find the sniffer task in the retrace queue and remove it.  It would have
to be found by hand for each version of the ROM, and an INIT resource
installed that removed the appropriate task depending on ROM version.
problem: future ROM versions.

(3) Laborious but possibly workable solution: remove each task from the
vertical retrace queue (inside an INIT resource) and then put them back,
running each as the only task in the queue.  While doing this, run your
stack in the application heap.  When one is running, SysError will be called
with the appropriate error status.  Patch SysError to hand control back to
your INIT resource and bingo, you've found the stack sniffer.  This isn't so
hot if Apple decides to hard-wire the sniffer into the vertical retrace
manager or something equally gross.

(4) Do what Mark did in MacIP: allocate your stacks from the stack region,
above the application heap.  This could be done either using your own
allocation routines (as Mark did) or by creating a new heap zone above the
application heap and below the main stack (which I always meant to do but
never got around to).

(5) Make the application run in user mode on the 68000 and create a separate
system stack.  Then the vertical retrace interrupt will shift onto the
system stack, which you can set to be above BufPtr somewhere.  This requires
a lot of hard work, hacking the trap dispatcher.

(6) Depending on your environment, you can fool it in other ways.  For
instance, you might be able to dispense with an application heap entirely,
setting the low-memory globals to some low value, so the stack sniffer will
be fooled.

I hope this helps.  Unfortunately, Apple did not include any documented way
to disable the stack sniffer.
-- 
Tim Maroney, Electronic Village Idiot
{ihnp4,sun,well,ptsfa,lll-crg,frog}!hoptoad!tim (uucp)
hoptoad!tim@lll-crg (arpa)

Give me food, or give me slack (or kill me).