🚀 go-pugleaf

In article <4141@bcstec.ca.boeing.com> silverm@bcstec.ca.boeing.com (Jeff Silverman) writes:
>From: silverm@bcstec.ca.boeing.com (Jeff Silverman)
>Subject: QEMM/386
>Summary: What is QEMM 386 actually doing and how does it do it?
>Keywords: QEMM memory management 80386
>Date: 9 Nov 92 05:44:05 GMT
>
>I am a VAX/VMS guru in the process of becoming an MS-DOS guru.  If these
>questions sound naive, it is because I am coming at these from a strange
>perspective.

Not naive actually - more like questioning the lack of common sense 
involved in DOS programming. Most of us have been asking the same things for 
years. It's because DOS was written way back when no one could imagine 
needing more than 64K of memory. To this day DOS is still configured to run 
on an 8086.

>In the VAX/VMS operating system, programs execute in their own virtual address
>space, which gets mapped on a page by page basis into physical memory.  2 VMS
>programs can be running at the same virtual address yet address different
>pages of physical memory, because the mapping is done on a per-process basis
>and the OS handles all that transparently.  The programmer sees 1, big pool
>of RAM.

Sounds like OS/2. :)
[I am a certified OS/2 addict. I admit it.]

>What does QEMM/386 do, from a programmers point of view?  It appears to move
>things around in both physical and virtual memory.  How is this possible?
>If something, say, a TSR, is moved out of low memory and into either High
>Memory (By high memory, I mean the region between 640K and 1 Mbyte) or extended
>memory, how does it keep working?  All of the destinations for jump instructions
>would change.  Or does the 80386 have additional memory relocation?
>How does QuarterDeck's OPTIMIZE program work?  Please don't say it works well,
>I KNOW it works well, I've been playing with it, I can see that it works, but
>don't know what it's doing or how it does it.  Well, I do know what its doing,
>it's modifying CONFIG.SYS to free up more RAM in the lower 640K region.

A program like a TSR would have been written from the start to execute no 
matter where it gets loaded - i.e. no absolute jumps. You'd use branches if 
you could or a jump table that you fill in at loadtime if not. (I'll 
admit there are probably more sophisticated ways to do that. Different 
strokes...I first did the jump table thing on my Apple //e... [ugh])

>Why does DOS have a 640Kbyte limit?

Ask Bill Gates. :)
Seriously, it's because DOS was written for the 8086 which had a physical 
address limit of 1M. (20 address lines - 2^19 = 1M.) Of this, the last 384K 
was designated as for use by the operating system and by system ROMs by the 
designers. (Like I said, they never dreamed people would actually have 64M 
of RAM in their _desktop_ machines.) This was a convenient number since 
640K = 10 64K segments, each of which can be addressed directly with 16 
bits. 

>Why would anybody ever want to use expanded RAM if extended RAM is available?

You wouldn't. It just hangs around since some older programs used it because 
at the time (pre-386) it was the only option. And there _are_ still people 
with big expanded memory cards in their machines that would whine 
incessantly if they were told to join the late 80's and chuck them.

>If I have a "C" program which runs properly under VMS, but needs, say, 3 Mbytes
>of RAM to store all its data structures, what is the best way to run it under
>DOS?  Under Windows?

Under DOS I would go for overlays. Either to XMS or to disk, depending on 
what's available and how nice of a pre-built overlay manager you use.

Under Windows all you have to do is ask the Windows memory manager for the 
memory. It does the virtual remapping thing you were talking about.

Under OS/2 you also just ask for it. You can get all the physical RAM you 
want plus as much virtual as the user has room for on their HD. Plus you 
don't have to deal with segmented addressing (which is really irritating to 
have to deal with under DOS and, to some extent, Windows.).

Really the nicest thing that QEMM does for the DOS user is pool all physical 
memory and reallocate whatever is needed to running programs as whatever 
type they want. In theory all of the other features are already in DOS 
(except for the stealth feature) - they're just better implemented under 
QEMM. (or 386Max for that matter)

Good luck with your quest. If you need more I'm sure someone else (or me for 
that matter) can fill in the gaps.

----------------------------------------------------------------------------
Steve Luzynski @ Case Western Reserve University in Cleveland, OH
sal8@po.cwru.edu             | voice calls cheerlessly accepted at x2153.
----------------------------------------------------------------------------
current mental status: more or less at peace with the world, all things 
considered.
----------------------------------------------------------------------------
this post was written while i should have been doing homework. therefore,
you are indirectly responsible for my current gpa, such as it is.

In article <4141@bcstec.ca.boeing.com> silverm@bcstec.ca.boeing.com (Jeff Silverman) writes:

 > What does QEMM/386 do, from a programmers point of view?

From a VMS perspective, or from the perspective of a programmer used to
UNIX or OS's with flat virtual memory, QEMM/386 doesn't do very much for
you directly. What QEMM/386 does is to act as a EMS, XMS, VCPI, and DPMI
server. What does this mean? Jump to a couple of paragraphs below. But
first, keep in mind that from the 80386 processor and up, Intel's CPU's
support multiple views of memory which can be confusing. In real mode,
the chips emulate the 8086 which cannot address more than one megabyte
of memory. Addresses are generated as a 16-bit segment and a 16-bit
offset, which are combined by shifting the former 4 bits to the left and
adding, to create a 20-bit physical address. This is a pain for compiler
writers and assembly programers because segments are 64Kb in size and
you often need to use different sizes of pointers (16-bit near pointers
and 16:16-bit far pointers). (Note that this description more or less
still applies in V86 mode, which these chips have in order to be able to
emulate an 8086 while still running a real operating system in protected
mode, except that the 20-bit address formed is a linear and not physical
address -- the next paragraph explains the difference.)

In protected mode (note I omit specific discussion of the 80286 here),
the chips deal with memory in a slightly more complicated way. Instead
of 16-bit segment values which translate directly to the physical
address, you load 16-bit segment selectors into the segment registers.
You can then use offsets up to 32 bits in size (each selector has
information associated with it in a table, indicating the 32-bit linear
address which the segment starts at, whether the contents are data or
code and how large the segment is to allow memory protection, and other
information), so a pointer is 16:32 bits, or 48 bits (making the virtual
address space of the chips some pretty large number:
281,474,976,710,656). However, commonly code which takes advantage of
the 80386 simply set the segment registers to selectors which point to a
common place in linear memory, set the limits to the full 4Gb, and
thereafter never twiddle the selectors again, thereby implementing a
flat memory address space with 32-bit "pointers".  In addition to this
segmentation organization, the chips are capable of paging. Once the
chips resolve a memory reference from a 16:32 pointer to a 32-bit linear
address, they go through page translation (standard 4Kb page size) to
the 32-bit physical address, e.g. actually goes out on the memory bus.

OK, back to QEMM/386:

(a) EMS and XMS: QEMM/386 takes charge of as much of the conventional
    and extended memory as you direct (defaults to all). It takes charge
    of conventional memory so that if you run Desqview, it will be able
    to page different collections of memory pages into the conventional
    memory space so that DOS programs can be multitasked. QEMM/386
    allows programs which understand how to make use of expanded (EMS)
    or extended (XMS) memory to do so, with the advantage that it will
    freely interconvert between the two as necessary because it acts as
    the EMS and XMS servers. For example, if you have 7 megabytes of
    memory free, you can sequentially run program A, which uses six
    megabytes of expanded memory, and then program B, which uses six
    megabytes of extended memory, without having to tell QEMM/386 to
    shift memory from the EMS pool to the XMS pool: it is automatic.

(b) VCPI and DPMI: in the late 1980's, a number of software applications
    were starting to take advantage of the protected mode features of
    the 386+ chips (which include more than just changing the memory
    model). However, these programs would get into each others way by
    clobbering each other's data because there was no protocol to manage
    the chip's resources. VCPI (Virtual Control Program Interface) was
    created by Quarterdeck in conjunction with a number of other
    companies who were writing system-level software (386 DOS extenders,
    multitaskers like Desqview) or applications so big that they needed
    to use protected mode to fit to allow multiple programs which used
    the 80386 to co-exist. VCPI is simple and mostly allows programs not
    to get into each other's way. DPMI (DOS Protected Mode Interface)
    was created by Microsoft, Intel, and a committee of other companies.
    A program which is compatible with DPMI calls the DPMI server
    software to access the special features of the 386+ chips. I believe
    only one of the most recent revisions of QEMM allows it to act as a
    DPMI server. Windows 3.x is the best known and commonest DPMI server.

 > How is this possible?  If something, say, a TSR, is moved out of low
 > memory and into either High Memory (By high memory, I mean the region
 > between 640K and 1 Mbyte) or extended memory, how does it keep working?

TSR's and device drivers which are loaded in high memory or the HMA (an
area of memory slightly less than 64Kb which isn't in the first megabyte
of memory but which can be accessed by opcodes executing in real or V86
mode due to an artifact of creating the 20-bit address: if you take the
segment value to be 0xFFFF, you can access 65536-16 bytes of memory past
the first megabyte because adding offsets greater than 16 when using
registers which are 32-bits yields an address with the twenty-first bit
turned on, two) don't really notice anything different than if they were
loaded normally. Yes, they have been relocated in the first megabyte
address space, but the normal DOS .COM, .SYS, and .EXE formats are
designed so that they can be loaded anywhere in that first megabyte.

If you are really asking how programs like Desqview which can multitask
DOS programs can work, Desqview at every task switch remaps the sets of
memory pages which contains the applications being multitasked into the
first megabyte of linear address space, either by calling on QEMM/386 or
alternatively by calling on any EMS 4.0 servers loaded which has the
hardware to remap memory pages.

 > How does QuarterDeck's OPTIMIZE program work?  Please don't say it works
 > well, I KNOW it works well, I've been playing with it, I can see that it
 > works, but don't know what it's doing or how it does it.

OPTIMIZE tries to figure out how much memory all the TSR's and device
drivers you use on your machine takes up, and which ones would fit into
high (640-1024Kb) memory and the HMA. Note that various memory managers
have come up with tricks to make more high memory available, most of
which plays tricks with mapping memory pages into the linear space that
would otherwise be taken up by ROM-BIOS code that isn't being used any
more in the operation of the computer.

 > Why does DOS have a 640Kbyte limit?  Why would anybody ever want to use
 > expanded RAM if extended RAM is available?  If I have a "C" program
 > which runs properly under VMS, but needs, say, 3 Mbytes of RAM to store
 > all its data structures, what is the best way to run it under DOS?
 > Under Windows?

IBM made a reasonable decision back in 1981 to reserve some of the
one-megabyte address space of 8088/8086 chip for future use and
expansion purposes. The correct limit to complain about isn't the 640Kb
limit, it's the one megabyte limit imposed by the 8088/8086 CPU, and it
is unfortunate that so much effort and software has been written to try
to wring the last tiny bit out of the 384Kb difference between 640Kb and
1024Kb when the horizons could be so much larger.

If you have to write programs which needs lots of memory space, want
excellent performance and don't need to worry about whether your
customers will be able to run your programs (e.g. your code won't be
redistributed or is an in-house application), your best bet is to look
at the 32-bit C compilers that support going past DOS and 8088/8086
limitations -- you get much better compatibility with code which makes
evil assumptions like sizeof(int)==sizeof(pointer) and that of course it
is OK to do a malloc(1048576). There are a couple of commercial C
compilers for DOS which support use with a 386 DOS extender, but there
are also freely distributable ports of the GNU gcc/g++ compiler for DOS
(DJGPP) and for OS/2 2.0 (actually two slightly different ports). The
reason that a number of commercial applications support using expanded
or extended memory rather than using a DOS extender, even though it
takes more effort to write such code, is that expanded memory can be
added to *ANY* PC-compatible, including 8088/8086 machines, and extended
memory can be added to 80286 machines. Therefore, by writing to these
lower denominators, the applications will work passably on a larger
number of machines than if you were to limit their use to 386+ CPU's.

--
| Ping Huang (INTERNET: pshuang@martigny.ai.mit.edu) speaking for himself.
| "One Thing to name them all, One Thing to define them,
|  One Thing to place them in environments and bind them..."