🚀 go-pugleaf

[John D. Burger]
:
|   We're developing a text understanding system that marks up the source
|   with semantic information in SGML (e.g. <VIOLENCE victim="...">).  What
|   we will eventually have is a SUITE of such processors, from simple (but
|   fast) to sophisticated (but slow).  Different users may run different
|   processors on the same source document, and may eventually wish to
|   merge or compare the results.
|
|   What this seems to require is exactly what you describe, namely,
|   keeping track of where every element (or at least every tag) is, in
|   terms of the original document.
|
|   Since processors might treat white space differently (e.g. two blank
|   lines vs. indentation for paragraphs), it even seems that we have to
|   index the SGML down to the individual characters, not just words or
|   other tokens.

I've thought a little more about this, and maybe someone out there can use
an idea I had.  It goes like this:

Assume that we have text files that do not contain any markup, i.e. so that
we won't need to parse their contents in any way if we refer to and extract
text from them, and, as a corollary, element contents are not split across
files (entities).

Assume that we have a document type which does not have both elements and
data in element contents.

Assume that we have an entity manager that can extract portions of an
entity given an entity, a starting point and a length, however specified.

For each element that contains PCDATA in the DTD, define three attributes

	contents ENTITY   #CONREF
	start    NUMBERS  #IMPLIED
	length   NUMBERS  #IMPLIED

(with the understanding that _start_ and _length_ are REQUIRED if
_contents_ is specified).

The start and length are both NUMBERS because we might have a record-based
storage system that must address a given character by record and position
within a record, as well as one that can address characters individually.

Then, the element structure can be kept in a "regular" SGML document, which
in the document type declaration subset identifies the entities to which
references are made in the start-tags in this document, and the application
can request the element contents from the entity manager while receiving
the element structure from the parser.

I know of no entity manager that allows this flexibility, but that doesn't
mean none exist, or can't exist.  (I also imagine that this kind of data
extraction service will be required by an entity manager to be used with
HyTime engines.)

Best regards,
</Erik>
--
Erik Naggum                 ISO  8879 SGML                    +47 295 0313
                            ISO 10744 HyTime
<erik@naggum.no>            ISO  9899 C                 Memento, terrigena
<SGML@ifi.uio.no>           ISO 10646 UCS             Memento, vita brevis