🚀 go-pugleaf

[Paul Prescod]
|
|   I really don't know much about sgml except what it's for.  I *DO*
|   however know what I would use it for.  But i need to know one thing:
|
|   Is there any standard for SGML wherein the text is seperated from the
|   formatting codes, so that for those that have no interest in SGML the
|   text is not mangled?

There are no _formatting_ codes in SGML documents to begin with.  (OK,
so I'm being pedantic, but it's an important point, often overlooked.)

Those who have no interest in SGML should use an SGML-aware editor that
doesn't display the markup, but instead uses some locally meaningful
form of presentation for the individual elements.  Note that formatting
is not in the scope of SGML, although formatting is certainly one of the
things to which you could subject an SGML document.

If you wish to separate the markup from the text completely, you will
need to establish a means to identify the start and end of all elements
in the separated text.  This is certainly _possible_, but you wouldn't
want to do it, because you would end up with more things to take care of
than writing an SGML-aware editor from scratch.

Some of us don't think text is "mangled" by SGML-type markup, but that
may be because we actually know the language, and use it to keep markup
at the level where it is still informative to us.  (It follows that I,
at least, don't use any SGML-"aware" editor, either -- they invariably
know less about SGML than I do and that's frustrating, especially when
they're wrong and being _very_ stubborn about it.)

Best regards,
</Erik>
--
Erik Naggum                 ISO  8879 SGML                    +47 295 0313
                            ISO 10744 HyTime
<erik@naggum.no>            ISO  9899 C                 Memento, terrigena
<SGML@ifi.uio.no>           ISO 10646 UCS             Memento, vita brevis

Still, if one wanted to try an "sgml" aware editor (for free!) - what
are the options?

Brian
--
       Brian Glendenning - National Radio Astronomy Observatory
bglenden@nrao.edu          Charlottesville Va.           (804) 296-0286

In <19930105.004@erik.naggum.no> SGML@ifi.uio.no (Erik Naggum) writes:

>|   Is there any standard for SGML wherein the text is seperated from the
>|   formatting codes, so that for those that have no interest in SGML the
>|   text is not mangled?
>If you wish to separate the markup from the text completely, you will
>need to establish a means to identify the start and end of all elements
>in the separated text.  This is certainly _possible_, but you wouldn't
>want to do it, because you would end up with more things to take care of
>than writing an SGML-aware editor from scratch.

Doesn't HyTime provide this with its CONLOC facility?
Mind you, I'm not suggesting that there are any HyTime editors out there!
-- 
L.Carr@ecs.soton.ac.uk               Les Carr
Tel: +44 703 593649                  Dept of Electronics and Computer Science
Fax: +44 703 593045                  University of Southampton SO9 5NH England

Erik Naggum (erik@naggum.no) writes:

  If you wish to separate the markup from the text completely, you will need to
  establish a means to identify the start and end of all elements in the
  separated text.  This is certainly _possible_, but you wouldn't want to do it,
  because you would end up with more things to take care of than writing an
  SGML-aware editor from scratch.

We're developing a text understanding system that marks up the source with
semantic information in SGML (e.g. <VIOLENCE victim="...">).  What we will
eventually have is a SUITE of such processors, from simple (but fast) to
sophisticated (but slow).  Different users may run different processors on the
same source document, and may eventually wish to merge or compare the results.

What this seems to require is exactly what you describe, namely, keeping track of
where every element (or at least every tag) is, in terms of the original
document.

Since processors might treat white space differently (e.g. two blank lines vs.
indentation for paragraphs), it even seems that we have to index the SGML down to
the individual characters, not just words or other tokens.

Are there any tools we can take advantage of that already do this?

John Burger
john@mitre.org

[John D. Burger]
:
|   We're developing a text understanding system that marks up the source
|   with semantic information in SGML (e.g. <VIOLENCE victim="...">).  What
|   we will eventually have is a SUITE of such processors, from simple (but
|   fast) to sophisticated (but slow).  Different users may run different
|   processors on the same source document, and may eventually wish to
|   merge or compare the results.
|
|   What this seems to require is exactly what you describe, namely,
|   keeping track of where every element (or at least every tag) is, in
|   terms of the original document.
|
|   Since processors might treat white space differently (e.g. two blank
|   lines vs. indentation for paragraphs), it even seems that we have to
|   index the SGML down to the individual characters, not just words or
|   other tokens.

I've thought a little more about this, and maybe someone out there can use
an idea I had.  It goes like this:

Assume that we have text files that do not contain any markup, i.e. so that
we won't need to parse their contents in any way if we refer to and extract
text from them, and, as a corollary, element contents are not split across
files (entities).

Assume that we have a document type which does not have both elements and
data in element contents.

Assume that we have an entity manager that can extract portions of an
entity given an entity, a starting point and a length, however specified.

For each element that contains PCDATA in the DTD, define three attributes

	contents ENTITY   #CONREF
	start    NUMBERS  #IMPLIED
	length   NUMBERS  #IMPLIED

(with the understanding that _start_ and _length_ are REQUIRED if
_contents_ is specified).

The start and length are both NUMBERS because we might have a record-based
storage system that must address a given character by record and position
within a record, as well as one that can address characters individually.

Then, the element structure can be kept in a "regular" SGML document, which
in the document type declaration subset identifies the entities to which
references are made in the start-tags in this document, and the application
can request the element contents from the entity manager while receiving
the element structure from the parser.

I know of no entity manager that allows this flexibility, but that doesn't
mean none exist, or can't exist.  (I also imagine that this kind of data
extraction service will be required by an entity manager to be used with
HyTime engines.)

Best regards,
</Erik>
--
Erik Naggum                 ISO  8879 SGML                    +47 295 0313
                            ISO 10744 HyTime
<erik@naggum.no>            ISO  9899 C                 Memento, terrigena
<SGML@ifi.uio.no>           ISO 10646 UCS             Memento, vita brevis

In <19930108.007@erik.naggum.no> Erik Naggum <SGML@ifi.uio.no> writes:
>I've thought a little more about this, and maybe someone out there can use
>an idea I had.  It goes like this:
>
>Assume that we have text files that do not contain any markup, i.e. so that
>we won't need to parse their contents in any way if we refer to and extract
>text from them, and, as a corollary, element contents are not split across
>files (entities).
>
>Assume that we have a document type which does not have both elements and
>data in element contents.
>
>Assume that we have an entity manager that can extract portions of an
>entity given an entity, a starting point and a length, however specified.
>
>For each element that contains PCDATA in the DTD, define three attributes
>
>   contents ENTITY   #CONREF
>   start    NUMBERS  #IMPLIED
>   length   NUMBERS  #IMPLIED
>
>(with the understanding that _start_ and _length_ are REQUIRED if
>_contents_ is specified).
>
>The start and length are both NUMBERS because we might have a record-based
>storage system that must address a given character by record and position
>within a record, as well as one that can address characters individually.
>
>Then, the element structure can be kept in a "regular" SGML document, which
>in the document type declaration subset identifies the entities to which
>references are made in the start-tags in this document, and the application
>can request the element contents from the entity manager while receiving
>the element structure from the parser.
>
>I know of no entity manager that allows this flexibility, but that doesn't
>mean none exist, or can't exist.  (I also imagine that this kind of data
>extraction service will be required by an entity manager to be used with
>HyTime engines.)

You can get the same functionality by using one level of indirection
and standard HyTime addressing elements.  Simply define a location
ladder that ends with a data location element that specifies the
location and extent of the data to grab (in these examples I've
simplified as allowed by HyTime for brevity):

<!-- First define common attributes to do content-reference: -->
<!ENTITY %conref.att
        "ContentSource IDREF  #CONREF  -- source of element content --
         HyNames       CDATA  #FIXED 'ContentSource linkend'
         HyTime        NAME   #FIXED clink" -- conceptually a clink --
>

<!-- Now define element to contain HyTime elements, for neatness: -->
<!ELEMENT ContentRefSpec  - - (nameloc, dataloc+) >
<!ATTLIST ContentRefSpec HyTime NAME #FIXED hybrid>

<!-- Now define nameloc and dataloc elements.  These are standard
     HyTime. DatalocSource defines source of data for dataloc. -->
<!ELEMENT (nameloc | DataLocSource)
                   O O (nmlist) -- nmquery omitted for simplicity --
>
<!ATTLIST (nameloc | DataLocSource)
                   HyTime NAME #FIXED nameloc
                   ID     ID   #REQUIRED
>
<!ELEMENT nmlist  O O (#PCDATA) -- lextype(NAMES) -->
<!-- Attlist omitted for simplicity -->
<!-- Now define Dataloc element, again, HyTime: -->

<!ELEMENT dataloc O O (dimspec*) -- Will define data locations -->
<!-- Content model reduced to 'dimspec' for simplicity -->
<!ATTLIST dataloc HyTime NAME #FIXED dataloc
                  id     ID   #REQUIRED
                  locsrc IDREFS   #REQUIRED -- no logical default --
          -- other attributes omitted for simplicity --
>

<!ELEMENT dimspec O O (marklist, marklist) >
<!ELEMENT marklist - O (#PCDATA) -- Lextype(snzi*) -->
<!-- Content model reduced to '#PCDATA' for simplicity -->

<!-- Now define our application-specific elements.  All will
     use the common content reference attributes: -->
<!ELEMENT MyData  - - (%mydata.el;)* -- typical element -->
<!ATTLIST MyData
          %conref.att;
>

I've defined the omission indicators such that for manual
typing, you only have to specify a few of the tags to
create a reference:

<!-- In the DTD, define the entity that contains the data: -->

<!ENTITY thedata SYSTEM "data.file" CDATA >

<!-- Now use DataLocSource to create the entity reference. This
     will be refered to from DataLoc with the locsrc= attribute: -->
<DataLocSource id=mydata nametype=entity>thedata</>

<!-- stream-oriented (one dimension) reference: -->
<!-- Note that ContentRefSpec is implied by nameloc. -->
<nameloc id=dataref1>dimspec1
 <dataloc id=dimspec1 locsrc=mydata>
 <marklist>10 100
<MyData contentref=dataref1>

<!-- line-oriented (two dimensions) reference: -->
<nameloc id=dataref2>dimspec2
 <dataloc id=dimspec2 locsrc=mydata>
  <marklist>1 1<!-- First record -->
  <marklist>10 -1<!-- 10th through last character -->
<MyData contentref=dataref2>

Interestingly enough, Erik's notation could be transformed
into what I've defined above on the fly by an application.
It would not itself be HyTime conforming or directly
processible by a HyTime system, but it could be transformed
into "virtual SGML" dynamically and then passed to a HyTime
system.  Note also that the above method allows the
use of multiple data locations that can be aggregated
into a single value when the location chain is resolved.

You can also define some application-specific simplifications.
For example, since I have namelocs and datalocs contained within
a common element, I could define the processing semantic for
my application such that the nameloc element is implied by
the existence of the dataloc elements within the ContentRefSpec
element.  The application defines the nameloc ID, perhaps by taking
the nameloc ID value from an attribute of the ContentRefSpec:

<!ELEMENT ContentRefSpec  - - (nameloc, dataloc+) >
<!ATTLIST ContentRefSpec HyTime NAME #FIXED hybrid
          id            NAME   #REQUIRED  -- defines nameloc ID --
          DataContainer ENTITY #REQUIRED  -- defines locsrc value --
>
<!ELEMENT nameloc O O (nmlist) -- nmquery ommitted for simplicity --
<!ATTLIST nameloc  HyTime NAME #FIXED nameloc
                   ID     ID   #IMPLIED  -- From ContentRefSpec --
>
<!ELEMENT dataloc O O (marklist) >
<!ATTLIST dataloc  HyTime NAME #FIXED dataloc
                   ID     ID   #IMPLIED  -- Defined by processor --
                   locsrc ID   #IMPLIED  -- From ContentRefSpec  --
>

The ID= attribute on ContentRefSpec is not of type ID, so
it's value will not be part of the SGML ID name space, but we
can define an application semantic that uses the ContentRefSpec
ID= value to provide the now implied nameloc ID.  Because
ID= is required on ContentRefSpec, it is effectively required
for nameloc, thus meeting the requirement of ISO 10744.  If
we further define the Dataloc ID to be implied by the containment
structure, we reduce the actual required markup to:

<ContentRefSpec id=dataref1 datacontainer=thedata>
  <marklist>1 1
</ContentRefSpec>

Similarly, the location source specification is simplified by the
ENTITY attribute DataContainer, which is used to create a virtual
nameloc for the ID-to-entity mapping needed by dataloc.  The nameloc,
nmlist, and dataloc elements are all omissible and the required
attributes are now implied unambiguously.  The ContentRefSpec clearly
identifies the application-specific semantic of this particular
hyperlink and the details of the HyTime stuff is hidden (or at least
hideable).  This does require that the HyTime elements not be part of the
general content of your application, because it would be a violation
to allow nameloc with implied ID outside a context that guarantees
specification of an ID by some mechanism.

Eliot Kimber                      Internet:  drmacro@ralvm13.vnet.ibm.com
Dept E14/B500                     IBMMAIL:   USIB2DK9@IBMMAIL
Network Programs Information Development     Phone: 1-919-543-7091
IBM Corporation
Research Triangle Park, NC 27709