🚀 go-pugleaf

mskuhn@immd4.informatik.uni-erlangen.de (Markus Kuhn) writes:

>At present nearly all public domain files in the netland are available
>with the FTP protocol. This is a widely used but not very comfortable
>solution.

There have been a number of relatively recent improvements, allowing FTP
servers to send informational messages concerning the contents of
directories, etc. For an example, here's an excerpt from the motd from
gatekeeper.dec.com:

>     Extended commands available via:
>        quote site exec COMMAND
>     Where COMMAND is one of:
>        index PATTERN   - to glance through our index (uses agrep).  example:
>                          ftp> quote site exec index emacs

For another example, an excerpt from a session with ftp.uu.net shows an
informational message for a directory:

    ftp> cd stone
    250-Files within this subtree are automatically mirrored from
    250-gate.fzi.de:/pub/OBST
    250-
    250 CWD command successful.

It should be simple to extend the site-specific commands to allow users
to request descriptions of files, etc. The problems I see are:

- keeping a lot of file descriptions up-to-date is a pain in the rear.
  A lot of times you just can't grab the readme file out of an archive;
  I'm not sure how many archive maintainers have time for much more than
  that.

- the user interface is atrocious - "quote site describe <file>" isn't a
  very intuitive way to get information on a file. A better FTP client
  would fix that.

Another solution would be to go with something like the Andrew File
System, so that you could examine files individually using native
operating-system operations, rather than being stuck within some
archive-browsing-specific application.

>What do you believe OSI has to offer as an alternative?

I know I'm going to inflame some of the OSI people out there, but I
would suggest: not a whole hell of a lot. FTAM doesn't give you anything
that FTP doesn't, except the ability to translate even more obsolete and
odd file formats that no one uses for archives, since usually files are
stored in some bundled, compressed form. It's likely that document
lookup systems, on the other hand, might not have enough
operating-system file support to handle the task. Finally, the
likelihood that FTAM will come into widespread usage as a remote
file-mounting system - ala NFS and AFS - is very low. In a perfect world
it would be even lower.

>[lots of good ideas for archive-browsing deleted]

You might try getting a copy of ftpd and hacking some of your
ideas into it. Combine it with a client user interface that hides
the "quote site" incantation from users, and see what people think... 

				Peter Desnoyers
-- 

peterd@pjd.dev.cdx.mot.com (Peter Desnoyers) writes:

>It should be simple to extend the site-specific commands to allow users
>to request descriptions of files, etc. The problems I see are:

>- keeping a lot of file descriptions up-to-date is a pain in the rear.

>- the user interface is atrocious - "quote site describe <file>" isn't a
>  very intuitive way to get information on a file. A better FTP client
>  would fix that.

Yes, these are the two best reasons, why extending ftp won't lead to a
very comfortable solution.

My vision is this:

Let's define two terms:

     a file       this is want we know from ftp, FTAM, the UNIX filesystem,
                  Andrew,... . It is just a byte sequence with a very
                  limited number of additional attributes, e.g. a
                  filename from a very restricted characterset, a date
                  and time of last modification, a owner ID, access rights, ...

     a document   this is all a file contains plus additional attributes,
                  e.g. a one-line-description, a more detailed description,
                  keywords, a describption of the file format, the compression
                  method, a version number, an author, a worldwide unique
                  identifier, ...

A well designed document server would also have ftp (and FTAM :-) access,
but with these access methods you only see files. If it were possible
to establish a standard for "documents", then each information object
published has only once to be stored in a "document server" with all
it's additional attributes. Then every manager of an other "document server"
could get a copy of a complete document with a kind of "document access
protocol". So the additional attributes only have to be entered once
in the first document archive. All other document archives will just
copy them as easily as this is done today with ftp.

The ftp users won't see anything from this new development, because
each document contains also everything a file contains. But a user
of a "document access user agent" will get a very comfortable universe
of documents for browsing.

E.g. a document might contain the following attributes (of course a
user agent would have to present a subset of them in a nicer way):

commonName:              ISODE non-Consortium Release
suggestedFileName:   *   isode80.tar.Z
version:                 8.0
shortDescription:        A wonderful OSI development environment for UNIX
longDescription:         <the text of README>
keywords:                UNIX|OSI|networking|FTAM|QUIPU|PP|...
lastModification:    *   199204311000Z
fileFormat:              <object identifier for *.tar.Z>
author:                  Steven Hardcastle-Kille|Marshall Rose|Collin Robins|...
size:                *   8765432
legalStatus:             <object identifier for "public domain">
uniqueID:                <c=GB, o=ucl, ou=cs, ou=document store, ...>

A ftp user would only see the * attributes.

In addition, this document would have a unique name in the world wide
X.500 naming tree. This unique name could then easily be used e.g.
in a hypertext news article and everyone interested in this document
would just klick a button in the hypertext article and his user agent
would use X.500 to search the nearest copy of the document and would
then automatically download it. (What "nearest" means has to be defined
by the document user agent administrator). ISO standards like SGML/HyTime
offer powerful hypertext referencing mechanisms for text documents.

Last week, I read the latest drafts of the OSI group communication standards
X.gc and X.acc. They use DFR for access to a documents store that belongs
to a news group. So I was wondering, whether DFR might not in general
be a nice protocol for the "document access protocol" I described above.

I hope a reader of this group could post a short description of DFR
and related standards (e.g. what is a DOR?), because I'll have to wait
several weeks on the ISO documents (why are they not available in a
document store??? @!%# ).

Markus :-)

---
Markus Kuhn, Computer Science student -=-=- University of Erlangen, Germany
Internet: mskuhn@immd4.informatik.uni-erlangen.de  |  X.500 entry available
Don't worry over what other people are thinking about you. ----------------
---------- They're too busy worrying over what you are thinking about them.

In article <1992Apr13.130106.8102@informatik.uni-erlangen.de> mskuhn@immd4.informatik.uni-erlangen.de (Markus Kuhn) writes:

>Both FTP and FTAM have far less capabilities and are not the optimal
>choice for a public file archive system.
>
>Can anyone tell me whether the ISO DFR (document filing and retrieval)
>protocol and service (ISO 10166) describes a better alternative?

Yes, DFR is EXACTLY what you are looking for and includes all the features
you describe in your later message. I will take the hint to provide a
summary shortly. To understand it properly, you also need to look at
ISO/IEC 10031, "Distributed-office-applications model" - both "Part 1:
General model" and "Part 2: Distinguished-object-reference and associated
procedures." Make sure you have ordered those as well.

Markus also writes (earlier):

>It would be fine to have a protocol which provides things like
>
>   - free names for files/documents (e.g. with spaces, ISO 10646 characters).

Of course. DfrTitle is defined as CharacterData which means choice of
GraphicString, T61String, PrintableString, GeneralString...

>   - for comfortable browsing attributes like a one-line-description of
>     the file contents

The "basic attributes" available on any DFR implementation include the
following sub-string searchable attributes as well as title and pathname:

Version-Name, External-Location, User-Reference, 
User-Reference-To-Other-Objects.

The "extension attributes" based on those for ODA documents include
many more. In addition, attributes can be added in exactly the same
way as for X.500 Directory.

>   - an attribute that contains a longer description like the README file
>     in many of today's *.tar.Z files. I want to be able to read this
>     attribute before I load the whole package.

Unless you want to make the README searchable on substrings, it might
be better to handle this by forming a "group" that contains both
the full item and the README file separately, with attributes for the
group. Incidentally, "references" like symbolic links can point 
to documents and groups (and search-result-lists) and have their own
attributes for searching so you can have as many "views" as you like. 

>   - attributes like author, version number, keywords, ... in order
>     to make browsing through the archive more comfortable.

All there. As well as arbitrary version titles there is also a DFR
maintained system of next and previous versions which could be used
like RCS and SCCS for version control (and could be implemented using
RCS or shapetool etc - the latter provides a complete "attributed file
system" which I think is rather similar to the DFR concept).

There is also a system for reserving documents to change them (comprehensive
read and write locks).

>   - search operations for keywords, author, names ... (a de luxe archie).

Of course, same arbitrary filter expressions as used in Directory Service
and Message Store. (In fact I would be inclined to use DFR as front end
for, and implementation of, both of those).

>   - distributed search operations

This is where you need to read ISO 10031. ISO 10166-1 just describes DFR as
though it was a single server, rather than a distributed system. Perhaps
that is why there is so little interest - people assume it is just a
fancy filing system. But this is the same as the description of the
X.400 Message Transfer System as a single "object", which only on 
"closer inspection" can be "decomposed" into a system of MTAs relaying
messages to each other. ALL the hooks needed for distributed operations
are there, but they are not explained. Instead there is a footnote (page v):

"...This International Standard governs the interactions of a Document Filing
and Retrieval client and a single Document Filing Retrieval server. Future
standardization will consider the facilities of a Distributed Filing and
Retrieval server system and the need for inter-server protocols and a DFR
administration protocol..."

With the aid of ISO 10031 one can read between the lines to understand
how this will work and the role that strange features like "Distinguished
Object References" and search lists will play in a distributed system.

Result should be more like "Prospero" than archie - i.e. you browse
within your own "home" DFR store organized how you want it, and it
transparently goes and gets information from other DFR stores when
you actually need it - either for an actual content, or to be able
to browse the attributes of a collection of documents.

>   - a well defined unique identifier for each document

They have a very interesting approach. Each document in a DFR store has
a Unique Permanent Identifier (UPI) which is PERMANENTLY unique (like
a Message Store message number, not re-allocated when a document is
deleted). These are not universally unique among other DFR stores and cannot 
be, since two separate copies can be made of the same document, with
different UPIs, even within one DFR store.

BUT, each document also has a path-name, similar to an ftp file pathname,
which can be unique if the DFR is setup that way. Access can be via
either UPI, path-name or "relative-path-name". The latter is just a
working domain for accessing via pathnames that are relative to a
sort of "current directory". This "base" is specified as a UPI.

So, if you have a system of mirror or cache sites, it is only necessary
to coordinate the "base" used by each site for a given sub-tree, and the
same "relative-path-name" accesses would apply. There are cumulative
"search-result-lists" that could easily be used for automatic updating between
archives (including user creation of such mirroring on the fly).

Now a "Distinguished-Object-Reference" (DOR) can be used to quickly
access a document on any DFR (or any other Distributed Office Application
- there will be protocols for Message Stores etc). This includes a
"local reference", (which could be a UPI or a "base" and "relative-path" as 
above), together with details of where to go to use that "local reference".

Now here comes the interesting trick. "Where to go" can of course be
the presentation address of a DFR store (or for that matter a gateway to
any ftp store, with simple pathnames used as "local reference"). It can
also be the Directory NAME of a DFR store or whatever, for independence
of changes in physical location and network address. Finally it can
also be a Directory POINTER to the name. The latter could easily be used
to implement a system that leads you to your local mirror/cache site
that updates the same "base" directory!

Sorry if that's too confusing, but read this again when you have
access to the standards - this is NOT spelt out clearly there either.

>   - all attribute contents should be as much machine readable
>     as possible in order to allow developping comfortable
>     browsing tools.

Same approach as MS and DS.

There has been a lot of discussion in the iafa mailing list and elsewhere
on issues that relate DIRECTLY to DFR, but only passing mention of it as
though a DOR was just another form of document ID to be incorporated. In
fact DFR is a framework within which archie, prospero, WAIS, WWW, ftp
areas etc etc can best be designed to function.

I believe it is also the key to designing adequate X.400 based news
software - (note Markus reference to use of DFR in "group communications").

The Distributed Office Applications architecture is also highly relevant
to OSI access for dialup users, PCs etc. It implies that simple Remote
Operations protocols can be used to gain indirect access to virtually
any "office application" - which basically covers the vast majority
of the sort of requirements people have for internet use. This reduces
the importance of both network and transport layer multiplexing and 
NSAP addresssing and relaying for dialup applications. A local DOA applications
gateway can do far more than a "transport bridge" - and do it transparently.

DOA also answers some of the concerns that have led to proposals for
a simplified OSI stack - but without violating the OSI Reference Model.

Thank you for your very interesting description of some aspects of DFR.

cmf851@huxley.anu.edu.au (Albert Langer) writes:

>The "basic attributes" available on any DFR implementation include the
>following sub-string searchable attributes as well as title and pathname:

>Version-Name, External-Location, User-Reference, 
>User-Reference-To-Other-Objects.

What is the semanitics behind external-location and the references?

>>   - distributed search operations

>Result should be more like "Prospero" than archie - i.e. you browse
>within your own "home" DFR store organized how you want it, and it
>transparently goes and gets information from other DFR stores when
>you actually need it - either for an actual content, or to be able
>to browse the attributes of a collection of documents.

May I compare this mode of operation a little bit with mounting
a file system?

>>   - a well defined unique identifier for each document

>So, if you have a system of mirror or cache sites, it is only necessary
>to coordinate the "base" used by each site for a given sub-tree, and the
>same "relative-path-name" accesses would apply. There are cumulative
>"search-result-lists" that could easily be used for automatic updating between
>archives (including user creation of such mirroring on the fly).

I got the idea between your lines, but I am still confused. Could you please
give a small example?

May a DOR be used to uniquely identify a document world wide (i.e. there
may be sereral copies of THIS document available and all have the same DOR),
and the system will try to get the "nearest location" of a document with this
DOR?

E.g. imagine I have a distributed hypertext system (as implemented on the
Internet with WWW). A link in a text file will show to a document. Is
the DOR the appropriate information that should be stored in the link?
Will it be possible in the ISO HyTime standard (will become final IS this
summer, see comp.text.sgml) to use a DOR as a link to an other remote
(e.g. again HyTime) document?

Together with powerful search methods and perhaps indexes (as planned
for the OSI group communication standard) hypertext will be a very
powerful tool helping you to find the information quickly.

Looking forward to easy browsing facilities through a large universe
of documents ...

Markus

[BTW: if you have never heard of WWW, the WorldWideWeb project, just
get the file /pub/WWWLineMode_v.vv.tar.Z (take the highest v.vv)
from the anonymous ftp site info.cern.ch (128.141.201.74) and install
it on a UNIX machine with Internet access. This is very easy (20 minutes)]

[[The above [...] text is a good example of a hypertext link with DOR.
Unfortunately, it's not very machine readable, so you have to start ftp
manually. ]]

---
Markus Kuhn, Computer Science student -=-=- University of Erlangen, Germany
Internet: mskuhn@immd4.informatik.uni-erlangen.de  |  X.500 entry available
Don't worry over what other people are thinking about you. ----------------
---------- They're too busy worrying over what you are thinking about them.