🚀 go-pugleaf

By popular request, here are some very rough outline notes I took at the
ACEDB Workshop in Boston last month. I cannot vouch for their accuracy, but
hope I have not misquoted anyone too badly. Please post corrections and
clarifications to this newsgroup.

The full set of notes are about 20 pages long. I have broken them into two
parts to accomodate my news reader and mail software.

If you have questions about particular points, you probably should send
them directly to the people who made those points at the workshop.  A list
of workshop attendees with Email addresses was posted to this newsgroup a
few weeks ago.

A copy of this document is available via anonymous ftp
on weeds.mgh.harvard.edu  under pub/acedb-dev/doc

-John McCarthy, Lawrence Berkeley Laboratory


==================================================================
ACEDB Workshop  Wed 28-31 July 1993 Boston, MA 
Highlight notes from John McCarthy (LBL) jlmccarthy@lbl.gov 

more or less in chronological order, with TOPICS IN CAPS
see list of attendees for full names, addresses, etc.

Wed 28 July 
=================================
COMMUNICATIONS, GOPHER
Mike Cherry is providing ACEDB archive on his gopher server at
Stanford -- genome.stanford.edu.

Mike asks if people could please send notes to bionet.software.acedb
another thing that would be nice to have is FAQ - Brad will do this

weeds.mgh.harvard.edu has anonymous gopher+  login/server
Also experimenting with remote anonymous AATDB login aatdb
user name c-aatdb used for MacX
Mike used Richard's acedump to create gopher files.
WAIS paragraph format / from acedump    also textace

Brad notes that his gopher archive includes USDA sysad archive.

Brad Sherman did research for secure anonymous ftp

Mike: weeds is a separate machine, not cross mounted to anything, and
not used for anything else

stty script --  Otto points out that you can trap interupts with Bourne
shell
Brad points out that most vulnerable spot is within application.

anonymous FTP setup instructions,  
thor.ece.uc.edu
/pub/sun-faq/anon-ftp.script
guidelines:  . cert.org   /pub/tech.tips/anonymous.ftp

=========================================================
PRINTING FOR REMOTE USERS
richard: set ACEDB_LPR to null -- code checks for this.
maybe could do a script that turns print script into Email.
Michelle: also may want to print to a file  
(richard:  currently always goes into /PS directory).  
Stan: might be nice to have switch to disable all writing
could do changeroot to keep people restricted to specified area.

=================================
ACEDB NEW DEVELOPMENTS - Richard Durbin: 
ORGANIZATIONAL STATUS, what Richard and Jean are doing:
RD  -worm sequencing (2,5,10 Mb/yr)
      -sanger Centre  (RD joint head of informatics)
      -CGC map -> Cambridge  (worm cytogenetics center)
            Mary O'Callaghan doing this (moc@mrc-lmb.cam.ac.uk)
      -in house use: sequence (worm, yeast, human)
                             worm genetic map
   so RD will have less time for kernal development

Jean:  -Human mapping - Frensal (Genethon)
          -IGD
          -ports to various unix systems
          - management of kernal  (want to keep unified version)
             incorporate changes from others & redistribute
             also would like to make other's tools & modules available

RELEASES:
        1-10  now  (still backwards compatible with old databases)
        2-0 September? (model changes for maps, sequences)
                             running this in Cambridge now
                      but want to make other changes before release
              would like to get some feedback from this workshop first
     developers can get 2-0 code but should not use for databases
          that will be distributed, etc.
Will have AWK scripts for database restructuring from 1-10 to 2-0
Extra things in specifications directories, PS fonts, Xfonts.wrm,


NEW ARCHITECTURE FOR MAP DISPLAY-- RD
  columnar architecture/format will help
  would also be nice to have developers style guide

Stan: would be nice to have another level of indirection, so people
could map different names to std display code names -- intermediate
layer to make it easier to change models and yet maintain
compatibility with display modules.

RD: not convinced that another layer of indirection is good idea
may slow things down

RD: people should not expect that all the good ideas that get
suggested here will get implemented.  Commitment to maintain kernal.

------------------- break-----------------------
NEW DEVELOPMENTS (mostly in 1-10)
SUBCLASSES 
  -can appear on main window
  -can use in queries, .ace files
  -no code change required
specification
  -wspec/subclasses.wrm
  -read after models.wrm
  -ace file
      class genome_Sequence
      Subclass-of Sequence  //can be a subclass  (can have 2 parents?)
      Visible            //indicates whether displayed on main window
      Filter     "Genomic_canonical"  query
  How it works:
       Have 8 bits free per key in memory 
       -uses 1 bit per filter
       checks on object save
       when filter changes -- rescans entire class with query
       can't change entries in class class directly, reads from
subclasses.wrm
        
May be possible in future:
      to have different default display type for different subclasses
      
Used in 2.0
?Map   -  Chromosome
?Locus - gene, RFLP, etc.
?Interval - rearrangement, Chrom_Band, Hybrid cell line, ...
?Df_dup_defic     Inside
                           Outside
?Probe

2-0 has a models class 
so won't have ?class as part of keyset
you will get this type of info in other ways

2 possible ways to do sub-maps
  1 scale with some linear transformation
  2 interpolate between shared loci (now done in 1-10 physical map)

Otto: IGD has used clone grid as general model for 2-D maps

RD: map display now automatically transforms contig info in this way
 maybe could do some graphic to indicate where there are conflicts
  between different versions of maps

GRAPH PACKAGE additions
   filled arcs
   filled polygons (from Suzanna)

IMAGES (grey level: 256-->128)
      (a) in subwindow    graphPixels()
      (b) PIXEL_SCROLL graph type   (coordinates are pixels) 
            graph Pixel Background()

Possible Development
      -ASCII dump format for draw stack
      -stand alone drawing editor
 full color images will require more thought!
now easier to attach sub-menus to boxes

Otto: can pick up X colors and coordinates to use from XV
Suzanna: has done some things to display TIFF images

MAP PACKAGE
map.h     w7/mapcontrol.c
A. drawing coordinates  (zoom, centre, scroll, etc.)
B. columns of data (user toggleable)
C Cursor

MAP  *map
Look -> map -> mapCreate(column, ...         )
{float   min, max
Coordinate Functions
Locator (small green bar)

Stan: is this the way you would draw multiple (synteny) maps?
RD: not yet..., in the meantime can use TableMaker multi-map

Planned Enhancements
1. add a help line per column for display control
2. allow >32 columns
3. give names to sets of columns
4  ?   Multiple ?MAP's per window

Code:
     separate columns further
             ==>library of columns
     separate SEG's array per column    (convert (store),  destroy)
   put PICK control in MAP package
          -select   /  follow
          - neighbor       on/off  (key)  [e.g., to light up related
objects]

Sequence Fetch/Align
--------------------------
a. summon up SwissProt/PIR entry(s)
b. show multiple alignment based on BLAST segments  =BLIXEM
uses unix command
   fetch  -Bdbname (SP,PIR,EMBL,GB)
             -a accession number
             -P Pearson (FASTA) format
             -s  just  sequence ... no new line
   our version uses the EMBL CDROM index structure

EXTERNAL COMMANDS
general idea of invoking external commands is now part of ACEDB
external Command( ) executes any unix command and puts output in text
window (can't do anything further with it at the moment)
[this would be even more useful if one could cut-n-paste window]
Otto: IGD stores entire EMBL entry as ?LongText

=================================
LBL ENHANCEMENTS (John McCarthy)
Query by Example, Query Builder, etc.
now working to integrate with Table-Maker

ENHANCEMENTS FOR FlyDB -- Suzanna Lewis
image display for polytene chromosome
data entry enhancements
STS contig assembly

RD: there was a capability in horizontal map display to work on sets
SL: lab dataflow
     need tools to massage data from Excel to .ace files
          translation
          validation
          composition  (composes string that goes into a single tag)
yet another contig assembly program - uses P-Q trees
      biologists may want to look at P-Q trees themselves
RD: saw Carrie Sutherland's package-might you use it?
SL: may be an attractive solution, except would need dif display
Stan: what about Pecherer tool?  SL: that's a bit different, before
ordering

=================================
MacACE- Frank Eeckman 
would like to make it available in a public domain way
RD: problem is whether we can provide high level of support,
documentation, etc. up to usual Mac standards private vendor might
help support that, but might constrain and even begin to be the tail
that wags the Unix version tail.
Program itself is only 500k for version 6-0.
Older SCSI I is fairly slow, but new SCSI is pretty fast for data.
RD: another issue is distribution. Unix is easily distributable via FTP.
Could do via binhex. 
NCBI has offered to distribute MacACE on its CD.
RD: maybe could do distribution every 6 months of ACEDB, acedb, aatdb,
IGD, etc.
RD: Sun and SGI are binary data compatible
[NOTE: subsequent to the workshop, LBL's Human and Plant Genome
projects have provided some supplemental support to help Frank Eeckman
get a production version of MacACE ready for distribution along with
ACEDB version 2.0; he will be spending the last half of September in
Cambridge working with Richard on this project]

=================================
IGD ENHANCEMENTS -- Otto Ritter had to introduce dummy locus to force
3 maps to same scale to show male, female, avg cm map from GDB.

IGD invokes external program to display postscript images (e.g.,
vector diagrams)

IGD future plans:
integration via dif dimensions
   static objects and processing info
   public and private data
  data objects and analysis methods
  data and knowledge about data (PROLOG interface-Hans Lerach)

more portability
WAIS and gopher servers

use OPM to define data models
FTP and Email communication
more data  (Martin Bishop -- cDNA, non-human, etc.)
pedigree data, displays, etc. (Steve Bryant, ICRF Clare Hall)

========================================================
METABOLIC PATHWAYS MODULE - Stan Letovsky

RD: Bump package may help solve layout problem

even more challenges as paths cross species, tissue-type,
developmental stage, etc.

includes some diagram routines that may be of general use
(though not all that well isolated as a package]

===========Thurs July 29=================================
ACEDB FUTURE -- Richard

CONTACTING EXTERNAL PROGRAMS
Pickme_to_call  <a>  <b>   calls system("a $ACEDB/external/b")

Proposed:
?External
     Command  UNIQUE X_prograpm   #Command
                                   Text_display   #Command
                                   Ace_parse       #Command
     Remark  ?Text

?Command U  Basic  U Text
                      External U Text U Text
                      From  U Text
?External has default display method to execute the appropriate
command [display (key, from) from is the object its linked to]

Brad & Otto: would be nice to be able to apply selected external
command to any arbitrary keyset. Could make this a menu option on the
keyset menu, which would call up graphselect to choose an external
procedure instance.

====================================
CLIENT-SERVER
tace command line version of ACEDB can run in server/client mode,
linked by RPC

Server                                               Client
           <--------------query

                                 acefile----------->
                                                          keyset
            <-------------expand

                                 acefile------------>
                                                          objects

Suzanna: now all you get back from server is formatted text strings
   rather than .ace files  and client does not build up its own database

Client and server must share classes, models, and tags

does not include any password protection, etc.

Client accumulates a local name table and database

X client later this year?

NB: can embed in scripts

should data persist on client? 
only problem is that it might grow too large

Question: how could client combine local data with that from server?

Otto: wants 2 modes-asynchronous and synchronous modes
  in synchronous mode, user could decide whether to save
  also would be good to have command to remove/delete keyset

currently will die if you overflow the cache and don't have write access.  

Otto: see Query buttons/menu - server search is synchronous

aceclient hostipaddress [-timeout secs] [-aceout] [-f scriptfile]

====================================
DATE STAMPING (COARSE GRAINED CONCURRENCY)
-sessions
-all objects are session stamped if non-empty

Proposal 1
-function to create a keyset of all objects changed since a previous
session plus all empty objects
-dump the keyset in new and old session and acediff

Proposal 2 (could be used for finer grain time-stamping)
 -make all addions by a pseudo command pointing to the session
       (ala comments, which can even be added between tag & value)
 -toggle option in tree display could show session # optionally
-?Deletions

Otto suggests that should be optional part of model whether or not to
include session stamp; Richard thinks it won't be bad -- under 2%

Dave Matthews: what kinds of information can be added to sess?
RD: session is a system object that cannot be edited, but can create a
parallel class that would be related and could contain comments

Logfile contains messages, names of files read in, etc.

at present, can use acediff  old.ace  new.ace > difface  (class at a time)

the way it works...
session object contains list of objects created or modified in sess

====================================
TEXT/?TEXT
Text is stored locally, like Int, Float
?Text is a class
      a key is stored locally
      the name of the corresponding object is the text. The object
contains a back pointer. This allows rapid searching of the names

Cost: 32 bytes/object, no garbage collection
Proposal: special mechanism just with forward and back pointers

====================================
CACHES:
cache2: B classes (trees)
#objects
cache1: copies of disk blocks exactly as on disk, and A classes
  A classes include DNA, keyset, ?Longtext and hidden for drawing
#blocks

cache2 has to be as large as # of objects you are updating at a time
          normally 1000 or so
cache1 has to be as big as the largest object you ever want to use
  (array of names for each class -- 
    lex (dictionary) subclass flags, ptrs
   voc - continuous stream of all the names
Otto's ?Text voc is 10Mbytes     acedb 
Symptom: crashes and says "cache is full..."

experience is that database runs in about 10% of its storage size.

====================================
TAG COMPILATION DEPENDENCE

Brad: why not separate out tags needed for compilation?
others could be made as needed 
or could do a utility that reads models.wrm and creates tags.wrm
(could be part of syntax checker and metadata creator)

====================================
SEARCHTEXT and ?LONGTEXT
Otto: IGD has OMIM in ?Longtext -- its great!
you can have a keyset and define certain classes as ?SearchText and
?Longtext will highlight them in context.

==========================================
SEQUENCES
?Sequence (B)  and ?DNA (A)
needs to be same name in both classes
any sequence can have parent and/or daughter sequences
   overlapping or non-contiguous

                                              200
  Y1                            ---------+------>
  Y2                  -------->              <-------------------
  Y3                  --------------------------------------
  X                  
                        1       800                                       
2000

if you want one to have priority, set Genomic_canonical

Other situation (see handwritten notes)

Mike Cherry:  how aatdb loads info from GenBank
script to parse GenBank flat files in GCG format and map to aatdb
written in C -- probably would be better in PERL
They have to do arithmetic for subsequences (ala RD explanation)

Mike: next do BLAST-X, non-redundant search, another script that reads
back into .ace files

REFERENCES
When you read in references, make up id that has 5 char 1st auth,
year, ____; best way to find duplicates is pages (Sam)
Otto: within EMBL and SWISSPROT, had seq # for id, then table with
author, year, title, etc -- thinks aatdb solution is much better

SCIENTIFIC NOTATION
Gregg: ACEDB is not printing very small Float numbers
RD: stored ok, but needs to be one line change in tree display to
display in scientific notation if too small for decimal representation

==========lunch=============
PHYSICAL MAPPING
Sam:
if you want to move into 3rd region, set boolean Band
can stack up to 10 cosmids -- curator has control
things at top only are linked by hybridizing to cosmids (YACs)
cosmids are given position within contig

below green line are General_remarks (highlights associated cosmid)
  Y remark (hidden remarks) - only shows if menu item is toggled
stripped out most other classes except clone to use as lab notebook
formerly used Excel- quite unhelpful
physical map display is better, lets user explore the data
use in conjunction with Table-Maker

e.g., to see which contigs hybridize with any given clone also using
clone grid for hybridization data, but not in conjunction with the
physical map also using contig editor (descendent of Contig9)
     can bring in new clones, vary size, attach comments, etc.
     would like editor to make more direct use of mouse, rather than
specifying length of cosmids, etc.
    at present not a way to join contigs easily RD: 1-10 release
pretty much the same, but Neale now has additional functions that help
the editing process (now in test phase at Sanger)
  it does correct calculations if you have fingerprint data
  several people expressed interest in getting new version
Otto: would be nice to have a scale (tag in contig model)
RD: there is a scale in the editing form, so should be easy
Otto: if you could position loci and comments on scale, then you could
put a restriction map below
RD: have another approach to restriction maps
Otto: would be nice to make it a recursive so you could show maps within
maps
RD: would be nice to have menu item for "next contig left [right]" to
get to other contigs connected via genetic map
RD: how it works...  
probably will keep this module with editing functions
  (units in this map are integers because based on fingerprint data)
in the future, use uMap to show physical maps more generally, with
floating point est Kb, etc.

RESTRICTION MAPS
For restriction maps, you want to be able to put sites on consensus
maps as well as on individual clones.  So idea for the model is...
?Clone
     RestrictionSite  ?Restriction Float  //site from start point in coord
same tag setup for ?Map, with
    Clone  ?Clone  Float
need to do this for David Bentley et al -- human X

also may want to have 
?Probe     with subclass STS, clone, 
      ______   ?Clone
      ______    ?Clone
Data   Positive_clone  ?Clone
          Negative_clone ?Clone
          Positive_pool  ?Clone
          Negative_pool ?Clone

also can relate to ?Pool, and even to virtual pools (all pools that
have ever been hybridized to it) also pool is a type of probe that has
subprobes
Suzanna: also sometimes pool the clones you are testing against
   RD: so also need a pool of targets...  probably unwise to make the
clone class recursive, so maybe need targetp
__:will this generalize to more complicated pooling schemes?
RD: it should work...
Otto: why not call it "reagent"  for probes, restriction sites, etc.
 probe can be subclass of reagent
  GDB has reagent as mechanism by which locus is placed on a map

Suzanna: describes flybase models
uses in situ info to localize to chromosome and find inconsistencies
likes idea of using clone grid to show what should and should not hit
Two-tiered pooling scheme allows prediction of what row/col should be
===============
MODELS
JMc: distinctions between lists, sets, and trees
Jean invites everyone to submit their models to common ftp site
for him and others to review, make suggestions, etc.

Lisa: would really like to have XREF's within #constructor
RD: problem is that back XREF would not know where to go if you have
more than one # in the same class Another problem is that you can't
XREF back because you'd have to know the location (the preceding
dependent data field)
Frank: would be nice to have a diagramatic way to represent
connections, and make it impossible to do unpermitted connects

problems that can occur...
RD: e.g., 3-point data
    Gene1 ?Gene XREF 3pt
    Gene2 ?Gene XREF 3pt
    Gene3 ?Gene XREF 3pt

e.g.   Gene1 unc-5
        Gene2 tra-1

then they decided they needed to change the order.
edited by deleting unc-5 and changing it to tra-1, then del tra-1
that lost a cross reference
moral is that its unwise to have this structure
==============MODELS (continued)=======================
Sam: what to do when you think you have a problem
a. send question to acedb newsgroup
b. try to describe as fully and precisely as possible
1. try to reproduce it in a stripped down database to isolate
2. try to eliminate as many other variables as possible
3. then can send a small sample dataset
4. provide login name for Richard and Jean to run local

Elizabeth: would like to be able to specify ordering of set items
RD: ACEDB will preserve order in which set was created
so author order will be preserved if read as part of paper
problem was back XREF from ?Author XREF back to paper
RD: used to do alphabetic order on sets 
     might be good idea to permit different sort orders

Elizabeth: would also like to have private tags for local comments
RD: could have 2 versions of database -- one gets screened via filter
that drops certain tags
Otto: could you have 2 wspecs pointing at same blocks.wrm? NO
    could specify display separate from model set vs list
(Frank and John also like this idea)
   Richard does not like this -- good to have user visual cue as to
what underlying logical structure is
JMc: could have active outline put hidden stuff collapsed at bottom
Lisa: also would like to attach reference to most every tag, handle
like comment...
RD: would be possible to have extra comment classes...
RD: another solution...
?Paper
    Corresponding_comment ?Comment

and add Corresponding_paper in the ?Comment class

?Used_in  Strain1
                Strain2

=====================================================
SPECIAL CHARACTERS
Dave M: users want Greek, super and subscripts, italics
could use SGML, TeX, or other escape sequences
Otto: might be good to include fonts or pointer to where to get them
we should look at what X would need to do Greek, etc.
RD: might be able to add a few things to ACEDB fontset for super and
subscripts.

=====================================================
ACEDB INTERACTION WITH OTHER DATABASES
Mike: going to use Genome Topographer for Yeast DB at Stanford
GT supposed to be multi-species db
costs money for underlying DBMS - GemStone
as a test, going to load aatdb into Genome Topographer
    going to teach GT to read and write .ace files
    this plus client-server might lead to .ace client for GT
Tom Marr (& Jackie, Elizabeth, Steve)
SmallTalk80 (ParcPlace), Servio Corp -- GemStone
data from GDB, yeast, Hood, Olson, Cytogen(Grey), Donis-Keller's
mapping group, Blanchard, etc.
Bob Robbins supposed to be trying to get site license for DoE

BUT, Mike does not like current GT interface as much as ACEDB, so will
be trying to link GT to ACEDB. ACEDB is easier to use.

Leland Ellis: 
-Filamentous Fungi IBT, Tamu, Baylor
   Aspergillus nidulous
   ......
   Neurospora crassa

-Bovine: TAmu
   Jim Womack  (cf Human)

-Crops: TAmu  Sorghum vs Maize, others

all of these need distributed, comparative support, rich GUI (b&w)
GT is quite different from ACEDB in its complexity
  700 classes and 40k methods to manipulate them!
  GemStone coded in OPAL
  
how to get around all this ...
-Mosaic  NCSA:  ftp.ncsa.uiuc.edu
-HTML  html.mode.el --> Emacs
-httpd  server  http://Keck.tamu.edu/ibt.htl  (universal resource locator
URL)
-runs under X, nice GUI, Access to WWW, Gopher, ftp,news,file

LE is converting GT 150page doc into html hypertext document
WWW: comp.infosystems.www, alt.___.www

Doug Bigwood: NAL PGD goals to bring together data, comparative
have been working in SyBase, APT -- painful development & use
(same as GDB)
ultimately want to have graphical user interface
DB went to Madison 2 years ago & looked at ACEDB, WCS, Jean
have talked to Otto about how to put ACEDB on top of SyBase.



-- 
+=====================================================================+
|  John L. McCarthy.               . | Internet:..JLMcCarthy@lbl.gov  |
|  Computer Science R&D  MS 50B-3238 | Bitnet: ...JLMCCARTHY@LBL      |
|  Lawrence Berkeley Laboratory     .| telephone: (510) 486-5307      |
|  1 Cyclotron Road                . |                                |
|  BERKELEY, CA 94720, U.S.A.      . | FAX:       (510) 486-4004      |
+=====================================================================+