🚀 go-pugleaf

All,

Please find enclosed the newsletter for April.....

cheers Ed
-- 
 ------------------------------------------------------------------------
| Ed Griffiths, Acedb development, Informatics Group,                    |
|        Wellcome Trust Sanger Institute, Wellcome Trust Genome Campus,  |
|               Hinxton, Cambridge CB10 1SA, UK                          |
|                                                                        |
| email: edgrif@sanger.ac.uk  Tel: +44-1223-494780  Fax: +44-1223-494919 |
 ------------------------------------------------------------------------


----------------------------------------------------------------------------
----------------------------------------------------------------------------

ACEDB User Group Newsletter - April 2003
########################################

If you want to have this newsletter mailed to you _or_ you want to make
comments/suggestions about the format/content then send an email to
acedb@sanger.ac.uk.

-----------------------------------------------

This month sees praise for AceDB from the Worm Advisory board, support for
alternative genetic codes, better searching in the Tree window, a tightening
up of GFF output, using the server with xinetd, some bug fixes and the usual
nerdy stuff for developers.

-----------------------------------------------

General News
************

AceDB powers the worm...or something like that...
+++++++++++++++++++++++++++++++++++++++++++++++++

AceDB received praise and support from the Wormbase Advisory Board report
for 2003 which included the following paragraph:

    SUMMARY REPORT FROM THE WORMBASE ADVISORY BOARD 2003

    2.  The advisory committee wishes to emphasize that WormBase's use of
    ACEDB as an integration and content delivery database is serving the
    project very effectively, and has been an important factor in enabling
    the project to achieve its impressive two-week data updating cycle. We
    make this point because this has been an issue of some controversy in
    the past, with some critics arguing that ACEDB is too ad hoc, or
    insufficiently robust, or cannot meet the performance demands of a
    production system. In point of fact ACEDB's performance on common
    bioinformatically important operations, notably retrieval of complex
    structured objects, is significantly faster than commercial R-DBMSs,
    while the ACEDB data model has become an object of formal study by
    database researchers who see in it valuable ideas for a new kind of
    DBMS (Ref 1). The success of the 2-week updating process shows that
    robustness has not been a problem. Furthermore ACEDB now comprises
    several person-decades of programming investment in
    bioinformatics-specific viewing and analysis capabilities which a
    generic DBMS alone could not replace. In summary, the committee feels
    strongly that the principle of not fixing what ain't broke should
    apply in this case.

Well done to the AceDB developers and the Wormbase group.

-----------------------------------------------

New Features
************

DNA FMap keyboard shortcut
++++++++++++++++++++++++++

Yet another FMap shortcut key, you can use the "d" key to toggle the DNA
display on/off.

AceDB now supports different genetic codes
++++++++++++++++++++++++++++++++++++++++++

(Thanks to Jean Thierry-Mieg mieg@ray.nlm.nih.gov who did the original
coding for this new feature.)

Up until now AceDB has only used the standard genetic code meaning that you
could not do translations of sections of dna that used a different code e.g.
mitochondrial dna. You can now specify your own genetic code for any dna in
AceDB.

Alternative genetic codes can be specified using the new Genetic_code
class:

?Genetic_code   Other_name ?Text
                Translation  UNIQUE     Text
                Start  UNIQUE   Text
                Base1 UNIQUE    Text
                Base2 UNIQUE    Text
                Base3 UNIQUE    Text

Here's how you might include it in your Sequence class:

?Sequence DNA UNIQUE ?DNA UNIQUE Int
                    .
                    .
                    .
          Origin  From_database UNIQUE ?Database UNIQUE Int
                  From_author ?Author XREF Sequence
                              .
                              .
                              .
                  Species ?Species
                  Genetic_code  UNIQUE ?Genetic_code    // specify a different
genetic coding.
                  Method UNIQUE ?Method UNIQUE Float

Here's an example in ace file format:

Genetic_code : "Ascidian Mitochondrial"
Translation
"FFLLSSSSYY**CCWWLLLLPPPPHHQQRRRRIIMMTTTTNNKKSSGGVVVVAAAADDEEGGGG"
Start
"-----------------------------------M----------------------------"
Base1
"TTTTTTTTTTTTTTTTCCCCCCCCCCCCCCCCAAAAAAAAAAAAAAAAGGGGGGGGGGGGGGGG"
Base2
"TTTTCCCCAAAAGGGGTTTTCCCCAAAAGGGGTTTTCCCCAAAAGGGGTTTTCCCCAAAAGGGG"
Base3
"TCAGTCAGTCAGTCAGTCAGTCAGTCAGTCAGTCAGTCAGTCAGTCAGTCAGTCAGTCAGTCAG"

Sequence : "MTCE"
DNA  MTCE
Genetic_code "Ascidian Mitochondrial"

If you displayed this sequence in FMap, then when you used any of the FMap
functions that translate the MTCE's DNA, the translation would be done using
the genetic code specified by the "Ascidian Mitochondrial" object.

If you don't specify the code, then the standard genetic code will be used
(i.e. the code that AceDB has used up until now).

If you use the old Source/Subsequence or the new SMap tags to create a
hierachy of perhaps "transcripts -> genes -> clones -> chromosome" for
your sequence and the sequence requires an alternative genetic code, then to
make biological sense you should specify just one alternative genetic code
for the whole hierachy. It may be sensible however to specify it in more
than one place in the hierachy: you would certainly want to specify it in
the sequence object that contained the actual dna so that you could
sucessfully export that object with all its neighbours from AceDB, but you
should also specify it in the topmost parent of the entire hierachy. This is
because when translating the dna for any subpart of the hierachy, AceDB
looks up through all the successive parents of that subpart for the
alternative genetic code. Hence if you also specify the alternative genetic
code at the highest level then all translations of objects within the
hierachy will be guaranteed to use the correct code.

Improved text search in tree display
++++++++++++++++++++++++++++++++++++

You can now search for just about anything in the tree display window: tags,
text, numbers. It doesn't matter whether the search target is hidden (i.e.
in a collapsed branch of the tree display) or whether you are in "Update"
mode, you can still search.

You can search either by selecting the "Find" button in the tree display
main menu or use the shortcuts:

Ctrl "f","b"             "f" to search forward, "b" to search backwards

Ctrl ",", "."            "," to find the previous occurrence of the target,
                         "." to find the next occurrence.

(For previous you can press either the "," key or the "<" key and for next
you can press the "." key or the ">" key, its just more convenient to press
the "," and "." keys because you don't have to press shift first.)

Note the following:

   * If the target can't be found you get a message.
   * When the search reaches the end of the object you get a message.
   * If next/prev will wrap to the top/bottom of the object on reaching the
     bottom/top.
   * Ctrl "f","b" always bring up the dialogue box allowing you to change
     what you're searching for and always start from the top/bottom of the
     object respectively.
   * If the target is in a collapsed branch of the tree then the search will
     expand the branch so you can see the target. There is now a "Collapse"
     option in the main menu to collapse back the branch.

Please note that the targets must be complete, searching for partial strings
will be introduced next month.

A new "-list" option for the "gif seqfeatures" command.
+++++++++++++++++++++++++++++++++++++++++++++++++++++++

You can use giface to build virtual sequences using the "gif seqget" command
and then dump the features on that virtual sequence in GFF format using the
"gif seqfeatures" command. As alternative you could also just list the types
of feature found in that virtual sequence. A new list option now allows you
to list all the object keys that were used to make that virtual sequence
with its features:

seqfeatures [-list [features | keys]]

             -list features    lists the feature types found in that sequence
(the default)

             -list keys        lists the keys of objects used to construct the
virtual sequence

You can use the "-list keys" function in combination with the existing
"-source" or "-feature" flags to find out which keys of a particular type
were used for a particular stretch of sequence.

-----------------------------------------------

Articles
********

Tightening up GFF output format from AceDB
++++++++++++++++++++++++++++++++++++++++++

GFF files are made up of single line records with the following format:

<seqname> <source> <feature> <start> <end> <score> <strand> <frame> [attributes]
[comments]

Leaving aside the "attributes" and "comments" fields, the rest are in some
senses mandatory although this is not completely clear in the GFF spec. In
the past AceDB has sometimes omitted some of these mandatory fields
altogether which makes parsing of the records more difficult. From now on
AceDB will adopt the following standard for outputting GFF records:

<seqname | "."> <source | "."> <feature> <start> <end> <score | "."> <strand |
"."> <frame | "."> [attributes] [comments]

In summary: the "seqname" to "frame" fields will always be present but
will default to a "." except for the "feature", "start" and "end" fields
which must values must be given. From this release AceDB will output all GFF
files in this format.

Running the AceDB server under the control of xinetd
++++++++++++++++++++++++++++++++++++++++++++++++++++

(Thanks to Jack Chen chenn@cshl.edu of wormbase for finally finding a
working configuration.)

In the past if you wanted the AceDB server to be automatically started when
requests came in to your machine you could do this via inetd, an operating
system utility which provided this service.

xinetd is an open source replacement for inetd, it provides many more
features and much security than inetd, you can read more about it at the
xinetd website. This has replaced inetd as the default service on Linux and
some other Unix versions.

Its configuration is a bit different from inetd and is as follows:

For this example we assume the following:

nickname for server/database -  MyfirstDB
             port for server -  20113
        user owning database -  mieg
       path to server binary -  /usr/local/bin/saceserver
            path to database -  /home/databases/aardvarkDB
   timeout values for server -  200:200:0

In /etc/services add the line:

MyfirstDB  20113/tcp

xinetd keeps its configuration files in the directory /etc/xinetd.d.
Create a file in this directory giving it some meaningful name such as
AceDB and add the following lines:

# file: /etc/xinetd.d/acedb
# default: on
# description: wormbase acedb database
service acedb
{
               protocol                = tcp
               socket_type             = stream
               port                    = 20113
               flags                   = REUSE
               wait                    = yes
               user                    = mieg
               group                   = mieg
               log_on_success          += USERID DURATION
               log_on_failure          += USERID HOST
               server                  = /usr/local/bin/saceserver
               server_args             = /home/databases/aardvarkDB 200:200:0
}

Note that you will need to change the port, user, group, server and
server_args fields for you setup. Lines beginning with a "#" are as usual
comments.

Warning: In the past there have been a number of problems with the way
xinetd worked which prevented users from running the AceDB servers with
xinetd. Currently the following combination of Linux, xinetd and AceDB is
known to work:

Linux: Red Hat Linux release 7.3 (Valhalla), kernal: 2.4.18-27.7.xbigmem

xinetd: xinetd-2.3.7-4.7x

AceDB: 4_9m

If you find that you cannot get AceDB working under xinetd control then you
should first check that you have up to date software.

-----------------------------------------------

Bugs Fixed
**********

DNA dumping
+++++++++++

The DNA dumping command had the wrong default, it should have been for the
spliced DNA unless otherwise specified.

The code also did not cope with when an object contained both a DNA tag
directly referencing DNA and a Source_exons tag specifying the exons to be
spliced out of the DNA.

Both now fixed.

GFF dumping the "reverse" strand
++++++++++++++++++++++++++++++++

You may have noticed occasionally that the GFF dumping function failed
reporting that dumping of the "reverse" sequence had not been implemented.
This usually happens if the sequence has been reverse complemented and then
the user tries to GFF dump. In fact this is a misunderstanding of how GFF
should be dumped in that for any one sequence, the reverse strand features
are always dumped but their strand is "-" instead of "+".

The "gif seqfeatures" command which does the GFF dumping code, when it
detects a reverse complemented sequence, automatically reverse complements
it for dumping and then reverse complements the sequence back. This is
correct for GFF format which is supposed to represent the features on a
known sequence (not its reverse complement) and the strand of features is
represented by the "strand" field.

-----------------------------------------------

Developers Corner
*****************

bIndexGetTag2Key() function
+++++++++++++++++++++++++++

As you will remember the bIndexNNN() functions are a set a of functions that
enable you to search objects for tags and retrieve keys while making the
best possible use of AceDB index to avoid opening the objects wherever
possible.

The bIndexGetTag2Key() function allows you to retrieve the trailing object
key in a tag2 system, i.e. a set of tags in this format: locator_tag
user_tag1 obj_type1_key user_tag2 obj_type2_key etc.

An example of this would be the SMap tags introduced last year into AceDB:
SMap S_Parent UNIQUE <ptag2> UNIQUE <key> // must be just _one_ parent.
<ptag2> UNIQUE <key>

Using the bIndexGetTag2Key() function you can get the key following the
"user_tag" but if the object does not contain the locator_tag then it will
not be opened which makes traversing large hierachies of SMap'd objects much
more efficient.

The function copes with the situation where there are a number of user_tags
following the locator tag.

The bindexNNN() functions are derived from code written by Jean Thierry-Mieg
and include (from wh/bindex.h):

/*
 * These functions find tags etc. making optimal use of the index,
 * only opening objects to get data if strictly necessary.
 */
BOOL bIndexTag(KEY key, KEY tag) ;                          /* Is a tag in an
object ? */
BOOL bIndexGetKey(KEY key, KEY tag, KEY *key_out) ;         /* Retrieve key
following a tag if
                                                               present. */
BOOL bIndexGetTag2Key(KEY key, KEY tag, KEY *key_out) ;     /* Retrieve key
following tag but
                                                               where tag is part
of a tag2 system. */

Getting graph and gex to co-exist for key bindings
++++++++++++++++++++++++++++++++++++++++++++++++++

Some AceDB windows are implemented using purely Graph package code, some
with purely Gex code and some with a mixture (e.g. FMap). There can be a
problem with clashes over key bindings if both layers have uses for the same
keys, e.g. the left and right arrow keys are used by the text entry boxes of
Graph for FMap origin and zone setting but also by the Gex layer for
horizontal scrolling. So what to do ?

The addition of two new functions to the GraphDev interface allows you to
solve this. graphDevDisableKeyboard() and graphDevEnableKeyboard() allow you
to disable/enable Gex layer keyboard code so that the Graph layer can
intercept the events. For FMap it works like this:

  1. Normally the left/right arrows control left/right scrolling of the
     entire FMap
  2. When the user clicks on the zone or origin boxes, Graph disables the
     Gex handling of the arrow keys so that it can use them to allow the
     user to cursor left and right in the text entry boxes.
  3. When the user presses enter to show they have finished, then the Gex
     left/right scrolling is enabled again by Graph.

This allows Graph and Gex to coexist as a crude approximation of what
normally happens in windowing systems.

sMapTreeCoords(): a more general sMapTreeRoot()
+++++++++++++++++++++++++++++++++++++++++++++++

sMapTreeRoot() is a function that allows you to find the coordinates of any
child in an SMap within the ultimate parent of that SMap, while this is
often useful, there are also occasions when its useful to find out the
coordinates of a child in any one of its parents. A new function,
sMapTreeCoords(), allows you to do this (from wh/smap.h):

/* Finds x1,x2 coords of key in target_parent, which must be a parent of
 * of key in keys smap tree, if target_parent is set to KEY_UNDEFINED it
 * does the same as sMapTreeRoot(). */
BOOL sMapTreeCoords(KEY key, int x1, int x2, KEY *target_parent, int *y1, int
*y2) ;

As can be seen from the description, sMapTreeRoot() is now simply a call to
sMapTreeCoords().

Getting the object from which an FMap seg was derived
+++++++++++++++++++++++++++++++++++++++++++++++++++++

An FMap seg mostly represents a single box drawn on the FMap screen but
going from this seg to the object from the seg was principally derived is
not completely straightforward, in fact the FMap code has a number of
different ways of doing this. The fmapSeg2SourceName() function takes a seg
and returns the name of the object from which the seg was principally
derived. Sometimes this is seg->key, sometimes seg->parent and sometimes
seg->source depending on the seg type.

-----------------------------------------------

April monthly build now available.
**********************************

You can pick up the monthly builds from:

Sanger users
     ~acedb/RELEASE.DEVELOPMENT
External users
     http://www.acedb.org/Software/Downloads/monthly.shtml

-----------------------------------------------

Next User Group Meeting - D319, 3.00pm, Thurs, 8th May 2003
***********************************************************

----------------------------------------------------------------------------
----------------------------------------------------------------------------
Ed Griffiths <edgrif@sanger.ac.uk>
Last modified: Fri May 2 09:55:53 BST 2003