🚀 go-pugleaf

All,

Please find below the June newsletter, sorry for the slight delay, I have been
away for a couple of weeks.

Ed
-- 
 ------------------------------------------------------------------------
| Ed Griffiths, Acedb development, Informatics Group,                    |
|        Wellcome Trust Sanger Institute, Wellcome Trust Genome Campus,  |
|               Hinxton, Cambridge CB10 1SA, UK                          |
|                                                                        |
| email: edgrif@sanger.ac.uk  Tel: +44-1223-494780  Fax: +44-1223-494919 |
 ------------------------------------------------------------------------


----------------------------------------------------------------------------
----------------------------------------------------------------------------

ACEDB User Group Newsletter - June 2003
#######################################

If you want to have this newsletter mailed to you _or_ you want to make
comments/suggestions about the format/content then send an email to
acedb@sanger.ac.uk.

-----------------------------------------------

The June release includes the first extension to the Query language for some
time, a new programmer's interface in C, Ace-C, reminiscent of AcePerl, but
such that the same subroutines can be used in distant clients, as stand
alone executables or embedded in tace/xace, an enhancement to the
xace/xremote interface, tips for models file editting, bug fixes for gene
making and acediff and a speed fix for the server.

-----------------------------------------------

General News
************

Jean Thierry-Mieg mieg@ncbi.nlm.nih.gov reports:

Very good news: Jean mieg@ncbi.nlm.nih.gov and Richard rd@sanger.ac.uk have
decided that they will from now on try to contribute regularly to this
newsletter, and encourage developers to use it as a forum to discuss planned
or desirable enhancements of Acedb.

Very sad news: Mark Sienkiewicz, after one year of excellent work on acedb
at NCBI, has left. In this short time, in collaboration with Jean, he has
made several lasting contributions to acedb. He wrote the support for
arbitrary genetic code, recursively hooked to the closest parent sequence.
This code is compatible with the NCBI recognized set of genetic codes, and
is therefore usable, modulo a simple declaration, for all organisms and
organelles. It is currently used in WormBase to translate the mitochondrial
genes. Mark then worked on tricky kernel issues, affecting the very large
acedb database that Jean uses at NCBI to support the annotation of the
complete human genome (4 million sequences, 16 million objects). He fixed in
particular a rare disk runaway and accelerated the client/server TCP
protocol. But his most important realization is his contribution to the
design and implementation of the new Ace-C programmer's interface described
below. "He went over to the dark side, but let us hope for the return of the
Jedi".

-----------------------------------------------

New Features
************

Expanding Keysets in the ACEDB query language
+++++++++++++++++++++++++++++++++++++++++++++

(This article is courtesy of Dan Lawson dl1@sanger.ac.uk (who originally
requested this new feature) and Jean Thierry-Mieg mieg@ncbi.nlm.nih.gov (who
coded the new feature))

ACEDB handles sets of objects called Keysets (you have all seen the xace
window termed 'Main KeySet' or the tace line '// 13 Active Objects'). We use
Keysets all the time during an ACEDB session, as transient lists of objects
of interest, and it is possible to save them for later retrieval/use, using
the 'save as' button in the 'Select/Modif' menu of the keyset window. They
then belong to the "Keyset" class and can be retrieved in xace from the main
window, or using the query: find keyset toto.

The problem was to get the objects contained in the keyset. In graphic mode
(xace), you click on the Keyset object and it expands in a new keyset
window, which you can query, intersect, dump etc. In text mode (tace), you
could only so far get inside the keyset using the tace command 'follow'.
However, a major limitation of the tace behaviour was that this command was
not part of the query language, so it could not be used in complex queries
or from the WormBase interface.

The new operator, 'expand', fixes this problem in the way you expect. Let us
consider a biological example. In each release of WormBase, there is a
canned keyset called CDS_with_RNAi. You can now say, from the command line:

acedb> query find Keyset CDS_with_RNAi; expand; where From_laboratory = HX;

// Found 8425 objects
// 8425 Active Objects
acedb>

This is especially useful when you use the WormBase single line query search
box:

find Keyset CDS_with_RNAi; expand; From_laboratory = HX;

(Note that you do not need to specify the leading "query" keyword when using
the Wormbase query dialog.)

This query returns those members of the CDS_with_RNAi keyset which were
predicted by the Sanger Institute.

In addition, since the expand operator is now part of the standard acedb
query language, it can be used in more complex queries as follows:

Expand: query language operator.
Expand takes no parameter.
It opens the keysets present in the active set and returns as the new
active set the union of all the objects that they contain.

examples: (at the tace prompt)

acedb> query find author; expand
returns an empty set, because authors are not keysets, but plain objects

acedb> query find keyset CDS_with_RNAi; expand
returns as the active keyset all objects contained in CDS_with_RNAi

acedb> query find keyset C* ; expand ; IS > b10 AND IS <= b100
returns all members of all keysets called C*; then filters those on
their names. Recall that, in acedb, b8 < b10 < b20 < b22a < b100.
So you will get objects from b11* to b100, possibly including b22a.

acedb> query find keyset ; COUNT { expand } > 12 ; expand
returns a keyset of all keysets containing at least 12 objects, then
expands just those

Getting the xace main window id for use with xremote
++++++++++++++++++++++++++++++++++++++++++++++++++++

To interact with a running xace program via xremote you need to know the
window id of the xace main window. This is now provided along with other
useful information in the "readlock" file for the running xace process.

Readlock files are held in the database subdirectory
"<db>/database/readlocks" and have a file name of the format:

      <session>.<host>.<pid>

e.g.      5.grifbo.457076

Typical contents of a readlock are:

Readlock file to prevent destruction of sessions still in use by other processes
Created: 2003-07-02_13:45:46
User: edgrif
Program: xace
Version: ACEDB 4_9t
Linkdate: compiled on: Jul  1 2003 14:00:08
WindowID: 0x6c00028

Note the window id as the final entry.

Ace-C: a new C interface to acedb
+++++++++++++++++++++++++++++++++

(This article is courtesy of Jean Thierry-Mieg mieg@ncbi.nlm.nih.gov)

Mark Sienkiewicz and Jean Thierry-Mieg designed a new C interface to acedb.
It is analog to AcePerl, but programmed as a C library. Like Janus, it has a
double face. The same Ace-C code can run as a subroutine in tace, or as a
distant client talking to the acedb server.

This is in fact the seventh interface developed for acedb, but we hope that,
together with AcePerl, Ace-C will last. The problem at stake is to use acedb
to support a Web site. We first had the Moulon server of the enigmatic Guy
Decoux which was a clever hack rather than a programmer's interface. Then we
had the -p style of Doug Bigwood et al. which exported full fledged Perl
objects, but was hard to use. Then Lincoln and Jean developed the -j style
to support first Jade, a Java universal interface to acedb, which died as a
victim of the Java war, then AcePerl, the clear success we are all familiar
with. In the meantime, another java interface, javace, was developed, but it
is not universal and exports only a particular acedb schema into a
particular biojava schema. There were also two C interfaces, one called
acelib, which was designed collectively at the Cornell acedb meeting but was
incomplete and clumsy, and a second one designed at NCBI, which was too
complex and never released. By the way, we would like to remove acelib, so
if anyone is using it, which I doubt, please let us know, we can help port
the code to the new Ace-C.

Why so many attempts ? Well, clearly it is a hard problem, but each time we
benefit from the previous tries. Why not just use AcePerl ? Because AcePerl
modules can only be used as distant slow clients, and cannot be embedded as
fast subroutines to develop new tools inside acedb. In fact AcePerl would
benefit from being rewritten above the new Ace-C, which is better optimized
than the current AcePerl communication layer and which offers new
functionalities.

The real nice thing is that the Ace-C programmer's interface is defined in a
single include file, wac/ac.h, and you may choose to link your program
either with the client Ace-C lib, called libaccl.a or with the standalone
Ace-C lib called libacs.a.

In the client case, data are exchanged with the server using the new 'show
-C' mode, born from ideas discussed years ago with Richard and Doug Bigwood.
In the standalone case, the Ace-C function calls are just wrappers for
standard acedb kernel calls, which means that Ace-C runs with a negligible
overhead, and can run standalone or as a subroutine imbedded in tace/xace.
The interface is powerful, streamlined, unambiguous and well defined, and
should be easy to learn and use.

Example (extracted from in wacext/makefile.acc):

kscount: kscount.o  $(LINK_ACC)
$(LINKER) -o $@ kscount.o $(LINK_ACC) $(LIBS)

skscount: kscount.o  $(LINK_ACS)
$(LINKER) -o $@ kscount.o $(LINK_ACS) $(LIBS)

kscount s:host:12345:anonymous   # runs as a client
skscount $ACEDB                  # runs standalone

Ace-C is now used in production underneath the NCBI AceView website
http://www.humangenes.org All the source code has been put in the CVS, the
library is well documented in wac/ac.h, although there is no user manual,
and there is a built in regression suite. However, the rarely used functions
may still need some tuning and we would appreciate feedback.

If you are interested, please use the acedb daily build and mail
mieg@ncbi.nlm.nih.gov)

Named Keysets
+++++++++++++

(This article is courtesy of Jean Thierry-Mieg mieg@ncbi.nlm.nih.gov)

As part of the support for Ace-C, we introduced 3 new commands to the acedb
command language, available from tace and aceclient.

kstore name  : store active keyset in internal tmp space as name
kget name  : copy named set as active keyset
kclear name  : clear named set

The name space and all the named keyset are per client and released when the
client quits.

Example: Build a Wen diagram

   query ......... // a very slow query
kstore L1
query ......... // a very slow query
kstore L2
....
kget L1
spush
kget L2
sand
spop
count  // counts the objects in the intersect

I remember that Lincoln had for years been asking for this functionality.

-----------------------------------------------

Articles
********

Editting wspec/models.wrm
+++++++++++++++++++++++++

A couple of times recently users have sent emails with questions about the
models file and how it should be edited. Here are tips/guidelines/rules.

The spatial layout of the models.wrm file is crucial as it describes the
structure of objects in the database. Here is a crude example:

 Class in models.wrm               Object in database

?Name   First_name Text        Ed---->First_name--->"Edward"
Surname Text                      |
V
Surname---->"Griffiths"

If you misaligned the class definition as below acedb would complain and
refuse to parse the models file:

?Name   First_name Text
Surname Text

This is fine in a way because you would quickly find out about the error and
nothing will be corrupted. Mistakes can however be less obvious if you have
a deeply nested object and you have used Tabs to space out your class. The
accidental deletion of a Tab can change the whole branching structure of
your class:

Original class:

?Map    No_cache // Don't cache segs for this map.
Display Non_graphic  // Prevents a graphic display!
Title UNIQUE ?Text
Flipped // Then coordinates go upwards
Unit UNIQUE Text
// e.g. kb, centiMorgan, MegaParsec
Centre UNIQUE Float UNIQUE Float
// default centre, width - else 0, 10
Extent UNIQUE Float UNIQUE Float
// min, max - else min, max gene/locus
Default_view UNIQUE ?View
Minimal_view UNIQUE ?View // use this when >1 map displayed
View ?View                // Columns to display
Inherits  From_map UNIQUE ?Map  // To locally edit
Author Text     // login name of who created it

Class after accidental deletion of Tab preceding "Centre":

?Map    No_cache // Don't cache segs for this map.
Display Non_graphic  // Prevents a graphic display!
Title UNIQUE ?Text
Flipped // Then coordinates go upwards
Unit UNIQUE Text
// e.g. kb, centiMorgan, MegaParsec
Centre UNIQUE Float UNIQUE Float
// default centre, width - else 0, 10
Extent UNIQUE Float UNIQUE Float
// min, max - else min, max gene/locus
Default_view UNIQUE ?View
Minimal_view UNIQUE ?View // use this when >1 map displayed
View ?View                // Columns to display
Inherits  From_map UNIQUE ?Map  // To locally edit
Author Text     // login name of who created it

This one character deletion changes the branching structure of the object
completely legally to something quite different from the original. There are
now three main branches in the object defined by the "Display", "Centre" and
"Inherits" tags where there used to be two. While usage of both spaces and
Tabs is completely legal, using Tabs is not a good idea !

If inadvertantly you read this wrong model file and start using it

   * You can still see your old data, because acedb uses the schema as it
     was when the data was loaded.
   * You can parse old ace files, because the list of tags did not change,
     just their organization. But now data under "Centre" go to the new
     place.
   * Queries concerning "Centre" and any tag below only retrieve newly
     entered data, because the old objects do not contain those tags in the
     current position

To fix the mess, please use tace:

tace $ACEDB << EOF
query find map centre // gets just the 'new' centre
show -a -f toto.ace centre // exports just that tag
edit -D centre // removes the tag in the 'new' objects
read-models // read back the correct models
parse toto.ace  // recover your data in the right place
save
quit
EOF

Here are some tips that will help you to avoid having trouble editting the
models.wrm file:

   * Only use a plain text editor, i.e. one that does not format or add any
     control characters to the file, e.g. "vi", "emacs" or some similar
     editor in Unix and "Notepad" in Windows. In particular note that if you
     are working on a Windows machine your editor will probably add DOS
     style "end of line" characters to the end of each line. This is fine if
     you only use the models file from a windows machine but you should be
     aware that this will make that models file unparseable from a Unix
     machine.
   * Use a constant width font to view the file so you can be sure your tags
     are correctly lined up.
   * Always use spaces instead of Tabs, that way it will be more obvious if
     you accidentally delete something. Chances are that you will produce an
     illegal models file and acedb will report the error.

-----------------------------------------------

Bugs Fixed
**********

Gene curation bug in FMap
+++++++++++++++++++++++++

A number of users have reported the error "Failed to update parent sequence
object: " when trying to make temporary genes from within the gene finding
package for the FMap. This has now been fixed and was a fault in the code
that tried to find a parent for the temporary gene.

Blixem not displaying homologies that extend to end of sequence
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

A trivial (as in the coding, not in the debugging sense !) off-by-one error
in coordinate calculations in blixem calling code resulted in the loss of
all homologies which extend right to the end of the displayed sequence.

acediff bugs
++++++++++++

acediff was erroneously interpretting "//" in a text field as the start of
an acedb style comment and hence getting the text field wrong. This is now
fixed, backslash escapes in comment delimiters are now honoured:

// this is a  comment
/\/ this is not (now) a comment

acediff had a problem processing files where the same tag was repeated due
to clumsy pre-processing and generated spurious -D lines in the output, this
is now fixed.

Printer list in print window
++++++++++++++++++++++++++++

The printer list in the print window used to contain blank lines if the
system file /etc/printcap contained incomplete entries. This is now fixed.

makefile shenanigans
++++++++++++++++++++

Replaced "test" with "/usr/bin/test" since the test builtin in some versions
of the shell doesn't understand "-L" -- step forward Solaris!

-----------------------------------------------

Developers Corner
*****************

Performance and acedb
+++++++++++++++++++++

(This article is courtesy of Jean Thierry-Mieg mieg@ncbi.nlm.nih.gov)

This could be interesting to all programmers. I recently fixed heavy
performance problems just by editing my makefile:

  1. I had an apparent memory leak in the acedb server underneath my
     web-site, affecting Solaris on sparc or on Intel. The leak went away
     when I removed from my makefile the pragma _-l malloc_
  2. On Linux, I considerably accelerated parsing files by removing a pragma
     _-l pthread_, although I was actually not using threads, acedb is NOT
     thread save, the library was listed in my configuration files, because
     of other programs, and this turned the getc(), used by the acedb parser
     to scan input files, from a macro into a very slow function. It is
     possible that this library is linked implicitly in many C++
     implementations, slowing down many of your programs, beware!
  3. Several months ago, qsort on Solaris was bugged, retrieving a negative
     record at a very low but non zero probability, and inducing very soon
     after that a flat crash. I fixed that by replacing in arraySort the
     call to qsort by an actually faster call to msort, a function that I
     added in freesubs. Conclusion: If you have any trouble with
     performance, do not forget to suspect the standard Unix functions, such
     as qsort, and standard libraries, such as lmalloc and lpthread. Also
     remember that I am always interested in hearing about performance
     problems (mieg@ncbi.nlm.nih.gov).

About catText catBinary
+++++++++++++++++++++++

(This article is courtesy of Jean Thierry-Mieg mieg@ncbi.nlm.nih.gov)

Let us re-explain what happened last month. Since its introduction,
catBinary was bugged, it was appending an extra zero at the end of the
buffer. This was apparent only if you were calling catBinary() twice on the
same stack, because you then had this extra zero in between the two pages.
Nobody ever complained, may be nobody ever tried. This bug was fixed by Mark
Sienkiewicz around May 25.

While fixing the bug, catText was inadvertently modified. This killed the
query language, was detected by Aquila the same night, and fixed the
following day. So, just to clarify:

   * The automatic Aquila test correctly reported the error, merci Detlef.
   * catText was not modified, except for a few hours, and only on the
     development version, during which acedb was brain dead (all queries
     failed). So nobody should worry about catText.
   * However, catBinary was purposely modified. This affects, in the public
     distribution, binary buffers exported by aceserver. If by chance you
     had a work-around the bug in your client code, you should be aware of
     this change and modify your clients to cope with the now correct data
     exported by the acedb server.

Server speed up
+++++++++++++++

(Thanks to Mark Sienkiewicz (formerly of NCBI) for spotting this one.)

Requests/replies can be greatly speeded up for the server by setting the
TCP_NODELAY flag for the socket. This prevents TCP from waiting for an
acknowledgement for each packet before it sends the next one. This is
particularly relevant for small requests on slow networks (the effect is
hardly noticeable on a local high speed network):

Jean Thierry-Mieg mieg@ncbi.nlm.nih.gov reports:

Mark Sienkiewicz greatly speeded up the server by setting the TCP_NODELAY
flag for the socket. This prevents TCP from waiting for an acknowledgement
for each packet before it sends the next one. The improvement is
particularly important for small requests on slow networks, as when one uses
AcePerl over the network across the Atlantic ocean. The way Mark found the
bug is interesting. Because it seemed too hard to retrofit the ace.4_9 TCP
protocol (let us call it the s-protocol) in the ace.4_7 code still used in
production at NCBI, he wrote, in the context of Ace-C, a new simple minded
TCP protocol (let us call it the a-protocol). Later the Ace-C client was
adapted to talk to the ace.4_9 saceserver, and it became immediately
apparent that the s-protocol was way slower than the a-protocol, even on the
NCBI intranet. After some testing, it all boiled down to raising a single
flag, and speeds were equalized.

Optimisations
+++++++++++++

(This article is courtesy of Jean Thierry-Mieg mieg@ncbi.nlm.nih.gov)

With data size ever increasing, it becomes necessary to optimize the code
once more. To this end, i revamped the acedb CHRONO and started pushing
fixes in the CVS.

  1. A glitch in the hashing function used by the dict library was fixed.
     This does not affect the acedb main classes, which have their own
     direct hashing. The improvement is visible only if you use a dict with
     over 100,000 entries.
  2. I modified disk/cache handling with a tremendous effect on very large
     databases. To parse the 16 million objects of the whole human genome
     annotation database on an alpha, time went down from 24 to 12 hours.
  3. The problem of disk runaways has been found by Mark Sienkiewicz but
     needs to be incorporated into the development code.
  4. File scanning optimization is under way, and will be put in the CVS
     shortly.
  5. Several types of query now use the internal indexing more efficiently.

Chrono
++++++

(This article is courtesy of Jean Thierry-Mieg mieg@ncbi.nlm.nih.gov)

Chrono is back and working. It is a very versatile way of profiling acedb.
It is programmed as macros, and does not affect the compiled code when
turned off.

Usage:
#define CHRONO       /* if undef, all chrono calls disappear */
#include "chrono.h"   /* must come after the define CHRONO */
...
f ()
{
chrono ("f") ;      /* or any message you like */
.........            /* do something */
chronoReturn () ;   /* carefully chronoReturn along each path*/
/*
beware of goto's and early return statements */
}
...
main()
{
chronoStart () ;
chronoStop() ;
chronoReport () ;
}

The nice thing is that you can chrono a whole function or just a few lines
of code. Nested chrono calls are allowed.

To start and stop a chrono session, there are four methods

   * call chronoStart/Stop/Report in main()
   * define CHRONO at the top of session.c to chrono the whole acedb
     session.
   * activate chrono from the tace prompt (to be folded in CVS very soon)
     Chrono {start | stop | show} affects all chrono-aware routines
   * activate and visualize chrono from xace->admin->chrono This option is
     available in xace up to 4_7, but seems lost in 4_9.

Example comparing acein and freecard on the same dataset:

// Total time : 4.42 s  system,  66.43 user  level = 0
//           # of calls   System       %          User       %
//   Chrono        0      0.00 s      0 %         0.00 s     0 %
//test_acein       1      0.65 s     14 %        14.50 s    21 %
//test_freecard    1      0.92 s     20 %        13.93 s    20 %

-----------------------------------------------

June monthly build now available.
*********************************

You can pick up the monthly builds from:

_Sanger users_
     ~acedb/RELEASE.DEVELOPMENT
_External users_
     http://www.acedb.org/Software/Downloads/monthly.shtml

-----------------------------------------------

Next User Group Meeting - D319, 3.00pm, Thursday, 10th July 2003
****************************************************************

----------------------------------------------------------------------------
----------------------------------------------------------------------------
Ed Griffiths <edgrif@sanger.ac.uk>
Last modified: Tue Jul 8 10:46:24 BST 2003