🚀 go-pugleaf

All,

After quite a difficult year which saw us a person short for most of the time
coinciding with a period of major code changes, we are at last reasserting some
sort of order.

The SMap code is in acedb and being used extensively by the worm group, there
have been many other changes and bug fixes.

The enclosed newsletter is by way of a catch up, do take a look at the website
and look through the Year 2002/2003 links which contain the fuller docs.

New years resolution: be more on time with builds and newsletters.....

cheers Ed
-- 
 ------------------------------------------------------------------------
| Ed Griffiths, Acedb development, Informatics Group,                    |
|        Wellcome Trust Sanger Institute, Wellcome Trust Genome Campus,  |
|               Hinxton, Cambridge CB10 1SA, UK                          |
|                                                                        |
| email: edgrif@sanger.ac.uk  Tel: +44-1223-494780  Fax: +44-1223-494919 |
 ------------------------------------------------------------------------



----------------------------------------------------------------------------
----------------------------------------------------------------------------

ACEDB User Group Newsletter - Catch up Newsletter
#################################################

If you want to have this newsletter mailed to you _or_ you want to make
comments/suggestions about the format/content then send an email to
acedb@sanger.ac.uk.

-----------------------------------------------

After rather a long time in the doldrums of debugging and other things we
have finally made a good monthly release of acedb that incorporates the SMap
code _and_ a stable FMap. Completely replacing the way FMap builds its
virtual sequences has been very hard as there are many subtle links between
the sequence building and the display code that got broken.

Monthly builds and newsletters will be back to just that from now
on...monthly.

This newsletter contains a summary of all the many changes made over the
last few months, you can see more details at:

   * 2002
   * 2003

I strongly recommend that you take the time to visit these web pages and
have a look through the article titles there to see if there is anything of
interest as I have not included everything in this newsletter.

Thanks for everyones patience.

-----------------------------------------------

General News
************

AceDB changes its license
+++++++++++++++++++++++++

Acedb has been licensed under the Gnu Public License (GPL) for some years
now which means that the code is freely available to all but cannot be
filched for commercial gain and that authors must be acknowledged. This
license has a major drawback however in that it effectively prevents the use
of the licensed code in any form in commercial software. As this was
restricting the spread of GPL code (synonymous with "open source software"),
a new less restrictive license was produced which meant libraries consisting
of GPL code could be incorporated into programs sold for profit provided the
inclusion of GPL code was acknowledged.

Acedb has now moved on to this less restrictive form of licensing allowing
greater use of the code for commercial programs.

Rob Clack joins us
++++++++++++++++++

At the end of last year John Sturdy unfortunately had to leave us and we
have finally managed to recruit Rob Clack as a replacement. Rob will
initially be taking over the acedb web site and test systems. You can send
email to Rob at rnc@sanger.ac.uk.

-----------------------------------------------

New Features
************

New "dna" command options
+++++++++++++++++++++++++

The tace/giface "dna" command now has options for unspliced/spliced/cds dna
dumping. The new command synopsis is:

Dna [-mismatch] [-unspliced | -spliced | -cds] [[-f] out_filename] :
   Fasta dump of related dna sequences.
        -mismatch          allow mismatches in construction of dna
        -unspliced | -spliced | -cds
                           for intron/exon objects dna can be dumped for exons +
introns,
                           the spliced exons or just the CDS within the spliced
exons.
        [-f] out_filename  output file for dna.

Improvements to column overlap mode in xace
+++++++++++++++++++++++++++++++++++++++++++

The different columns in the fmap correspond to the different Method objects
in the database, on the forward strand they are given the name of the Method
object, on the reverse strand the name is prepended with a "-". Some of the
Method object data is shared for both columns, some is column specific. Up
until now the overlap mode was shared which meant that if you selected
"bump" it would be applied to both forward and reverse strands which could
be very irritating as the whole display would be shifted by a large amount
to the right. Now overlap mode can be set independently for the forward and
reverse strands. The mode is specific to the strand so that if you select
"overlap" for the reverse strand, then it will remain as overlap when you
for instance reverse complement the display.

Changes to the EMBL dumping code within AceDB
+++++++++++++++++++++++++++++++++++++++++++++

(This article is courtesy of Dan Lawson dl1@sanger.ac.uk)

AceDB has the ability to dump a genome sequence in EMBL format. One facet of
this code is the addition of /note qualifies for matching EST/mRNA
transcript evidence to a CDS feature. AceDB 4.7 checked whether each of the
cross-reference EST objects had a Database line with an EMBL accession and
would output one of the following:

 if the EST has an EMBL accession stored.

      /note="cDNA EST EMBL:D72339 comes from this gene"

 else the object name is returned.

      /note="cDNA EST yk97e6.3 comes from this gene"

AceDB 4.9 no longer performs this check and will always return a note line
which use the AceDB object name.

e.g.  /note="cDNA EST yk97e6.3 comes from this gene"

This ensures that a curator has control over the transcript name in the EMBL
file such that consistency is maintained within the AceDB namespace rather
than the EMBL namespace.

First round of keyboard shortcuts
+++++++++++++++++++++++++++++++++

There have always been some keyboard shortcuts in acedb, recent discussions
have shown that there is a pressing need for some kind of rationalisation
and extension of these. I have collated all the emails that I received about
existing and desired shortcuts, this is round one with more to come in the
next few weeks.

The first set of shortcuts are available from any of the main acedb windows
and include:

   * Ctrl-S:  save the database

   * Ctrl-W:  close this window

   * Ctrl P:  print this window

   * Ctrl Q:  quit acedb

   * f1/f10:  help

   *  f2/f9:  print this window

The job of implementing the shortcuts has passed through a number of hands
and hopefully now its going to get done, thanks to everyone for their
patience, send mail direct to me, edgrif@sanger.ac.uk if you get fed up with
waiting or have suggestions/bugs etc.

Splash window preference
++++++++++++++++++++++++

Acedb displays an initial "splash" window when it starts up which with some
window managers remains on top of all the other windows on the screen which
can be irritating. There is a command line option "-nosplash" which can be
used to turn off display of the window. Now you can also set a preference to
not display the window at all, see "SPLASH_SCREEN" option in the preferences
window which can be reached from the "Admin" menu on the xace main window.

Killing the AceDB server with SIGTERM
+++++++++++++++++++++++++++++++++++++

Server programs, especially those run in the background will normally accept
certain signals as "commands" to get them to take certain actions, the de
facto standard for unix includes:

   * SIGHUP: reread server configuration file(s).
   * SIGTERM: terminate gracefully.

The AceDB server will now respond to SIGTERM and shut down correctly. This
is done by by setting a global flag from a server callback routine called by
the acedb signal handler function. The server then checks for the global
flag in the main processing loop of the socket layer, and then initiates the
save and exit of the database. This seems a bit contorted but avoids any
issues of trying to use issue calls that are not signal safe from a signal
handler or of trying to exit perhaps half way through an interrupted partial
acedb save.

Support for SIGHUP will be added if required.

Some FMap changes
+++++++++++++++++

A number of small changes have been made to FMap:

Support for "Show_in_reverse_orientation" tag: The worm database version
of the ?Sequence class contains a Show_in_reverse_orientation tag which is
used to indicate that the 3' sequence of an EST paired read that has been
sequenced in reverse. The FMap code now looks for this tag and will reverse
the strand that the sequence is shown on so that it appears downstream from
its 5' pair. To take advantage of the tag you need to add this tag to the
?Sequence class, you can put the tag anywhere in the class but in the worm
version it is under the "Properties" tag:

Properties    Pseudogene Text
              Genomic_canonical Gene_count UNIQUE Int

              etc.

              Show_in_reverse_orientation   // Draw 3' reads in reverse
orientation

              etc.

Improved display of homologies with a common clone parent: Clicking on a
homology box in fmap will now hilight all all the homologies in that column
that have a common clone parent. In addition if you select to have the
column displayed in "cluster" mode, homologies with a common parent get
sorted into the same vertical alignment within the column, making them
easier to see.

Empty Strings in Tree Display.
++++++++++++++++++++++++++++++

The tree display has been altered so that it will now display for text
fields with null strings, previously they were not displayed at all and
hence not edittable.

New Server PID file
+++++++++++++++++++

A new _PIDFILE_ option has been added to the wspec/serverconfig.wrm file,
uncommenting this will cause the server during initialisation to write a
file into the database/ subdirectory containing the pid of the server. The
file will be deleted when the server terminates. You can use this file to
control the server via a number of unix utilities which are commonly used to
provide general control of server programs.

As with other new configuration options in acedb, to make use of the new
options you should:

  1. Get a copy of the new wspec/serverconfig.wrm from the acedb web site.
  2. In your database merge your old wspec/serverconfig.wrm with the new
     one.
  3. Uncomment the new PIDFILE flag in serverconfig.wrm
  4. Run the new server against the database and you should see a file
     called database/SERVER_PIDFILE.wrm which should contain the pid of the
     server.

Some things to note:

   * In normal operation the server creates the pid file at start up and
     removes it at shutdown.
   * If the server crashes in an untrappable way then the pidfile will not
     be removed of course. In this case when the server starts up, if the
     pid file is still there it will complain but _will_ start up. If you
     don't want it to restart after crashing you should take a look at the
     NO_RESTART option in serverconfig.wrm.

New "Save" and "WriteAccess" commands/options
+++++++++++++++++++++++++++++++++++++++++++++

Previously users have had more control over saving and write access when
using xace (e.g. "Save session and keep write access") than with the command
line tace and giface programs. New options have been added to the "Save"
command and a new "WriteAccess" command has been added:

Save : [-drop | -regain]
    Save current state otherwise the system saves every 600 seconds
       -drop   (default) drop write access after saving
       -regain regain write access after saving

Writeaccess : [-gain | -drop [save | nosave]]
    Set write access status
       -gain (default) get write access
       -drop lose write access, with or without saving

These commands are particularly useful for scripts that need to update a
database and perhaps need to hold on to write access while parsing in
several files of data.

New options for giface commands to give more power to xremote
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

The giface program allows the creation of "virtual" displays (e.g. fmap,
gmap) and then the dumping of those displays in various formats, e.g. GFF,
postscript, etc. Perhaps less well known is that xace can be driven by the
same commands via the xremote program distributed with AceDB.

For instance:

xremote -remote 'gif ; seqget bA314N13 ; seqdisplay'

Will get you an FMap of clone bA314N13 displayed by xace (you must have xace
running _before_ you issue the xremote command.

This mechansim is used as a core part of the "lace" annotation tool
maintained by James Gilbert James Gilbert <jgrg@sanger.ac.uk> for the human
annotation project at the Sanger Institute.

The new options include:

seqdisplay ..... [-new] [-title title]
     Works on current sequence, use gif or ps to actually dump the display
        -new     make a new sequence, current one is preserved (for use only
with xremote currently)
        -title   give your own title to the sequence

seqrecalc
     Causes the sequence to be recalculated for the active graph

You could for example use the new commands/options to construct a script
which:

  1. Starts up an xace for the user, and then automatically displays a
     sequence in an fmap:

     xremote -remote 'gif ; seqget sequence_name ; seqdisplay'

  2. Adds new data into the database and gets the fmap to recalculate and
     show the new data by:

     xremote -remote 'gif ; seqrecalc'

  3. Replaces the sequence in the fmap with a new sequence:

     xremote -remote 'gif ; seqget new_sequence_name ; seqdisplay'

  4. Displays a new sequence in an entirely new fmap (which becomes the
     current fmap, the existing current one is preserved):

     xremote -remote 'gif ; seqget another_sequence_name ; seqdisplay -new
-title "Super New Sequence'

Gene Finder changes
+++++++++++++++++++

Kevin Howe <klh@sanger.ac.uk> has made some changes to the Gene Finder code
(originally written by Phil Green of Washington University Medical School)
which while not strictly affecting acedb are worth documenting here (most of
the changes affect the stand alone version of the gene finder code).

Kevins comments:

You can do a diff on the file yourself to see what I've changed. The
standalone-only stuff I've protected with "#ifndef ACEDB", but I'll just
summarise the other changes here:

  1. The array "cumVec" is now double instead of float (and this has some
     knock-on effects in places where this array is used/passed, parameter
     types etc). This is because the float precision is inadequate for
     sequences more than 10kb or so. You may want to change this back.
  2. I've changed the constant SEGSCORECUTOFF from a #define to a static
     float. This is so that the value can be modified by a command-line
     option in the standalone version.
  3. The main calculation of the coding segments and genefinder features in
     "aceFeatures" and "aceSites" is now protected with checks that that
     tables for each site-type are non-null. This will have no impact in
     ACEDB, because the tables are always non-null, but I've extended the
     standalone version to be more flexible in what it expects in the tables
     file. If you don't want ATG predictions, then you can make a tables
     file with no ATG tables, and the code will now happily not try to
     predict ATGs.
  4. In aceMaxSegs, there was a mistake which meant that some GF_CODING
     segments were calculated in the wrong frame, which I've fixed. In
     aceFeatures, I also added a fix which ensures that coding segments that
     bridge a stop codon are broken into two segments, one either side. I
     discussed this one with Richard and he agreed that this is the now the
     correct behaviour.

Better FMap and Tree display interactions
+++++++++++++++++++++++++++++++++++++++++

When displaying objects in the Tree display window, its always been possible
to click on a sequence object key within the tree display and see the FMap
display of that object. The problem with this in the past has been that in
the case where the sequence object is actually the parent of the object in
the tree display the subsequent fmap is not that helpful in that it displays
the whole of the object instead of the part that relates to the object in
the tree display. The acedb display system will now show the sequence object
zoomed and positioned around the object in the tree display.

New -tsuser command line option

All AceDB programs now accept a -tsuser commandline option of the format:

-tsuser "new_username"

By default the username for a database session is the userid you logged in
with. You can use -tsuser to set a different username for the session. This
is useful in at least two situations:

  1. If you are using various scripts to interface to tace or giface to make
     updates to the database, you can use the -tsuser option to use the name
     of the script instead of your login name for the updates, that way you
     can be sure which updates were made by which scripts.
  2. If you have a number of developers who all access the database from the
     same userid for access permissions reasons, by setting the username for
     each database user you can keep track of the updates made by the
     different users.

New "Nosave" command
++++++++++++++++++++

You can now turn on and off saving by acedb using the "Nosave" command:

NoSave [-stop | -start]
   Stop or start all database saving, the -stop option prevents any saving by
any command,
   you must use the -start option to turn saving on again.
      -stop   (default) turn saving off
      -start  turn saving on.

This is particularly useful if you want to test scripts that use tace for
large scale database updates without accidentally altering the database
during testing. By setting saving off at the start of the script, you can
get all your data parsed by acedb so see if its correct but it won't be
saved in the database.

New giface "smap" command
+++++++++++++++++++++++++

SMap has been made available via the giface interface so it's possible to
find out the positions of one feature in its parent. The command interface
is:

smap -coords x1 x2 -from class:object -to class:object

The command maps the coordinates x1, x2 (default is the whole object) in the
"-from object" to coordinates in the "-to object", the former must be a
child of the latter via SMap or Sequence tags.

The command returns a string in the following format:

"SMAP  class:name y1 y2 nx1 nx2 status"

Where

   * class:name is the class and name of the parent object.
   * y1 y2 are the coordinates of the child in the parent object.
   * nx1 nx2 are the coordinates in the child that were actually mapped to
     produce y1 and y2, these will have changed if the original
     coordinates had to be clipped to fit in the range of the parent object.
   * status is an smap status code which indicates whether the mapping was
     perfect or clipped in some way.

New Blixem Preference
+++++++++++++++++++++

>From the xace "Preferences" dialog you can now specify a path for the script
that launches the stand alone version of blixem (you need to set the
BLIXEM_EXTERNAL preference to use the stand alone version). This allows
users to select which version of blixem they want to run which can be useful
for testing new versions.

Display of Homologies and EST read pairs in FMap
++++++++++++++++++++++++++++++++++++++++++++++++

The FMap display now supports drawing of lines between the blocks for a
single feature, e.g. all the blocks of a single homolgy match. This is very
useful when there are lots of homologies in a column or when a homology is
spread out over a long distance as it makes it easy to see which blocks
belong with each other or whether there are further blocks off screen. This
is extended for EST read pairs so that the blocks for each match are joined
by lines and the pair are also joined with a dashed line. The line drawing
is turned on when the display mode for a feature/homology column is set to
"cluster" (cluster mode already vertically aligns the blocks for each
feature).

Line drawing for all features is controlled by:

Join_blocks:This is a new tag in the Display section of the ?Method class,
if present in the Method object for a feature then blocks will be joined by
lines when the column display mode is "cluster".

In addition to specifying the Join_blocks tag, connecting of EST read pairs
by a dashed line is controlled by the following tags within the two Sequence
class objects representing the EST read pair:

Paired_read ?Sequence XREF Paired_read: each EST Sequence object must cross
reference the other.

EST_5 or EST_3: one EST Sequence object must have the EST_5 tag and one the
EST_3 tag to indicate whether they are the 5' or 3' read.

AceConn a C package for accessing the Socket Server.
++++++++++++++++++++++++++++++++++++++++++++++++++++

Previously there were the following ways to access the socket server:

   * saceclient: a command line based program that could be driven by
     scripts.
   * AcePerl: perl package interfacing directly to the socket server.
   * BioJava: contains a package interfacing directly to the socket server
     (but also see next article).
   * BioPython: contains a package interfacing directly to the socket
     server.

There was no package for a client written in C to interface to the server.

This gap has now been filled with a new package called AceConn. This is a
small library of C routines enabling a C based client to interface to the
server. The main components of the interface are:

AceConnCreate():
     creates the initial connection structure which is the handle for all
     subsequent calls to the package routines.
AceConnConnect():
     makes the intial connection to the server doing all the necessary hand
     shaking, if this call succeeds then the client can start sending
     requests.
AceConnRequest():
     Sends a request to the server and returns the reply.
AceConnDisconnect():
     terminates the connection to the server doing all the necessary
     handshaking.
AceConnDestroy():
     Frees the connection allocated by AceConnCreate().

The interface sends text commands just as the user would type in when using
saceclient and then returns the reply from the server in a buffer. Usually
this reply is text but it could be an image as produced by some of the gif
subcommands. The client must parse the returned data itself.

Each request will wait for a client specified number of seconds before
timing out. The timeout may be set to zero for the client never to timeout,
or negative to reset to the default timeout (120 seconds). This allows the
client code to retain control of how long it remains blocked waiting for a
request.

If a client makes a request that produces a very large reply, the server
returns the reply in slices so that other requests from other clients can
also be processed. The AceConnRequest() call automatically batches up these
requests into a single reply. While this is convenient you should be aware
that this can lead to the client using up considerable amounts of memory for
large requests.

You can see further documentation by downloading the Ace-Conn package from
the acedb website.

A useful new Java package for accessing the socket server.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Kemin Zhou formerly of Affymetrix has contributed a Java package for
accessing data from the socket server. You can download the package from the
acedb web site: Java-Zhou package

Kemin's description:

"This simple package is meant to give a basic interface to ACEDB. You can
view the documentation with webbrowser

Type netscape index.html then you are there.

You need to put the acedb directory in your CLASSPATH Or just use the
ace.jar file. In you java program say import acedb.*;

You have to set up your own Acedb and Saceserver, that is not difficult.

This package is to be used by programers who knows both Java and ACEDB."

-----------------------------------------------

Articles
********

SMap Model clarification
++++++++++++++++++++++++

A small but important clarification has been added to the SMap model for
gluing together sequences and features, it is fundamental to the SMap that
each child has one and only one parent. Objects that are to be SMap'd must
form strict trees, i.e. each object has a single parent (except for the top
object which has no parent). Objects may, of course, have multiple children.

The part of the SMap model that must be put in the child object to point to
its parent should be:

          SMap S_Parent UNIQUE <parent_tag> UNIQUE <parent_key>
                               <parent_tag> UNIQUE <parent_key>

As an example the current wspec/models.wrm Sequence class includes:

?Sequence

          SMap S_Parent Canonical_parent UNIQUE ?Sequence XREF
Genomic_non_canonical
                        Genomic_parent UNIQUE ?Sequence XREF Nongenomic
                        AGP_parent UNIQUE ?Sequence XREF AGP_fragment

This means that any _one_ of Canonical_parent, Genomic_parent or
AGP_parent could be the parent of this object.

Dotter and Blixem binaries
++++++++++++++++++++++++++

The downloads page on the acedb web site now includes a separate downloads
package for the belvu, dotter, and blixem binaries. These are available for
Unix, Mac OSX and MS Windows.

Socket server performance improvements
++++++++++++++++++++++++++++++++++++++

A while ago a request was received to add time stamps to each log file
message written by the server which seems reasonable. Unfortunately this
turned out to be a very bad idea performance-wise since the operation of
finding the current time is always very expensive. This simple change had a
major impact on server performance, especially where a client issued a whole
series of AceDB commands, each one requiring a time stamp !

Having a timestamp for every log message is now optional, the default is to
have them only for session start/end messages. In the future an option will
be added to allow the server administrator to specify a timestamp per every
NN messages.

Socket server performance is now significantly better than the RPC based
AceDB 4_7 server.

Changes to Sanger Centre binaries
+++++++++++++++++++++++++++++++++

As of this month we are building acedb on the latest versions of OSF for the
Alpha machines and Linux for the PCs:

   * OSF1 V5.1 732 alpha
   * Red Hat Linux release 7.3 (Valhalla) Linux 2.4.18-10 i686

If you download binaries from the Sanger site then please make sure that
your operating system is at these levels at least otherwise the binaries may
not run.

If you are running older systems you will need to download the source and
compile it yourself. Please report any problems with doing this to:
acedb@sanger.ac.uk.

Acedb, MAC OSX and "Darwin"
+++++++++++++++++++++++++++

(This article is courtesy of Simon Kelley srk@sanger.ac.uk)

Acedb is now available to run on Apple Macintoshes running Mac OS X. We are
using the Fink packaging system to distribute Acedb for Macs. This is
available from http://fink.sourceforge.net/

Acedb depends on a few other bits of software which don't come with OS X but
are available from the Fink system. The main one of these is the X windows
system.

To get Acedb on a Mac, first install fink by following the instructions at
http://fink.sourceforge.net/download/index.php

Download the package for acedb from http://www.acedb.org/Software/Downloads

Install the extra software required by running the following command in a
Terminal Window:

fink install xfree86-base xfree86-rootless gtk+ readline

and finally install acedb by runnning this command:

dpkg -i <path_to_acedb_package>

Removal of Match_Table support
++++++++++++++++++++++++++++++

Prior to the introduction of SMap, there was a problem with large numbers of
homologies being processed/displayed in the FMap because the homologies had
to be embedded in the Sequence objects. The Match_Table tags could be used
to specify which homologies were processed/displayed. Few users made use of
this tag and since the SMap now allows homologies to be held outside of the
sequence objects, support for these tags has been removed from the code (as
advertised in the acedb newsgroup some months ago).

-----------------------------------------------

Bugs Fixed
**********

ACEDB.wrm should not be writeable
+++++++++++++++++++++++++++++++++

Sadly a bug was introduced into the code a couple of years ago and recently
reported by a user. model.c erroneously checked to see if database/ACEDB.wrm
was writeable as a test to see if current user should be allowed reread the
models file. This is a bad idea, users like to make ACEDB.wrm read only to
prevent accidental reinitialisation of database. There was already an
internal routine in session.c which tests for the admin user and does not
require this incorrect predicate, I have added an external function to
session.c to make this visible to other bits of acedb code so that the check
is correct now.

Users should keep database/ACEDB.wrm as a read only file to prevent
accidental erasing of this file followed by possibly catastrophic accidental
reinitialisation of the database.

SMap/Fmap bugs
++++++++++++++

This month has seen a lot of bugs discovered and fixed which come down to
problems with interactions between SMap and FMap and also the problems of
introducing "recalculates" into parts of FMap where their used to be redraws
which has revealed that FMap contains quite a lot of state that goes stale
when this happens and produces bugs that are hard to notice and find the
cause of.

I include the below simply as pointers to the type of things that have been
going on:

   * Bug in sMapLength, if there are mixed sMap and ACE4 tags, we should use
     both.
   * Fix up sMapFindSpan so that it knows only about 1-based coords, this is
     the standard for smap stuff. Make sure that other smap calls used
     sMapFindSpan correctly.
   * Fix bug in sMapFindSpan(), meant that it would never find the span for
     the given coords.
   * clear look->chosen hash on recalculate - it contains stale data.
   * fix setStartstop to cope with 1,0 coords - as when called in giface
   * Fix code that does coords for recalculate, it was broken because we
     don't use fmapFindDNA any more which used to frig the coords. The new
     rules are encapsulated in setStartStop().
   * Add code to fMapDoRecalculate() to make sure that any new start/stop
     include current active zone, this became a problem with smap code
     because we get hold of the sequence a bit differently, but was also a
     problem as far back as 4_7 if you scrolled in just the right way.
   * Fix up setTempGene() so that it works out reverse complement coords
     correctly because we now use keys, not segs, to do coords stuff.
   * Fix bad bug which sometimes corrupted the dna array by inserting a C
     "end of string" char but then not removing it.
   * initialise() was not called from all the places it should have been,
     this did not matter previously because bsUpdate() and other operations
     were done on _VSequence, now a static holds the class, if the static is
     not initialised the code eventually crashes because the key does not
     have the correct class for the bsNNN calls.
   * fMapFindSpan() is now really a call to sMapFindSpan() but this is all
     messed up because of confusion over coordinate systems: smap stuff
     should ALL BE ONE BASED. Otherwise you get bugs like the one in
     fmapgene where everything is suddenly off by one.
   * fix bug in seg flipping for complementing when something is clipped by
     smap, need to reverse which end is clipped.
   *

Inappropriate "Show As" actions
+++++++++++++++++++++++++++++++

The code to decide what to do when a user tried to display an object from
the keyset window when the "Show as" display type was incorrect did not
always work very sensibly or, in some cases, did not work at all.

The code to decide what to do when there was an object/display type
mismatche was scattered across some 15 files which compounded the problem.
All the code to choose what to do is now in display.c and takes a consistent
series of steps to cope with this problem, ending with the Tree display if
no other appropriate display can be found. In addition the code will now
warn the user when the object simply cannot be displayed.

Logging stops after saving from tace
++++++++++++++++++++++++++++++++++++

Dan Lawson reported that logging stops after saving from tace. It turns out
that closing of the log got introduced into sessionClose() at the beginning
of 2000, as part of the conversion to use aceOut for accessing the logfile.
One can see why this was originally done: sessionClose() is called when the
application is exitting. Unfortunately its also called when applications are
just saving. To avoid this problem calls to sessionClose() have been
replaced with calls to sessionMark(), sessionMark() no longer calls
sessionClose(), instead they both call an internal routine closeSession(),
which saves the session but does not close the log file. sessionClose() is
now the routine to call when the application is going to exit.

Query Fixes
+++++++++++

Number evaluation has been removed from string compares, this was producing
errors when obj names actually were numbers because they got rounded !

An optimiser has been added in [] calcul clause after a bug report by Dave
Matthews about the slowness of this. This has reduced a 10 minute query to <
1 second.

Ancient acediff bug
+++++++++++++++++++

Two bugs fixed:

cmpFileParts() was altered at some time to do error checking for EOF (which
was a good idea) but in doing the check read _twice_ as many chars as it
should have because it was doing a separate getc() to test EOF from the
getc() to just get the next char from the file. This caused the program to
do anything from fall off the end of the file to erroneously report
differences.

oneLine() did not count lines in the file it was processing correctly, this
is because a) the freeread() routine it called was incorrectly translating
pseudo-newlines into just 'n' (now fixed) and b) it didn't increment the
line count when it encountered one of these (which in acefiles signals a
multiline text entry).

-----------------------------------------------

Developers Corner
*****************

If you wish to make comments/suggestions about any of the below, please mail
them to acedb@sanger.ac.uk

Aquila work
+++++++++++

Rob and Mark (Sienkiewicz) have been doing good work on aquila on serveral
fronts:

   * Verifying the test results to make sure that output is actually correct
     rather than just different.
   * Introducing the new aquila program (written by Robs predecessor but
     never fully integrated into the test system). This system enables the
     running of individual tests or multiple tests because test script files
     can now "include" other test script files to build up sets of tests.
   * Putting all the aquila and build scripts into cvs.
   * Creating a document to describe how simple it is to add new tests !

Replacing bIndexFind() with bIndexTag()
+++++++++++++++++++++++++++++++++++++++

The bindex package in acedb provides a call to use the index to test whether
a particular tag exists in an object:

BindexFindResult bIndexFind(KEY originalKey, KEY tag) ;

But there is a trap for the unwary here in that the call can return one of
three possible values (BINDEX_TAG_ABSENT, BINDEX_TAG_UNCLEAR,
BINDEX_TAG_PRESENT) and its very easy to forget this and write code that
assumes that a Boolean value is returned:

if (bIndexFind(key,tag))

This call will sometimes return BINDEX_TAG_UNCLEAR and sometimes
BINDEX_TAG_PRESENT with consequent confusion in the code following the test.

In fact Jean had already added some functions (keyGetNNN) to bssubs.c to use
bIndexFind() but then open the object and look for the tag if the return
from bIndexFind() was BINDEX_TAG_UNCLEAR. These calls returned an
unambiguous Boolean value to indicate whether the tag was present or absent.

These calls have now been renamed and moved to bindex.c as it seems more
logical/obvious for them to be with bIndexFind, they are:

BOOL bIndexTag(KEY key, KEY tag) ;

returns TRUE if tag is found in object key.

BOOL bIndexGetKey(KEY key, KEY tag, KEY *key_out) ;

returns TRUE if tag is found in object key and will return the key
following tag in key_out if key_out is non-NULL.

New aceOutPrintStr() function
+++++++++++++++++++++++++++++

I've added a new aceOutPrintStr() call for printing simple strings. This is
try and dissuade people from using aceOutPrint() for this purpose which
leads to bugs if the string happens to contain a "%" which is then
interpreted as a formatting instruction with usually bad results.

qsort/msort routines
++++++++++++++++++++

(This article is courtesy of Jean Thierry-Mieg mieg@ray.nlm.nih.gov)

Replaced qsort by self contained msort, incredibly qsort crashed (rarely but
reproducibly) under solaris 5.7 on sparc machines.

New version of dict
+++++++++++++++++++

(This article is courtesy of Jean Thierry-Mieg mieg@ray.nlm.nih.gov)

Wonderful new version of the dict with stable memory that means after you
add in dict and get a name out of the dict that char* is stable until the
dict is destroyed. Whereas before it went out of scope if by chance the dict
was realocted by an implicit arrayExpand.

All the relocatable memory addresses are now fully hidden from the user by
the modified obscure dict.h

Truly, it is a tremendous improvement !!!

Some messubs changes
++++++++++++++++++++

messcrash message format: The format of the message now contains
information about the program running, userid etc. in a more easily
parsed/uniform way:

// FATAL ERROR (ruid: XXX, euid: XXX, nodeid: XXX, program: XXX, version: XXX,
file: XXX.c, line: XXXX)
   -  Unable to update database/database.map file (system error 13 - Permission
denied)

messcrash optional abort() call: You can control whether messcrash calls
abort() or not using the messAbortOnCrash() routine prior to the messcrash
call. This call is used in the implementation of the messAssert() macro
described below.

A safer hprintf(): hprintf() (and other XXprintf style routines) have been
a problem in Standard C because there was no way to tell whether you had
allocated a big enough buffer to hold the string that they produce. Some
operating systems and the latest version of the C standard library overcome
this to some extent with XXnprintf versions of the calls which print at most
"n" characters. Glib already provides these type of calls, e.g.
g_vsnprintf(), and hprintf() now makes use of this.

Some messNNN macros(): The following macros have been added to regular.h
which should help with producing concise but error-checking code:

messCheck()
     allows more succint (one line) checking of whether calls such as
     printf() worked. Can be used in either of these formats:

     messCheck( var = func(arg1, arg2), cond) ;
     messCheck( func(arg1, arg2), cond) ;

     Note that you don't have to assign the function result to a variable
     and that "cond" should be a simple equality test like != 0 or == NULL.
messAssert()
     this macro works just like the standard C library assert() but calls
     messcrash() instead of directly calling abort().

If the checks fail then messcrash is called with a string which shows the
condition that failed.

Graph package changes
+++++++++++++++++++++

You can now set the graph line style via the graphLineStyle() call to either
GRAPH_LINE_SOLID or GRAPH_LINE_DASHED.

There is a new graphBoxSetDim() function which you can use to set the
dimensions of a box, this enables you to do things like draw a single pixel
wide line but in a box that is several pixels wide so that the user can more
easily click on the line (this is used in the homology block line drawing
code so that the user can click on the lines between homology boxes to
select that particular homology match.

SMap changes
++++++++++++

The smap code now makes use of the NO_OVERLAP_EXTERNAL status flags to
detect which end of a feature has been clipped if the feature extends beyond
the end of the current smap. We can make use of this to draw at least part
of very long features that extend beyond the ends of the current FMap
context for instance.

Improvements to AceIn/AceOut package
++++++++++++++++++++++++++++++++++++

The AceIn/Out packages could be expensive to use in the case where the
caller wanted to replace the text in an existing AceIn/Out object, to do
this required destroying the existing object and recreating a new one with
the new text. This was expensive for instance in the server where new
commands are constantly read in and new replies sent out.

Two new calls have been added which allow replacement of existing text with
new text in an existing AceIn or AceOut object. The calls are:

void aceInSetNewText(ACEIN fi, char *text, char *params) ;

void aceOutSetNewStack(ACEOUT fo, Stack s) ;

-----------------------------------------------

February monthly build now available.
*************************************

You can pick up the monthly builds from:

Sanger users
     ~acedb/RELEASE.DEVELOPMENT
External users
     http://www.acedb.org/Software/Downloads/monthly.shtml

-----------------------------------------------

Next User Group Meeting - D319, 3.00pm, Thursday, 13th March, 2003
******************************************************************

----------------------------------------------------------------------------
----------------------------------------------------------------------------
Ed Griffiths <edgrif@sanger.ac.uk>
Last modified: Thu Mar 6 18:10:38 GMT 2003