🚀 go-pugleaf

Tom Mead's important questions about reference retrieval procedure 
descriptions and completeness for meta analysis have received 
appropriate responses from several in this thread.  From the point 
of view of an Information Science researcher I would re-emphasize 
several points: Clearly, meta-anaylsis or any analysis suffers to 
extent relevant references are ignored.  Retrieval recall is often 
quite low (e.g., 50-95% of relevant documents missed) and 
generally suffers if potentially relevant sources are ignored 
(databases, multiple document and indexing types -- including grey 
literature; full, free-text indexing; etc.).

I was appreciative of pointers to the Cochrane Collaboration 
(CC)efforts which made some of these points. (Note: Bill Hersh's 
URL to the CC web home page -- http://hiru.mcmaster.ca/cochrane -- 
works whereas Thomas Lumley URL to the [included] CC Handbook, 
with much of the relevant material, was faulty.)  I was unable to 
get access on the web to Appendix 5 of Section 5 of the Handbook 
entitled "Optimal MEDLINE search strategy for identifying <RCT 
references>."  Perhaps this appendix explains what is not fully 
elucidated in the body of the Handbook:  a full description of a 
search strategy must include the specification of details such as 
search field (title, abstract, full text, index terms --- e.g., 
Mesh -- etc.), term proximity, weighting, stemming, truncation, 
proximity, etc., as well as full Boolean search structure for each 
database searched.  Also, one can manipulate these parameters to 
modify recall/precision balance.

Our own research at MIT on intelligent computerized search 
assistant systems supports these various points.  A particularly 
relevant point in this respect is that it is now possible with a 
technique known as "Broadening by Dropping Factors (BDF)" to make 
good estimates of the number of relevant documents not yet found 
in the search, as well as indicate how to capture the missing 
documents.  BDF employs sampling of purposely (overly) broadened 
searches for new relevant documents.  To the extent such 
techniques prove reliable and are incorporated into operational 
search systems we, as a society, can insist analysts of all types 
should attain reference recall up to some given level.

Reference: Richard Marcus, "Intelligent Assistance for Document 
Retrieval Based on Contextual, Structural, Interactive Boolean 
Models," Proceedings of the RIAO 94 Conference, Volume 2, pps 27-
33, Center for Advanced Study of Information Systems (CASIS), New 
York, NY,  October 11-13, 1994. ISBN 2-905450-05-3.

Richard S. Marcus, Prin Rsch Scientist, Lab for Information and Decision
Systems, MIT, Cambridge, MA 02139-4307. marcus@LIDS.MIT.EDU