🚀 go-pugleaf

Thomas L. Mead wrote:
>> 
> My questions:
> 
> ==> Is a published meta-analysis paper to be considered adequate if it
> gives an inadequate explanation of the literature search (the
> resources, the techniques, strategies, dates & files & details galore)
> ...?
>

You would probably find more discussion of these points on the evidence-based-health
mailing list (at mailbase in the UK). There are guidelines for "systematic reviews"
produced by the Cochrane Collaboration and various other people which do point out how
important the initial selection of papers is. In particular, restricting an analysis
to just those papers found in a MEDLINE search is not a highly regarded approach:
MEDLINE searches, even conducted by experts, miss an enormous number of studies.

The Cochrane Collaboration has a Web site at
http://hiru.mcmaster.ca/cochrane/default.html

In regard to your questions about the need to report search strategies and so on, here
is a quote from the Cochrane Collaboration handbook.

------------
As with any research study, a number of methodological decisions must be made when
undertaking a systematic review, and there are potential threats to validity
associated with each decision. These include: how studies are identified and selected
for inclusion, how their quality (validity) is assessed, how much weight each one is
given, and how their results are combined and interpreted. 

If reviewers do not say how these decisions were made, there is no way of knowing how
much faith to put in their conclusions.
-------------

There is a lot more similar information and recommendations in the handbook, which is
available online at the Web site.

Hope this is helpful

thomas lumley
University of Washington

<Article 17271 of 17292 (sci.stat.consult) (57 lines) Wed Apr 10 21:20:17 1996
<Subject: Meta-analysis literature searches
<From: Thomas.L.Mead@dartmouth.edu (Thomas L. Mead)

<Sometimes, when I read the METHODS sections of published meta-analysis
<research (in any field), I'm disappointed by what I consider to be
<cursory or inadequate descriptions of the literature search which
<precedes the statistical work.



Your concerns are well founded and I believe outlined in 

L'Abbe KA, Detsky AS, O'ROURKE K.  Meta-analysis in clinical research. Annals of Internal Medicine
107:224-33, 1987.

More current references should also talk about study registries so that searches are less critical.  

It might not be a silly idea to "insist" that all published meta-analysis include a 1-800 number and a
request that anyone aware of any study, published or not, call and leave a message.  

<Working with these projects is interesting -- indeed exciting -- and when I've been a part of an M-A
<team I have taken my role very seriously.  I don't understand the statistics work;

You may wish to look at the following for some further encouragement (the critical issues in a
metaanalysis are usually not statistical)  


O'ROURKE K, Detsky AS.  Meta-analysis in Medical Research:  Strong encouragement for higher
quality in individual research efforts.  Journal of Clinical Epidemiology 42(10):1021-1024, 1989.

Keith O'Rourke
The Toronto Hosp.

Thomas.L.Mead@dartmouth.edu (Thomas L. Mead) wrote:
>This is posted to   "sci.stat.consult"   and   "medlib-l"  and 
>"bit.listserv.asis-l"
>
>
>Sometimes, when I read the METHODS sections of published meta-analysis
>research (in any field), I'm disappointed by what I consider to be
>cursory or inadequate descriptions of the literature search which
>precedes the statistical work.   ("We searched MEDLINE..." just doesn't
>say nearly enough to me!)

This is a very important point. You are quite correct that the more 
detailed the description of the search process and selection criteria, 
the better and more useful the meta-analysis can be.

>Working with these projects is interesting -- indeed exciting -- and
>when I've been a part of an M-A team I have taken my role very
>seriously.  I don't understand the statistics work; I do understand
>medical literature searching and the library support-issues involved in
>large M-A projects.  

>My questions:

I highly recommend that you look into a book called _Methods of 
Meta-Analysis_ by John E. Hunter and Frank Schmidt. I think it came out 
about 1990 or so. It discusses many of the issues you ask about below and 
also gives explanations and examples of the relevant mathematics. Some 
background in basic stats and math is helpful, but you can read the book 
for content then loan it to whichever statisticians you work with if they 
haven't seen it.

>==> Is a published meta-analysis paper to be considered adequate if it
>gives an inadequate explanation of the literature search (the
>resources, the techniques, strategies, dates & files & details galore)
>...?

I wouldn't consider it adequate without such details.

>==> Without an adequate & detailed description of the search, a M-A
>study would not be repeatable/verifiable, etc.    Is that important ...
>?

Definitely. If someone cannot hunt down all the sources if they think 
your analysis needs to be repeated with different criteria, then your 
work is sloppy. I would go so far as to provide an appendix with a list 
of studies on the same topic that were excluded, with a short code 
denoting the reason for exclusion.

>==> Without an adequate & detailed description of the search, can peers
>really competently review a M-A study?  Wouldn't they be limited to
>critiquing statistical methods ... ?

Depends how well the reviewers know the topic of interest. Typically, a 
journal editor will send articles to people skilled in the area of the 
article for peer review prior to accepting or rejecting the paper for 
publication.

>==> Is it not true that the SELECTION of which studies to submit to M-A
>techniques is important?  Doesn't the selection process begin with the
>literature search itself?  (I can easily NOT GET lots of stuff...)  If
>that process if faulty, does it spell doom for the subsequent work?

Yes, the search is the place selection starts. Probably the most likely 
error in selection is failure to search in the journals and indexes of 
other disciplines that might address the same problem but which do not 
normally cross paths in terms of contact at the level of which journals 
people read on a regular basis.

-- 
Steven J. Pierce
Master's Student in Experimental Psychology
Mississippi State University
E-Mail: sjp2@ra.msstate.edu
WWW: http://www2.msstate.edu/~sjp2/index.html

Tom Mead's important questions about reference retrieval procedure 
descriptions and completeness for meta analysis have received 
appropriate responses from several in this thread.  From the point 
of view of an Information Science researcher I would re-emphasize 
several points: Clearly, meta-anaylsis or any analysis suffers to 
extent relevant references are ignored.  Retrieval recall is often 
quite low (e.g., 50-95% of relevant documents missed) and 
generally suffers if potentially relevant sources are ignored 
(databases, multiple document and indexing types -- including grey 
literature; full, free-text indexing; etc.).

I was appreciative of pointers to the Cochrane Collaboration 
(CC)efforts which made some of these points. (Note: Bill Hersh's 
URL to the CC web home page -- http://hiru.mcmaster.ca/cochrane -- 
works whereas Thomas Lumley URL to the [included] CC Handbook, 
with much of the relevant material, was faulty.)  I was unable to 
get access on the web to Appendix 5 of Section 5 of the Handbook 
entitled "Optimal MEDLINE search strategy for identifying <RCT 
references>."  Perhaps this appendix explains what is not fully 
elucidated in the body of the Handbook:  a full description of a 
search strategy must include the specification of details such as 
search field (title, abstract, full text, index terms --- e.g., 
Mesh -- etc.), term proximity, weighting, stemming, truncation, 
proximity, etc., as well as full Boolean search structure for each 
database searched.  Also, one can manipulate these parameters to 
modify recall/precision balance.

Our own research at MIT on intelligent computerized search 
assistant systems supports these various points.  A particularly 
relevant point in this respect is that it is now possible with a 
technique known as "Broadening by Dropping Factors (BDF)" to make 
good estimates of the number of relevant documents not yet found 
in the search, as well as indicate how to capture the missing 
documents.  BDF employs sampling of purposely (overly) broadened 
searches for new relevant documents.  To the extent such 
techniques prove reliable and are incorporated into operational 
search systems we, as a society, can insist analysts of all types 
should attain reference recall up to some given level.

Reference: Richard Marcus, "Intelligent Assistance for Document 
Retrieval Based on Contextual, Structural, Interactive Boolean 
Models," Proceedings of the RIAO 94 Conference, Volume 2, pps 27-
33, Center for Advanced Study of Information Systems (CASIS), New 
York, NY,  October 11-13, 1994. ISBN 2-905450-05-3.

Richard S. Marcus, Prin Rsch Scientist, Lab for Information and Decision
Systems, MIT, Cambridge, MA 02139-4307. marcus@LIDS.MIT.EDU

< tom.mead@dartmouth.edu >
Thomas L. Mead (Thomas.L.Mead@dartmouth.edu) wrote:

: I've done some work with medical meta-analysis ("M-A") studies, where
: my job is to perform the literature search (usually rather vast &
: comprehensive & intentionally overbroad).   Then I do some
: post-processing of the results, such that references are in a database
: program.  I also get involved with filtering these results (determining
: which studies gotten from the literature search are likely to be
: subjected to M-A techniques...)

: Sometimes, when I read the METHODS sections of published meta-analysis
: research (in any field), I'm disappointed by what I consider to be
: cursory or inadequate descriptions of the literature search which
: precedes the statistical work.   ("We searched MEDLINE..." just doesn't
: say nearly enough to me!)

Meta-analysis in the area of psychiatric treatment seems to suffer 
from two problems  *besides*  what you mention, both of which make the
outcomes ABSURD, rather than just incomplete.

 -  The Treatments in the study do not match the stated intent.  
 -  The populations under study differ vastly, but that is not even
          mentioned.
 -  The outcomes are not firmly defined.

The only decent medical meta-analyses that I think of, off-hand, 
are the ones that follow the prescription of using SIMPLE 
treatment definitions, and SIMPLE outcome (namely, Death within
a stated interval).

Oh, and the performance of various meta-analyses is often carried 
out without confirming that the results on hand are homogenous 
in a  *statistical*  sense.  That is, the SAME treatment will not 
legitimately show a 1 S.D.  effect in BOTH directions, in two 
different studies that legitimately can be combined for an average.

(I don't know if this helps you directly, but there is a message
here, that at least a few of us have a lot of *scorn*  for what
some meta-analyses have pretended to show us...   just another
inept literature review, but decorated with numbers).



Rich Ulrich, biostatistician              wpilib+@pitt.edu
Western Psychiatric Inst. and Clinic   Univ. of Pittsburgh

<(I don't know if this helps you directly, but there is a message
<here, that at least a few of us have a lot of *scorn*  for what
<some meta-analyses have pretended to show us...   just another
<inept literature review, but decorated with numbers).



<Rich Ulrich, biostatistician              wpilib+@pitt.edu
<Western Psychiatric Inst. and Clinic   Univ. of Pittsburgh

Perhaps what such metaanalyses actually show is that the individual 
studies can not be interpreted ... not an inept literature review 
but a inconclusive quantitative review of inept research (and research 
reporting). 

If you can't interpret studies as a group how can you interpret them 
individually?

If you want to be able to interpret individual studies conduct and report 
them so that they can be conclusively metaanalized.

Keith O'Rourke
The Toronto Hosp.