🚀 go-pugleaf

Thomas.L.Mead@dartmouth.edu (Thomas L. Mead) wrote:
>This is posted to   "sci.stat.consult"   and   "medlib-l"  and 
>"bit.listserv.asis-l"
>
>
>Sometimes, when I read the METHODS sections of published meta-analysis
>research (in any field), I'm disappointed by what I consider to be
>cursory or inadequate descriptions of the literature search which
>precedes the statistical work.   ("We searched MEDLINE..." just doesn't
>say nearly enough to me!)

This is a very important point. You are quite correct that the more 
detailed the description of the search process and selection criteria, 
the better and more useful the meta-analysis can be.

>Working with these projects is interesting -- indeed exciting -- and
>when I've been a part of an M-A team I have taken my role very
>seriously.  I don't understand the statistics work; I do understand
>medical literature searching and the library support-issues involved in
>large M-A projects.  

>My questions:

I highly recommend that you look into a book called _Methods of 
Meta-Analysis_ by John E. Hunter and Frank Schmidt. I think it came out 
about 1990 or so. It discusses many of the issues you ask about below and 
also gives explanations and examples of the relevant mathematics. Some 
background in basic stats and math is helpful, but you can read the book 
for content then loan it to whichever statisticians you work with if they 
haven't seen it.

>==> Is a published meta-analysis paper to be considered adequate if it
>gives an inadequate explanation of the literature search (the
>resources, the techniques, strategies, dates & files & details galore)
>...?

I wouldn't consider it adequate without such details.

>==> Without an adequate & detailed description of the search, a M-A
>study would not be repeatable/verifiable, etc.    Is that important ...
>?

Definitely. If someone cannot hunt down all the sources if they think 
your analysis needs to be repeated with different criteria, then your 
work is sloppy. I would go so far as to provide an appendix with a list 
of studies on the same topic that were excluded, with a short code 
denoting the reason for exclusion.

>==> Without an adequate & detailed description of the search, can peers
>really competently review a M-A study?  Wouldn't they be limited to
>critiquing statistical methods ... ?

Depends how well the reviewers know the topic of interest. Typically, a 
journal editor will send articles to people skilled in the area of the 
article for peer review prior to accepting or rejecting the paper for 
publication.

>==> Is it not true that the SELECTION of which studies to submit to M-A
>techniques is important?  Doesn't the selection process begin with the
>literature search itself?  (I can easily NOT GET lots of stuff...)  If
>that process if faulty, does it spell doom for the subsequent work?

Yes, the search is the place selection starts. Probably the most likely 
error in selection is failure to search in the journals and indexes of 
other disciplines that might address the same problem but which do not 
normally cross paths in terms of contact at the level of which journals 
people read on a regular basis.

-- 
Steven J. Pierce
Master's Student in Experimental Psychology
Mississippi State University
E-Mail: sjp2@ra.msstate.edu
WWW: http://www2.msstate.edu/~sjp2/index.html