🚀 go-pugleaf

Re: Responses to 'sample split question'

Since I received so many requests for the responses to the 'sample split'
question I decided to post them to the net instead of sending them out
individually.  Several responses were "second responses", coming after I told
the respondent that (1) no trend or pattern could be seen in the scatterplot
and (2) regression approaches would need to deal with the fact that the
x-variable, in this case test score, was not measured without error (the
"errors in variables" problem.

The responses are slightly edited, mostly to save space.  I included
"signatures" when available.  My thanks to all respondents.
------------------------------------------------------------------------
The question:

I'm helping a research psychologist who wants to show that a questionnaire
score is predictive of the degree of blood cholesterol lowering.
The questionnaire measures a person's belief in their ability to
to adhere to a cholesterol lowering-diet.  The psychologist wants to use
a technique that I am unfamiliar with.  She wants to use the median
questionnaire score to divide the sample into a lower 50% group
and an upper 50% group, then calculate the mean blood cholesterol lowering
for each group and compare them using the t-test.  After doing this
we get the following:

            MIN  MAX  RANGE  N   MEAN CHL LOWERING  MEAN TEST SCORE
lower group  19  104   85   240       6.1                90
upper group  105 135   30   249      11.9                117

(t-test significance < .000)

Would you be convinced if you saw this presentation?  Are you bothered
by the different ranges in the two groups?  If I told you that the
correlation coefficient is 0.19 (p<.000) would you change your mind.
========================================================================
The Responses:
========================================================================
From: cole@waldron.stanford.edu

Median-splitting the sample seems like a terrible waste of information.
Why not simply regress the choleosterol change scores on the test scores?
Median splits treat people at the 1st percentile on the test as being
equivalent (for all intents and purposes) to people at the 49th percentile,
while those at the 49th percentile are treated as somehow very different
than peple at the 51st percentile (who are considered similar to people at
the 99th percentile, etc.).  Also, techniques for dealing with skewed
distributions on the test scores present themselves naturally within a
regression context (e.g., transformations).  In addition, since the
psychologist admits interest in "prediction," the standard errors of
measurement, coefficient of determination, etc. which emerge naturally from
a regression would be of some interest.  I guess the main point is simply,
why dichotomize a variable we know is continuous when the techniques for
treating the continuous variable are more helpful in the first place?

[second response]
Ah, the old errors in variables problem.  Unfortunately, there's probably
not much you can do unless you have some other measure of the construct you
are interested in.  (If you do, then there are a host of techniques for
generating estimates of a "true" score from multiple indicators of the
construct you are interested in.)  But cheer up!  The errors in variables
approach is just as problematic for a 2-sample t-test (which is equivalent
to a regression on a dummy variable indicating group membership), so you
haven't really escaped anything by doing a median split.  In fact you're
better off because you haven't made the huge error of estimating the "true"
score for a datum at the 1st percentile as equivalent to the "true" score
at the 49th percentile, which is what you do in a median-split 2-sample
test.  Unless you have another predictor of chol-change besides the test
score, there's not much you can do.  For more on the problem in general,
see any good regression primer (I like Neter, Wasserman, & Kutner, Applied
Statistical Linear Models, published by Irwin).  If you're comfortable with
matrix algebra, try Seber's Linear Regression Analysis, published by Wiley.
You might also check out a paper by Davies and Hutton, The effects of
errors in the independent variables in linear regression, Biometrika, 62,
383-391.  That's a 1975 paper I believe, so there is probably some more
recent work that I'm not up to date on.  But if you have only one
predictor, there's no way of estimating a "true" score independently of the
"observed" score, so there's no real hope of debiasing.  Of course, in
practice, most of us just ignore the errors in the regressors and assume
that no disasterous bias has been introduced.  You could flip the equation
around to predict test scores from chol-change and take a look at the size
of the residual mean squares compared to the regression mean squares as a
VERY loose heuristic for how much variability test scores does not relate
to variability in chol-change.  Of course I would never really mention this
to anyone else, since now you assume chol-change is perfectly measured, but
it might be consoling to you and/or your client to have some notion about
how much "error" there really is in the test scores.  Let me know if
somebody else has a better idea about errors in variables, but with only
one estimate of the "variable", I have no idea what else one could do.

Probably you'll end up arguing 1) there's nothing else we can do without
more information, and 2) noise in the IV's will in general work AGAINST
finding a relationship with the DV.

======================================================================== 53
From:         "Michael H. Prager" <mprager@PONTUS.SPAN>

This seems to be a variant of a method proposed by Wald (1940) for
an unbiased slope estimate when the explanatory variable is measured with
error.  According to Thiel (1971, p. 610ff.), Wald  "divides the n
observations into two groups, one consisting of the n/2 observations with
the largest values of the explanatory variable and the other containing the
smallest values....  The slope of the straight line which connects the
centers of gravity of the two groups is the estimator of beta."  He then
goes on to discuss properties of this estimator, and to discuss further
research along this line, the most important of which seems to imply that
the data should be divided into 3 groups and the middle one discarded.  I
recommend you look at Thiel's book, which has a much better treatment of
this question than most introductory or mid-level statistics books I am
familiar with.

I don't know what's been done about testing hypotheses on beta.  I think
that bootstrapping would provide a simple nonparametric hypothesis test.

Thiel, H.  1971.  Principles of Econometrics.  John Wiley & Sons.

Wald, A.  1940.  The fitting of straight lines if both variables are
subject to error.  Ann. Math. Stat.  11 284-300.

*====================================================*
*  Michael H. Prager                                 *
*  U.S. National Marine Fisheries Service            *
*  Southeast Fisheries Science Center                *
*  75 Virginia Beach Drive                           *
*  Miami, Florida  33149  USA                        *
*  Internet:    mprager%pontus.span@umigw.miami.edu  *
*  Telephone:   305-361-4218  (FTS 350-1218)         *
*====================================================*
======================================================================== 30
From:         Ronan M Conroy <RCONROY@IRLEARN.UCD.IE>

The researcher seems to be replacing a general hypothesis -
that there is a relation between belief in your ability to
lower your cholesterol and you success at doing so - with only
one of a number of operationalizations that are possible: that
mean cholesterol drop in those who scored above-median would
be greater than those who scored below-median.
What about exploratory data analysis? Is there a linear relationship,
that might be summarized using a regression? Are those who believe
in their power to control their diet more consistent in the
degree of reduction? (Is there any relationship between initial
cholesterol and the measure of confidence?). My guess is that the
psychological measure doesn't behave like a linear scale, but maybe
there is a score above which you have 'enough sense of control' and
variation above that doesn't matter.
I'm just strewing the page with possible interrelations that would
be of interest, that might exist and that haven't been checked
for. What I'd do first, though, is look at a scatterplot.

======================================================================== 58
From: spector@stat.Berkeley.EDU (Phil Spector)

While there's nothing wrong with this as an exploratory technique when
no other method is available, I don't think it's the technique which
provides the most insight into the problem at hand.  If you had administered
five different tests and wanted to know which ones to continue studying, I
would advise you to concentrate on those tests for which the t-test was
significant.  The significant value is certainly indicative of something.
But in terms of an in depth study, it would be of interest to know if the
effect of belief is consistent across the scale, that is, does a test score
of 130 really mean anything more than a test score of 110, or does the effect
level out.  Similarly on the low range.  By arbitrarily grouping the subjects
this way, that information is lost.

You imply that a correlation coefficient has been calculated, indicating
that linear regression has been considered.  This may be a good start, but
since there are only two variables here (as far as I can see, anyway),
namely test score and cholesterol lowering, a simple scatterplot will
give you more insight into what's going on than any statistical test.  I
would personally be more convinced of what's going on from the
scattergram than the t-test.

With a sample of around 500 observations, if I wanted a statistical test
which wasn't entirely dependent on linearity, I would divide the data
into, say five groups, and perform an analysis of variance, calculating
linear, quadratic and perhaps cubic orthogonal contrasts on the effect
of test scores to get more insight into the exact nature of the
relationship.  You don't report on the range of CHL lowering in the
two groups, but a few very large drops in CHL for the upper group might
have a major effect on the t-test.  This would be somewhat reduced in
the ANOVA approach, and, of course, a non-issue for the scatterplot.

[second response]
If you can't see a discernable pattern in the scatterplot, I would
suggest that the t-test difference is an artifact, possibly caused
by a few unusual observations.  I'm personally not too concerned
about the error on the test score - it's the only measure you've got
and with 500 subjects if a trend emerges, I think it can be trusted.
Good luck!                             - Phil Spector
                                         Statistical Computing Facility
                                         Department of Statistics
                                         UC Berkeley
                                         spector@stat.berkeley.edu
======================================================================== 18
From:        KNIGHT@unb.ca

No answers - only questions.

(1) How about looking at a scatterplot?

(2) How about looking at histograms of the CHL lowering for both
groups?

(3) Since a regression is a cinch to give a better prediction than
the two valued predictor proposed, (Predict 90 for one group, 117 for
the other.) I'm blamed if I see any reason to go the proposed road.
HOWEVER, the client may have some good reason reason to look at the
data that way; though discovering this reason is likely to need both
persistance and tact.
======================================================================== 21
From:         Tom Backer Johnsen <backer@CC.UIB.NO>

The procedure of (a) making a dichotomy from a continuous variable like the
scores  on a questionnaire and then (b) using this variable as an independent
variable in a t-test is in my view wrong.  For one thing you are basically
throwing out information in order to be able tp perform the t-test in this
manner.  If you compute the signifcance of the normal correlation coeff.
you use more of the information.  Secondly, you could have a non-linear
relation between the two variables, that is ignored in the test you pro-
pose.
Tom Backer Johnsen
Psychometric Unit
University of Bergen
NORWAY
======================================================================== 36
From:         R.A.Reese@HULL.AC.UK

I agree with the comment that forming a divide into upper and lower halves
loses information. And looking at the linear correlation (I presume of the
raw questionnaire score and the cholesterol drop), if that was converted
to a regression, it would explain 3.6% of the variation in cholesterol drop.
I guess that someone pointed out to the psychologist that cholesterol was
measured on a ratio scale but the questionnaire scores are at best ordinal,
i.e., don't form a metric. So the linear correlation doesn't necessarily
mean anything. In any case, I'd rather see the points plotted.

What is not made clear is the underlying question. If it is purely "that
the higher the score on the questionnaire, the higher will be the
cholesterol drop", the scatterplot should convince the researcher one way
or the other, and only if it is not clear-cut is there a real need for a
"significance test". (However, journals and especially referees do seem to
still crave the odd p.) The t-test idea is about the weakest answer to the
question. You could split the sample into, say, deciles, and apply one-way
anova with a test for linearity. But I'd also look at the scatterplot with
an idea of an alternative - "the higher the questionnaire score, the larger
the POTENTIAL cholesterol drop". This would express itself in increased
variance, or in the appearance of outliers (suppose just a few higher
motivated individuals achieve massive drops, but most people achieve just
a small drop). Mechanical statistics says "if heteroscedastic, apply
trasform". I suggest "whatever the data, look at the data and apply thought".

In general, whenever faced with a rsearcher who says "I've got to apply an
XYZ test to this data", I ask "Why? Why that test? What would it show that
several other tests would not also show - and if this test gives different
answers why should I believe this one, that one or any?"
======================================================================== 23
From:         PLOCH@UTKVX.BITNET

Following Reese's tack one could split the sample into 3 or more
categories based on the questionnaire score and within each
category create box and whisker displays of the cholesterol drop.
This would show the median and inter-quartile range allowing some
estimate of linearity and heteroscedasticity.

Don  (BITNET: Ploch@UTKVX)
     (INTERNET: PLOCH@UTKVX.UTK.EDU)
Sociology, University of Tennessee, Knoxville TN 37996
(615)974-7022
======================================================================== 51
From:         "Karl L. Wuensch" <PSWUENSC@ECUVM1>

     It is hard to evaluate your question without additional data.  Why
did your associate want to dichotomize the questionnaire scores?  Why not
just do a Pearson r, Spearman, Kendall tau, or other such correlation
coefficient between the questionnaire scores and the cholesteral scores?
Does a scatterplot show a nonlinear effect?  A nonmonotonic effect?  For
the t-test done, do the within-group distributions appear normal?  Is there
great heterogeneity of variance and if so were df corrected therefor?
In my experience psychologists often needlessly categorize quantitative
variables.

[second response]
     You might want to suggest that your colleague present both the t-test
(having dichotomized one variable), with the seemingly interesting means,
and the Pearson r (leaving both variables continuous).  It might also be
instructive to compute a point biserial r between the dichotomized variable
and the one left continuous.  I wonder if the variance accounted for with
the t-test/point biserial is much different from that with the standard
Pearson r.

      Karl L. Wuensch, Dept. of Psychology, East Carolina Univ.
      Greenville, NC  27858-4353, PSWUENSC AT ECUVM1 (BITNET)
======================================================================== 29
From: Ed Haupt <haupt@pilot.njin.net>

Please! do a scatter plot first, then figure out what makes sense.  The
large N makes the result trivial. R without a scatterplot, doesn't
say anything, and the negative skew makes everything weird.

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Ed Haupt                |                         | OFFICE (201) 893-4327
Dept. of Psychology     |      "Kein Bier,        |   HOME (201) 783-0561
Montclair State College |        keine Arbeit."   | INTERNET:haupt@pilot.
Montclair, NJ 07043     |                         |            njin.net
              USA       |                         | BITNET:Haupt@njin
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
========================================================================
-----------------------------End of File--------------------------------

+---------------------------------+-----------------------------------+
| Richard Leickly                 | Internet:richard@u.washington.edu |
| Northwest Lipid Research Clinic | University of Washington          |
| Seattle, Washington   98195     | Mailstop: ZA-36  Ph:206-223-6267  |
+---------------------------------+-----------------------------------+