Chapter 15: Meta-Analysis
Chapter Outline
Outline Concepts
I.
Meta-Analysis as a Method of Summarizing
meta-analysis:
a tool by which quantitative
Quantitative Research Literature
results from previous studies are combined
--Meta-analysis is an effort to provide a way
to summarize research findings by looking at
the actual statistical
effects found in
quantitative research studies.
A.
Contrasting Standard Literature Review
Argument with Meta-Analysis Evidence
--alternatives
· Primary analysis is produced by single primary analysis: “the original analysis
studies
of data in a research study” (Glass,
1976, p. 3)
· Secondary analysis involves returning secondary analysis: “the re-analysis of
to past studies and reanalyzing their data. data for the
purpose of answering the
original research question with better
statistical techniques, or answering new
questions with old data” (Glass, 1976, p. 3)
·
Narrative reviews
are the most typical
narrative reviews: reviews of literature
forms of literature reviews published in in which published
material (typically) is
the field. They provide nonquantitative examined and nonquantitative
analyses of past research. assessments are
made about the state of
--strengths: 1. narrative reviews rely
on research.
strong inductive arguments that reason
from past research to general
conclusions, and that suggest theoretic
directions for future research. 2. Though
most narrative reviews emphasize
quantitative studies, qualitative studies
also may be considered.
--weaknesses: 1. narrative reviews
completed by researchers with strong
biases or poor reasoning skills may not
draw sound conclusions. 2. Researchers
using narrative reviews sometimes reach
conclusions that differ from each other.
In contrast, meta-analyses examine
many quantitative studies to arrive at
conclusions about a body of research.
B.
The Problem Statement in the Modern
Meta-Analysis Study:
1. Research questions that ask about
strength of relationships;
2. Research questions that ask about
comparisons of results from studies
of different designs;
3. Research questions that ask about
the impact of moderator variables;
4. Research questions that ask about
how well different theories can
explain phenomena explored in past
studies.
II. Assumptions for a Meta-Analysis Study
Meta-analyses can only look at empirical
studies (descriptive empirical studies and
experiments). The studies must be
reported in enough detail to permit the
researcher to compute effect sizes. To
complete a meta-analysis, you need to
know such things as:
· sample sizes
·
actual effect sizes—or there must
be
access to all the coefficients necessary to
compute the effect sizes.
· reliability for measured variables.
B. In meta-analyses, the empirical studies independence: the statistical assumption
must be independent that groups, samples, or (in meta-
analysis) studies are unaffected by each
other.
C. Studies must be comparable
III. Steps in Meta-Analysis
--After stating the problem question,
researchers may examine effects by
looking at different ways of gauging
effects:
·
For studies dealing with continuous
variables, using the effect size of a
correlation r usually is preferred.
· When studies deal with categorical
independent variables, the d statistic
that looks at differences in means
often (but not always) is
recommended.
--Based on past research and theory, the
direction of the relationship is specified
in the hypothesis.
The hypothesis should not simply ask if
there is a relationship that is different
from zero. Given the sample sizes typically
used in meta-analysis studies, this approach
virtually guarantees that needs to be identified.
Instead, meta-analysis hypotheses often deal
with minimum effect sizes that deserve
attention. For instance, a hypothesis could be
stated as “Studies reveal slight or greater
positive relationship between teachers’ levels
of nonverbal immediacy and students’ recall
of information.” Since a minimum
correlation
of .20 (equal to explaining at least 4% of the
shared variance among variables) was
identified as “slight” in Chapter 12, support
would be claimed for the hypothesis only if
this minimum were reached.
A. Sampling Quantitative Studies.
--Deciding whether to include a study
sometimes depends on considering the
assumptions underlying the use of
meta-analysis.
1. The number of studies drawn from a
single
article or research report should
not be too great. Researchers usually
limit themselves to
sampling
no more
than two or three studies from the
same research article or report.
2. Studies
should not be included if they
are radically different from others.
--There
must be an argument to show
that the excluded studies were from
a different population than the
included studies.
a.
Finding
published and unpublished
studies
--There is controversy regarding
publication bias:
the tendency for
whether unpublished studies should research
publications to favor
be included, but the tendency for empirical studies reporting
publication bias
(the preference for
statistically significant effects
research journals to favor studies and to deny
publication to studies
with statistically significant results finding no
statistically significant
relationship between variables.
·
Finding published work: use of
standard tools (e.g., ComAbstracts
[(www.cios.org], Communication &
Mass Media Complete, and
PsychINFO may be used. For
speech and hearing science, the
indexing systems known as
PubMed and Linguistics and
Language Behavior Abstracts)
·
Finding unpublished work, such as
convention papers, researchers have
options such as the use of ERIC (also
published in abstracts called
Resources in Education); Dissertation
Abstracts International and Masters
Abstracts International permit
researchers to identify worthy
research that may not have made the
transition to formal publication; the
International Communication
Association provides access to full
texts of
papers presented at its
annual convention; archives of the
National
Communication Association
also contain special reports not
published
elsewhere; in addition, the
Communication Institute for Online
Scholarship
hosts ComWeb MegaSearch
and the website for this book includes
COMFile!, which lists communication-
based websites organized according
to
the categories of the National
Communication Association; a search
of individual
authors’ websites where
unpublished research papers may be
posted can be
searched.
b. Accounting
for the Possibility of
Unavailable Contrary Studies
--Researchers concern themselves with
weather they have identified enough
studies so that any undetected studies
would not make any difference. This
problem often is called the file drawer file drawer
effect: the tendency
effect and assumes that a researcher who
for studies that fail to find
completes a study and finds no significant relationships to
relationships usually tosses the study into remain unpublished and become
a file drawer, since there is little chance abandoned in hypothetical “file
that it can be published. drawers”
--Though there logically is no way to
be fail-safe number: the number of
sure that a body of unavailable studies additional
studies showing a zero
does not exist, computation of a fail-safe effect that
would have to exist to
number is an attempt to deal with this reverse the
reported relationship
problem. size pattern.
· The fail-safe number computes the
number of studies showing no
relationship that would have to
exist for the researchers to turn
back and conclude that there is, in
fact, no relationship between
variables
·
One piece of advice is that a
researcher should feel confident in
the sample of studies if the fail-safe
number (symbolized N_{L}) is at least
5N_{L} + 10.
· Criticism of the fail-safe number:
1. There is more than one way to
compute a fail-safe number (such
as the alternative method by
Iyengar & Greenhouse, 1988),
and the approaches produce
different numbers.
2. Omitted studies may not show
only “no differences,” but they
may show differences in the
opposite direction. If unreported
studies are in the opposite
direction from the average
reported effect size, the actual
fail-safe number could be lower
than reported.
B. Computing Relationship Sizes.
Choosing a Measure of Effects
1.. Examining mean differences statistics
--Hedges’ d
It sounds simple, but many studies
include more than two experimental
groups. Under such circumstances,
researchers need to decide which
experimental condition is the most
important one to compare with the
control group. Then a contrast may be
computed with this key comparison.
2. Examining effects in correlation form
--There are multiple methods to compute
effects expressed as correlations from
different sorts of test statistics.
--Sample correlations (r) often are corrected
for sampling bias in estimating population
correlations (ρ).
--Sample correlations often are corrected correction for attenuation in
for attenuation in reliability of measures. reliability: a method of adjusting
(Note: this matter of reliability observed correlation coefficients
coefficients really only applies to for imperfections in reliability of
measured variables. When it comes to the measurement of variables
experimental variables, it is not possible
to compute such reliability coefficients.
As a matter of convenience, researchers
simply assign experimental variables
reliability coefficients of 1.0. Though
test-retest measures can be taken for
demographic variables such as age, sex,
and academic class, most researchers
correcting for attenuation also assume
that these matters have reliability
coefficients of 1.0.
C. Identifying Essential Differences in
Studies:
Separating the Apples from the Oranges
--Since research rarely involves
simple replication
of past work, meta-analysts are wise to code
different characteristics of research studies and
to use methods called “focused comparisons” to
test whether there are any differences produced
in research effects.
1. Coding moderator variables
--coding typically includes: years
studies
were completed; whether they were
laboratory or field studies, numbers of
independent variables; soundness of
measurement.
2. Other Moderator Variables:
there are two
other ways researchers may identify study
characteristics.
a. Theoretic differences
b. Researchers might rely on archival and
historical sources.
c. Perceived study quality
--Though these quality ratings are not
objective, they are intersubjective,
intersubjectivity: the degree to which
means that individuals may
different researchers
with
share common agreement despite essentially different
beliefs draw
any differences in their individual essentially the
same
perceptions.
interpretations of the meaning of
observations.
D. Assessing Mean Relationship Sizes
--A decision must be made regarding whether
to weight studies by sample sizes.
If done, weighting by the inverse variance
weight is preferred.
--Interpreting mean effect sizes as correlation
or
standardized differences
E. Making Diffuse Comparisons diffuse comparisons of effect
--If the effect sizes are inconsistent, the
sizes: measures of the
researcher has reason to believe that there is homogeneity of effect
at least another variable that is making a sizes in meta-analyses.
contribution to the variance associated with
different study outcomes.
F. Making Focused Comparisons focused comparisons:
--Focused comparisons permit the researchers assessments in meta-analyses to
to explore the relationships between determine whether differences in
study effect sizes on one hand, and any other variables are related to
moderator variables that may have been
differences in the sizes of study
operating, on the other hand. effects
on primary variables.
1. Continuous Moderator Variables
·
As one indication, researchers
could
correlate effect sizes (expressed as
Fisher’s Z) with each of the continuous
measures.
·
Focused comparisons using a form of
trend analysis may be applied to the
continuous variables, such as experts’
ratings of the soundness of measurement
in the studies to be analyzed.
2. Continuous Moderator Variables
·
Using tied contrast weights to use
a
form of trend analysis
--If the researcher finds a statistically
significant focused comparison, the
researcher completes new diffuse tests of
the groups of effect sizes created by each
category level of the moderator variable.
If the diffuse tests reveal that the effect
sizes in each category still are not
homogeneous, the researcher concludes
that the critical moderator variable has
not yet been found. The process
continues with other moderator
variables in the search for category
groups with homogeneous effect sizes.
--If moderator variables are highly
correlated, then the contributions that
each one actually makes to explaining
differences in effect sizes will not be
clear.
IV. Assessing Meta-Analysis in Communication
Research
A. Advantages of Meta-Analysis
1. By
attempting to structure the
examination of study findings for
statistical analyses, researchers get
some distance from their own biases.
2. A sound meta-analysis is capable of
being replicated statistically by other
researchers.
3. The statistics of meta-analysis allow
studies with large sample sizes and
accompanying reduced sampling
error—those whose findings are most
likely to show stable results—to be
represented according to their degree
of reduced sampling error.
B. Disadvantages of Meta-Analysis
1. Meta-analyses
cannot study the
influences of moderator variables
unless researchers have already
decided to make them repeated
objects of research.
2. Meta-analyses may not be as valuable
as primary research efforts.
3. Meta-analysis do not promote
assessing conceptual issues
underlying the concepts of inquiry.
4. Meta-analyses are restricted to analyses
of quantitative research studies alone.
--Some attacks on meta-analysis are mistaken:
·
The charge that meta-analyses
combine studies
that are so fundamentally different that they
really are trying to equate “apples” with
“oranges” is a misunderstanding
(Response: in fact, meta-analysts code
differences in study methods to avoid
mixing “apples” and “oranges”).
·
The complaint that meta-analyses
typically combine studies of dubious
quality is an exaggeration (Response:
many meta-analysts screen research to
include only the best studies; the vast
majority of meta-analysts usually code
message quality and use focused
comparisons of effect sizes to assess
whether studies of different quality
produce different effect sizes).
·
The criticism that meta-analyses
are
biased in favor of published research
that shows statistical significance is
generally unfounded. (Response: not
only do researchers typically attempt to
find unpublished studies, but they
regularly test for the potential number
of conflicting studies that would be
required to reject the meta-analytic
conclusions they draw).
·
The assertion that meta-analyses
emphasize single-variable influences
and “main effects” is largely incorrect.
(Response: coding moderator variables
and conducting focused comparisons
helps researchers discover important
sources of interactions).