Up A Quiz for Chapter 15 Meta_Analysis Spreadsheet Meta-Analysis Activity Fail-Safe Numbers

                                                     Chapter 15:  Meta-Analysis

                                                              Chapter Outline

                               Outline                                                                    Concepts

I.   Meta-Analysis as a Method of Summarizing                    meta-analysis: a tool by which quantitative
     Quantitative Research Literature                                
results from previous studies are combined

      --Meta-analysis is an effort to provide a way

to summarize research findings by looking at
the actual statistical effects found in
quantitative research studies. 

A. Contrasting Standard Literature Review
Argument with Meta-Analysis Evidence


·  Primary analysis is produced by single                   primary analysis: “the original analysis

studies                                                               of data in a research study” (Glass,
            1976, p. 3)

·  Secondary analysis involves returning                    secondary analysis: “the re-analysis of

to past studies and reanalyzing their data.             data for the purpose of answering the
             original research question with better
             statistical techniques, or answering new
             questions with old data” (Glass, 1976, p. 3)

·  Narrative reviews are the most typical                  narrative reviews: reviews of literature  
forms of literature reviews published in                 in which published material (typically) is
the field. They provide nonquantitative                 examined and nonquantitative
analyses of past research.                                      assessments are made about the state of

--strengths: 1. narrative reviews rely on                 research.
strong inductive arguments that reason
from past research to general
conclusions, and that suggest theoretic
directions for future research. 2. Though
most narrative reviews emphasize
quantitative studies, qualitative studies
also may be considered.

--weaknesses: 1. narrative reviews
completed by researchers with strong
biases or poor reasoning skills may not
draw sound conclusions. 2. Researchers
using narrative reviews sometimes reach
conclusions that differ from each other.
In contrast, meta-analyses examine
many quantitative studies to arrive at
conclusions about a body of research.

B.   The Problem Statement in the Modern
Meta-Analysis Study:

1.   Research questions that ask about
strength of relationships;

2.   Research questions that ask about
comparisons of results from studies
of different designs;

3.   Research questions that ask about
the impact of moderator variables;

4.   Research questions that ask about
how well different theories can
explain phenomena explored in past

II.   Assumptions for a Meta-Analysis Study

  1. Meta-analyses can only look at empirical
    studies (descriptive empirical studies and
    experiments). The studies must be
    reported in enough detail to permit the
    researcher to compute effect sizes. To
    complete a meta-analysis, you need to
    know such things as:

·  sample sizes

·  actual effect sizes—or there must be
access to all the coefficients necessary to
compute the effect sizes.

·  reliability for measured variables.

B.   In meta-analyses, the empirical studies                      independence: the statistical assumption

      must be independent                                               that groups, samples, or (in meta-

                                                                                   analysis) studies are unaffected by each


C.  Studies must be comparable

III.  Steps in Meta-Analysis
--After stating the problem question,

researchers may examine effects by
looking at different ways of gauging

·  For studies dealing with continuous
variables, using the effect size of a
correlation r usually is preferred.

·  When studies deal with categorical

independent variables, the d statistic
that looks at differences in means
often (but not always) is

--Based on past research and theory, the

direction of the relationship is specified
in the hypothesis.

The hypothesis should not simply ask if

there is a relationship that is different
from zero.  Given the sample sizes typically
used in meta-analysis studies, this approach

virtually guarantees that needs to be identified.

Instead, meta-analysis hypotheses often deal

with minimum effect sizes that deserve

attention. For instance, a hypothesis could be

stated as “Studies reveal slight or greater
positive relationship between teachers’ levels
of nonverbal immediacy and students’ recall
of information.”
Since a minimum correlation
of .20 (equal to explaining at least 4% of the
shared variance among variables) was
identified as “slight” in Chapter 12, support
would be claimed for the hypothesis only if
this minimum were reached.

A.  Sampling Quantitative Studies.

      --Deciding whether to include a study

sometimes depends on considering the
assumptions underlying the use of

1.   The number of studies drawn from a

      single article or research report should
not be too great. Researchers usually
limit themselves to
sampling no more
than two or three studies from the
same research article or report.

2.   Studies should not be included if they
are radically different from others.

--There must be an argument to show
that the excluded studies were from
a different population than the
included studies.

a.   Finding published and unpublished

--There is controversy regarding                          publication bias: the tendency for
whether unpublished studies should                  
research publications to favor

be included, but the tendency for                      empirical studies reporting

publication bias (the preference for                  statistically significant effects
research journals to favor studies                      
and to deny publication to studies
with statistically significant results                    
finding no statistically significant

                                                                      relationship between variables.

·        Finding published work: use of
standard tools (e.g., ComAbstracts
[(www.cios.org], Communication &
Mass Media Complete,
PsychINFO may be used. For
speech and hearing science, the
indexing systems known as
PubMed and Linguistics and
Language Behavior Abstracts

·        Finding unpublished work, such as
convention papers, researchers have
options such as the use of ERIC (also
published in abstracts called
Resources in Education); Dissertation
Abstracts International
and Masters
Abstracts International
researchers to identify worthy
research that may not have made the
transition to formal publication; the
International Communication

Association provides access to full
texts of papers presented at its

annual convention; archives of the
National Communication Association

also contain special reports not
published elsewhere; in addition, the

Communication Institute for Online
Scholarship hosts ComWeb MegaSearch
and the website for this book includes
which lists communication-
based websites organized according
to the categories of the National
Communication Association; a search
of individual authors’ websites where
unpublished research papers may be
posted can be searched.

b.   Accounting for the Possibility of
Unavailable Contrary Studies

--Researchers concern themselves with
weather they have identified enough
studies so that any undetected studies
would not make any difference. This
problem often is called the file drawer                 file drawer effect: the tendency
and assumes that a researcher who             for studies that fail to find

completes a study and finds no                             significant relationships to

relationships usually tosses the study into               remain unpublished and become

a file drawer, since there is little chance              abandoned in hypothetical “file

that it can be published.                                      drawers”

--Though there logically is no way to be                  fail-safe number: the number of
sure that a body of unavailable studies                  additional studies showing a zero    
does not exist, computation of a fail-safe              effect that would have to exist to
number is an attempt to deal with this                  reverse the reported relationship
problem.                                                             size pattern.

·        The fail-safe number computes the

number of studies showing no
relationship that would have to
exist for the researchers to turn
back and conclude that there is, in
fact, no relationship between

·        One piece of advice is that a
researcher should feel confident in
the sample of studies if the fail-safe
number (symbolized NL) is at least
5NL + 10.

·        Criticism of the fail-safe number:

1.   There is more than one way to
compute a fail-safe number (such
as the alternative method by
Iyengar & Greenhouse, 1988),
and the approaches produce
different numbers.

2.  Omitted studies may not show
only “no differences,” but they
may show differences in the
opposite direction.  If unreported
studies are in the opposite
direction from the average

reported effect size, the actual
fail-safe number could be lower
than reported.

B.   Computing Relationship Sizes.

Choosing a Measure of Effects

1..  Examining mean differences statistics

      --Hedges’ d

It sounds simple, but many studies
include more than two experimental
groups. Under such circumstances,
researchers need to decide which
experimental condition is the most
important one to compare with the
control group. Then a contrast may be
computed with this key comparison.

2.   Examining effects in correlation form

      --There are multiple methods to compute

      effects expressed as correlations from
different sorts of test statistics.

--Sample correlations (r) often are corrected
for sampling bias in estimating population
correlations (ρ).

--Sample correlations often are corrected                      correction for attenuation in

for attenuation in reliability of measures.                    reliability: a method of adjusting

(Note: this matter of reliability                                   observed correlation coefficients

coefficients really only applies to                                 for imperfections in reliability of

measured variables. When it comes to                         the measurement of variables

experimental variables, it is not possible
to compute such reliability coefficients.
As a matter of convenience, researchers
simply assign experimental variables
reliability coefficients of 1.0. Though
test-retest measures can be taken for
demographic variables such as age, sex,
and academic class, most researchers
correcting for attenuation also assume
that these matters have reliability
coefficients of 1.0.

C.  Identifying Essential Differences in Studies:
Separating the Apples from the Oranges

--Since research rarely involves simple replication
of past work, meta-analysts are wise to code
different characteristics of research studies and
to use methods called “focused comparisons” to
test whether there are any differences produced
in research effects.

1.   Coding moderator variables

--coding typically includes: years studies
were completed; whether they were
laboratory or field studies
, numbers of

independent variables; soundness of

2.   Other Moderator Variables: there are two
other ways researchers may identify study

a.   Theoretic differences

b.   Researchers might rely on archival and
historical sources.

c.       Perceived study quality

--Though these quality ratings are not
objective, they are intersubjective,                      intersubjectivity: the degree to which
means that individuals may                                    different researchers with
share common agreement despite                           essentially different beliefs draw
any differences in their individual                           essentially the same
perceptions.                                                         interpretations of the meaning of


D.  Assessing Mean Relationship Sizes

--A decision must be made regarding whether

to weight studies by sample sizes.

If done, weighting by the inverse variance
weight is preferred.

--Interpreting mean effect sizes as correlation or
standardized differences

E.   Making Diffuse Comparisons                                               diffuse comparisons of effect

--If the effect sizes are inconsistent, the                              sizes:  measures of the
researcher has reason to believe that there is                    homogeneity of effect

  at least another variable that is making a                           sizes in meta-analyses.

  contribution to the variance associated with
different study outcomes.

F.   Making Focused Comparisons                                             focused comparisons:

      --Focused comparisons permit the researchers                    assessments in meta-analyses to

to explore the relationships between                               determine whether differences in

study effect sizes on one hand, and any                            other variables are related to          

moderator variables that may have been                          differences in the sizes of study
operating, on the other hand.                                          effects on primary variables.

1.      Continuous Moderator Variables                                 

·  As one indication, researchers could
correlate effect sizes (expressed as
Fisher’s Z) with each of the continuous


·  Focused comparisons using a form of
trend analysis may be applied to the
continuous variables, such as experts’
ratings of the soundness of measurement
in the studies to be analyzed.

2.   Continuous Moderator Variables                                 

·  Using tied contrast weights to use a
form of trend analysis

--If the researcher finds a statistically
significant focused comparison, the
researcher completes new diffuse tests of
the groups of effect sizes created by each
category level of the moderator variable.
If the diffuse tests reveal that the effect

sizes in each category still are not
homogeneous, the researcher concludes
that the critical moderator variable has
not yet been found. The process
continues with other moderator
variables in the search for category
groups with homogeneous effect sizes.

--If moderator variables are highly
correlated, then the contributions that
each one actually makes to explaining
differences in effect sizes will not be

IV. Assessing Meta-Analysis in Communication

A.  Advantages of Meta-Analysis

1.   By attempting to structure the
examination of study findings for
statistical analyses, researchers get
some distance from their own biases.

2.   A sound meta-analysis is capable of
being replicated statistically by other

3.   The statistics of meta-analysis allow
studies with large sample sizes and
accompanying reduced sampling
error—those whose findings are most
likely to show stable results—to be
represented according to their degree
of reduced sampling error.

B.   Disadvantages of Meta-Analysis

1.   Meta-analyses cannot study the
influences of moderator variables
unless researchers have already
decided to make them repeated
objects of research.

2.   Meta-analyses may not be as valuable
as primary research efforts.

3.   Meta-analysis do not promote
assessing conceptual issues
underlying the concepts of inquiry.

4.   Meta-analyses are restricted to analyses

of quantitative research studies alone.

--Some attacks on meta-analysis are mistaken:

·  The charge that meta-analyses combine studies
that are so fundamentally different that they
really are trying to equate “apples” with
“oranges” is a misunderstanding
(Response: in fact, meta-analysts code
differences in study methods to avoid
mixing “apples” and “oranges”).

·  The complaint that meta-analyses
typically combine studies of dubious
quality is an exaggeration (Response:
many meta-analysts screen research to
include only the best studies; the vast
majority of meta-analysts usually code
message quality and use focused
comparisons of effect sizes to assess
whether studies of different quality
produce different effect sizes).

·  The criticism that meta-analyses are
biased in favor of published research
that shows statistical significance is
generally unfounded. (Response: not
only do researchers typically attempt to
find unpublished studies, but they
regularly test for the potential number
of conflicting studies that would be
required to reject the meta-analytic
conclusions they draw).

·        The assertion that meta-analyses
emphasize single-variable influences
and “main effects” is largely incorrect.
(Response: coding moderator variables
and conducting focused comparisons
helps researchers discover important
sources of interactions).