ANOVA Science Education
Corporation Professional Development Program for K-6 Teachers:
Teaching Science Literacy
through Inquiry- The Research Investigation Process (RIP™) , Program
Evaluation.
RIP Implementation
and In-Classroom Follow-up
Honolulu District, Honolulu,
Hawaii
June 25, 2003
The purpose of this professional development program was to
introduce K-6 teachers to the teaching of science through true scientific
inquiry, using the research investigation process (RIP ä) and to explore the RIP as
a tool for addressing the Hawaii Science Content and Performance Domain I
standards in the classroom.
Specifically, it was designed to guide teachers in the use of the
inquiry process; to have teachers learn how to design and conduct scientific
research studies; to have them become familiar with techniques to assist in
guiding students through the scientific inquiry process; to have them examine,
practice, understand, and become competent in the ability to apply data
analysis techniques to decision-making in science; to increase confidence in
using scientific research in their approach to instructing students in science
and in addressing the scientific inquiry benchmarks and science inquiry content
standards; to have them implement the RIP as a tool for instruction in the
classroom; and to increase student interest in learning science.
Over the course of the initial three-day workshop session,
the research investigation process (RIP) was introduced and teachers were
provided the opportunity to develop an understanding of each of the elements of
the RIP through their participation in and development of actual research
investigations. Teacher participants
were guided through a number of activities related to making observations;
posing research questions; obtaining, examining, and evaluating background
information; constructing hypotheses; and designing the methods for a research
investigation. Techniques in data
summary, analysis and presentation were explored in the context of hypothesis
testing and decision-making in science.
Teachers were then expected to introduce workshop-related concepts and
activities learned into their classroom and guide their students in conducting
their first RIP over the subsequent three months. During the three-month implementation period,
half-day individual teacher/small group follow-up sessions were available to
the participating teachers upon request.
The individual teacher/small group follow-up sessions involved modeling
of instructional techniques and practices with students, assisting teachers on
curriculum development, and/or clarifying concepts presented in the initial
three-day workshop session. The
participants met together again in a final follow-up session at the end of the
three month implementation/individual teacher follow-up period to share their
inquiry-based instructional experiences and student outcomes. All aspects of this workshop were aligned
with the State of Hawaii Science
Content and Performance Standards.
The data for this program evaluation were obtained from
assessments of 11 of the 12 elementary teacher-participants at the beginning of
(Pre-Assessment) and again at the end (Post-Assessment) of the 3-day initial
workshop, from questionnaires administered along with the Post-Assessment
(Post-Workshop Questionnaire), and during the follow-up session at the end of
the program (Post-Follow-Up Questionnaire, N=7 participants). (One of the program participants was
eliminated from the evaluation because they were unable to attend all of the
workshop sessions and, therefore, did not complete both of the
assessments.) Items on the assessments
required demonstration of knowledge about the scientific inquiry process, data
analyses procedures, and
decision-making in science.
A number of these items required teachers to demonstrate their knowledge
through application. Self-report items
measured teacher confidence levels in understanding and using scientific
inquiry in the classroom and in comprehending and applying the scientific
inquiry content standards to their instruction.
The response scale for the confidence items included “not at all
confident” (‘0’-value), “somewhat confident” (‘3’-value), “confident”
(‘6’-value), and “completely confident” (‘9’-value). A concept inventory determined teachers’ familiarity
with and ability to teach elements of scientific inquiry and data summary and
analysis techniques. The answer scale
for the concept inventory items included “I am completely unfamiliar with this
concept” (value=1), “I am somewhat familiar with this concept, but do not
really understand what it means” (value = 2), “I am familiar with this concept,
and have a fair understanding of what it means” (value = 3), “I am very
familiar with this concept, but would have some difficulty teaching it to others”
(value = 4), and “I am completely familiar with this concept and could easily
teach it to others” (value = 5). The pre-workshop and post-workshop assessment
items were the same. The Post-Workshop
Questionnaire containing five items was also administered to assess the
teachers’ perceptions of how much their understanding of scientific inquiry and
the research investigation process changed and improved as a result of
participation in the workshop. Finally,
the Post-Follow-Up Questionnaire, containing a number of the teacher confidence
and perception items on the Pre- and Post- Assessments, as well as additional
items related to the impact of the individual/small group teacher follow-up
sessions and activities on teacher perceptions, was administered. The Pre- and Post-Assessment data were
statistically analyzed one-way repeated measures ANOVAs to determine
significant differences (indicating change) between pre- and post-assessment
mean values. ANOVAs were also used to
compare responses on items from the Post-Workshop Questionnaire with the
Post-Follow-Up Questionnaire, and were to compare responses from common items
on the Pre-Assessment, Post-Assessment, and Post-Follow-Up Questionnaire. In the latter cases, following a significant
effect, Tukey’s tests were used for multiple comparisons. The criterion for statistical significance (a) for all tests was set at 0.05.
Teacher Knowledge and Understanding of the Scientific
Research Investigation Process (RIP), and Confidence in Teaching Scientific
Inquiry
Workshop participants demonstrated a large, statistically
significant increase in their knowledge and understanding of the individual
elements of the RIP, almost doubling their assessment scores by the end of the
initial 3-day workshop session (Figure 1, below). This included the logical order of the RIP
elements, understanding of components involved in each element, and
demonstration of the ability to construct testable hypotheses.
Figure 1. Demonstration of knowledge and understanding
of the elements of the RIP. There were
a total of 25 points available on this portion of the assessment.
*Mean post-assessment score
is significantly greater than mean pre-assessment score [F (1,10)
= 67.13, p<0.001].
The post-3-day workshop increase in teacher-participant
knowledge and understanding of the research process was accompanied by a
significant increase in teacher’ self-reported familiarity and understanding of
concepts related to the scientific research process in the concepts inventory
(Figure 2, below). By the end of the workshop, the average participant’
response rose from “familiarity with fair understanding of concept” to “very
familiar with concept and could teach it to others.” This showed that teachers recognized their
increased knowledge and understanding.
Mean (+SEM) RIP Concept Inventory Score
|
|
Figure 2. Familiarity and
understanding of concepts related to elements of the RIP. The answer scale for the concept inventory
items included “I am completely unfamiliar with this concept” (value=1), “I am
somewhat familiar with this concept, but do not really understand what it
means” (value = 2), “I am familiar with this concept, and have a fair
understanding of what it means” (value = 3), “I am very familiar with this
concept, but would have some difficulty teaching it to others” (value = 4), and
“I am completely familiar with this concept and could easily teach it to
others” (value = 5).
* Mean post-assessment score is significantly greater
than mean pre-assessment score [F (1,10) = 32.74, p<0.001].
By the end of the initial 3-day workshop,
participants’ self-reported confidence levels for
their ability to use scientific inquiry, their
understanding of teaching science through inquiry, and their ability to teach
and engage students in scientific research activities all significantly increased,
doubling compared to pre-workshop levels (Figures 3, 4, and 5, below).
Mean (+SEM) Confidence
Score
|
|
Figure 3. Self-reported
confidence levels for participants’ ability to use scientific inquiry. The response scale for the confidence items
included “not at all confident” (‘0’-value), “somewhat confident” (‘3’-value),
“confident” (‘6’-value), and “completely confident” (‘9’-value).
*Mean post-assessment score
is significantly greater than mean pre-assessment score [F (1,10)
= 17.16, p=0.002].
Mean (+SEM) Confidence
Score
|
|
Figure 4. Self-reported
confidence levels for understanding of teaching science through inquiry. The response scale for the confidence items
included “not at all confident” (‘0’-value), “somewhat confident” (‘3’-value),
“confident” (‘6’-value), and “completely confident” (‘9’-value).
*Mean post-assessment score
is significantly greater than mean pre-assessment score [F (1,10)
= 15.75, p=0.003].
Mean (+SEM) Confidence
Score
|
|
Figure 5. Self-reported
confidence levels for ability to teach and engage students in scientific
research activities. The response scale
for the confidence items included “not at all confident” (‘0’-value), “somewhat
confident” (‘3’-value), “confident” (‘6’-value), and “completely confident”
(‘9’-value).
*Mean post-assessment score
is significantly greater than mean pre-assessment score [F (1,10)
= 15.80, p=0.003].
Teacher Understanding of and Ability to Apply Data
Summary, Presentation, and Analysis techniques for Decision-Making in Science
By the end of the initial 3-day workshop, participants
demonstrated a dramatic, statistically significant increase in their knowledge
and ability to correctly organize data into a summary table and to construct a
bar graph for comparing the central tendency of two groups of data (Figure 6, below).
Mean (+SEM) Data Summary & Presentation Score
|
|
Figure 6.
Demonstration of understanding and ability to apply data organization
and presentation techniques to data.
This section was worth a total of 10 points.
* Mean post-assessment score
is significantly greater than mean pre-assessment score [F (1,10) = 18.47, p=0.002].
Workshop participants also demonstrated a very large change
in their knowledge and ability to apply data analysis techniques to research
data. Comparison of the pre-and
post-assessments revealed that by the end of the 3-day workshop, participants
significantly increased their understanding of how to calculate descriptive
statistics and their ability to determine which measure of central tendency is
most appropriate for a group of data (Figure 7, below).
Mean (+SEM) Data Analysis Score
|
|
Figure 7.
Demonstration of understanding the calculations for descriptive
statistics and ability to determine the most appropriate statistic to represent
central tendency for a group of data.
This section was worth a total of 10 points.
* Mean post-assessment score
is significantly greater than mean pre-assessment score [F (1,10)
= 16.96, p=0.002].
Teacher-participants demonstrated a statistically
significant increase in their ability to interpret data presented in
scatterplots and summarized in bar graphs by the end of the 3-day workshop
(Figure 8, below).
Mean (+SEM) Graph Interpretation Score
|
|
Figure 8.
Demonstration of ability to interpret scatterplots and bar graphs. This section was worth a total of 10 points.
* Mean post-assessment score
is significantly greater than mean pre-assessment score [F (1,10)
= 19.17, p=0.001].
The post-3-day workshop increases in teacher-participant
knowledge of and ability to apply data presentation and analyses were
accompanied by significant increases in teacher’ self-reported familiarity and
understanding of concepts related to data presentation and analysis (Figures 9
and 10, below). By the end of the
workshop, the average participant’ response for the three measures of central
tendency rose significantly from “somewhat familiar with concept, but do not
really understand what it means” to “very familiar with concept, but would have
some difficulty teaching it to others” (Figure 9).
Mean (+SEM) Central Tendency Concept Inventory
Score
|
|
Figure 9. Familiarity and
understanding of concepts related to measuring central tendency. The answer scale for the concept inventory
items included “I am completely unfamiliar with this concept” (value=1), “I am
somewhat familiar with this concept, but do not really understand what it
means” (value = 2), “I am familiar with this concept, and have a fair
understanding of what it means” (value = 3), “I am very familiar with this
concept, but would have some difficulty teaching it to others” (value = 4), and
“I am completely familiar with this concept and could easily teach it to
others” (value = 5).
* Mean post-assessment score is significantly greater
than mean pre-assessment score [F (1,10) = 51.76, p<0.001].
Similarly, the average participant’ response for tables and
graphs rose significantly from “familiar with concept with a fair understanding
of what it means” to “very familiar with concept, but would have some difficulty
teaching it to others” (Figure 10).
Mean (+SEM) Tables & Graphs Concept Inventory
Score
|
|
Figure 10. Familiarity and
understanding of concepts related to tables and graphs. The answer scale for the concept inventory items included “I am
completely unfamiliar with this concept” (value=1), “I am somewhat familiar
with this concept, but do not really understand what it means” (value = 2), “I
am familiar with this concept, and have a fair understanding of what it means”
(value = 3), “I am very familiar with this concept, but would have some
difficulty teaching it to others” (value = 4), and “I am completely familiar
with this concept and could easily teach it to others” (value = 5).
*Mean post-assessment score
is significantly greater than mean pre-assessment score [F (1,10)
= 21.20, p<0.001].
Benchmarks and Standards
General teacher
confidence and awareness of ability to understand and apply scientific inquiry
to the teaching of science and in ability to successfully address the
scientific inquiry standards were also affected by participation in the initial
3-day workshop. Teacher-participant
self-reported confidence in ability to address content standards in the
classroom rose significantly, from “somewhat confident” to “confident” by the
end of the workshop (Figure 11, below).
Mean (+SEM) Confidence
Score
|
|
Figure 11. Self-reported
confidence levels for ability to address content standards in the
classroom. The response scale for the
confidence items included “not at all confident” (‘0’-value), “somewhat
confident” (‘3’-value), “confident” (‘6’-value), and “completely confident”
(‘9’-value).
* Mean
post-assessment score is significantly greater than mean pre-assessment score [F
(1,10) = 10.96, p=0.008].
Similarly, by the end of the workshop, participants’
confidence scores in their ability to accurately and completely address the
scientific inquiry benchmarks doubled, increasing from below “somewhat
confident” to near “confident” (Figure 12, below).
Mean (+SEM) Confidence
Score
|
|
Figure 12. Self-reported
confidence levels for ability to accurately and completely address the
scientific inquiry benchmarks. The
response scale for the confidence items included “not at all confident”
(‘0’-value), “somewhat confident” (‘3’-value), “confident” (‘6’-value), and
“completely confident” (‘9’-value).
* Mean
post-assessment score is significantly greater than mean pre-assessment score [F
(1,10) = 18.51, p=0.002].
Finally, by the end of the 3-day initial workshop,
teachers significantly increased their familiarity and understanding of inquiry
standards from being “somewhat familiar with this concept, but not really
understanding what it means” to being “very familiar with this concept, but
would have some difficulty teaching it to others” (Figure 13, below). This increase was statistically significant
and was consistent with the increase in teacher-participant confidence
regarding addressing the scientific inquiry content standards and benchmarks
(Figures 11 and 12, above).
Mean (+SEM) Inquiry Standards Concept Inventory
Score
|
|
Figure 13. Familiarity and
understanding of the inquiry standards concept. The answer scale for the concept inventory items included “I am
completely unfamiliar with this concept” (value=1), “I am somewhat familiar
with this concept, but do not really understand what it means” (value = 2), “I
am familiar with this concept, and have a fair understanding of what it means”
(value = 3), “I am very familiar with this concept, but would have some
difficulty teaching it to others” (value = 4), and “I am completely familiar
with this concept and could easily teach it to others” (value = 5).
*Mean
post-assessment score is significantly greater than mean pre-assessment score [F
(1,10) = 15.80, p=0.003].
Teacher
Perceptions of Overall Impact of the Initial 3-Day Workshop on Understanding of
and Ability to Implement Standards-Based Inquiry
The Post-Workshop Questionnaire administered with
the Post-Assessment contained five self-report items designed to assess how
much teacher-participants believed their knowledge and abilities regarding the
scientific research investigation process (RIP) and scientific inquiry were
impacted by their participation in the initial 3-day workshop. The results from these items are presented in
Figures 14-19 below.
A majority of the workshop-participants (70%)
claimed that their understanding of the RIP was changed a “large amount” to “completely”
as a result of their participation in this workshop, while three participants
claimed it was changed a “moderate amount” (Figure 14, below).