The Ala Wai Watershed Inquiry (AWI)
teacher-development workshop consisted of eight daylong meetings spanning the
period between March 1, 2001 and May 12, 2001. This workshop served as one in a
series of workshops designed to equip teachers with skills necessary to promote
inquiry and critical thinking in their classrooms. The main purpose of the AWI Workshop was to
strengthen teacher confidence and skills in standards-based teaching through
inquiry, problem solving, and critical thinking centered on the concepts of the
watershed and environmental stewardship.
Specifically, the AWI was
designed to address the concept of watershed stewardship in relationship to the
Ala Wai Canal, to assist teachers in understanding how to use the Hawaii State
Content Standards and Benchmarks in designing their curricula, to promote the
understanding and use of the inquiry process in science and other fields of
middle school education for addressing the content standards, to begin to
increase teachers’ understanding of how to develop critical thinking skills in
students, and to encourage the use of the scientific research investigation to
broaden the approach and scope of the current science curriculum used by
teachers.
In addition to the goals mentioned
above, the AWI Workshop was also designed
to strengthen those teacher communication skills necessary to
successfully engage in learning through inquiry. Teachers were required to
deliver a group oral presentation of their groups’ research investigations that
were conducted over the course of the workshop. They were also required to
submit a written research report of this investigation. Finally, teacher-participants were required
to develop and submit a written outline of a plan for an inquiry, standards-based
unit for future implementation in their classrooms. The workshop content was supported by the
expertise of an Environmental Education Specialist, a Resource Teacher, three
professional scientists (including two Environmental Biologists and a Neuroscientist/Science
Research Education Specialist), and two spokespersons for the Ala Wai Watershed
Association.
Teacher Demographics
Seven middle school teachers of
science, social studies, math, english, and physical education participated in
the workshop. Six of the teachers were
from
Assessment and Evaluation
Techniques for the Workshop
The seven teacher-participants were asked
to complete an attitude and knowledge Workshop Assessment prior to their
participating in the workshop (Pre-Workshop Assessment), during their
participation midway through the workshop (Day 4, Mid-Workshop Assessment), and
following their participation in the workshop (Post-Workshop Assessment).
Self-reported confidence items required teachers to place a vertical line on an
18 cm scale. These responses were then
measured and quantified for data analyses.
Teachers were also asked to write occasional “temperature checks” and a
“reflection” following each workshop. A
brief questionnaire was given to the teachers after they engaged in research
scenarios designed to provide them with experience and practice to select and
use the correct research tools for testing particular hypotheses about
AWI Program Evaluation
Methods
Teacher reflection and “temperature
check” comments were placed into categories based on whether they addressed the
workshop in general; Ala Wai Watershed, watersheds in general, and
environmental stewardship; scientific inquiry and the research investigation
process; or standards-based education, including content standards and the
standards-based unit. (Note that all
teacher comments are quoted exactly as expressed, but may include additions by
the evaluator, indicated with brackets, for clarity.) The comments were then used, where appropriate,
to support and clarify quantitative data from the Pre-, Mid-, and Post-Workshop
Assessments, Program and Teacher Expectation Surveys, and PD-Credit
Evaluations.
The data from the items in the
Workshop Assessments (Pre-, Mid, and Post-Workshop Assessment), were summarized
and graphed. Where appropriate, one-way
ANOVAs with repeated measures were used to determine differences between the
pre-, mid-, and post-workshop data. When the results were statistically significant,
Neuman-Keuls multiple comparison tests were used to determine differences
between pairs of means. The Teacher and
Program Expectation Surveys used Likert-scale items requiring the
teacher-participants to decide whether various expectations were not at all
addressed, somewhat addressed, or completely addressed in the workshop. The
data from these surveys were summarized and graphed as percent to determine the
extent to which teachers felt that the expectations were met. Items from the research scenario
questionnaires were analyzed using the dependent t-test. The PD-Credit
Evaluation consisted of 10 elements of Quality Professional Development
arranged as a Likert-like scale. The evaluation scale for each element ranged
from 1=does not meet the standard, 2=partly meets the standard, 3=meets the
standard, 4=more than meets the standard, to 5=meets the standard to a high
degree. Teacher participant data from the PD-Credit Evaluation were summarized
using the mean and range for each item.
Evaluation Overview and
Summary of Findings
This evaluation focuses on the
impact of the AWI workshop on enhancement of teacher confidence and knowledge
in engaging students in standards-based learning about environmental
stewardship through scientific inquiry.
To this end, the evaluation specifically examines teacher-participant
understanding of, and confidence in teaching about and/or through, the concepts
of watershed and environmental stewardship, scientific inquiry and the research
investigation process, and content standards and standards-based units.
General comments gathered from
teacher reflections and “temperature checks,” as well as quantitative data from
assessments, surveys, and evaluations indicate that the workshop engaged
teachers in a successful professional development experience.
The major findings in this evaluation are that participation in the AWI
Workshop resulted in overall increases in both teacher knowledge about, and
teacher confidence to use in their teaching 1) the Ala Wai Watershed and
environmental stewardship, 2) scientific inquiry and the research investigation
process, and 3) content standands and standards-based units. General teacher-participant reflection
comments made at the end of the workshop support the quantitative data obtained
from assessments and surveys, indicating a gain in knowledge in the three focal
areas of the workshop (see Tables 1-3 below).
Table
1. Teacher-participant general
statements about the Ala Wai Watershed and environmental stewardship.
|
“This workshop was
completely successful at addressing the speakers, provided background
information, and involved us in doing field studies and our own research study on the |
|
|
|
“Learning about the |
|
“For so many years I drove past the |
Table
2. Teacher-participant general
statements about Scientific Inquiry and the Research Investigation Process.
|
“My knowledge/understanding of research design is refined
because of this workshop.” - Teacher #1 |
|
|
|
“I will definitely
be able to use the research process in all of my classes.” - Teacher #2 |
|
|
|
“I now know how to
use the research investigation process.” -
Teacher #3 |
|
“Now I have some
ideas of how I can use scientific inquiry in my math, language arts, and
health classes. I am excited
about teaching scientific inquiry.” |
|
“I understand the research process better than I did in
college! |
|
|
|
“This workshop did
well in helping the teachers to learn and to understand each step of the
process in doing [a]
research investigation.” - Teacher #4 |
|
|
|
“I became
acquainted with some of the tools scientists use in their investigations and
the sequence of steps within the research investigation
process.” - Teacher #6 |
|
|
|
“I am grateful for
this workshop for it has challenged me to expand my knowledge of inquiry
process and the skills
needed... .” - Teacher #7 |
|
“Research Investigation provides the teacher with an
awesome instructional tool to help implement
standards-based learning and support the emphasis on what is learned
rather than what is taught.” |
Table
3. Teacher-participant general statements about Standards and the Standards-
Based Unit.
|
“I finally understand what content standards, benchmarks,
performance indicators, and performance standards
are! It took this program to finally
make sense of all of these terms.” - Teacher
#4 |
|
|
|
“I gained the following as a teacher from this workshop:
Looking at standards and attempting to write performance
indicators.” - Teacher #5 |
|
|
|
“This area [of the workshop] really helped to relate the
scientific process and the standards in my mind.” - Teacher #7 |
|
“I now fully understand and appreciate how engaging
students in research can address a number of the HCPS
II... .” |
As teacher knowledge increased, so
did overall teacher-participant confidence in the areas covered in the
workshop, as indicated by the teachers’ self-reported confidence level in the
accurateness of their responses on the Workshop Assessment questions (Figure
1). By the end of the workshop, general
teacher confidence in understanding and being able to apply the concepts
covered in the workshop more than doubled to a level above “confident,” which
was a statistically significant increase over the substantially below
“confident” level exhibited by teachers prior to participating in the workshop.
Mean (+SEM)
Confidence (measured in cm) Completely Confident
Confident Not at all Confident Time of Assessment
Figure 1. Teachers’ self-reported confidence in their answering all of the items on the Workshop Assessment accurately before (Pre), during (Mid), and after (Post) their participation in the workshop. Confidence ratings were made by teachers after they completed the items on the assessment. A one-way repeated measures ANOVA indicated that teacher confidence in the accuracy of their responses to items on the assessment increased significantly by the end of the workshop (F2,12= 6.43, p < 0.02). This indicated that teachers were becoming more comfortable with the concepts covered in the workshop.
*significantly
different from Pre-Workshop (p <
0.05)
Teacher
comments from reflections at the end of the workshop described this
post-workshop increase in confidence: “In the area of addressing standards in
the AWI program..., I personally feel a lot more confident about implementing
it in my class; or, to be more specific, ‘I get it now!’” and “It [the workshop] has helped me feel a lot
more confident about research investigation... .”
A general goal of the AWI
Workshop was to instill teacher confidence in the use of the concept of
environmental stewardship applied to the
Responses from an overwhelming
majority of the teachers indicated that the teacher expectation centering on
learning and understanding watersheds was completely addressed (Figure 2). Teacher #4 reflected, “I understand what the
Ala Wai Watershed is and what a watershed is.”
However, one teacher (Teacher #6) indicated that this component of the
workshop was not at all addressed (Figure 2).
This teacher reflected, “What did this class teach us about a
watershed? What did you want us to learn
about a watershed? I have no idea.”
Figure 2. Teacher-participants’ responses (shown as percent) regarding the extent to which the AWI Workshop addressed the expectation for teachers to learn about and understand watersheds.
Five of seven of the workshop
participants responded that the teacher expectation of implementing and
observing a model of a watershed project was fully addressed, while one teacher
felt that it was somewhat addressed (Figure 3).
Teacher #7 reflected, “I feel that the AWI Program did address this
portion of the [teacher] expectations.”
However, similar to how she felt about the previous teacher expectation,
Teacher #6 responded that this expectation also was not at all addressed.
Figure 3. Teacher-participants’ responses (shown as percent) regarding the extent to which the AWI Workshop addressed the expectation for teachers to implement and observe a model of a watershed project in action.
As can be seen in Figures 4-6 below,
there was unanimous agreement that the three project expectations related to
the Ala Wai Watershed--conducting a research investigation, presenting an oral
report and producing a written report-- were completely addressed in the
workshop.
Figure
4. Teacher-participants’ responses (shown as percent) regarding the extent to
which the AWI Workshop addressed the expectation for teachers to conduct a
research study (with physical and social components), from beginning to end,
that centers on the Ala Wai.
Figure 5. Teacher-participants’ responses (shown as percent) regarding the extent to which the AWI Workshop addressed the expectation of teachers conducting an oral report on their research investigation of the Ala Wai.
Figure 6. Teacher-participants’ responses (shown as percent)
regarding the extent to which the AWI Workshop addressed the expectation of
teachers writing a research report on their research investigation of the Ala
Wai.
Teachers’ knowledge about
watersheds and environmental stewardship were assessed in the Workshop
Assessment (Figures 7 and 8). As shown
in Figure 7, participation in the workshop resulted in a dramatic 70% increase
in teachers able to demonstrate a good understanding and a decrease from 90 to
30% of the teachers having only a poor understanding of the term “watershed.”
Figure 7. Percent of 7 teachers having a good, fair, and poor understanding of the term “watershed” before (Pre-), during (Mid-), and after completing (Post-) the workshop.
One teacher-participant (Teacher #7) reflected on her Ala Wai Watershed learning experience with, “Learning about the Ala Wai Watershed was quite eye-opening... .”
In contrast, the AWI Workshop did not
appear to have a major influence on teacher understanding of environmental
stewardship (Figure 8). By the end of
the workshop, only one teacher (Teacher #7) increased her understanding of this
concept (from poor to good). At the end
of the workshop, Teacher #7 reflects her new knowledge about the Ala Wai
Watershed in a statement emphasizing stewardship: “For so many years, I drive
past the AW and never knew its needs and dilemma.”
Figure 8. Percent of 7 teachers having a good, fair, and poor understanding of the term “environmental stewardship” before (Pre-), during (Mid-), and after completing (Post-) the workshop.
Although teachers’ understanding of
environmental stewardship did not appear to be greatly influenced by their participation in the
AWI Workshop, teacher confidence in the ability to teach and engage students in
environmental stewardship was dramatically changed. Figure 9 shows the mean teacher confidence in
ability to teach and engage students in environmental stewardship before,
during, and after participating in the workshop. Teacher confidence level
increased significantly from near not at all confident before the workshop
(mean + SEM = 2.63 + 0.71) to confident by the end of the
workshop (7.43 + 1.92) [Figure 9].
Mean (+ SEM)
Confidence (measured in cm) Completely
Confident Time of Assessment Not at all Confident Confident
Figure 9. Teachers’ self-reported confidence in their ability to teach and engage their students in environmental stewardship before (Pre), during (Mid), and after (Post) their participation in the workshop. A one-way repeated measures ANOVA indicated that participation in the workshop significantly influenced teacher confidence in their ability to teach and engage their students in environmental stewardship by the end of the workshop (F2,12 = 4.22, p = 0.04).
*significantly
different from Pre-Workshop (p <
0.05)
Scientific Inquiry and the
Scientific Research Investigation (RIP) Process
One primary goal of this workshop
was to assist teachers in learning about and developing skills in the research
investigation process so that teachers could begin using scientific inquiry in
standards-based science teaching.
Specifically, it was expected that teacher-participants would become
familiar with and be able to begin to guide their students in the steps
necessary to conduct scientific research to gain an understanding of the world
around them. It was also expected that teachers would gain an appreciation for how
the inquiry process used in science can be generalized to other academic
courses. Increasing teacher comfort
level in emphasizing the use of statistics in data analyses and decision-making
by their students was a primary goal of the research investigation/inquiry
content of the workshop.
The teacher-participants unanimously
indicated on the Program and Teacher Expectation Surveys that all workshop
expectations related to scientific inquiry and the research investigation
process were at least somewhat addressed, with a majority of teachers
indicating that five of these six expectations were fully addressed (see
Figures 10-14 below).
All seven teachers responded on the
Program Expectation Survey that they had engaged in research investigations
which generated new knowledge (Figure 10).
Figure 10. Teacher-participants’ responses (shown as percent) regarding the extent to which the AWI Workshop engaged them in research investigations which generate new knowledge.
The teachers were also unanimous that
the program expectation for teachers to understand how the parts of a research
investigation relate to each other was fully addressed in the workshop (Figure
11). For example, Teacher #5 stated, “I
have learned the step-by-step process of research investigation. I now know how each part is connected;” and
Teacher #6 claimed, “Our teachers did a good job in teaching us the
interrelationship between the parts of a research investigation.”
Figure 11. Teacher-participants’ responses (shown as percent) regarding the extent to which the AWI Workshop helped them to understand the interrelationship between the parts of a research investigation (introduction, methods, results, discussion and conclusion, and action plan).
The majority of the participants
expressed that the program expectation of understanding the interrelationship
between research design and data analysis was completely covered in the
workshop; however, three of the seven teachers also felt that this expectation
was only partially addressed (Figure 12).
Figure 12. Teacher-participants’ responses (shown as
percent) regarding the extent to which the AWI Workshop addressed the
understanding of the interrelationship between research design and data analysis.
All participants felt that the strengthening of communication skills necessary for researchers was addressed in the workshop, with three out of seven indicating that this program expectation was addressed completely (Figure 13).
Figure 13. Teacher-participants’ responses (shown as percent) regarding the extent to which the AWI Workshop addressed the strengthening of all communication skills that are necessary for researchers.
Five of the seven
teacher-participant responses on the Teacher Expectation Survey indicated that
the learning of statistics for use in the scientific inquiry process was
completely addressed in the workshop (Figure 14). Teacher #1 claimed, “Standard deviation
[is]...among the most valued things that I gained;” and Teacher # 7 reflected,
“Regarding the area on statistics, I truly learned something new here...” and
“I [had] often wondered how scientists came to the conclusions they came to
using data.” One teacher (Teacher #7)
stated, “Although understanding the null hypothesis is simple, it was confusing
for my brain and I found it a fun and challenging concept to grasp.”
Figure 14. Teacher-participants’ responses (shown as
percent) regarding the extent to which the AWI Workshop engaged them in the
learning of statistics.
The Majority of the teachers also felt that their
expectation for learning about specific statistical tests and directional and
non-directional hypotheses (one- and two-tailed tests) was completely addressed
(Figure 15). One teacher reflected, “How
thrilling to learn [about]something like the ‘null’ [hypothesis].”
Figure 15. Teacher-participants’ responses (shown as percent) regarding the extent to which the AWI Workshop engaged them in learning about correlation and t-tests and using different tailed tests.
The percent of teachers
demonstrating a good understanding of the role statistics play in the research
process increased from 0 % to 60 % by the end of the workshop, with a
corresponding six-fold decrease in the percent of teachers with poor
understanding (Figure 16).
Figure 16. Percent of 7 teachers having a good, fair, and poor understanding of the role statistics play in research before (Pre-), during (Mid-), and after completing (Post-) the workshop.
For decision-making based on data
collected in the research investigation, the AWI Workshop emphasized graphing
and interpretation of experimental and correlational data, covering the bar
graph (histogram), line graph, and scatterplot.
The Workshop Assessment examined the impact of the workshop on the
teachers’ ability to describe, understand and interpret graphs for making
decisions based on research investigation data.
The assessment data indicate that teacher understanding and
interpretation of the graphs increased substantially over the course of the AWI
Workshop. For example, as shown below
in Figures 17-19, teacher-participants’ abilities to describe and interpret bar
graphs were dramatically improved over the course of the workshop. The percent
of teachers able to provide excellent descriptions of the data presented in a
bar graph increased from zero to sixty percent, while poor descriptions
decreased from 70% (Pre-Workshop) to 0 % (Post-Workshop) [Figure 17].
Figure 17. Percent excellent, good, and poor responses for 7 teachers describing a bar graph (histogram) comparing plant growth under different fertilizer conditions before (Pre-), during (Mid-), and after completing (Post-) the workshop.
By
the end of the workshop, teachers also exhibited a large increase (0 %
Pre-Workshop to 55 % Post-Workshop) in ability to correctly interpret the data
presented in a bar graph (Figure 18).
This large percent increase in
interpretation ability corresponded with a dramatic percent increase (from 0 % to
70 %) in understanding of the concept of error in decision making and a dramatic
decrease (from 100 % Pre-Workshop to 10 % Post-Workshop) in lack of
understanding of error (Figure 19).
Figure 18. Percent Correct and incorrect responses for 7 teachers interpreting data presented in a bar graph (histogram) comparing plant growth under different fertilizer conditions speed before (Pre-), during (Mid-), and after completing (Post-) the workshop.
Figure 19. Percent excellent, good, and poor responses for 7 teachers recognizing the role (importance) of variability (error bars) in interpreting the data presented in a bar graph (histogram) comparing plant growth under different fertilizer conditions speed before (Pre-), during (Mid-), and after completing (Post-) the workshop.
Figure 20 shows the mean confidence
of the teachers in their ability to success-fully engage students in research
in their subject area before, during, and after their participation in the AWI
workshop. As can be seen in the figure, prior to the workshop, teachers rated
themselves as being slightly less than confident (mean + SEM = 7.01 +
1.11 compared to 8.25), while after their participation in the workshop, they
felt slightly more than confident (9.24 + 1.43 compared to 8.25). A repeated measures ANOVA indicated that mean
teacher confidence for engaging their students in research did not differ
across the three workshop periods.
However, because of the small sample size (n=7) and relatively small variation among the means,
the ability for the ANOVA to detect a significant workshop effect on teacher
confidence was very weak. Thus, the negative statistical finding on workshop
influence on teacher confidence level in successfully engaging students in
research should be cautiously interpreted. (See Figure 20 caption for further
explanation of the power issue.)
Not at all Confident Confident Completely
Confident Mean (+ SEM)
Confidence (measured in cm)
Time of Assessment
Figure 20. Teachers’ self-reported confidence in their ability to successfully engage students in research in their subject area before (Pre), during (Mid), and after (Post) their participation in the workshop. A one-way repeated measures ANOVA indicated that the teachers’ confidence was not significantly different across the three workshop assessment periods (F 2,12 = 1.75, p > 0.05). Due to the small sample size and relatively small variation among the means, the power of this test to detect a significant workshop effect on teacher confidence was very low (power of performed test with alpha = 0.05: 0.14) compared to the desired power of 0.80.