|
OSESA reports 1999-2006
|
|
|
|
|
 |
OSEDA evaluation reports, 1999-2006
Missouri's Office of Social and Economic Data Analysis (OSEDA) worked with the eMINTS National Center and participating schools to evaluate the eMINTS program implementation in the first five cohort groups of schools.
The eMINTS evaluation project was recognized as meeting
the requirements for the U.S. Department of Education's "Scientifically-Based
Research" outlined in No Child Left Behind legislation and featured as
a model program at the State Educational Technology Directors Association (SETDA)
"National Leadership Institute on Evaluation" in December 2002.
Types of reports
The OSEDA evaluation project team produced a variety of reports
to document their findings about eMINTS' impact on Missouri's teachers, students,
classrooms and educational system.
- Survey reports summarize quantitative
data from surveys or district data sources.
- Focus group and interview reports summarize
issues from focus groups and school interviews, and present qualitative data and
summaries of teacher, principal and parent communities.
- MINTs
project evaluation from OSEDA includes data about the pilot project that evolved
into eMINTS. (PDF file; 56 pages)
Survey reports
The analysis of student MAP scores in the FY04 cohort of eMINTS schools shows significant differences by eMINTS enrollment status on the MAP Communication Arts and Mathematics tests. Analyses of MAP scores for special education students, students receiving Title I services and students receiving free and reduced lunch suggest that eMINTS enrollment significantly increases their scoring on the MAP tests. These results support previous analyses of eMINTS cohorts.
There is no doubt that for the FY04 eMINTS cohort, enrollment in an eMINTS classroom is beneficial for increasing MAP scores for both the Communication Arts and Mathematics tests.
 Executive summary (PDF 36 KB)
July 2006 — Results of 40 school observations in Title I eMINTS FY03 Schools indicated that positive school climate could be distinguished by the nature of interaction between the building principal and students. Building principals who were classified as student-centered positively affected the climate of the school. In this context, student-centered leadership was defined as maintaining high visibility with the students: visiting classrooms, relating to students in school settings, monitoring student achievement and progress. Four correlations were flagged for statistical significance between student-centered principals and school climate indicators. Student-centered principals were more likely to lead schools where student work was displayed in the classrooms, where visitors were welcomed, where office staff related positively to students and where teachers attended to and instructed student behavior in public areas. Analysis of student MAP scores also shows significant differences by principal leadership status on the mathematics tests. These results support previous analyses of the importance of principal leadership in eMINTS schools.
November 2005 — This report presents results from an examination of the professional development of eMINTS teachers in the FY03 cohort. The purpose of the study was to assess differences and similarities between the efficacy of professional development delivered by eMINTS Staff (Cluster Information Specialists - CIS), and by participants in the eMINTS Professional Development for Educational Technology Specialists (PD4ETS) program, a train-the-trainer program. Previous analysis of eMINTS teachers’ perceptions of their professional development found that CIS played a crucial role in the development of teachers. Evidence from the current study suggests that both types of professional development delivery produce similar results and assist in preparing eMINTS teachers to support student learning.
 Executive Summary (PDF 32 KB)
April 2005 — Previous analyses of Missouri Assessment
Program (MAP) data have consistently indicated a positive interaction between
the eMINTS and Title I programs. This report examines the MAP performance of
students who receive Title I-funded educational assistance and are enrolled
in eMINTS classrooms, using data from the FY03 eMINTS cohort of schools. While
the results were mixed, there was evidence of a smaller achievement gap between
Title I and non-Title 1 eMINTS students. These quantitative results are supported
by observations from Title I teachers regarding the ways in which eMINTS classroom
enrollment supports the performance of the students with whom they work. These
results suggest that a closer coordination between the Title I and eMINTS programs
will have a beneficial impact on the performance of Title I students.
January 2005 — The analysis of student MAP scores in the FY03 cohort of eMINTS schools shows significant differences by eMINTS enrollment status on the MAP Communication Arts test and smaller, but positive, differences on the MAP Mathematics test. Analyses of instructional practices suggest that participation in the eMINTS program helps all teachers more effectively support higher levels of student achievement. Finally, analyses of MAP scores for Black students, special education students and students receiving Title I services suggest that eMINTS enrollment helps reduce the achievement gaps between these groups and other students. These results support previous analyses of eMINTS cohorts.
Executive Summary (PDF 32 KB)
November 2004 — This report analyses the impact of the nonrandom selection of eMINTS schools, teachers and students in the FY02 eMINTS cohort. The report summarizes the evaluation-design choices made by the program in light of the simplistic study design endorsed by advocates of “scientifically based research.” It also proposes a modeling strategy to estimate the potential impact on MAP scores of the nonrandom selection of teachers and students. The analyses suggest that no identifiable selection biases exist among the schools, teachers and students participating in the FY02 eMINTS cohort.
November 2004 — This report presents Missouri Assessment Program (MAP) test results from the FY00, FY01 and FY02 eMINTS cohorts. The purpose of the study was to examine the relationship among factors of race, enrollment in an eMINTS classroom and performance on the MAP examinations. Findings from this study showed that, while eMINTS enrollment did not impact the achievement gap between Black and White students in a statistically significant way, the performance of both Black and White students was higher and the gap was reduced.
September 2004 — This policy brief examines the quantitative impacts of the eMINTS expansion project.
The report analyzes 2001 and 2002 Missouri Assessment Program (MAP) results for
eMINTS students in classrooms with teachers who have completed the eMINTS
professional development training and are serving as Mentors to students in classrooms
with teachers undertaking the eMINTS training (Mentored teachers), students in
Mentored teachers’ classrooms, as well as non-eMINTS students in classrooms in that
grade in the same school building. Additional analysis explores the relationships of
teacher status, student average MAP scores, and classroom instructional practices
categorized according to the eMINTS lesson typology. Results suggest that the structure
employed in the expansion project may have helped newly mentored teachers adopt
inquiry-based instructional practices more quickly, but the pace of these adoptions did
not necessarily help student performance.
September 2004 — This report presents data from observations of classroom climate, instructional practice and mentorship experience in the eMINTS Expansion program classrooms. Data from the entire two-year program is analyzed. The analysis describes clear relationships between a positive classroom climate and the use of effective behavior management strategies. The analysis indicates that these classroom characteristics are also related to the observed instructional practices of teachers. Differences between the classrooms of mentor teachers and those of mentored teachers over the two years of the expansion program are considered. The two-year data suggests an overall improvement in the observed classroom climate of eMINTS classrooms.
May 2004 — This report presents Missouri Assessment Program (MAP) test results from the FY02 eMINTS cohort. The analysis highlights statistically significant differences on the Communication Arts, Mathematics and Social Studies MAP tests, but not on the third grade Science test. These differences can be seen for all students, for low income students, and for special education students. There are also significant differences for students enrolled in Schoolwide Title I schools.
Executive Summary (PDF 53 KB)
July 2003 — The eMINTS evaluation has highlighted the role of inquiry-based instructional practices in supporting high levels of student achievement. As the eMINTS program has developed, "instructional practice" has been assessed in three ways: through direct observation using the eMINTS Lesson Typology, through coding each lesson along Bloom's Taxonomy of Educational Objectives and as part of a general assessment of the classroom environment using the "Hallmarks of an Effective eMINTS Classroom" rubric. This paper assesses the validity of each of these measures by analyzing their interrelationships. Data for this analysis is taken from a set of classroom observations conducted with teachers participating in the eMINTS expansion project. Results show significant correspondence between each measure at both the lesson and classroom levels.
July 2003 — Many ingredients go into being an effective eMINTS teacher. Results of 99 classroom observations in eMINTS Expansion Schools indicated that effective and less effective behavior management could be distinguished on the basis of teachers' sensitivity to five areas in creating their classroom environment: exhibiting work that was student-produced; arranging space for flexible use; providing elements that afford a soft, sensory, tactile option; maintaining physical comfort; and interacting in a respectful and encouraging manner. Teachers observed conducting facilitated lessons were also more likely to have classrooms with these positive aspects. The combination of a positive classroom and the use of facilitated, inquiry-based instructional practices helped to construct a well-ordered and effective learning environment.
July 2003 — The eMINTS expansion project was designed to identify successful strategies for extending the impact of eMINTS instructional resources beyond the third and fourth grades. This report summarizes the early experiences of the expansion schools as they developed the capability to expand and sustain the eMINTS innovation in their buildings. Analysis of these schools' experience revealed three common factors that are key to creating an educational environment that supports the eMINTS innovations: an instructional plan, efforts to involve students in the school community and programs to bring community volunteers into the school. Expansion schools that used these three factors as resources to create learning environments where students can achieve, showed strong positive results on the 2002 MAP test and are identified as successful, proficient eMINTS schools.
January 2003 — This policy brief examines the
quantitative impacts of the eMINTS Program on the second cohort of participating
classes. The analysis compares 2002 Missouri Assessment Program (MAP) results
for eMINTS students and non-eMINTS students in the same school building and for
eMINTS students and all students statewide. eMINTS students in the fourth grade
scored significantly higher than non-eMINTS students. Similar results were not
seen among third grade students. eMINTS students in classes characterized by teachers
who consistently implement the inquiry-based teaching practices emphasized by
the eMINTS professional development program scored higher than students in other
types of classes. While there were fewer significant differences in some aspects
of the program, these results in general substantiate the positive trends associated
with eMINTS enrollments reported in the policy brief "Analysis
of 2001 MAP Results for eMINTS Students," published in January 2002.
October 2002 — This report classifies FY00 eMINTS
principals according to their leadership orientation. School visits were conducted
during the 1999-2000 and the 2000-2001 school years. On-site visits including
interviews with the principals and the eMINTS classroom teachers, illustrate four
general leadership orientations among the FY00 principals. Principals with a collaboration
orientation acted as catalysts to the full implementation of eMINTS. Principals
with a regulation orientation limited the full implementation of eMINTS. Principals
with a disconnection orientation had little impact on the implementation of eMINTS
in their buildings. In addition to these three main orientations, a fourth, "hybrid"
orientation was identified. These leadership orientations are clearly related
to the ways eMINTS teachers conduct their lessons, and have clear impacts on student
performance. Students in schools with collaborative principals score higher than
students in schools with other types of principals. This report describes FY00
principal leadership orientations and to what extent leadership orientation affected
the implementation of the eMINTS program.
January 2002 — This policy brief examines the
quantitative impacts of the eMINTS Program on the first cohort of participating
classes. The analysis compares 2001 Missouri Assessment Program (MAP) results
for eMINTS students and non-eMINTS students in the same school building. eMINTS
students scored consistently higher on the MAP than non-eMINTS students. eMINTS
students classified as having special needs scored higher than special needs students
in non-eMINTS classes. Finally, higher MAP results among eMINTS students were
found to be associated with the instructional practices.
November 2001 — This report summarizes classroom
observations conducted in FY00 classrooms. It describes a general typology of
instructional lessons based on 85 classroom observations conducted in 2000 and
2001. The observations, which serve as the empirical basis for the typology, were
conducted during site visits made by the eMINTS evaluation team in two waves,
between March 1 and May 20, 2000 and between November 11, 2000 and April 18, 2001.
During each lesson, evaluation team members observed teachers' interaction with
students, their use of computers and their general conduct of the lesson. The
discussion outlines the rationale for the lesson typology and the key characteristics
of the categories that constitute the typology. Comparisons between observations
conducted in 2000 and 2001 shows a general increase in the occurrence of facilitated
lessons.
July 2001 — This report presents results
from the eMINTS teacher surveys administered between 1999 and 2001. During this
period the eMINTS Program refined its professional development instruction and
improved its service delivery procedures. Over the survey period the results show
general improvement in teacher computer usage in areas related to instruction.
By the spring of 2001 eMINTS teachers report that they are doing more with computers,
that they feel more skilled in using computers, and that they use computers more
frequently in their work. Teachers also feel that they are more effective teachers,
as indicated by the overall increase in the average score on the MSIP Instructional
Efficacy scale. The pattern of responses suggest that this improvement in teacher
opinion is related to programmatic improvements in its second year.
September 2001 — Results from a spring administration
of the MSIP Elementary Student Questionnaire show that fourth grade students enrolled
in eMINTS classrooms rate their school climate and expectations for learning higher
than students not in eMINTS classrooms. However, there were no statistically significant
differences among third grade students. These results will be used in estimating
the impact of the eMINTS experience on student performance on the MAP test.
June 2001 — This policy brief examines the qualitative
and quantitative impacts associated with the installation of computer classrooms
in the Jennings School District.
January 2001 — FY01 teachers and their approach
to lesson plans.
December 2000 — A comparison of FY01 and FY00
teacher opinions.
Focus group and interview reports
- FY00
teachers' summary impressions of their experiences with eMINTS and factors
they identified for its successful implementation, 10/31/01 (PDF 170 KB)
- Parents'
perceptions of the eMINTS program and its effect on their children, 9/4/01
(PDF 277 KB)
- Teacher
perceptions of student changes in FY00 eMINTS classrooms, 2/25/01 (PDF 81 KB)
- Teacher
experiences with technical support (12/8/00)
- Teacher
comments about working in an eMINTS classroom (12/8/00)
- Teacher perceptions
of Cluster Information Specialists (8/7/00)
- Teacher perceptions
of eThemes services (8/2/00)
|
