Examining Effective Characteristics of Professional Development in K-12 Education Since the Inception of the No Child Left Behind Act of 2002: A Meta-Analytic Investigation
by Eugene M. Thomas and Karen H. Larwin
With the dire financial crisis facing our national and state economy, schools are forced to reduce budgets while simultaneously improve program delivery. Professional development is the mechanism that is generally used to facilitate improving educational delivery and subsequently student achievement results. This investigation examines the influence of professional development on student achievement since No Child Left Behind. Results indicate that professional development can have a moderate impact on student achievement. A number of moderators were found to have a positive significant impact on this effect including the level of students, the duration of the professional development, the discipline area focus of professional development, attendance requirements, delivery mechanism, and strategies. The implications of these findings are discussed.
Listen to Eugene Thomas, lead author discuss effective characteristics
of professional development, Youngstown State University.
Professional Development Defined
The term professional development is a universal phrase used by various fields to describe training of employees. Specifically for this investigation, "Professional development can be defined as a career-long process in which educators fine-tune their teaching to meet student needs" (Maggioli, n.d., p. 2).
History of NCLB, Parameters of the Law and Accountability Measures
The United States Department of Education was formed in 1867 to collect data and ensure that public education was successful by assisting states to develop effective systems (United States Department of Education, 2012). Even with this formation of the Department of Education, inconsistencies amongst the educational institutions that were funded with public tax dollars in the United States existed (Smith & Wohlstetter, 2001; Young, 2001).
Throughout the 130 years of existence, the United States Department of Education changed its form and functions under the executive branch to influence educational reform and create a better education system for all citizens. One major movement from 1979-2002 that had an influence on No Child Left Behind (NCLB) and improving teacher quality, which is deemed as a primary indicator of a student's academic success in school (Darling-Hammond & Berry, 2006; Dash, Magidin, O'Dwyer, Masters, & Russell, 2012; Dickson, 2002; Geringer, 2003; Lasley, Siedentop, & Yinger, 2006) was America 2000 that later evolved into Goals 2000 which derived from former Governor of South Carolina Richard Riley who was appointed as education secretary by President William Clinton.
A second event that contributed to the creation of NCLB occurred in 1987: A Nation at Risk. In an unparalleled report in 1983 the status of public education in the United States compared with other nations was deteriorating (A Nation at Risk, 1983). This report, combined with political pressures prompted the government to take action, led to the creation and implementation of NCLB.
Prior to NCLB, government granted states authority to institute public education under the 10th amendment of the Bill of Rights in the United States Constitution. With these provisions in place for almost 100 years, the conditions of inadequacy, lack of accountability, and inconsistent standards in public education led to the necessity for the federal legislation of NCLB. Even with the highly politicized and criticized law, organized federal influence and guidance in public education reached well beyond the spectrum of NCLB.
On January 8, 2002, the bipartisan support of Congress to reauthorize the Elementary and Secondary Education Act (ESEA) passed NCLB (VerBruggen, 2012). It was one of the most influential and organized movements in public education in an attempt to restructure and improve public education throughout the United States. NCLB had various sections addressing improved academics and teacher accountability. Among the components of NCLB was the aspect of high quality professional development as a key strategy to improve teaching and learning thus produce highly qualified teachers. Explained in NCLB under title nine, schools were to maintain that activities, "Are high quality, sustained, intensive, and classroom-focused in order to have a positive and lasting impact on classroom instruction and the teacher's performance in the classroom; and are not 1-day or short-term workshops or conferences" (NCLB, Title IX, Section 9101(34)).
With the onset of new guidelines and strategies for schools to implement, it was evident that NCLB had a direct on schools and districts. Essentially, the response by education organizations had been proactively addressed to meet the required standards through teacher training, specifically in the form of professional development.
NCLB Warrants the Need for High Quality Professional Development
Since NCLB's inception into law in 2002, there were a series of changes needed by schools and districts to implement the requirements of highly qualified teachers. In 2004, Margaret Spellings, Secretary of Education for the United States Department of Education, outlined "A Roadmap for State Implementation." Spellings (2004) explained that since its creation in 2001, education has been fundamentally changed: "NCLB was a national endorsement of the conviction that every child matters and that every child can learn" (p. 2). With her words, she described NCLB as a law of principles that involved all students in grades three through eight that measured student achievement annually leading up to 100% proficiency by 2013-2014 with a strong emphasis on teachers to reach this goal. She explained, "States are responsible for implementing a rigorous system for ensuring teachers are highly qualified, for making strong efforts to ensure that all students have access to highly qualified teachers, and for providing support for recruiting and retaining the best and brightest teachers for our schools" (Spellings, 2004, pp. 2-3).
Among various academic and funding features of NCLB, the law's heightened accountability measures for schools, districts and teachers required them to move well beyond the scope of their typical daily routines. NCLB influenced schools and districts to evaluate student data in specific academic content areas, develop plans for improvement in areas of weaknesses, and implement the plans including staff training. These new unforeseen measures forced districts to respond quickly in order to attain compliance with the law.
Through the new accountability standards under NCLB, the law was redirecting school districts to identify specific academic areas with specific student groups to raise student achievement. Schools that failed to raise achievement were faced with multiple sanctions by the state system, some severe. Nevertheless, the building of capacity to address the demands of NCLB was viewed through various lenses. Funding, reasonable goals, teaching strategies, assessments, and school reform all contributed to the success of attaining the compromised goals of improving student achievement for all students under NCLB. Thus, it was evident that NCLB encompassed more than just accountability among students and administrators. Teachers were required to accept leadership roles and take responsibility for their own learning and development that built capacity, stability, and civility among their students, classrooms, and individual buildings (Freiberg, et al., 2009). Respectively, every strategy under NCLB of school and academic improvement involved staff training (Guskey, 2003; Holloway, 2003; Hunt, 2006; McCarthy, 2006).
Progression of Effective Practices of Professional Development
In response to the high stakes accountability measures of NCLB to increase student achievement for all students, professional development was implemented as an instrument to improve instruction putting teachers at the forefront to attain increased results in student achievement (Blank & Alas, 2009; Smith & Gillespie, 2007; Wilson & Berne, 1999). Consequently, the professional development implemented had received mixed reviews about what worked and what was effective. With the various purposes that professional development was performed, it was difficult to narrowly define high quality professional development. With NCLB still at its infancy for reshaping what schools and teachers typically touted as success, the professional development activities that were implemented prior to and after NCLB fell short of the expected outcomes. Inconsistent results were prevalent across the different implementations of professional development, and reported effects on student achievement and teacher growth (Bullough, Kauchak, Crow, Hobbs, & Stokes, 1997; Cordingly, Bell, Rundell, & Evans, 2003; Lustic & Sykes, 2006; Ross, Bruce, & Hogaboam-Gray, 2006).
With the dilemma of inconsistent results and practices of professional development, the nation's educators faced a new challenge: find what works. With this, it was determined that a range of controllable characteristics such as curriculum development and teacher efficacy were deemed to have more of an impact on teachers rather than on student scores (Huffman, Thomas, & Lawrenz, 2003; McBride, 2006; Ross & Bruce, 2007).
Several experts identified subject area knowledge as a factor of student achievement and teacher effectiveness was identified. For example, teachers with a math or science degree have higher test scores than teachers who are teaching these subjects without expertise in the field (Brewer & Goldhaber, 2000; Meyer & Sutton, 2006; Monk, 1994; Monk & King, 1994; Rowan, Chiang, & Miller, 1997; Santau, Maerten-Rivera, & Huggins, 2011; Young & Lee, 2005). However, even with the ability to control and improve subject area knowledge, there were still mixed results in teacher performance.
Concurrent with content area expertise, a wide range of effective characteristics began to emerge from practices prior to NCLB and beyond. Many researchers discovered that professional development had similar effective characteristics across a wide range of subject areas taught in schools. Collectively, the strategies of enhancing knowledge and skills, modeling, creating a culture for professional growth, professional learning communities, empowering staff within to become leaders, and providing context to the various areas of educational systems were becoming prevalent as a norm for teacher professional growth and student achievement (Birman, Desimone, Porter, & Garet, 2000; Bruce, Esmonde, Ross, Dookie & Beatty, 2010; Darling-Hammond & Berry, 2006 Jeanpierre, Oberhauser, & Freeman, 2005; Johnson, Fargo, & Kahle, 2010; Niess, 2005; Scantlebury, 2008). Correspondingly, professional development that improves teaching generally results in increased student achievement, but these results are not consistent (Birman et al., 2000; Burkhouse, Loftus, Sadowski, & Buzad, 2003; Garet, Porter, Desimone, Birman, & Yoon, 2001; Guskey, 2002; Hill, Rowan, & Ball, 2005; Yoon, Duncan, Lee, Scarloss, & Shapley, 2007).
So, the question arises as to what are the aspects of successful professional development that create positive impacts on student achievement? With the barrier of time and human and financial resources, many districts struggled with providing professional development programs that reflected the effective characteristics of traditional practices. The overarching impact of professional development is not yet fully understood. In an effort to shed light on conflicting results and in an effort to examine what aspects of professional development impact student achievement, this dissertation meta-analyzes the existing research of professional development for teachers in the K-12 grade level since the onset of NCLB. As such, this meta-analytic study is the only know investigation that analyzes the impact of professional development on K-12 student achievement since the inception of NCLB (Hattie, 2009).
The analytical method for this study was a meta-analysis. A meta-analysis is defined as the, "Analysis of analyses" (Glass, McGaw, & Smith, 1981). The purpose of the meta-analysis is to analyze multiple studies in order to determine the significance of multiple variables against the outcome variable of student achievement. Glass et al. (1981) explained that a meta-analysis allows for studies with smaller sample sizes to be combined thus produce a much larger sample size, which in turn will increase the statistical power.
According to Glass et al. (1981), there are three necessary steps when performing a meta-analysis. The first step involves collecting research studies to analyze against the outcome variable. The studies collected must fit parameters of the overall analysis as well as match the data on the specified research topic. While performing and analyzing the search for relevant studies, it is likely that bias will be discovered. In order to minimize the bias, it is imperative to continue deeper into the search for available studies.
The second step, according to Glass et al. (1981), is to analyze the data. By the analysis, it is suggested that the studies be described, classified, and coded. An important aspect of this step involves measurement consistency. In order to obtain this, Glass et al. suggested coding the studies twice in order to establish rater agreement, which essentially is a score of homogeneity for the ratings. In order for this to occur, it is important to clearly define the moderator variables so that apparent differences are evident between the different classifications. This process creates reliability of the coding processes in data and is found to be reliable in the classifications more than 95% of the time.
The final step in a meta-analysis, according to Glass et al. (1981), is the analysis of the complete mean effect size measures including each individual mean effect size measure for each research variable being studied. Once all of the effect size measures are calculated, the results are analyzed, interpreted and reported as findings.
The current investigation sought to answer the following research questions:
1. What is the impact of professional development on student achievement?
2. Is that impact moderated (a) by school level (elementary, middle, high school, mixed); (b) by the duration of the teacher training or professional development; (c) across different delivery methods; (d) across different sample sizes; (e) across different subject areas (science, mathematics, or reading); (f) across the different characteristics of programs; (g) across the different characteristics of strategies; (h) across internal or external providers; (i) across different funding structures; across voluntary or mandatory participation; and, across different exam types?
3. Is the impact of professional development on student achievement moderated across different regional sources of data; across different years of published data; and across different sources of data?
The studies chosen for inclusion in this meta-analysis were found through extensive online searches utilizing databases over a nine month period included Google, Google Scholar, Educational Resources Information Circuit (ERIC), EBSCO, Electronic Journal Dissertations (EJC), edgov, and JSTOR. Most of the research spanned from 2000-2012 with some of the research spanning from 1992-2000 in order to establish a historical perspective of studies prior to NCLB.
The specific descriptive search criteria included history of NCLB, opponents of NCLB, accountability measures of NCLB, NCLB professional development, NCLB certification, influence of NCLB on professional development, effective practices of professional development, ineffective practices of professional development, online professional development student achievement, advantages of online professional development, disadvantages of online professional development, and online professional development and student achievement. Abstracts, summaries, and table of contents of articles were reviewed in order to select which studies to include. The criteria to include articles were: (1) articles that addressed public schools grades K-12, (2) articles that addressed student achievement, and (3) studies that addressed improvement in student achievement scores in various subject areas.
Once a substantial number of articles, dissertations, and presentations were found, these were downloaded electronically and printed. The articles were carefully analyzed in order to determine whether the information and data were pertinent to this study. The articles that were not relevant were discarded. Reference lists from the various articles were examined in order to produce any additional references that did not surface during the initial search process in which the screening to include or discard newly identified references was repeated. Overall, there were more than 500 initial resources identified for this study in which more than half were discarded due to generalized claims and discrepancies in the research. The elimination process left 204 resources to be considered in the study. After careful consideration and discretion, 115 studies were selected to be used primarily in the literature review as well as other chapter sections. From the 115 sources, 90 mention professional development and student achievement but only have data on efficacy or results about teachers' scores on some measure. Also, 17 studies that included inferential quantitative data such as means, standard deviations, variances, t-tests, f-tests, and chi-square data were selected from the 115 studies for the meta-analysis and are denoted in the reference list with an asterisk.
The primary purpose of this study was to identify characteristics of professional development that impact student achievement, since the inception of NCLB. The study yielded a total of 17 studies for the meta-analysis which, included nine research journals or research articles, five research reports or government reports, two dissertations, and one conference presentation. Many of the studies contained multiple measures, resulting in a total of 53 effect size measures from the 17 studies. There was a student sample size of n = 69,556 and an overall total of 14 moderators extracted from the studies. The effect size measures within the study range from -0.523 to 1.613, yielding a grand mean overall effect size measure d = 0.353, p <.001, a significant moderate-to-large effect according to Cohen (1992).
Forty-seven of the 53 effect sizes (89%) that were used in this study were positive which implies that many of the moderators had a positive impact on student test scores. Six of the 53 effect sizes (11%) that were used in this study were negative which implies that few moderators demonstrated little impact on student test scores. The analyses also revealed that five (29%) of the 17 studies had a mean effect size of 0.5 or greater, which implies that the effects of professional development on student achievement according to Cohen (1992) is considered large. Table 1 provides a detailed breakdown of the 17 studies that met the criteria to be included in the study.
Meta-Analysis Results by Moderator and Levels
Analysis of moderators revealed a number of interesting results. Research on the impact of professional development on student achievement reveals the greatest impact is found in studies including high school and middle school age students. As expected, professional development that was extended over a period of time produces larger effects. Greatest impacts were found for professional development in the discipline of science, followed by reading. And, packaged professional development programs, those presented by external providers, and those which were internally funded revealed the greatest impact. Also noteworthy, mandatory attendance to professional development provided the largest impact measures. The specific results for each moderator, broken out for each level of the moderator, are presented in Table 2.
The basic issue of publication bias is that not all completed studies are published, and the selection process is not random (hence the 'bias'). Rather, studies that report relatively large treatment effects are more likely to be submitted and/or accepted for publication than studies which report more modest treatment effects. Since the treatment effect estimated from a biased collection of studies would tend to overestimate the true treatment effect, it is important to assess the likely extent of the bias, and its potential impact on the conclusions (Egger, Smith, Schneider, & Minder, 1997; Sterne & Egger, 2001).
Begg and Mazumdar (1994) suggested an inverse correlation approach as a statistical test for publication bias. Concretely, they suggest that computing the rank order correlation (Kendall's tau b) between the treatment effect and the standard error (which is driven primarily by sample size). The current investigation reveals a Begg and Mazumdar Rank Correlation, τ = .103, p = .279, revealing a non-significant correlation and the large sample size of the study supports that publication bias is not a concern in the current investigation.
Summary of Findings
This investigation examined the effectiveness of 14 moderators derived from the studies of the characteristics used in professional development. The dependent variable of student achievement was selected as the measure of the effect, if any, of each moderator. Overall, the investigation included 17 research studies for inclusion in the meta-analysis in which 14 moderators were identified. Within these 17 studies were 53 effect size measures. The overall student sample size was 69,556. Forty-two of the 53 effect sizes yielded significant positive results. Out of the 14 moderators, 12 revealed significant results.
Analysis of Moderator Variables
The moderator variables of funding and publication year revealed that achievement is not significantly different across different funding levels or publication years. Contrary to these findings, the moderator variables of school level, dosage, delivery, sample size, subject area, programs, strategy, providers, attendance, exam type, location, and source revealed a significant effect across all levels. Correspondingly, some levels within the significant moderators revealed a large significant effect that is relevant and necessary to consider when developing professional development models. Other levels that typically heed attention did not measure up to expectations. Analyses of the studies provided in this investigation and research from other sources present a clearer understanding about the findings of the current investigation.
The Levels of Efficacy, Content, and Pedagogy
During the review of research for this investigation, there was a resonant presence of variables of teacher and student efficacy. Specifically, Huffman, Thomas & Lawrenz (2003), McBride (2006), Ross and Bruce (2007), and Ross et al. (2006) suggested that professional development focusing on efficacy can have some positive impact on student achievement. Due to these connotations, the level of efficacy under the moderator of strategies was included as part of the meta-analysis. The results of the meta-analysis revealed that under the moderator of Strategy, the level of Efficacy, d = .070, p <.001 had no effect on student achievement when examined comprehensively across the included studies.
Contrary to the level of Efficacy, the moderator of Strategy revealed the mean effect size of the levels Pedagogy, d = .461, and Content and Pedagogy, d = .382, suggested a significant moderate-to-large effect. These two levels represent the importance of in-depth knowledge of content areas (Birman et al., 2000; Bruce et al., 2010; Darling-Hammond, & Berry, 2006 Jeanpierre et al., 2005; Johnson, Fargo, & Kahle 2010; Niess, 2005; Scantlebury, 2008) combined with the art of teaching are likely to have a positive impact on student achievement (Birman et al., 2000; Burkhouse et al., 2003; Garet et al., 2001; Guskey, 2002; Hill, et al., 2005; Yoon et al., 2007).
With the heightened focus of teacher accountability in relation to student test scores since the inception of NCLB, states and districts were positioned to make a swift shift in focus and begin to hold educators more accountable for student results (Blank & Alas, 2009; Smith & Gillespie, 2007; Wilson & Berne, 1999). As a result, school leaders were presented with the daunting task to design professional development programs for teachers in a strategic manner. A clearer quantified understanding of the impact of teacher and student efficacy in the specific subject areas and instructional strategies would provide much needed insight about the types of targeted professional development that would assist teachers to attain improved student results. Some studies suggested that there is logical and sense of intuitive connectedness between professional development and student achievement (Borko, 2004; Loucks-Horsley & Matsumoto, 1999). This investigation revealed that effects that occur in efficacy may not be sustained or powerful enough to impact student learning. Other studies of professional development caution that effects of efficacy has not been examined in a manner that is quantitative and replicable (Desimone, Porter, Garet, Yoon, & Birman, 2002). With this said, teacher and student efficacy is a variable worthy of further investigation as the nation rapidly shifts toward data driven results to measure success.
The Levels of Face-to-Face and Online
Today's leaders and educators are faced with the obstacles such as time and funding when developing and implementing professional development programs. Compounded by these obstacles and the statute of NCLB requiring high quality professional development (NCLB, Title IX, Section 9101), the delivery method of online training is an emerging practice. In this investigation, the moderator of Delivery revealed that achievement is significantly different across the three levels with the level Online having no impact. These results are consistent with the study of Dash et al. (2012) regarding the level of Online is flat. Darling-Hammond (2005) suggested that two key features that are present in highly effective professional development including collaboration and teacher reflection. When compared with traditional professional development, online professional development lacks these two key components. Also, this may be the result of the use of online delivery being rather infantile and with time this delivery model can improve. Further, when compounded with other factors such as lack of technology literacy, poor infrastructures, participant technology experiences that interfere with independent learning and availability of specifically targeted topics or groups, it is evident through this investigation why the level of Online is flat (Armstrong et al., 2000; Collins & Berge, n.d; Dede, 2006; Galley, 2002; Stanford-Bowers, n.d.; Tyler-Smith, 2006).
Contrary to using online strategies as a solitary method of professional development the traditional method of face-to-face is recommended. In this investigation the mean effect size of the level of Face-to-Face and Online,d = .613, revealed significant large effect on achievement. Even with the large effect, only two effect size measures support this anomaly. With this said, it is reasonable to suggest that there is insignificant research available to justify a combination of the two levels. The other level of Face-to-Face, d = .348, revealed a significant moderate effect. Respectively, Face-to-Face is more than justified as an essential characteristic being that 50 effect size measures make up this level. Specifically, this investigation supports Pritchard and Marshall's (2002) claims that leaders who embed teacher training during the work day increase the probability of increased student achievement.
The Levels of Math, Reading, and Science
Since the onset of high stakes testing, many leaders believed that professional development focused on reading as a strategy to increase test scores across all subject areas. Specifically every core subject area tested involved the ability to comprehend and analyze literature. A national study about the implementation of NCLB at the state and local levels reported that 80% of elementary teachers participated in 24 hours of professional development for reading instruction or less during the 2003–2004 school year (U.S. Department of Education, 2012). According to reading experts, this dosage level raises concerns that the level of reading suggesting that it is not intensive enough to have an impact and that it does not focus enough on subject-matter knowledge (Cohen & Hill, 2001; Fletcher & Lyon, 1998; Foorman & Moats, 2004; Garet et al., 2001). This investigation supported these findings under the moderator of Subject Area and the level of Reading, d = .392, revealing a moderate-to-large effect. The idea that an emphasis on reading improves mathematics speaks at length about the impact of reading on mathematics (Larwin, 2010) but falls short of the expectation with this investigation revealing the level of Math, d = .192, having a small-moderate effect.
Worthy of note is the mean effect size of the Science level, d = .653, which revealed a significant large effect on achievement. At the onset of NCLB, the testing movement focused primarily on reading and mathematics. States were required by the 2005-2006 school year to measure all students' progress in reading and math in grades three through eight and at least once in grades 10 through 12. More recently, states were required by the 2007-2008 school year to have in place science assessments to be administered at least once during grades three through five, grades six through nine, and grades 10 through 12. The shift of focus from the subjects of reading and math to science compounded with the evolution of healthy professional development practices and focus on pedagogy (Desimone et al., 2002) suggested support for the large effect size measure.
The Levels of External and Internal
The moderator of Providers revealed that the mean effect size of the level External, d = .455, revealed a significant moderate-to-large effect. Amazingly, this supports a shift in philosophy that professional development be implemented in-house with existing staff. Contrary to what is often practiced in schools, this investigation revealed that the level of Internal has no effect.
A recent study examining the effectiveness of professional development schools (PDS) looked specifically at using external experts at the university level for training purposes (Creasy, 2011). Creasy (2011) found that, "Classroom teachers and teacher preparation institutions have identified a gap between research and practice. Teachers and university personnel in professional development school settings seek to build the bridges that allow schools and universities to benefit from this mutual relationship" (p. 19). This idea of reconnecting practitioners with external theorists over an extended time period (French, 1997) can reinvigorate the identification of effective methodologies and lead to teacher change.
Overall, when considering the levels within the 14 moderators are meta-analyzed, it is determined that the impact of professional development on student achievement since the implementation of NCLB is significant, d = 0.353. This significance level alone is a phenomenon that warrants acknowledgement. Generally, it is rare in the field of research to show an impact on student achievement with interventions that are not directly delivered to the student. Even so, it is imperative to reiterate that student test scores are not always considered the best measure of student achievement (Bell et al., 2010; Coleman et al., 1966; Jencks et al., 1972; Popham, 2001), however this is the reality of our current educational system.
In examining ways to improve student achievement by using professional development in this investigation and the countless hours of reading and dialogue with colleagues, it seems that many would like to find a "magic bullet" a single variable that increases student achievement. This investigation overwhelmingly supports the claim that improving student achievement is a multi-faceted issue and the answers are equally as complex.