It is Difficult to Review the Quality of Education if You Don’t Know What Children are Learning: ERO and Self-Review of Early Childhood Services
By Ken Blaiklock
Blaiklock, K. (2012). It is difficult to review the quality of education if you don’t know what children are learning: ERO and self-review of early childhood services. NZ Research in Early Childhood Education Journal, 15, 1-10. Retrieved from http://www.childforum.com/research/2012-nzrece-journal-articles/865-self-review-education-review-office-early-childhood.html
A critical look at the New Zealand Education Review Office’s draft guidelines for reviewing early childhood services is provided in this paper. ERO’s guidelines provide a range of “evaluation indicators” to inform centres as to what is considered to be effective practice. The indicators that ERO provides in relation to the assessment of children’s learning are supportive of current practices in the use of narrative assessment or Learning Stories. This paper argues that such practices provide little evidence about what children are learning in early childhood centres, thereby making it very difficult to assess programme quality. The paper also argues that ERO needs to make more extensive use of international research on effective assessment strategies rather than limiting its theoretical and research base to mostly local approaches.
Key words: Education review, quality, self-review, learning.
The Education Review Office (ERO) has a crucial role in evaluating the programmes provided in around 4,439 licensed early childhood education services. The government and public rely on ERO to ensure services are providing programmes that cater for the well-being and learning of children. The role of ERO has become increasingly significant with more children being enrolled in centres, often for at least 20 hours per week. Currently, 94% of New Zealand children have some participation in early childhood education before they turn 5 years old (Ministry of Education, 2011).
As child participation rates have risen, and the number of centres has grown, the proportion of ERO’s resources devoted to reviewing early childhood services as opposed to schools has increased. In 2011, ERO spent $9.6m on reviewing early childhood services compared to $15.5m for primary and secondary schools (Education Review Office, 2011). Recently, ERO announced a revision of the methodology that it uses to review early childhood services (Education Review Office, 2012a). The draft guidelines for this new approach are aimed at ensuring “that reviews are undertaken efficiently and effectively in a growing sector” (p. 1) and include a greater emphasis on making use of the self-review processes undertaken, or expected to be undertaken, in every licensed centre.
ERO defines self-review as “the use of robust processes to systematically inquire into and evaluate the effectiveness of policies, programmes and practices. Self-review findings are used to inform decision-making, improve the quality of practice and promote positive outcomes for all children” (Education Review Office, 2012a, p.12). Self-review in centres has been promoted and supported by earlier Ministry of Education publications such as The Quality Journey (1999) and Ngā Arohaehae Whai Hua (2006). ERO has now signalled that it intends to carry out its reviews in ways that build on and complement the internal reviews carried out by centres. (The new guidelines do not apply to home-based services, kohanga reo, or hospital-based services.)
An increased emphasis on self-review in early childhood services follows on from the success of this approach in primary and secondary schools in New Zealand. ERO’s approach to evaluating school effectiveness was recently endorsed in an OECD report on school evaluation (Nusche, Laveault, MacBeath, Santiago, 2012). The report commented favourably on the use of a collaborative approach where a school conducts their own internal review followed by an external review carried out by ERO. The report also noted the importance of evidence-based assessment for ongoing improvements in teaching and learning in schools.
Although self-review may have been shown to be a useful part of the evaluation process for New Zealand schools, there is little evidence about its effectiveness for improving early childhood programmes. In 2009, ERO published a report on how self-review was being carried out in a sample of 397 services. The report examined “how well self-review was understood, supported and implemented in each service and the extent to which it led to improved management and educator practice” (Education Review Office, 2009, p. 1.). ERO found wide variation between services in self-review practices. Apart from a few short anecdotes (pp. 15-16), the report contained no evidence that self-review results in higher quality care and learning for children in centres.
In this article, I will present an argument that ERO is misguided in its move towards a greater focus on self-review in early childhood centres. The argument will not be on the processes of self-review but on the lack of valid information that centres have on the effectiveness of their programmes for enhancing children’s learning. A lack of valid information makes it very difficult for centres, or for ERO, to evaluate the quality of the programmes being provided.
The draft guidelines for the Education Review Office’s new methodology for reviewing early childhood services are published in a document titled ERO’s Approach to Reviews in Early Childhood Services” (2012a). ERO anticipates that the final version of the document will be published in 2013, following consultation with interested parties. The guidelines set out the principles underpinning ERO’s reviews, including “equity of educational opportunities and outcomes” and being “informed by evidence” (p.4). ERO states that each review of a centre will examine compliance with regulated standards but “focuses particularly on the quality of provision” [original emphasis] (p. 6).
In relation to quality, ERO reinforces the value of Te Whāriki (Ministry of Education, 1996) as a framework for guiding children’s learning. But ERO’s own description of the generalised nature of Te Whāriki points toward the difficulty in using the curriculum as a guide to what children should be, or indeed are, learning. ERO states:
According to Te Whāriki the outcomes of a curriculum are knowledge, skills and attitudes that combine together to form a child’s working theory and help children to develop dispositions that encourage learning. Positive outcomes for children include high level competencies as well as more specific knowledge, skills, attitudes and dispositions for learning (Education Review Office, 2012a, p. 7).
The outcomes in Te Whāriki, however, are usually phrased in very general terms that provide little guidance as to what children in a particular age range could be expected to learn. Most of the outcomes appear to have been written to apply to any child from birth to five, regardless of age or developmental level. Hence, the outcomes provide no information about typical developmental sequences that occur as children grow from infancy through to school age. When describing what children are learning, the outcomes use phrases such as “an understanding of …” “a capacity to …”, increasing knowledge about…”, and “familiarity with …”. The vague nature of these descriptors makes it very difficult to use any of the outcomes for assessment purposes. Furthermore, the learning outcomes in Te Whāriki are “indicative rather than definitive” (Ministry of Education, 1996, p. 44). No learning outcomes are prescribed as being mandatory to include in a programme.
In spite of a lack of clear or specified learning outcomes in Te Whāriki, the Education Review Office claims, “the focus on how well children learn is central to all ERO reviews” (2012a, p. 7). Moreover, ERO suggests that its entire approach to reviews is linked with “research information on how different factors and features of early childhood practice contribute to positive learning outcomes for children (p. 7).
Such statements reflect ERO’s stated emphasis on evaluating how programmes contribute to children’s learning. But if Te Whāriki does not provide assessable outcomes, how then does ERO suggest that outcomes for children should be evaluated? A major part of ERO’s approach to Reviews in Early Childhood Services is dedicated to outlining ‘evaluation indicators’. These indicators are provided “to clarify the basis on which ERO evaluates early childhood performance and to assist in the early childhood service’s self-review” (p. 23).
The evaluation indicators are categorised into four interconnected areas that reflect the structure of the overall review framework, known as Nga Pou Here. The framework adopts the Maori concept of pou to represent each of the four areas.
- Pou Whakahaere (Governance and Management)
- Pou Ārahi (Leadership)
- Mātauranga (Curriculum and Assessment)
- Tikanga Whakaako (Teaching and Learning)
The indicators for the four Pou are many and varied, filling 13 pages of the document. In this article I will focus on the indicators that ERO uses to evaluate whether an early childhood service is gathering appropriate assessment information about children’s learning. This is a crucial area because, as noted above, a key focus of ERO’s work is to evaluate whether programmes are promoting positive learning outcomes for children. The indicators that ERO uses are important for signifying what ERO regards as effective assessment practice. Hence, the indicators also act as a guide for what ECE services should be doing to monitor and document children’s learning, and as criteria to consider when carrying out self-reviews.
The evaluation indicators for assessment are included in a subcategory of the Mātauranga (Curriculum) Pou. Twenty indicators are listed (2012a, p. 33). (See Appendix.)
ERO claims its
indicators for education reviews in early childhood services are based on current national and international evaluation and research, including ERO’s national evaluations, evaluation theory and practice, and many years of experience within ERO. The relevant findings of the evaluation and research have been analysed to identify indicators of high quality early childhood education (p. 23).
Such statements would suggest the indicators point the way to effective assessment practices that result in centres collecting valid evidence about what children are learning. However, a closer look at the indicators reveals they mostly consist of general statements reinforcing existing practices in the use of narrative assessment or Learning Stories. For example, “assessment information focuses on enhancing dispositional learning, as well as skills and ways of knowing”, “reflects a credit-based approach that pays attention to children’s strengths, interests and dispositions,” and “is available to the children so that they can revisit and share their learning with others.” “Assessment builds children’s identity [sic] as a successful learner” and “assessment processes support children to understand and contribute to decisions about their learning” (p. 33).
Learning Stories are a locally developed assessment technique requiring teachers to write narrative descriptions of children’s experiences in particular settings (Carr, 1998, 2001; Carr & Lee, 2012). The focus is on dispositions towards learning rather than on knowledge and skill development. Learning Stories are usually accompanied by photos and can provide anecdotal reports of a child’s involvement in selected aspects of an early childhood programme. There is little evidence, however, that Learning Stories are a practical and effective way of assessing the complexity of children’s learning in the early childhood years (see Blaiklock, 2008, 2010).
Some of the concerns with Learning Stories include:
- problems defining particular dispositions.
- a lack of rationale for the links between particular dispositions and the strands of Te Whāriki.
- a high level of subjective interpretation when describing and analysing a child’s learning.
- a lack of guidance on what learning areas to assess and when.
- a lack of evidence that Learning Stories are effective in showing changes in children’s learning and development over time.
- problems in using a Learning Story about a specific incident as the basis for planning future learning in different contexts.
The Education Review Office has previously endorsed the use of Learning Stories through its reviews of early childhood services and in a national report on The Quality of Assessment in Early Childhood Education (2007). Given that the Ministry of Education has invested considerable resources in professional development to promote the use of Learning Stories, ERO may have thought it was appropriate to continue to support this approach in their latest guidelines. ERO could make use of its special role as an independent arbiter of educational quality, but to date it has not. ERO is a separate government agency from the Ministry of Education and is in a position to question current early childhood assessment practices in New Zealand, especially when current approaches are not supported with sufficient evidence to show their effectiveness.
The Education Review Office claims that the evaluation indicators are based on “current national and international evaluation and research” (2012a, p. 23). They further note, “ERO conducted a wide ranging literature search on research related to early childhood education over the past 5-10 years. The search was broad enough to encompass diverse theoretical and methodological perspectives” (p. 23). Such a search of the research evidence should have alerted ERO to the limitations of a focus on narrative approaches. However, ERO goes on to acknowledge, “priority was given to New Zealand research in order to provide localised perspectives on best practice in early childhood education” (p. 23).
The priority given to ‘localised perspectives’ may explain why ERO appears to be unaware of much of the international research on the assessment of young children. The list of references ERO consulted to develop the evaluation indicators is substantial, but largely limited to New Zealand sources (see Education Review Office, 2012b). Hence, it is not surprising that ERO has supported current local practices. Fifty eight of the 65 references provided for the Mātauranga Pou (Curriculum and Assessment) are by New Zealand authors. Of the remainder, 5 are to authors from Australia, 1 from England, and 1 from Canada. No references from the United States are included, in spite of this being a major source of research evidence on effective assessment. (e.g., Bagnato, 2007; Brassard & Boehm, 2007; Snow & Van Hemel, 2008).
ERO has taken a very selective approach in its review of research. Although ERO (2012a) sought to cover “diverse theoretical and methodological perspectives” (p. 23), this is not evident in the references that are provided. The international research on effective ways of assessing young children is largely ignored in ERO’s report. This may explain why the evaluation indicators make no mention of fundamental tenets of assessment such as validity and reliability. Validity, or whether an assessment measures what it claims to measure, is a crucial quality when evaluating children’s learning. Reliability, the stability or consistency of an assessment, is also very important (Snow & Van Hemel, 2008).
The evaluation indicators for assessment also make no mention of developmental patterns in growth and learning, the importance of assessing specific areas of learning (e.g., language development) and the need to use valid means to identify children who may benefit from early intervention. These considerations currently receive little attention in narrative based assessment in New Zealand. ERO’s new methodology for reviewing centres does nothing to change this situation.
No textbooks on assessment are included in ERO’s references for the evaluation indicators. If ERO had consulted some of the widely used international texts (e.g., Beaty, 2005; Gullo, 2005; Martin, 2007; Wortham, 2012) they may have become more aware of the limitations of New Zealand’s current emphasis on Learning Stories as the dominant, and often the only, form of assessment in early childhood centres.
In communications with teachers, ERO has promoted the evaluation indicators as “representing an in-depth look at best practice and research” (Bleasdale, 2012, p. 3). The indicators for assessment may represent existing practice in New Zealand but it is questionable that they represent best practice.
The Education Review office has called for comment on its new methodology for reviewing early childhood services. It is to be hoped that ERO will reconsider the advice they are providing on assessment before finalising the guidelines. Taking a less insular approach, and learning from international research and evidence, will result in more effective reviews and, in turn, enhanced quality of programmes for children.
Bagnato, S. J. (2007). Authentic assessment for early childhood intervention: Best practices. New York: Guildford Press.
Beaty, J. (2005). Observing development of the young child (6th ed.). Upper Saddle River, NJ.: Prentice Hall.
Blaiklock, K. E. (2008). A critique of the use of learning stories to assess the learning dispositions of young children.New Zealand Research in Early Childhood Education, 11, 77-87.
Blaiklock, K. (2010). The assessment of children's language in New Zealand early childhood centres. New Zealand Journal of Educational Studies, 45(1), 105-110.
Bleasdale, K. (2012). Revising ERO's approach to early childhood reviews. Education Gazette, 91(12), 2-3.
Brassard, M. R., & Boehm, A. E. (2007).Preschool assessment: Principles and practices. New York: Guilford Press.
Carr, M. (1998). Assessing children's learning in early childhood settings: A professional development programme for discussion and reflection. Wellington, New Zealand: NZCER.
Carr, M. (2001). Assessment in early childhood settings: Learning stories. London: Paul Chapman.
Carr, M., & Lee, W. (2012). Learning Stories: Constructing learner identities in early education. London: Sage Publications.
Education Review Office. (2007). The quality of assessment in early childhood education. Wellington: Education Review Office.
Education Review Office. (2009). Implementing self-review in early childhood services. Wellington, New Zealand: Education Review Office.
Education Review Office. (2011). Annual report. Wellington: New Zealand Government.
Education Review Office. (2012a). ERO's approach to reviews in early childhood services. Wellington, New Zealand: Education Review Office.
Education Review Office. (2012b). References for ECE evaluation indicators Wellington, New Zealand: Education Review Office. Retrieved from http://www.ero.govt.nz/Review-Process/For-Early-Childhood-Services-and-Nga-Kohanga-Reo/ERO-Reviews-of-Early-Childhood-Services/ERO-s-Approach-to-Reviews-in-Early-Childhood-Services-DRAFT-2012
Gullo, D. F. (2005). Understanding assessment and evaluation in early childhood education. New York: Teachers College Press.
Martin, S. (2007). Take a look: Observation and portfolio assessment in early childhood (4th ed.). Toronto: Pearson.
Ministry of Education. (1996). Te whāriki: He whāriki mātauranga mō ngā mokopuna o Aotearoa: Early childhood curriculum Wellington: Learning Media.
Ministry of Education. (1999). The quality journey. He haerenga whai hua: Improving quality in early childhood services. Wellington, New Zealand: Education Review Office.
Ministry of Education. (2006). Ngā arohaehae whai hua: Self-review guidelines for early childhood education. Wellington, New Zealand: Learning Media.
Ministry of Education. (2011). Annual report 2011. Wellington, New Zealand: Ministry of Education.
Nusche, D., Laveault, D., MacBeath, J., & Santiago, P. (2012).OECD Reviews of evaluation and assessment in education: New Zealand 2011. OECD Publishing.
Snow, C. E., & Van Hemel, S. B. (Eds.). (2008). Early childhood assessment: Why, what, and how. Washington, D.C.: National Academies Press.
Wortham, S. C. (2012). Assessment in early childhood education. Boston, MA.: Pearson.
Evaluation Indicators for Mātauranga. Assessment and Planning Processes (Education Review Office, 2012a, p. 33)
- Teachers take account of bicultural approaches to assessment.
- Assessment acknowledges the social and cultural worlds, and ways of learning of all children.
- Assessment supports the development of a strong Maori identity in Maori children, through reflecting Maori perspectives of the child, their world and their place in the world.
- Maori children’s cultural capital is acknowledged and valued and their learning achievements are celebrated.
- Assessment practices value and respond to Pacific cultures, knowledge and ways of learning.
- Assessment information
- focuses on enhancing dispositional learning, as well as skills and ways of knowing.
- shows deepening of, and the increasing complexity of, children’s learning.
- reflects children’s relationships with the environment, building connections between children, their communities and physical environments.
- reflects a credit-based approach that pays attention to children’s strengths, interests and dispositions.
- is available to the children so that they can revisit and share their learning with others.
- Assessment includes multiple perspectives that enhance the interpretation and analysis of learning.
- Assessments illustrate and support continuity in learning and demonstrate children’s progress in a range of contexts.
- Assessment builds children’s identity as a successful learner.
- Ongoing observation of children in everyday activities builds a picture of what children know, understand, feel, are interested in, and can do.
- Teachers have processes to analyse assessment information to understand children’s learning pathways and then plan to continue and strengthen them.
- Assessment processes:
- support children to understand and contribute to decisions about their learning
- provide parents and whanau with a way of contributing to their children’s learning.
- Planning is driven by evidence-based formative assessment, for individual and groups of children.
- Information from individual and group narratives is used to plan the curriculum
About the Author
Ken Blaiklock is a senior lecturer in the Department of Education at the Unitec Institute of Technology in Auckland. His research and teaching interests focus on assessment, and the development of children's language and literacy skills.
Oops ... you are attempting to view an article or a resource in the member-only area for Educators. This is available exclusively to our Educator members.
To keep reading, you need to login with your Educator membership login
Not a member? Look below ↓ for the click here button ↓ It will take you to the membership page to sign up and choose your own unique username and password.