OFSTED has recently appointed a new head of research. Judging by the recently launched early years report: Bold beginnings: the Reception curriculum in a sample of
good and outstanding primary schools this appointment has not come too soon.
The report has been criticised by educationalists for being politicised and ideological. In this BLOG I highlight some dos and don’ts of evidence-informed reporting using the report cited above as an example. I conclude with the view that without a rigorous approach to data, policy reports can appear ideological as opposed to evidence-informed.
1. Do be clear on the research question(s)
The first, and probably the biggest problem with the report, appears on Page 2 Paragraph 1. For the purpose of clarity, I have differentiated between the commissioned review and what appears to be the intention of the report (see below):
In January 2017, Her Majesty’s Chief Inspector (HMCI) commissioned an Ofsted-wide review of the curriculum. Its aim was to provide fresh insight into leaders’ curriculum intentions, how these are implemented and the impact on outcomes for pupils.
This report shines a spotlight on the Reception Year and the extent to which a school’s curriculum for four- and five-year-olds prepares them for the rest of their education and beyond (Page 2).
Is the report about leaders’ curriculum intentions or the link between the curriculum and how it prepares four and five years old for an unsubstantiated period into the future? The aim of the report is not clear, and the rest of the report reflects that lack of clarity.
2. Do make sure that the methodology, purpose, and aims of the report are consistent
Point 1 is re-enforced by the stated methodology and purpose of the review Page 31 para 83.
Her Majesty’s Chief Inspector commissioned this thematic survey as part of a wider review of the curriculum in England. Its purpose was to gain greater insight into leaders’ curriculum intentions and how these are implemented (P31 para 83).
It is easier to envisage a thematic review offering insight into leaders’ curriculum intentions but less easy to see how it could provide evidence for how the curriculum prepares four- and five-year-olds for the rest of their education and beyond.
Similarly, there are 4 different types of data included in the document:
- Government documents;
- Unstructured interviews;
- Online questionnaire;
- Participant observation.
There is, however, no description of coding methods or analytical approaches. It is perfectly feasible to take data from numerous sources and create a substantive dataset but that requires an appropriate methodology and coding structure. There is little evidence of either in the report.
The research decisions made during the review are not clear either (see below). For example, data collected for grade 1 and 2 schools (participatory observation, documentary evidence, and online questionnaire), is different from the data taken for grade 3 and 4 schools (OFSTED inspection reports). A clearer explanation is required as to why that decision was taken and its impact on the dataset.
In addition, HMI analysed more than 150 inspection reports of primary schools inspected in the 2016/17 academic year where early years provision, which includes the Reception Year, was judged as requiring improvement or inadequate.
This information was compared with the information and observations
undertaken in the 41 visits to good and outstanding schools in order to identify any differences in practice (Page 4 paragraph 9).
The reader is left wondering not only what question(s) the review intended to ask (point 1); but also how those questions were answered?
3. Do offer a rationale for sampling
Schools sampled for the review are predominantly in areas of deprivation with 70% of the sample in the 40% most deprived areas of England. No explanation is given nor does it seem to be appropriate given the intention of the commissioned review. A rationale for the sample and a clear explanation of sampling anomalies would be useful.
One insight into the sampling method suggested that:
The schools were chosen because they typified the findings of longitudinal research
into the importance of high-quality, early education; namely, that children who do well by the age of five have a greater chance of doing well throughout school (Page 31 para 86).
The statement begs a number of questions:
- The name of the Longitudinal report(s)?
- The authors of the longitudinal report(s)?
- What were the findings of the report(s)?
- What were the characteristics of the schools that meant they were selected?
- Why select those characteristics?
- How does the sample correlate with the stated intention of the report: “to provide fresh insight into leaders’ curriculum intentions”?
- What efforts were made to ensure pre-conceived notions of high-quality did not unduly impact upon the research findings?
Again the question is, how does this meet the original aim of the report? Did the sampling intend to triangulate the findings of the longitudinal report(s) or was the sampling conducted in order to reflect the views of leaders in schools with those characteristics? Does the report do either, or does it just reflect a prevailing view based upon earlier research findings?
4. Do be cautious about inferring causality from qualitative data analysis
Thematic analysis generates a rich description of a phenomenon. It does not necessarily confer causality onto the description generated.
For example, observing the “act of”, “reference to” or “belief in” holding a pencil correctly does not confer causality onto the act. It is not possible to know that sitting at a table improves letter formation without research that offers evidence of such a causal link; for the report to infer that such a link was established from data taken during the review (see below) is problematic:
In schools visited where writing was of a high standard, the children were able to write simple sentences and more by the end of Reception. They were mastering the spelling of phonically regular words and common exception words. These schools paid good attention to children’s posture and pencil grip when children were writing. They used pencils and exercise books, while children sat at tables, to support good, controlled letter formation (Page 5).
Another example is cited below: “reading was at the heart of the curriculum in the most successful classes”. One obvious question is whether data was taken at “class level”; if so, how and how much?
Reading was at the heart of the curriculum in the most successful classes. Listening to stories, poems and rhymes fed children’s imagination, enhanced their vocabulary and developed their comprehension. Systematic synthetic phonics played a critical role in teaching children the alphabetic code and, since this knowledge is also essential for spelling, good phonics teaching supported children’s early writing (Page 4).
5. Do not use woolly terms: “some of”, “most of”, “too many” and “too few”?
A lack of specificity does not help to identify themes and patterns in the dataset. In the example below “one” leader morphs into “leaders” who collectively claim that a “five-a-day” programme revived reading for leisure and improved attainment (see below).
There is no valid evidence presented that the statement is true nor is there a description of the relationship between “a leader” and “leaders”, and whilst beliefs are a legitimate concern of thematic qualitative research the opinion of one leader does not constitute a theme.
One early years leader referred to their ‘five-a-day’ read-aloud programme, engaging children in five stories, narratives and information texts each day. This included the story told during whole-school assembly, the texts shared as part of children’s literacy and the text chosen for the story time before lunch and/or at the end of the school day. The children were immersed in the sounds and experiences of the stories. Leaders believed this had revived reading for leisure and made a significant difference to children’s attainment in reading and writing (Page 19 para 41).
Similarly, there is a reference to “some schools” recognising the need for broader experiences in the curriculum. How many did?
Some schools recognised that the broader experiences and opportunities they offered children formed part of their curriculum:
■ the beginnings and ends of each day
■ snack and lunchtimes
■ hygiene routines
■ outdoor learning
■ the use of the school hall for physical education.
(Page 13 Para 14)
6. Do not make unreferenced claims
There are numerous examples of claims made that do not seem consistent with the research methodology adopted. For example, the claim is made that curriculum in the reception year can be the difference between 7 Cs and 7 Bs at GCSE (see below). A claim of this nature needs referencing.
The research is clear: a child’s early education lasts a lifetime. Done well, it can mean the difference between gaining seven Bs at GCSE compared with seven Cs. What children are taught during Reception – the curriculum – is therefore hugely important. Such rewards are by no means guaranteed (Page 4 para 5).
7. Do not generalise unless there are sufficient grounds
Claims to generalisable findings need caveats in qualitative research unless there are sufficient grounds to make those claims. Data collected from, Grade 1 and 2 schools compared to data collected in the same way from grade 3 and 4 schools could lead to generalisable recommendations. As previously described, the review did not do that; rather, it compared data from visits to schools with data from inspection reports, The dataset and research approach does not warrant generalisable claims.
8. Do focus on the salient points generated by the data
The title of the report is Bold beginnings: The Reception curriculum in a sample of
good and outstanding primary schools. The report, however, concerns itself with a number of issues including deprivation, the expertise of NQTs and offers a diverse range of advice from how to hold a pencil to the teaching of synthetic phonics.
It seems to me that what it does not do, with any clarity, is provide fresh insight into leaders’ curriculum intentions, The intention of the review is lost amongst a number of seemingly unrelated points with little evidence to suggest that they were generated by the data.
It is argued that a post expertise era has emerged for inspection regimes (Freeman & Sturdy, 2014). OFSTED simply cannot sustain the view that HMIs have sufficient expertise to offer an objective view of a primary school, secondary school, FE College or other institutions included in its inspection portfolio. The illusion of expertise disappeared partially as a result of fierce criticism from teachers on social media and partially because policymakers seek, notionally at least, evidence-informed policies.
OFSTED does have a role to play; its inspectors have a unique insight into how different providers are performing relative to one another. It has an unparalleled dataset and the opportunity to engage with policymakers and researchers throughout the world.
The purpose of writing this blog is not to pour scorn on the report or to offer advice to Professor Mujis (who does not need it) but to make the point that without a rigorous approach to data the report appears ideological as opposed to evidence-informed, which is not a sustainable position for a modern inspectorate. The appointment of a new head of research, therefore, gives grounds for optimism.
Freeman, R., & Sturdy, S. (Eds.). (2015). Knowledge in policy: Embodied, inscribed, enacted. Policy Press.
Thanks to Christian Bockhove, our twitter exchange provided some of the ideas for the BLOG.