Contracting out schools at scale (13 Oct 2020)

We're very excited to have you here, and thank you so much to Dr. Lee Crawfurd for joining us. Before I go on to introduce our guest today, I just want to go over some housekeeping rules. So we'll be starting in just a minute, and Dr. Lee will speak for about 40 minutes. During this time, if you have any questions, you can use the question and answer function to post your questions. We will also have about 15 minutes dedicated to question and answer. Thank you. Answers at the end of the talk, and during this time you can use the raise hand function. So Dr. Lee will be talking about a very important and a very interesting topic that we all deeply care about. He will be talking about education and schooling, and he is a senior research associate with CGD's global education team. Previously, he's worked as an advisor with governments in different developing countries, including Rwanda, South Sudan. And he's also worked with the government of the UK. He has also served as a consultant for various international organizations including the World Bank, African Development Bank, as well as the Asian Development Bank. He has a PhD from University of Sussex, and he has also studied at the University of Oxford and SOAS. So I will now hand over to Dr. Lee to sort of continue the session. Thank you so much for joining everyone. I will now stop my share, and then Dr. Lee, you can go ahead and share your screen.

Wonderful, thanks very much. It's a pleasure to be here virtually. I really appreciate the opportunity to talk to you all. So yeah, this is joint work with Abdullah Alam from ISAPS, the Institute for Social and Policy Sciences in Islamabad. So it was part of my PhD at the University of Sussex. And yeah, we're hoping to submit to a journal very soon, so any feedback would be warmly appreciated. And we're really interested in as well engaging with my team at the Centre for Global Development. We're really interested in engaging with the government. Our president from CGD met with the former Education Secretary before the new one came in, but we're keen that our work is useful to policymakers as well as being academically useful. So a bit of background to start. This is the percentage of students attending private schools in different parts of the world. And there's been this— it's been relatively steady in high-income countries, but there's been this quite rapid growth across low- and middle-income countries, from around 10% of students doubling to 20% of students over the last 20 years. And in Pakistan, you can see it's been even more rapid and over a third of students in primary school now attending private schools. And, you know, in some urban areas it can be even more, half of students. Similarly, in India, there's up to half of students in Delhi and some major cities attending private schools. So it's something that is an important phenomenon when we're thinking about education. And it's something where we've just done a survey recently of bureaucrats, of officials from ministries of education working on education, about their engagement with the global education system and the global education architecture. And by and large, officials were content with the advice they were getting from international organizations about most areas of policy, with the distinct exception of private schooling, where they just weren't getting the advice they needed. So I feel like it's an area where the that's increasingly important and an area where, because it's so politically contentious or can be, and ideological, countries aren't really getting the advice they need from international organizations about how they should be engaging with this private sector. And so one way of engaging obviously is through public-private partnerships, through which in education we tend to think of as public financing and private management of schools, which can be a little different to how we think of PPPs in other sectors where you might be thinking about private financing for public goods. In education, we tend to be calling about— talking about public financing for schools that are either private schools or they are public schools that are then managed by a private organization. And this shows— this slide shows the growth as well in PPPs in Pakistan, which I think has been a real leader in this space. And, you know, was it one of the panelists here who was just mentioning one of the first schools was contracted out in the late '90s in Pakistan? So there's this long history of engagement with the private sector. in Pakistani schools. And these lines show, you know, it's now over approaching 2 million students enrolled in the Foundation Assisted Schools program under PEF, which is students who are attending private schools and don't have to pay fees because the cost is supported by the government. So large numbers of children, and then thousands of schools across these different schemes. So FAS is the Foundation Assisted Scheme, and NSP is the New Schools Programme, so where private schools are subsidized to open an entirely new school. EVS is the Education Voucher Scheme, which is similar but slightly different to the Foundation Assisted Scheme, in which the voucher follows the pupil so they can then choose which school they want to attend. And then the newest of the programs is the PSSP, the Public School Support Program, and that's going to be the focus of this paper. And it's a slightly different program where you're taking existing government schools and you're trying to turn them around by handing over management to a private operator, either an NGO or an individual, often a teacher or a headteacher. who thinks they can manage their own school. And there's a really rapid increase in the, the PSSB program, growing to over 4,000 schools in a single year from 2016 to 2017. So it really, you know, came on the scene very quickly and at quite large scale. So what we do in this paper, we evaluate this scheme, the Punjab Public School Support Program, which, as far as I can tell, is the largest ever transfer of management of public schools to private operators in the world. We use a difference-in-difference approach, so we're comparing early converting schools versus later converting schools. And to give a quick preview of the findings, we see this large increase in enrollment in these schools that I think can be reasonably attributed to being fresh enrollment, but some reduction in quality and learning. And we'll get into that in more detail. So this is the genesis of this paper. The idea came from some colleagues of mine studying a similar program in Liberia, which was very high profile. It made the front page of the Financial Times magazine, was on the front page of The Economist as well, I think. And it was this controversial experiment, particularly controversial because there were large American NGOs and companies involved as some of the outsourcing organizations. And the initial pilot ended up being 93 schools that were given from you know, public schools that were given to private organizations to manage. And in the letters to the editor of the FT, there was this letter from one of the trustees of TCF saying, oh, this is very interesting, at TCF we've just adopted 271 failed government schools, which is, you know, 3 times the size of this very high-profile experiment in Liberia, and this is just one operator. So my eyes were really like, wow, this is amazing scale. So we need to go and see what's going on over here. And so, you know, to put this into kind of broader perspective, is the Liberia program, which had 93 schools. There's a few small programs in Colombia that has a handful of schools. New Zealand have a few schools, South Africa. I think the model is similar to the charter schools in America and academies in the UK, which have grown over 15, 20 years to become quite large. But it's been a slow process over 15 or 20 years. So, you know, my kids go to the local school across the road that's now an academy, and most schools, the majority of schools in the UK are now academy schools, but it's been a slow process, and this is really a big bang, short, sharp shock over one year with the PSSP. And there's been similar programs in the past in Pakistan that have been smaller scale and more slowly rolled out, but I think the PSSP really stands out for speed and scale. And so—

Dr. Lee, sorry to interrupt. There's one question with regards to— since we're talking about trends and the background. So Wayne is asking, what do you think happened in the late '90s to boost private school percentage, private school enrollment, so suddenly from 17% to 30%? Do you think there was a particular government policy that was promoting this or anything else that you are aware of?

It's a good question. I don't think there's a perfect explanation. I mean, what I've left off that first graph was the low-income countries, but most of the growth in this initial figure comes from middle-income countries rather than low-income countries. So I think it's, you know, It's this new middle class. It's people who have a little bit more money than they had before and are feeling dissatisfied, I think, with government service provision and are wanting to— who can afford a little bit to try and find something better for their children. And the very poorest countries and the very poorest people in Pakistan and elsewhere will continue to go to the free government school option. But I think that's where the— I think that's more or less why there's been this explosion. It's this gap between people's demands and expectations and the government's response in service delivery. And it's a bit of a puzzle. We have some other research from Delhi in India showing that actually the quality of low-fee private schools that are quite popular is really bad and is certainly no better than government schools. And one reason that they are popular is that they at least purport to teach in English, and parents find that attractive, even if the teachers are not exactly fluent in English and they aren't teaching the children English any better than government schools. But the fact that they advertise themselves as English medium is quite attractive as well. So those are my hunches, but I don't have hard evidence for that. And so, yeah, I think PPPs is one way the governments are thinking about responding to this growth. And it's, yeah, as I said, it's controversial, and there are some people who think that we should just ignore the private sector and focus on improving public schools so people can attend their local public school and not have to worry about that, and others who think it's worth trying to explore whether there are benefits from private schools that the public sector can support those children. And, you know, the government's ultimately responsible for the education of all children, whether they happen to be in a private school, I think, or a public school, and so bears some responsibility there. This big table, which is quite noisy and there's a lot going on here, but it shows a kind of spectrum from the different types of public-private partnerships. So on the left-hand side, the most public partnerships, and on the right-hand side, the most private. So it shows kind of how the PSSB fits into these types of schemes around the world. So you have And I'd say the foundation-assisted scheme in Punjab fits more on the right-hand side. It's existing private schools that are out there that are being subsidized with some regulation, but it's, you know, they're very much private schools versus in the Liberia case and even here in the UK, I think the schools are very much seen as public schools still. They just happen to be managed by a private organization. And there's, you know, a whole different set of design features of these programs about whether they are allowed to charge top-up fees, which they are in some case, whether the teachers remain government teachers or can be freely hired. And there's a whole different set of different configuration of design features that you could vary, and I don't think we have great evidence yet on which of these features really matter. But to focus on the PSSP program that we're looking at today, so these schools are not supposed to be charging any top-up fees, they're not supposed to be selective, the teachers can be hired by the private operator, so the existing government teachers were reassigned to other public schools. And that's the basic setup. And to look a little at some of the academic literature in this space, so reviews from the US show that overall there's been a a bunch of different experiments with charter schools, quite now large-scale, and various experiments with voucher schemes similar to the PEF schemes. And overall, they find that the charter school model seems to be more promising than the voucher model, I think because they manage to exercise a bit more control and contracts, I guess, are more complete. The study which made it to the front page of FT is now in the American Economic Review, Romero et al., looking at the Liberia Partnership Schools, and they did see some improvements in quality, which was really the focus of the scheme and the contracts which they signed, but there was also apparently some gaming of the contract, so Um, one operator, um, you know, they really kind of stuck to the letter of the contract. And when it said that, um, class sizes were supposed to be 40, um, and there happened to be 50, 50 students enrolled in that class in the prior year, um, they basically kicked out some of the students and stuck to their contracted, um, class size, which, um, I don't think, uh, you know, a local public school would be would be doing that. So it kind of highlights some of the risks of writing down a contract explicitly, that contracts are never complete. There's a couple of other studies focused on Pakistan by Felipe Barrera-Osorio and co-authors. So he shows that the subsidy scheme in Sindh looks to be very positive, really. So giving subsidies for students to attend existing private schools in SINs led to really big increases in both enrollment in those areas and in learning. And he also has a paper on the Foundation Assisted Schools program, which showed a large increase in enrollment but didn't look at learning in that paper and wasn't able to say conclusively where that increase in enrollment in the private schools was coming from. So it may have been coming from students who were previously out of school, or it may have been coming from other nearby schools. And that matters quite a lot because, you know, from a policymaker perspective, we care about reducing the number of children who are out of school and not just shifting students between schools. And then finally, there's this seminal paper by Hart, Schleifer, and Vishny, who provide a theoretical framework for thinking about the proper scope of government and when it's appropriate to contract out services to the private sector and when it isn't. And, you know, according to their theory, which I then apply to the case of prisons in America, a private provider will typically have a stronger incentive to engage in quality improvement and also in cost reduction. And that cost reduction can be both a positive and a negative. So, you know, you do want to provide services as efficiently as possible. Where you can. But the trouble is when you're engaging with a private provider, as I said a moment ago, contracts are always incomplete, and a private operator will have a stronger incentive to economize on costs that are not included in the contract, and that might be quality in the case of some services. And I think that certainly has been shown to be an issue in Liberia and is clearly an issue in the economics of education more broadly, that quality is really hard to observe. It's really hard to observe and measure the quality of teaching and the learning that's occurring. So, you know, we know that standardized tests don't capture everything that happens in a classroom. And there's studies showing that various effects on non-cognitive outcomes of children can be really important for their later life well-being and success. And so there's lots that isn't captured by test scores. So that's something that has been tried quite extensively in Punjab and Pakistan as well is— trying to kind of manage contracts and manage the performance of both public and private schools, um, with test scores. Uh, and, you know, in theory there are maybe problems with that, um, and I'll, I'll show some, some data on that here. Um, so this is a map showing, um, the distribution of the PSSP schools across the province, labeling the— what I'll call the treatment schools in the transparent circles and the— what I'll call the comparison or control schools in dark circles. And these are— the program was phased over 3 or 4 phases to begin with, and I think more were planned. I'm not sure there have been any new phases, but essentially these schools were all selected according to the same or similar criteria. And as I understand, it was reasonably arbitrary which schools were included in the very first phase versus the second or third phase. And so that allows us this quasi-experimental setting to be able to compare these treatments and comparison schools. And so this map really just shows you the— shows you that there was no particular clustering of these schools across the province. It's not like it started in one city and then went to another city. These schools were really scattered in a semi-random way across the province, which is good from a research standpoint. And so these are those criteria I just mentioned with which schools were selected into one of these phases. We don't know precisely which school was selected according to which criteria, but it was— if you met any one of these criteria, you could be included for selection. And it includes being multi-grade and overcrowded, so having needing more classrooms or teachers and having a large number of students, being underutilized, so having very few students, having poor learning outcomes. And so here's where there was a bit of a difference. So in the initial phase, it was schools with a 0% passing rate on the grade 5 exams, the Punjab Exams Commission, PEC exams. And in phase 2 and phase 3, it was under 25%. Um, so these schools aren't identical, but, you know, they have to be just similar enough. Um, and some that were basically non-operational at the time. Um, and so there was a total of, uh, 10,000 schools that were— that meet one of these criteria and were eligible to be selected. Um, and 3,500 were selected for this, for one of the initial, uh, 3 phases. Um, so this, this table shows the timeline, um, uh, which hopefully helps to make a bit clearer, um, what we're going to be doing in the analysis. Um, so the program Phase 1 started in at the start of the school year in April in 2016. Phase 2 then started in the middle of the school year, so we, we've kind of not focused on Phase 2 for that reason. And Phase 3 started in April 2017. Oh, sorry, I'm just— I hadn't been following the questions in the Q&A. On the SIND subsidy program, um, uh, I think it's similar to the Foundation Assisted Schools, um, scheme, uh, under PEF in, um, uh, in Punjab. Um, but I think, yeah, if you check the details of that paper, it will, um, explain in detail. But there's a similar kind of subsidizing existing. Um, private schools, and then maybe I'll come back to some of these other questions as we go on.

That sounds like a good idea.

Yeah, tricky to read and try and keep on track. So we're comparing Phase 1 schools and Phase 3 schools that started one year later. And I've brought together various sources of administrative data to try and do this. So there's EMIS, which is an annual school census, and so we have this for, you know, going back to 2015 before and after treatment of all of these schools. And so that's going to be the main source on enrollment. We also have patches of data that I use essentially to kind of corroborates the numbers reported by schools. So, you know, administrative data has some strengths and weaknesses compared to survey data that a lot of researchers prefer to use. So we have this data going back before the policy, and then it covers every school in the province, and it doesn't cost us anything. So, you know, we typically spend hundreds of thousands of dollars on— to go and do surveys. But this data is kind of self-reported to the government, and the school has certain incentives to, to report in a certain way, perhaps. And so, you know, there are some concerns around it. So we, we also use the monitoring, the MEA data, which is observed directly by an independent monitor. as a kind of check on the enrolment numbers that are self-reported by schools. And then similarly in the ACER data, there's unfortunately quite small overlap between the schools that they sample in that survey and the PSSP programme schools, but there is some, a little bit of data there for a small number of schools that allows us to check how reliable those enrolment numbers seem to be. And then for the— on the learning side, we use the same from the independent monitors, the Literacy and Numeracy Drive data, which unfortunately doesn't— we don't have before Phase 1, but we do have before and after the Phase 3 schools were converted. There are a couple of other, again, pieces of data. The quality assurance tests, QAT, are tests that are high stakes for the school, low stakes for the student, and they're used— ultimately, schools have to be getting a 50% pass rate on this assessment or they can lose their funding. So that's really important for schools. And again, we don't have that for before and after for all of this, but we do have a couple of rounds. And then the Punjab Exams Commission Grade 5 exams, which we have a couple of rounds of, well spaced, but ultimately we want to see parallel trends is the key assumption for a difference-in-difference analysis. which means that these 2 groups, they don't have to be identical before the reform, but they have to be on a parallel path, and that assumption doesn't seem to hold for that, but I'll show you that more clearly in a second.

Dr. Lee, before we move on, so there was one question about selection criteria by Kate, and she was asking about non-functional schools, since only one-third of these schools were selected of the total 10,000 schools that you talked about. So she's asking, how should we think about these other closed schools which were probably not incorporated into the analysis? And also, is there any reason why such a large percentage of nonfunctional schools was not selected in the sample? And presumably the closed schools don't appear in the sample at follow-up, and do the treatment for such schools is more like opening a school rather than privatizing a school. So is that— could that be one of the reasons why that's the case?

Yeah, that's a good question. I should probably dig into that. I'm not sure we have, but I could definitely do that analysis excluding those schools. There's— yeah, I think they are a kind of— it's 500 of the 3,500, so it's a minority. But that's a good point, that it's quite a different reform potentially. Thanks. Okay, so let me press on. So this is— so the key assumption for this analysis is that the the trends are parallel, so the changes are the same. So this is kind of a balance test, which— and we don't need these schools to be identical, but just to show you that they are— there are some statistically significant differences between these groups, but I think they're quite small differences. So both have between 70 and 80 Pupils on average attending these schools are really quite small, with an average of 2 teachers, about a 65% pass rate on average, despite that difference in the eligibility criteria on the pass rates, and similar average marks. And yeah, so these are kind of 2 or 3 classroom schools. So essentially, yeah, you can think of these as being quite small, quite small schools. And here are, here are the main results on enrollment. So on the left, this is showing average enrollment for the phase 1 treatment schools in black. and the phase 3 schools in the dashed line. And you see this really clear and stark jump in enrollment in the treatment schools, roughly a doubling in enrollment. And the comparison schools then continue on more or less the same path. And then when they are treated a year later, they see the same sharp increase in enrollment. I mean, this isn't totally surprising because these, um, the private operators are being paid on a, on a per-student basis, so they have a really strong incentive to go out and increase enrollment. And you hear, you know, anecdotally people were going around and kind of rounding people up and really marketing and trying to advertise these schools quite heavily. So it's— yeah, it seems plausible to me that this is real. On the right-hand side, we see the estimated effect, or the difference between the 2 groups, growth by month. And yeah, it's the same result as on the left. You see this big increase of 60 students in the treated schools, then drops away in the second year when the comparison schools are also treated and they start to catch up. The big concern here is about where this enrolment is coming from. Is it coming from students who were out of school, or is it coming from the neighboring school, in which case, is it really that much of a win? So the first thing I do is break down the increase in enrollment by grade. And so this table shows the regression coefficients on the difference-in-difference model. And so it's broken down by grade. And so you can see that most of the increase is in Catchy. So an increase in 40 students in Kachi, and an increase in 18.5 students for the whole rest of grades 1 to 5, and that also being larger in grade 1 and grade 2 than grade 5, a much smaller increase. So that seems, I think, consistent with the idea these schools are enrolling children who weren't previously enrolled at all rather than just poaching students from other schools, which is somewhat reassuring. Um, next I look at the schools that neighbor PSSP schools. So this is still focusing on government schools, and what happens to a school that's neighboring a Phase 1 school, its closest neighbor. So we use GIS data to identify the, the closest school to each of these schools, and here we see these nice parallel trends with nothing really shifting when the schools are converted. So that's reassuring. If we thought that these schools were poaching lots of students from the neighboring government school, you might see a big drop in enrollment here, and we don't seem to see that at all. But this is only focusing on public schools. So there are also obviously lots of private schools around, as we've discussed, and these newly— these government schools that have been newly taken over by private management that are free might be quite attractive to people who are paying to send their kids to a private school. Unfortunately, we don't have the same panel data on private schools tracking them over time to be able to do the same analysis as this. What we do have access to is a census of private schools at a single point in time, so we aren't able to look at how enrollment changes over time in private schools. But we can— the one thing we can do is look at how enrollment changes in PSSB schools depending on how many private schools there are near to you. So we break that down here by the number of neighboring private schools within— I can't remember the exact distance we used here, but within a narrow geographical area of 1, 2, 3, 4, or 5 or more private schools near you. And you might expect, if this was mostly poaching students from private schools, you might expect a bigger increase in PSSB school enrolment where there are more private schools than when there are fewer. And you do indeed see a difference here, so the biggest increases are in our— for schools that have 4 or more private schools nearby, but you still see this increase for those that only have 1 or 2 private schools nearby. So I think, you know, this is consistent with the idea that there is some poaching of schools— of students from private schools, but not entirely. So there's still some, I think, real increase in enrollment going on. I skipped over one table here, which was looking at the distance to— this is the same analysis focusing on public schools, but looking at how close that public school is. So this is first looking at enrolment in all schools that neighbour a PSSP school, and there's actually, if anything, a slight increase in enrolment. And then we break it down, so if you're within 3 kilometres of a PSSP school, or within 2 kilometres, or within 1 kilometre. And these results vary a little and are all positive, which we don't have a great explanation for. I think why you would see enrollment to go up in the public school that's neighboring a newly converted PSSB school. It's quite a small increase. It's nothing like the big increase we see in the PSSB school, but that is a bit of a puzzle. Perhaps there's some positive spillover onto some confusion about which school had been— had a change in management. We don't— yeah, suggestions for that would be great.

So, Dr. Lee, we— so we're out of time for the presentation, but we can perhaps take another 5 minutes or so for you to wrap up, and then maybe we can move on to the questions.

Great, okay, I'll speed through this if that's right, um, last few slides. Um, so this is, um, looking at test scores, um, from the monthly monitoring test data. So this is a small sample of 5 pupils that are tested in each classroom, a different 5 each time, um, on this random test. And here you see this really sharp drop, um, in the— this time it's Phase 3 schools that are newly converted, you see this big drop in the average percentage of questions that are correct. It's about a 0.1 standard deviation decrease. And so, I don't know, we struggled a bit with this about whether it's the composition of pupils has changed because enrolment has gone up. So it may be that you're enrolling some weaker students that may be driving part of this. But in grade 2 or grade 3, it's not such a big increase in enrollment, so that doesn't explain this entire effect. Most— because, you know, most of the increase is in Catchy and in grade 1, and this is testing students in grade 2 or 3. So I think this is real. I think this is, you know, it's quite a disruptive thing to this school. You have people newly running schools who haven't been doing it for a long time. It seems they're just— yeah, they've got the big increase in enrollment that they've been contracted for, but the learning outcomes are dropping. You see this across different subjects, math, they do English. Then moving to the QAT test, so this is interesting. The baseline test rate for schools entering the program is normally distributed on the left, as you might expect. The endline pass rate is really not normal at all. This is really kind of all over the place. So there's 100 schools that have got 100% pass rate. And if you look at 40 and 50, which are these important thresholds, you see there are, you know, 80 schools that have a 50% pass rate and just 25 schools that have a 45% pass rate. So there's clearly some manipulation going on there around that threshold, which is, you know, natural and understandable that these schools may lose out on their funding if they get 49, but if they get 51, then they're fine. But the overall shape of this distribution just looks kind of very different to the one on the left. So there's clearly where tests matter for the school, there's some risk of manipulation there or gaming. This is looking at the, um, the grade 5 exam, which is nice because we have the full time panel, but you can see the trends are really not parallel to begin with, so we don't have the same nice picture here. So kind of less confident in these results because of that, but they are consistent that there does seem to have been a negative drop after conversion. And then just looking at mechanisms for what else was happening, so we also have data on teachers. So these schools are recruiting an average of 3 additional teachers, which means that despite the big increase in enrollment, the pupil-teacher ratio has actually gone down by 8 or 9 pupils. And then we can break this down to the different providers, and overall the NGOs have seen a smaller increase in enrollment than the individual providers. but perhaps a bigger, a bigger drop in test scores compared to the individuals. And so yeah, to conclude, this apparent large increase in enrollment, we can't rule out all spillovers, but you know, the data we have does suggest that at least some of this enrollment is new, which is great because there are still lots of out-of-school children in Punjab. But we see this decline in test scores because you can't contract for everything when you're writing a contract. And then finally, I haven't talked about this very much, but the potential for cost efficiency gains from cheaper teachers. So the per-pupil reimbursement is about a third of government per-pupil spending in public schools, so it's much less per pupil. But you haven't replaced the teachers who are being paid. They've been reallocated to other schools. So there hasn't been any immediate actual fiscal cost saving. But there's— it depends on what happens to the rest of the system. If you needed more teachers in any case, then this is a cost-effective or a lower-cost way of recruiting newer teachers into the system. But there are all sorts of questions about, you know, how sustainable that is and how desirable it is to be, uh, to have teachers on very different pay levels, um, uh, doing ostensibly the same job within the public, public system. And yeah, people have very strong views about that one as well, um, I won't get into. Um, but yeah, I'll stop there. Thanks very much. Um, and here's my email address if anyone wants to get in touch directly.

Thanks. Thank you so much, Dr. Lee. That was a very interesting set of results. I was fascinated by the way you presented, and as well as the stark difference in the impact on enrollment versus the impact on test scores. There are a couple of follow-up questions that Kate had, which I think are interesting comments, and would be happy to hear what your thoughts are. So she's asking that you've shown impacts in the local market as far as enrollment is concerned, but do you know more about the effects on other schools, particularly with regards to the kind of work that Andrabi et al. have shown in their studies of the competition effect that might happen in close-by areas? So I have— do you have the data, or do you think it would be possible to see how perhaps scores might have changed in nearby schools or fee might have changed in nearby schools. And also on gaming, could you look at the number of students who are enrolled versus the ones who appear for the test? So these are 2 comments.

Yeah, so on competition, yeah, it's a real shame we don't have the same panel data on private schools. I mean, one option could be that we haven't really explored is the Anjabi et al. panel survey, but I'm not sure that you would find enough schools that would match the PSSB sample there. So yeah, we, you know, we— I showed results on the closest neighboring public school showing a slight increase, so maybe that's a kind of competition effect. But yeah, I think less confident in those results than the main results. I think it's a hard thing to show. And then on the test scores, so it's a— as I understand, it's a randomly selected set of pupils from grade 2 and 3 in the QAT— sorry, not the QAT, the MEA Literacy and Numeracy Drive tests. So that shouldn't be affected by who happens to turn up. It's just a random— a day that they turn up and test a random set of kids, so it shouldn't be affected by that. You know, the PEC exams clearly will be affected by who turns up, and we do see an increase, a small increase, in the number of candidates enrolling in the PEC exams. It's, you know, it's 1 or 2 pupils extra enrolling, because in a school of 70 children, you don't have many in grade 5 to begin with. But yeah, that's an issue there.

Dr. Mozum has a question. So Dr. Mozum, I'll unmute you and you can perhaps go ahead and ask your question.

Thank you. Hello, can you hear me?

Yes, we can hear you.

Thank you, Habna, and thank you, Lee. I thought that was a very interesting And, and a very arduous investigation. May I say, let's just separate out 2 aspects of the problem, one enrollment from the test scores. The test score results are a pretty, pretty strong indictment of this whole experiment, but let's put a pin on it and put it aside. I am actually very concerned about the enrollment issue. And I was approaching it from the opposite point of view that you are, which is to say, I would want to see poaching because— let me give you the argument for it. Let's assume that you have a normal public school and you have different income groups around, and only the slightly better off working-class kids can go to the school, the other ones have to go to work, right? And then you say we're going to set up a charter school to improve quality, blah blah blah. And if we're going to do that, then you would— and if there is some extent or perfect payment of fees, you would expect the slightly better-off working kids to go leave the public school and go to the charter school. So I would expect to see poaching, and I would expect to see competition between the public school and the private school. Now, you're not finding evidence of that. You're finding slight evidence of that. That to me is worrying, and it sounds to me counterintuitive. And perhaps the better way to look at it might be, in this case, an RCT where if we use Euler's theorem, you say that let us take a catchment area of a public school and a private school, and let's sum up all the kids in that area. And you say 30% will not go to school because they will be working, and 70% went to— or maybe out of the 70%, 40% went to the public school, and then 30% went to the charter school. Then you have much more conclusive proof of who's doing what. But my comment there on your puzzle. Thank you.

Thanks very much. Yeah, it's a real weakness that we don't have this full data on all of the pupils, so we just don't know how many are attending private schools. And yeah, I think from a policy perspective, we wouldn't want to say that this program has increased enrollment by 50%, um, if that enrollment was just shifting from different schools. Um, so I think that's, that's quite an important distinction. And, you know, I think it's positive that the takeaway is that this does seem to have increased, um, overall enrollment. Um, and yeah, I think there probably is some, um, coaching going on. I mean, maybe that's why there seems to have been this increase, this small increase in enrollment in neighboring public schools. Maybe they have felt the competition and had to go out and drum up some business themselves. I don't know.

Thank you, Dr. Lee and Dr. Mazum, for those comments and discussions. So there's another question by Maryam Haroon. So we've been talking about enrollment and how perhaps that might have changed as a result of this program. Do you have any evidence on dropout rates? Do you think that one possibility for the big impact on tests— test scores that we're noticing is that students dropped out from schools and therefore hence may have led to a change in composition, and that might have contributed towards a drop in test scores?

Yeah, it's certainly possible that you could have. So we know that overall aggregate enrolment has increased, and we don't know whether it's the same children or, you know, there could have been a lot of turnover. You might have been uncertain about these or unhappy with this new management and moved your children to a different school. And so, yeah, that disruption, I think, could have been a factor. I mean, it'd be great to, yeah, to try and follow this over a longer period. Unfortunately, the kind of natural experiment that we've used here just lasted for a year before the other schools were converted. There might be another way of— it's been suggested we could try and find matching schools that were similar on other observable characteristics, but were never converted at all to allow us to look at longer-term effects on learning where maybe some of that change and disruption has kind of worked itself out. In any case, I think, you know, 0.1 standard deviation effect size, like, what does that mean? And that's the kind of— that's an average-sized effect size I think we see from literature, but it can often translate into quite small changes in the actual skills of pupils, what they can actually do. And I think there are big question marks about how we should value learning gains that are ultimately quite marginal, or learning losses that are quite marginal, compared to access to schooling that we know has labor market impacts. Like, it changes people's life just attending school, even if it's quite a low-quality school in which you don't learn a huge amount. It can still have lots of benefits. So, yeah.

Thank you, Dr. Lee. There's some comments in the chat window with regards to the quality of the data that you are using. So there are perhaps concerns with regards to the MEA data as well as with regards to the QAT data. So are you doing anything to kind of address those, or do you have any ways of kind of using robustness checks to establish whether the results that you're getting are actually robust too?

I mean, I think, you know, we try to triangulate a little across these these different data sources. Um, um, but yeah, I'm not sure we have, um, otherwise a great answer to that. Um, I think, yeah, for sure there are issues with data quality, um, uh, with this data. Um, yeah. Okay.

Thank you. I think our panelist had a comment or a question, but she dropped out. So let's just see if she's joined back in. Yeah, there's Asya. Can you hear us?

Are you talking to me?

Yes, Asiya, can you have— can you hear me?

Yes, I can hear you. Thank you.

I believe, yeah, I believe you had a comment, so please feel free to go ahead.

Yes, I, I just wanted to, um, punch in a few, um, free basic things about, uh, these schools that the Punjab Education Foundation has in Punjab. And some of them are, well, one is the EVS, which is the voucher scheme. And then we have the foundation-assisted schools, which was basically— you need to understand a little bit of the history behind this, that these were basically those dying schools which were not picking up enrollment, nor were they rightly managed in the sense that the students were— the parents weren't paying fees, the infrastructure was bad, everything. So the government decided to take up those schools which were below the fee of ₹4,000, and it was a complete— Uh, and it was, it was, it's basically a flagship program. And then this was basically, this was like, it basically hit off in the south of Punjab where for example in Bahawalnagar and Bahawalpur we had 8 to 9 schools on maybe one road. So it really hit off very well. It started with in 2013 I think it had a 16 lakh enrollment of the entire PEF. And which I think increased to 25 lakh or something. So the numbers were quite large considering that. And then we had this school, the new school program, which was picked up by the entrepreneurs and in partnership with the schools. And we have, I think, 2,203 schools with again a large number of students. And even in, in the foundation-assisted schools, we have 1.8 million students right now currently enrolled. And but the issue over here is, in all this scenario, is I have, I have a few questions. Now I'm— I am not speaking as an advisor to the Minister of School Education, I'm, I'm barely speaking as an educationist, looking at these schools, trying to evaluate and trying to understand why are— what is the basic mechanism that we are deriving out for, like Lee said, what about the academic outcomes? How do we measure those? The LND tests, well, they are conducted by the MEAs, and the MEAs are retired Army JCOs kind of people. So I'm not sure if most of you understand JCOs, but so if they are going to be conducting LND tests on a laptop, And marking the attendance. Marking attendance is okay, fine. It probably doesn't need a lot of academic issues to, you know, test. And then making the students and the communities— so basically our measurements are evaluated, the students' outcomes, the academic outcomes are evaluated based on these academic outcomes of how much learning is happening here. And how much are these teachers being looked after? Now, from my understanding is that these foundation-assisted schools and most of the PEF schools that the teachers that I have at least spoken to or interacted with have a— get a salary of below ₹5,000, some of them. So how do you possibly expect? I work pro bono, so I— it's, you know, it's a different thing. But a teacher who's living for— who's working for a livelihood is getting paid less than, or maybe 5,000 or a little more. The maximum, I think they have a cap limit, if I'm not wrong. So how can you— what kind of learning is this teacher giving Or what kind of qualification does this teacher have? Forget that also. So my question is that how do you measure the the learning outcomes correctly of these students? Like I I saw in the lease one of his presentation was very remarkable really done. Some of it I've seen, Mr Abdullah. I think they probably. they did a third-party evaluation also on these same PEF schools. So I went through that also, but I'm still trying to figure out where are we missing the link, how can we improve these schools, and because right now currently we have 7,000 PEF schools and somewhere around 4,200 PSSP schools currently enrolled in Punjab. So in order to take these up, and then also we, because like the minister's office keeps getting a lot of these days, we have a lot of issues with the PEF partners and they keep sending us pictures, both sides, uh, the, the PEF schools also, and, uh, and the people who are against the PEF schools. And, uh, they, to me, some of them are doing really well and some are doing— well, I don't want to show you the pictures, but I do have some of them. And, uh, so everything which is done, even if it's, if you call it a teacher training, if it's a CPD program, If it's anything else regarding learning of a child, it has to be measured, whether it's an assessment, whether it is learning of the imparting knowledge of the student— of the, sorry, the teacher in the classroom, what kind of pedagogical skills she is giving, you know. So all these things they need some, they need some sort of an evaluation. And I frankly, um, because I work very closely with these schools and I've been to a lot of these schools also, so if you see the ground reality is something extremely different from what we see in our, uh, in our documents. So, um, I don't know. So I, I don't know where what part have I answered or not, but I just wanted to share this small data with you guys. So, uh, it, you know, it brings some, um, reality to it. Thank you.

Thank you.

And this was not an official statement, by the way. Even if it's recorded, I won't let the minister see this part.

Um, thank you so much for your insights. from the field and from the actual sort of context that you're working in, Asiya. Relatedly, Dr. Fariha was also asking, in trying to understand this puzzle that Asiya was also talking about of why these test scores are so low, so she was asking whether the subsidy initiative in Sindh that I think at some point, Dr. Lee, you mentioned in your presentation that that led to increased enrollments and an improvement in quality So do you have any ideas as to why we see these differences between these, given the overall context is pretty similar? It's the same country. So could there be any sort of potential explanations as to why we're observing improvements in quality in Sindh but not in Punjab? So any thoughts on that, as well as on Asiya's comments?

Yeah, I mean, firstly, like, kind of speculating here, but as you know, some of the research by Andrabi and his team shows that competition can be important. And I think in this case with PSSP, the school operators were given a very clear kind of mandate and instruction, which was you're going to be paid for every student, and that's subject to them meeting a certain minimum average pass rate on this exam. So the focus was very much— and these are really small schools to begin with, failing schools— and so the focus in this initial year was clearly going to be on getting enrollment up, and they seem to have achieved that. But, you know, they were— they kind of did what they were asked in a sense. And I think it's, it's a really hard— going back to the beginning of the theory, it's you can't contract for everything, and there's always going to be incentives for private operator to try and game that. Why have you seen more positive results than SIND? Maybe it's because these schools are in more dynamic markets. If there's, you know, if there's 5 nearby private schools and you're competing against them, you kind of need to compete on quality as well as as everything else. So that's potentially one answer. And yeah, it'd be interesting to see a study, a similar study on the foundation-assisted schools in Punjab. But I think, yeah, going to Asiya's comments, I think she's right that it's a really hard challenge, I think, measuring learning and quality, and there's massive variation between students and between schools within all sectors. And when you look at the differences between private schools and public schools, it can often be shifting the distribution from here to slightly further across, and it's really hard to pick that up with your bare eyes and barely statistically. And I think, yeah, we have to think about what all that adds up to. But yeah, I hope with this study, despite all the limitations of the data quality, we do see this quite distinct change in learning.

Thank you so much, Dr. Lee. I think we're over time, so thank you everyone who stayed back and for a very sort of insightful and engaging discussion. Thank you for joining us, Dr. Lee, and for everybody else, please don't forget to tune in 2 weeks later for Tim Doberman's talk on ending the circular debt and trying to reform Pakistan's energy sector. So looking forward to seeing you guys again in 2 weeks.

Thank you.