Outcomes and JFF: The Story Behind an Acquisition

Welcome to the Future of Education. I'm Michael Horn. Particularly excited about today's show because get to talk about a passion project of mine, quality outcomes and assuring that we actually know the real data for how learners are doing in their post secondary programs with an organization that I helped co found and serve on the board of that was recently acquired by Jobs for the Future, JFF and to talk through Equos and JFF and this new partnership. Tremendously excited to welcome to the show today Steve Yudzinski, who's the Senior Innovation Officer at JFF and is currently helping manage the transition and integration of Equos into JFF and really working to expand Equos partnerships and develop sustainable models for the organization that will really have far reaching impact on learning and workforce systems. And we'll talk more about what that means and how Equos operates in a moment. And then also excited to welcome Tamisha Bridges Mansfield, who's the Vice President for Workforce Innovation at JFF and most recently joining me as a member of the board at Equals at equos@jff. She oversees the workforce development and future of work initiatives, particularly around innovation and systems transformation. So I think this is going to be basically just the two right people to have at JFF to have this conversation with that I'll bring them to the metaphorical or virtual stage, whatever we're calling it these days in Covid. But Steve, Tamisha, it's good to see you both. Tamisha, let me start with you because I gave a very high level of what Equos is based on outcomes. But let's start with the why and then we can get into the brass tacks a little bit more like why is JFF interested in Equos? What brought you to this conversation in the first place?

No, I think that's a really good starting point and it's great to meet you, Michael. I'm looking forward to working with you. Really. I think the why is the question that folks are noodling in their heads right now. It's really about JFF really wanting to put a flag down and really looking to be a more powerful catalyst for change rather than merely just a conduit for change. With this perspective shift, it's really, really forced us to kind of take a look at, to reexamine what our big picture and organizational strategy is and really think about how we can drive innovation and impact in the education and workforce system. So you know, here comes Equos and we took a look at it and really came to see that in the two years that it has really developed a robust framework to really measure program quality and impact through outcomes as real, as well as really helping to build a respective platform for thought leadership. So it allows us to have some credibility as we talk about the importance of quality in the training and workforce ecosystem. And so with equals on board, we really see it as a way for JFF to strengthen our advisory services and practices with states, with higher education networks, with workforce systems, and really, really pursue a research agenda that is really grounded in core quality and outcomes of career pathways with really increased clarity and focus where we're not trying to like name and define things, but we have something that we know works. And so really by combining forces with equals, it will really allow us to really extend our current work, both JFF's current work and equals current work and impact to more sectors and more communities, hopefully across the country.

Makes total sense. Steve, do you want to add anything to that since you've been riding shotgun to all these conversations?

Yeah, I mean, it's a great quick, it's a great overview of what we're up to with equos. The thing to mention is we've been partnered with Equos for some time at JFF through a data initiative. We've also been working to evaluate and create catalogs of quality providers on a regional basis. EQUOS has been the foundation of that vetting process and so forth. So we've really been built a familiarity with the team, with the framework itself, and we're excited to bring it here into the family of JFF so that we can grow its capabilities.

So that starts us grounding, I think, with why JFF cared and got excited about Equos and vice versa, frankly, from the perspective of board and founding member, why we got excited about the partnership with JFF as well. But let's broaden the lens a little bit. Tamisha, I'd love to hear, you know, why should the field pay attention to this? Why is this important more broadly as we think about impact and this transformation that you all at JFF really want to lean in on and shape in an active way?

Yeah, no, I think. I think probably the first place to start with answering that is really just saying that, like we're right now going into what is a third year of a pandemic economy and labor market. And I think as we begin to look forward and really begin to work towards an equitable recovery, the field really has to take a closer look at the outcomes that are being produced by the training and workforce and education system. I think we're at a point where we can't just blindly say any training is good training and we really have to have to take a closer look. And so I think because of that, the learning and workforce ecosystem is really reaching a turning point. And also I would add that this marketplace is really learning large and has become really confusing and challenging for learners, for workers, for opportunity seekers, particularly folks who are from disadvantaged or marginalized community, to really make sense of it and to find the programs that are going to be right for them, that are going to meet their interests and needs, that are going to meet the demands of their local labor markets, to really help them cultivate skills and find high paying jobs. So it's really about being the field, being responsive to what, what people are asking for, what funders are asking for in terms of understanding quality. And so in many ways I think EQUALS is already rising and risen to that huge challenge through the quality assurance framework that really kind of drills down and looks at, well, what are five things that we would say count as quality, what for student learning, what is the learning outcomes, what are the completion outcomes, what does placement mean, how do earnings look? And really for workers, what does satisfaction look like of those training experiences? So you take all of that together and I think it provides a really compelling message. I would also say that states are also getting curious and needing to know this as well as they begin to vet their, their learning offerings at the state level. And so it's really about meeting the needs of both learners and systems and bringing folks together to understand that. And so it's already kind of demonstrating the need that the field has for new and innovative training models. And I also think that program leaders and training leaders realize that they have to rise to the occasion now and are looking for easy mechanisms to be able to measure and understand their outcomes for themselves. Right. To help tell their story of their own impact. And I think they need to do that with access to better data collection, more means of transparency, all of it that points the way for people. So what we really hope is that this acquisition really helps signal like JFF's intention to be more of service in more tangible ways in terms of moving progress forward as it relates to understanding the importance of outcomes for the field.

I feel like I'm like listening to all I had hoped equos would become listening to you. So I'm just like loving this and just to double click on it. I think you talked about the different areas that we've created measurements for around learning, completion, placement actual salary growth relative to investment and expected growth, so that we're not creating incentives to skim or cream, you know, who you serve. But then also that last piece, the learner satisfaction, and then you also talked about how important it is for the institutions themselves to say, hey, this is the value we provide and be able to customize it to what they would claim. I think both of those pieces are important because my observation often from a lot of these conversations I have about quality and value in the post secondary and workforce training spaces is that a lot of times we sort of thrust a definition onto an institution or an expectation of what we think the learner is trying to achieve. But in point of fact, they want something totally different from the experience that we don't have good ways of capturing. And so hopefully that customizability you spoke about will also help the field get more focused on what the learner's goals actually are in all this. But, but I think that's actually a good segue because, Steve, it allows you to maybe talk about if those are the standards, what does Equos actually do on the ground and what does it, what does its work actually look like itself?

Yeah, it's, it's a fabulous question. So Equos work to date has been about partnering with state departments of higher education in New Jersey, Indiana as well Colorado, and really advising partners in those states around best practices, how to aggregate data, how to build data systems. As a matter of fact, there was just, I just left a meeting with the New Jersey Department of Higher Education office that will be releasing a report that was built in partnership with Equos at the end of the month which details those recommendations moving forward. You know, why did JFF bring EQUOS here? We're really interested in furthering the practical utilization of this framework. We're really interested in pairing, as Tamisha was talking about, a research agenda with upgrades to the framework. So, for example, we have our five elements of the framework JFF is looking to build, equity component to that. One of the key headings that we'll be looking at, how do we understand economic advancement and what kinds of recommendations might we make to our partners around additional types of information that are important for learner success. So, for example, access to financing options, access to career connection, how effective that is, access to other supports such as child care or flexible scheduling, and so forth. One of the things that is true with the pandemic coming into year three is that there are permanent changes here that influence how we're working, but also how we're learning. And to meet those changes, a dynamic range of options are being built and launched, new credentials and just the complexity of that alone. So being able to work around the framework itself, evolving how we define what is quality, what are some of those additional sort of more subjective components of offerings, how would people be able to use them to make the right choice for them moving forward, for example? The other thing to mention is that the Equos framework, as I mentioned, was being it's the foundation of a vetting process that we've used to create a provider catalog for Skill up, for example. There are other partners that are beginning to use that data and so really being able to expand that that data set so that we can understand some of those more dynamic nuances as people are evaluating the quality of programs as we progress here.

It's interesting hearing you say that because you talked about the notion of evolving the standards also and keeping them up to date, but then also leveraging them in ways or I guess also allow other providers to leverage them in ways to surface the right data at the right time to someone making a choice, basically. Right. So that they can make a good choice, an informed choice for them. On my one of my podcasts, Future youe, we've had a lot of conversations lately about how a lot of the ranking systems and reports right now on ROI that have been coming out about college aren't terribly useful to an individual learner trying to make sense of where they ought to go. Because it's not comparing, you know, school, two schools that they might be comparing against each other. Right. To each other. It's sort of looking at this in a very national way and not within particular labor markets or particular geographies and so forth. And so I think that segues where I want to go next to you, Tamisha, which is let's dig in a little bit more about how JFF itself might use this quality assurance framework, because you said it will inform a lot of the, you know, the work streams that you all have going on actively as well. And presumably this tighter coupling allows you to do some things that you wouldn't have otherwise been able to do.

Yeah, no, I think that's true. And to your last point about how does equals help people make sense of what's happening in their local labor market? We really see the C equals really fitting seamlessly within our current work that we've been doing to develop a quality provider index, as Steve mentioned, to really evaluate the quality of learning providers, programs and models at a local level. We were able to Use that in the development of skill up and equals and working with equals prior to the acquisition helped us to be able to understand and refine that. So I think it does provide an opportunity, you know, once we get usage at scale, at any local level, to be able to vet and understand the quality of providers in the context of what's happening on the ground in local communities. But then there's a question of like, well, how do you get the information to people? And so I would say the other thing that we're really interested in looking at and wanting to plan, to dig into is really using the basis or the framework as a basis for new digital tools and infrastructure that will really help us to collect, to track, to analyze, which is, you know, great, but also to share. Right. The insights that we're learning from what we're collecting about post secondary quality data. Data is only as good as you make it accessible to people to help people make sense of the world, to help stories and tell stories about what's happening. And so really figuring out what are the right tools to make the, the data and information that can come from the framework and that can be realized from the FR real for people and communities in real time. And then I would say also as it relates to our advisory services, it's also kind of can be seen as like a baseline of looking at a program or looking at a series of programs to say, you know, how are they performing along these metrics and then being able to provide services to help people raise their scores, improve, improve their, their standing as it relates to, to the met are there. So it's a way of thinking about how do we have our advisory services that are informed across a consistent definition of quality. So we're not having to like recreate what quality means and looks like per provider, per their wishes, right. Or poor, what we think is relevant. But really we have a basis to begin and a level of expertise to start from that really helps inform any advice, recommendations or implementation support that we would provide. So it's in many ways it's the data that, that, that we're hoping to kind of undergird that additional service and support that we provide to states, to workforce systems and to communities and understanding what's happening.

When I hear you talk, Tamisha just reminds me how much JFF does across the landscape, right. Just how big the scope in remit is and how many different sort of levers you have at your disposal, which I think is frankly really exciting. One of the things when we built the standards and you mentioned it as well, it's to really provide the data and then let others figure out, okay, what's our threshold for is this good or bad and so forth. Steve, maybe turn to you on this question because I want to go a level deeper on how you might see learners, you know, leveraging this to navigate the post secondary education and training marketplace. And I want to come at it with like a, a specific angle, which is we know historically the most marginalized learners in our, in our groups have been the least likely to consume some of the data that's out there to help them inform spaces. I think we've all offered some hypotheses already about why that is that, the utility of the data and so forth. But how do you see what you all now have in equos as really helping change that dynamic and helping individuals navigate?

Yeah, I think there are a few ways. So the first thing to say is you mentioned that there's a wide range of capabilities that JFF has. So our core competence of convening and thought leadership as well as policy, so being able to advocate for those policies, those systems at the federal and state levels that aggregate information and make it available at scale, that helps people navigate and make those, those tough decisions, making it available to as many people as possible. So one, one piece is really around that convening and policy component. The other thing that we're really interested in is that practical use of some of these standards. So being able to draw the field together around a point of view around what quality looks like today as well as in the future. So being able to account for demand and being able to integrate that into some of, some of the work is just one example of how we might be able to better indicate, you know, this might be a better bet for an individual versus another, another option. The other thing that I think is important to help people sort of navigate is really building our partnerships for those platforms that are using some of the standards. So we mentioned Skill Up Guild is a partner who has used the framework to evaluate some of their partners. Climb Credit is a partner of ours. So a number of platforms can also integrate this. If you can think about sort of syndicating this to their own audiences above and beyond what JFF can do individually. So it's, it's a bit about sort of a movement building, you know, agreeing around the right measures, the right framework, building a sort of policy advocacy. What, what do we think should happen? What do we think can help this scale and then also bringing partners together to advocate for the use, you know, above and beyond JFF programming.

So I actually want to tee off on two things for both of you there. One piece you just mentioned is the ability for these standards to really exist outside of just the immediate JFF use case, if you will.

Right.

And so in my contemplation when we built the standards, we were also thinking, hey, this doesn't just have to apply to say, workforce training providers. It could be your liberal arts college, right, that's trying to understand the value that they provide. Or right now there's a ton of flak being given to arts programs that perhaps charge a fair amount of money and the outcomes are highly unclear. How do you all think about that as jobs for the future, focusing on the workforce and training element of this? But, but sort of the sense that maybe this could be the answer to help a lot of sectors of higher ed and post secondary training and so forth better define the value that they're actually providing and what they're doing and how will JFF support that? Or maybe say, hey, great that you all want to run with this, it's yours. How do you all think about that? Steve?

Yeah, I'll start off. And Tamisha, please add on. One of the things we're thinking about in terms of further developing the framework is it's been used for sort of short term credential programs to higher ed programs. And so it's been used on a wide range of programs and we believe that over time we will want to sort of further specify that based on the particular audience or use case in some regard. So one thing to say on our roadmap is sort of regular releases or versions of the framework that are more appropriate for use and higher ed short term credentials, state systems and so forth. So that's, that's one thing to say, the other thing to say, by the way, I have an art background, kind of proud of it, you know, coming out of here. So I do have this sort of progressive set of experiences that bring me literally to this conversation today. And being able to help those institutions understand the value and the cross transference of the skills that are gained in those, those programs I think can be a really compelling sort of, you know, research component. How do we define that, but also how do we bring the partners together so that we can legitimately understand sort of the longitudinal, you know, things that could take years or even decades to really understand what the trends are. That's a little bit of what's starting to bubble up in our research agenda. How do we do some of the immediate components Informing navigating a complex market. But also how do we understand the outcomes of things that often take years. I mean adult education is another example where it can often take a long time to see the economic advancement realization there. So it's a little bit of how we're thinking about it here.

Tamisha, do you want to add anything to that?

I mean other than being a liberal arts graduate? I'm like there is value in that credential. Exactly. Right?

Here's the history major.

Right. As about, I would say there better be some value in it beyond like student loans. But, but, but yeah, I mean I think it's, I think it is like an evolving thing that we're, that we're wanting to understand. And I think as I think about it in response to like this immediate moment that we're in where you have millions of people literally either trying to figure out their next path, figure out what their next job is going to be, figure out what additional skills do they need. I, I think of like an immediate use case is in that kind of shorter term. And then I think we can think about the evolution of it down the road and there might be opportunities to pilot it with folks who are frankly willing, right. To put their programs under the microscope to even do this at that larger kind of four year institutional level. So I, you know, I agree, I mean of course I agree with Steve that it is going to be an evolution to figure out like how does this fit in that, like broader higher education marketplace. But in terms of back to your earlier question about what is the field? What is the response to the field? I think there is a more immediate response in the field as it relates to the short term training programs and those shorter term credentials and certificate programs in the immediate. So that's kind of where my eye is focused to be able to, you know, to get up and running and begin like continuing to expand it even broader.

So, so one more question on this and then, and then we'll do one more question to wrap up. But the, the question in my mind out of that because Tamisha, you just talked about the short term programs. Been a big debate on Capitol Hill. You all know it better than I do. I suspect around short term pell and uncertain results and outcomes. And then simultaneously there's been some noise maybe at last around a college transparency act and getting more data between post secondary programs and workforce outcomes nationally or at least releasing some of the restrictions that have been in place there. How do you see equos fitting into those policy worlds? Does you Know, would some of this render equos not as important or does it become more important as we have more transparency and ways to connect this data? So Tamisha, maybe you first.

Yeah, I mean I think it actually does become more important from a transparency perspective. I think of things like the eligible training provider list that is, you know, the requirement under wioa. And is there, are there states who are willing. Right, because it's, it's about, it's about like the voluntary, you know, receptibility of, of institutions and organizations. Right. And systems for that matter. But are there states who, who once they have a better understanding of equals and how it works, who might be willing to, to put their ETPL list right through the framework and see kind of what, what rises and what stands and how do they make that information then more transparent, right, to local workforce boards, to, you know, to, to individuals who walk through the doors of American job centers? I mean, I think there is, there is a broader like kind of public system opportunity and conversation that then of course then translates to that. But I think it is important, particularly as you're looking at long term pell, like what is the quality standard within that and could equals be a piece of that or a testing ground for foreseeing that. So I do think that there is an opportunity to add this into part of the policy conversations as a solution from a policy and system perspective to be able to solve the question of what is actually working and what actually is resulting in good outcomes for learners and participants.

Pension programs makes a lot of sense, Steve.

Yeah, yeah, I'll build on that really quickly. I mean, thinking about the, the relevance of equos, does it make it sort of moot or more relevant? In my own point of view is that it becomes more relevant. So you know, it's a data rich field, but information poor is, is kind of the, the way to think about it. And Equos can help organize that data. Adviser states, institutions, state workforce agencies, regions, as they make this information available. You know, being able to say, hey, it's, it's not just the data coming out of providers that's important. It must be cross referenced with demand data. It must have deeper understanding of the value of these credentials. Because while two credentials may seem very similar, if there's a demand signal that one is more preferred gives you a better opportunity professionally. You know, those are the types of information that must be cross referenced as we, as we, as we implement hopefully some of this legislation. It's absolutely essential to do this work.

Well, it's interesting hearing you say that also because I love the dynamic nature of what you're doing, thinking about. Right. In terms of responding to local conditions. It's always felt to me that sometimes policymakers want to set a bar, if you will, of it's either good or bad, if it's above this. And yet if we're in a really flush economy, placement rates for all programs might be rising above that minimum bar. But we'd still want to know where the better programs are, right. In that context to help people find those programs and lift those programs up. So. So I think that actually brings me into the place where I want to wrap up with both of you, which is focused on equity. And Steve, start with you, but then, Tamisha, I'd love you to jump in right afterwards, which is. We've alluded to it a couple times during this conversation, but specifically, how do you see the EQUOS acquisition helping JFF drive greater equity in post secondary education and training?

It's perhaps the most important component of some of the work that we are embarking on here with equos. So I talked a little bit about some of the mechanics of updating the framework and so forth. To be more explicit about how we describe whether this is sort of an equitable program, equitable pathway and so forth. We have our racial economic equity work that is here at jff, we have a large report coming out very soon led by Michael Collins and his team, which describes how different populations of folks have totally different opportunities based on their access to different kinds of pathways, whether they're tech or other in demand pathways. And so I think being able to be very explicit about what type of information must be collected to be able to adequately evaluate the equity of one program or pathway over another, as well as going back to some of what we were talking about in terms of making this information available to as wide a range of individuals as possible through the platforms, through state agencies, through other ways to interact with the information. So I think that, you know, for Equos, it. It really needs to become the heart of how we are recommending the evaluation of this work. And it's not easy to do. We were on, you know, a call with a very large tech platform not long ago. They do not collect information on demographic for probably good reasons, but it also makes it incredibly difficult to understand the efficacy at that level. So I think we will build on that from a technical perspective within the framework and the work itself. And it's JFF's leadership in these areas that we will use to inform the Framework moving forward. Tamisha, I wonder if you have any.

Yeah, I mean, I don't think that you can have a conversation or like a standard or a framework about quality if it isn't grounded and willing. Right. To take a look at data through the lens of race and gender and other levels of demographics. And so I see the application of equals connecting back to racial equity by that kind of dynamism of the framework in terms of what you add on to. And I think part of what we would hope to add on to it is the information about participants, demographic information. You cannot understand, right. How quality impacts different communities if you don't have an ability to disaggregate the data. Right. And so as much as we are able to build on equals with that as another element in the, in what goes into and what programs and providers put in, that is what will be able to make it be applicable and be able to understand what. How quality is experienced by people. How is it experienced by black people? How is it experienced by people who Latinx people housed, Experience by women like you have to get the data in there to be able to do it. It's not going to happen without that. But as I, as I look at this and the potential of it, that that's, that's a piece of it.

Yeah, I think that's a tremendous set of points. And to me it sometimes feels like no Child Left Behind. You're a sort of conversation from the K12 world. But I think that subgroup information is incredibly important to get that holistic picture and also make sure, frankly that these standards aren't used in a way, and I said it earlier, but I'll say it again, to incentivize the skimming or creaming. Right. Of students just in an effort to goose their results. And it's a key reason, I think, why we want to understand both the subgroup information, but also why are students going coming to a particular program like what is their circumstances and their motivations so that we can really put these things together and make sure that learners can find the right program for their situation that's most likely to result in success. So I think what you all are saying brings this together incredibly nicely. And I'll just say thank you to you both. I'm tremendously, personally excited to see equals find a home in the JFF family, if you will. And Steve Tamish, looking forward to working with both of you days, weeks, months, years to come to expand the impact in the ways that we discuss. And for all of you, link, if you have follow ups. Feel free to ping us if you liked this video, if you liked the conversation with Steven, thumbs up so we can get more content like this and this content can be discovered. And frankly, if you know folks who are ready to go,