Using Data to Improve Your Work With Students with Dr. Jordan!

Hello everybody, welcome. I see some familiar names and others that I don't know yet. First off, I'd like to welcome you. Parallel is a community and we are so happy you are joining us today. I want to start by saying thank you to our providers. You are the face of Parallel and the ones that make a difference in the lives of our students daily. We are so grateful to have you. And to our HQ team, we are so appreciative of the work you do daily to help support different learners and thinkers. If you're not a current Parallel provider, we are hiring, and we'll talk a little bit more about that later on. So stay tuned. Now, a few quick housekeeping items before we start. We encourage you to ask questions throughout the webinar, and we will get to them at the end. Please put your questions in the Q&A. Also new this time, we have some interactive polls and yes/no questions. We're so thrilled to be an official NASP CPT credit offer provider, and so what we would like to do is to make sure that you do interact with those polls and stay for the entire duration of the webinar. in order to receive a certification of completion at the end. Now with that, I'd like to share a short video. I'd like to introduce you to some of our providers and share a little testimonial.

I'm Dr. Jordan Wright. I am the Chief Clinical Officer here at Parallel Learning. We believe that within every kid there is a capacity to learn, there's a capacity to grow, and we need to figure out the best way to help them move forward in doing that. Parallel is an organization that is dedicated to increasing access to evaluations and special education services to kids who often do not have ready access to these services.

I am Kimberly Cortez, and I'm a speech-language pathologist here at Parallel. Parallel really stands out to me for the way they support Their community. There is open communication at all times. There is not a time where I have not felt supported by Parallel. My name is Stacy Quarter. I'm a school psychologist with Parallel Learning. There's tools and resources that allow us to be more efficient so then we can meet more students' needs.

We understand that we are all here to support those who are supporting our kids.

And they do a great job at making sure that you have the resources you need.

We have a clinical machine at Parallel that supports our providers. We've got clinical leads who offer quality assurance and support and office hours.

Parallel has been just a great place where I can connect with students, connect with families, and just build community with my coworkers. Of course, working remotely has a lot of benefits. There's the flexibility. You can make the schedule work for you. the way you need to for you in your life and what's going on. I feel supported and I feel I have the flexibility to work and grow in my career on my own terms.

We offer a lot of flexibility. We offer a lot of resources and a lot of tools.

They're very mindful that the providers that they bring in are a match for the company's culture and that they can meet the students' needs effectively.

We ask our providers to bring their whole authentic self into the work that they do with our kids. We want to make sure that you believe that every kid has the capacity to learn and grow.

All right. And now, without further ado, I would like to introduce you to award-winning author, Founding Director of the NYU Center for Counseling and Community Well-Being, and Parallel's Chief Clinical Officer, Dr. Jordan.

Thank you so much, Corrine. I really appreciate it. And thank you everybody for joining this evening. I am excited to nerd out with you a little bit about data and using data to improve the work that we're doing with the kiddos that we're working with. So thank you for joining. All right, here's what we're going to go through this evening. We're going to start with what the research says are the benefits for collecting data, using data to monitor the progress that our students are making in the work that we're doing with them. We're going to talk about ways of collecting that data, different ways that we can look at the data. And then how to mindfully use it, how to really think about being very deliberate about taking the data, being very careful with it, because data are messy, and reinforcing what it is that we're doing, how we're working with them, so that we can try and do the best work possible. So what does the research say are the benefits of collecting ongoing data? for working with our students. And I'm specifically going to be talking about all different types of services, from specialized instruction to counseling to speech therapy to OT, all of the individual work we're doing. We've looked at all of the data together to try and see, are there benefits across the board when we collect data about our kiddos? The first ones are about the process, how they help the process. One thing that's emerged in the research is that we just get more feedback about student progress. This seems like a no-brainer, right? The more we ask how a kid is doing, the more we are tracking very specifically how a kid is doing. If we ask teachers, if we ask parents and caregivers, the more we know about the actual progress that kids are making. Not just what we see in that small time we're with them, but is it generalizing out? We're just getting more feedback. It's also opening up different doors for kids to communicate how they're doing. So we are very used to monitoring IEP goals. We're very used to monitoring more objective measures, like how many times did a kid do this, or how many prompts did a kid need to display inappropriate behavior? We're very good at that. We're very well trained to think in this way. But when we think about asking kids how they're doing overall, when we're thinking about generalizing those behaviors out to the real world, to their families, to their classrooms, all these things, We tend to be less tooled up to do so, and this gives kids a different way of letting us know exactly how they're doing. It also allows us a new way, a different way of communicating directly with all of the different stakeholders, not just students themselves, but school communities, the IEP teams, It gives us a way— the more hard data that we have about the kids we're working with, the more we can do those quarterly progress reports, the easier it is to communicate with parents and caregivers about the concrete gains that we're seeing with their kids. Now, there are some outcomes that I— some of the research I call sort of these hybrid process and outcome variables. So these are great outcomes in and of themselves, but they are also facilitating. So I think of them as process and outcome. So we know that the more we ask students how they're doing, the greater the engagement they have in the work that they're doing with us and their relationship with us. This has been seen pretty consistently across the board, actually asking kids, How is this going? How are we doing? How are you doing? In a systematic way makes them more invested in the work that they're doing with us, which is a great side effect. One of the biggest outcomes of that, when they are more engaged, they are much less likely to skip out or be non-compliant with the work that we're doing. I know I'm a psychologist, so I often give kids sort of feelings homework or thought homework or things to do between this week and next week, or between this session and next session. And some kids just don't do it, right? They're kids. I mean, adults don't do it either. There are many reasons that kids and adults and everybody are non-compliant. with what we are doing, with what we're trying to accomplish with them. This is true across the board. There are gonna be some kids who, if you give them speech practice to do between now and next session, they will absolutely do it. There are gonna be other kids who absolutely won't. What we've found in the research is that the more you ask kids how they're doing, how this is going, the more data we're collecting on the kids, the more likely they are to really engage in the work. Do the homework, do the things that we're asking of them from beginning to end. And a little line of research that I've really liked is, has seen an increase in growth mindset for these kiddos. So when we ask them, how are you doing now? How is this going? It gives them one, some ownership. Like, oh, You care how I'm doing. I need to work at this. So we work together to get better. But it also gives them that hope, that growth mindset that they can do better. They can gain skills. They can learn whatever it is they need to learn in whichever service that we're providing with them. They feel more like they can grow. When we ask them how they're doing. And then there are some outcomes, just some pure outcomes. And one of the best outcomes that I, in the literature that I like the most, is the relationship is better. The relationship between me and the kids is everything. We know that in the therapy literature, in the speech literature, in OT and instruction, in teaching in general, both General ed, in special ed, we know that that relationship between the kid and the person who is helping the kid learn or grow accounts for a huge proportion of how much they in fact grow and improve and learn. So this relationship is key, is absolutely key. And when kids think that you care, about how they're doing and they get that hint through you asking, how are you doing? You're collecting the data on them. It improves that relationship. They tend to be more satisfied with the work that we're doing. When you actually ask them, when they, when you give them the opportunity as young as 6 years old, 5 years old, 6 years old, when you give them the opportunity to to give you feedback. And when you take that feedback in, they are just more satisfied with the services. It grows the older they are, right? The more— the older they are, the older the kiddos are, the more reliable their self-report is on how they're doing, how they think the services are going. So it's a better predictor with a little bit older kids and definitely adults. But across the board, we see that students are way more satisfied with services When we are consistently asking and monitoring, how are you doing? And how is this going? And then the, the sort of holy grail of this all is everything improves. Our outcomes improve when it's psychotherapy or counseling. They learn those skills. They practice the coping mechanisms. Their behavior is better. With speech, they learn and grow in articulation or pragmatics or whatever it is they're working on faster and more effectively. In education, they learn better, deeper, and more quickly when we are deliberate about asking about their data, looking at their data, tracking it, and using that information to better the work that we're doing with them. So all of this In aggregate tells us we should probably be doing this. We should probably take the time to collect data on each kiddo as we're working with them and on each student as we're working with them in counseling, speech, OT, executive function coaching, instruction, direct instruction, just academics. We should be checking in with them and making sure they're learning and growing the way that we want them to. Both from our perspective and their perspective. To me, the research shows it's a no-brainer. However, we're gonna do our first poll here. Let me see if I can launch this poll. The question is, how nervous does collecting this kind of data make you? No, that's a lie. It was the wrong poll. So thinking about your students, here we go. Thinking about your students and the work you're doing with them, how satisfied are you with their progress and outcomes? How satisfied are you actually, honestly? Now I've just launched the poll. It should pop up and you can respond to it. Now I've given some Silly responses. I'm hoping we don't get a lot of, I'm a terrible, terrible clinician and none of my students make any progress. I doubt we're gonna get any perfect clinicians here. My guess is we're gonna hover somewhere in the middle, which is great. We'll give it another 30 seconds. Great, we're plateauing. Thank you. And about 5 more seconds if you haven't answered yet and you want to. All right, I'm gonna share the results. So you should be seeing the results. We had about 70% saying mostly satisfied, 30% saying a little satisfied. That is where I would probably hover between those. I'd probably, if I could find a medium in between a little and mostly, like kind of satisfied. That's sort of just the way that I think and work. But this is exactly where we expect to see most providers. Like, yeah, we're making progress, we're doing some good work, but we could probably be doing better.

All right.

So to me, it's a no-brainer. We should do it. The research says we should do it. However, there are some barriers to collecting these outcome data, especially when we're asking kids to self-report how they're doing every single session. We're, you know, we're asking them on an ongoing basis. There are some barriers. The first one is buy-in. Now, I'm all in. Right? I'm all in because I've seen the research. I've seen how I can— I've used it and I've improved the work that I've done with some kiddos based on it. But if you think it's just an exercise, if you think it's going to be used to evaluate you somehow and be used against you in some way by your school or by the company you work for or anything, Or, and this is probably the biggest one, if you think it's a waste of time, right? If providers think it's a waste of time, this is a huge barrier because it does take some time out of a session. When we have, you know, service minutes to deliver, each one of those minutes is important. It's sacred in certain ways. We hate giving up a minute and a half at the end of a session. Now we waste a minute and a half. on all sessions, in all sessions with people being late, and like that minute and a half a week probably isn't going to make a big impact on how much progress a kid is going to make or how much they're going to learn or grow. But this is a pushback that I definitely hear a lot, like, no, I can't spare that minute, that minute and a half at the end of a session to have a kid rate how they're doing or how the work we're doing. is going. Another barrier is student buy-in. They may not really understand what they're being asked. We have to be very careful how we ask these questions so that we're getting decent data. They may just think it's an evaluation of them. They may not really understand how or why you're doing this. And this is a barrier. We want to make sure that kids understand Why we're doing what we're doing. We want to make sure that they understand that it's a genuine ask. We really want to know from your perspective, how are you doing? How is this going? How do you feel about it? If we're genuinely asking that and we're genuinely using that information to do better work, they buy in. The last one, and this is not a small issue, which is resources. The original way of collecting these data in person are with little surveys on pieces of paper, right? You hand over a piece of paper, you have a kid write it out, they hand it back to you, you score it and do it like that. Adds a lot of work. It adds a lot, kills a lot of trees. It adds an onus, a burden to the provider, to the clinician. or to the teacher that they may just not have the resources to be able to do. So we want to make sure that if you're going to do this in earnest, if you want to give this a shot, we want to make sure that you have the resources to make it as easy as possible, to automatically score it if possible, to track trajectories over time, to graph it for you. We want to make sure that you have those resources. to be able to use it appropriately and easily. So we're going to talk about 2 types of outcome data collection. These are 2 things that we use here at Parallel. These are not the only way of collecting outcome data. It's not the only way we collect outcome data. We— but I'm just going to use 2 as examples because I think they illustrate really nicely The different ways we triangulate data. We're not going to rely entirely on my perspective, my perspective as a clinician. We're not going to rely entirely on a kid's self-report, right? We want to collect it from multiple angles so that we can try and really understand what is going on. So I'm going to start with something we have called the Enhanced Goal Tracker, where we are tracking how well kids are doing against their IEP goals. This is— we're very, very used to thinking in this way. How much progress is this kid making against their IEP goal? IEP goals, whenever possible, are quantitatively defined, right? We want to really think about they display this behavior 90% of the time, or they need this type of support less and less and less to get down to needing no support to make this certain thing happen or to demonstrate this skill or this learning, right? It's very quantitative, which means we need a tool to evaluate it quantitatively. When we look at outcomes data against IEP goals, we are, we very often are thinking session to session, but really aggregating it and looking maybe once a month, maybe every quarter, so that because we have to do our progress reports in our IEP systems, and we're probably relooking at those goals and thinking, oh yeah, this kid has made some progress here and not so much here on these goals. We at Parallel really want to look at it at every single session. Sounds like a big burden, right? A real pain in the butt. But if you have the tooling, if you have the appropriate resources and way to do it, it can be very easy. Here's an example. If you have a target behavior that you wanna track in a session, we've got a counter, you've got a counter and you can say, oh, out of 10 attempts, they got 5 correct in this example, 5 incorrect, and needed 5 prompts. Another example is how accurate were they at different tasks? So I can slide this bar up and down and say for this task 1 activity, they were, I have here 10% accurate, or I can slide that up and say, oh, they did really well this session. They were 90% accurate on that one task, on that one skill demonstration. This helps us monitor and say we're ready to move on to the next skill. And we may only have 2 tasks or 2 skills that we're working on in a session. We may have 4, we may have more than that. Another example, because these are all quantitatively defined, but they're all defined in slightly different ways. I know in counseling we use this a lot when we're trying to get them to do multi-step tasks. We do this in executive function coaching as well when we're training somebody to think through The different steps to solve a problem, which is a great executive function. I always point at my prefrontal cortex as if everyone knows what that means, but my executive functioning right here at the tippy front of my brain is regulating how well I can break this larger task down into smaller steps. And then I can track, oh yeah, you did task 1 accurately, task 2 accurately. In this case, Step 3, you just didn't do. You skipped it. And so step 4 was sloppy, or it didn't— you did it, but the output or the outcome wasn't as strong as it could have been had you done step 3. This helps us track it because if you are week to week consistently missing step 3, I need to know that and work on step 3, whatever that is. I need to know, oh, you're really good at breaking it down, which might be task step 1. At listing out the different steps, at solving them one at a time. Oh, you're not good at solving them one at a time. You want to jump to the answer, which is step 4. And we— so we need to slow down. And I know that because I'm tracking it in real time during a session. So it doesn't add any workload or time demands to me as a clinician because I can do it in session. Again, if I have the resource, we use this enhanced goal tracker to help us do that work. Now, routine outcome monitoring. Routine outcome monitoring is a— this is what a lot of that research was based on that I presented at the beginning of the presentation about the benefits of asking kids, of getting their perspective. There are different ways of doing routine outcome monitoring. Again, I'm just going to show you the way we do it here at Parallel. Because that is easy access to me. It is student-reported at every single session. This is really important. It is routine. It is not every 3 months we ask them how they're doing. We really want to see week to week how it's going, and we want to use that information week to week. It's crude, right? It is, it is a blunt, but it's an— we want to use an easy-to-understand rating. of both functioning and sentiment. And I'll show you exactly what I mean by both functioning and sentiment in a second. But we want to make this easy enough to use for our younger kiddos, clear enough to use for those with disabilities, those with learning differences, those with ADHD. We want to make it easy enough to understand and use. That, of course, means it's going to be less precise, and it's going to be less perfect data. That's okay. I'm probably not going to use one data point from one session to mean all that much. But I am going to start aggregating the data maybe month to month and looking at trends and thinking about what it means for my work. So this is our scale. We go from frowny face to smiley face. I love it. I love a good frowny face to smiley face scale. We do translate this. We automatically— it automatically translates into a scale of 1 to 10. To the nearest tenth. So we can get a 3.7, we can get a 6.9. Kids aren't having to worry about those numbers. They're just like, yeah, I was pretty— I'm leaning more toward the smiley face, or I'm leaning more toward the frowny face. And this is what helps us understand what they are thinking and feeling from week to week. We ask 4 questions. Again, you can ask 1 question, you can ask 7 questions, different routine outcome monitoring systems work a little bit differently. We've broken it down to ask 4 questions in these 4 areas, 2 about functioning and 2 about sentiment. So for global functioning, we'll say, how am I doing overall? I kiddo, how am I doing? And I'm going to slide. I kiddo, I'm the student. I'm going to slide. Yeah, I'm doing okay. Or no, I'm pretty grumpy today. I'm gonna go left and right. This is global functioning. How are you doing in general? Then we have specific functioning. How is my whatever it is we're working on? Reading. How is my reading going at the moment? How is my math going at the moment? How are my feelings going at the moment? How is my talking in class going at the moment? How is my control over my Behaviors going at the moment. This is specific functioning. It's specific to the work we are doing. Session. How did today's session go in general? I'm curious how sessions are going. And how do I feel about my therapist, my speech therapist, my teacher, my whatever? How do I feel about you? And I get to slide. I, student, get to slide back and forth. And yes, I may have a bad session and be all the way on the grumpy, frowny face on both session and provider sentiment one week and then up the next week. So I may not need to take every single one of these seriously as its own data point. But once we start aggregating the data together, we want to see, are we in general seeing some trend up? Or probably more importantly, are we making sure we're not seeing trends down? We, we definitely do not want to see any trends down if at all possible. So we're gonna do another poll. This is the one I accidentally almost launched last time. How nervous does this make you? How scared are you to ask these questions honestly? All right, I have launched it. Take a few seconds. Let me know how nervous. Doing this kind of makes you good. I see some terrifieds. I like it. We will give it a few. I will be honest, I was nervous to begin with. I was, I was not super on board and doing it. All right, we'll give it 10 more seconds if you want to answer this. How nervous? All right, look at this. We've got some people who are ready to go. I love it. We've got some people who are a little nervous, a lot nervous, and terrified. I get that as well. I absolutely understand that getting this kind of information is sometimes a double-edged sword, right? We may find out things we don't want to know. And we may not know who has access to this data and if they're going to use it nefariously. We are at Parallel, we're very clear, we're using this to improve the work we're doing, right? I, as a provider, want this information so I can do better work with this kid. And we'll talk about how to do that. Right. Thank you for doing these polls. How do we use these data? How do we use it in practice? First of all, think about a session. If you've got a low alliance score, like they don't feel good about you as a provider, don't worry. That's okay. We're not always about— especially if you're a teacher. Sometimes teachers are not popular. Sometimes therapists are not popular. Sometimes we do tough work, right? So the work in the session may have been just tough that day. They may be frustrated with themselves. And they may be frustrated with their lack of getting it, whatever it is you're trying to work on them with. It could, though, especially if you see it as a trend, it could be a misattunement, a misalignment between you and that kid, right? We want to make sure that— I tell my doctoral students, I teach in the doctoral program at NYU, and I tell them all the time, the more decades that I am training students to become therapists, the more I think that 90% of it is just being nice. Be nice to kids, be nice to parents, be nice to your colleagues, be nice to school staff, and you're already 90% of the way there. And then everything else is like, do you know what you're doing? Like, do you know the techniques and the interventions? But, you know, we want to make sure that our kids like us and see us as good, helpful helpers. So it could be misattunement, it could be something else entirely. Don't jump to conclusions that just because you got a low score one week or even one month that it's a horrible thing. You may just be doing tougher work with that student, and that's okay. Think about an intervention. So if you're not seeing good scores on the IEP goals progress or on the functioning, how, how am I doing overall? How is my How are my feelings doing? Or how is my reading going? If we're seeing low scores, then maybe the interventions are not landing well with that student, right? Maybe they're just not the right way of interacting with that kid. I tend to be a little more irreverent, and I tend to like make jokes and that kind of stuff. And there are definitely kids out there who are not joking around, and they do not jive with my personality, and I have to change myself. I have to be like, okay, For this kid, I need to change my demeanor a little bit, change my interventions. The IEP goals and the functioning scores, the progress in the ROM functioning scores, also tell us which interventions are actually influencing that student's functioning outside of the room.

Right?

So if I'm, you know, seeing progress in a speech goal in the moment, but they're reporting, my speech is not going well outside of this room, that's important to know. It's not generalizing, and we need to know that. And it helps us understand what works best for that particular student. We can look sort of week on week like, oh, that was a great week. They really made progress and had some high ratings. What did I do that week? Let's replicate that. Let's do more of that. Let's be more that way. And think about patterns. This is for me the scariest thing, and the hardest thing for me is to look across all of my students and see, are there things I'm doing better than other things? Right? Are there sessions where all my ratings are really good and my IEP goal progress is up? And so I'm doing better with that kind of thing. But when I'm trying to intervene on something else, I'm not great at it, right? So I'm really, really Good at helping people cheer up, right? I'm good at working with depression. Depressed kids, throw them at me all day long. I love working with depressed kids. Anxious kids, I'm less good at, probably because I'm a little bit like Muppety. You can tell I'm a little manic. And, and so that neurotic style probably just isn't as strong with anxiety. I know how to do it. I know the skills. I know what to do. But I'm just a little less effective with anxiety than I am with depression? That's hard for me to know and find out. We can also look at types of students. Are you better at working with younger students than older students, or older students than younger students, or girls instead of boys? Like, we can look and say, across all of my caseload, who am I thriving with? Who is the work really working out with well, and when is it not? But then also think about, are there trends where This happens. I'm using a counseling example. Like, sometimes kids cry in therapy, and sometimes they don't like crying in therapy, so their ratings may be a little bit lower, and that's okay. That's just a trend we see. It's not something I'm going to change necessarily. I'm not going to be like, never cry. That's not what I believe in. But we can look and look about the— look at these patterns, look at these trends to help me grow. As a clinician, as a provider. When you're not seeing progress across IEP goals, across ROM responses and trends, change your strategies. Try something else. Try something new. Think outside of the box. Go crazy, but change something if you're not seeing progress. When you are seeing progress, And this is why kids do better with these ROMs. Champion it, cheerlead it. Do those big claps, those big hoorays on whatever platform you're working on. Cheerlead the heck out of it, right? We want to make sure kids feel proud of the accomplishments that they are making in school, in therapy, in speech, in whatever they're working on. Like, oh, I've seen this trend. Let me show it to you, kiddo. Look how much you look better. You're doing. Look at that line. Look how high it's going. That's awesome. I love it. Thumbs up. When progress is stalled, you may also want to revisit your IEP goals to make sure they're appropriate, to make sure you're not maybe aiming too sophisticated, aiming too high, and maybe there's some foundational work you need to do before. you can make actual progress on where, where you're aiming. So it's always good when things are plateauing to revisit those goals. And then you can use the data to communicate progress. So I said this at the very beginning, you can always use these data in your progress reports, in your IEP systems, but you can also communicate this back to parents and caregivers, IEP teams, teachers, People who care. Anyone who cares about this kid, you can use these data to show them, look what this kid is doing. Look at this growth, or look at this dip that happened in March. What happened in March to this kid that I don't know about? Because April they went back up and now we're making progress again. We can collaborate using these data with the people who are important in these kids' lives. All right, I'm gonna do an example. This is an example I've pulled from our data at Parallel. I'm gonna— I don't even know, I don't even remember who the provider was, but I'm gonna thank our provider and all of our providers because you've done great work and data, and I'm excited about it. So this is a 12-year-old boy who's in speech therapy. Obviously, I'm not giving any more details than this, so you will not be able to identify this kid. And I, I've picked one of the goals, which is to articulate clearly his needs and feelings in the context of sessions, ultimately to generalize out to his classroom communication. This kid was having problems articulating what his needs and feelings were. And so the goal is, how can we better put language to this kid's needs and feelings? So we start— I'm just going to start randomly with some ROM scores. You can see here for October, we've averaged all of the scores in October, and we see the sessions went pretty well, right? We're hovering around a 9 in terms of the, the how he felt about his sessions. However, how do I feel about my therapist is hovering right above 4. Across all of October, we've averaged it across all of October. What would you do if you saw these data? I'm going to launch this poll. What would you do if you saw a slightly above 4 in how much this kid liked you? Obviously, I've given some silly responses. Would you freak out? Would you ignore it because you're awesome and everything will be fine? Would you be nicer? Would you think about why the student may not be happy with you? Getting some very quick responses here. Give it another couple seconds. 5 more seconds if you want to respond. I love this. Look at this, 94% think about, you know, why the student may not be as happy as you want them to be, and a couple people freaking out. I would freak out too. I think I'd probably, if I could, if I could have responded both, I'd probably be like, a 4? I'm not a 4. I need to be much better than a 4. So let's look at what this provider did. This provider took that data and made some changes. Now, I don't know exactly what she did, but she used this for and said, okay, we need to get better here. So she switched something around. And when we look at that trajectory, we see improvement November to December to January. And now we're well above 8, which is where we want to— like, we definitely want to be above 8, 8 is excellent. This, right? This is a fantastic score, 8 and above. 7 and above, I'm happy with. Below 7, like I need to make some changes or something needs to happen. Remembering that there may be individual sessions where they were higher or lower, like maybe they had a bad week, but because we're averaging these scores across an entire month, we're seeing some trends and we do need to look at trends. more than probably an individual session. Now, this is good. This is good progress in terms of the relationship, in terms of the, the how this kid feels about the work that they're doing together. But that's great. But does it align with actual improvement in functioning? So let's look at the ROM scores for their global functioning, this kid's global functioning, and His clarity of speaking, and we see some real growth, right? We're seeing overall he was doing fine. Like, we're asking global, like, maybe his speech is not the thing that's driving his overall well-being. Maybe that's one small part of who this kid is, and he was doing okay. He was at a 6, but he's gotten better. He's, he's well above 8. Um, how well is the clarity of speaking going? That's a hard thing. Right? That takes more than a month to, uh, to grow in articulating outside in your classroom, like, your needs. And, um, but he's feeling pretty good about it by January. He's really feeling the progress. And are we seeing that progress? So let's look. This is our enhanced goal tracker, and we're seeing the percent of statements that are clear in session. In the moment, we're tagging these. Yep, clear, not clear, clear, not clear has gone significantly up, right? We're we're gaining toward pure clarity in needs and feelings, and I'm not at 100%. I guarantee that, right? And I am much older than 12, and we're seeing a decrease in the need for prompting, need for clarification. Say, ooh, say that again. Those are prompts. Say it in a different way. That's a good prompt. But we're seeing a decrease in that need. So across all of these metrics, the sentiment metrics getting stronger, the functioning ROMs getting better, and our looking at the IEP goal progress getting better, needing fewer prompts. Which is fantastic. I love this provider, whoever you are out there. I'm gonna stop here. We've got about 15 minutes, about 10 minutes probably for Q&A, and then we'll have a couple more announcements. But I'm gonna invite Kareem back and see if there are any questions for me, any freakouts, any clarifications about how to use these data.

Amazing. Thank you so much. That was very informative. We do have a couple of questions. So providers are wondering, how can they get the results of the ROMs? And I guess there's a lot of interest and curiosity to see how they can use the data to make appropriate adjustments. So the question is, how can a provider get the results of the Yes, it's coming.

That is all coming. We are— you know this, you providers who are our backbone and our face and the strongest part of who we are. You know we're a young company. We are working on it. So we are collecting ROM data. We are looking. I will tell you one of the things we at Parallel are also thinking about is not really about like the provider themselves, but across all of our speech, how are we doing across all of our counseling? I'll give you an example in counseling when we have a curriculum. So we are using an evidence-based curriculum for all kids who have anxiety as an example. And we can look at all of these ROMs and the Enhanced Goal Tracker data to see, is our curriculum working? Because if it's not, we need to swap out our curriculum. So we can use these data at HQ on our side to evaluate how are we doing overall as a company? Are we supporting our providers enough with curricula, with tools, with games, with whatever it is? Because if we're not, we're going to see that show up in the ROMs. This was a long way of saying it is coming. We are, we are working on trying to figure out the best way to get you more usable real-time ROM data so that you can see that progress as it's happening. So stay tuned.

All right, thank you so much. Please continue to put your questions in the chat. We have a few more. So there is— somebody is asking about the phenomenon of survey fatigue or respondent fatigue and wondering how can they ensure that students aren't affected by that if they're being asked at every session about what they're feeling and etc.

Yeah. So the research around survey fatigue is shaky at best. We know it's a thing. I get fatigued by too many Studies and research, but what what what the small body of research out there says shorter surveys don't contribute to survey fatigue. So yes, it sounds like a lot like every session, but if I'm seeing them once a week, I'm asking them four questions once a week. They're you know probably more fatigued by me and my energy than they are by answering those questions. If you have a kid who is seeing five providers in a row. And they're all asking them to do these sorts of things, then I might worry about survey fatigue. I'm not worried about it if they're getting a 4-question survey once a day, or once a week, or twice a week, or something like that. I'm just not worried about it. The other thing to think about is, if you suspect survey fatigue, first of all, you can always ask, right? We're in a relationship with these kiddos. We can ask the question like, Is this too much? But you also can track that. So if you see a kid 4 days a week and you notice that every Thursday it's a lower ratings, then you can be like, huh, I wonder what's going on on Thursdays, or I wonder if this might be they're just sick of doing this survey and then they have a weekend to recharge and they're fine again on Monday. But again, you always want to ask the question. You want to use the data.

All right, this definitely generated some additional questions. Somebody is asking regarding their particular caseload that they have middle school students with emotional impairments and behavioral issues. They're concerned that these students would fudge the survey purposely. What kind of comment would you have for that?

Absolutely. Look, survey data, all data are problematic, right? You're always going to have that one kid who just loves frowny faces and he's like, haha, I'm just going to frowny face you all the time. If you decide that the data are not useful, they're not useful, right? If you decide for this particular kid— now there are a lot of kids where we as clinicians might be like, oh, I don't think this is appropriate for them because they have this impairment or They're too young, or this kind of stuff. And the research has shown it's actually not the case. We don't want to make excuses for kids beforehand, but if they start using it and you're like, yeah, they're not really understanding, or they're just fudging it, or they're being jerks, or they're being overly compliant, or whatever, then that's okay. You can either stop doing it, or you can address it with them and be like, look, we don't have to do this. This is up to you. This is not mandatory in a way that like you're, you're gonna lose your license if you don't do it, right? When, when we do this, this is to make us better. So data are imperfect. Know that the data are imperfect. If you get a 0 on all 4 items for 3 months straight, don't go and like say, I should not be a therapist anymore, right? Like, I'm gonna give up my whole profession. Like, No, maybe that kid is just kind of being a jerk in the moment. Sometimes kids are jerks. It happens. You can address it with that kid, and you may decide to say, this kid, it's not useful. Or you may use an intervention and say, you're answering zero on everything. You're answering frowny face on everything. Tell me what that's about. Let's talk about this, right? Let's get to the bottom of this and see if you can use it.

Yeah, and I think that's where getting or receiving that feedback is going to help the providers tremendously. I do see here some concerns about working or having special needs students rate adults. I'm not clear exactly what the question is here, but I think there's some concern about maybe the accuracy or efficacy. Yeah, I think that's a really good question. Of that. Any comments about that? I mean, yeah.

Yeah, I, I'm a big believer that all data are flawed, especially survey data, especially self-report data, right? We're asking them to introspect, think about their experience, translate that into a slider. The data are imperfect, and the data are Useful. They are not the truth, right? Your scores on ROMS are not the truth. They are not the end-all be-all of how you're doing, of how that kid's doing. They are just a tool to use. They are useful to use. So you just need to kind of remind yourself over and over and over, this is a tool. It's not precise. It's a clumsy tool, right? This is, this is a giant mallet. Instead of a little precision hammer. So get the gist of it. Don't worry about the difference between a 6 and a 7, or an 8 and a 9, or a 1 and a 2. Like, don't worry about that. Worry when you see, you know, a kid who is significantly dipping, right? Because you know they knew how to do it. You know they were engaged in a way. And when you see a dip, That is when something's going awry in their life. It may not be the work you're doing, but something is happening, and they are communicating that to you.

Thank you. That was very helpful. A question here about the enhanced goal tracker. What flexibility do providers have in selecting and/or using enhanced goal trackers?

Yeah, so our enhanced goal tracker is great because you You create the goal. So you type in whatever that goal is, and then you choose which kind of tracker you want to use. Is it going to be just correct, incorrect, and you just tally them? Is it going to be a slider of accuracy? Is it going to have the prompts? You select those tools to decide how you're going to track that particular goal, because these goals are all over the place in terms of how we need to track them. And whatever it is, with the one exception of if it's a purely qualitative goal, if there's no number associated with it at all, then that's hard to tally things. It's hard to like do it well in session. You may have to take your 15 to 20 seconds after session and sort of type a little mini note in there against that purely qualitative goal. But when it is a quantitative goal, which is what we aim for in our IEP plans, right?

Yeah.

Then we have all sorts of different ways. You just do use that dropdown menu, choose the most appropriate tracking device in that moment, and you can use it in session.

All right, and I think we'll just take one more question before we wrap up here for the evening. How can we involve other stakeholders in the students' lives, such as the teachers, the parents, In the data-driven improvement process? So in other words, how can we kind of enlist the other players, the other key players in the student's life to help things?

Oh, I don't know who asked that, but I love you. So when we think about triangulating data, right, we now have Enhanced Goal Tracker. That's my report, my clinician report, my teacher report.

It's me.

We have the ROMS, which is kid report. But there are 2 other big constituencies here, right? The everyday teachers, the classroom teachers, whatever, and the parents and caregivers. So we certainly can't send a ROM to them every week. They're not going to do it. That we're going to get, you know, some survey fatigue. If, you know, if I have a special ed teacher and I'm working with all their students, and once a week I send them a survey of how is every single kid in their classroom doing, It's not gonna look— they're not gonna do it. They're just not gonna do it. Parents aren't gonna do it. So we need to think of a better cadence, and there are ways of creating specialized targeted surveys that we send again in a routine way, not every week, maybe every 2 months, to look at their functioning in those settings. So I might send a survey if I'm doing an executive functioning training with a kid, I might send an executive functioning survey like how well is this kid controlling their behavior, controlling their attention, control, like working with multi-step problems at home and send that to parents and caregivers. I'm going to keep it short. It's not going to be as short as the 4 questions, but it may be 10 questions every 2 months. And I'm going to try and get their buy-in. I'm going to do the same thing with teachers. We may or may not get those back. That's tougher because they're not captive in our session with us like kids are. So that's a little tougher sometimes, but we are going to use that to help triangulate and figure out how well this kid is growing, learning, and thriving. I have put my— yeah, sorry, Karina.

I was just going to say, I said it was the last one, but I have to ask a question for my own curiosity. So in that context, would you recommend using Google Forms, or what's your opinion on that?

Yeah, it depends. It depends what resources you have access to, right? If you have a platform that can send it out in a really great privacy-protected way, great. Google Forms can kind of do that. You can find ways to do it. Just be careful about data, right? when we're distributing data back and forth online, we just need to be very, very careful about protecting the data. All right, I have put my email address here. Please feel free to reach out if you want to nerd out about some of this stuff. I do want to say before we leave and before I say thank you, we are hiring. I can't overstate this. Our providers are amazing. Our provider— you saw the video at the beginning. That was just a little That was the tip of the iceberg. We have amazing speech therapists, social workers, school psychologists, special education instructors. Like, we've got amazing people who are connecting with our kiddos, who are doing the day-in, day-out work that make us look good. So I want to say a huge thank you to our providers, to all of you out there, the ones who are not watching live, who are going to watch later. I am so grateful for the work that you're doing with our students, with our kids, and I want you to know just how much I appreciate you. We are hiring. If you know, um, if you are not one of our providers and you're listening and you want to reach out, please come to our website. If you are a provider and you know somebody who you think would be a good culture fit with us, it would be a good match, um, for what we're doing here at Parallel, helping these kids thrive, um, we do have that, uh, referral program. We've got an incentive if you make a referral and we hire them. And, um, I just want to encourage folks who want to do the best work possible to come join us. Come work with us because we want you. And with that, please also join our socials. Please like, subscribe, whatever that is. And with that, I want to thank everybody here. Thank you, Corinne, for moderating this. And I truly appreciate you bearing with me while I talked at you for an hour. And have a wonderful, wonderful evening.

Thank you.