Academic Research From the University of California, Irvine Finds an AI Digital Assistant Helps Teachers Decrease Stress and Save Time

Levi: This is Supervised Learning, a podcast where the Merlyn Mind team learns from experts in artificial intelligence, technology, and education. We hope you enjoy learning with us through these conversations with those who know. Time to learn. It's Emily and Kylie. I am thrilled to have this conversation because it's, I guess we just looked, about two and a half years in the making, right Kylie? Do you want to tell us? We started with a strange email a long, long, long time ago. Why don't you guys introduce yourselves, Kylie?

Dr. Kylie Peppler: Yeah, so my name's Dr. Kylie Peppler. I'm a professor of both informatics and education here at the University of California, Irvine. And I do a lot of research at the intersection of new technologies and designing them for high quality teaching and learning experiences.

Levi: And Emily.

Dr. Emily Schindler: Great. My name is Dr. Emily Schindler and I am the Assistant Director of Creativity Labs here. And my background is in understanding how teachers learn with technology and in innovative environments. And on this project, I got to do a lot of the data collection and research, so.

Levi: Great. And for those of you listening, who don't know exactly what we do, let's just kind of tell you the story of why as a startup, trying to bring AI into education and do something that was very different, right. We wanted to say, how can use a voice assistant to assist teachers in the orchestration of teaching to control the tools they're using. Let them walk around the classroom freely without being tethered to the laptop or to the device. How do we do that and do it right and make sure it actually helps? Well, we said we should probably talk to experts, right? Like researchers who actually look at this stuff and understand what does technology do? What's it good for what it's not good for and how does it really help teachers and then by extension impact student and learning outcomes? So we kind of canvased the world and found these incredible researchers at the University of California, Irvine in the Creativity Labs. And I sent a cold email to Kylie two and half years ago and we... So maybe let's pause there and say, why does research matter? Why does it matter that we do these things?

Dr. Kylie Peppler: Yeah. That's a great question. And I just want to say when you walked in the doors in 2019 with this audacious idea, totally futuristic, like how are we possibly going to envision one device is going to transform the classroom experience? And so part of what we do as researchers is that we try to distill the learnings of the field and distill that into high quality theory. And this is what we do as learning scientists. And then we use that theory to inform the design. And so it was just really cool to be part of the parade and some of the early inquiry that you had. And I remember back to that discussion, we were just kind of talking about sort of the guiding theory about sort of constructionism about what would we hope for in a device, like what you were building and how might you go about that design process to inform that product? I didn't know you were going to walk back a few months later and say, hey, let's work together. But that was just a really exciting proposition, especially. Emily, you want to just talk a little bit more about what you think about the role of research and product development?

Dr. Emily Schindler: Sure. I mean, for me, because my interest is in teachers and teaching, I look at it from the point of view of, well as a teacher, what is available to you in terms of tools? And there's just lots of kind of hints in the system such as teachers aren't the ones who are generally purchasing the technology. But they are the people who have to use it every day. And so I think in terms of closing the loop between the user and the developer, I feel like research has a major role to play in that aspect.

Dr. Kylie Peppler: Yeah. Because we, as we start to think about building ed tech and just building sort of the future of innovation and education, that requires a lot of things that we've been studying in academic realms. Like how do we work together? What's that design process? So we have new processes in the field called research practice partnerships, where researchers are working more closely with teachers, but industry partners need that same kind of insight. We've got things called design based research, where we take the theory, we design in a local context, and then we refine the theory and we refine the design. And then we also have things around implementation research. We call DBIR, so design based implementation research. And so how do we prepare for scaling? So what we know about learning, what we know about teaching, what we know about design, how do we bring this together? Because there's been kind of a silo happening. We can't get good innovation out of academia. A lot of industry partners have been unable to work with researchers. So how do we actually start to design at this intersection, which is requires new work processes, new ways of working in the 21st century?

Levi: Yeah. So are we the norm? Do most startups like us come and join forces with powerful academic researchers like yourselves and your institution to go make sure that the new technology that's coming out is going to be effective and do what it hopefully could do in a classroom?

Dr. Kylie Peppler: Absolutely not. We rarely see it. Now, that doesn't mean we never see it, right. We've worked with a lot of partners over the years. I got started working with the MIT Media Lab team on the design and development of Scratch. I've worked with Leah Buechley and SparkFun on refining a lot of the LilyPad Arduino and the wearable technologies. So we have a long history of working at this intersection, but a lot of times these are kind of few and far between experiences and mostly they're kind of about scaling innovations that happen in academia, outside of academia. But that, you look at the national level, those counts are just... It's less than 1% of academic projects that actually move out of the market. And I would say it's probably less than 1% of industry partners that actually partner with researchers. And so it's just a missed opportunity there.

Levi: Right. I think there's good data out of Brookings. I believe on that. Less than 1% of ed tech companies are running these kinds of efficacy tests to see if it truly has impact on outcomes. And you could almost like, so why did we do it?

Dr. Emily Schindler: Yeah. That's what I was going to just ask. Why did you guys decide to do this?

Levi: I mean, we had great leadership internally right across the team. It's a team of researchers, right? So our founders were all AI researchers who've spent their careers in IBM, in research labs, looking at let's create a theory and a hypothesis on how we can do something effective and then let's go test it and rigorously prove if it works. And we had other great learning scientists and folks on the team who you guys know like guess who, who said, like let's do it in the learning sciences. We've done in my past kind of opportunities and places where I've worked at, at Hewlett Packard and other things like that. So we basically said, look, we're really trying to push the envelope here on what is possible with cutting edge, artificial intelligence, voice control, and automation to save steps and help teachers use technology more effectively in classrooms. We have really clear hypotheses on why that matters. Why if you make it easier for teachers to use tools and simplify workflows and orchestrate what they're doing in simpler and less steps, that extra time is going to be used productively and effectively and have outcome on students. But that's a lot of like in our minds three, four years ago, right? And so we said, you know what, right, because it really matters. And if we work with the right people like yourselves and we test this and we work with schools and we work with teachers and we work with administrators, then when we move forward to go bring this to the market. There's not as much like we got to sell you something and you just have to take, believe it by faith. We can get to the point where it's like, oh, there's actually evidence behind this. You're not just selling me some. We wanted to show that this really works. So it's scary. I think before this, you guys were asking me like, why is it scary? And if we're just very honest, right, as a startup, you're already kind of on life support more or less. You're a temporary organization. You're not necessarily going to be around forever. You have to create something that really works to continue fulfilling that vision. And that means if you get data back from a research project like this that says this product doesn't work, it doesn't have impact, it's not helpful, you have to course correct dramatically, which you might not have enough time left in the life of that company to do, right? So it was kind of a big, bold bet for us to say, we believe enough in this, we want to get it right. Let's work with the right people and let's prove that it works. And it wasn't all smooth sailing. You guys can talk more about this, but there are a lot of things that didn't work early on and we worked together and we made progress. And eventually, I think what we'll talk about today is there are good strong signals that we're on the right path.

Dr. Kylie Peppler: Oh yeah. Yeah, definitely. And that's a two way street, right. A lot of researchers don't want to get into working with industry partners because they, A, can't ... We have to change a lot of what we do normally, right. So we've got to work at a faster pace, right? We've got to think about things, like you said, the clock is ticking, right? We can't launch a product if we take too long, if we do too laborious of methods, if we can't get the product to a state that would be viable before launching, right. So it's a lot of pressure, but I would encourage researchers to start to develop the methods and the approaches to being able to do this, right. And so there's a lot of trust, I think that has to be built, because you have to have those hard conversations. You got to kind of really invest. And so both sides, this is a phenomenal relationship, in terms of just having shared expectations about what it meant to be doing research and to be inviting us into the table on that. But we've learned a lot from the games industry and other industries as well, is that typically a research project could take three to five years to find something, right. That's just too slow. No product would ever launch, right. And so you can have this perfect paradigm in your head, but at the end of the day, we need to get the evidence to administrators, to teachers to make the right decisions. And most of the decisions they make is based on marketing. It's not based on actual value to the classroom. I mean, it's a whole industry sort of built on marketing. And once you get folks invested in one platform, one approach, not because it's better, that they're resistant to changing, right?

Dr. Emily Schindler: inaudible.

Dr. Kylie Peppler: And most of the most of the evaluation is very thin, right, and it's usually done by the company itself. And so there is no consistent kind of FDA for EdTech. There's no third party platform helping to make that decision. But I think for companies that are trying to future proof themselves, having the firm foundation. If you get the product right in the beginning, right, the very first initial stages, then it's so easy to kind of build on those insights and kind of be moving in the right direction. If you don't get those foundational layers, it's just really hard. You end up either having to pitch the entire product or launch something new. So I think in the end when we get these relationships right, you can save time, you can save money and really have a clear value proposition to offer to schools.

Levi: So before we jump into the meat of this conversation, what we actually researched, what we learned, let's just kind of bring back who were the participants? So you had Merlyn Mind is the industry partner. That was the startup led by the former founders of the IBM Watson team who said, let's bring AI into education. Then you had Kylie Dr. Kylie Peppler, Dr. Emily Schindler, the UCI Creativity Labs. And then we also had another partner involved. Well, I guess at least two other groups of partners, right? We had the schools themselves, the administrators, the practitioners, the teachers, and then there was also Digital Promise. Can you tell us a little bit more about Digital Promise and their role in this partnership?

Dr. Kylie Peppler: Yes, absolutely. Digital Promise, for those of you that don't know Digital Promise, they really have... I mean, they've been around sort of building sort of a national league of innovative schools. They've done so many different projects and efforts at sort of really building a nationwide sort of community of educational leaders, of cohorts, of teachers that are really investing. And then they also have a strong research capacity. And so this is kind of the partner that kind of held our fists in the fire and kind of we're reviewing our internal reports, helping us to kind of make solid decisions and forming our theorization around thinking about classroom orchestration, around what it meant to do time savings. And a lot of what we do, especially in the learning sciences of these kind of closed loop design cycles. And so we want to get the product and we want to get our measures to the right research project, right. So you don't want to to launch at the beginning because you don't have that high confidence yet. And so working with Digital Promise really brought a lot of teacher voices to the table, strong, high quality research voices and helped us to really refine our processes moving forward. And so I think that in and of itself has been one of the innovations that we've put forward from this product, from this relationship is really what does it mean to be working at the intersection of this research practice industry partnership?

Levi: Wonderful.

Dr. Emily Schindler: I will also say just briefly about Digital Promise, they also come with a long history of collaborating with industry partners. And so they also kind of knew that landscape and were able to just advise and partner with us as we kind of walked through it.

Levi: Okay, fantastic. So now we know the group that embarked on this journey. We're going to learn, are we going to research together?

Dr. Emily Schindler: Yes.

Levi: We talked about the importance of research generally in education technology, but why was it important that we research Merlyn and Merlyn Mind's artificial intelligence approach to the classroom? Why did this deserve its own research project? What was unique about this?

Dr. Kylie Peppler: Well, I mean, to my knowledge, there's nothing comparable, right? And like I said before, it's a wild proposition, right? To bring AI into the classroom, first of all, to bring a voice assisted technology, to do things that don't have a prior model, right? So it's a radical break from what we know about classroom technologies. And then the other radical proposition is it's really a hub for the entire classroom, right? So you're orchestrating all of the technologies and real life humans in the space over time. So how do you get that, right? Just even what does that mean? Unlike Alexa, you've got to then now orchestrate teacher and children voices and what activities? What has the highest value proposition for teachers? And a lot of times that was unexpected, right? Kind of at the beginning, probably we would've had a very different hypotheses than the features we see them using all the time now, right. So Emily, what struck you?

Dr. Emily Schindler: Well, I mean, I think this technology is one that as I talked to participants, as they were kind of thinking about getting involved in our research project, many of them were afraid of this technology. There's a lot of fear and well-placed concern around bringing AI into the classroom because of data privacy, because of just the idea that what happens in a classroom ought not always to be recorded and then researchable later on, right? And so the fact that your business model does not sell data was a huge, huge selling point for those participants. I mean, and once I got into classrooms, I actually found lots of these teachers had covert Google Homes or Alexas in their classroom that they were like, "I've been secretly using this because it helps me so much." And I was like, "Well, that's amazing that we now have a device that you can use that gets you the benefit without all of the concerns about data privacy."

Levi: For us, it's very exciting to have people that represent industry in the room for us having these conversations, right. Because at that point, it was hard. I mean, and people don't realize like the research started during COVID, right. So we met you guys before COVID, but then COVID happens and it's like, how are we going to build this technology and test it and learn from teachers when we can't get into schools? And so I think there's another thing to be said for the ability for researchers and academics and former educators and current educators working with industry as the trusted go between, that in a time of chaos and uncertainty and COVID and everything else, and dealing with a product that many people were afraid of and weren't certain it was legal to bring this into the classroom, we're able to trust you and say, oh, okay, now I see. Now I see why this works. So I don't think we can underestimate how important that was in our development at that stage of the company, right?

Dr. Kylie Peppler: Yeah. That's wonderful to hear. And I think it's so true that like... But you can imagine as researchers go into this, we don't really want to ruin all of our relationships with schools either, right. So I think, that trust is really a two way street in terms of having products that we believe in, companies that we feel like have really high, strong capacity to design, to respond to the teacher comments and in real time, right. There was always a product upgrade being pushed, there was always somebody being available for the teachers. And we saw that time and time again in the research reports, especially is that they were really, to their knowledge, that was the very first time that they had really been listened to and-

Levi: Wow.

Dr. Kylie Peppler: ...that's a game changer really. And I think for ed tech companies that are listening, you really can't undersell that. Teachers really feel like they don't have a voice in the design of technology and the choice of what technologies enter their classroom. And so just really building the company very differently and future proofing it in a lot of ways. I think you've got a lot of insights to share too, and just the way that you work, not just in the product innovation.

Dr. Emily Schindler: I also do just want to put in a quick plug for qualitative educational researchers. That is an area of training that often goes, I don't know, sort of underappreciated, which is that we developed very strong relationships with the participants in our research. And so we're able to kind of we undertake explicit training in building trust and ethics and all of preserving that relationship intentionally. And so I think that in this case, that proved to be really crucial.

Dr. Kylie Peppler: Yeah. And it didn't hurt that it was a voice activated technology. Back when all the classrooms were... all the surfaces were being wiped down and the barriers inaudible all of the classrooms. So that really helped.

Levi: Yeah. Let's segue that into what did we decide to research? What were the research questions? What were we trying to learn here? What was our hypothesis on the value this product could bring into the classroom?

Dr. Kylie Peppler: Yeah. And that, I just wanted to kind of just talk a little bit about the methodology of how we got there because part of it was the listening that was happening at Merlyn Mind to like, what do administrators, what do teachers want, similar to over to Digital Promise? And then kind of our early findings from being in the classroom with the technology, what were we seeing? What were the safe bets to start to place, right? And so at that intersection, it really kind of boiled down to two things, really. It was around techno stress. In the pandemic, when you're rolling out all of these LMSs, you're introducing all sorts of new technology to solve problems, there's a growing really probably the height of techno stress happening. And then failures of technology right and left limited kind of support for those technologies. So you have the educators and teachers just really so stressed out on the job. And yeah, they're kids too. I mean, they had to learn so much. And then the second kind of area that we thought we were starting to see was really a time savings. We tried to operationalize that. And so in what ways were they able to spend time in the classroom more efficiently? And for those of you that know teachers, this is a really important issue. Emily definitely translate that for our audience. It's not to be underestimated.

Dr. Emily Schindler: Yeah. So in terms of techno stress, it was actually in phase one of the research when we first started. I think our goal was really just to see, how does this product work and what are the constraints surrounding it? It Was more almost exploratory. And what we noticed, because we happened to be implementing during perhaps the largest increase in tech use in education in our history, the teachers that I saw were facilitating synchronous learning with half of their students on Zoom and half of their students in person. And if you can just reflect on every meeting that you've had during the pandemic over Zoom and how many times somebody's muted, somebody doesn't have their camera on, somebody's linked isn't showing up properly, or you don't have access? Imagine that happening with 34th graders and trying to troubleshoot that. And that's not the goal of that interaction. You're trying to learn things. And so that's what we were observing. And the stress levels were really, really high, and time expenditures as well.

Levi: Let's come back to time expenditure next. I actually want to ask about techno stress as a word. First of all, what does techno stress even mean for those that may not have heard that term before? And then I want to ask about how did it manifest before the pandemic and then how do we believe it... Does it matter after the pandemic, or was this a unique to like it was a problem in the pandemic, but you don't need Merlyn and a digital assistant after the pandemic? Talk about techno stress and what it means and why it's present before, after, during, and probably for a long time to come.

Dr. Emily Schindler: Sure. I'm happy to take that one. So techno stress is actually a fairly old construct. The original research on it was back in the early 2000s, late'90s. And it came out of organizational psychology, which was really looking at how technology impacts people's job performance and feelings at work. But recently, it started to be applied to educational spaces because as we mentioned, there is just a huge proliferation of things to do and manage related to technology in the classroom. And rightly so. As we're preparing students to live in a world where this is the case, it makes absolute sense that we would also try to replicate that in educational spaces. And so I can't speak too much to before the pandemic, except to say that it was sort of one of those things where it was loosely defined, we all knew it was happening. If you talk to teachers about this, it's one of those concepts where they're like, uh- huh. Yeah, thank you.

Levi: I can just shed a little light on what we had discovered before we got to you guys.

Dr. Emily Schindler: Sure, yeah.

Levi: It's like, we've been spending a few years looking at how could AI help in the classroom? We had all this like there was many, many different ways we could have brought AI into the classroom. We thought like we could build a learning app, we could build a math app, we could build a reading app. There's all these things we could have done with it. And we went into classrooms and we sat with teachers before COVID, and we watched. And what we saw was, wow, they're spending a lot of time going back and forth to their computer, accessing a ton of different applications. A reading app, a math app, a quiz app, a video, a presentation. And it was like almost this constant race around the room to go back and forth, back and forth, while they're also paying attention to their students and all the applications and their students and the applications. And many times it included unplugging things, plugging things in, rolling carts around the room. And we talked to the teachers and what they said was, man, don't give me another app. I have so many good apps, just help me with the stuff I already have. And we thought about it and we said, there's something interesting here around AI simplifying how they use all of those things. We didn't know the word, techno stress. We weren't thinking of the term classroom orchestration. All we were thinking of was like, there's automation opportunity there. You could have the teacher focus on the things they're best at, eyes on students, connecting with them emotionally. And then instantly saying, hey, Merlyn, start my presentation now. Okay class, watch the video. Hey, Merlyn, pause the presentation and open Google Drive. Being able to do all of that stuff without have to looking or touching or walking, we thought could have big impact, before we ever saw COVID. Then COVID happened and we think it still matters after. But yeah, talk to me about like that insight then the research and how those things kind of came together.

Dr. Emily Schindler: Sure. It is a really wonderful feeling as a researcher because it, I have to say rarely happens where, when you say what you're researching to teachers just in plain language, they're like, yes, thank you very much and keep it up. Because often, there, we experience a little bit of a disconnect with the academy and practitioners. And so, techno stress is really just this idea that there are... Technology introduces all of these different sort of stressful factors into your life that you have to negotiate in order to accomplish your goals. And so everything from switching inputs between a peripheral like a document camera and your projector to, like you said, having to physically touch your computer so that you can negotiate some sort of issue that has arisen in the middle of your lesson. These are common every day occurrences for teachers.

Dr. Kylie Peppler: Right. Yeah. And all of this, it really significantly impacts your overall work productivity, right? And for especially in schools, what that means is that the kids don't have time to learn. So all of these are distractors and noise and they also have the consequence of distracting the teacher. And so that train of thought that actually they can't follow the question or what they would like to see happen in the school day kind of happening there. But I think Levi, you're also hinting at like one of the strong value propositions for why to bring in researchers, right? Is that we talk about this as a problem of practice, right. And what we try to do with our orientation, this is not every researcher out there, but these are learning scientists that we are oriented towards not inventing our own problems to research, right. But really starting with a true problem of practice, just like you started with your industry insights, is that we want to start with that. And then we want to give that language. And that language, we want to show up with something that's existing in the literature. And in this case, we're talking about techno stress, but then that gives us a methodology, it gives us other industries, it gives us comparison points. And now all of a sudden it can start to do something really powerful, right. We didn't just invent this ourselves, but building on that legacy of research.

Levi: So now let's talk about the research. What did we learn as we dug in and looked at Merlyn assisting teachers, does it have an impact on techno stress? Does it have an impact on time savings? Maybe just start walking us through the findings and what we learned during this.

Dr. Kylie Peppler: Well, and I also want the audience to know this is an audacious time to run any type of research project, right. I mean, A, most researchers are shut out of schools, right. So to get people to say yes in this moment is you've got to have something that is going to help, right? Otherwise, all the Chicago Public Schools shut down. They're like, stop it, don't even come near us, researchers. So large swaths of the country. And it was relatively easy for us to get these doors to open, because they're like, yeah, actually that could really help us. So there was a strong value proposition there to be able to open up that door. But we also launched, right, when we thought things were kind of... The coast was clear, right, last fall. So this is like November inaudible.

Levi: This is the second phase of the research because we've been researching for a year.

Dr. Kylie Peppler: Oh yes. Yeah. Yeah, the first phase of the research was really qualitative inquiry like Emily had said. And that just means that we're in the classroom, we're seeing what works, what doesn't work, what are people using most often? How do we translate sort of the data security? Different ways of, okay, we're fending questions, but ultimately we got to think about different onboarding processes, sort of working out all of those kinks. And sometimes we only think about the product there, but it really is how are you going to get questions? And what are the onboarding materials? What are the questions that are going to be asked? And so, there was a really nice kind of process that started to develop there, where we were reviewing a lot of those things. And at first maybe we fielded some of those questions and then what was working, we kind of passed over to your team to kind of continue to develop as you were thinking about scaling. And then the second phase of the research was really doubling down on some of these bets. So it was like, actually, the product's working consistently in classrooms, the teachers are reporting that they don't ever want to part with their devices. Now, I'm like we got something right. And that's when to pull the trigger to kind of say, hey, can we start to provide evidence around the efficacy of the product, right. And that's where we developed this quasi experimental study. You start to get to sort of the golden silver standards of the research industry, that's where you want to start to build for scale. And so the quasi experimental allowed us to really start to look at what was happening in the Symphony Classrooms compared to a matched control set. And so you could just imagine this is two different schools that could be matched or two different classrooms that are roughly the same, servicing the same demographics. And I think this is also the strength of the company, as you were kind of thinking about where do you want to start to move the needle? You really were committed to working in public schools, right, with high needs learners. And we didn't want to be looking at private schools or very exceptional kinds of settings for this research. And so that became really important too, as we started thinking about this. So we shored up that methodology. We had a survey battery to kind of go out. We kind of pre- tested folks at the beginning in November at some point. And then the holidays happen, which is a high stress time of year as we all know, right. And then we kind of thought post- test would kind of come back in January or February and along comes Omicron outbreak. And it was probably one of the most stressful time periods in educational history, maybe even topped the initial outbreak in 2020. So how crazy is it to go back in it and do... seven weeks later and do a post- test and expect to see any reductions in stress, right. But it's also the perfect experiment, right? So is this a device that only helps in the best of times, or is this a device that helps even in the worst of times? Because I think we've all kind of can grow to expect now that we have to expect the unexpected. So how do we future proof our classrooms and really give teachers the support? We've got educators and nurses. A lot of these are the frontline workers that are exposed every day to the pandemic. You're rippling. So and so is out and I got to get homework ready. All of these things that are just accumulating. So how do we leverage AI and technology to start to tinker at some of the... at the edges of some of these problems, right? So we were over the moon when we got these results back, right. I mean, as a researcher, the data's just going to say what it's going to say, right. And especially this early, we're not even sure if we have the right instruments yet. Sometimes you got to worry about that. We're not sure if the pandemic just threw us for a loop. But what we found was that close to two thirds of the teachers reported a reduction, this is in the Symphony Classrooms, a reduction of techno stress in that seven week period. And then that inaudible.

Levi: So that's in the classrooms that are teachers using Merlyn, the digital assistant to help them, yes. Keep going.

Dr. Kylie Peppler: Yeah. So teachers using Merlyn, and techno stress is this large kind of umbrella and there's different kind of areas within that. And so we actually saw significant declines in techno stress across a variety of areas in that short period, and significantly more so than what you're seeing in your control group as well. Emily, you want to talk a little bit more about why you think that might have... What were some of the key features of the product that probably started to contribute to that techno stress reduction?

Dr. Emily Schindler: At one of the sites where we did, we also followed up with some qualitative observations and interviewing. When we installed or by we, when Merlyn Mind installed the Symphony Classroom in these spaces, what we found was that in order to switch inputs between peripherals in this particular setup, they had to open a whiteboard and go. The switch to do that was behind a whiteboard. And so what it meant was that, and I interviewed all of the participants, and every single one of them said," Well, if I can't configure my technology to be accessible through," they were running it through a doc cam, basically, using their doc cam as their hub for all of their input switching." If I can't do it through the doc cam, I don't use it." And you think about that, it's like you're missing just a huge amount of capability there by having just a random placement of the input switch behind several layers of actual wall that you had to go and get it out. And so just that small thing, which just you think about that decision to put that input panel there, right? That was made years and years ago by someone who was really far removed from actual teaching and learning. And what Symphony Classroom did is it allowed them to remove all of those steps, going to the wall, opening it, finding the right button to push, waiting for the HDMI input to pick up and then going back to your computer. And by the way, what was I even doing? I mean, in that time, that's when students are off task, that's when students are like what is going on. And so I watched teachers eliminate that challenge from their day. And the quotes that they come back with are like," I would recommend this, it saves me milliseconds and that is so important." One of the quotes was like," I would recommend Symphony Classroom. We need it." Every single person that I interviewed and said," Please do not take this from me," which I was like," Are you sure?" As a researcher, that's a suspicious result there. But even upon my second observation, it was really clear that after seven weeks, even of the study, they were... the device was part of their practice.

Dr. Kylie Peppler: Yeah. So Emily, in that example you gave like, so if they wanted to switch to something that was on the whiteboard or the interactive whiteboard, what would they have to do with Merlyn?

Dr. Emily Schindler: All they have to do is have remote. They don't even have to say, hey Merlyn with the remote. They just say, switch to HDMI 1. And there's a new update where you can say, switch to my laptop. You can name your HDMI sources. And so it's getting easier and easier all the time. I mean, and it was, we're talking about probably a two minute task as opposed to three seconds with Symphony Classroom.

Dr. Kylie Peppler: And you know how it is. We've all been on those Zoom calls and you can't get to yourself unmuted or whatever it is. And even if it's 20 seconds, the stress load that's there and the distraction, you're not thinking about what Johnny, the important question that Johnny asked you 20 seconds ago. You're just sweating bullets to try to make this lesson happen before the end of the period and before these other things. Because you've got other things that are then waiting. So just compiles on that stress, especially when you have large numbers of young minds to sort of waiting for that to happen. And so when we say techno stress, it's really just that amount of fluidity in the environment, the ability to kind of have a train of thought and just the exciting things you could do with that time save.

Levi: And so you mentioned, we actually saw real statistical evidence that this saves time and that it decreases stress. I think it was in the report it says 14% decrease in techno stress after just seven weeks. Would you expect to see that with a new product that early? Why is that unusual?

Dr. Kylie Peppler: No. I mean I was kind of prepared, especially when we picked this instrumentation to kind of be like, well, let's test them again in another seven weeks and the pandemic will be over, they'll be used to using a product. So you're looking at a product that has a quick adoption rate, right? And so a lot of products are kind of like stick with us a year or two. And then when I started using Canvas, for example, the time savings was not in the first year of use. I had to build out my class, I've got to do these things. It definitely wasn't in the first year that I built my online class. But in subsequent re- administration, now I just mod this, right. And so it saves me a lot of time unless I have to go for a new build, right. So that there is frequently with innovative new products, there is frequently a large startup cost. And you saw that with smart boards, you saw that with... Lots of it, they require a lot of professional development and I'm talking days not hours, right? Repeated kind of upskilling that a lot of times people are using these parts of the product and not moving over here. So part of the AI technology, part of what you've built is an intuitive technology, right. And that, from an HCI standpoint, that's what we're looking for, right. We're looking for something that does that, but you wouldn't see these kinds of findings, especially in this small of a period. And we haven't started to look at that dosage effect, but it may even happen earlier, right. We might been able to go in within two weeks and seeing that techno stress decrease. And we don't know what will happen in a year's time or two years time, once you really, you know all of the different functions that you might not even be using to be able to save you time in other ways as well.

Levi: Okay. So our first question or research question was, can you decrease stress if you have a digital assistant helping a teacher, making it easier for them to use technology? And we were shocked, I guess, at how true that was, right?

Dr. Kylie Peppler: Yes.

Dr. Emily Schindler: Yeah.

Dr. Kylie Peppler: Yeah. Absolutely.

Levi: How do you explain that? Why do you think that is? What is it about AI and a digital assistant that just kind of is intuitive, that fits into the way teachers work today?

Dr. Kylie Peppler: It's so humanizing to have a backup, somebody to be helping you in, and I personify Merlyn here, but to have an entity that's really there with you in the classroom, right? That's not coming in or coming out or training you to do what you're doing differently, but really to do the things that you were hoping to accomplish with ease and insight. And we haven't talked about some of the other features, but the timer feature for example, is very popular. And just being able to kind of set a time to kind of bring the group back together so that you're not having to mark time. Especially as a teacher, you might have to respond to the principal. You might have to go pull Maria out of the classroom to go get checked for COVID. You don't realize that you only meant to give them three minutes, but 10 minutes had lapsed. And so just having a partner in crime I think is just really important. And Emily, what are some of the other things that you were starting to see?

Dr. Emily Schindler: I think my bias is towards project based and student centered pedagogy. So what I noticed is that when you could reduce the cognitive load, what we saw in the qualitative data is that teachers started to wonder what else was possible. And so that's a really great indicator that they're about to make a big switch in their teaching because once a teacher gets curious about something, like people who become teachers are obsessed with learning. And that's really what drives them. And so what you want is for a teacher to feel like they are constantly looking at the next horizon that they're going to go after. We saw that pretty consistently in the qualitative data as well, that once teachers felt like a couple of tasks or just a tiny reduction in complication in their day was enough for them to say, well, I wonder what else I could do with the device and I also wonder what else is possible for my students? Might I be able to make a switch from more teacher centered pedagogies and kind of tap down presenter, assessor type things to using the tools to facilitate authentic inquiry with my students?

Dr. Kylie Peppler: Yeah. I think the quantitative data also backs us up too, but one thing we haven't touched on that's huge we can talk about professional development a lot. And oftentimes it comes down to lead the desk, lead the sage on the stage model and become the guide by the side, right. And that's easy to say, right. But if your technology does not support that movement, you're arrested to the front of the room and you're leaving those likely kids in the back of the room to their own devices, and to feel like they're not really invited into the classroom community and to be off task and to not notice. And you're not looking at what the kids are doing on the computer, or kind of noticing kind of maybe the unique insight and being able to build on that. Merlyn does not tether them further to the computer, but allows them to use the full space of the classroom and to move around. And that, for the project- based learning, for the kind of things that we know about high quality teaching and learning, that's a huge freedom doesn't afford. And so we can't kind of underscore that enough, but there's some quantitative data I want to talk about too.

Levi: So that kind of takes us into, it saved time. And we can talk about like evidence on just how much time it saved and how impressed for it saved time, but then what they did with that time, right. Because you could do different things with times you saved, right. What did we learn from this? You guys are kind of teasing it now, but maybe just talk about it did save time, that's a big deal, and then what people did with the time.

Dr. Kylie Peppler: Right. Yeah, there was two areas, right. And you can kind of see them. The instrumentation we had is kind of like how do they spend their day? And so roughly folks have kind of looked at four areas. And so you can kind of think about it, a teacher might spend time on administrative tasks, on managing the technology, on maintaining discipline, right. And so that things don't go awry in the classroom and teaching and learning. And these things, they're all interrelated, right? So when you shift time from one thing and importance of one thing, you kind of free up time. And in general, we want to see more time on teaching and learning. Teachers, administrators, parents, we all want to see more time on that and these other things can then become the distractor that keeps these other things from happening. So what you start to see in the Merlyn portfolio is that huge reductions, a significant reduction in administrative tasks, right. You can imagine just being able to even say, hey, call the front desk or ask the technology folks to come in, or especially because it's working across the classroom, just being able to communicate there. But a significant reduction in administrative task while significant increases are happening in teaching and learning. And so at the same time, and remember this is the seven week during this pandemic period, you're seeing your control teachers spend significantly more time on maintaining discipline and more time being spent on managing technology, right? And you can see that Merlyn had no... The teachers were worried at the beginning like, well, is this going to create discipline problems? Are my kids going to try to run Merlyn? And the answer was no, there was no impact on discipline, right. And even just to minimize the new disciplinary problems that were emerging in these other classrooms, that's quite cool, and no impacts on increased burdens on managing technology, which we hinted at before. But to see it in the data is just really amazing.

Dr. Emily Schindler: Yeah. And just kind of again, back filling with the qualitative data, what you see is that even, I'm a nerd about time, especially because the idea of whose timeline or whose perception of time matters in the classroom and how that comes to be constructed as truth is very interesting to me. I think students often perceive time really differently than teachers do, even though they're experiencing the same thing. And so when you think about measuring time or saving time, right, it complicates that, okay. I know that that is not helpful for the world, but I am at the end of the day a researcher. So all of that to say, I'm really interested in the fact that teachers were reporting the perception of saving time, whether or not I would go in as a researcher and say like, you spent two seconds less than you did the last time I saw you is really sort of irrelevant. Because what I think they're saying is I felt as though I could focus more intently on teaching and learning, rather than administrative tasks, rather than managing technology and maintaining discipline. I perceived that because we got more of the things related to teaching and learning done during the class time when I had Symphony Classroom there. And that's whether or not it is actually saving minutes and seconds, isn't the issue. The issue is whether or not you feel like you have the brain space to do what you know is right for kids.

Dr. Kylie Peppler: Yeah. And I think it might also ask what are they reporting spending more time on, right? And so we're seeing some really significant impacts highly, highly significant impacts on, that they're presenting tasks for which there is no obvious solution, right. And so that in November they were presenting more tasks with obvious solutions because that's easier, right. And then we're just all going to learn how to do this math problem this predictable way. And seven weeks later, they're able to do some of these things that are unbounded, right? They felt like they had time and resourcefulness to do that. They're also reporting that they ask students to decide on their or significantly more often to decide on their own procedures for solving complex tasks. So not telling kids what to work on and how to work on it, but really kind of opening it up for the complexity of learning and the processes to emerge. So really much higher quality strategies around teaching and learning. And then lastly, you're also kind of seeing them say that they're allowing students to use the ICT, the information and communication tools like laptops for projects or class work. And so you see lots of evidence that the teachers are using this time to do the things that they weren't able to do previously. And you hear this all the time from teachers is that, yeah, that would be great. I would love to offer this project. I would love to teach in this way, I just don't have time. I've got to make sure that they perform well in the standardized test and I've got to teach to what I know that they need to be tested on.

Levi: So one of the fundamental beliefs of Merlyn Mind, our company, was that teachers are amazing and teachers can teach. And if we get other stuff out of their way, more students are going to learn more and be more successful in the world. Our ambitions are pretty big, it's going to be a better place. We can help the whole world learn more, do more, be better. So what you're saying is, even though we weren't really looking for that as the outcome of this project, we're already seeing initial findings that if you free a teacher up and give them more time and make it easier for them to administer and teach, they can teach in a higher level than the kind of way they want to teach. Is that what you're saying?

Dr. Kylie Peppler: Absolutely.

Dr. Emily Schindler: Yes.

Dr. Kylie Peppler: Absolutely. And I mean that, again, it's such an audacious thing that when we met in 2019, I was like, hey. But really, it makes sense, right? I mean, you start to see these things and that all of a sudden, if you do feel like you're not constrained on time, you don't have all of these behavior issues, all of these administrative tasks that are just sucking the life out of your school day. Now, all of a sudden, what would you do at that time, right? And getting that time savings back and sort of restoring the humanity to teaching is just so important.

Levi: And what about like is it just one grade level? Is it just one subject? Who does it work for?

Dr. Emily Schindler: I mean, it really isn't about grade level and subject area so much as about teaching style. So if you are interested in student centered pedagogies, then this is going to help you free up your ability to implement that work. I think we can think about applications from kindergarten teacher who is using Symphony Classroom to help mark time for reading and guided reading groups. We can think about a high school science teacher who is doing labs and can't be touching their computer, but needs to be able to use the tech for other purposes.

Dr. Kylie Peppler: inaudible pull up that information inaudible.

Dr. Emily Schindler: Music teacher. Yeah, absolutely. I mean, it really is more about enabling hands on learning in learning environment. So if that can work for any space, pre- K through adults.

Levi: And we saw that in the research, correct? We looked across K to 12, across subjects and the findings were similar. You're saving time, you're decreasing stress. It doesn't matter what you're teaching, how old your students are, this is helpful.

Dr. Kylie Peppler: Yeah. We thought it'd be outset. This is in 2020. We thought we might be able to help you hone a product market fit of like is this for late elementary, is this better for middle school, high schoolers. And is this better for your techno savvy teachers or your less techno savvy teachers? Is there a kind of a floor or a ceiling that we can advise? And the answer was no, we did not see any significant clustering. We saw teachers as happy in the art classroom as they were in the science classroom, in kindergarten as well as sixth grade, right. So I think you're talking about a general use technology that has wide application. So we are really happy. I mean, again, these things that we hope for, we can't determine as the researcher in the situation, but we hope that the design will hold up like this. We get really excited to partner ourselves with technology that doesn't have that constraints. We don't want to have Merlyn only in classrooms and the late elementary. And then all of a sudden, the teachers and the students, it no longer works for across that divide. So we were really delighted that the data really spoke to that, and the teachers testified to that as well.

Levi: You mentioned there a comment about like, maybe it would only work for the technology proficient teachers and maybe not the other ones. Did we see any evidence in the research that teachers that maybe are late adopters are afraid of technology also benefited from Merlyn?

Dr. Emily Schindler: Yeah. So one of my favorite stories from this research is of a teacher I call Luis, who upon interview said," I actually don't want to do this. I don't want to participate in this trial, but the rest of my team, my fifth grade team really wanted to. And so I'm going to do it." And he ended up being randomized into the treatment group. So he was one of the first receivers of Symphony Classroom. But what he said is that he, first of all, it satisfied his desire to participate with his team, which is really wonderful. And second of all, he described himself really as a very reluctant user in sorts of technologies that he was talking about using pre and during the pandemic, he said," I would really use a Google Doc once in a while, but that was about it." And he even described during the pandemic that there was a time where he was teaching virtually where he was... He had been teaching for 20 years and he said," I was afraid I wasn't going to be able to do my job anymore, because of that." So we would characterize him as evasive towards technology, right. But what we saw is that even as an evasive user, he was able to, first of all, approach use with Symphony Classroom. He undertook it and he turned it into this really interesting learning opportunity for him and his students, where his students helped him to co- negotiate the technology in the classroom. And it turned into the space where that doesn't really happen very often in classrooms, where the students got to share authentic expertise with their teacher. And in doing that, he was actually modeling, learning more authentically than he might have otherwise done, right, because most of the time teachers just teach things they already know. And so for him to take that risk, to model that risk in front of his students, and then use his own vulnerability and be supported by Symphony Classroom while he was undertaking that, I mean, that's magic, that's the holy grail, so.

Levi: What a cool story.

Dr. Kylie Peppler: Yeah. And he wasn't alone, right. We also saw-

Levi: Oh, no.

Dr. Kylie Peppler: ...significant changes. There's a widespread fear amongst educators that they're going to be replaced by technology, replaced by AI, something that's going to look like flashcards or school of one, right. And that they're going to be outmoded by technology. And what we saw actually in this study is that fear took a significant decline, right. And so we saw over time that they felt like they weren't going to be replaced. They did not feel like it was too complex to understand. So what a great kind of gateway into understanding what the positive role of AI could be in the future of the classroom. And again, it's got to be in the right hands of somebody who doesn't want to use AI to replace a teacher, right. And understands the intelligence that a teacher brings to teaching and learning. But really thinking about augmenting the teacher and not replacing them. And so I think that this is I think a really strong example. And why the field needs to be led by exemplary products and models in this way. So that becomes the paradigm, not something that then replaces the human in the model, right. But it shows us how we can augment and how we can actually help teachers to achieve what they would like to see and really help them to be at their best in the classroom.

Levi: Oh, I love that. That is our ambition. We believe teachers can change the world and we're going to help them do it. So, one thing that we didn't touch on and we're about done here. One of the few last questions, you talked about how much impact this can have in the classroom, decreases stress, saves teachers time, allows them now to maybe teach with higher level teaching practices and approaches and pedagogy. Does it also potentially help them outside of the home with all of the stress and workload? Can it help them holistically as a human? Which sounds like a very audacious thing. It could do everything for you, which is not what we're saying, but what came out of the research here?

Dr. Kylie Peppler: Right. Yeah. I mean, again as a researcher and you're just seeing this data, you're just like, wow, okay. This is just amazing, but yeah, I mean, that's exactly what we started to see is that the reports on, again, techno stress is that if you're stressed about what happens in the next day is that you're going to prep a lot the night before. And so a lot of the items within the techno stress measure started to say things like I... They felt like their personal life was being less invaded by classroom technology. We were starting to see this significant impact beyond the school day. Again, who would be that crazy to throw in a question like that. So that was just amazing. We wouldn't even hypothesize, but it makes sense, right? If you feel like what we're seeing is that they're able to improvise in the classroom, right? They're able to respond to things. They're able to tackle harder things, right? The things they've always been wanting, and they're spending less time on the administrative work, which is having... When a teacher gets ready for class, you have all of your tabs open, you've predetermined where the conversation is going to go, right? You have these very tight windows of moving between things. And then you kind of get thrown a couple loops, either from an oddball question coming from the kids, or your administration kind of comes over the loud speaker and asks for something, right. And so you're constantly juggling that kind of thing. And so a lot of teachers spend a lot of time prepping for the school day before or after school in their break. I mean, their lunches are short. They really optimize. And so if you feel like things are going to succeed, you don't need to do all of that prep time, right, and really start to hyper- engineer the school day. But that also that engineering, what do you... I mean, then you're going to take your best guesses and especially for seasoned teachers that have been there, you kind of know where things are going to go. But all of that, a priority decision making is not what we'd like to see in teachers. We want to see them in real time responding to what the kids are noticing. What's kind of emerging? Being really present with the kids. And so that's part of what we're seeing that shift of really kind of being able to be responsive in the moment.

Levi: Wow. So let's end with kind of one last question then. So we're very excited about what we've learned from this, because we've always believed that this technology belongs in every classroom in the world. We believe every teacher would benefit from having Merlyn, a digital assistant to help them as they teach, that would free them up. It would allow them to have more impact on their students. And this is great for all of us and it opens up opportunities to change and improve and do lots of things in the classroom maybe we didn't think was possible before. But we're obviously very biased, right? We've devoted our lives to this, we've been spending years and years doing this. We'd love to have kind of your perspective as independent outside researchers who lived through this with us for a few years. What do you think about the future of this technology, of digital assistance in the classroom, of how it can help teachers and what do you hope comes out of this going forward?

Dr. Kylie Peppler: Oh, yes. I mean, it is fun to be part of this chapter, right. Sometimes we play this game of like imagining five to 10 years ahead. And this is still in its infancy, right. But again, when you have a base product and you've got that kind of strong market fit, you've got a real value add, right, that's not just... It's not something that you're trying to sell teachers, but really meets them with where their needs are, right? I really believe that the company will be highly successful, right? Everything that we're seeing right now is that you've got that fit happening, that you're ready for that scale. And it's not just for teachers too. I think all of the tech companies, teachers are able to use more of that technology that has already been sold and is sitting there in the classrooms and are more in an adventurous spirit to embark on it. We saw this year, a lot of things from engineering to computer science, getting pushed out because teachers don't feel like they have the time for it. And so they're doubling down on the three Rs and maybe even more just reading and writing to be specific and moving this to the periphery. So there's a lot of parties that will benefit from really rethinking what Merlyn Mind could be doing in the classroom in the future. So thank you and thank you for making this audacious idea a reality.

Levi: We have a great team, great founders, a lot of people that have brought it here. Emily, any kind of thoughts on your end?

Dr. Emily Schindler: Yeah. I think anything that you can do to go after the administrative burden that comes along with following student interest and undertaking project- based learning in classrooms is the ticket. It's the missing piece for us moving forward, I think, in our classrooms, but also in our workspaces where we are in this space where we want to... We have so many new technologies. The world is new every single day when we wake up. We want to be able to negotiate those and figure those out in real time, and this allows teachers to do that with their students. So it changes the power dynamics of whose knowledge counts in the classroom. It changes what it means to be a technology- enhanced teacher and how you learn the technology in front of your students. So it's not just the product, and I would even say, it's your way of working? It's your development that you undertook and will continue to, I think, moving forward and teacher, our data says that teachers saw that, that it mattered. This was the first time for some of them they'd ever been asked about what they need. So keep it up. Do it over and over again. I hope it changes everything.

Levi: Well, I want to say from the whole Merlyn Mind team, all of our current customers and teachers working with, and hopefully every future person that benefits from Merlyn, thank you so much to Dr. Peppler, Dr. Schindler to the Digital Promise team, to all the schools that participated in the research study, because you are our partners. And it was a phenomenal journey to work in the dirt together. We were getting dirty and working through this and working through problems. And the whole time we felt, I think mutually, we all care about the outcome that this could have, that we really could help teachers. We really could then help students. And hopefully, we see evidence of that. I don't think any of us expected to see so much evidence so soon. For us, it's thrilling and we wouldn't have been able to do this without you guys, so all of the wonderful people that you brought to bear on this journey that we're on. So we very, very much appreciate this and we hope other startups can benefit just as much working with incredible researchers like you all.

Dr. Kylie Peppler: Yes. I hope we can share the learnings in terms of just these new ways of working. So yeah, thank you for inviting us to be part of this journey. Yeah, there was a lot of trust in our team and we really appreciate that. So thank you.

Levi: All right.

Dr. Emily Schindler: I loved it. I learned a lot. Thank you.

Levi: Until our next research project together, here we go.

Dr. Emily Schindler: Yes, inaudible.

Levi: Okay. Have a good one. Thank you guys. Thank you for joining us for this episode of Supervised Learning. Until next time, keep learning.