User experience (UX) product metrics – such as number of user downloads or clicks on your platform – are a compelling source of data for EdTech companies: they are easy to obtain and give an indication of how folks are engaging with your product. But what if these product metrics don’t paint an accurate picture of how your product is being used? What if relying on only UX product metrics led to erroneous conclusions about your product?
A great example is a recent EdTech research evaluation of an app product that was newly implemented in a middle and high school. The UX product metrics indicated that more students from the high school had downloaded the app compared to students from the middle school. If we took these metrics at face value, we might have concluded that there was something about the middle school that impeded adoption and use of the product.
When we dove into conversations with administrators and students, however, we learned that the UX product metrics were misleading. The culture of technology adoption at the middle school was actually more positive than the high school. We learned even more from talking with the students from the middle school who were surprisingly more familiar with the product and its features, and seemed to use the app much more frequently than students from the high school.
Our research team at Labs learned the importance of understanding the context in which an EdTech product is being used and the value of qualitative data from users. Even the best designed product is constrained by the site it is implemented in – or what we refer to as the “technology adoption culture” of the school an EdTech product is implemented in.
Given the impact of the tech adoption culture on understanding how users engage with a product, here we share three things to consider about a school’s technology adoption culture that may affect how your EdTech product is used:
1 – Can Students Access Your Product During School Day?
If your EdTech product supplements the academic experience outside actual learning instruction, consider whether students can access your product when you need them to. Does effective usage depend on students accessing the product at specific times throughout the day? Is your product reliant on phones or other tech devices?
For example, if your product requires students to check their phones to receive updates or complete tasks, consider whether students are allowed to access their phones or devices during the school day. If the school does not allow access to devices during or between classes, you must consider how this constraint of the research site may impact the interpretation of your UX product metrics compared to another school that permits phone access. UX product metrics must be understood in context.
2 – Is Your Product Duplicating Other Technology at the School?
Schools are implementing more EdTech tools now than ever before, especially in the wake of the virtual learning wave. It’s crucial to consider whether the utility of your product might be dampened because it duplicates aspects of other products being used at the school.
If students have access to multiple EdTech tools or apps that play very similar roles in their learning experience, what about your product stands out from the rest? Will students become fatigued during their decision making process of which product to use for similar purposes? Your UX product metrics alone can’t answer these questions. To increase the likelihood of your EdTech product being used and users gaining unique value from it, thoroughly investigate how your product will coexist with other EdTech already in place.
3 – Does Effective Use of Your Product Rely on Other Tech?
Many EdTech products rely, in part, on other technologies or input from instructors. If effective use of your product depends on, for example, linking to a learning management system or an instructor inputting assignments that link with your product.
It’s necessary to consider how impactful these dependencies are, and how they may skew the interpretation of your use metrics. How reliable are these dependencies over the academic term? If instructor behavior impacts your product use, how are you supporting them in the use of your product in addition to the student users? Without this context, your UX metrics may not show a full or accurate picture of product engagement.
As much as we would like EdTech to always be flawlessly implemented and used, the reality is that these products are not used in a vacuum. Hurdles will always appear that prevent the most effective user engagement. By understanding the realities of the context your product operates in through mixed-methods research – which requires going beyond your UX product metrics – you will be able to more accurately understand how your EdTech product is used in real academic environments.
The Accelerator at WGU Labs leverages the talent of 30-plus education and business professionals to boost your company’s impact and business scale. Interested in learning more? Let’s chat.
