Evaluating an emerging media experience longitudinally
Client feedback
“Your work on this project has been so wise, thorough, flexible, and engaging. I really hope we pick this back up in the near future, plus you are firmly top of mind for any future R&D work. You’ve delivered beyond expectation.”
The challenge
A national broadcaster wanted to understand how users respond over time to a prototype media player that enables a more personalised viewing experience.
To find this out, an 8 week trial was conducted in which 12 participants were invited to watch a specially produced version of a weekly football highlights programme via the prototype.
Participants were able to change the running order, as well as other elements of the programme, according to their preferences.
What we did
After watching each week’s episode, participants were asked to respond to a recurring set of questions about their viewing experience.
This was supplemented with three rounds of interviews to probe deeper into their responses (after the first week, half-way through, and at the end). By combining a weekly diary with in- depth interviews, we were able to build-up a highly detailed picture of their viewing habits and preferences.
Due to its longitudinal nature, we were mindful that participants were liable to drop out before the end of the study, so every aspect was optimised for engagement. We kicked-off with a webinar to onboard participants to the project, and to underline their importance to the study. We used WhatsApp to administer the diary element as we felt its familiarity and immediacy could help to facilitate relationship-building between participants and researchers. All of the twelve participants that began the project saw it through to its conclusion.
What we found
As the prototype was iterated following each round of interviews, we bracketed our analysis around each iteration. This enabled us to track the impact of individual changes on user sentiment.
At the conclusion of the study, we then aggregated our analysis to arrive at a holistic assessment of the prototype.
We observed an overwhelming preference for the prototype version of the programme over the original because it allowed participants to watch more football highlights in less time, primarily by bypassing ancillary elements such as interviews and analysis.
This was true even for participants who had previously expressed a fondness for those elements of the show, indicating the potential of the prototype to reshape viewing preferences. In our recommendations, we therefore recommended that users are able to select interviews and analysis for individual games (previously it was a blanket opt-out), to balance brevity while still maintaining the central appeal of the format.
The outcome
Based on the success of the trial, the technology was applied to other genres and content formats. However, due to existing contracts with rightsholders, the version of the football highlights programme tested during the trial has yet to emerge.