How to analyse video data
One of my favourite user research tools to use as a PM is Hotjar. It gives you fantastic screen recordings of your user’s activities, identifies behaviour such as rage-clicking and generally gives you a window into their mindset when they’re using your product. There’s no better tool to remind yourself that ‘you are not your target audience’! However, it can be difficult to know where to start when you’re faced with mountains of video data which cover your entire site or app. It’s our job to derive actionable insights from this research, and it’s a surprisingly hard one to get your teeth into.
Two caveats before I jump into my tips for carrying out this type of analysis: firstly, this advice can apply no matter what screen recording tool you’re using. I’ve heard great things about Fullstory and other competitors, it just so happens that Hotjar is the tool I’ve used most in my career. Secondly, this doesn’t apply to formal user research which uses either moderated or unmoderated video recording to capture insights about specific questions - these tips are designed to help you sift through unstructured video data. I’m hoping to do a collab with a UX Research colleague soon, to think through how to get the most out of working with your research friends.
With that said, here are my top tips for working with unstructured video data.
Have a specific goal in mind
When you’re faced with potentially hundreds of hours of video data to watch, your goal is to prioritise your own time to get the best insights possible. This really helps if you have a particular question that you’re trying to answer, because you can just watch the video sections which relate to that topic. My most effective example of this was whilst trying to improve the Date of Birth input field on a particular sign-up form. I’d watched several colleagues, including engineers and senior delivery managers, struggle with this section of the form, because their eyes kept glancing over it. I wanted to use video recordings to validate whether this was happening with other users as well. It turned out that it was, because the DOB field was formatted as a drop-down whilst all the other fields were empty text boxes - the drop-down with hint text looked like a completed field when someone was quickly scanning the page. Knowing that I had narrowed down my question not just to the specific page, but to the specific field, meant that I could be very strategic about how I was spending my video-watching time.
Don’t look at a long flow all at once
It’s really easy to look at rich, contextual data like screen recordings and want to extract every ounce of value at once. However, this is a sure-fire way to misdirection, as you’ll end up watching one or two flows all the way through, writing down everything you see, and then stopping because you’re tired. This is not the goal of using high-volume video data; you want to watch as many different people interact with your product as possible to increase your confidence in the conclusions you’re drawing. I once watched a senior leader go into my video notes spreadsheet and write really extensive notes next to the top two videos before giving up - this is the wrong way to do it! It’s much better to spend an hour watching 2mins each from 60 videos, than spend that same hour watching 20mins of flow but only making it through 3 videos. The great strength of these tools is the scale at which they can deliver insights - don’t sacrifice that with your analysis.
Categorise, categorise, categorise
When you’re watching screen recordings, your goal is to turn unstructured data into structured data from which you can draw insights. It really helps to have a set of questions with categorical answers that you can populate as you watch. This means that you’ll be able to extract quantifiable insights from the most qualitative data of all, and look at long term trends as well as a snapshot in time. For example, when I was examining how people interacted with the Date of Birth field, I had a simple Boolean question (Yes/No) which asked, ‘Did this user miss the field?’ By using this categorical format, I was able to work out that of the 100+ recordings I watched, over 40% of users scrolled past the problematic field. It can be hard to restrict yourself to categorical data, and, by all means, take notes on things that you find interesting - but this is a really effective way of honing your focus to make sure that all your video-watching time gives you valuable insights.
Use the features the tool gives you
You don’t have to come up with these categorical variables all on your own - the recording tools will give you great insights that you can use in combination. Things like entry page, exit page, time on site, rage clicks are all useful features which are recorded automatically. I didn’t have to do any additional analysis to work out that of the people who skipped my tricky Date of Birth field, a significant percentage of them dropped out on that page because they couldn’t progress - a HUGE insight for a sign-up flow. I also watched some really painful videos where users struggled with a field for over 15 minutes - this would have been even more torturous if it weren’t for the playback speed button. These tools are here to make your life easier; make sure you take advantage of those features.
Take regular breaks
How many people have sat down to a lecture at university, or an online class, and taken really excellent notes for the first ten minutes before drifting off? It’s really hard to stay observant during these types of scenarios, especially when you’re watching the same thing over and over again. This is very normal and human, so do make sure you give yourself plenty of breaks, and potentially even spread your watch time over multiple days to make sure the quality of your analysis stays high.
I hope you find these suggestions useful - personally, I found this type of analysis a struggle when I first started as it can be difficult to find a way in. Let me know in the comments if there are any other tricky areas of the PM life that you’d love to see a deep-dive on!