Observing, Recording, and Sharing Usability Studies on Mobile Apps

When doing usability studies on websites or Windows or OSX applications, it’s easy to see what’s going on on-screen since you can just split the video signal to a display in another room and into some recording equipment. However, when doing usability studies on mobile apps, it can even be hard for the moderator to see what’s going on, since mobile device screens are relatively small and participants have a propensity to move them around as they perform their tasks.

For our latest mobile app project, I decided to get creative about how we record and allow remote observers for our sessions.¬†We don’t have a big hardware or software budget for usability equipment, plus, I’m thrifty. ūüôā

I put together a pretty reasonable workflow that worked really well this past week.

Software

  1. AirPlay receiver software; we used AirServer (http://www.airserverapp.com)
  2. Screen recording software; we used ScreenFlow (http://www.telestream.net/screenflow/overview.htm)
  3. Skype (http://skype.com)

Hardware

  1. Computer for the participant room, preferably a laptop w/ a built-in webcam
  2. One of these iOS devices: iPhone 4S / 5 or iPad 3, 4, or Mini
  3. WIFI router to create a private network for the computer and the iOS device to join
  4. Optional computer in an observation room for other people to watch the sessions
  5. Optional external microphone to connect to the computer

Procedure

  1. Install AirServer, which makes a computer act as an AirPlay receiver. We used the trial license this week during our tests, but it’s really reasonably priced.¬†This allows you to send the iPhone’s screen to the computer’s screen. It will also display the view from the computer’s webcam on the computer’s¬†screen. This allows the moderator to see what’s happening on the phone’s screen without having to crane over the participant’s shoulder. We put piece of paper that we put over half the screen to hide the view from the camera so that it doesn’t distract the participant.
  2. Since our corporate network blocks AirPlay, we used a separate wifi router that’s also plugged into the network. Both the computer and the phone are attached to this separate wifi router. This reduces the chances of network problems between the computer and the iOS device.¬†And since the WIFI router is plugged into the network, the participants can still perform tasks that need internet access.
  3. We’re doing screen recording and capturing the audio from the computer’s mic. Since AirServer is displaying the iPhone’s screen and the computer’s camera view on the computer’s screen, we’re getting both of those in the screen capture. We’re using ScreenFlow , but you could likely use any similar product.
  4. To allow remote viewing, we’re using Skype and sharing the computer’s screen. This transmits the audio as well. We have the remote side mute their mic.
It works pretty well. Right now though, we’re only remotely viewing from a few rooms over and the observation computer is plugged into the network. Sometimes there’s some chop in the video.

Tips

  1. During the study, the setup on the computer in the participant room is a bit complex, so make sure to write the steps down and practice.
    1. Start AirServer
    2. Start the Skype connection and make sure the remote side is muted
    3. Start the screen recording software
    4. Mirror the iOS device to the computer
  2. Set the phone to never time out. If the phone is connected to an exchange mail server, policy might force a max 5 minute timeout. We are using a development phone that doesn’t have any email accounts set up. If the phone times out / shuts off its screen, this will cancel the AirPlay mirroring.
  3. Disable all notifications on the phone and shut down all other apps on both computers to reduce the chance of interruptions.
  4. Charge the phone during the session with a cable plugged into the computer
  5. If the computer’s a laptop, make sure that it’s powered.
  6. ScreenFlow captures the screen recording in a proprietary format. An application that uses a standard format would be preferable. On our MacBook Pro 15″, exporting the video to an editable format is 0.75 the video length.
  7. I suggest creating a new screen recording for each participant. The exporting and editing of the larger files that are created when creating recordings w/ multiple participants is really annoying and slow.

Areas for Improvement

  1. Figure out how to do this for Android
  2. Improve the quality of the desktop / audio sharing with the remote computer
  3. Sharing with multiple remote computers; perhaps Skype multi-party chat, perhaps Google Hangouts
  4. Depending on the app or mobile web site you’re testing, you may not get tap targets, so it can be difficult for the observers to know exactly what the participant is doing unless they are very familiar with the test matter.