Lessons from user testing

Posted by on Dec 3, 2013 in Lessons learned, User experience

While rebuilding TED.com, we conducted over a dozen rounds of usability testing – sometimes testing multiple parts of the site on the same day. We learned the limits of our teleconferencing system, plus how much bandwidth it takes to run a prototype, two videocameras and our UX designers ragged. As the person tasked with making sure we had lots of users for each testing session, I learned which of my user recruitment methods were more potent for each type of test, as well as how to spot willing participants on the streets of DUMBO, the neighborhood in Brooklyn where our design agency was located.

Here are some other things we learned, listening to our users.

1. Video is the center of attention.

Of course this shouldn’t come as a surprise. An unusual number of people here at TED remember what it was like to try to create this video format the first time, back in 2007 when Hollywood was young and Youtube had just been launched. Since then, of course, TED has grown a lot — as has the world of online video. We know that our talks are our cornerstone, but it was really interesting to see how tangibly that played out for our users on the TED talk pages.

As we work to provide surrounding content for ideas, the takeaway has been: if ideas or supporting materials are going to come in another form — written articles, photo essays, bibliographies — then the page structure needs to give real context and sense of place.

2. Curation-lovers want to curate.

Releasing playlists was a really exciting moment for us, giving us the ability to bring together related talks. It also created a clamoring from users to create their own playlists. This has been in the pipeline from Day 1 of the project — it is definitely an objective for developing playlists as a whole. But it was fantastic to see such a united request for a tool from so many of our different communities. This is in the works.

3. Autoplay is loud and disconcerting.

We decided to kill autoplay. This decision had everything to do with hearing from users that it startles them in their workplaces and confuses them as their browsers are habitually full of open tabs. A few even noted that TED’s autoplay has turned them into audio technicians, experts in turning down the volume of our opening sequence, or that it keeps them poised in preparation to press “pause.” We hear you. Even over our intro music.

4. No one wants to be a member, if they don’t understand the club.

We knew that our identity service, the foundation of profiles and activity on TED.com, needed work. We built the old thing back in the day, and until now someone like Logan Smalley — a TED-Edster, a TED Fellow, a staff member and an uber use case — had three different profiles on the site. But we were surprised at how confused users got when we asked if they were “a TED.com member” or had “a TED.com profile.” Veteran community members, who regularly log in to monitor their TEDx licenses or translations, weren’t sure if they had a profile — they instead may have called it something else. And brand new users didn’t understand what membership on TED.com might look like. All of this told us that our enthusiasm to encourage people to log into their accounts — to help them keep track of their content, and help us know what they like — couldn’t stick without a more clear model of the kind of members and membership TED.com has available. Adding features like Watch Later and a user Dashboard will certainly go a long way toward clarifying the point of it all. But so will making profiles more dynamic and changing up the (we hope) delightful questions that provoke thought and identification in the TED universe. (Stay tuned for another post about how we’ve been developing profiles.)

5. If you want a volunteer, target those not in a hurry.

Surprisingly, coffee shops nearby were a terrible place to make new testing friends. In DUMBO, everyone is between meetings — mostly in the same three buildings. But I quickly found that, in this tourist-friendly neighborhood, slow walkers and people without bags were often game to try our user testing exercises. My rule became: find someone looking up at the sky, and you’re set.


hello_haleyHaley Hoffman is TED’s product development analyst.

She is undertaking a statistical analysis of the most welcoming coffee shops.