User feedback, by the numbers

Posted by on Oct 5, 2013 in Process, User experience


People can surprise you: they find hidden gems you never noticed, point out problems you were blind to, and offer up left-field ideas that spark new directions. They have nothing to gain by sugarcoating their feedback, and while it’s not always easy to hear, it’s necessary.

We began the process of rebuilding TED.com with weeks of internal brainstorming, planning and soul-searching. From there, we were off and running to seek out what users — old and new — had to say. And, once the design process began, we needed more opinions to find out: Is what we’re creating working?

Here, a look at our process of gathering input to inform our rebuild, by the numbers:


The TED staff is brimming with ideas. My team has reams of giant notepaper as evidence now: In a weeks-long series of small meetings, we conducted almost 16 hours of brainstorming sessions with teams from all four corners of the TED office, asking for input on what the new TED.com could look like. In our meetings, we saw blue-sky thinking at its finest, as well as a surprising number of cat drawings:

11: The number of TED internal brainstorm sessions
82: TED staffers who participated in brainstorm sessions
990: Minutes spent brainstorming
41: Dream homepage “redesigns” drawn in marker, by staffers in pairs
6: Number of cats drawn by staffers in magic marker in their visions for what TED.com could look like.


We dove into our treasure trove of advice left by TED.com commenters and people who’d sent emails to customer support. We sent out a survey to TED community members, including TEDx hosts, volunteer translators, and TED Fellows to hear their wishes and criticisms. And we sought the feedback of our online community with a survey on TED.com itself.

93,742: Emails written to TED customer support in the past 365 days
641: Feedback emails containing specific suggestions for redesigning the site
746: “How do I find ______?” inquiries, which helped us identify where our current site navigation could use improvement
250: People who independently suggested some kind of “watch later” feature
80: People who wrote simply to ask us to turn off autoplay in our video player
3,696: Total survey responses


User testing

Staff feedback is great, but our team knows what all good design teams know (and which was confirmed by a brave cadre of new TED staffers who participated in an on-the-fly card sort, which infamously produced exactly the same confusing navigation bar we already had): Once you’re on the inside, it’s too easy to wear organizational blinders – to show favoritism for the work you do. So rounds and rounds of usability testing with outside testers were critical to help us make sure what we were doing made sense. Luckily, many people — both TED fans and even a few brand-new to the site — came out in full force to help:

23: Rounds of usability testing
8: Rounds of talk page testing
2: Rounds of identity service, member profile, and dashboard testing
11: Record-breaking rounds of navigation testing
110: Total usability testing participants
6: Conference rooms dominated in the process
110: Glasses of water, tea or coffee offered to testers
4: Glasses of water, tea or coffee accepted
36: Mighty Wallets given out as thank-yous for user testing. Also 15 tote bags, 25 notebooks, 10 gift cards and 8 sets of Bananagrams
2: Testers who were just minding their own business, walking around the neighborhood, but were game to come up and try anyway.

hello_haleyHaley Hoffman is TED’s product development analyst.

She likes numbers and words. Not equally, but she’ll never tell them that.