{"rowid": 13, "title": "Data-driven Design with an Annual Survey", "contents": "Too often, we base designs on assumptions that don\u2019t match customer perspectives. Why? Because the data we need to make informed decisions isn\u2019t available.\n\nImagine starting off the year with a treasure trove of user data that can be filtered, sliced, and diced to inform new UI designs, help you discover where users struggle the most, and expose emerging trends in your customers\u2019 needs that could lead to new features. Why, that would be useful indeed. And it\u2019s easy to obtain by conducting an annual survey.\n\nAnnual surveys may seem as exciting as receiving socks and undies for Christmas, but they\u2019re the gift that keeps on giving all year long (just like fresh socks and undies). I\u2019m not ashamed to admit it: I love surveys! Each time my design research team runs a survey, we learn so much about customer motivations, interests, and behaviors. \n\nSurveys provide an aggregate snapshot of your users that can\u2019t easily be obtained by other research methods, and they can be conducted quickly too. You can build a survey in a few hours, run a pilot test in a day, and have real results streaming in the following day. Speed is essential if design research is going to keep pace with a busy product release schedule. \n\nSurveys are also an invaluable springboard for customer interviews, which provide deep perspectives on user behavior. If you play your cards right as you construct your survey, you can capture a user ID and an email address for each respondent, making it easy to get in touch with customers whose feedback is particularly intriguing. No more recruiting customers for your research via Twitter or through a recruiting company charging a small fortune. You can filter survey responses and isolate the exact customers to talk with in moments, not months.\n\nI love this connected process of sending targeted surveys, filtering the results, and then \u2014 with surgical precision \u2014 selecting just the right customers to interview. Not only is it fast and cheap, but it lets design researchers do quantitative and qualitative research in a coordinated way. Aggregate survey responses help you quantify the perspectives of different user segments, and interviews help you get into the heads of your customers.\n\nAn annual survey can give your team the data needed to make more informed designs in the new year. It all starts with a plan.\n\nPlanning your survey\n\nBefore you start jotting down questions to ask users, spend some time thinking about the work your team will be doing in the coming year. Are you planning new mobile apps or a responsive redesign? Then questions about devices used and behaviors around mobile devices might be in order. Rethinking your content strategy? Then you might want to ask a few questions about how your customers consume content.\n\nYou can\u2019t predict all of the projects you\u2019ll be working on in the coming year, but tuck a couple of sections in your survey about the projects you\u2019re certain about. This will give you the research you need to start new projects with solid foundational data.\n\nGoogle Drive is a great place to start collaboratively building survey questions with colleagues. Questions that seem crystal clear in your head get challenged, refined, or even expanded quickly when the entire team can chime in. \n\nAs you craft your survey, try to consider how you\u2019ll filter it once all of the data is compiled. Do you need to see responses by industry, by age of an account, by devices used, or by size of company? Adding the right filter questions can help you discover fascinating patterns in user segments. Filtering on responses to a few questions can surface insights like: customers in non-profit companies with more than 100 employees are 17% more likely to use an Android phone and are most attracted to features A, D, and F. A designer working on the landing page for a non-profit would love to have concrete information like this. Filter questions are key, so consider them carefully. But don\u2019t go overboard \u2014 too many of them and you\u2019ll start to hurt your survey response rate.\n\nMultiple choice questions are the heart of most surveys because respondents can complete them quickly, which increases response rate, and researchers can analyze them without a lot of manual categorization. Open text field questions are valuable too, but be careful not to add too many to your survey. You\u2019ll hate yourself after the survey\u2019s done and you have to sort through and tag thousands of open responses so patterns become visible. Oy vey!\n\nAn open-ended question works well towards the end of the survey. At this point respondents have a lot of topics swirling around in their head and tend to say weird things that will pique your interest. This is where you\u2019ll find the outliers who are using your product. They\u2019ll be fascinating to interview, and on occasion will help you see your work in a brand new way.\n\nConclude your survey with a question asking permission to get in touch for a followup interview so you don\u2019t pester people who want to be left alone. \n\nWith your questions nailed down, it\u2019s time to build out that survey and get it ready for sending!\n\nBuilding your survey\n\nThere are dozens of apps you could use to build your survey, but SurveyMonkey is the one that I prefer. It lets you pass in variables for each respondent such as user ID and email address. Metadata about respondents is essential if you\u2019re going to do any follow-up interviews with your customers in the coming year. SurveyMonkey also makes it easy to set up question logic, showing questions to customers only if they responded in a certain way to a prior question. This helps you avoid asking irrelevant questions to some respondents.\n\nDetermining survey recipients\n\nOnce you\u2019ve chosen a survey tool and entered all of your questions, you need to gather a list of recipients. Your first instinct will be to send it to everyone. You might say, \u201cI need maximum response and metric shit tons of data!\u201d But this is rarely the best approach \u2014 broad distribution almost always leads to lower response rates, increased noise, and decreased signal in your data. Are there subsets of customers you could send to, like only those who are active, those who are paying, or have been with you for a certain length of time? Talk to the keepers of your customer database and see how they can segment it so you can be certain you\u2019re talking to just the people who will have the most relevant responses for your needs. \n\nIf you want to get super nerdy when finding the right customer sample to survey, use a [sample size calculator]. Sampling is a deep subject best explored in other articles. \n\nCrafting your survey email\n\nAfter focusing your energies on writing and building your survey, the email asking your customers to respond seems almost trivial, but it will greatly influence your response rate. Take great care when writing your subject line and the body of the email. If you can pull it off, A/B testing subject lines can greatly improve the open rate of your email and click-through to your survey. My design research team has seen a ~10% increase in open and click rates when we A/B tested. We\u2019ve found that personalizing subject lines and greetings with the recipients name (ie. \u201cHey, Aarron. How can we make our app work better for you?\u201d) gave us the best response rates. Your mileage may vary.\n\nThe tone of your email is important \u2014 be friendly, honest, and to the point. Those that are passionate about your product will be happy to share their perspective. Writing a survey email that people will actually respond to ain\u2019t easy \u2014 in fact, they\u2019re almost always annoying. But Ben Chestnut found a non-annoying way to send a survey email and improve response rates.\n\nThe email sent for the 2013 MailChimp survey let customers know what we\u2019d been up to in the previous year, and invited feedback on what we should work on in the coming year.\n\nThe link to your survey should be a clear call to action. A big button with a label like \u201cAnswer a few questions\u201d generally does the trick. The URL linking to the survey will need to include some variables like user ID and email. It might look something like this if you\u2019re using SurveyMonkey:\n\nhttp://surveymonkey.com/s/somesurveyid/?uid=*|UID|*&email=*|email|*\n\nAs each email is sent, the proper data will be populated in the variables, passing it on to the survey app for inclusion in each response. This is the magic that will help you pinpoint customers to interview down the road, so take special care to test that all is working before sending to all recipients. How you construct the survey link will vary depending on what survey tool and email service provider you use, so don\u2019t take my example as gospel. You\u2019ll need to read the documentation for your survey and email apps to set things up properly.\n\nPilot before sending\n\nBy now, you\u2019ve whipped yourself into a fever pitch over your brilliant survey and the data you hope to collect. Your finger is on the send button, poised for action, but there\u2019s one very important thing to do before you send to the entire list of customers: send a pilot email. How do you know if your questions are clear, your form logic is sound, and you\u2019re passing variables from the email to the survey properly? You won\u2019t, unless you send to a small segment of your recipients first. \n\nThe data collected in your pilot will make plain where your survey needs refinement. This data won\u2019t be used in your final analysis, as you\u2019re probably going to make a few changes to your questions.\n\nSend the pilot survey to enough people that you can really stress test the clarity of the questions and data you\u2019re gathering, while considering how much data can you comfortably throw out. If you\u2019re sending your final survey to a few thousand people, you might find a couple of hundred recipients for your pilot will give you enough insight into what to improve while leaving the vast majority of the recipients for your final survey.\n\nAfter you\u2019ve sent your pilot, made your survey adjustments, and ensured the variables are being passed from your email into the survey app, you\u2019re ready to send to the remainder of your customers. This is your moment of glory!\n\nAnalyzing your results\n\nAfter a couple of weeks you can probably safely close the survey so no other responses come in as you transition from data gathering to data analysis. Any survey app worth its salt will chart responses to your multiple choice questions. Reviewing these charts is a great place to start your analysis. Is there anything particularly interesting that stands out? Jot down some of your observations. I like to print screenshots of the charts for each question, highlighting areas of interest. These prints become a particularly handy reference point for the next step in your analysis. \n\nPrinting results from a survey makes comparing different customers easy.\n\nViewing aggregate data about all responses is interesting, but the deltas between different types of customers are where the real revelations happen. Remember those filter questions you added to your survey? They\u2019re the tool that\u2019ll help you compare customer segments.\n\nMost survey apps will let you filter the data based on response to a question. If the one you\u2019re using doesn\u2019t, you can always export your data and create pivot tables in Excel. Try filtering your data based on one of your filter questions, such as industry, company size, or devices used. Now compare those printed screenshots of baseline responses to the filtered data. Chances are you\u2019ll see some significant differences in how each group responded to your questions, giving you clues about the variance in interests and motivations in customer segments and a leg up as you work on future design projects. \n\nOpen-ended responses are equally interesting, but much more time-consuming to analyze. Yes, you need to read through thousands of responses, some of which are constructive and some of which are not. Taking the time to tag each open response will help you see trends and filter out the responses that are unhelpful.\n\nUnlike questions with predefined answers, open-ended responses let users express unique ideas and use cases you may not be looking for. The tedium of reading thousands of response is always cut by eureka moments when users tell you something fascinating that changes your perspective on your app. These are the folks you want to pull out for follow-up interviews. Because you\u2019ve already captured their email addresses when you set up your survey and your email, getting in touch will be a piece of cake.\n\nFilter, compare, interview, and summarize; then share your findings with your colleagues. Reports are great for head honchos, but if you want to really inform and inspire, create a video, a poster series, or even a comic to communicate what you\u2019ve learned. Want to get really fancy? Store your survey results in a centrally accessible location so anyone in your company can research and discover the insights they need to make more informed designs. \n\nGood design researchers discover valuable insights. Great design researchers turn those insights into stories.\n\nConclusion\n\nAs we enter the new year, it\u2019s a great time to reflect on the work we\u2019ve done in the past and how we can do better in the future. Without a doubt, designers working with a foundation of insights about customers can make more effective UIs. But designers aren\u2019t the only ones who stand to gain from the data collected in an annual survey\u2014anyone who makes things for or communicates with customers will find themselves empowered to do better work when they know more about the people they serve. The data you collect with your survey is a fantastic holiday gift to your colleagues, one that they\u2019ll appreciate throughout the year.", "year": "2013", "author": "Aarron Walter", "author_slug": "aarronwalter", "published": "2013-12-13T00:00:00+00:00", "url": "https://24ways.org/2013/data-driven-design-with-an-annual-survey/", "topic": "design"}