Survey Research & Your Master’s: Best Practices for Grad Students

1. What are some of the general best practices to follow for designing effective survey questions?

I think good survey questions start with establishing a clear understanding of your research objectives. This sounds straightforward, but in my experience translating my high-level research question into specific topics and components was challenging and took more time than I had anticipated. However, investing in this area paid dividends later on because I had a solid reference for when I developed my survey questions to make sure they were relevant and focused.

In a similar vein, it’s important to refine your questions so they are as simple and concise as possible. Even for open ended questions, it’s important to have focused questions for a variety of reasons. It makes it easier for your respondents to understand, it leads to higher quality and relevant answers, and it makes analysis more straightforward. In my research, I used a mix of closed ended and open-ended questions, using the latter in a more targeted manner to help add context to topics I was asking about with my closed ended questions. This was helpful not just for drawing conclusions in my research, but provided strong quotes that helped bring my quantitative results to life in the paper.

The structure of the questions also matters and arranging your questions in a logical order helps respondents complete the survey quickly and also impacts how your respondents answer questions. In my research, I had several topics I wanted to cover and there was a natural order in how one topic led to the other as I was structuring my research plan and I made sure the finished survey reflected this. Within each subsection it’s also useful to consider the individual order of questions. I found it helpful to start with more general questions and then follow up with more specific questions. This also makes it easier to use features like branching logic or conditional questions in your survey platform. Those features are useful for preventing respondents from answering questions that are not relevant to their experiences which helps with answer quality and completion rate.

2. Based on your experience, what are some of the common pitfalls to avoid when conducting survey research?

I think one of the most significant things to look out for is sampling bias. Sampling bias is when the people who responded to your survey don’t actually represent the population you hoped to study. This is a really challenging aspect of survey research because completing surveys takes effort and it can be hard to convince people that it’s worth their time. There may be certain groups that are more likely to participate than others and it can be easy to leave people out. For this reason, it’s really important to spend an adequate amount of time on recruiting respondents, making sure your survey is straightforward and easy to understand, and building a survey that is as short as possible while adequately capturing the information you need for your analysis.

Failing to pre-test your survey is another common mistake researchers make. Even if you feel like you have a strong understanding of the people you are surveying (or are even part of that cohort yourself), sharing your survey with a few people in your study population before you actually conduct research is extremely helpful for catching issues like poorly worded questions, confusing instructions or technical errors. It’s so much easier to fix those problems before you send out your survey for real.

It’s also helpful to have at least one research expert review your questions for common issues and biases that may negatively impact your research. This could come from an advisor, faculty member, or even a peer. Again, doing this review in advance is vastly preferable to trying to fix issues after you’ve conducted your survey. At Georgetown, we had research experts in our library staff that were very helpful resources on the technical side of things.

3. How do you ensure the validity and reliability of survey data?

First and foremost, I think it’s very helpful to build your survey around tried-and-true research methods that are in common use. Looking at the research methodologies used in similar research can provide an extremely helpful starting point for your own research and help ensure that you’re following best practices and are producing good quality research. You’ll already be reviewing prior art in your literature review, so it’s easy to pay some extra attention to methodologies and think about what strategies seem to have worked well for researchers and what their benefits and limitations are and how those might apply to your research. Especially when it comes to quantitative analysis, it’s important to avoid reinventing the wheel and be conservative about your methodology choices.

Along the same lines, even though a good survey may use multiple types of questions and measurements, it is helpful to avoid too much variation to ensure there is some overall consistency in your survey questions. This is especially helpful if you are exploring multiple dimensions of a topic and wish to compare and contrast results between different topics or questions. It is much easier to make comparisons when the format and measurement methodology are the same.

4. What are some of the ethical considerations you've encountered when conducting survey research?

My research has been focused on the Emergency Management and Public Safety space and not only deals with some sensitive topics around emergency response, but also deals with people who may not be able to talk about some aspects of their work in certain contexts. Being very clear about the purpose of the survey and my research was important to building trust with my respondents. It’s useful (when you are able) to allow respondents to answer your survey anonymously. It’s also important to understand the specific ethical guidelines and rules you’re operating within. Your academic program or institution likely has their own policies and procedures, and in some cases, you may need to present your research plan to a review board. Understanding these requirements at the start of your research is an important logistical step as well as crucial for ensuring you are conducting research ethically.

5. Do you have any tips for encouraging higher survey response rates?

Two things have worked well for me. First, when you are conducting outreach to recruit participants it’s very helpful that your communications are clear and make a compelling case for why responding to the survey is important. While you want to avoid being overly wordy, it increases response rates when respondents feel some connection to the research. Second, all things being equal, shorter surveys have a higher likelihood of being completed.

Expect to go through an iterative process as you develop your question set where you tweak and remove questions that are not essential to your research question. Many online platforms will give you an estimate of how long your survey will take to complete on average, use this along with feedback from pre-testers to determine if your survey is too long. Like I mentioned earlier, it’s also helpful to use branching logic and/or conditional questions to make sure your respondents only need to respond to questions that are relevant to them.

6. How can students use survey data to draw strong conclusions and make meaningful recommendations?

An important part of building your survey is determining the types of data analysis methods you plan to use. These are key to being able to actually draw conclusions from your data. Even simple analysis like calculating frequencies, percentages and averages can be helpful in drawing meaningful information out of your raw data. Similarly, identifying patterns, trends and relationships in the data is crucial to exploring your research question. Using visual aids like charts and graphs is usually extremely helpful for presenting your analysis clearly and illustrating your findings in a more impactful way to your audience.

7. What are some techniques you used to report survey results in a clear and meaningful way in your capstone project?

I used a lot of visual aids in my paper and presentations. Especially when illustrating things like trends and comparisons, showing charts and graphs was a more impactful way of conveying my results compared to simply recounting numerical results. When doing data visualization, it is important to keep in mind basic best practices to ensure you’re illustrating your data in a clear and concise way that doesn’t overwhelm or confuse your audience. In my charts and graphs, I used simple color schemes and selected formats that were appropriate to the data and analysis I was depicting. I also reviewed my graphics to ensure they were relevant and did indeed help illustrate my findings and enhance my work rather than duplicate words already on the page.

8. How did you adapt traditional survey methodologies to your study of emergency management practitioners' use of social media?

For my specific research questions, there wasn’t a lot of prior study so my focus was on conducting foundational research. That meant that I could use fairly simple survey and analysis methods and still draw useful conclusions from my data. I looked at other survey-based studies and looked at how they structured their research, identified their study population and built their surveys and used this information, along with established best practices, to develop my own research. My primary survey consisted of 40 closed ended questions grouped into six thematically distinct sections and a set of screener questions to confirm respondents were in my target study population. In the screener section, I gave respondents the choice to answer anonymously or identify themselves and also asked for their consent for a follow up qualitative survey conducted via phone to explore my area of study in further detail.

9. Did you face any unique challenges as a part of designing or conducting your survey? How did you overcome them?

Recruiting participants was very challenging. My goal was to include a representative sample of emergency management practitioners at all levels of government including federal, state, tribal, and local organizations. I was only able to achieve this goal through a significant amount of research to identify and validate potential respondents and I spent a lot of time building my outreach email lists. I also identified specific people that could themselves identify and recruit qualified respondents and used them to increase my overall reach. Overall respondent recruitment took a lot more time and effort than I had originally anticipated, but it was critical to ensuring I had the data I needed to adequately explore my research questions. Leveraging my own network within the Emergency Management industry, and using the resources of my advisor and other faculty helped immensely with the recruiting effort.

10. What advice would you give to researchers looking to study the impact of social media on specialized fields? How can they ensure their research is effective and relevant?

Social media is a unique area of study because it is both an established communication channel while still rapidly changing and evolving. For this reason, it’s important to understand some of the potential limitations of your research and that survey data you collect is really a snapshot in time and attitudes and behaviors are liable to change significantly over time. While this is something to watch out for, it also has some upsides in that social media users have lived through major changes and shifts in technology and usage. If you ask questions about how things have changed and their experiences over time, you can often readily identify clear trends and make strong comparisons that will shed light on your research topic.