Discovering what successful email marketing means to small businesses

Summary

My team was presented with the question “How do users track ROI for digital marketing?” However, we learned through our initial research that what mattered most to users went beyond the scope of the original question. Defining success and feeling successful were much more nuanced than originally assumed. We were able to reveal through exploratory research this disconnect as well as blind spots in the organization’s product offerings.

Starting with lots questions

Although the organization wanted to learn something specific, we asked questions to better understand their broader needs. Through our conversation we learned about the organization's broader goal of making their users productive, effective, and efficient.  This helped us understand that they wanted to know how users tracked effectiveness in general, not just ROI.

Building a framework with secondary research

We conducted a literature review to understand the problem space of small businesses and marketing.  We learned about the competing challenges facing small business owners and the number of barriers they face when it comes to marketing (limited time, resources and expertise), which are not unique to the organization's users. This background information helped us to frame our research questions and to eventually design our moderation guide to focus on the user’s broader business goals and activities not just marketing.

Getting to know the users

To better understand the users we listened to recorded customer service calls. These calls shed light on potential areas of exploration as well as provided contextual information about the product landscape, consumer expectations, common pain points, and the customers’ language when talking about the product.  We observed that users talked about their marketing efforts, goals, and tools in a completely different way than the organization presented their services.

Reframing the question

Based on our initial research, we reframed the question to something more meaningful to users: “How do users calculate success?” We needed to determine how users defined and measured marketing success, rather than specific metrics. Answering this question required us to learn how they actually used marketing tools and their expectations for digital marketing products. To gather this type of information we focused our research on semi-structured interviews with a moderation guide designed to prompt discussion of behaviors, expectations, and pain points in an open-ended manner.

Recruiting the right participants

We recruited from existing users, specifically users identified internally as either “successful” or “at-risk” with the goal of understanding what top performers and poor performers were doing and how they might differ.  We recruited through a screening survey on email marketing behavior, which we used to group participants by characteristics such as their level of usage and expertise with the product.  Results from the screener also provided general quantitative data about marketing tools and practices. 

Conducting the interviews

The screener survey also provided background information about the participants, allowing us to focus the interview on more open-ended questions in order to learn about the participant's mental model. A more standardized approach of questioning would prevent us from following spontaneous conversation threads and identifying valuable insights.  

We completed 19, 1-hr long semi-structured interviews via video conference.  Participants were compensated for their time with visa gift cards sent by mail directly from the organization following the completion of all interviews.

Analyzing the data

After all the interviews were completed, we took detailed notes and extracted quotes from interview recordings.  With the nature of this research focused on mental models and experiences, data were analyzed using thematic analysis methods.  Our participants were a diverse group (commercial real-estate brokers to psychics) with a wide range of needs and behaviors. Identifying the connecting themes was not immediately clear.  We used the moderation guide as a framework to ensure that relevant information was covered.  After discussing some common findings, we established categories and conducted an affinity mapping exercise. We coded findings into themes and determined the impact of the theme as it related to “user success”. From this process we had three core insights about how users employ a range of digital marketing tools, how they seek more skills, and how feeling successful is closely tied to feeling confident in what you do.

Findings & Recommendations

Our findings provided the organization with an understanding of ​how users use digital marketing tools, the skills they seek related to digital marketing, and their nuanced approach to defining success.   However, we also saw a major gap in user needs and the products and services being offered. Based on the findings (and reinforced by the literature), we developed three user archetypes to focus our recommendations.  From the perspective of the organization's goal to make their users productive, effective, and efficient, we made priority based improvements for the organization based on "Processes & Tools" and "Guidance & Support", giving the organization recommendations on how to better align their product with user needs.