Your organization has designed a new training program, and everyone is eager to see business results. You have worked with your stakeholders to define what “success” looks like by creating a Measurement Map®, specifying what to measure to indicate that the program is working. Now what? Making the jump to how to measure the program can be challenging, especially if you aren’t a “data person.”
A recent Watershed study found that while 94% of respondents want to measure business impact, less than half reported having any analytics capabilities on their learning and development (L&D) teams, and only 7% are actively seeking to add analytics staff to their teams. Further, when asked what the biggest challenges are to measuring impact, two of the most common answers were not knowing how to start and not having access to the data. If this sounds like your situation, how do you proceed with measuring the business impact of your new training program? Here are five tips to help you tackle the data dilemma:
1. Find a Data Analyst to help you
Data Analysts are your data wranglers. They have the skills to help you determine what data you need and can either pull that data for you or help find someone who can. Most likely, your project will require multiple data sets, such as learning, HR, and business data. Your Data Analyst can organize all of those files for you, cleaning them and merging them together. They can provide you with descriptive analyses of the data (counts, averages, etc.) and may be able to help with more in-depth statistical analyses, where needed. Maybe you don’t have anyone in your group specifically with that title. (In the Watershed study, only 20% of respondents had a Data Analyst or statistician on their team.) But might you have a person with access to data and an interest in analysis who can help? (Think about someone who provides you with reports and has a proclivity for Excel and start there.) Engage a Data Analyst in your project as soon as you can. Ideally, the Data Analyst would be part of your stakeholder Measurement Mapping session in which you and the stakeholders discuss the project’s goals and choose the metrics that will best show success. That way, they can hear firsthand what the goals for the project—and the metrics—are.
2. Be open to suggestions and state that to your Data Analyst
Your Measurement Map lays out your ideal metrics for showing impact. But there may be times when a business metric isn’t available or reliable. Maybe your desired metric wasn’t captured for the entirety of the time frame you need, but another similar metric was and could be used instead. Perhaps some regions define a metric differently, so you shouldn’t directly compare them. Your Data Analyst can help figure that out. Approach this situation as a conversation about the data and make it clear that you value the input. It’s better to find out about any potential issues with the data early rather than after you have presented your plans to the stakeholders!
3. Collaborate on your operational definitions
You want the most accurate, complete data possible. To get that data for you, a Data Analyst needs a description of exactly which data to pull and what calculations to make, which requires that you both agree on definitions. So, expect a discussion around:
- Who and what should be included? Does “all Customer Service Representatives” mean everyone who is active on the date you requested the data, the month the training program rolled out, or the beginning of this year? What about people who have been terminated or changed job roles? Should the list of training completions include all courses or just a subset? What if someone took a course twice; do you want both to count as “course completions,” or just the first or most recent?
- What’s the time frame? What are the starting and ending dates you’re interested in? You said you wanted quarterly average survey results, but you asked for data starting in March. Do you still want this data aligned to a calendar quarter, or every three months starting in March? What does “weekly” mean? Monday through Sunday, Sunday through Saturday, or something else?
- Do you want raw data or aggregated data? Sometimes, you’ll want details on every record, while other situations might call for the data to be summarized in some way. For example, do you want each individual training record (so that you know exactly who took what course when), or do you prefer the total number of courses completed, total hours of training, or percentage of training completed? If you are looking at sales data, do you want each individual sales transaction or the total sales per month (in dollars or in other units)? If the data are summarized, is that per person, per department, per region—and on what time interval (daily, monthly, annually)?
It’s fine if you don’t know the answers to all of these questions going into the conversation. Use this discussion to determine what options will yield the most applicable data to answer your research questions.
4. Ask the Data Analyst to write your data requests
Now that you know what data you want, you’ll probably find that you have to go to more than one data source to get it. (For example, personnel data is often stored in a different system than business data, such as sales or error rates.) Let the Data Analyst take the lead in writing up the specifications for the data requests and working with the various data owners to get the data.
5. Work together on the interpretation
Your Data Analyst will likely create many graphs and tables while analyzing the data. Go through the various data visualizations together and compare your views on what story the data is telling. For which groups is training making the most difference—for example, new hires or those who started out as lower performers? (Are those the same people?) Did the northwest region see a bigger change in sales than the other regions following the latest product training? Do the teams that increased their customer satisfaction scores also show improvements in customer retention? Use your knowledge of your program and your organization to uncover and highlight key insights into your story.
A good Measurement Map is an excellent start to a measurement project, defining what to measure. Taking the steps above to partner with a Data Analyst can help with how to measure, allowing you to obtain relevant, trustworthy data that answers your stakeholders’ business questions.
About the Authors
