How to analyse and report on a tracking study

I was going to call this blog article ‘Handling a tracking study efficiently’, but I thought the word ‘efficiently’ might get overlooked. Why? Because software vendors often like to claim things are easy, intuitive and ‘not a problem’. You will frequently hear or read words like ‘seamless’ and ‘automated’, coupled with statements like ‘instantaneously updating dashboards in real time’. Is that the truth? Well, it might be, but it might not be. Let’s start by considering the fundamental differences between typical customer feedback surveys and typical market research surveys with, say, 10 to 1000 questions.

Customer feedback surveys

Most of us receive customer feedback surveys from time to time; it sometimes feels like daily. You barely have to read the questions as you know what to expect, although I sometimes struggle with ‘Would I recommend this computer paper supplier to a friend or a family member?’ when it was the most unmemorable transaction of the last year and the least likely topic I will discuss with anyone. But that’s another story. The critical point is that customer feedback surveys are short, usually one to four questions, and unlikely to change for long periods, if ever.

Market research surveys

Let’s contrast typical market research surveys of ten, and often many more, questions. If you have ever managed a tracking study of this type, you will know that the research buyer will likely change the questionnaire regularly, even if they have claimed there won’t be many changes. A situation will soon arise where a new sub-brand needs to be added to the questionnaire, or an advertising campaign will push a new service or message that needs reporting. Things are likely to change. And possibly dramatically. Can the software you use to analyse and report from this type of tracking study cope?

First of all, a story

I attended an ESOMAR conference recently and approached a company with a trade stand that claimed to be experts in handling tracking studies. I was interested in exploring how data from our MRDCL data analysis software could be handled in their dashboarding system. People at trade stands tend to be positive, and the person talking to me certainly was – at least initially. I asked a series of questions, which drew a series of responses that I have summarised below. You will see what I mean!

My Question	The response
What happens if you add a question to the survey?	That’s OK, as long as the question has a unique name like Q3a.
What about if I want the new question to be Q4 and all the question numbers from the previous waves to move forward by one question.	(Rubs chin) That’s tricky. One of our engineers could write a recode to deal with that.
What happens if I need to add a response for a new brand or sub-brand	No problem
So, if I want to keep the brands alphabetically and the new brand is fifth, meaning that code 5 becomes code 6, etc., is that OK?	No, the code would need to go on the end of the other codes; otherwise, the data will get confused. (I think he muttered ‘obviously’ after that, but I  may be wrong)
Is there no way around it?	Yes (looking delighted), one of our systems team could recode all the data, including the back data.
What happens if a question disappears?	(He’s looking fed up with me now) Why would it disappear? (He’s hoping I go away)
If the data is in ASCII format and the data location changes, can your system manage that?	Let me give you the email address of our systems manager; he will be able to advise you. (Conversation ends)

What is my point?

I think there are some crucial points to make here:

• There is a big difference between a tracking study with a short questionnaire of a handful of questions and a tracking study with many questions. A customer feedback questionnaire is unlikely to change; if it does, it is most likely a relatively simple operation and a one-off need.
• A market research survey is likely to have multiple changes over its lifetime. I have worked on surveys with literally hundreds of changes over the course of one year.
• Software platforms that efficiently automate customer feedback surveys from data collection to analysis and automated online reporting or report distribution are entirely different beasts from market research survey processing systems.
• While it may be possible to program around questionnaire changes or recode data, this can soon become a tangled web or require expensive experts to handle.

Where does MRDC Software sit in this?

Our MRDCL software product is designed to handle questionnaire differences in tracking studies. There are tools to make this as automated as possible, but there is a far more important capability within MRDCL that avoids this ‘tangled web’ situation that I refer to or the need to get experts involved. Some products may be able to handle some of the problems I posed for the exhibitor at the trade stand, but few have all the tools that MRDCL has. What are these tools? Let’s explore more.

MRDCL templates

At the heart of the problems I raised at that trade stand are things conceptually very simple, but when it comes to computer systems, they are fiddly and awkward. It’s no coincidence that spreadsheets are computing-friendly, as things are nicely organised in rows and columns. This concept has been adopted by MRDCL, using its templating tools. If you want to tell MRDCL that what was q5 for waves 1 to 6 has become q6 for waves 7-10 and q7 for waves 11 onwards, it is as easy as filling in a spreadsheet.

What you need to make the automation of your tracking study analysis and reporting easy

1. Understand the scope – ensure that you understand how the data will arrive and how reporting is required.
2. Anticipate changes – if data or reports may change or be structured differently, be ready for these changes wherever possible.
3. Don’t postpone problems – if something major changes, don’t ignore the problem and find workarounds. You are likely to pay a premium for this in the long run.
4. Have a budget for changes – make sure there is a budget to streamline systems every 6 to 12 waves. This means annually for a monthly project and about every two years for a quarterly project.

Similar can be a lot more expensive than the same

Clients have sometimes been surprised that there is a such a big difference in cost between repeating a process and reproducing something that is similar. Repeating the analysis and reporting from a tracking study that is identical should be a matter of pressing some buttons and checking the output. Where things change, a lot more work is needed. Seemingly, minor differences can mean a lot of minor programming changes, which add up to a whole new re-test procedure.

Making changes to a computer programs is a difficult task, which is prone to error, even with the best documentation and standard operating procedures. Program code is as hard to read as a fairly unfamiliar foreign language.

Think costs

Having worked in data management, analysis and reporting for many years, I have encountered a number of badly handled tracking studies that MRDC has, in many cases, helped to reorganise to make efficient.

One recent project that I encountered was costing just under US$10000 in staff time to process each month. By reorganising the project at a cost of US$15,000, the project was reduced to a monthly cost of about US$1,500. In other words, savings were made within two months.

Organise the project from the start or, if it’s started, organise it now!

The example above is not uncommon. Time and costs can soon run out of control where a tracking study is not well managed. We are not talking about 10% or 20% improvement in efficiency; we are talking about massive potential savings.

Need help or advice? Just ask.