Online survey software reviews: reliable or unreliable?
You are ready to do your first online survey or i…Read more
I was a speaker at a conference in Jakarta recently and as I left the conference room, someone came up to me and asked “What’s the best tabulation software for me?” I could see the man was short of time, so I had to think fast. My reply was “Tell me three things. Firstly, describe what you consider to be the types of thing that you need the software to do (I emphasised need). Secondly, describe the skill levels and potential of the users. Thirdly, does the software need to do anything else, for example, data collection or charting, or does have it to fit in with other software that you use?”
The reply made it easy for me. He just wanted to do some counts and percentages for each question with an option to run a simple cross tabulation by region, age and gender, for example. Almost as a warning, he told me that some projects could have over 1000 respondents. This statement made me realise that I should have asked what he considered a big project. So, let’s look at those four things – “what does the software need to do?”, “what skill level do the users have?”, “how does the tabulation process fit in to the company’s overall process?” and, the one I missed, “what is a big project?”
What are the candidates?
It’s a tough choice. Even at MRDC, we offer a range of options – and, there are many alternatives on the market –some good, some not so good. If you have experienced data processing professionals that can use or have the potential to use a scripting language, MRDCL might be your choice from our software portfolio. If you want an easier user interface, but a good level of flexibility and power, QPSMR might be the choice. If you want flexibility but online data collection, Snap might the solution. Or, if you have data in Excel, Excel itself might give you all the cross tabulations you need. So, let’s dig a little deeper.
1) What does the software need to do?
This is probably the most difficult one to pin down in all honesty. Something that might seem trivial could be a game changer if the software does not have a particular function. For example, as a buyer of software, you might take it for granted that any tabulation software for market research would easily calculate a standard error, but if this was important and you hadn’t checked, you may be buying software that is going to be fall short of what you want.
Have a checklist
The only successful way, in my experience, is to have a long check list of features and functions that you must have, so that potential software suppliers can confirm whether the feature is available.
Don’t be fooled by what is possible. What is easy?
It doesn’t end there though. There are some software packages that may be able to offer a lot potentially, but, in practice, it is very difficult to do. I have known of tabulation software that allows you to include code in a programming language, which potentially means anything is possible. So, to use the previous example, a standard error might be possible, but might be a lot of work. Therefore, in addition to features being available, the ease with which they are applied is equally important.
Be precise with your questions
Next, you need to be sure whether there are any limitations. I recently came across a company that had bought tabulation software and had checked that significance tests were easy to apply. They were, indeed, easy to apply, but you could only apply one significance test at 95%, whereas MRDCL, for example, allows two tests on one table at any percentage level.
Higher level considerations
There’s also some higher level considerations, which are easier to gauge when choosing software. Can you specify complex filters/routing? Can you construct complex (or even simple) variables from questions? Can the software handle grid rotations and product tests easily? What types of data can be exported from the software? Can you produce summary tables of rating scales? And, a summary of mean scores easily? These sorts of questions are easier to remember as they are likely to be fundamental to your requirements. However, it is important to be know what your priorities are.
2) What skill level (or potential skill level) do the users have?
It sounds obvious, but the most advanced software product in the world is not much use if no one is skilled enough to use it. Expensive mistakes can be made here. The software might be relatively low in cost to buy or lease, but if it needs new employees on a high salary, the real cost of using the software suddenly increases dramatically.
On the other side, don’t underestimate the potential of staff. If someone understands basic market research and has good logic skills, they may well have the potential to advance to something more complex to use, which is more powerful.
If you are considering an upward path in terms of complexity, it may make sense to take on a software product where more training is needed, but where greater long term gains are available. If not, avoid overly powerful software that takes days (or much more!) to learn.
3) Does the software need to do anything else or need to fit into a current process?
This is two questions really, I’ll accept that. So, to take the first part, do you expect the software to do anything other than tabulations? Does it need to do charts, for example? Or, can it produce figures in Excel so that charting is relatively easy? Do you want the software to allow data entry? Do you want the software to allow you to conduct online surveys or handle CATI (computer assisted telephone interviewing)?
Some products may offer nothing other than the ability to produce tabulations; some may offer a little more or a lot more. What do you need? What might also be desirable in the foreseeable future? Don’t worry if the software offers something you don’t need; you will rarely be paying extra for that – and it may be useful in the future.
The second part of the question may or may not be important. If you already have data collection software in place, for example, is it easy to get data from that software to the product you are choosing? Generally, you will want your data in Excel/CSV format, ASCII, Triple-S or SPSS format, but you will find that some tabulation software packages will handle only some (or even none) of these. Triple-S is a good route to go as it should mean that texts, questions/variables and data are easily moved to the product of your choice.
4) What is a big project?
This can be an important question. I have seen the extremes here because it is so subjective. In the case of the enquirer I mentioned in the opening paragraph, he thought a sample of 1000 respondents was big. At the other extreme, I’ve known people who work on national census data who think that half a million respondents is small.
The fact of the matter is that some software packages will genuinely struggle (some would say ‘fall asleep’) once things get too big. The number of questions and variables, the number of respondents and the number of tables are typically the key figures that may make one software package more suitable than another. Software tends not to be like a long book. If you usually read novels that are 300 pages long and pick up one that is 900 pages, you can be fairly sure it will take about three times as long to read. With software, the increase in time to process may be an upward curve with lesser software packages. The advice is to check and see whether your biggest project is a bridge too far for the software package you intend to use.
Choosing the right software can be tough, but hopefully this blog article has given some indicators of what to look for. Excel is an excellent tool for cross tabulations (Excel calls them pivot tables) – it’s fast, easy to use, but very limited. But, if it’s enough, my advice is to use it.
As for our products, here’s how I would describe our products:
MRDCL can do anything (more or less). It does it efficiently as long as you are a skilled user of scripting languages. It’s packed with features and handles big data sets and large volumes of tables well.
QPSMR and Snap are much easier to use. They both have plenty of features and can handle complex needs. QPSMR is strong for paper questionnaires and CATI whereas Snap is good for online/mobile surveys, charting and reporting.
DataDynamic has tabulation tools but would be most appropriate if you wanted to automate PowerPoint reports or produce basic online dashboards efficiently.
Whatever you choose, try to ensure that the software does what it needs to in the medium term. Make sure the key features are available and easy to use. Make sure that the software can comfortably handle projects of the size and complexity that you have. Above all, ask questions - suppliers should have nothing to hide, in our view.
As ever, I am always here to help and to advise at email@example.com