nt's black box. Therefore a basic principle in questionnaire design is for the researcher to ask a battery of questions to capture each of the intended concepts.

A related issue concerns the type of data that is collected by each question in a questionnaire or research instrument, as it is often called. In simple terms it is often sensible to avoid the yes/no type of questions as these are generally simplistic, not good at capturing constructs and limit what statistical analysis and testing one can or cannot undertake. Wherever possible it is useful to use scales as these allow for more capture of variance, wider statistical analysis and testing and ultimately more useful conclusions.

A second important aspect to do with questionnaires concerns issues of validity and reliability of the research instrument. This is one aspect that often receives limited attention, yet it is a major source of error. Throughout I have used the terms research instrument and questionnaires interchangeably because in the social sciences and in management research, questionnaires are our instruments and are equivalent to the thermometers, spectrometers and what have you that are used in the physical sciences.

Like these instruments, our questionnaires must meet two basic requirements - they must be valid and reliable. Using the thermometer idea, the user of such an instrument would want to be confident that it p

What makes a market research survey a good survey? There is no simple answer to this question, and it is not one aspect, but various aspects together that make for a good piece of research. Indeed, market research is very much a balancing act where the researcher often has to deal with decisions that have conflicting consequences. The researcher needs to balance out the various elements to ensure that much of what is gained on the swings is not lost on the roundabouts. This, after all, is what the research process is all about, and involves a dedicated attempt at reducing error, in the knowledge that one can never eliminate error completely. Many devote much attention to sampling issues yet paper over questionnaire issues that are often a much larger source of error. This article starts by looking at two important sampling issues and then proceeds to consider two other issues in questionnaire design that often receive scant attention.

Sampling Concerns

Whenever one is asked to undertake research, one of the first issues that tends to crop up is that of sample size. At least two aspects need to be highlighted in sample size considerations. First, are the results of the sample to be generalised to an entire population? This sounds like a trivial question and a somewhat shocked 'of course' refrain is to be expected. However, the corollary to this is, with what degree of accuracy and confidence do you want this to happen? Here lies the rub because these are researcher or management decisions and on this basis one can justify a wide range of sample sizes! In fact, more often than not, statistical considerations for sample size determination are of secondary importance.

A more relevant question to ask for sample size determination is perhaps what is the intended use of the findings? Will they be used to make a critical decision; as part of a PR exercise; or is it just a case of a 'nice to know' situation? Only the first objective is likely to require a representative sample. However, irrespective of sample size it is very desirable that a random sample is collected as this allows for statistical analysis of the data. Any random sample, even if the sample is not generalisable, can be analysed statistically. Moreover, there is nothing wrong with findings from a sample that are not generalisable as long as the conclusions are clearly bounded by the limitations of the sample.

A second aspect concerning sample size that is useful to consider relates to what type of analysis it is intended to undertake with the data collected, so as to answer your research question or questions. The size of the sample may seriously limit what analysis is possible to undertake. Different research issues or models will demand different statistical analytical tools and different sample size requirements to ensure the robustness of the analysis. For example, a multiple stage model involving constructs or concepts affecting another concept or concepts that in tum affect some other concept would require a particular analytical technique like structural equation modelling.

Such models are by no means far fetched and represent the type of situation encountered in trying to understand consumer decision making processes. Often they are best handled with structural equation modelling that has its own sample size requirement irrespective of generalisability issues. Indeed too large a sample may give a wrong answer because the statistical technique may be oversensitive with large samples. This may sound a little daunting if all the research that one has come across consists of percentages. But, there is more to good research than percentages. Indeed, percentage type analysis and results are often the outcomes of a decision the researcher would have made about the data collection employed in the questionnaire used.

Pitfalls of Questionnaire Development

There is no doubt that sample size has its importance in the research process but there is much more to be concerned about besides issues of sample size. Let me start by dispelling one myth that always sends a shiver up my spine. This is the idea that anybody can sit down and write a good questionnaire. Yes of course anybody can put together a questionnaire and there is nothing stopping anyone using it either. The emphasis in my statement is however is on a good questionnaire. Since this is not the place to go into the intricacies of questionnaire development, I will only underline two aspects to show how much is often overlooked.

I will start by first highlighting how one goes about seeking to capture a concept when designing a questionnaire. What a researcher is often trying to do when conducting market or management research is to capture something that resides in the respondent's mind or black box as it is sometimes called. It is by no means an easy feat and is often not likely to be achieved by simply asking a single question. A useful analogy is of the lecturer trying to capture student knowledge about a subject. A lecturer would normally seek to determine student knowledge by asking more than one question to capture the knowledge concept. If one asks a student just one question one may have hit the one area that the student just did not study. However, if one asks a couple of questions on different aspects of knowledge the student was supposed to have learned and it turns up dry then one is far safer in drawing conclusions about the student's knowledge. An analogous multi-question procedure is similarly worthwhile to pursue in questionnaire development to capture concepts like service quality, loyalty, etc., that are similarly resident in the respondent's black box. Therefore a basic principle in questionnaire design is for the researcher to ask a battery of questions to capture each of the intended concepts.

A related issue concerns the type of data that is collected by each question in a questionnaire or research instrument, as it is often called. In simple terms it is often sensible to avoid the yes/no type of questions as these are generally simplistic, not good at capturing constructs and limit what statistical analysis and testing one can or cannot undertake. Wherever possible it is useful to use scales as these allow for more capture of variance, wider statistical analysis and testing and ultimately more useful conclusions.

A second important aspect to do with questionnaires concerns issues of validity and reliability of the research instrument. This is one aspect that often receives limited attention, yet it is a major source of error. Throughout I have used the terms research instrument and questionnaires interchangeably because in the social sciences and in management research, questionnaires are our instruments and are equivalent to the thermometers, spectrometers and what have you that are used in the physical sciences.

Like these instruments, our questionnaires must meet two basic requirements - they must be valid and reliable. Using the thermometer idea, the user of such an instrument would want to be confident that it pr

you want this to happen? Here lies the rub because these are researcher or management decisions and on this basis one can justify a wide range of sample sizes! In fact, more often than not, statistical considerations for sample size determination are of secondary importance.

A more relevant question to ask for sample size determination is perhaps what is the intended use of the findings? Will they be used to make a critical decision; as part of a PR exercise; or is it just a case of a 'nice to know' situation? Only the first objective is likely to require a representative sample. However, irrespective of sample size it is very desirable that a random sample is collected as this allows for statistical analysis of the data. Any random sample, even if the sample is not generalisable, can be analysed statistically. Moreover, there is nothing wrong with findings from a sample that are not generalisable as long as the conclusions are clearly bounded by the limitations of the sample.

A second aspect concerning sample size that is useful to consider relates to what type of analysis it is intended to undertake with the data collected, so as to answer your research question or questions. The size of the sample may seriously limit what analysis is possible to undertake. Different research issues or models will demand different statistical analytical tools and different sample size requirements to ensure the robustness of the analysis. For example, a multiple stage model involving constructs or concepts affecting another concept or concepts that in tum affect some other concept would require a particular analytical technique like structural equation modelling.

Such models are by no means far fetched and represent the type of situation encountered in trying to understand consumer decision making processes. Often they are best handled with structural equation modelling that has its own sample size requirement irrespective of generalisability issues. Indeed too large a sample may give a wrong answer because the statistical technique may be oversensitive with large samples. This may sound a little daunting if all the research that one has come across consists of percentages. But, there is more to good research than percentages. Indeed, percentage type analysis and results are often the outcomes of a decision the researcher would have made about the data collection employed in the questionnaire used.

Pitfalls of Questionnaire Development

There is no doubt that sample size has its importance in the research process but there is much more to be concerned about besides issues of sample size. Let me start by dispelling one myth that always sends a shiver up my spine. This is the idea that anybody can sit down and write a good questionnaire. Yes of course anybody can put together a questionnaire and there is nothing stopping anyone using it either. The emphasis in my statement is however is on a good questionnaire. Since this is not the place to go into the intricacies of questionnaire development, I will only underline two aspects to show how much is often overlooked.

I will start by first highlighting how one goes about seeking to capture a concept when designing a questionnaire. What a researcher is often trying to do when conducting market or management research is to capture something that resides in the respondent's mind or black box as it is sometimes called. It is by no means an easy feat and is often not likely to be achieved by simply asking a single question. A useful analogy is of the lecturer trying to capture student knowledge about a subject. A lecturer would normally seek to determine student knowledge by asking more than one question to capture the knowledge concept. If one asks a student just one question one may have hit the one area that the student just did not study. However, if one asks a couple of questions on different aspects of knowledge the student was supposed to have learned and it turns up dry then one is far safer in drawing conclusions about the student's knowledge. An analogous multi-question procedure is similarly worthwhile to pursue in questionnaire development to capture concepts like service quality, loyalty, etc., that are similarly resident in the respondent's black box. Therefore a basic principle in questionnaire design is for the researcher to ask a battery of questions to capture each of the intended concepts.

A related issue concerns the type of data that is collected by each question in a questionnaire or research instrument, as it is often called. In simple terms it is often sensible to avoid the yes/no type of questions as these are generally simplistic, not good at capturing constructs and limit what statistical analysis and testing one can or cannot undertake. Wherever possible it is useful to use scales as these allow for more capture of variance, wider statistical analysis and testing and ultimately more useful conclusions.

A second important aspect to do with questionnaires concerns issues of validity and reliability of the research instrument. This is one aspect that often receives limited attention, yet it is a major source of error. Throughout I have used the terms research instrument and questionnaires interchangeably because in the social sciences and in management research, questionnaires are our instruments and are equivalent to the thermometers, spectrometers and what have you that are used in the physical sciences.

Like these instruments, our questionnaires must meet two basic requirements - they must be valid and reliable. Using the thermometer idea, the user of such an instrument would want to be confident that it p

e robustness of the analysis. For example, a multiple stage model involving constructs or concepts affecting another concept or concepts that in tum affect some other concept would require a particular analytical technique like structural equation modelling.

Such models are by no means far fetched and represent the type of situation encountered in trying to understand consumer decision making processes. Often they are best handled with structural equation modelling that has its own sample size requirement irrespective of generalisability issues. Indeed too large a sample may give a wrong answer because the statistical technique may be oversensitive with large samples. This may sound a little daunting if all the research that one has come across consists of percentages. But, there is more to good research than percentages. Indeed, percentage type analysis and results are often the outcomes of a decision the researcher would have made about the data collection employed in the questionnaire used.

Pitfalls of Questionnaire Development

There is no doubt that sample size has its importance in the research process but there is much more to be concerned about besides issues of sample size. Let me start by dispelling one myth that always sends a shiver up my spine. This is the idea that anybody can sit down and write a good questionnaire. Yes of course anybody can put together a questionnaire and there is nothing stopping anyone using it either. The emphasis in my statement is however is on a good questionnaire. Since this is not the place to go into the intricacies of questionnaire development, I will only underline two aspects to show how much is often overlooked.

I will start by first highlighting how one goes about seeking to capture a concept when designing a questionnaire. What a researcher is often trying to do when conducting market or management research is to capture something that resides in the respondent's mind or black box as it is sometimes called. It is by no means an easy feat and is often not likely to be achieved by simply asking a single question. A useful analogy is of the lecturer trying to capture student knowledge about a subject. A lecturer would normally seek to determine student knowledge by asking more than one question to capture the knowledge concept. If one asks a student just one question one may have hit the one area that the student just did not study. However, if one asks a couple of questions on different aspects of knowledge the student was supposed to have learned and it turns up dry then one is far safer in drawing conclusions about the student's knowledge. An analogous multi-question procedure is similarly worthwhile to pursue in questionnaire development to capture concepts like service quality, loyalty, etc., that are similarly resident in the respondent's black box. Therefore a basic principle in questionnaire design is for the researcher to ask a battery of questions to capture each of the intended concepts.

A related issue concerns the type of data that is collected by each question in a questionnaire or research instrument, as it is often called. In simple terms it is often sensible to avoid the yes/no type of questions as these are generally simplistic, not good at capturing constructs and limit what statistical analysis and testing one can or cannot undertake. Wherever possible it is useful to use scales as these allow for more capture of variance, wider statistical analysis and testing and ultimately more useful conclusions.

A second important aspect to do with questionnaires concerns issues of validity and reliability of the research instrument. This is one aspect that often receives limited attention, yet it is a major source of error. Throughout I have used the terms research instrument and questionnaires interchangeably because in the social sciences and in management research, questionnaires are our instruments and are equivalent to the thermometers, spectrometers and what have you that are used in the physical sciences.

Like these instruments, our questionnaires must meet two basic requirements - they must be valid and reliable. Using the thermometer idea, the user of such an instrument would want to be confident that it p

there is nothing stopping anyone using it either. The emphasis in my statement is however is on a good questionnaire. Since this is not the place to go into the intricacies of questionnaire development, I will only underline two aspects to show how much is often overlooked.

I will start by first highlighting how one goes about seeking to capture a concept when designing a questionnaire. What a researcher is often trying to do when conducting market or management research is to capture something that resides in the respondent's mind or black box as it is sometimes called. It is by no means an easy feat and is often not likely to be achieved by simply asking a single question. A useful analogy is of the lecturer trying to capture student knowledge about a subject. A lecturer would normally seek to determine student knowledge by asking more than one question to capture the knowledge concept. If one asks a student just one question one may have hit the one area that the student just did not study. However, if one asks a couple of questions on different aspects of knowledge the student was supposed to have learned and it turns up dry then one is far safer in drawing conclusions about the student's knowledge. An analogous multi-question procedure is similarly worthwhile to pursue in questionnaire development to capture concepts like service quality, loyalty, etc., that are similarly resident in the respondent's black box. Therefore a basic principle in questionnaire design is for the researcher to ask a battery of questions to capture each of the intended concepts.

A related issue concerns the type of data that is collected by each question in a questionnaire or research instrument, as it is often called. In simple terms it is often sensible to avoid the yes/no type of questions as these are generally simplistic, not good at capturing constructs and limit what statistical analysis and testing one can or cannot undertake. Wherever possible it is useful to use scales as these allow for more capture of variance, wider statistical analysis and testing and ultimately more useful conclusions.

A second important aspect to do with questionnaires concerns issues of validity and reliability of the research instrument. This is one aspect that often receives limited attention, yet it is a major source of error. Throughout I have used the terms research instrument and questionnaires interchangeably because in the social sciences and in management research, questionnaires are our instruments and are equivalent to the thermometers, spectrometers and what have you that are used in the physical sciences.

Like these instruments, our questionnaires must meet two basic requirements - they must be valid and reliable. Using the thermometer idea, the user of such an instrument would want to be confident that it p

nt's black box. Therefore a basic principle in questionnaire design is for the researcher to ask a battery of questions to capture each of the intended concepts.

A related issue concerns the type of data that is collected by each question in a questionnaire or research instrument, as it is often called. In simple terms it is often sensible to avoid the yes/no type of questions as these are generally simplistic, not good at capturing constructs and limit what statistical analysis and testing one can or cannot undertake. Wherever possible it is useful to use scales as these allow for more capture of variance, wider statistical analysis and testing and ultimately more useful conclusions.

A second important aspect to do with questionnaires concerns issues of validity and reliability of the research instrument. This is one aspect that often receives limited attention, yet it is a major source of error. Throughout I have used the terms research instrument and questionnaires interchangeably because in the social sciences and in management research, questionnaires are our instruments and are equivalent to the thermometers, spectrometers and what have you that are used in the physical sciences.

Like these instruments, our questionnaires must meet two basic requirements - they must be valid and reliable. Using the thermometer idea, the user of such an instrument would want to be confident that it provides the same reading when used among healthy individuals and therefore the instrument can be said to be reliable. In addition, you want to be sure that it is valid and is measuring what it is supposed to be measuring which in this example is clearly temperature. That a thermometer measures temperature is fairly self-evident but this aspect is not always so clear. Using the earlier example of testing student knowledge it could be argued that rather than capture knowledge, the instrument used in the form of tests and exams are really capturing the student's recall ability.

This is a legitimate validity concern. Validity and reliability appear trivial in the case of the thermometer but less so in the case of tests to capture student knowledge and far less so when the researcher is seeking to capture marketing or management constructs. Here, concepts like service quality, value, satisfaction, loyalty and so on, are demanding to conceptualise and to convert to questionnaires (termed operationalisation). Testing for validity and reliability is often limited in much of the commercial research that one encounters, however it is critical if worthwhile conclusions and directions for management decision making are to be indicated.

Both validity and reliability can be tested statistically and supported if they are within established parameters of acceptance. Indeed validity is not an absolute but a question of degree. Some questionnaires are more valid than others. Yet, many are neither valid nor reliable, rendering any conclusions drawn from the data dubious.

Developing a good questionnaire to capture a concept, be that concept brand personality, service quality or one of the many other concepts we use in marketing and management, is often a full Ph.D. dissertation. It requires pursuing a demanding process involving more than one data collection exercise allowing for the testing of the questions used to capture the concept of interest, as well as the testing and retesting of the instrument's reliability and different aspects of its validity.

I have seen too many questionnaires that seek to capture constructs by asking one or a few 'off the head' questions. Such an approach is hardly a solid basis for strong and meaningful data capture, analyses and recommendations for decision making by management. This article has sought to highlight some of the complexities that are papered over and that can introduce considerable error. Many show extreme concern over sample size issues and pay little attention to questionnaire issues that are often a much larger source of error.

Rigorous research is undoubtedly challenging. Yet, faced with what at first hand may seem like insurmountable complexity, determination is required to avoid sliding back into doubtful past practices. If one is to spend time, money or both on research this is an appeal to use tried and tested instruments and invest some effort in undertaking analysis that goes beyond the provision of a few percentages. Done properly, survey research represents an indispensable tool that can provide useful inputs to management's decision making process.