Reaction Paper: Interviews and Questionnaires

Crafting questionnaire and conducting interview are complicated but key to survey research. Both require a very careful methodological consideration but in the end it is simply an art (Converse, 1986: 7). Many readings discussed general guidelines on how to craft the standardized questionnaire, problems and challenges need to be considered in mind when crafting it, and problems in conducting face-to-face interview and how to deal with them. In this short note, I would like to raise several issues that can be problematic when crafting the questionnaire and conducting interviews.

Meaning. How to make sure that the question in the questionnaire is interpreted by the respondent exactly the same as intended by the researcher/the question writer is the main problem in crafting the questionnaire. Meaning relates to many things. It can relate to certain concept/reference, to certain context (time and space), etc. Converse (1987) suggests that the question needs not only to provide a frame of reference (since it may not be that the respondent commonly uses) but also a detailed explanation, such as: “By ‘family’ we mean…” (18). A more general guidelines from Converse to deal with this issue is by providing clear and straightforward questions by using simple language, common concept, and widespread information and then pretest it.

Tourangeau (et.al), Schwart (et.al), and Feldman reiterate that understanding the process of how respondents react to the question is a key issue so that we will be able to predict the answer to the question (the theory of response process or response effect). Respondent, according to this theory, will react through the process of comprehension, retrieval, judgment, and response to the question. These whole processes are not necessarily sequential and can be affected by many things such as the wording of the question, the order of the questions, and types of response options. Respondents, for instance, “are likely to draw on the content of preceding questions in interpreting subsequent ones” (Schwart, et.al,:37).

In my view however, crafting straightforward and clear questions and understanding the response process can only predict reactions of the respondents in general. In the end, the questions in the questionnaire will be answered by each respondent with his/her unique situation and response process. This is still problematic to the meaning of the researcher’s question. The desire to come to exactly the same interpretation between researcher and respondent is still difficult.

Social and cultural context. This issue is not very much discussed by the readings. Standardized questionnaire assumes that all questions are valid and appropriate to all respondents. What if the survey is conducted in a country governed by repressive regime or military junta such as in Burma? Can we ask standard questions? Or we can simply answer that the survey is not possible to be conducted in such a situation.

In certain cultural context, such as in some countries in Southeast Asia, asking clear and straightforward question as suggested by Converse is not always useful. For example, individual information in this situation is categorized into several layers. First, individual information that can be shared to others such as age and number of children. Second, information that individual hesitates to share such as whether or not she/he uses condom. Third, information on which individual is not sure enough and therefore she/he does not want others to know such as: Is he/she religious enough? Fourth, when evaluating others (including the government), the people tend to say it in indirect way. In the situation number 2 to 4 above, the questions cannot be straightforward. Therefore in crafting the question the researcher should also think of whether using proxy-question or not. For example, when asking whether or not someone like the president, the researcher can ask how he/she feels about life under the current government.

Public mood. According to Feldman, in interpreting the question and responding, respondents will activate only one interpretation (out of his/her multiple interpretations) to provide the answer. Therefore, “when respondent hear or read a survey question, the first thing they do (most likely automatically) is to activate an interpretative framework to make sense of the question” (10-11).

This process, however, assumes that the respondent’s situation is normal. It will be problematic when the question is asked in a certain public mood. What I mean by public mood here is a general feeling of the public related to certain situation. For example, again, in countries in Southeast Asia which are still affected by the economic crisis, the pessimistic feeling about economic life is widely spread across the country. In this kind of situation, conducting a survey on public opinion of economic performance of government tends to result in pessimistic evaluation of the government economic performance.

Interview: meaning or strict procedure? Face-to-face interview is commonly used in conducting a survey. The main issue here is how to make sure that the intended meaning is gained while the process of gaining it does not fall into daily conversation between interviewer and interviewee (thus violating the research procedure). Schuman and Jordan, however, found that meaning and procedure in interview are not always compatible each other. When a strict procedure is emphasized, there is a high possibility of having “error of the third kind”, that is “error that arises from the discrepancy between the concept of interest to the researcher and the quantity actually measured in the survey” (262). When the meaning is emphasized, the interviewer should have flexibility in clarifying and modify the question in a way that daily conversation happens. The question then whether or not the interview is still part of scientific research.

To overcome the problem, Schuman and Jordan suggest a collaborative approach in which not only the questions are crafted properly to maximize the intended interpretation, but also there is a flexibility for the interviewer to clarify the meaning when necessary (262). The fact that it will fall into ordinary conversation, following Schuman and Jordan, is not a problem as long as the interview is still structured as the standardized questionnaire required. The problem here however, how do we know that the conversation is still structured, and how do we know that the interview needs to clarify the meaning, and how do we make sure that it does not happen arbitrarily.

Other issues that related to interview are bias from the interviewer when several of these situations happen: 1) interviewer is not neutral or having and expressing certain interests through the ways of asking question, intonation, and attitudes; 2) interviewer does not have the same interpretation as the researcher; 3) interviewer does not qualify for the survey (too curious beyond the need of the survey, poor communication skills); 4) interviewer’s appearance is too attractive; 5) interviewer lies. To deal with these issues, the researcher usually conducts trainings for the interviewers and controls the procedure of survey through auditing.

Having said those several issues, I would like to acknowledge here that the usage of survey research is obvious. Standardized questionnaire is its main instrument of gauging data. Given its ultimate goal of having the same meaning between the researcher and the respondent, endeavor of getting better guidelines of crafting questionnaire needs to be encouraged more so that we do not have to say that it is simply an art.