In my experience with methods reading, and in particular, during a closer reading of many arguments for how to choose or design particular aspects of a study’s methodology, there seem to be a few central things to the core of the decision. It appears that to understand choices of methodology there are two major steps that one must take in order to be successful.
Firstly, researchers need to shed absolutist notions of “better” or “worse” methodologies. In discussions with students, often the first concern of students is to discuss what is “better” or “worse” methodology, or what kinds of methodologies they accept as “scientific” or “ethical” without regard to the kinds of questions involved. Oddly, these claims are entirely organized around past situational education. Students often want to market the methodological skills they have developed in their education, and downplay those that others have in comparison as a way to compete, or justify the value of the education they have attained prior. I know that I have made these claims early in my education, and I fully accept that in doing so, I was wrong.
The existence of various methods of the sciences and the humanities seem to be centrally organized around more situationally specific practical concerns than they are about more epistemologically accurate or inaccurate discoveries. Epistemic beliefs on methodology often should be the secondary concern after consideration of the practical concerns. Certainly, we might be able to make an argument for why gathering and using data in one way is situationally better than another given epistemic claims, given we already know the existence of the better data is reliable and feasible to attain. However, as implied, first one must discuss reliability and feasibility under the situation. One cannot simply discredit, say, survey or interview data if there are no better options given the circumstance. Specifically a researcher should not do so if getting any kind of answer is unquestionably more helpful than having no answer at all.
Secondly, researchers need to understand the aspects of practical importance in their particular interest. There appears to be a common thread of which all researchers appear to crawl along like ants, not really seeing the whole, in order to explain in detail how they came by their chosen methods, and why their openness to the unknown lead them to success. In as few words as I can, I will describe these principles as practically reducible to accuracy, psychology, and economics. Although someone else might describe them as reducible to other things, I find that these are terms of which we can all understand to some extent.
Notice, the first step should become clear in the mind of a researcher as soon as there is pressure to publish. They must follow where the threads go for their particular areas of inquiry. They discover this necessarily. It’s the second step which seems to be difficult to explain or understand even by those seasoned in methods. Thus the rest of this article is a discussion on this second step.
3 OPTIMIZATIONS: Economic, Psychological, & Accuracy
Although someone else might see that these ultimately reduce to some other set of basic categories, the point here is to discuss a process of choosing methodologies in a way that most could understand. I will start with economic as first the chosen method must be feasible given resources at hand or else the study cannot be done. Second I will discuss psychology, which is more interested in the psychology of participants than the academic. Lastly I will discuss accuracy of method.
Economics of Method:
In some ways, this consideration is mostly chosen for the researcher. Every academic has been trained (or trained themselves) using the limited resources available to them throughout their education and career. In fact, they are trained to think that it is more productive and efficient if one can stick to the methods and heuristics that their field proposes is meaningful and believable. For example, one does not frequently have the time or money to re-educate themselves to do advanced statistical work after a PhD that was specifically qualitative, especially if that is not the standards the field seems to be able or wants to use. This person would have to acquire several years of statistical training, learn to understand how to translate this work into a statistical software package or two, and explain those results to others in a statistical way. Because of this, most people must maintain a specialized set of skills, and design their questions and hypotheses in such a way that is best suited to the skills they already have. This is even more explicit in the natural sciences where laboratory capital is a large part of the costs. To ask questions that are outside of the visual range of the equipment at hand is to ask very expensive questions for the researcher.
This is important to understand when reading research work. Many times a particular method was used, not because it was the theoretically best method, but because it was the best one, or perhaps the only one economically feasible for the researcher. The economics of research has a considerable history. In fact, this has been discussed as a matter closely resembling a problem of pragmatic thinking for the researcher.1 Also it has been considered as a competitive problem of applying knowledge and capital resources among many people vying to solve similar problems with uncertain outcomes.2 It would seem to me that choosing a method might be somewhat of a separate economic problem from designing the correct question or hypothesis, but in economics these things are generally bundled together, and probably rightfully so. One cannot hope to attempt to explain a substantial, modern physics problem without the proper tools, so why would academics without them attempt to answer those questions themselves?
This is no different for social scientists, but I believe it is much less immediately clear since much of the skills and capital of social science appear, at first glance, to be intangible assets. However, one still needs a place to do interviews, software of which they have rights to use and they need options which allow for specific things. In fact, many of the things that could help validate in person interviews are particularly troublesome when dealing with online participants. In online research, many of these limitations suddenly become clear again. One might forgive one for not realizing the cost and time to have a location for people to be interviewed, and perhaps even a researcher might overlook these costs given the way that their program lets them use school property, but this is certainly not the case online. One immediately becomes aware that they cannot be sure of who they are interviewing or surveying, and perhaps will have numerous other problems, and a development of trust is much more difficult to come by.3
Psychology and Ethics:
This is most closely related to ethics and risk management. After one knows the economics (costs and potential gains) of a study, a researcher and ultimately the IRB can then evaluate psychological risks relative to the potential gains. However, this is not strictly a psychological point. There can also be financial losses to society from a study, especially if you are able to identify the subjects in question. So it is partially social as well. But in the case of interviews and survey, it seems that the dominant interest is in long term psychological impression. Similarly, online, this can be difficult. There is often a real separation between the participant and the researcher that can lead to problems of validity, but this can be ethically necessary in many cases.4 However, if one is to discuss the methods of second hand data, or something like simply using census data, or even very large datasets scraped online that is not at the individual level of granularity, perhaps one should consider this relative to sociology and economics instead as the researcher really has no first hand understanding of what happens to individuals and individuals likely would have difficulty in giving a good explanation of what does occur when this data is used.
After the economic feasibility and ethical concerns have been addressed to a reasonable level, then one can to decide what method would meet certain epistemic standards. If after the prior two, one decides they have the option between several methods, then they can decide what will most validate and make accessible the findings of such a study. One should not consider this first if there is no ethical or feasible way to get data from the methods they believe are meaningful. In fact, if one cannot make a question or hypothesis economically feasible or ethically permissible, in either case, the study should not move forward. Perhaps there are better and worse methodologies, but given specific factors, different methods are tools to meet those factors of research process. Certainly one should not judge a method without understanding the goals of the publication, the field it speaks towards, and the particularities of the object or objects of study.
Certainly one would not reasonably claim that the most important part of building a home is having a good hammer. One cannot build a home with a hammer alone. So one should not be dogmatic about the use of a hammer over a saw for example. Certainly no one wants to cut a board with a hammer. Likewise, few would want to try to only do online interviews to get to all solutions to social and psychological questions online when perhaps a emailed survey structure will perform much better and be much cheaper. Notice this, as stated in the economics section, can be also relative to the field one is in. If the heuristic of the journals or academic community of interest see value in doing something a particular way, this might be a serious consideration regardless of the researcher’s opinion.
As stated, it seems to me that there is a somewhat unstated order to making methodological decisions when conducting a piece of research such as interviewing or surveying. First one must consider what can actually be done with the time and resources at hand. Then one must consider whether they and others (specifically the IRB) would consider it to be psychologically damaging (or economically or socially damaging if one considers other methods than direct observation). Then, and only then, is it reasonable to judge the value of a method over another. Once a possible set of methods are established as useful to the given topic, then we may decide pragmatically what works to solve the question or hypothesis of interest.
Peirce, C. S. “Note on the Theory of the Economy of Research.” Operations Research 15, no. 4 (1967): 643–48. ↩
Arrow, Kenneth. “Economic Welfare and the Allocation of Resources for Invention.” NBER Chapters. National Bureau of Economic Research, Inc, 1962. ↩
O’Connor, H. & Madge, C. (2017). Online interviewing. In Fielding, N., Lee, R. & Blank, G. The SAGE Handbook of online research methods (pp. 416-434). 55 City Road, London: SAGE Publications Ltd. doi: 10.4135/9781473957992 ↩
Dean, E., Cook, S., Keating, M, & Murphy, J. (2009). Does this avatar make me look fat?: Obesity and interviewing in Second Life. Journal of Virtual Worlds Research, 2, 3-11. ↩