Research Design

Week 4 - Research Design: The ability to meaningfully select research topics, pose research questions, sample populations, and securely manage data are part of the research design process. Throughout Week 4 we will work through the different elements of a research project’s design.

The slides for this week’s lecture can be found here

Overview

Research design is an umbrella term used to describe the activities that go into documenting how, when, and where research will be practically carried out. The design phase also helps a researcher to reflect on the representativeness and ethics of their proposed work. In short, a design phase has (at least) five considerations: What will be studied? (Units of analysis and observation); How will the study be representative? (Sampling of a population); How will researchers ensure the integrity of their research? (Internal and external validity) What will the study use to collect data? (Research Instruments); and How will data be stored for analysis and long-term access? (Data management plan).

In this week’s lecture we will address the first three considerations - leaving research instruments and data management planning to the specific research methods and traditions that we will discuss in later weeks. In doing so, we’ll be able to discuss the design of a research project with specific examples and specific methods that make sense for data collection.

Before we get started it may be helpful to define one key term you will often hear or encounter in the design phase of research: operationalize. In the simplest definition - operationalize simply means to take a plan and put it into action. It may help to consult the diagram below to understand how exactly we ‘operationalize’ a set of research questions. Moving from the Planning to the Designing stages in our research means that we’re starting to make concrete or actual the broad concepts that we describe in our research questions. So, when we move from one stage to the next we’re making operational decisions, and adding to the specificity of our research project. The Designing stage does just that - it lays down designs for how a research question can be operationalized. In the next stage we’ll actually Execute these research designs.

Units of Analysis

After writing a set of research questions and reviewing the literature - the next step in our research process requires that we identify what exactly we will collect data about. In other words - regardless of our epistemic or ontological commitments, we need to clearly document what we will study in terms of actual, real, observable phenomena that exist in the world. For example, if we were to write research questions about digital literacy we might want to study this concept through undergraduate students. If this were the case - the students would be whom we were looking to gather data about.

In research design we think of these subjects or participants in terms of units that will be analyzed or observed.

One way I think about the differences between units of analysis and units of observation is that they are like measurement and counting - I can measure my desk (unit of anlaysis is my desk), but I do so through using a tape measure with inches (that is, I count the number of inches long or wide that my desk is)… The same is true of a research subject or entity. I may be interested in studying the formation of cliques in a high school cafeteria. The unit of analysis then might be cafeteria tables where cliques congregate, but what I actually choose to gather data from - that is the indivisible unit of observation - will likely be students that sit at one particular table or another. The difference being - I have selected a broad entity that I will analyze (tables of cliques) and a specific entity that I will observe (students at tables).

By deciding and specifying our units of analysis and observation in advance, we often come to some clarity about how we can know (justify our beliefs) through the systematic collection of data. So in deciding our units of analysis and observation we can move on to thinking about the quantity of observations we need to make to comfortably justify what we may come to believe about our subjects.

Sampling

Again, regardless of our epistemic and ontological commitments - when making a decision about who and what to study we aim to produce research that is accurate, rigorous, and informative. When we select an entity to analyze or observe we want to uphold this accuracy, rigor, and informativeness by choosing entities that are representative of the thing we argue they are meant to shed light on. For example lets return to our study on cliques in a high school cafeteria - if we want our study to be representative of the high school, then we likely can’t study just one clique, or one person in a clique and claim to provide accurate data about clique formation. We likely need to select a sample of different students, sitting at different tables, and members of different cliques. Only then can we likely make a reliable claim about cliques in an entire high school.

When designing research it is necessary to first define the population being studied. In our example research project about cliques, the number of students in a high school is our total population. It would be infeasible, both for our time and likely our budget, to study each and every student in the high school’s population. Instead, we can use a strategy to sample a smaller number of students that may stand-in for, or represent the larger population.

In lecture we discuss three types of strategic sampling that can be used to ensure representativeness:

  1. Random - selecting a discrete sample based on a method of non-biased or random selection (e.g. every n-th person, drawing numbers from a hat, selecting from a list using a randomization algorithm, etc.)
  2. Stratified - if a population can be organized by some variable or trait in advance of beign sampled, then we can create strata and sample strategically so that we include a representative sample from each strata.
  3. Cluster - a population may already be organized, or clustered, into meaningfully groups before we begin sampling. If our analysis depends upon respecting or even understanding these clusters then we can, like strata, attempt to develop a representative sample by selecting from each cluster. A simple analogy here might be the study of a city where blocks represent a cluster of residents. We may wish, instead of sampling from an entire city to sample from blocks of the city in order to ensure we represent each neighborhood.

It is extremely important to note - these are really the building blocks of sampling methods. Population sampling can combine these techniques, and expand upon them in important ways. We will talk more about these variations in sampling when we discuss particular research methods in the coming weeks.

Validity

All research that attempts to generate new knowledge faces challenges, internally and externally, to the legitimacy of ‘how we know what we know’ - In our discussion of epistemology we talked about their being criteria for valid knowledge that consists of true justified beliefs. If our research planning and designing stages are about carefully documenting what and how we plan to gather data to make a new knowledge claim, then partially what we are engaged in is justifying our beliefs (informed by our literature reviews). Once we begin to gather data, or to operationalize our study, we need to be careful about the criteria we use to justify what we believe. Put another way, if we want to be accurate, rigorous, and informative then we need to have representative data that is valid - that is, actually about what we claim it to be about.

In research design we typically think about guarding against biases by using measures of external and internal validity.

We will, like sampling and units of analysis and observation, continue to revisit these concepts as we begin to explore quantitative, qualitative, and design-based research methods in the coming weeks. But, by simply recognizing and being aware of these concepts you can continue to build research plans and designs that are accurate, rigorous, and ultimately informative to the audience identified in your research statements.

Readings

Read the introduction, and then choose two types of research designs to read about (for clarity - each type is a method): USC LibGuide on Designing Research Projects - HTML

The design of a research project doesn’t need to be fully about creating brand new knowledge - it may also be for the purposes of conducting an evaluation of an ongoing program. Read a short introduction to Evaluation and its different design considerations at this link (HTML)

LIS Research Spotlight

Suggested

Exercise

Forthcoming