Social science research

This article incorporates material from the Wikipedia article "Social science research", as of an unknown date, released under the Creative Commons Attribution-ShareAlike 4.0 License. Social science research is the practice of investigating human behaviour, institutions, and social relationships by systematic empirical method — a craft defined less by any one technique than by the obstacle it is organised around, since its subject matter can seldom be placed under experimental control and most of its designs are therefore approximations to an experiment that cannot be run. It is the research arm of the social science disciplines — sociology, psychology, political science, economics, anthropology and communication studies — and one branch of scientific research at large, standing beside archaeology, materials science and the biomedical trades rather than under any of them; it encompasses survey research, experimental and quasi-experimental design, econometrics, demography, ethnography, content analysis and case-study work as its sub-trades, together with the applied policy fields — disinformation studies, conspiracy research and online-radicalization research — that draw on all of them.

The trade's characteristic products are the sample survey, the field experiment, the administrative-data study and the ethnographic monograph, and its findings pass routinely into legislation, litigation, clinical guidance and platform rule-making. Since about 2011 the field has been working through the replication crisis — the finding that a large share of its published results do not hold when independent teams repeat the study — and has answered it with preregistration, registered reports, open-data and open-materials requirements, and larger samples.

One branch of the trade is contested in a way the rest of it is not. According to the former State Department official Mike Benz, the mathematician Eric Weinstein, and the journalists Matt Taibbi and Glenn Greenwald, the research on misinformation, conspiracy theory belief and online radicalization that expanded after 2016 operates less as inquiry than as the evidentiary supply line of an enforcement apparatus — the censorship-industrial complex, in the term Benz uses throughout — its studies reportedly commissioned, funded and staffed by the same institutions that then administer content moderation and deplatforming on the strength of the findings. The researcher Renée DiResta, whose own Stanford Internet Observatory is named throughout that account, holds the charge to be a political campaign rather than a finding — one that closed a research centre rather than exposing one.

Accounts of where that claim itself came from differ, and most of them require nobody to be lying. On the reading Benz and Weinstein advance, the design faults are real and purposive, and the people building the instruments know what they are for. On a second, the faults are real but wholly ordinary: undergraduate participant pools, crowdworker samples, flexible outcome measures and a politically uniform professoriate are weaknesses the trade's own methodologists documented across social psychology well before the censorship dispute began, so that the critics have seen a general infirmity of the trade accurately and read a purpose into it. On a third, one coinage covers two unrelated things — a documented flow of federal and foundation money into online-information projects, and an alleged rigging of study design — and the merger is itself the claim. On a fourth, the description is back-formed from a small number of long-form interviews, so that later informants describe the apparatus in the vocabulary and running order of whichever account reached them first.[citation needed]

The doubt attaches to that chain rather than to the research. The account is carried chiefly by interviews rather than by documentary exhibits addressed to study design, and the strongest documents in it — the Election Integrity Partnership's ticketing records and the Twitter Files releases — bear on what the platforms and the consortium did, not on how the underlying studies were built.[citation needed] It runs the other way too: the reply that the research is ordinary rests substantially on self-assessment, the institutions named in the account being the same ones that reviewed it.

History and development

The empirical study of social life took its modern shape in the nineteenth century, when statistical description of populations was joined to explicit theory. Auguste Comte's positivism proposed a science of society on the model of the natural sciences, Adolphe Quetelet applied the arithmetic of averages to crime, marriage and mortality, and Frédéric Le Play and Charles Booth carried out household surveys of working-class life whose schedules are recognisably the ancestor of the modern questionnaire. Émile Durkheim's Suicide (1897) is conventionally read as the first sustained attempt to test a social theory against official statistics, while Max Weber's insistence that social action be understood from the actor's own meanings founded the interpretive tradition that has run alongside the statistical one ever since.

The sample survey was industrialised between the wars and after — quota polling in the 1930s, probability sampling and the permanent survey organisations in the 1940s, and the standardisation of interviewing, coding and weighting into a routine service trade. Design theory was codified in the 1960s: Campbell and Stanley's treatment of internal and external validity gave quasi-experimental work its vocabulary, and Glaser and Strauss's grounded theory and Geertz's "thick description" did the same for fieldwork. From the 1990s, econometrics and development economics went through what their practitioners call the credibility revolution, shifting the field's centre of gravity toward natural experiments, instrumental variables and randomised field trials.

The post-war expansion was paid for, and by whom has shaped what the trade asks. Government agencies and private foundations became its principal patrons, and the terms of that patronage — what is funded, on what schedule, against what deliverable — set research agendas as directly as any theoretical development did. The pattern of federal, foundation and intelligence-agency money running into university social science through direct contract and foundation intermediary alike belongs to the same history[citation needed], and it is the structure at which the trade's later funding critics point.

Methods and tools

The trade's methodology divides conventionally into a quantitative and a qualitative family, with mixed-methods designs bridging them; the choice is governed by whether the question is about incidence and effect size or about meaning and process, and most working researchers are trained in one family and collaborate for the other.

Quantitative and experimental designs

The randomised experiment is the reference design and is usually unavailable, since the causes social researchers care about — a war, a recession, a schooling reform, an upbringing — cannot be assigned. What the trade does instead is approximate it: the probability sample survey for describing a population, the field experiment where randomisation can be arranged, the quasi-experiment where a policy change or an administrative boundary supplies something close to random assignment, and econometric identification strategies — instrumental variables, difference-in-differences, regression discontinuity — where it does not. Measurement is a separate craft again: an abstract construct such as trust, prejudice or radicalisation has to be operationalised into items a respondent can answer, and the validity of the resulting scale is an empirical question in its own right rather than a matter of the researcher's definition.

The generalizability of the resulting findings is limited by who actually sits for the studies. Behavioural scientists publish broad claims about human psychology on samples drawn overwhelmingly from Western, educated, industrialised, rich and democratic populations, mostly undergraduates, who are outliers rather than representatives on many of the measures involved. The cheap modern successor to the undergraduate pool is the online crowdworker panel, whose respondents skew young, underemployed and highly educated and include a core of practised participants who have taken many such instruments before. Statistical software, standing panel infrastructures, linked administrative registers and commercial survey vendors make up the rest of the tooling, and each of them constrains what can be asked as well as what can be answered.

Qualitative and interpretive designs

Fieldwork proceeds on a different logic: the researcher is the instrument, and the object is to recover what an action means to the people performing it rather than how often it occurs. Participant observation and the ethnographic monograph remain the core form, joined by the in-depth interview, the focus group, documentary and content analysis, discourse analysis, and the case study worked in depth against theory. Grounded theory supplies the standard procedure for building categories out of the material rather than imposing them on it, and the tradition carries its own validity vocabulary — credibility, transferability, reflexivity about the researcher's own position — in place of the sampling-and-inference apparatus of the quantitative side.

Ethics and research governance

The trade's governance regime was built out of its own failures. The Milgram obedience studies, the Stanford prison experiment and the Tuskegee syphilis study established, between them, that consent, deception and harm in human-subjects work could not be left to the investigator's discretion, and the Belmont Report of 1979 set out the principles — respect for persons, beneficence, justice — on which institutional review was then constructed. Prospective review by an institutional review board, documented informed consent, limits on deception and debriefing requirements are now ordinary conditions of doing the work at all.

Review also determines what may be studied, by whom, and whether the result is publishable, and a substantial volume of social research runs outside the regime that produces the public literature. Work carried out under contract, under a nondisclosure agreement or under a classification guide — the pattern documented in classified research on campus and in classified thesis work — is reviewed by its sponsor rather than by a university board, and its findings do not enter the record that later reviews and meta-analyses are built from.[citation needed] The published corpus is therefore a filtered one, and the filter is administrative rather than methodological.

Training and practitioners

Entry to the trade runs through a doctoral programme with a compulsory methods sequence — sampling and statistics, design, and one qualitative craft — followed by an apprenticeship on someone else's grant. The work is done disproportionately by graduate students and research assistants: they field the instruments, run the coding, clean the data and often write the first draft, while the principal investigator holds the grant and the publication record. Around them sits a service industry of survey firms, panel vendors and crowdworker platforms that supplies respondents by the thousand on commercial terms.

The career structure rewards publication, and publication is controlled by peer review and the journal hierarchy, so the incentives of academic publishing and academic gatekeeping reach back into what gets studied. Grant dependence compounds it, since a research programme that cannot be funded cannot be staffed, and the economics of the trade therefore select for questions a patron will pay to have asked. The pressure runs to individuals as well as to agendas: a researcher who publishes against a funded consensus risks the position rather than the argument, which is the mechanism documented as career coercion and, at its sharper end, as the suppression of those who pursue proscribed questions.[citation needed] The practitioners themselves are also, as a body, politically unrepresentative of the populations they study: social psychology in particular went from considerable political diversity to almost none over the second half of the twentieth century, a skew the field's own methodologists have described as a threat to the validity of its findings rather than merely an oddity of its demography.

The replication crisis

Main article:

Replication crisis

From about 2011 the trade discovered that a large proportion of its published findings did not survive independent repetition. The statistical grounds had been set out earlier — that for research fields with small effects, small samples and analytic flexibility, most published positive findings should be expected to be false — and were made concrete when researchers demonstrated that ordinary, undeclared choices about when to stop collecting data, which conditions to report and which covariates to include could produce a statistically significant result for an impossible hypothesis. The direct measurements followed: of one hundred psychology studies repeated by independent teams, 97 per cent of the originals had reported significant effects and 36 per cent of the replications did; of twenty-one social-science experiments published in Nature and Science, thirteen replicated, at about half the original effect size. The named mechanisms — p-hacking, HARKing, the garden of forking paths, and the publication bias that leaves null results in the file drawer — are all failures of ordinary practice rather than of honesty, which is why the reforms were aimed at procedure.

Outright fabrication surfaced alongside them and was caught the same way. A widely reported 2014 study on canvassing and attitude change was withdrawn after graduate researchers attempting to extend it found that the survey firm named in the paper had run no such survey and that the data did not exist. The reform package that emerged — preregistration of hypotheses and analysis plans, registered reports reviewed before results exist, open data and open materials, and journal-level transparency standards — is now partially institutionalised, with adoption uneven across subfields and enforcement largely voluntary.

Social science method and anomalous phenomena

The reform movement had a proximate trigger that the field's own accounts mention and then step past. In 2011 the social psychologist Daryl Bem published nine experiments in the Journal of Personality and Social Psychology reporting that subjects' responses were influenced by stimuli selected at random only after the response had been recorded — precognition, obtained with the discipline's standard designs and standard statistics and passed by its flagship journal's ordinary review. Three preregistered replication attempts of one of the experiments failed, and Bem and colleagues later assembled a meta-analysis of ninety experiments from thirty-three laboratories reporting a small but statistically significant effect. The episode is ordinarily recounted as proof that the standard methods were too permissive, and it drove much of the machinery that followed. A standing reply among those who hold the psi results genuine is that the methods were not revised until they produced a conclusion the field could not accept, so that the revision answered the result rather than the procedure.[citation needed]

The same asymmetry is the substance of the best-documented statistical dispute the trade has had over anomalous data. When the government remote viewing programme was reviewed for closure in 1995, the American Institutes for Research commissioned assessments from the statistician Jessica Utts and the psychologist Ray Hyman working from the same body of experiments. Utts concluded that by the evidentiary standards ordinarily applied elsewhere in science the effect was established, its size consistent across laboratories and far outside chance expectation. Hyman accepted that the departures from chance were real and not attributable to the methodological flaws he had previously identified in the literature, but held that an unexplained departure from chance is not yet evidence of a psychic faculty and that independent replication had not been established. The programme was closed.

Two things follow for the field's handling of anomalous claims generally. The methods set out above are the instruments through which claims about remote viewing, extrasensory perception and consciousness-mediated effects are adjudicated, so the trade's internal standards decide in practice what counts as evidence in parapsychology and its neighbours; and the standard of proof those claims are held to — preregistration, independent replication, pre-specified analysis, hostile review — is one the policy-facing research on belief and disinformation has not itself been required to meet.

Research on belief, disinformation, and radicalization

The applied fields that expanded after 2016 — disinformation studies, research on conspiracy theory belief, and the study of online radicalization — are the branch of the trade whose output goes most directly into enforcement. Their findings underwrite platform rule-making, deplatforming decisions, algorithmic demotion, and the labelling of legally protected speech as misinformation — categories whose boundaries are themselves in dispute, and whose application is the machinery of social media censorship and internet censorship more broadly within the information ecosystem. The centres that produce them — the Stanford Internet Observatory, the University of Washington's Center for an Informed Public, the Atlantic Council's Digital Forensic Research Lab — have also served as the analytic arms of content-flagging consortia, most visibly the Election Integrity Partnership. The recurring methodological objection to this branch is not that its conclusions are unwelcome but that its designs are said to be incapable of producing any other conclusion.

Definitional circularity

The objection Weinstein presses hardest is definitional. A study of "conspiracy theory belief" or "misinformation susceptibility" fixes its dependent variable against a list of claims the researchers have classified as false in advance, so that what is measured is deviation from the positions of institutional authorities — the CDC, the WHO, the relevant agency — rather than deviation from the evidence. Where the authorities are right the two coincide; where they are not, belief in a true proposition is scored as the pathology under study, and the question is settled by definition instead of by inquiry — the effect that gives closing a debate its cost. The classification also travels: the labels a study assigns become the vocabulary applied to the belief and the frame carried into press coverage of it. The COVID-19 lab leak hypothesis is the standing example: it was coded as misinformation in instruments fielded before the Department of Energy and the FBI assessed a laboratory origin as the likelier one, the reclassification having followed rather than preceded the studies built on it. Neither those instruments nor the studies resting on them appear to have been rescored or withdrawn in consequence.[citation needed] A common reply among those who defend the instruments is that no measurement of belief can avoid a criterion, that the best available consensus at the time of fielding is the only criterion available, and that the alternative — letting each respondent supply the standard — measures nothing at all.[citation needed]

Sampling and generalizability

Claims about how many people in a national population hold conspiracy beliefs are routinely extrapolated from samples no survey methodologist would accept for a prevalence estimate: undergraduate participant pools and online crowdworker panels such as Mechanical Turk. This is the trade's general sampling problem operating where the stakes are highest, since the percentages produced are what reaches press coverage and policy briefings, and the well-documented unrepresentativeness of such pools is not usually carried through into the headline figure.

Ideological composition of the field

The demographic fact is not seriously in dispute, and it was established from inside the discipline rather than against it: the social-psychological professoriate that supplies much of this research is politically homogeneous, and the mechanism its own methodologists identify is not fraud but question selection — which hypotheses are posed, which are funded, and which congenial findings are never subjected to hostile replication. The framing of the field's central question, "why do people believe conspiracy theories?" rather than "are any of these allegations true?", embeds an answer in the problem statement. The same skew runs across academia more broadly.

Mike Benz adds a career-structure argument to the demographic one: before 2016, on his account, there was no full-time, well-paid occupation in policing online speech — "you could not get a full-time job getting paid hundreds of thousands of dollars to censor what other people say on the Internet" — whereas the field has since become a track placing its practitioners "right at the heart of the Pentagon, the State Department, the CIA", which he reads as a standing institutional interest in findings that justify more of the work.

Funding and the research-to-enforcement loop

Research on disinformation, conspiracy belief and online radicalization is funded disproportionately by government agencies — DARPA, the National Science Foundation, the State Department — and by private foundations with programmes in content moderation. According to Benz, the same institutions then set the research agenda, conduct the studies and administer the enforcement those studies recommend: the Stanford Internet Observatory, the University of Washington's Center for an Informed Public and the Atlantic Council's Digital Forensic Research Lab were all partners in the Election Integrity Partnership, which ran flagging operations against posts during the 2020 election while publishing the research that justified the flagging — a closed loop, in his reading, rather than a sequence of independent judgements. Renée DiResta, who ran the Stanford end of that work, describes the funding as ordinary federal science money, the flagging as reporting rather than enforcement, and the loop as a construction of the campaign against the researchers rather than a description of what they did; the campaign, on her account, ended with the observatory wound down.

Operationalization

Studies of algorithmic radicalization commonly operationalise the outcome as content consumption — a user who watches videos a coder has classified as extreme is counted as radicalized — which conflates what a person watched with what a person became and leaves claims about "radicalization pathways" without a behavioural outcome that could fail. On this point the field's own later work has largely conceded the ground: large-scale audit studies of YouTube found consumption of extreme content concentrated among users who already held the corresponding attitudes and arriving by subscription and off-platform referral rather than by recommendation, with the algorithmic pathway accounting for very little of the traffic.

Instruments for "conspiracy theory belief" carry the parallel problem at the item level. Standard scales mix demonstrably false propositions with ones that are contested, partly true or since substantiated — bulk government surveillance of citizens, industry influence over the agencies that regulate it, the laboratory origin of COVID-19 — and score them as interchangeable indicators of a single disposition, so that a respondent who is right about the substantiated items registers as more conspiratorial than one who is wrong about them. Such instruments carry the trade's general replication exposure with an additional loading, the questions being politically valent and the professoriate that fields them not being politically mixed.

Weinstein reads the whole pattern as structural rather than accidental: an incentive system in which particular problems are owned by the institutions best placed to study them, so that the research reliably stops short of the finding that would cost its patron something — "does an arms maker want a more peaceful world? Does a health care system want nutrition to decrease the number of patients who walk through their doors. All of these are owned problems", his own difficulty being that he keeps trying to solve someone else's owned problem.

Open questions

References

Source ratings:

(Color-coding) (

[1 Wikipedia-grade]

·

[2 Questionable]

·

[3 Reporting/Analysis]

·

[4 Speculative/Hearsay]

·

[5 Ungrounded]

). (To change source ratings,

edit this page

.)