It has been argued (Van Casteren & Davis, 2007), that a fully automated procedure to select stimuli is desirable to avoid the experimenter bias
described by Forster (2000).
This paper critically examines the methodology of informal experiments employed in ordinary language philosophy and much of contemporary philosophy of language and linguistics, and discusses the role that experimenter bias
can play in influencing judgments about informal and formal linguistic experiments.
By tightly controlling procedures and observations, experimenters make sure that their data are as unaffected as possible by experimenter bias
Moreover, the African studies were closer to a lowest common denominator than a "gold standard," suffering from numerous critical flaws including selection bias, randomization bias, experimenter bias
, inadequate blinding, supportive bias, participant expectation bias, lack of placebo control, inadequate equipoise, attrition of subjects, failure to investigate non-sexual HIV transmission, lead time bias, and time-out discrepancy.
As such, there may have been an effect of experimenter bias
on the outcome of the behavioral measure.
Burnham JR (1966) Experimenter bias
and lesion labeling.
Spelke dismisses the study, noting that it was not peer reviewed, has never been replicated, and did not control for experimenter bias
Despite a few threats such as experimenter bias
, pre-test treatment interaction, and instrumentation, the study is significant.
A number of conclusions drawn from the reviewed studies were unfounded because the experimenters inferred causation from correlation, lacked standardized measures, had poor sampling procedures, failed to control for threats to validity, had very limited assessment of anxiety, had experimenter bias
, and had poor operationalization of religious constructs.