Recently, I decided to check out the section “Skeptical Beginner“. Someone was asking for magical experiments. I would like to share some information in relation to their question based on scientific ideology that I have learned during my Research Methods courses.
If someone wants to determine the validity of a magical experiment, they should consider at least the bias called observer bias. Our observer is the person conducting experiments. Observer bias translates itself when the experimenter sees something that is not there, or doesn’t see something which is there (confounded variables).
There are ways to avoid observer bias. One method is called blind observer, in which an experimenter has someone else who doesn’t know what to look for make observations on the experiment. Another method is through multiple observers. Also, one can avoid observer bias by using objective measures. Objective measures means that while an experimenter can not necessarily infer why Y occurs, they can measure something like how long it takes for Y to occur after introducing X.
Two ways to test reliability are the test-retest, and ABA reversal techniques. ABA is when an experimenter first measures some kind of situation before applying any manipulations to it. After that they manipulate the situation somehow and take another measurement. They should hope to see a change (increase/decrease or appearance/disappearance) in some part of the situation over time. To be sure it was the manipulation which caused a change (not something else), our experimenter then removes or stops manipulating and takes another measurement. If the manipulation works, the line on a graph marking change over time should look like a letter U(upright or inverted). If one keeps applying ABA over and over, they should see a wavy line on a graph. ABA reversal can be really fun to test magical efficacy, but I don’t find it helpful when applying magic in relation to time.
Another bias that a magical experimenter might like to look out for is inferential bias. Inferential bias points to when an experimenter doesn’t know whether two things are related by correlation or causation.
I hope this sheds a little bit of light for someone who wants to make up magical experiments. A final note though, lets not forget about sweet intuition!
Sincerely,
Dave
