Setup
Astrology is very popular — both Gallup and YouGov report that about 25% of Americans believe that the position of the stars and planets can affect people’s lives, with an additional 20% of people reporting being uncertain about astrology’s legitimacy. If it’s true that a person’s natal astrological chart contains lots of information about their character or life, then it stands to reason that astrologers should be able to match people to their charts at a rate that is at least moderately better than random chance. If they can do that, then that would provide substantial evidence that astrology really works! Even if you’re not interested in astrology, per se, we think that designing a test of astrology serves as a nice example of the scientific method put into practice - that is, how to go from a claim to a scientific test of that claim. We’ll explain how that process works.
How do you test a claim scientifically?
Before we get into the details of our astrological charts study, let’s talk about how to test a claim scientifically. The methods of science, while imperfect, are some of the most powerful ideas ever invented by humanity. Here’s one approach for using scientific methods to test claims:
Make the claim precise: usually an ambiguous claim can’t be tested, or if a test is made, it’s hard to tell if it truly was a valid test of the claim. For instance, saying “Your month of birth says a lot about you” isn’t testable because it’s too ambiguous. But specifying “people born in February are more empathic than those born in other months” makes the claim precise enough to be testable.
Choose a measurement: to test the claim, you must be able to find something you can measure - the result of which will vary if the claim is true compared to if the claim is false. If you were testing the claim that people born in February are more empathetic, you could measure the empathy levels of people born in February as well as those born in other months using a well-validated empathy questionnaire.
Design a study: once you know what measurement you want to make, you need to design a study to actually make that measurement (while ruling out alternative explanations)…
Run the study and analyze the result: finally, you have to put the study into action by recruiting study participants, collecting the data, and analyzing it to see whether it supports or contradicts the original claim.
How do we apply the scientific methods to astrology?
To apply scientific methods to astrology, we first have to figure out what astrology claims… One of the most fundamental claims that astrology makes is: a person’s natal chart — reflecting the position of the celestial bodies at the time of their birth — offers insights about that person’s character or life.
Figure 1: If you’re curious what natal charts look like, here’s an example in two popular styles known as Placidus and Whole Signs:
If you’ve never seen an astrological chart before, you may find that it comes across as quite mysterious. Natal charts are generated based on a birth date, time and location. Reading such a chart requires specialized skill. Someone with no experience is not going to be accurate, even if astrology turns out to be effective. An additional challenge, when it comes to study design, is that astrologers differ in how they interpret these charts.
But since nearly all astrologers agree that natal charts can reveal insights about a person’s life or character, this is the claim we designed our study around, asking astrologers to identify which chart belongs to a given person from a set of options. What’s appealing about testing this core aspect is that mainstream science considers such predictions about a person to be impossible (since none of the known forces of physics could account for the relationships between an astrological chart and a person’s life). So if astrologers can actually do this task successfully, that’s a strong demonstration that they have a skill that science can’t currently explain.
How did our test for astrologers work?
Our test for astrologers consists of 12 multiple choice questions. For each, we show lots of information about one real person’s life… These questions were chosen by asking astrologers what they would ask someone if they wanted to be able to accurately guess that person’s astrological chart.
After showing this information about a study subject, we showed each astrologer 5 astrological charts. Only one of these is the real natal chart of that person (based on their birth date, time, and location), and the other four are “decoy” charts that were generated based on random dates, times, and locations. The astrologer’s task is to predict which one of the five charts is the person’s real chart.
If astrologers were randomly guessing (i.e., if they had no skill whatsoever), they would get 20% of questions correct (an average of 2.4 out of 12). If, on average, astrologers can get at least, say, 33% correct (4 out of 12) that would provide substantial evidence that astrology works. And even if most astrologers don’t do better than chance, but just one astrologer can get at least 11 out of 12 right, that would provide strong evidence that that astrologer has genuine skill
What do we mean by “astrologer”?
328 participants took our test, including 152 astrologers. We define an astrologer (for these analyses) as a participant with at least some astrology experience who also predicted, just before starting the task (but after the study design was explained to them), that they would perform better than chance (i.e., that they would get at least 3 out of 12 questions correct). The reason for these exclusion criteria are because if someone doesn’t have astrological experience, then their performance on this test obviously says nothing about astrology, and if someone with astrological experience does not believe they can do better than random guessing at the task, then it’s not fair to use their inability to do the task as evidence that astrology itself doesn’t work.
We recruited astrologers through a variety of methods:
We reached out to dozens of notable, well-trusted and influential astrologers telling them about the project and asking if they would like to participate in the challenge.
We promoted the challenge to our >200,000 newsletter subscribers as well as on our social media accounts.
We posted the challenge to a variety of popular astrology Facebook groups.
We also attempted to post the challenge to the two largest astrology subreddits, but unfortunately in both cases the administrators would not allow us to post the study there.