Using Statistics To Establish (Or Disprove) Relationships

February 2011

Statisticians describe independence as whether the occurrence of one event or characteristic makes it neither more nor less probable that other event(s) or characteristic(s) occur(s). The chi-square and Fisher exact tests described below are the most widely used tests for evaluating independence of variables.

The classic tool for teaching probability involves placing different colored balls into a container, and then randomly removing the balls. The objective is to determine the probability that a certain color of ball will be removed. If everything were to occur “perfectly”, each removed ball would always bear the same relationship as the starting percentage of each color of ball in the container. Of course, this cannot occur because (i) only one entire ball (not a percentage of a ball) is removed at a time, and (ii) random events could cause one color of ball to be removed with a greater frequency. With small sample sizes, the outcome of a single ball removal dramatically affects the percentage of a particular colored ball that has been pulled.

One could calculate the probability of every possible permutation of balls being pulled. This is possible when small sample sizes are involved, since the possibilities increase geometrically as the number of available options increases. For this reason, this method of performing the tests is usually limited to a two by two table, with small numbers in each quadrant. Such a test is called the Fischer exact test (named after its founder). The Fisher exact test calculates the probability of observing a particular table result or group of related results. The chi-square test approximates the Fisher exact test when larger number of observations must be addressed.

Most of the rest of this article involves using these two statistical tools to determine whether state budget challenges are related to how typically vote as “red”, “blue” or “purple”. There is a general notion that states in the biggest fiscal messes are the “blue” democratic states, generally because these states tend to have more significant unfunded (or underfunded) state employee pensions, and are more generous with social programs. However, a statistical analyses disproves the notion that there is a relationship between es in the largest fiscal messes and their political tendency; i.e., whether it is a blue, red or purple (battleground) state.

We previously described three different “solutions” to three states’ financial crises, and provided a list of the top ten states (i) projecting the largest shortfalls for fiscal year 2012, and (ii) with the largest 2012 projected shortfalls relative to their most recent budget. These lists are expanded in the following two tables to include the states’ political tendency. The classification of red, blue and purple (i.e., battleground) states was determined by compiling average margins of victory in the last five presidential elections. Three of these past elections were won by Democrats, Bill Clinton in 1992 and 1996, and Barack Obama in 2008; two were won by Republican George W. Bush in 2000 and 2004.

The following table lists the 10 (actually 11, since position 10 is a tie) states that are projecting the largest dollar shortfalls for fiscal year 2012, and each state’s political tendency based on average victory margins of the last 5 presidential elections.

The following table lists the 10 states with the largest 2012 percentage projected shortfall relative to their most recent budget and each state’s political tendency based on average victory margins of the last 5 presidential elections.

With large numbers and/or relatively large number of variables, an approximation of the “brute force” calculation performed in the Fisher exact test can be performed more easily. With larger observations, an approximation of the Fisher exact test is called a chi-square test. However, the chi-square test may yield unreliable results with tables that (i) contain less than 50 observations or (ii) contain less than approximately 5 observations in any cell in the table. In such circumstances, the Fisher exact test usually remains feasible and is reliable.

Both the Fisher exact and the chi-square test the hypothesis that the two groups (i.e., voting tendency and largest budget shortfall) are unrelated. To determine the validity of this hypothesis:

1. The chi-square compares the “actual” values (values that actually were observed) with the “statistically expected” values (calculated values based on the sum of actual values that one would expect assuming the hypothesis is valid). Based on differences between the actual values and the statistically expected values, the test result either accepts or rejects the hypothesis with a certain confidence level. Here is our online interactive chi-square calculator.
2. The Fisher exact test calculates the probability of getting deviations as extreme as the “actual” values (values that actually were observed) under the hypothesis that political tendency and largest budget shortfalls are not related. Based on the probability result, the test result either accepts or rejects the hypothesis with a certain confidence level.

The following two tables summarize the respective data in two tables shown above.

Because the data in both of the above summaries violates one of these chi-square reliability criteria (i.e., a cell contain less than 5 observations), both the chi-square test and the Fisher exact test were performed to determine whether or not a relationship exists between states in the biggest fiscal messes and states' political tendency.  Both statistical tests indicate no relationship exists at the 95% confidence level.

Stated otherwise, (i) using the chi-square test, the actual values are NOT significantly different than the statistically expected values at the 95 percent confidence level, and (ii) using the Fisher exact test, that chance of obtaining more extreme deviations from the observed values is not high enough to support the hypothesis that a relationship exists.  Importantly, these results remain despite the way the data is “sliced and diced” (e.g., use the top 5, top 15, top 20 states rather than top 10).

A separate article uses these same tests in the litigation-related application of whether employment discrimination is occurring (or not occurring).

Fulcrum Inquiry performs statistical analyses in litigation.