Baily Deeter '22
Introduction
Introduction: It is widely accepted that quarterback is the most important position in professional sports. Any team with a good quarterback will be able to move the ball on offense and score points at will, making them an instant contender. It’s no coincidence that some of the best quarterbacks of all time, such as Peyton Manning, Tom Brady, and Joe Montana, have all won multiple Super Bowls. In search of the next great quarterback, teams often use high draft picks to select the best quarterback from the incoming draft class. For some teams, it works. Manning, John Elway, Drew Brees and Andrew Luck were all selected in the top 5 of the draft, and all five have been phenomenal. However, Mark Sanchez, JaMarcus Russell, Ryan Leaf, and many other busts were also selected in the top 5 but have not performed as well in the NFL. Sure, players at other positions selected in the top five have proven to be busts, but it seemed to me going into this project that a lot of the biggest busts were quarterbacks. This led me to believe that teams might often draft quarterbacks too high because the payoff of landing a top-tier quarterback is so huge when they’re actually taking a player who really isn’t talented enough to warrant a high draft pick.
General Hypothesis
Quarterback is the most important position and bad teams (who theoretically are bad because they don’t have strong quarterback play) have the top draft picks. Therefore, teams will be more inclined to “reach” for a quarterback, translating to a lower mean career grade for quarterbacks relative to non-quarterbacks since they are often not as talented as players that they are picked ahead of.
Methodology
I decided to study the top five picks in each of the past 10 NFL Draft classes (2009-2018), giving me 50 total players in my data set. I split up the 50 players into quarterbacks (14) and non-quarterbacks (36). To measure career effectiveness, I turned to a reputable website called Pro Football Focus that uses highly trusted analytics to grade each and every NFL player on each and every snap. PFF is used by all 32 teams, and NBC’s Sunday Night Football uses PFF’s grades to introduce each player in the starting lineup.
PFF gives each player a grade for every season, and while it hasn’t calculated career grades for players, I used this data to make that calculation by hand. I multiplied each season grade by the player’s snaps that season, added up these values for each of the player’s seasons, and then divided by the total number of career snaps. I did this because I wanted to place more weight upon seasons with more snaps, valuing each snap equally. Once I calculated a career grade for each player, I calculated the mean career grade and the standard deviation for the two groups (quarterbacks and non-quarterbacks) by hand. I then used a two-sample T-test to compare the two means. I used a T-test instead of a Z-test because we don’t know the true population standard deviation and because my quarterback group didn’t meet the requirement of 30 samples.
However, after obtaining the results for the first test, I came across a key realization: not every top 5 pick is created equal. Theoretically, the top overall pick should be better than the fifth overall pick. Since PFF only has data going back to 2006, I originally chose the top 5 instead of the top 1 or 2 to increase my sample size. However, in seven of the 10 drafts I studied, a quarterback was selected first overall, and 12 of the 14 quarterbacks from the initial test were selected in the top 2 of their respective drafts. So, I decided to run a second test with the same methodology except for the key difference of only studying top two picks rather than top five picks. I wanted the sample size for the non-quarterback group to be in the double digits, so I expanded the sample size to players drafted in the top 2 since 2007. This gave me a group of 13 quarterbacks and a group of 11 non-quarterbacks, so I ran the same two-sample T test with these slightly altered populations.
In both cases, I elected to use a two-sided test. While I expected quarterbacks to have a lower career grade, it is also possible that quarterbacks could have had a higher career grade than non-quarterbacks for a number of possible reasons (such as being less injury prone or being given more time to develop). Also, a two-sided test is generally more trustworthy. I chose a standard 0.05 significance level before conducting each test.
Results
Neither of the two significance tests yielded a significant result at a significance level of 0.05, although both yielded p-values close to that level. The top-5 test generated a p-value of .1134 with a test statistic of –1.64, whereas the top-2 test generated a p-value of 0.0542 with a test statistic of –2.0402. The confidence interval for the difference of mean grades top-5 test was (-9.4731, 1.0731), whereas the confidence interval for the top-2 test was (-15.7505, 0.1505). The top-2 test yielded a particularly low p-value that was almost sufficient for us to reject our null hypothesis. Interestingly, if we had run a one-sided test at the same significance level, we would have been able to reject the null.
| Mean QB Career Grade | Mean Non-QB Career Grade | P-Value of 2-Sample t-test | Significant at α=0.05? | |
| Top 5 Picks | 69.5 | 73.7 | 0.1134 | No |
| Top 2 Picks | 69.3 | 77.1 | 0.0542 | No |
Conclusion
We were unable to reject the null hypothesis that quarterbacks selected in the top five perform just as well as non-quarterbacks. Our first of two tests compared the mean player grade of quarterbacks picked in the top 5 of the last 10 draft classes to the mean player grade of non-quarterbacks picked in the top five. For this test, we got a p-value of 0.1134, so we couldn’t reject the null at a significance level of 0.05. For our second test, I used the same methodology but with top-2 picks (and with a sample size of 12 draft classes) and got a p-value of 0.0542, which meant we still couldn’t reject the null. While our results were not significant, it was interesting that the p-values were close to the significance level and that the mean grade for non-quarterbacks was higher for both tests, indicating that this topic could be worth further exploration.