I am going to predict that, leading up to the NFL draft, you are going to hear many draftniks and fans say that (insert bad team name from last year here) should draft OT Joe Thomas and not take a QB or RB early in the draft, because you build the team the right way by going for offensive line, and some variation on QB's being more risky, and RB's being fungible.
Off and on over the next month and half leading up to the draft, I am going to try to look at the history of the draft to see whether this actually holds water. My guess, based on a quick review of the early first round draft picks over the last 25-30 years, is that it does not. In my opinion, a big part of the rationale for this is the same situation noted by Doug Drinen in his post "Reputation and Information", re-posted here http://www.pro-football-reference.com/blog/wordpress/?p=243. Except, it is the opposite side of the coin, since we don't have many meaningful stats on offensive linemen, the bad ones may not be perceived as bad as other positions where we have more statistics to measure.
Who is the bigger bust, Ryan Leaf or Robert Gallery? No-brainer, you say? I was at the KC-SD game at Arrowhead in 1998 when Leaf went something like 1 for 15. He was definitely a disaster. But living in an AFC West market, I have seen more than enough of Gallery as well. His performance in the season opener against San Diego was every bit as much a disaster, as he served as an express highway to the quarterback all game long. Could the Raiders horrible performance over the last few years, and inability to keep a qb standing for more than 2 seconds this year, have anything to do with drafting Gallery with their high pick a few years back?
Well, those are but two isolated examples. I am going to try to incorporate all the evidence, great pick, so-so, or bust. How to do it is the problem. As Drinen's article points out, it is not going to be possible to compare positions by raw statistics, because some QB/RB have lots of data, while others have little direct data. And I don't think Games Played or even Games Started is a viable option because of the differences in position, and how they are utilized. If a quarterback and offensive lineman are equally bad, the lineman will probably start more games. Take Mike Williams at Buffalo for example. If the Tackle is bad, he can move into the guard spot to see if he works there. He would have to be worse than roughly 5 other players to get benched. He may still get benched, but out of necessity and depth, he is probably going to start more games than a similar QB.
So, that leaves us with something that I think is the best we have at this point: wins and losses, and how the team performed in the years after a pick of a certain position was made. It is by no means perfect. One player alone does not determine a team's future. Nor is a team's success directly tied to the quality of one pick. If we have enough data points, some of that stuff will even out. Acknowledging it is not perfect, I am going to try to look at each position by the following method:
Look at all top 12 picks in the NFL draft since 1978 (the year of passing liberalization and expansion to 16 games), look at the record the year before the pick, and gauge the team's success over the next 5 seasons, by win-losses, percentage of team's making playoffs, and percentage advancing to at least a conference championship game.
I will not have the time to look at each individual's career with the drafting team. So, if the Packers start getting good at the end of the 5 year's after Tony Mandarich is drafted, even though he is out of the league, so be it. I guess I am not looking at how good the player drafted was directly, but rather, what the effect of drafting a certain position has on the franchise over the next five years.
Anyway, I will try to start with QB, OL, and RB, and move on to the others.