How do you measure a play the data cannot name?

By Dr. Chester Ismay · November 3, 2026

The tush push is the most argued-about short-yardage play in the NFL: a quarterback takes the snap and teammates shove him forward. In episode 2 of The Monday Morning Agent, an AI analyst searched the play descriptions and found no play with that name, so it had to write a definition before it could measure anything. This post asks what a data scientist would ask next. How good is that definition? Does the result survive adjustment? And does a higher conversion rate mean what people assume? The edge is real and larger than the show reported, because this work found an error in my own tool. Nothing here settles whether the play should be legal.

The error I found in my own tool

Rebuilding the analysis in Python and in R, I could reproduce the episode's numbers exactly, and I could also see why they were wrong. The tool's filter line says that two-point tries are out, but 195 of them stayed in: 60 on the defined side and 135 on the other. A two-point try has no first-down flag and is not marked as a touchdown, so the tool counted every one of them as not converted, including the ones that succeeded.

The episode's verdict ("holds, as a steady edge") does not change. Its size does. I have fixed the tool and corrected the episode's pages, and the agent's ruling stays there as it was given on air, with a note. Everything below uses the corrected plays.

Testing a definition against human charting

The agent's definition is a rule the data can apply: any run by the team's primary passer, not from shotgun, with one yard or less to go. It is a proxy, and a proxy can be tested if a separate label exists. FTN Data's charters mark quarterback sneaks by hand, so I joined their flag to 6,020 of these rushes from 2022 to 2025.

Precision is 89.6% (95% interval 87.7% to 91.2%), and recall is 89.1% (87.2% to 90.8%). The errors are informative. Of the 122 plays that match the definition without being charted as sneaks, 59 are scrambles, which the rule cannot exclude because it has no scramble clause. Removing them raises precision to 94.3%. Of the 128 charted sneaks the definition misses, 124 were run by someone other than the primary passer, such as a backup or a tight end taking a direct snap.

The bigger limit is what neither source can see. FTN's flag says sneak, not push. So this test validates "quarterback sneak," and the data still cannot tell a pushed sneak from an unpushed one.

The answer barely depends on the definition. On the same four seasons, the rule reads +11.1 points and the charted sneak reads +10.7. A rough definition is fine when, as here, you can show that its errors do not move the answer.

The raw gap understates the edge

The defined play is not called in the same situations as other rushes. 81% of defined plays come on third or fourth down, against 57% of other rushes, and the first two downs are a different kind of decision.

So I compared the two groups within each combination of season, down, and goal to go, and combined the differences with Mantel-Haenszel weights. The gap rises from +10.1 to +12.6 points. Plays from one team are also not independent, so I got the interval by leaving out one team at a time: +9.5 to +15.7. That is the headline answer, as the plan set it in advance, and it excludes zero. Without the Eagles it is +11.7 (+8.8 to +14.5). Leaving out any single team keeps the adjusted gap between +11.7 and +13.0.

The comparison group can change the story

"Every other rush" is a mixed group, so I also compared the defined play with each part of it. One row looked like a major finding until I checked it.

Against shotgun runs by the same quarterback, the gap is only +2.1 points (−2.6 to +7.4), which suggests that quarterbacks convert short yardage well from any alignment. But that group includes scrambles, and in the charted seasons the 59 scrambles in the definition's errors converted 100% of the time. That rate is not believable as skill. A likely reason, which I have not tested, is that a failed dropback gets recorded as a sack or an incompletion, not a failed scramble, so only the successes appear as runs. Without scrambles, the group converts 73.4% on 229 plays, and the gap returns to +8.2 points (+2.5 to +14.6). I added this row after seeing the first result, so treat it as exploration.

This is not one team's play

The argument usually centers on Philadelphia. The Eagles do run the play the most (166 times) and show a raw gap of +15.1 points. But team-level gaps are noisy, so I fit a random-effects model to the 29 teams with at least 20 defined plays. The mean gap is +11.5 points (+8.8 to +14.1), and the spread between teams is small: tau is 2.6 points, and I² is 13%. Once each team's estimate is pulled toward the league according to how noisy it is, Philadelphia's falls to +12.3.

These data do not separate Philadelphia from the league. They also could not show a team difference that is small relative to this much noise.

Conversion rate is not the same as value

A conversion rate counts every first down the same. Other rushes gain more yards, and they fail more often. So I compared several outcomes with the same adjustment and the same team-based interval.

The defined play gains fewer yards and is still worth more per play. A likely reason, which fits the table but is not a separate test, is that it fails less often at a distance where failing is expensive. Lost fumbles run about twice as high in the raw rates, though the adjusted interval includes zero. Expected points come from nflverse's model, so this comparison inherits that model's assumptions.

The season-to-date numbers point the same way. In 2026 through the games of November 1, which I held out and read last, the corrected gap is +12.7 (+4.6 to +19.6) on 145 defined plays, and +15.7 (+7.3 to +24.1) when adjusted. This season is no test of the Eagles' version: Jalen Hurts has missed their games since week 6, and 3 of those 145 plays are theirs.

What this does not show

This is a measured association, not the effect of calling the play. Coaches choose it when they like the look, and defenses choose how to respond, and neither choice is in these columns. The adjustment does not cover field position beyond goal to go, the score, the clock, or the personnel on the field. FTN's label covers sneaks, not pushes, so "tush push" remains an inference throughout. The data say nothing about injuries, and the case for or against a ban rests partly on questions of safety and fairness that no column here can answer. The definition rows and the scramble row were added after I saw results, as the plan records.

What carries over to your own analysis

Many business metrics are proxies too: "active user," "qualified lead," and "resolved ticket" are all definitions someone wrote down.

• Validate a proxy against a label, even a small one. A few thousand hand-labeled records can tell you whether your definition's errors change the answer.
• Rebuild important numbers independently. Two pipelines agreeing is useful. The moment they should agree and a filter's description does not match its behavior is more useful still. That is how the error in this post surfaced.
• Check the outcome behind the rate. A higher rate can still be the worse choice if it costs more when it fails, so measure value as well.

If your team relies on a metric like these, I would be glad to talk.

Method, code, and checks

Data: nflverse play-by-play, CC BY 4.0, and FTN Data via nflverse, CC BY-SA 4.0, used here only as aggregates. Conversion means the play gained a first down or scored. Intervals are Wilson for rates, Newcombe for gaps, a leave-one-team-out jackknife with a t quantile for adjusted differences, and DerSimonian-Laird for team differences. Every table is computed separately in Python and in R. Across both posts, the two agree on all 1,314 cells in 20 tables within 1e-9, and the corrected rows match the show's tool, now that it is fixed. Every number on this page is read from those tables when the page is built. I built this analysis with Claude Code, Anthropic's coding agent, and checked it with the tests and reconciliations above. The code and tables are on GitHub.

The terms on this page

Conversion

The play gained a first down or scored a touchdown.

Under center and shotgun

Two places the quarterback can stand. Under center means directly behind the center, taking the ball by hand. Shotgun means several yards back.

Primary passer

The player with the most pass attempts for that team in that game.

Precision

Of the plays the definition picks, the share the human charters also marked as sneaks.

Recall

Of the plays the human charters marked as sneaks, the share the definition picks.

Mantel-Haenszel

A way to combine differences across groups so that a comparison happens inside each group, with more weight on groups that have plays on both sides.

Random-effects model

A model that treats each team's true gap as a draw from a league-wide distribution, which separates real differences between teams from noise.

tau

The estimated spread of true gaps between teams, in the random-effects model.

I²

The share of the observed spread between teams that the model attributes to real differences and not to noise.

Expected points added

How much a play changed the number of points a team can expect to score next, from nflverse's model. EPA for short.

Scramble

A called pass where the quarterback runs instead of throwing. The data count it as a rush.

95% interval

A range computed so that, over many repeated samples, about 95% of such ranges would contain the true value.