PUL's Pulls
Some data and analysis on pull strategy
I have some thoughts on how teams should be handling pulls, so to try to get some data to back up my opinions, I recorded stats for the 35 pulls from the 2022 PUL Championship game (DC Shadow vs. Medellin Revolution Pro).
The conclusions I could draw just from the data weren't as clear as I would've liked, for reasons I'll discuss. So, this article will be partly data-based but mostly analysis of what I see in the game film.
Let's get into it.
1. Data and explanations
This section is the boring one, where I explain my methods and define the vocabulary I use. I've uploaded the raw data to a Google Sheet here.
Here's the meanings of some words I'll use both in this article or in the linked data:
"Opportunity". One thing I was interested in is, how do teams take advantage of an opportunity to gain yards without the defense being set? Some of the data I've normalized "per opportunity". An "opportunity" to advance the disc is a pull that lands in bounds, and is long enough that the receiving can get to it, before the defense is set. The pulls excluded are ones that land out of bounds, or land & stop more than 30 yards from the end line that the receiving team is standing on.
(Note: DC has 5 pulls that don't give Medellin an opportunity to advance: 3 extremely short, and 2 out of bounds. Medellin has 6 pulls that don't give DC an opportunity: 5 out of bounds, 1 very short.)
"Raw". Any data labeled "raw" refers to every single pull, even the ones that are out of bounds or short, and not just pulls where there was an opportunity to advance the disc.
"D is set". Since one of my goals was to see how the offense took advantage of the time before the defense was fully set, I had to define what it meant to be set. This is one place where the data is messy, and I had to make judgement decisions about who was or wasn't being guarded. That said, I've defined "D is set" as the first pass where both the thrower and receiver are being guarded by a defender. (For example, if one person on the defense runs down to cover the pull, and marks a person who is throwing to an uncovered receiver, the D wasn't set.)
"Free yards" are yards gained before the defense is fully set. For example, if the receiving team catches the pull 5 yards from their endzone and moves the disc to a spot 25 yards from their end zone before the defense is covering them, they've gained 20 free yards.
"[Time to/Location of] [catch/stop]". I didn't record the hangtime or distance of pulls in the air, but rather the time/distance until the disc was in the offense's hands. So a pull that lands 15 yards from the receiving end zone, and rolls/skids until it was 6 yards from the receiving endzone will be recorded as being caught/stopped 6 yards from the receiving end zone.
Note that this means data of this type can be a combination of a few things: both how far the pull is thrown and how aggressive the offense is about going to pick the disc off the ground. More on this later.
Finally, it's important to note that the lines were painted on the field wrong in this game, and the circled 'X' that looks like it should be the brick mark is actually too far forwards. In this pull, you can see the players take a bricked disc to a spot a few yards before the circled 'X'. In recording the data, I assumed that the 'X' mark is 25 yards downfield, and the actual (unmarked, at least to my eyes) brick mark is 20 yards downfield. I used these landmarks to approximate the yardage data I recorded.
All that said, here's a summary table of the data I'll discuss in the rest of this article:
Note that the columns are arranged based on the Pulling Team, but some of the rows refer to offensive stats. So if you want to see how many "free yards" Medellin's offense gains after a DC pull, you need to look in the column labeled DC.
2. The data is always messy
As I referred to above, some of the data points I recorded were somewhat subjective (was the D set, or not?) and that has an effect on the conclusions I'm able to draw. Let me give you the most obvious example.
Watch this play where Medellin pulls to DC with the score 3-5:
Arguably, DC catches the pull 6 yards from the endzone, and the defense isn't "set" (according to my definition above) until 39 yards from the endzone—DC did a great job getting those free yards, right? This one data point was more than 20% of DC's "free yards" gained.
The only problem is, it's pretty obvious that the reason Medellin was so "slow" to cover the pull was not that they were slow jogging down the field to cover the pull, but that they miscommunicated on defensive assignments, resulting in a blown coverage. In the screenshot below, I've labeled the 5 defenders who run to guard 4 players, leaving DC with an uncovered player downfield.
So should I give DC those 33 free yards, or try to guess how many free yards they would've gotten if Medellin's D didn't miscommunicate?
I'm going to include these yards in the data, because I'm going to argue below that Medellin is superb in pulling situations, and I'd rather give you a conservative estimate of their excellence than be accused of adjusting the data to make my opinions look stronger.
3. Medellin has longer pulls
Medellin is widely regarded as having some of the best throwers/best pullers in the world, so this isn't much of a surprise. When DC was pulling, Medellin on average caught or stopped the pull 21 yards from their endzone. With Medellin pulling, DC was getting the disc in their hands only 13 yards from their endzone.
4. Medellin's offense aggressively gains free yards
This is a bit subjective in the data, but I think between the data and the film it's clearly true: Medellin is aggressive with moving the disc forward before the defense gets set.
Here's what the data says: both teams, on average, gained about 12 "free yards" on the pulls where they had an opportunity to advance. (If we adjust the data point where Medellin has a blown coverage, it's more like a 1.5 yard advantage in Medellin's favor). However, Medellin is catching the disc 9 yards closer to the DC defense, and still manages to gain just as many yards!
There could be a few reasons why this happens, and Medellin's offense being aggressive is just one of them. Another possibility is that Medellin, when pulling, does a better job of running downfield to cover the pull and set their defense. There might be some truth to this, but if so, it's not obvious—by the data, Medellin takes about 2 seconds longer to set their D (still 1.5 seconds longer even if we throw out the blown coverage play). This is exactly in line with both of the teams covering the pull at about the same running speed, but Medellin needing to run 9 yards further.
Also, since DC's pulls are shorter, and on average wind up in the offense's hands (slightly) sooner, they would need to run slightly harder to get down the field in time. I'm not sure about the math here: does Medellin needing to run a longer distance cancel out with DC needing to run slightly quicker?
But overall, I think it's pretty obvious that Medellin is aggressive with getting to the pulled disc and getting as many free yards as they can. Watch this pull, where you can see the Medellin offense literally sprinting forward to pick up a short pull.
Compare that to this pull from Medellin that lands in the center of the field, about 10 yards from the endzone, and DC doesn't even try to catch it:
Of course, these are two of the most extreme examples, but they showcase what I see as a pretty clear trend.
For another example, here's a pull where DC uses a centering pass, but then there's no one nearby looking to catch another pass, even though the defense isn't set. It seems that, philosophically, the DC offense is thinking "after the centering pass, we set up our offense". But the Medellin offense is thinking, "let's keep getting free yards until they stop us, then we'll set up an offense".
Look at the screenshot below, where the DC player is about to catch the disc with lots of free space in front of them...but with no teammates interested in taking advantage of that free space:
Another aspect of this "aggressiveness" is that Medellin does a great job giving the disc to the open players. Covering the pull against Medellin has to be a full-team effort. If not, Medellin will get free yards using whichever players haven't been covered yet. In the next section below you can see a screenshot of a play where a few players on DC have run down to cover the pull, but they're covering players who don't have the disc, while Medellin just gives the disc to uncovered players to continue racking up free yards. They are flexible in taking advantage of what the defense is leaving open for them.
5. Medellin isn't scared to work it down the sideline
I went into this analysis knowing just from watching the game that Medellin was generally good at getting the free yards. One thing that stood out to me analyzing this game that I hadn't noticed before is how Medellin doesn't worry about "centering" the disc, and instead is happy to work it down the sideline if that's where the free yards are being given to them.
In my opinion, in a high level of ultimate like this where you can trust your players to dump it off the sideline, Medellin is making the right decision by getting the yards.
Here's a pull where Medellin gains yards going down the sideline:
In fact, you can see that the DC defense ran down the center of the field, while Medellin is moving the disc towards the sideline continuing to get free yards:
This pull is another example where Medellin works the disc down the sideline, in the process passing by DC defenders who are in the center of the field. They get past half field before they're fully being defended! Watch it here:
6. DC could do better handling pulls on the sideline
Two of the plays where the DC offense started closest to their own endzone involved pulls near the sideline that weren't caught. Watch this one:
It's a very catchable pull, but because the person catching it started near the center of the field, they didn't manage to run over to the sideline in time to catch it.
This pull is another play where DC got pinned deep on the sideline, although I have a little more sympathy here since it was extremely close to going out of bounds. But knowing that pulls landing near the sideline are going to be one of the worst outcomes for DC, the way they field the pulls looks suboptimal.
Look at this screenshot, taken from the first pull in this section:
All seven of the DC players are standing extremely close together! Mosquera puts the pull in the farthest spot from them, and they are slow to react.
I would suggest to have a system with three players tasked with catching pulls: one player near each sideline, and a third to catch most of the pulls in the center of the field. If you start standing near the sideline, it'll be much easier to mentally calculate whether a pull is going to land out of bounds.
7. Throw the dang disc out of bounds
I'm not the first person to say this, and I won't be the last. This Understanding Ultimate post has a similar argument to what I'm about to say, which is, pull the dang disc out of bounds if you care about winning. The brick mark is 20 yards from the endzone line. Medellin, on average, didn't face a set DC defense until they were 29 yards from the endzone line. That includes pulls where Medellin made it 39, 40, 43, and 45 yards from their own endzone before DC's defense was fully guarding them (yes, past half field).
Only one pull out of 17 ended with DC's defense being set closer than 20 yards from the end zone. Every other pull would have been improved (or at worst just as good) if it had been thrown out of bounds past the brick mark. If we care about "actually winning frisbee games" and not just "looking like we're skilled at frisbee", we need to be serious about pulling out of bounds.
DC, on average, was facing Medellin's defense 21 yards from the endzone. And Medellin had more positive outliers (DC facing a set defense 5, 8, and 10 yards from the endzone) than they had negative outliers (31 and 32 yards from the end zone). So Medellin makes the right choice in general by pulling in bounds. DC, I believe, is making a strategic error by not trying harder to throw the disc out of bounds.
As pointed out in the Understanding Ultimate post, the best situation is to have the disc roll out of bounds. But even throwing it out (past the brick mark) in the air would be a huge improvement for DC's pulling game.
8. Do pulls even matter?
Some people, like the Understanding Ultimate post I linked above, take it as pretty obvious that forcing the offense to have a longer contested path to scoring is good for the defense:
Which of these [pulls] do you think is a better situation for the defence? The one that is ~5-10 yards longer, and quickly picked up in the middle of the field, or the one that is slightly shorter but rolls out of bounds, and is fetched, and played from tight to the sideline?
But others argue that pulls don't really matter. Well, at least I thought that when I remembered a Reddit thread I had seen recently. In this thread, Reddit user mdotbeezy starts a comment saying "Pulls don't matter" (there is also some more discussion in this thread and in this one). But, it turns out that their later comments add a little more nuance, to the point that I only partly disagree with them.
Here's the part I do disagree with:
...field position isn't important for good teams...the odds of scoring from 5 yards back in your own endzone and from the brick mark aren't very different. There's a reason teams on the goal line will routinely dump the disc repeatedly back to midfield - because possession > position...
...And no, AUDL is not different enough that it's "disingenuous".
Here's my response:
1. First, I appreciate a little exaggeration, but I doubt that anyone can find me a filmed example where a team is "on the goal line" and "dumps the disc back to midfield" and does it "routinely". (Though, this will partly be because any defense good enough to force a defense to dump it back to midfield is probably also good enough to stop that team from getting to the goal line in the first place) Yards are easy to gain, but they are not infinitely easy to gain. The more yards a team needs to gain (with defensive pressure), the more often they'll turn it over before scoring.
2. I strongly disagree that the AUDL is "not different enough". The best AUDL teams have an 80% hold rate. Last year's club national championship game had a 92% hold rate! (2 breaks out of 25 points) The hold rate in the PUL championship game? 56% (I count 14 breaks out of 32 points).
It seems obvious to me that, yes, if a team can maintain possession indefinitely and patiently work the disc down the field, it doesn't matter that much if you make them work for an extra five yards before scoring. But in a game where there will be lots of turnovers, gaining extra yards will help the team win.
One semi-mathematical way to think about it: on each possession, a team goes a certain number of yards before turning the disc over. You can plot the probability of them going a certain number of yards before a turnover on a bell curve. With some AUDL teams having a 60+% clean hold percentage, they might go, on average, 120 yards before a turnover. In this game, there was a 56% hold percentage, and certainly more turnovers than just the one per point required for a break, so maybe the average was around 60 yards before a turnover.
With some made-up numbers, we can calculate the probability of each team making it more than 80+ yards before turning (fielding a pull on the goal line & going to score) vs. making it 70+ yards before turning (fielding a pull 10 yards from the end line and marching down the field to score). In other words, how much does the chance of a turnover increase when you need to go 10 extra (contested) yards?
Using our made-up AUDL numbers, the chance of scoring falls by only about 3%. While using our made-up PUL numbers, the chance of scoring before a turn falls by 13%. Even though everyone's playing the same sport, 10 yards could become 4x more important just based on which part of your team's performance bell curve you're in. This is shown in the images below by the different size (area) of the two red boxes. (I used this calculator for the numbers in this paragraph & the images below.)
This is obviously a very simple and imperfect model, for a number of reasons (for example: some yards are harder to gain than others). But I still wanted to share it to give some people a slightly better feel for what we mean when we say that field position matters more in a game with more turnovers.
3. Even small percentages matter. mdotbeezy's comment says that the odds of scoring "aren't very different". But, even if it's just a 1% benefit I can gain from better pull strategy, I'm taking it. If I'm trying to win, I make strategically winning decisions, and expect them to add up over the course of my playing career.
On the other hand, here's what I do agree with in their comments:
If anyone was trying to pin it to the sideline then that'd be an argument, but that rarely happens - people still value distance. Again, very few players are intentionally rolling the disc out of bounds or intentionally taking a brick, which would at least be defensible. Instead it's max distance, hope for the best.
This is exactly what Understanding Ultimate and I are arguing: throwing the disc as far as possible and hoping for the best is a bad strategy (especially in the PUL). Pulling the disc out of bounds (either directly or after bouncing in bounds) is often the right play.
To put it another way, I actually kind of agree that "pulls" don't matter. It doesn't matter if you can pull it 85 yards if your defense isn't set until the offense has worked it back 20 yards in the other direction. That's why I've focused my data recording on where the disc is when the defense is set — I think this obviously matters much more than purely how far the pull goes. So, I'd say that "pulls" don't matter but "pulling plays", i.e. the whole combination of the pull itself along with how the offense and defense manage it, is what matters.
One final point before summing this section up: the defense's fitness is an important factor here, too. I don't think teams should sprint down to cover the pull if they don't have the cardio to continue to play hard defense after that! And a team that has better cardio will both be able to cover the pull better and play defense better, in general. So I don't think DC should automatically invest more in covering the pull more aggressively—instead they need to balance the benefits of making the offense work harder against the negatives of using up their own energy running down the field. (But as I mentioned above, there was no clear evidence in my data that Medellin covered the pull any faster than DC did. Instead, I think it's clear from the tape that Medellin is mostly just smarter about taking advantage of pull plays).
To be clear, none of the data from this game is any kind of proof that being better on pull plays caused Medellin to win. There are lots of different things Medellin does well. I think it's obvious that pull plays matter (note that I'm not just saying "pulling longer matters", I'm saying "making strategically correct decisions on pull plays matters"). In other words, it's an assumption of this article that pull plays matter, not a conclusion of this article. And Medellin, among their many other areas of excellence, does a great job handling pull plays strategically (on both offense and defense).
That being said, I do think it's fun that Medellin won by two goals and also was better by about 8 yards per pull over 18 pulls...or just about two field lengths, matching the two goals they won by. Again, this is totally not science, just a fun coincidence! If I had to guess, I'd say the advantage they had on pull plays this game was worth 1/4 to 1/2 of a point. It's not that much, but give me those points every time.
2022-06-25 Edited to add:
In their Deep Look podcast, Ultiworld discussed this championship game, saying:
Charlie: [DC]'s offensive line ... was the weak link in this game...
Keith: [Medellin] got way too many opportunities from, like, midfield...I feel like that's where DC's offense really let them down, giving up a lot of opportunities where it just wasn't a lot of yardage for Revo to work, lets them go into that small ball game that they do so well.
But as I've discussed above, DC's defense, while doing other things well, also gave Medellin too many opportunities from midfield: five out of seventeen pulls ended with Medellin starting within 5 yards of midfield, with a few even past midfield.