Turnover analysis has its limits

But it's still fine to do

Hive Ultimate has a series of videos analyzing turnovers from high-profile ultimate games (Melissa Witmer also had a recent Ultiworld article focused on turnover analysis). Here's Hive’s latest video, for example:

There's some insightful analysis here. I especially liked the discussion of how one of Germany's set plays was consistently ineffective. That's good, actionable advice.

A friend recently told me about a similar turnover-analysis-focused drill their club team did — the team stopped play after each turnover to discuss whether the right decision was made. I like this Hive series, but I do think there are a few fundamental flaws in doing an analysis like this. Of course I encourage people to be mindful about their turnovers. But there are a few points to remember before trying to take too much away from an analysis like this:

---

You can't always imagine the plays that didn't happen

I enjoy reading about economics, and a theme I've learned from there is that it's often important to consider things that didn't happen. A few examples might help explain what I mean:

Economists say that a downside of taxes is the things that they cause to not happen. If a 10% tax is charged on top of the cost of a $1000 iPhone, there are some people out there who are made worse off. They would have purchased (and benefited from owning) an iPhone at the $1K price point, but can't afford it at the higher price and choose not to buy. It's a transaction that didn't happen. It's not recorded in our statistics anywhere. But there is a loss (a person was unable to obtain something useful to them) due to the added tax. It's trivial to find a person who did buy an iPhone, but next to impossible to find the specific person who made the choice not to buy an iPhone at the higher price. In frisbee there are also 'important' events that we'll never be able to record statistically.

Another example in the news recently was the effect car seat regulations have on how many children a family decides to have (article here, maybe paywalled). It's relatively easy to track how car seats save lives, but much harder to track the lives that never even happened due to car seats.

A final example: here's a quote from the book The Captured Economy that discusses how the housing shortage in America's most economically vibrant cities have harmed our national economy:

As to the cause of growing geographic inequality, Moretti and Hsieh conclude that the main culprit is land-use restrictions. Truly striking is that the lion’s share of the harm is being caused by the highly restrictive policies of just three cities: New York, San Francisco, and San Jose. If regulatory barriers to new housing construction in those three cities had been pared back to just the median level of restrictions nationwide, Moretti and Hsieh estimate that the resulting influx of workers would have raised overall US output by 9.7 percent over the period in question.

Our whole economy could have been 10% bigger (assuming their calculations are correct). Everyone in the country could’ve had a 10% raise! But we don't feel it as a loss, because it's a gain that didn't happen, instead of a loss that did. If our job cuts our pay by 10%, we're going to feel that viscerally.

Thanks for sticking with me through all that! In frisbee, analyzing our turnovers suffers from the same flaw. We can see what did happen — the cut, the throw, the turnover. But we can't see what didn't happen. We can't see the cutter who could've started their cut earlier to maintain better flow. We can't know that the best decision was actually throwing a dump on stall 1. Some turnovers are probably "caused" by a series of throws that brought the disc to a disadvantageous position on the field, and not what the final thrower does with it once it's there.

Sometimes, I'm sure, the "actual cause" of the turnover is the thing that did happen. And our straightforward analysis is correct. But we have no consistent way to determine which turnovers are due to things that did happen, and which are due to things that didn't happen. And if we can't confidently say which is which, we can never be confident we're right when we say a turnover was caused by the thing that did happen and not the thing that didn't happen.

The Hive video does reckon with this, to some extent, for example the discussion at 2:30 mentions another option the thrower may have had. But for every "alternate future" that you can see on the tape, there are more that are fundamentally unknowable.

---

Overall efficiency is driven by game plan

The next issue with turnover analysis: turnovers emerge out of the team's offensive strategy in a way that can never be fully accounted for. One way to reduce your team's turnovers is to adopt a more effective game plan. But you can't look at any one specific turnover and attribute it to the game plan, because the entire path the offense takes would change.

I think Felix and the people at Hive understand this point—after all, a variant offensive philosophy is their site's main reason for existence! I'll make up some numbers to highlight the point: say Team USA had 15 turnovers in the game. But if they'd adopted and trained the Hex system, they would have only had (let's say) 12 turnovers. The problem is that we can't point out any specific turnovers and say that those three were the ones due to playing vert stack instead of Hex. The entire system is different in a way that's impossible to compare side-by-side —the particular turnover-causing arrangement of players+disc wouldn't have happened in a different system. Switching systems would help your team's turnover problems, but not in a way that's possible to understand through analyzing each individual turnover.

Although I couldn't find a quote that fits perfectly here, this section is inspired by Ben Taylor's book Thinking Basketball. (His YouTube channel by the same name has been in a number of my posts.)

---

Update (2024-01-29):

The play highlighted in my article Don’t lose more yards than necessary on dump throws is an example of the phenomenon I’m discussing in these first two sections. The US team turns it over at the end of the highlighted possession on a ~30 yard forehand that is just barely incomplete. But a few passes before that, they had unnecessarily lost 15+ yards on unpressured dump throws:

One of the “causes” of the turnover was arguably the dump throw that happened many seconds before the turnover was thrown. But at the same time, if the US hadn’t lost so many yards on dump throws, it wouldn’t have necessarily turned a 30-yard forehand incompletion into a 15-yard forehand score. The disc would’ve ended up somewhere else entirely! We can never be sure exactly what would’ve happened in this hypothetical universe.

---

Decision making is a spectrum, too

As I've pointed out before, decision making happens on a spectrum, just like throwing accuracy does.

The Hive video understands that sometimes people just make inaccurate throws. Five percent of anyone's throws are going to among their 5% least accurate. Watching one of these throws on tape, it's easy for most of us to not over-coach it. We say something like "good decision, poor execution", and move on. We understand instinctively that every once in a while, we're just going to have an inaccurate throw.

Decision making is fundamentally the same. Five percent of our decisions are always going to be at the "5% worst decision" level. When we analyze turnovers, we're almost always going to be drawing those examples from a player's worst decisions. But we can't magically remove the bad part of our decision making bell curve. We can only shift the entire curve towards better decisions.

But there's no video evidence of the result of our average decisions, so we're sort of stuck focusing on only the worst decisions we make.

For throwing accuracy, there's no magic shortcut to shifting our performance bell curve towards improved accuracy—we just have to practice throwing a lot. And there is similarly no magic shortcut to shifting our decision making bell curve towards better decision making other than lots of practice making decisions.

Telling someone "that was a bad decision, make a different decision next time" isn't fundamentally much different from telling someone "that was an inaccurate pass, make more accurate passes next time". My intuition is that these aren't exactly the same, and there actually is some "mental component" to the decision making that isn't there for the throwing accuracy. Watching a video like this could potentially help us redefine our sense of what is a good or bad decision in a way that helps us make better decisions. But I do think we tend to forget how much decision making and throwing accuracy are similar.

---

Expected value is what matters, but it’s unknown

In frisbee all that matters is whether we score. An individual decision shouldn't be judged only on the basis of whether it was "risky". Rather, the question is whether the expectation of the number of points we'd score on that possession has gone up or down.

A slightly riskier throw that greatly improves the team's chances of scoring can be more effective than an easier throw that doesn't help the team get closer to scoring. An 80% throw that leads to a 99% chance of scoring is better than an 85% throw that leads to a 90% chance of scoring.

There has been previous work done in quantifying which areas of the field have better expected value. But this is another issue that's just fundamentally very difficult to solve—a true expected value would require knowing not just where the disc is on the field but also the alignment of the players (and any expected value map is only valid for that particular team). We can make educated guesses, but no one really knows how any particular decision is affecting the offense's expected value of scoring. It's a much harder problem to solve than asking "was this throw a turnover?", so we often tend to just solve the easier problem instead.

As a specific example, let's go back to the same play I mentioned, at around 2:30 in the Hive video. The narration says "this throw is pretty risky, and not for a whole lot of gain". They understand we care about expected value—"risk" vs "gain". But without way more data tracking in ultimate, these are really just opinions. Perhaps the throw was a little less risky than it seemed to the video's author (our analysis of it is inevitably biased by the fact it didn't work). And perhaps the payout was higher than the author assumes—a 15 to 20 yard gain on the break side is nothing to sneeze at. We just don't know for sure.

I'm sure there's a good amount of correlation between the decision we can analyze and the unknowable changes to expected value. But again, it's a difficult problem to solve and should always make us a little more unsure whether we're analyzing a turnover correctly.

This is especially true in World Games-level competition, where turnovers happen in extremely tight margins, and the throwers are some of the best decision makers in the world.

---

I still like being mindful

Again, I don't think this type of turnover analysis is useless. Any player who wants to improve needs to reflect on the decisions they made, turnovers included.

I think meditation is very analogous to thinking about turnovers. We know we'll get distracted even when trying our hardest to focus during a meditation session—and we know we'll always throw turnovers even though we don't want to.

Meditation teachers tell us not to overreact when we get distracted—just note it calmly and return our focus where it belongs. Likewise, a calm attentiveness is the best approach when we turn the disc over.

Be mindful, but with a nonjudgmental attitude of gentleness and kindness.

---

Final thoughts

To recap, here's what I see lacking in turnover analysis:

• Turnovers can be caused by things that didn't happen, and thus don't show up on tape

• Turnovers can be caused by our overall game plan, and thus can't be optimized by analyzing individual turnovers

• Decision making is a spectrum, so suggesting someone "make a better decision next time" isn't much different than suggesting someone "make a more accurate pass next time".

• Analysis is driven by expected value, which we don't have the data to accurately judge. Obviously bad decisions are not so interesting to talk about, and borderline decisions have the uncertainty of not knowing what an accurate expected value calculation would say.

Turnover analysis has its limits, but it also has its place, as long as we use it carefully and don't forget about the assumptions we make when using it.