When the AI says the teams are balanced and they are not
Five pull requests to footbolski, a match-organising app for our football group, including tracing why its AI team split contradicted its own explanation.
footbolski is the app our football group uses to organise matches: people register for a game, an admin keeps player ratings, and an AI splits the confirmed players into two teams. Between August and September I sent it five pull requests. The most interesting one taught me something about where an LLM belongs in a system.
The bug: reasoning that does not match the result#
Issue #30: the AI’s explanation said the two highest-rated players were on opposite teams, but the generated teams had both of them on Team A.
I traced the pipeline from the team service through the name-matching code to the prompt. The name matching was correct: it case-folds both sides consistently, so this was not a lookup bug. The real problem was that the reasoning text was saved and shown exactly as the model wrote it, and nothing checked its claims against the team lists the model returned in the same response. The prompt told the model not to put the strongest players together, but nothing enforced that.
My pull request added a deterministic backstop. After the AI’s split is mapped to real players, it checks whether the two highest-scoring players landed on the same side. If they did, it swaps the weaker of the two with the strongest player on the other team, keeping team sizes equal, and adds a note to the displayed reasoning so the admin can see a correction was made.
What the maintainer found next#
The maintainer tested the pipeline against a real upcoming match, five live runs. In four of them, the model put the two strongest players on the same team and claimed the opposite. Asked to rank the squad, it returned the correct order, so it knew who was best. And an exhaustive search over every possible split found a far more even one in milliseconds than anything the model produced.
Their conclusion: the backstop works, but it patches one rule and ships the contradiction to users with an apology attached. They pushed a redesign onto the same branch. The model no longer picks teams. It reads the scouting notes and returns structured judgments about each player. Code validates those judgments, a solver finds the actual best split, and only then does the model write an explanation, from the finished split. It cannot describe teams that were not produced.
The other four pull requests#
- Join WhatsApp group step. New players register in the app but are not necessarily in the WhatsApp group where matches are organised. I added a step to first-time registration: open the group link, then confirm. The confirm checkbox stays disabled until the link has been opened. It is a nudge, not verification, because WhatsApp gives the app no signal when someone joins.
- Aerial on player cards, a new Passing rating. The admin had to open every player’s detail view to see the Aerial rating, so I put it on the card. I also added a Passing rating for the team splitter to weigh and, at the admin’s request, removed eight free-text style tags the AI could not meaningfully compare. In the same area I found an import of a function that did not exist, which would fail at runtime and likely meant the AI split path was already broken, and fixed that too.
- Backfill for older player cards. The Passing column had no value for players created before it existed, so their cards showed “-/10”. A one-time database migration sets those to the default of 5. Its downgrade is deliberately a no-op, because undoing it could erase real ratings an admin had since set to 5.
- Existing members blocked by the WhatsApp step. On a new phone or in a private window, existing members were asked to join a group they were already in. The root cause: “new user” was decided purely from what this browser had stored. The fix also checks the typed name against the real player list, using a full-name match so a stranger who shares a first name with a member still sees the step.