The Sideline Analytics Gap
Professional sports clubs spend thousands of dollars on multi-camera tracking rigs and specialized sensor gear to measure ball possession and movement heat maps. For amateur futsal players renting a community court on a Friday night, match analytics usually means arguing after the final whistle about who controlled the game.
During Challenge 3 at Apple Developer Academy, I wanted to explore a practical alternative: turning an ordinary iPhone placed on a sideline tripod into an autonomous futsal match tracker.
The goal was to run live ball detection and player skeleton tracking entirely on-device, calculating match analytics without relying on cloud connections or expensive dedicated hardware.
Sideline Tripod iPhone
│
▼
AVCaptureSession (CameraService.swift)
│
├── Dual Pixel Buffer Dispatch ──┐
│ │
▼ ▼
BallDetectionService.swift PoseDetectionService.swift
├── YOLO11m CoreML Model └── Apple Vision Framework
├── Calibrated Confidence (18 Skeleton Joints)
└── IoU Threshold Filter │
│ │
└───────────────┬────────────────┘
▼
SessionView.swift Overlay
(Real-time ball box + player pose wireframe)
│
▼
Post-Match Analytics & Heat Zones
The Real-Time Motion Blur Problem
Detecting a stationary futsal ball in a test photo is trivial for modern computer vision models. Tracking that same ball across a fast-paced 5v5 futsal match on a compact court is a completely different engineering challenge:
- Futsal balls move at high speed across tight spaces, creating severe motion blur across video frames.
- Players crowd around the ball during physical challenges, causing frequent partial occlusions.
- Floodlights and indoor shadows cause erratic lighting shifts across the court surface.
In our first prototype, we took a standard COCO-pretrained YOLO model, converted it to CoreML, and fed raw camera frames straight into inference using default confidence thresholds (0.50).
The result was unplayable. Every time a player kicked a fast pass, the ball blurred across four frames, the model’s confidence dipped to 0.38, and the bounding box vanished. A second later, the tracker reappeared, but latched onto a player’s white sneaker or a round spotlight glare on the shiny court floor. The bounding box jittered wildly across the screen.
How do we keep the bounding box locked on a fast-flying ball without mistaking players’ shoes for the ball?
Calibrating Confidence and IoU for Sports Velocities
To stop the flickering, I had to separate model capability from post-processing thresholds.
I selected YOLO11m over smaller variants like YOLO11n. The medium variant provided the extra parameter capacity needed to distinguish partially occluded balls from white shoes while running smoothly on the Apple Neural Engine.
In BallDetectionService.swift, I customized the Non-Maximum Suppression (NMS) parameters specifically for futsal dynamics:
final class BallDetectionService: ObservableObject {
@Published var ballBoundingBox: CGRect?
// Lowered threshold accommodates high-velocity motion blur
private let confidenceThreshold: Float = 0.35
// Tight IoU prevents duplicate boxes during close challenges
private let iouThreshold: Float = 0.45
private var visionModel: VNCoreMLModel?
init() {
setupCoreMLModel()
}
private func setupCoreMLModel() {
guard let modelURL = Bundle.main.url(forResource: "yolo11m", withExtension: "mlpackage"),
let coreMLModel = try? MLModel(contentsOf: modelURL),
let visionModel = try? VNCoreMLModel(for: coreMLModel) else {
return
}
self.visionModel = visionModel
}
func processFrame(pixelBuffer: CVPixelBuffer) {
guard let visionModel = visionModel else { return }
let request = VNCoreMLRequest(model: visionModel) { [weak self] request, _ in
self?.handleDetections(request: request)
}
request.imageCropAndScaleOption = .scaleFill
let handler = VNImageRequestHandler(cvPixelBuffer: pixelBuffer, options: [:])
try? handler.perform([request])
}
}
Lowering the confidence threshold to 0.35 allowed the model to maintain tracking continuity through motion-blurred frames. To prevent this lower threshold from spawning false positives, I tightened the IoU threshold to 0.45, ensuring overlapping candidate boxes collapsed into one stable center point.
Could we track player positioning in the same frame without stalling the video feed?
The Dual Vision Pipeline
Ball location alone only tells half the story. To understand passing lanes and field positioning, we needed player skeletons.
Instead of running a second heavy neural network, I tapped Apple’s native Vision framework via VNDetectHumanBodyPoseRequest. Because Vision’s body pose detection is optimized directly by Apple for iOS silicon, running it concurrently with YOLO11m did not drop frame rates:
final class PoseDetectionService: ObservableObject {
@Published var detectedJoints: [VNHumanBodyPoseObservation.JointName: CGPoint] = [:]
func processPose(pixelBuffer: CVPixelBuffer) {
let request = VNDetectHumanBodyPoseRequest { [weak self] request, _ in
guard let observations = request.results as? [VNHumanBodyPoseObservation],
let firstPerson = observations.first else {
return
}
self?.extractKeypoints(from: firstPerson)
}
let handler = VNImageRequestHandler(cvPixelBuffer: pixelBuffer, options: [:])
try? handler.perform([request])
}
}
CameraService.swift acts as the single capture coordinator, receiving buffers from AVCaptureSession and dispatching them cleanly to both services. The coordinates feed into SessionView.swift, drawing live bounding boxes and player skeleton overlays with negligible rendering delay.
Turning Coordinates into Match Analytics
Once coordinate logging became reliable frame by frame, the data mapped naturally into match insights:
- Ball possession: Tracking the percentage of time the ball stayed on either half of the pitch.
- Heat zones: Aggregating coordinate clusters to highlight whether attacks favored the left wing, right wing, or central corridors.
- Active playtime: Detecting stoppages automatically when the ball left the camera’s field of view for consecutive seconds.
What I Learned
Building Footsal showed me that in computer vision, offline evaluation scores rarely match physical deployment conditions. A model that scores well on static benchmarks can easily fail when kicked across a sunny, shadow-filled sports court.
The key breakthrough came from understanding physical motion: passes will blur, players will step in front of the ball, and court lighting will fluctuate. Tuning threshold boundaries around those physical realities, rather than relying on default model parameters, was what turned an unstable experiment into a steady tracking tool.