Motion input · August 2026

Steering a fractal with an iPhone, a Watch, and AirPods

I built an app that draws a fractal and zooms into it forever, and you steer it without touching the screen. Tilt the phone to pan. Tilt your wrist to pan hands-free. Turn the Watch crown to set zoom speed, or wiggle it to change colors. Pitch your head to zoom. Flick your head sideways to change colors too. All three devices work at the same time. Here's the video:

The fractal was a good test subject because it never ends and it looks great. But what I wanted to learn was how to make motion control across several devices feel good on hardware people already own, and where the problems are. That applies to more than fractals: cameras, simulators, accessibility, museum installations.

Who controls what

The rule that made it work

Tilt is measured against gravity, so it doesn't drift. Head yaw has nothing to measure against, so it does. That one fact ended up deciding the whole control layout.

At first I had head yaw panning the view. It felt fine for a minute, then the "center" slowly wandered until I was cranking my neck sideways just to make the view hold still. I tried to filter it out. The real fix was a design rule: drifting axes only get to trigger events, never steer. So yaw now flips the color palette when you flick your head past a threshold. A drifted center can't hurt that. Everything that steers continuously — phone tilt, wrist tilt, head pitch — rides an axis with gravity as its reference.

Related rule I learned the same way: one gesture gets one continuous job. I tried having wrist tilt control zoom and pan at the same time. It felt like the app was fighting me. Splitting the jobs across devices fixed it.

The walls I hit

Each of these cost me real time and none of them are documented loudly. I'm naming them so you can budget for them. The specific fixes are in my private implementation.

• AirPods head tracking can silently give you nothing. Connected, worn, API reporting success — and zero motion data, no error. There's an undocumented precondition. Until you find it, all you see is silence.
• watchOS owns certain wrist motions. I built a wrist-flick gesture and it worked three times, then the system gesture it collided with kicked my app back to the clock face. There's no API to turn that off. Check what the OS already claims before you design a Watch gesture.
• Keeping a Watch app running while the wrist moves around is its own project. The obvious API isn't enough. The working setup takes two session types together, started in a specific order. Wrong order and they conflict.
• Half my axis directions were backwards. They looked right on paper and only showed up wrong on a real wrist. Head tracking has a correct convention — look left means pan left, same as FPV drone pilots and assistive head-mice use — and it's not what joystick habits suggest.
• One thing I never solved: detecting a physical tap on the Watch case from accelerometer data. Two approaches, zero detections on real hardware. If you've solved this, I'd like to hear how.

Where this goes

The reusable part is small: a control law that shapes tilt into velocity, pan speed that scales with zoom depth, and the drift rule above. The same layer that steers a fractal could steer a growing tree in a museum, a camera in a simulator, or a cursor for someone whose hands are busy or don't cooperate.

If you're building motion controls on Apple hardware and could use help, email me: captaincolinr@gmail.com