AI-First iOS Development From Scratch

Hey 👋

Last week I did something I never thought I’d do: I built a feature in an iOS app.

Some context: I’m a web developer. Backend preference. I’ve never touched Swift, never opened Xcode with intent, and definitely never worked on the Clay codebase (our internal iOS app at Automattic).

But I had an experiment in mind: how far can AI take me in completely foreign territory?

To make it interesting, I did this alongside Ryan — a colleague who’s an experienced iOS developer. Same feature, same timeframe. He used his expertise. I used… Claude.

The Feature

Contact Photo Matching. The idea is:

• Scan the user’s photo library for faces
• Match them against contact avatars
• Show matched photos in the contact view
• Let users confirm/reject, share, or set as avatar

All on-device using Apple’s Vision framework and Core ML. I had never written a line of Swift.

Act 1: Surprisingly Fast Start

Day 1, morning. I started by creating a PRD with Claude — phases, scope, architecture decisions. Within a few hours I had a working prototype: photo grid, face detection, basic UI.

I remember thinking: “This is going to be easy!”

I was wrong.

Act 2: The Wall

Day 1, afternoon. Face matching results were… random. Same image would give different confidence scores every run. 0.85, then 0.91, then 0.78. For the exact same face.

I went in circles with Claude for hours:

Me: “Results are inconsistent” Claude: “Let me add more logging…” Claude: “The values look correct” Me: “But they change every time!” Claude: “That’s strange. Maybe it’s a Vision Framework bug on the simulator?” Me: “That can’t be right…”

Claude kept insisting. I kept dismissing. We tried a dozen other fixes. Nothing worked.

Act 3: Fresh Start

Day 2. Fresh conversation, fresh mind.

This time I tried a different approach: “Build me a debug view.”

I asked Claude to create a visualization showing:

• Face bounding boxes overlaid on images
• Landmark points (eyes, nose, mouth)
• Raw numerical values
• History of values across runs

And suddenly I could see the problem. Landmark values were drifting on every detection run:

Run 1: leftEye.x = 0.3421
Run 2: leftEye.x = 0.3398
Run 3: leftEye.x = 0.3445
Run 4: leftEye.x = 0.3367

Same image. Same face. Different landmarks.

Ryan happened to be nearby. “Want to test on my device?”

Seconds later: values consistent. The simulator was the problem.

Claude was right. It’s a known Vision Framework bug on the simulator (since 2021!). VNDetectFaceLandmarksRequest with CPU-only mode returns inconsistent landmarks. Only on simulator. Works fine on real hardware.

I had wasted hours dismissing something the AI correctly identified early on.

Looking back

Would I have learned Swift, Xcode, the Vision framework, AND built this feature in two days on my own? No way. I still needed Ryan’s help for the simulator thing, and his code is probably much cleaner than mine. But I got something working. In two days. With zero iOS experience.

A few things that stuck with me:

Claude suggested the simulator bug early on and I kept dismissing it because it felt wrong. I should have just tested it on a real device right away instead of going in circles for hours. When the AI insists on something, at least rule it out properly.

Day 1 ended with frustration and a polluted conversation context. Day 2’s fresh start brought immediate progress. I think both humans and AI benefit from “sleeping on it.” Context pollution is real.

And one fun side effect of running multiple Claude sessions in parallel: while waiting for agents to finish their tasks, I had enough free time to build a Bomberman clone in a separate session. Because why not.