How AI calorie trackers actually work (a non-magical explanation)

Every week, someone emails us to ask whether CalorieScan AI is "real AI" or just a very confident lookup table. The honest answer is: it's both, in a specific order, and the order matters.

This post is the version of the explanation we'd give a friend who knows what an iPhone is but not what a transformer is. No hand-waving, no "the AI does the rest" cop-outs.

Step 1: the camera is the input

The first thing the app does is unglamorous: it looks at the photo you took.

A vision model — in our case a fine-tuned variant of an open multimodal model, paired with a small in-house segmentation network — has two jobs. First, it has to find the food in the frame and ignore the table, the napkin, the cat. Second, it has to name what it sees.

The naming step is harder than people think. "Chicken" is not a useful label. "Pan-seared chicken thigh, skin on, with what appears to be a chimichurri" is. The more specific the label, the more accurate the macros downstream, because the difference between chicken breast and chicken thigh skin-on is roughly a 70% jump in calories per gram.

Step 2: portion size, the hardest problem in food AI

If you take only one fact away from this post, take this one: portion estimation is the hardest part of any food tracking app, AI or not. A bowl of rice can be 100 grams or 400 grams. A "burger" can be 220 calories or 1,400.

Newer iPhones include a LiDAR sensor on the Pro line, which we use when available to get a real depth map of the plate. For everyone else, we fall back to a model that estimates portion size from visual cues: the size of the plate, the angle of the photo, the relative size of utensils, and a learned prior about how much food typically appears in a given dish.

We are not going to pretend this is perfect. It isn't. It's good enough — somewhere in the 80% accuracy range for typical meals — and it's correctable. Which brings us to the next step.

Step 3: the natural-language editor

After the photo is analyzed, you can talk to the result. Type "no croutons", "double the olive oil", "swap the chicken for tofu", and a small language model rewrites the underlying ingredient list. It then re-runs the macro calculation against our nutrition database.

This is the step that matters most for long-term users. No vision model is going to nail every plate every time. But almost everyone can type "make the rice half a cup instead." The combination — AI for the boring 90%, you for the 10% it gets wrong — is what makes the product usable seven days in a row instead of seven days total.

Step 4: the boring nutrition database

Underneath all of this is a 42,000-entry nutrition database, pulled and reconciled from USDA FoodData Central, OpenFoodFacts, and a curated set of restaurant menus. There is nothing AI about it. It's a table. The AI's job is to map the photo to the right row in the table and the right portion size against it.

This is also why "the AI just makes up calories" is a misconception. The AI is naming and measuring; the calories themselves come from a deterministic, inspectable source.

What this means for accuracy

Photo recognition is a multi-step pipeline. Each step compounds error. Be skeptical of any app claiming 99%.

We tell users to expect 80% accuracy on the first pass and 95%+ accuracy after a five-second edit. That tracks with the published literature on consumer food-tracking accuracy: even a careful human with a kitchen scale is in the 90s, not the 99s.

Why we built it this way

A lot of nutrition apps assume you have ten minutes a day to log food. We assumed you have ten seconds. Optimizing for the ten-second user — fast capture, fast correction, fast save — meant we couldn't lean on classical workflows like "search the database, find the food, enter grams."

The AI isn't a gimmick. It's the only way the ten-second workflow works at all.

What's next

We're working on multi-meal scenes (think buffets, family-style dinners), better restaurant disambiguation, and an opt-in feedback loop where confirmed plates can teach the model to be less wrong next week. We'll publish accuracy numbers as we ship.

In the meantime: the app is on the App Store, it's free to start, and we love getting bug reports about plates we got wildly wrong. The wrong ones are how the right ones happen.