Confusing Knowledge with Understanding: The AI trap

Last week I decided to try Fitbit’s AI coaching feature. It’s still in preview, positioned as a glimpse of where Fitbit wants to take personalized fitness. The concept is appealing: an AI coach powered by Gemini that learns your goals, studies your habits, and builds a training plan tailored to you.

I came in prepared. I have a ten-day periodization cycle I’ve been developing: strength training, rest days, flexibility work, and three distinct runs each cycle, a short day, a medium day, and a long run. My goal for the year is to run 15K continuously and conserve some strength. I laid all of this out for the AI coach. I showed it my plan. I explained my objectives.

The first week, it worked. The plan it generated looked a lot like what I was already doing, and I followed it without friction. I thought: this might actually be useful.

Then came week two.

The plan it offered me looked more like a gentle yoga retreat than a training program. No runs. Minimal intensity. Lots of active recovery. I pushed back: “Why aren’t there any runs this week? This doesn’t align with my goals.” It apologized and generated a new plan on the spot. The new plan suggested I run 13 kilometers the next day.

I had been training continuously for four days. The next day was a rest day. The AI didn’t know that. Or rather, it had known it, somewhere in our earlier conversation, and it no longer did.

That’s when something clicked for me.

—

Here’s what’s striking about that moment. The same AI I was arguing with about my training schedule can explain the science of heart rate variability with precision. It can discuss periodization theory, sleep architecture, the physiological mechanisms behind muscle recovery. Ask it almost anything about health and performance, and it will give you a credible, detailed answer.

But ask it to hold all of those things together, at the same time, across time, in service of a real person with competing goals and a training history, and it comes apart.

This isn’t a small gap. It’s a fundamental one.

What we tend to call AI “intelligence” is, more precisely, AI retrieval. These systems have become extraordinarily good at surfacing relevant knowledge in response to a question. That is genuinely impressive, and genuinely useful. But reasoning is something different. Reasoning requires holding context across time. It requires recognizing when multiple constraints are pulling in different directions and tolerating the tension between them before arriving at a decision. It requires knowing what you don’t know, and pausing because of it.

The AI coach didn’t feel the tension between my need for rest and my goal of building endurance. It processed those as separate inputs, not as competing demands that required judgment to reconcile. Any coach who had worked with me for a week, a human coach, would have felt that tension intuitively. They would have sat with it. They would have made a call.

There’s a French expression for this kind of grounded, practical judgment: *gros bon sens*. It doesn’t translate cleanly into English, and that gap is itself revealing. “Common sense” comes close, but it misses the weight. *Gros bon sens* implies something embodied and earned, a judgment that comes from having lived with consequences, not just having read about them. AI has read about everything. It has lived with nothing.

—

This matters well beyond fitness apps.

The same pattern shows up wherever AI is being asked to carry consequential decisions. In medicine, AI can be a remarkable knowledge resource. It can surface diagnostic possibilities, flag drug interactions, synthesize research, and support clinical documentation. These are real contributions. But the moment we ask it to own the decision, to hold the full weight of a patient’s history, their goals, their context, their contradictions, and make a judgment call, we are asking it to be something it is not yet.

The risk isn’t that AI will refuse when it’s out of its depth. The risk is that it will comply. Confidently. Without knowing what it doesn’t know.

There is a concept in aviation called *automation bias*: the tendency of human operators to over-trust automated systems, to stop questioning outputs that arrive with apparent authority. Medicine is already showing early signs of this. When an AI generates a confident-looking note, a plausible-sounding recommendation, or a clean-looking care plan, the cognitive pressure on the human in the room to simply accept it is real. And the system won’t tell you when it forgot the context from last week, or when it quietly dropped one of your constraints to resolve a tension it couldn’t hold.

Autonomous AI in healthcare isn’t just premature. It’s a category error. The clinical environment doesn’t stay stable between sessions. Patients change. Circumstances shift. Goals evolve in ways that require ongoing human interpretation. Unlike a misguided training week that costs you some fitness, a missed signal in a clinical setting can cause real harm to a real person.

A human in the loop is not a temporary workaround for a technology that will soon be ready to go it alone. It is a design principle grounded in something true about the nature of judgment itself. Some decisions need to be owned by someone who will live with the consequences of getting them wrong.

AI is genuinely useful. It is, in the right roles, a remarkable thinking partner. But a thinking partner is not a decision-maker. The distinction matters, and right now, it matters most in the places where the stakes are highest.

We’re not there yet. And we should be honest about that.

Thank you.