The Problem Was Never the GPU

Introduction

It started the way most performance investigations start: with a complaint that was completely reasonable and almost entirely misleading. Scanning a medication package with FarmakoMed's on-device AI sometimes felt instant, and sometimes felt sluggish, a beat too long before the camera preview handed off to a result. No crash, no error, nothing a bug tracker would flag on its own. Just an inconsistency that made the feature feel less trustworthy than it actually was.

The instinct, and it is a reasonable one, was to ask about the inference backend. Was the model running on the GPU or falling back to CPU? Was the device one of the ones known to have flaky GPU driver support? Should the default backend change? Was quantization costing more than it should? These are good questions. They are also, in this story, almost entirely the wrong ones, and figuring that out is what this article is actually about.

The broader lesson, and the reason this is worth writing down even if you have never touched a language model, is one that shows up in every performance-sensitive system eventually: the visible bottleneck is rarely the real one. The real one tends to live one abstraction layer higher than wherever you first pointed the profiler.

Chapter 1 — The Benchmark Trap

If you go looking for why on-device inference might be slow, the literature hands you an answer almost immediately, and it's a genuinely useful answer for a large class of problems. On-device runtimes like Google's LiteRT support multiple execution backends for the same model, typically CPU, GPU, and on some devices a dedicated NPU, and the throughput difference between them can be substantial. GPU delegates parallelise the matrix operations that dominate transformer inference, and public benchmarks routinely show meaningful speedups over CPU-only execution for the same model and the same prompt.

Inference on a language model, on-device or otherwise, is also usually split into two phases with very different cost profiles: prefill, where the model processes the entire input (in FarmakoMed's case, an image plus a short instruction) in parallel, and decode, where it generates output one token at a time, each token depending on the last. Prefill parallelises well and benefits heavily from a faster backend; decode is inherently sequential and tends to be latency-bound rather than throughput-bound. Add to that the distinction between a cold start, where the runtime has to load weights, allocate buffers, and initialise a delegate before it can run anything, and a warm start, where all of that is already resident and the same request completes in a fraction of the time, and you have a genuinely rich set of backend-level knobs to investigate.

None of this is wrong, and it's exactly why a backend investigation is the natural first move. When a system's benchmark numbers show a clear gap between two configurations, it is entirely rational to assume that gap explains a real-world slowdown you're observing. The benchmark isn't lying. It's just answering a narrower question than the one you actually have. It can tell you which backend is faster per inference. It cannot tell you how often your application is paying that cost, or whether the backend is even the thing that changed between a fast run and a slow one.

Chapter 2 — Following the Wrong Lead

So the investigation followed the benchmark's lead, and it found real things worth fixing. This is an important part of the story: the GPU thread wasn't a wild goose chase invented for dramatic effect. It surfaced two genuine, shippable bugs.

The first was a platform-level gap. Since Android 10, the OS restricts an app's access to non-NDK native libraries unless they are explicitly declared, a restriction documented in Android's own Android 10 behavior changes. The GPU delegate depends on vendor-supplied native libraries to talk to the device's OpenCL or Vulkan driver, and the app manifest was missing the declarations needed to load them on some OS versions. The practical effect was quiet: the delegate would silently fail to initialise, fall back to CPU, and the diagnostics logging the failure hadn't been wired up to say why, so it looked indistinguishable from "GPU wasn't requested" rather than "GPU was requested and failed."

The second was hardware fragmentation, which is the industry's polite term for "not every phone's GPU driver behaves the same way." Internal device-compatibility testing turned up specific Samsung Exynos chipset and GPU pairings where delegate acceleration was only partially supported, meaning some operations in the model's graph would run on GPU and others would silently fall back per-op, a configuration that can be slower and less predictable than simply running the whole model on CPU. Neither of these findings was imaginary. Both were fixed, quickly, and the fixes were correct and worth shipping on their own merits: the manifest declarations were added so the delegate could actually be evaluated instead of failing invisibly, diagnostics started capturing the device's chipset so failures could be attributed rather than guessed at, and a device-compatibility list was added to steer known-risky hardware away from a GPU request before the runtime had to discover the failure itself.

And yet, once those fixes were in, benchmarking the two backends against each other on FarmakoMed's actual model and actual devices produced an anticlimactic result: on a cold engine, the GPU delegate's own initialisation overhead ate a large fraction of whatever throughput advantage it offered on the inference itself, and on flagship-class hardware the net difference between "CPU, correctly configured" and "GPU, correctly configured" was smaller than the team had assumed going in. It wasn't nothing. But it wasn't the multi-second gap users were describing, either. Something else was adding that.

Chapter 3 — The Turning Point

The question that actually cracked the investigation open didn't come from the benchmark work. It came from a much duller source: diagnostics instrumentation added, almost as an afterthought, to understand a different complaint entirely, about how the app behaved when a user pressed the hardware back button mid-scan.

Once that instrumentation was in place and logging every transition of the AI runtime's internal state, a pattern showed up that nobody had gone looking for: the inference engine was being released and re-initialised far more often than the number of distinct user sessions would predict. It wasn't loading once per app launch, or even once per scanning session. It was tearing itself down and rebuilding itself repeatedly, sometimes multiple times within a single medication scan that a user would describe as one continuous action.

That observation reframed the entire question. The team had been asking which backend is faster. The more useful question turned out to be why is the engine being rebuilt this often in the first place? A GPU delegate that initialises in a fraction of a second is still paying that cost on every single rebuild, and if the app is triggering ten rebuilds where one would do, the backend choice was never going to be the lever that mattered. You cannot out-optimise a problem of frequency by tuning the cost of a single occurrence.

graph TD
    A[Warm inference · CPU]
    B[Warm inference · GPU]
    C[Backend gap chased]
    D[One cold engine rebuild]
    style D fill:#22C55E,color:#052E1B

Chapter 4 — The Invisible Bottleneck

So why was the engine rebuilding itself so often? The answer had nothing to do with AI at all. It was in the application lifecycle, and specifically in an assumption that felt reasonable in isolation and broke down completely in practice: that when the app moves to the background, it is safe, even correct, to release the loaded model and reclaim its memory, and reload it when the app comes back to the foreground.

That assumption runs into a quirk of how camera and photo-picker flows actually work on Android. Opening the system camera, or the system photo picker, or a file picker, is not, from the operating system's point of view, staying inside your app. It is a transition to another activity, and the host app is demoted to the background for the duration, exactly as documented in Android's own activity lifecycle and mirrored in React Native's AppState API as a transition away from active. To a lifecycle listener with no further context, launching the camera to scan a pill bottle looks identical to the user switching to their email app for ten minutes.

A naive backgrounding policy cannot tell those two situations apart, and FarmakoMed's engine-lifecycle manager, in its early form, didn't try to. Every transition away from the foreground released the resident engine to be a good citizen of the device's memory. Every transition back to the foreground triggered a fresh cold start, delegate initialisation and all, on whatever backend was configured. For a user genuinely leaving the app for a while, that's correct and desirable behaviour. For a user tapping the camera button to photograph a medication label, it meant the engine was destroyed and rebuilt as a direct, unavoidable side effect of the exact gesture that was supposed to start the scan.

No amount of backend tuning could fix this, because the backend was never the layer where the cost was being paid repeatedly. Optimising GPU delegate throughput while the engine is destroyed and rebuilt several times per scan is optimising a number that gets multiplied by zero savings, because the fixed cold-start cost dwarfs the variable per-inference cost you just improved.

Chapter 5 — The Half-Finished Solution

Here is the detail that makes this story about engineering practice rather than just about one bug: the fix for this had already been built, days earlier, for exactly one screen.

Alongside the original engine-lifecycle rewrite, an engineer had already noticed that opening the camera or gallery picker during a medication scan was backgrounding the app and tearing down an in-flight scan. The fix at the time was a suspension flag: a signal the medication-scanning flow could raise immediately before launching the camera or picker, telling the lifecycle manager "the app is about to background, but don't release the engine, this is expected and temporary," and clear immediately after control returned. A safety-net timeout was built in too, so a suspension could never wedge the engine open indefinitely if something went wrong downstream.

It worked. Medication scanning stopped losing its warm engine to the camera handoff. But the fix lived entirely inside the medication-scanning screen's own code, called directly, by that screen, around that screen's own camera and picker calls. It was a solution to a specific symptom, not a policy applied to the mechanism that caused it.

Chapter 6 — The Second Bug

FarmakoMed doesn't only scan medication packaging. The same underlying camera, gallery, and file-picker calls are used to capture receipts for expense tracking and to attach documents to appointments, screens built separately, by the same small team, reasonably close together in time, but never revisited once the medication-scan suspension fix shipped.

Those screens had no idea the suspension mechanism existed, because nothing in their own code path called it. They called the same underlying camera and picker bridge that medication scanning did, which meant they triggered the exact same background transition, which meant they hit the exact same naive release-and-rebuild behaviour the suspension flag had been built to prevent, just with nothing there to prevent it. Diagnostics later confirmed it plainly: expense and document capture were paying the identical cold-start tax medication scanning had already been fixed for, on every single capture.

This is the moment worth sitting with, because the tempting fix, the one most teams would reach for under deadline pressure, is to copy the suspension call into the expense-capture screen and the document-capture screen and call it done. It would have worked. It also would have meant the same lifecycle knowledge now lived in three separate places, and a fourth capture flow added next quarter would silently reintroduce the exact same bug, because the fix was a convention, not a guarantee.

What shipped instead moved the suspension logic out of the medication-scan screen entirely and into the shared camera and picker bridge that every capture flow, present and future, already calls to talk to the native camera and gallery APIs. Every caller of that bridge now gets suspension for free, without having to know the mechanism exists. Because the medication-scan flow's own broader suspension and the bridge's narrower one could now overlap on a single scan, the flag was made reference-counted, so an inner suspension resuming early can never prematurely release an engine an outer caller is still relying on.

This is the same lesson the original medication-scan fix had already half-learned, taken one step further. The first fix solved a real problem with a targeted patch. The second fix solved the same class of problem by asking where the patch actually belonged.

Chapter 7 — The Reload Nobody Questioned

The Chapter 5 and 6 fixes shipped, and they were correct. The suspension mechanism, moved to the shared camera and picker bridge, stopped the AppState-driven backgrounding policy from mistaking a photo capture for the user leaving the app. Scans that should have stayed warm, stayed warm. And medication scanning was still slower than it should have been.

Not slow the same way as before, and not by the same margin, but the same diagnostics instrumentation that had cracked open Chapter 3 kept showing a familiar shape: the engine, torn down and rebuilt, right around the same camera launch the suspension fix was supposed to protect. This wasn't a resurgence of the fixed bug. It was a second, unrelated finding, hiding in the same trace, that the first fix's noise had been masking.

It turned out to be code that had never been broken, and had never been questioned either. Immediately before the medication-scan screen opens the camera, it explicitly, deliberately releases the loaded model from memory, before a single frame of the camera preview appears. Not a mis-detected background transition. A reasoned trade: a multi-gigabyte resident model competing with a live camera activity for RAM is a real way to get the process killed by the operating system, and freeing that memory ahead of time is a legitimate way to avoid it. That trade shipped, worked exactly as designed, and then simply sat there, unexamined, while everything else in the surrounding lifecycle changed underneath it.

A trace from the production device made the cost impossible to ignore. With the release-before-camera behaviour active (today's shipped default), resolving the vision engine after a capture, the step that reloads the model into memory before any inference can start, took 5.7 seconds by itself. The full round trip from tapping the shutter to seeing a result was 25.9 seconds, of which the model's own decode step accounted for roughly 20. The reload was adding real, measurable time on top of a decode cost that was already the dominant factor.

The fix wasn't to remove the release outright. Freeing memory before a camera intent is still a legitimate concern on a phone with less headroom than a flagship, and skipping it blindly would just trade one un-measured assumption for another. Instead, the app now checks how much RAM is actually free right before the camera opens, using the same native memory-snapshot bridge already wired up for diagnostics, and only skips the release when there is genuine headroom. Everywhere else, the original release-before-camera behaviour is untouched.

Making that check testable took its own small detour. The team's existing feature-flag system gates its debug overrides behind React Native's __DEV__ flag, which evaluates to false in any release build, including the exact release build the fix needed to be validated on. The toggle for this had to live somewhere that survives a release build instead: a persisted, on-device preference reachable through the app's internal developer tools, gated on the actual Android product flavour rather than the JS bundle's debug flag.

With the toggle on and the RAM check passing, same device, same model, same medication scan: engine resolution dropped from 5.7 seconds to 1 millisecond, because the model was still resident and simply didn't need reloading. The full round trip fell from 25.9 seconds to 18.5, a 29 percent reduction, with the decode step itself unchanged either way, exactly as Chapter 3's turning point would have predicted.

graph TD
    A[Engine resolve, reload active: 5.7s]
    B[Engine resolve, reload skipped: ~0s]
    C[Total round trip, reload active: 25.9s]
    D[Total round trip, reload skipped: 18.5s]
    style B fill:#22C55E,color:#052E1B
    style D fill:#22C55E,color:#052E1B

Chapter 8 — What the GPU Actually Cost, On a Real Device

Chapter 2's GPU investigation was accurate, and it was also, necessarily, general. The manifest declarations and the Exynos/Xclipse device-risk list address real, known failure modes, but neither says much about what GPU acceleration costs on a device that isn't on that risk list at all. The Galaxy S24 Ultra's Snapdragon 8 Gen 3 and its Adreno 750 GPU aren't Exynos, aren't in the device-compatibility table's list of known-risky pairings, and by every existing check in the codebase should have been a straightforward candidate for GPU acceleration.

So the same production device that produced Chapter 7's reload numbers was used to answer a narrower question directly: request the auto backend, let the runtime's own candidate list try GPU before falling back to CPU, and read the trace.

The GPU delegate initialised cleanly. It did not fail, did not hit a missing manifest declaration, did not silently fall back to CPU the way a broken delegate normally would. Delegate initialisation succeeded; acceleration genuinely activated. And the same engine-resolution step that took 5.7 seconds on CPU took 34.9 seconds requesting GPU, on a device the compatibility checks had no reason to flag.

The native runtime's own logs said exactly where that time went, almost as an aside: OpenCL sampler not available, falling back to statically linked C API, immediately followed by the identical message for WebGPU. Before the delegate can run a single decode step, it has to stand up a compute context and compile kernels against this specific device's driver, a cost the public benchmark literature on GPU delegates rarely isolates, because most of it measures steady-state throughput in a loop, not the one-time setup cost of a fresh context.

Once the engine was up, decode itself told a quieter story: 18.4 seconds on GPU, against 17.7 to 20.1 seconds measured on CPU across multiple runs on the same device. Statistically indistinguishable. The parallelism advantage GPU delegates are built to provide over CPU never showed up for this workload, on this device, at a batch size of one.

None of that makes GPU broken. It makes it a bad match for an interactive feature that opens once, runs once, and closes, rather than a sustained inference loop that could amortise a one-time setup cost across thousands of calls. CPU remains FarmakoMed's production default, not because the silicon underneath GPU is deficient, but because on this specific interaction pattern decode throughput was never the constraint that mattered, and the setup cost GPU pays to get there was.

Chapter 9 — The Door That Wasn't There

If GPU acceleration exists on the silicon but doesn't pay off for a use-once interaction, the next honest question is whether the phone's dedicated NPU could do better. The Snapdragon 8 Gen 3 ships a Hexagon Tensor Processor built specifically for this class of workload, addressable through Qualcomm's QNN SDK. LiteRT-LM's own Kotlin runtime already has a stubbed backend branch for it, deliberately excluded from the automatic backend-selection path with a comment noting the dispatch provider isn't bundled yet, a clear sign the door had been scoped and deliberately left unopened.

Two of the usual obstacles to shipping a vendor-specific accelerator turned out not to be obstacles at all. The two Maven dependencies NPU support needs, com.qualcomm.qti:qnn-runtime and qnn-litert-delegate, sit on plain Maven Central, no vendor repository, no interactive registration gate. Both are licensed under the Qualcomm AI Hub Model License, which explicitly permits distributing the compiled libraries inside a commercial application's release build. On dependencies and licensing, adding NPU support looked entirely tractable.

The third obstacle was the real one. NPU inference through this stack doesn't run a portable model file the way CPU or GPU execution does. It runs a model that has already been ahead-of-time compiled against one specific chip's instruction set, a step that happens once, offline, before the app ever ships. Checking the places such a compiled artifact would plausibly already exist, Google's own litert-community collection on Hugging Face and Qualcomm's AI Hub, turned up chip-specific vision-capable Gemma variants for a neighbouring chip generation. For this exact chip, the Snapdragon 8 Gen 3's SM8650, the only pre-built artifact found anywhere was a text-only model. Useless for a feature whose entire job is reading a photograph off a medication box.

Producing one would mean standing up an AOT compilation pipeline as its own piece of infrastructure: LiteRT's Python compiler CLI, the QAIRT SDK, a source model pulled from Qualcomm AI Hub, and an ongoing commitment to re-run that pipeline every time the chip family or the runtime version moves. There is also, at the time of writing, no published compatibility table pairing LiteRT-LM releases against QNN SDK versions; the right pairing is discovered empirically, and getting it wrong is a documented, live failure mode, not a hypothetical one.

None of this is a verdict on Qualcomm's NPU hardware, or on QNN as a technology. The underlying tooling is real, actively maintained, and documented by Google as production-track, not experimental. It simply hadn't produced a usable artifact for this specific chip and this specific model family at the time of this investigation. "The chip supports this" and "there is something we can run on it today" turned out to be two different claims, and the distance between them was a model-compilation project, not a configuration change.

The pattern from Chapter 8 repeats here, one layer further out. GPU's problem was a warm-up cost that a benchmark measuring steady-state throughput would never surface. NPU's problem sits further upstream still: there was nothing to benchmark, because the artifact required to run the benchmark didn't exist.

Chapter 10 — The Real Lesson

Strip away the model, the delegate, and the Android-specific lifecycle quirk, and this stops being a story about AI at all. It becomes a story about where performance work actually pays off, which is a question every engineer working on a layered system eventually has to answer for themselves.

Benchmarks measure components. A GPU delegate benchmark measures the GPU delegate, honestly and correctly, on the question it was designed to answer. It cannot tell you how often your application actually invokes that component, under what conditions, or whether something above it in the stack is quietly multiplying a small per-call cost into a large user-facing one. Users don't experience components. They experience the system, end to end, including every policy decision, lifecycle transition, and abstraction boundary that sits between "I tapped the button" and "I got an answer."

This is why the benchmark trap is so easy to walk into and so hard to notice from inside it. Every step of the GPU investigation was legitimate engineering. The manifest fix was real. The chipset-risk data was real. The decision to default to CPU was a defensible, well-reasoned trade-off. None of it was wasted work, exactly, and none of it was wrong on its own terms. It just wasn't where the user-facing latency was actually coming from, and no amount of rigor at that layer could have revealed that on its own. Revealing it took an entirely different kind of instrumentation, one that measured how often a policy decision was firing, not how fast a chip could multiply matrices.

The second pass, months later, sharpened that same lesson into two distinct shapes. Chapters 4 through 6 were an accident: a lifecycle policy misreading a camera launch as the user leaving, fixed once it was found. Chapter 7 was never an accident at all, a deliberate, reasoned trade-off that simply outlived the conditions that had made it necessary, and cost real seconds precisely because nobody had gone back to check. Chapters 8 and 9 add a third shape again: GPU's cost was a warm-up tax invisible to any benchmark that measures a hot loop instead of a single interactive call, and NPU's cost sat further upstream still, in a supply chain for compiled models that hadn't caught up to the hardware sitting in the phone. Four different failure shapes, one underlying cause: every one of them lived one layer above wherever the team had first pointed the profiler.

Conclusion

The fastest inference backend in the world cannot compensate for an application that quietly rebuilds its own engine every time the user taps the camera button. That single sentence was the entire first investigation, compressed. The second pass added a corollary worth remembering just as much: the fastest accelerator is also useless if it takes thirty seconds to wake up for a feature that only needed it once, and no accelerator at all helps if the model it would run doesn't exist yet for your chip. It took a benchmark, a manifest fix, a chipset-compatibility list, a diagnostics pass built for an unrelated complaint, a second bug in a screen nobody had thought to check, a deliberate trade-off nobody had re-measured, and a real device pushed through GPU and NPU paths that had never been tried before, to find all four.

The visible bottleneck is rarely the real one. It's usually one layer up, in the policy nobody benchmarked, quietly deciding how often the fast thing gets to stay warm, or one layer further out, in the accelerator that never got a chance to prove itself before its own setup cost, or the model artifact it needed, spent the entire budget on its own behalf.