A week ago I finally got access to publish my app to production in the Google Play Store. Getting 12 Android testers to keep the app installed for at least 14 days was much harder than I imagined, but it finally happened.

I still haven’t published it though because I’m probably overthinking it I knew those “testers” didn’t actually test much. As always, I had to rely on my friend Codex and its gpt-5.6-sol model.

I gave Codex a terrible test plan

I just said:

to see what it would find.

I didn’t tell it which features mattered most, how many paths it should test, or what “done” meant. I basically pointed it at the app and said, “Go find problems.”

I almost immediately added that it should take before-and-after screenshots and attach them to PRs. And make sure it’s one PR per bug fix.

The first stretch ran for 26 hours. By the end of it, Codex had opened 35 pull requests.

I thought that was the experiment. It turned out to be just the first half. Or maybe even quarter.

The app is Oneiras, my AI-powered dream journal. It has a Flutter mobile app, an AWS Amplify backend, guest and signed-in accounts, Pro subscriptions, private dreams, image generation, notifications, sharing, search, and enough weird state transitions to make “just test it” a questionable request.

Codex did a lot of accessibility passes. It found things like missing accessibility labels and what the app looked like when text scaling hit 200%. Something I’d definitely not think to test myself.

It also found plenty of normal bugs: stale validation errors, buttons with no feedback, and broken offline states.

But the more interesting failures happened when the technical fix passed and the product was still wrong.

The screenshots kept catching bad assumptions

At one point, Codex told me shared links worked. I looked at the screenshot. They very clearly did not.

The test had opened the route, seen some expected UI, and treated that as success. But it had never created a real shared dream with a valid token because it was too afraid of privacy issues.

So I told it:

Once it did that, the cold-start link actually failed. That helped it fix the issue properly.

There was another similar instance where the screenshots didn’t actually confirm the fix, but I won’t bore you with the details on that one.

Then I finally made a real checklist

After the first exploratory run, I asked Codex whether we had caught most of the bugs (I know, I am great at prompting).

There wasn’t really an honest way to answer. Exploratory testing is good at finding whatever you happen to bump into. It does not tell you what you never tried.

So I asked Codex to create a full test checklist. Not just failures. Everything. What passed, what failed, what was only partially tested, what Codex could test independently via an emulator and my sandbox, and what still needed me or a real device.

This changed the kinds of tests Codex started running.

The first phase was mostly screen-by-screen: open something, tap it, change the text size, turn off the network, and see what happens. The checklist started asking different questions.

What happens after Android kills the process? Does deleting something also delete every file connected to it? Does a test that passes with two dreams still pass with 500? What happens when two devices edit the same dream? Can private content leak somewhere outside the app itself?

Earlier, Codex had tested account deletion with two dreams. It deleted the account, then verified the Cognito identity, user rows, dreams, share token, and recording were gone.

That test passed.

With 501 dreams, the app reported a timeout after about 33 seconds. The backend eventually deleted the identity and User row, but it had only fetched and deleted the first 100 dreams.

Exactly 401 orphaned Dream records remained.

Even worse, a second emulator kept showing the deleted account’s fake dreams after a foreground refresh and a cold restart.

The small test wasn’t wrong. It just wasn’t enough.

The checklist found several other bugs in the same category: leaked audio files after dream deletion, unsaved recordings disappearing after process death, private dreams showing up in Android Recents (another thing I’d never think to even test), two devices silently overwriting each other’s edits, and reminders shifting after a time-zone change.

These were different from the earlier bugs. Missing labels and unclear errors matter, but these involved privacy, data cleanup, persistence, concurrency, and scale.

The numbers show the shift. Only four of the first 35 PRs were mainly about privacy or data integrity. After the checklist, eight of the next 22 product-fix PRs were.

That may have been the biggest improvement. I guess we did learn from the last newsletter that planning helps. What we’re learning today is that it applies to testing too.

Then the numbers got ridiculous

The first autonomous stretch lasted 26 hours. The full campaign lasted eight days and four hours once you include my reviews, the GitHub outage, deployments, rebases, and all the times I came back and said “next.”

I merged 66 pull requests.

Fifty-seven fixed product behavior. Five repaired release or deployment machinery. Three were for the checklist and QA reporting. One added an improvement I realized was missing after a previous refactor.

The Flutter test suite grew from 173 tests to 289, about 67%.

Twenty-eight PRs included visual evidence, with 75 unique GitHub attachments.

I sent 171 messages across those eight days. But 104 of them, 61%, were 12 words or fewer. Twenty were basically just “Merged,” “Next,” or “Merged. Next.”

So no, this was not a one-prompt autonomous miracle. But it also wasn’t 171 detailed instructions.

Most of my messages approved the last change, challenged the evidence, made a product decision, or told Codex to continue.

The long technical middle, reproducing the bug, finding the code, building the fixture, deploying the sandbox, writing the fix, running the tests, cleaning up the data, and preparing the PR, was done without me.

I approved at least 80% of those PRs from my phone too.

Did I actually review all that code?

You’re probably wondering if I actually reviewed the code in every PR.

Kinda.

I intentionally made Codex keep the PRs small enough that they were easier to review. But I don’t think I sent more than a couple back because of the code itself. Usually it was because Codex made an assumption that didn’t turn out to be correct.

The next step is to connect my older phone and let Codex use a real device for testing billing and other things that can only be tested on real hardware.

The useful setup

Codex was very good at persistence. It could move from authentication to notifications to offline handling to accessibility without getting bored. It also did all the unglamorous work: deploying sandbox builds, creating fake data, restoring emulator settings, rerunning tests, rebasing PRs, and uploading screenshots.

The useful setup wasn’t “give the agent a vague goal and trust whatever it says.” It was having a real local setup with an emulator and sandbox, along with the checklist and automated checks. And of course me, willing to say when something didn’t look right.

The checklist gave the agent a definition of what was left instead of letting it decide that the screens it happened to visit were “enough.”

Instead of manually discovering every failure path, I spent more time challenging assumptions, judging whether the app actually felt good, and remembering the occasional piece of historical context that wouldn’t have been obvious from the code unless you looked at the commit history.

Honestly, that is a pretty good trade.

Have you let an agent test a real app this way? What would you trust it to do alone, and where would you still want to take over?

Cheers!

Evgeny Urubkov (@codevev)