There’s a Russian saying: “Measure seven times, cut once.” It generally means that spending more time preparing helps you achieve a better result in the end.

I wanted to see if the same logic applies to AI agents. So I gave two agents using the same model the same task, a real one-hour deadline, and almost the same prompt. Except one agent was told to plan first. Then I compared what they built…

The experiment

In the last couple of weeks, I’ve wanted to play “Command & Conquer: Red Alert 2.” But the last time I tried to download it, many years ago, my computer caught a virus. So I figured this would be a good task to test with AI. I also wanted a task complicated enough that it couldn’t be realistically perfected in an hour.

The agents needed to build a functional game. I needed to be able to move units, construct buildings, produce an army, fight enemies, and hopefully survive long enough to understand what I was doing.

Both agents received the prompt below. The italicized paragraph appeared only in the planning agent’s version:

I didn’t tell the planning agent how long it should spend planning. I wanted it to decide for itself. I expected it might use 10 or 15 minutes.

It spent roughly two. Guess it decided that measuring seven times isn’t worth it. Or it’s just that 15 minutes of planning feels different to a sand god than it does to a human.

Giving an AI agent a clock

I also wanted the one-hour deadline to be real. I had Codex create a small command the agents could call to find out exactly how much time remained. I wanted to avoid forcing them to set up their own timer loops. More importantly, I wanted to see how often they checked the time, what they were doing when they checked it, and whether the remaining time caused them to change their priorities.

The planning agent checked seven times. The agent that started immediately checked eleven times.

Neither checked on a regular schedule. They generally checked after builds, tests, or implementation milestones. The second agent checked more frequently partly because it ran more regression passes.

But both agents understood that the timer existed and used it without being reminded. I was actually a bit surprised that they remembered.

Testing the games

After the hour passed, I tested the two games one at a time without knowing which agent had produced which one. I had my best friend Codex create that part too. I just ran a Python script, and it built both repos and opened them on different ports in the browser.

One thing I was afraid of was that the builds would be broken. Surprisingly, both worked. That was probably my biggest surprise from the whole experiment. Maybe I should just never question Codex’s ways.

But this wasn’t a perfect blind test. The first game worked well enough that when I opened the other one, it became fairly easy to guess that it was the one told to start immediately.

In the first game, I could move units around the map, construct buildings, choose where to place them, and build an army. It wasn’t a finished game, but I actually wanted to play it further.

The second game looked better. But it didn’t feel complete.

Buildings were placed automatically instead of letting me choose where they went. I don’t think I managed to build any army units, and my base was attacked and destroyed so quickly that I didn’t have much opportunity to understand how the game worked. So maybe my army was being produced, but the units just got killed too soon for me to know.

It’s possible that neither version “worked” properly if I had evaluated them as complete games. But the second one definitely felt worse.

After running another script that revealed the results, the first game turned out to be the one made by the planning agent. The better-looking but less playable version came from the agent that started immediately.

I had scored the games using categories for playability, resemblance, completeness, visuals, and stability. The planning version scored 38 out of 50. The other version scored 36.

That sounds extremely close, but I think my scoring system understated the difference. Both games received full marks for stability because neither crashed. The second game also made up points through its stronger visuals. But “the application didn’t crash” isn’t the same as “I wanted to continue playing it.”

If I evaluated them again, I would spend more time figuring out how each game works and seriously trying to win. Although, honestly, I’m not sure I would enjoy doing that with the version that didn’t plan. That reluctance is probably useful evaluation data by itself.

The agent that planned didn’t build more

The planning agent didn’t win because it wrote dramatically more code or spent a long time thinking. It won because it defined a more useful target before it began.

The planning phase established requirements such as selecting units, placing buildings, managing resources, producing an army, and fighting an enemy. It gave the agent something relatively stable to work toward.

The other agent effectively made more of those decisions while it was already implementing the game. That may be the strongest argument for asking an agent to plan an open-ended task.

The plan didn’t necessarily help it build more. It helped it decide what needed to be built—and, more importantly, what not to build.

But did planning produce better code?

After comparing the games, I also reviewed the actual implementations without using the strategy labels. This produced a less convenient result.

The planning agent did not clearly write better code.

Its project had better documentation, a much smaller browser bundle, and tests that were more closely aligned with some of the core interactions I cared about. But most of its game lived inside one large class responsible for rendering, input, combat, economy, AI, audio, and the user interface. It also contained some significant bugs that my initial evaluation didn’t uncover.

For example, its construction queue could silently discard a completed building, and its unit collision logic could allow opposing tanks to drive through each other.

The agent that didn’t plan had a more reusable gameplay core and somewhat broader automated tests. At the same time, much of its main application code was extremely dense, its browser bundle was much larger, and its tests missed the most important question: was the game meaningfully playable?

It had more tests, but those tests helped it prove that its implementation behaved as expected. They didn’t prove that it had built the right experience.

My conclusion from the code review was essentially a tie. Planning produced the game I preferred, but it didn’t clearly produce better code.

Both agents stopped early

This was the result I found most surprising. Neither agent used the full hour.

The planning agent stopped with roughly 25 minutes remaining. The other stopped with roughly 16 minutes remaining.

They both knew how much time was left. They had checked the timer repeatedly. Yet once their own tests passed and their self-defined requirements appeared complete, they treated the task as finished.

The planning agent had even completed all of its original milestones with around half the budget remaining. It added a couple of optional improvements, checked that everything still passed, and stopped.

In hindsight, my prompt gave the agents a deadline, but it didn’t explain how that deadline should influence their definition of done. They interpreted one hour as a maximum runtime, not as a budget they should use to make the result as good as possible.

For a normal work task, that may be exactly what you want. If the requirements are clear and the result is correct, there’s no reason for an agent to keep changing things just to consume the rest of the clock.

But this task was open-ended. I asked them to reproduce a game “as closely as possible.” There was always more to improve. With 25 minutes left, the planning agent could have spent more time play-testing, improving the visuals, or fixing issues like tanks passing through each other.

A better approach might have been to create both core and stretch milestones. After completing each milestone, the agent could check the timer, evaluate the current result, and decide whether there was enough time to safely attempt the next improvement.

That would be different from blindly working until the clock reached zero.

So, should agents plan first?

Based on this experiment, I would tell an agent to plan an open-ended project before it starts.

Not because a plan guarantees cleaner code. It clearly doesn’t. I would do it because planning forces the agent to define the requirements it will work against instead of inventing them as it goes.

The planning agent produced the less attractive game, but it understood the assignment better. The other produced better visuals and more tests, but it missed parts of the experience that made the game feel like a game.

There’s an important limitation here: this was one experiment with one task. It doesn’t prove that planning will always win, and my evaluation process could be improved. But it did answer one of my original questions.

Yes, it’s possible to give AI agents an external deadline. They can check it autonomously and use it while working. Getting them to use the available time intelligently is a separate problem. And perhaps the stranger result is that the two-minute plan mattered more than the extra tests and better visuals.

What about you? Do you always tell your agents to plan first, or just let them build as they go?

Cheers!

Evgeny Urubkov (@codevev)