The trap of local optimization

One of the most significant shifts in the software development industry over the last two decades has been the move from shipping distinct update versions to continuous iteration.

If you worked in tech twenty years ago, you remember how it used to be.

When a new version of Photoshop or Windows came out, you bought a box. Inside was a stack of floppy disks or CD-ROMs. You spent hours installing the update. Because distribution was physical and expensive, publishers could only afford to ship updates once or twice a year.

Today, software is a service in constant evolution. If you use WordPress.com today and come back tomorrow, the code has likely changed in the background. Google Chrome is the perfect example of this. It doesn’t ask you to install version 142.0.744; it simply updates silently in the background.

This shift unlocked a powerful capability: the ability to test, measure, and iterate in real-time.

Plus, it gave rise to the culture of A/B testing.

You show Group A the current version of a button. You show Group B a version that is bright orange. You measure which group clicks more. If the orange button wins, you keep it. You have used data to make a better product.

It is a phenomenal tool. It removes guesswork and validates hypotheses. But like any tool, if you use it without understanding the underlying mechanics, it can lead you into a dead end.

The Hill Climbing Problem

Imagine you are standing on a foggy hillside. Your goal is to reach the highest peak possible. You have a simple algorithm to help you: take one step in any direction. If that step takes you higher, stay there. If it takes you lower, move back.

If you follow this process strictly, you will eventually reach a peak. You will be at the very top of the hill you are currently standing on.

But because of the fog, you cannot see that five miles away, there is a mountain twice as high.

Do you get where this is going?

This is the trap of data-driven iteration. If you only take steps that improve your immediate metrics, you will never walk down into the valley to cross over to the higher mountain. A/B testing prevents you from taking a temporary dip in performance, even if that dip is necessary to reach a much higher long-term summit.

To reach the highest peak, you have to be willing to descend. You have to accept that metrics might get worse before they get better.

The central limitation of iterative testing is the difference between local and global optimization.

Local optimization will get you to the top of the nearest peak.

Global optimization will force you to choose the right peak.

Data-Driven vs. Data-Informed

This is why I don’t like the phrase “data-driven”.

When you are data-driven, the data holds the steering wheel. You might see your conversion rate go from 10% to 12% and feel successful.

But you might be optimizing a product that is fundamentally obsolete.

If we were back in 2006, you could have A/B tested a BlackBerry keyboard endlessly. You could have found the perfect key spacing and the perfect tactile click. You would have built the best BlackBerry keyboard in history.

But you would never have arrived at the iPhone.

Going from a physical keyboard to a multi-touch glass screen is not an iterative step. It is a paradigm shift.

Data cannot tell you to delete the keyboard. That requires product sense, intuition, and the courage to ignore the immediate data in service of a larger vision.

We applied this thinking recently with our domains product at Automattic.

When I looked at our domain search experience last year, it was not good.

My team asked if we should test changes one by one. I said no, because we were on a low hill — we didn’t need a test to tell us that we were not on Everest. If we had tested small changes, most would have failed or shown negligible impact because the foundation was broken.

Instead, we rebuilt the entire experience from scratch to match the quality expected in the market. We changed thousands of variables at once.

It was necessary to move from the small hill to the base of the big mountain. Once we launched that new baseline and confirmed we hadn’t tanked our metrics, then we could return to iteration.

The Human Signal

How do you know when to jump to a new hill?

Often enough, data won’t be able to tell you. But people will.

You might have 100,000 data points telling you that your conversion rate is stable. But five emails from real humans telling you that your product feels slow or confusing can be a stronger signal than all that aggregate data.

We recently noticed a competitor launch a search interface that felt incredibly fast. The results were actually poor quality, but the feeling of instant feedback delighted users.

That was a strong signal that a new mountain was found. We are now building a multi-layered search experience that gives that instant feedback for keywords, while still taking the time to generate high-quality semantic results in the background. For now, no A/B test is involved — data wouldn’t be of help right now.

The goal is not just to improve metrics on the current path. The goal is to understand the landscape well enough to decide which hill is actually worth climbing.

Moral of the story: use data to optimize your ascent, but do not let it stop you from exploring the rest of the map.

If someone forwarded this to you, you can subscribe.

I also publish on paolo.blog and monochrome.blog.