Optimal Illusions: The False Promise of Optimization Hardcover – September 12, 2023

I was taught optimization in school and then I worked on compilers. That was 40 years ago. Later I worked at various times, for Apple, Amazon and 9 different startups. So in some ways I come from a similar background to Coco Brumme’s, but I am older.She covers all the important ground in her book.There are several ways to understand satisficing, the term Herbert Simon used. Brumme describes it as stuff you do to patch up a model when your optimizations have outcomes you don’t like. But I read some other author who said natural selection does not optimize, but satisfices, and the main rule to satisfy is “don’t die”. You don’t have to be the best at anything: just don’t die.My own spin on her book is that we and future humans will live in the junkyards of today, and reuse and recycling from those junkyards will be the focus of our lives. That includes media junkyards like old libraries.(And perhaps you should ignore the five stars and wait for cheap used copies of this book to come out in three or four years. Or get it from your public library. Who knows what else you might find, if you wandered through the stacks of your library? Or shopped at a Goodwill Store? Or scavenged stuff from a dump?)

Whenever I begin reading something and encounter phrases like “everyone knows” and “something is lost”, then I do not expect to learn much. Unfortunately, that is the case with “Optimal Illusions”. The book really ought to be titled “Optimal Delusions”: an illusion is a mistaken perception, and a delusion is a mistaken belief. The author writes that “everyone” is deluded into believing that “optimization” must be the primary criterion for improving the human condition, and because of this “something is lost”. However, the benefits of optimization are not delusions. It is hardly possible for a competent researcher to believe that optimizing a model to improve a process does not have trade-offs. The reason is because developing a good model is impossible without reviewing all factors that affect a process. Models must leave out some factors, and emphasize others. Most researchers understand that “all models are wrong, but some are useful”, and “a model should be as simple as possible, but no simpler”.One wonderful aspect of mathematical optimization that is not discussed in the book is its use for better understanding cause and effect. In physics the laws of nature are expressed mathematically as optimizations. For example, of all the possible ways that a physical system can evolve, the only one that actually occurs minimizes energy. Instead of including such discoveries, the author dwells on worst-possible outcomes for making better widgets or providing better services. She admits that improving crop yields has saved billions of people from starvation, but nonetheless mentions only hypothetical outcomes such as soil depletion or the risks of mono-culture (and, of course, the subjective feeling that “something is lost”). So how much of Earth’s landmass is used to raise crops, and how much of that land has depleted soil (whatever that means)? Has mono-culture ever failed? If so, what was the economic impact? A mathematician should be able to quantify these downsides to optimization (and upsides too). The author does not discuss her own past optimization work and what she would do now to compensate for possible negative consequences. So is her solution to simply not optimize? Let people unnecessarily starve? Keep product prices high to the detriment of the poor? Of course, what will actually happen is that bad outcomes (and possibly hypothetical ones) will be eliminated by more inclusive optimizations! The incentives for doing so are altruistic (it’s the right thing to do) and for business success (more profit); both are powerful motivations and are morally defensible, and therefore will occur.The book describes her travels to determine the causes of optimization “failures”, which basically means collecting anecdotes from people who have decided to go off-grid and use traditional non-optimized methods. Such anecdotes have been reported for decades and are nothing new (remember “hippies”?). The book can be read as a personal-growth journal, which might interest some readers. But the book is not about mathematics, nor a detailed historical review of the evolution of optimization methods, and it does not offer solutions to the problems discussed. It seems to be the misguided product of an unwarranted guilty conscience. A much better book (with math and history) is “When Least Is Best: How Mathematicians Discovered Many Clever Ways to Make Things as Small (or as Large) as Possible” by Paul J. Nahin.

A collection of engaging, very human stories written with a breezy, approachable, easy to read style. I think the main point here is the unforeseen consequences of an approach to efficiency that solely relies on a data for decision making, seeking to maximize or minimize particular metrics. While one may well ask "isn't optimization gone awry just optimizing for the wrong thing?" I think there is a deeper point here, that *just* looking at data for decision-making will always be susceptible to the imperfection of any model, and we - professionals/citizens for whom this is relevant - are asked to consider supplementing anlaysis with other methods that depend on a deeper understanding of what humans, present and future, might value.

The entire time I was reading this book, I couldn't push the image of a snake eating its own tail from my mind. Krumme does a beautiful job of taking the reader on a deep dive journey into the systems that have managed to infiltrate nearly every aspect of our lives. Through winding stories that stretch and circle across the US, she reveals how, bigger, faster, and more, is essentially an illusion that keeps us disconnected and discontent. And, in our attempt to optimize the mistakes of optimization, we find ourselves running in place.

The statistician George Box quipped that “all models are wrong, but some are useful”. So useful, perhaps, that they (or more accurately their use) have come to shape much of not just our economic but also our political and social views. We justify much of our behavior on the basis that it is optimal based on often brittle and limited models of reality. But there are costs to the use, often unconscious, of this optimization. In a series of beautifully written case studies, the author, a recovering high-level practitioner of mathematical modeling, big data and optimization, examines some of these costs. Very moving especially for a book on such a seemingly cold idea as optimization.

The topic was interesting but the primary thesis was not supported by very good examples. Not too pleased with author’s writing style which wandered from topic to topic. Disappointed because it was potentially a very interesting subject since I have a background in statistics.