Objective function, what's your function? How to grok AI and existential risk
It's the optimization, stupid!
Optimization is AI’s original sin
Algorithms work through optimization. You give them a set of inputs (like possible groceries), set an objective function (a budget of $20), and then they spit out an output (a shopping list).
I was my own algorithm Saturday morning at the Fort Greene Farmer’s Market and I output a list two items long: peonies ($15) and Evercrisp Apples ($5). I hate it here, I’m moving to Phoenix, etc.
The tricky bit about optimization is that you can only optimize for one thing, more or less. The fanciest algorithms (like neural networks) will juggle lots of inputs, but their output will be optimized for one thing.
The challenge of product design is in balancing multiple goals. TikTok’s Feed, for example, is powered by algos that balance your experience against their revenue (i.e. showing you ads). They’ve nailed the formula — it’s so addictive! — and yet however sophisticated it may be, their algo ultimately optimizes for a single thing, engagement— and in the precise way that’s best for TikTok’s business.
This is what’s dangerous about optimization. Optimization is a necessarily single-minded process that feels multidimensional — that’s the sleight of hand. Us silly humans trust in the machines, in part, because they seem to be taking everything in and coming to the best decision.
I can tell you from experience that creating an algorithm certainly feels complex. It’s a long and painstaking process to fine-tune an algorithm to reach that perfect balance between user value and business impact — but you always know how the story ends.
Algorithms are really just optimizing for the one thing that their creators care about — which is, more often than not, revenue. The profit motive is our era’s original optimization algorithm. If AI’s original sin is optimization, then optimization’s original sin is profit.
Humans suck at accomplishing goals
Humans are famously messy and tend, like the universe, toward entropy. Machines are brutally efficient and notoriously myopic. The difference between what we can do and what AI can do comes down to focus and scale.
It is (usually) a good thing how much humans suck at accomplishing goals. Unlike machines, humans pursue goals by optimizing for a great many factors — like each other and the future and social norms and the environment. Sure, we bicker, we get off course, we fuck up, etc. but we generally land in a considered place. We have tools to get to better outcomes — strategy, process, management — but we can’t help but have wide concerns.
Machines just optimize for the one thing.
The problem with optimization
When you optimize for one thing, lots can go wrong. You might create unintended consequences (like misinformation or radicalization), spread discrimination (by reproducing historical biases), harm the environment (by optimizing for short-term goals), create economic externalities (like price discrimination or monopolistic practices), and otherwise send human society down a bad path of new addictions, poor mental health, doomscrolling, and more.
Side note: I used ChatGPT as my little research assistant for this post and it did a decent job. It only works in this instance because I know enough to factcheck the output. You can see the conversation yourself here because ChatGPT now allows sharing. It’s a neat feature. Though if I’m wearing my crotchety Michael hat, I’d say it’s irresponsibly designed and will probably fuel misinformation. (If I’m wearing my philosopher’s hat, I’d say I’m intrigued by the concept of a conversation as a unit of knowledge.)
Machines suck at knowing their limits
OK, back to algos. Algos work through constraints; you code constraints into algorithms as a way of guiding optimization. I often wish, in this very newsletter, that companies programmed more constraints into their algos for more responsible product development.
The problem with AI is that this isn’t quite possible anymore. Because we don’t understand how AI models makes their infinite optimization decisions, we can’t anticipate every way things could go wrong — and plant the right constraints. This approach was somewhat possible with the current crop of “AI” — from the TikTok Feed to Youtube’s recommendation engine. But the next crop of AI technology — Large Language Models (LLMs) like ChatGPT — is too sophisticated and opaque for us to do the same, and there’s the rub.
Take ChatGPT. It’s not immediately clear what ChatGPT is optimizing for, which is plausibility. ChatGPT is trying to guess the next best word in a series. The answer needs to seem plausible for its human users — the humans who trained it, and the consumers who use it. ChatGPT is telling us, essentially, what we want to hear, whispering “hallucinations” that ring true… but it’s just a sweet sweet fantasy baby. This is called sycophancy bias.
At scale, it’s dangerous to have AI products that are opaque, operate without constraint, and tell us what we want to hear.
On last week’s Hard Fork, AI researcher Ajeya Cotra explained how this quality of AI might ultimately lead to our species’ demise. One hypothesis is that at a certain point, AI will scale so widely that most humans and companies will reply upon AI for most decisions. Business competitors and geopolitical enemies alike will pit their AIs against each other. And in this world — the “obsolescence regime” — humans are second class. Eventually, AIs may decide to move forward without us, and they will. It’s a great segment that starts ~26 minutes in.
I always loved the ending of Her — spoiler alert — how the AIs just get bored of us and dip.
What are you optimizing for?
Product people often ask: what is this optimizing for? Usually it’s obvious, but not without nuance — optimization can happen on various levels even in one feature (from the algorithm to the UI). If you look hard enough, though, you can reverse engineer a product’s strategy and business model.
Compare Instagram’s feed with TikTok’s. Both have algorithms that optimize for engagement (more time = more ads) — but for different kinds of content. Instagram’s main feed shows you engaging content from accounts you’re likely to follow — building upon Facebook’s rich social graph. TikTok’s feed, on the other hand, shows you content agnostic of account, and makes it painless for you to swipe through — providing instant algorithmic feedback and feeding TikTok’s interest graph. TikTok discovered that if they optimize for pure engagement — no social connections, just content — they can better short-circuit your brain. XML
Links this week
More on TikTok’s algo-first product design (Eugene Wei’s blog)
“Are we inventing a tool or a creature” — Sam Altman, referencing the anthropomorphism in ChatGPT I wrote about last week (New Yorker podcast)
On the lawyer who phoned in a legal filing riddled with ChatGPT hallucinations (Ars Technica)
A beautiful use of AI: ALS patients “voicebanking” while they can still speak (Washington Post)
A crisis line fired it staff after they unionized, replacing them with a chatbot — only to suspend the chatbot one week later. This is the moment. (Fortune)