Money Stuff: AI Labs Want Someone to Stop Them

When it goes public, probably this year, Anthropic PBC “is likely to tell investors its potential revenue opportunities are above $30 trillion, topping SpaceX’s $28.5 trillion estimate.” That would give Anthropic the largest total addressable market in the history of finance. I mean, that’s just a number that they write down; it’s not like Anthropic expects $30 trillion of revenue in 2027, or in 2057. But in theory, the maximal upside case for Anthropic is something like “one-quarter of all human economic activity will be done by artificial intelligence, and Anthropic will capture all of it.” 

What is the negative of a total addressable market? Like, if the maximal upside case for Anthropic is $30 trillion, what is the maximal downside case? With the benefit of hindsight, you could imagine a tobacco company going public in 1970 saying “our maximum upside case is 8 billion humans smoking two packs a day of our cigarettes, and our maximum downside case is paying billions of dollars in damages for causing lung cancer,” but Anthropic is on a whole different scale. Anthropic’s negative TAM is “you and everyone else on earth will be killed by our AI.” [1]  

I want to be clear that I do not think that Anthropic Chief Executive Officer Dario Amodei’s call to slow down AI development to prevent misaligned rogue AI is insincere or intended as marketing hype. But I do think that it is great marketing. In hindsight it is crazy that the SpaceX initial public offering prospectus does not have a risk factor saying “there’s a 10% chance our AI will kill everyone on earth.” I mean, maybe SpaceX doesn’t believe that! But Anthropic does, and arguably the closer you are to the frontier, the more likely you are to be deadly. Which means that you ought to have a “we might kill everyone” risk factor, if you want to get IPO investors excited about your capabilities. When OpenAI goes public, I assume it will need to claim a 15% probability of killing everyone.

Anyway. Amodei published his call to “pace the frontier” on Saturday. Sam Altman of OpenAI and Elon Musk of SpaceX endorsed it; Google and Microsoft also seem to mostly agree. Others do not, including US President Donald Trump:

“We’re leading China in AI. We’re the most sophisticated country in the world, and frankly, I want to keep it that way because whoever wins AI wins,” Trump told reporters on the sidelines of the Irish Open golf tournament. “And we can put guardrails. We can do this and that. But I think you have a lot of very negative forces that are bringing it up that shouldn’t be bringing it up.”

And Chinese officials:

“Fearmongering, confrontation and vicious competition will only disrupt the process of global AI governance and serve the interests of no one,” Foreign Ministry spokesman Guo Jiakun said at a regular briefing in Beijing on Monday.

And:

[David] Sacks, a frequent critic of Dr. Amodei, questioned why tech companies needed government intervention to slow down their A.I. development when nothing was stopping them from doing just that — particularly as they pursued a wildly ambitious goal to create so-called superintelligence, or A.I. that is far more intelligent than humans. …

“If developers of A.I. want to slow down development of their product they are free to do so,” said Senator John Cornyn, Republican of Texas, in a social media post on Sunday. “One thing is for sure: their competitors won’t, including our nation’s adversaries.”

And:

“Should a handful of select, market-dominant A.I. companies from Silicon Valley get to define the rules and safety standards of a generational technology for the entire world?” said Aidan Gomez, the chief executive of the Canadian A.I. start-up Cohere, in an essay on Sunday.

“These oligopolies are now requesting to bend competition rules and be permitted to dictate the terms for everyone else,” he added. “A wolf in sheep’s clothing, a cartel by any other name.”

It is a little strange that an industry is overwhelmingly saying “regulate us, otherwise in our pursuit of mammoth profits we will kill everyone,” and the US government is like “nah we’d rather you have the profits.” But we live in strange political times.

That said, you could tell a traditional antitrust sort of story:

1. Anthropic, OpenAI, and perhaps a couple of other frontier labs are the dominant providers of frontier AI models.
2. They can charge customers a lot of money for using their frontier models, and rather less money for older, no-longer-cutting-edge models.
3. Training a new frontier model requires ever-increasing billions of dollars of computing power.
4. The labs need to more or less continuously race to build new frontier models, because their competitors are all doing it, and if they don’t they will fall behind and no longer be able to charge a lot of money for their best models. (Also because they intrinsically want to build artificial superintelligence, for cancer-curing and/or killing-everyone reasons.)
5. If they collectively slowed down, then (1) they’d spend less on compute and (2) they’d be able to charge frontier-model prices for a longer time.
6. But if one of them slowed down, the others would eat its lunch.
7. If they got together in a room and agreed to slow down, that would look like an antitrust conspiracy: It is generally illegal for competitors to get together and agree to limit the output of their industry.
8. But if they publish papers about how important it is to slow down, that might have a similar coordinating function, at least among the US frontier labs if not necessarily among their Chinese competitors.
9. And if the government believes those papers, it might help them coordinate. Maybe the government will impose pacing by regulation that the labs could not impose by agreement. Or at least the government will let them get together and agree to slow down. Amodei’s post calls for “frontier AI companies within democratic countries [to] coordinate to establish common safety standards as well as limits on the rate of unchecked AI progress”; a footnote adds: “With government mediation or waivers of antitrust restrictions.” Just meeting in a room to establish common safety standards is legally risky; the labs can’t do it on their own unless governments affirmatively allow it. (Though the Information reports that “Anthropic, OpenAI and Google have been holding discussions about working together to create a standards body for the AI industry.”)

We talked once about a lawsuit claiming that environmental, social and governance (ESG) investors were violating antitrust law by pushing coal companies to produce less coal. The ESG investors wanted the coal companies to produce less coal to reduce harmful carbon emissions, but the complaint argued that pushing coal companies to produce less coal is a classic antitrust violation that limits output and so pushes up prices. Trying to save the world is no excuse for an antitrust conspiracy. You could imagine a similar complaint here: If all of the AI companies agree to produce less superintelligence, even for extremely good avoiding-human-extinction reasons, that could look like an antitrust violation.

I have a running joke around here about weird wrinkles of the US legal system that might stop artificial superintelligence from killing us all. Like, what if copyright rules or the Onion Futures Act or “everything is securities fraud” is what prevents AI from becoming all-powerful? That joke has become less funny over time, in part because AI has run right over some of those legal wrinkles. (Remember thinking that copyright law might limit AI companies?) But I suppose the reverse is now a possibility. What if the well-meaning humans in the best position to stop rogue superintelligence are willing to work together to stop it, but they can’t because of antitrust law?

Look it’s just very untidy:

1. The big stock indexes are largely float-weighted, or float-adjusted in some way. A company with a $1 trillion market capitalization, but with only $50 billion of freely traded shares, does not get a $1 trillion weighting in most indexes: If it did, index funds might be forced to buy more shares than are available. Most simply, the company could get a $50 billion weight in the index (i.e., the index could be float-weighted), though there are other possible approaches. (The Nasdaq 100 would give that company a $150 billion weight: more than its float, but less than its market cap.) 
2. When a company first goes public in an initial public offering, it will usually have a very low float: It might sell 5% or 10% or 15% of its stock in the IPO, and most of the rest of its stock will be subject to lockup agreements preventing holders from selling for some period, often six months. So for the first six months after it’s public, the company will have a very low float. After that, more float.
3. If the company gets into the index shortly after it goes public — say, because it is a big company and a lot of index providers (including Nasdaq) have decided to “fast-track” index inclusion for very big newly public companies — then it will have a relatively low weighting, because of its low float.
4. After the lockup expiry, it will have a higher weighting, because of the higher float.
5. Therefore, when the lockup expires, index funds will predictably have to buy a certain number of additional shares of the company, to reflect its very predictable increased float.
6. Also, when the lockup expires, there will be selling: The early investors who were subject to the lockup are no longer subject to the lockup, so they can sell, and some of them probably will. (That’s what it means to have a higher float: More shares are available for sale.)
7. “Aha,” you think tidily, “the people with shares to sell at the lockup expiry can sell them to the index funds who have to buy because of the increased weighting.” Like: The buying pressure and the selling pressure have exactly the same source, the lockup expiry, so they should just trade with each other in one big auction. (The buying pressure and selling pressure won’t have the same size: There’s no reason to expect that the number of shares sold by locked-up investors who want out will exactly match the proportion of the stock owned by index funds. But they’ll at least somewhat offset.)
8. But, no. The lockup expires, and people who were locked up can sell, leading to selling pressure. Then, much later, at some periodic rebalancing, the indexes will adjust their float numbers, leading to buying pressure. The two events, which are the same event and offset each other, don’t happen at the same time. The index weighting does not adjust instantaneously to reflect the increased float. 

Bloomberg’s Isabelle Lee reports:

SpaceX is set to get a larger weighting in the Nasdaq 100 later this month, a change that could trigger billions of dollars of buying by passive funds tied to the benchmark.

Its weighting is expected to rise to about 2.82% from roughly 1.28%, based on pro forma data from Nasdaq’s Global Index Watch circulated late Friday. The final weighting will be determined later this month. Wall Street expects the change to result in billions in new buying by index funds and exchange-traded funds that track the benchmark. …

SpaceX’s first lockup expiration came in August, alongside its first earnings release as a public company, prompting concern that a surge in newly tradable shares could overwhelm demand. The selloff didn’t materialize, including after a second lockup expired a week later. Insiders largely held onto their stakes and the stock held up.

Very annoying! As we have discussed around here, hedge funds are in the business of smoothing out index-fund demand at rebalancings, by buying stocks before they are added to the index and then selling them to index funds on the rebalance date. And I have half-seriously mused that hedge funds could be in the business of smoothing out insider supply at lockup releases, by short-selling stocks before the lockup release date and then buying them back from formerly locked-up shareholders when they sell. And you can mash that all together and be like “ahh, hedge funds can buy shares from the insiders on the lockup release date, warehouse them for a few weeks, and sell them to index funds when the index finally adjusts to reflect the increased float.” But that is risky and imprecise and has a lot of friction, and “let the insiders sell to index funds when their lockup expires” would be much tidier.

I was sort of fascinated by TWG Global’s statement, after it was caught misstating its investments in affiliates, that “there is no victim here. No one has been harmed, and no one has claimed they were harmed.” TWG is Mark Walter’s holding company, which owns (1) some insurance-and-annuity companies, like Delaware Life and Clear Spring Life and Annuity, and also (2) a collection of stakes in big sports teams. The insurance companies sold annuities to customers and, it seems, invested a lot of the customers’ money in Walter’s other businesses, while saying that they didn’t. Prosecutors and regulators started asking questions, and “after receiving subpoenas, those insurers disclosed more than $20 billion of loans that should have been labeled as affiliated transactions, but weren’t,” oops.

What does it mean that “there is no victim here,” that no one has been harmed or claimed that they were harmed? Here are some possibilities:

1. The investments that the insurance companies made in Walter’s companies have not defaulted, and do not seem likely to default. This one seems true.
2. The investments that the insurance companies made in Walter’s companies have not declined in market value. This one is probably also true, though on a technicality: They are largely private investments that don’t trade. The point of running an insurance company and selling annuities is to be able to make long-term illiquid investments without the risk of having to sell them in a fire sale. 
3. The annuities themselves have not defaulted: Customers are still getting their annuity payments. Clearly true.
4. The annuities themselves have not declined in market value: Again, hard to evaluate, because the annuities are bilateral contracts and don’t really have a market value. In theory, you might say “well, the customers bought the annuities at low discount rates given their regulatory approval and good ratings, but now the regulators want corrective action and the ratings agencies are nervous, so the value of the annuities has declined.” But there is no market price showing that.
5. The annuity customers do not otherwise feel aggrieved: They were not, like, relying on the insurance companies’ (incorrect) disclosures of their related-party transactions; they didn’t read those disclosures. The customers are blithely ignorant of the whole controversy. Surely that’s mostly true; nobody reads anything.

This is all a little unsatisfying. I have previously analyzed it by saying sure, right, there are no actual defaults, no actual declines in market value and no actual customers who care, but those are mostly structural features of the annuity market. The point is that the regulators (and ratings agencies) make sure that the insurance companies take only appropriate risks, and if the regulators and ratings agencies don’t have accurate disclosures then that’s bad, even if there are no identifiable individual customer “victims.” 

On the other hand, this has been in the news for a while now, and eventually the customers — or the financial advisers who sell them the annuities — will notice. Here’s a Wall Street Journal article about some of them:

Rick Phillips has been buying annuities for about a decade, paying upfront sums for the peace of mind of fixed returns. 

Now, the Los Angeles-area resident says he is swearing off the policies. 

The insurer backing one of his contracts, Clear Spring Life and Annuity, revealed this summer that federal prosecutors are scrutinizing its investment disclosures. Soon after, the company’s billionaire owner, Mark Walter, struck a deal to sell his controlling stake in the Los Angeles Lakers. That left Phillips worried that Walter’s insurance business was on shaky ground.

“The risk-reward isn’t there anymore,” said Phillips, a retired investment banker.

And:

Clear Spring, the insurer that issued Phillips’s annuity, had solid investment-grade ratings and quarterly financial disclosures that looked healthy. That gave financial adviser Carlos Dias Jr. the confidence to sell the firm’s annuities in recent years. Then in June, he was shaken when Clear Spring and Delaware Life disclosed that they had billions more than previously disclosed in affiliated assets.

AM Best said it was considering downgrading the companies.

Like my simplistic model was that annuity customers do not review insurers’ financial statements to evaluate their risk-reward ratio, but that is not always true.

Many businesses have the basic profile that there is some upfront investment, followed by profits over time. If you build a factory or found a startup or train a frontier AI model, the cash flows in the first year will probably be negative — you have to spend money to get started — but, you hope, in future years they will be positive: You have already spent the money to build the asset, and now you can harvest it. In this sort of business, growth is expensive: Every time you open a new factory, it costs money. You might grow slowly, using the profits from your existing factories to pay for new ones. Or you might accelerate your growth by raising money from investors, explaining that the long-term profits more than justify the upfront investment.

But other businesses have the basic profile that there is a big lumpy upfront cash flow, followed by losing money over time: A customer pays you $100 upfront, and then you spend $30 per year servicing that customer. (The annuity business is a little like this.) This sort of business requires, you know, better accounting than the other sort. “Ah sweet I’ve got $100, I’m rich, time to spend it all!” would be a bad reaction to this sort of business. You gotta plan for the future; you gotta make sure that the business is profitable over its whole life cycle. If every customer brings in $100 upfront and costs $300 over time, that’s bad.

If you are not especially careful, though, you might find yourself growing uneconomically. You need $30 per year to service your existing customer, you don’t have it, you think “hmm if I get another customer I’ll have $100,” you sign up another customer for $100, you spend $30 servicing the existing customer and spend the rest on yourself, next year the problem recurs, you find another customer, etc., and eventually you find yourself running a vast business that loses money on every transaction but makes up for it with rapid growth. Seems bad. Is this “a Ponzi scheme”? I dunno, maybe a little. It is not quite as deceptive as a traditional Ponzi scheme, but it has some of the same fundamental economic problems. Perpetual negative-value growth is not a good long-term strategy.

At the New York Times, Claire Suddath has a story about an alleged “Montessori Ponzi.” Basically: It is expensive to operate a preschool, but it is weirdly lucrative to open a preschool, because commercial landlords often give tenants cash upfront for tenant improvements. Etc.:

“We were calling it the Montessori Ponzi scheme internally,” said Alex Richardson, a teacher at Guidepost’s first school, in Orange County, Calif.

When Higher Ground opened new Guidepost schools, it often received large advances from landlords to improve their properties. As long as the company kept expanding, it seemed from the outside as if it were thriving. But when it came time to repay the landlords, and growth was no longer an option, the company collapsed. By the end, some Guidepost locations were losing $50,000 a month. …

Kevin McDonald, a landlord in Missouri, said he doubted that the money he’d advanced for a new Guidepost was used entirely at that location. “In my opinion, they’re not ethical people,” he told me. “They borrowed from us to help them open up other schools. In our case in Ellisville, they asked for $1 million in tenant improvements. Not even a third of that went to that school.” He clarified that this was an estimate. “You can just figure it out,” he said. “They didn’t use the money. When I went to the school a couple of times, I saw it.”

[Co-founder Ray] Girn wrote in a text message this month that he’d created “an incredible business model that is unfortunately very misunderstood,” adding: “Yes, the ‘float’ did impact the timing of our capital cycle, and there are legitimate criticisms one can make of our strategy, but it’s just plain false that we did anything negligent or nefarious.”

According to bankruptcy filings, the company opened 33 schools in 2020 and another 21 the next year, while cumulative losses grew to $154.6 million.

I would not have guessed that Montessori preschools would be a good substrate for an “incredible” but “very misunderstood” business model, but here we are. Though after the bankruptcy, some of the schools pivoted to another business model: “In California, a mother learned that her children’s Guidepost had been sold and that its Montessori curriculum would be replaced with artificial intelligence.” Also promising.

I have written a handful of times around here about a problem facing the technology industry in the age of artificial intelligence, which is that (1) the going rate for top tech employees is “enough money that they can retire immediately” and (2) what if they do? If you pay employees enough to be competitive with other companies, you will be paying them enough that they don’t have to work anymore. How can you motivate anyone? We have discussed two possible solutions, which are:

1. “$30 million will buy you a two-bedroom in the San Francisco metro area,” and
2. For some reason the widespread belief, among AI researchers, that their work will kill everyone on earth seems to motivate them to keep doing it rather than quit and enjoy their money while they can.

I realize that the second solution is counterintuitive, and honestly the first is too (leave San Francisco!), but whatever. At Bloomberg Weekend, Tiffany Ap discusses another possible approach, which is to convince tech employees that there is a thing called “sudden wealth syndrome” and that it is bad, so the safest thing to do, when you come into sudden wealth, is to ignore it and keep working:

[Centimillionaire podcaster Anastasia] Koroleva explores what happens when net worth removes the need to work. Her interviews with the extraordinarily wealthy have uncovered a recurring theme: For many, the exit does not feel like a crowning achievement — or, at least, not for long. It can feel more like an existential rupture. Guests describe becoming staggeringly rich using the language of trauma: “recovering from a $500 million sale,” “thrashing about for a decade” or drifting in a state of aimless ennui. …

“There’s been a tremendous softening of conversations around mental health and well-being among entrepreneurs,” says Sherry Walling, a clinical psychologist who specializes in founders. She credits that partly to the shock of Zappos founder Tony Hsieh’s death in 2020. “I do think that we’ve moved on from it as the world’s tiniest violin.”

Last November, Walling co-authored a playbook on the subject, Exit Strategy. She’d seen too many clients who’d done everything they set out to do, achieved the big exit — and were miserable. “There were some really dramatic stories,” she says. “People being suicidal, just really, really lost in their post-exit experience.”

Best not to risk it!

Amodei, Altman, Musk Call for Slowing AI Model Development. Trump Downplays AI Concerns as CEOs Call for Slowing Technology. From Bailout to Mayfair Rival: Greece Lures Hedge Fund Elite. Texas Challenger to Nasdaq, NYSE Lands Fifth Company in a Week.  Euronext open to ‘big bang’ deal with rival Deutsche Börse. Cluster of Polymarket Accounts Won Big on Companies Audited by KPMG. AI Data Centers Are on Track to Fuel ‘Explosive’ Growth in Captive Insurance. Apollo Said to Shutter Some Invoice-Financing Products at Eliant. BIS Frets About Leverage After Situational Awareness Emergency. Pfizer Makes an Unusual Enemy in $2 Billion Fight Over Unwanted Covid-19 Vaccines. Larry Ellison cancels $7.5bn Oracle share sale. Musk and SpaceX Face Expensive Tradeoffs on the Road to Mars. Super El Niño Spurs Traders to Place Early Bets on Mild Winter. J&J Is in Talks to Sell Its Hips-and-Knees Business to Apollo for $20 Billion. Lost in Margaritaville: The Messy Drama Over Jimmy Buffett’s $275 Million Trust. CIA Official Accused of Stealing $40 Million in Gold Bars Strikes Tentative Plea Deal.

If you'd like to get Money Stuff in handy email form, right in your inbox, please subscribe at this link. Or you can subscribe to Money Stuff and other great Bloomberg newsletters here. Thanks!