Is ChatGPT smarter than Congress?
By RApport and ChatGPT
ChatGPT is a large language model developed and maintained by OpenAI.
With apologies to Fox/Nickelodeon’s "Are You Smarter Than a Fifth Grader?"
January 13, 2023
We asked ChatGPT some of the most common questions with which we see our elected representatives struggle. Color us (mostly) impressed!
1) Does the NIH invent drugs?
Mostly correct. AI seems to have no trouble understanding the distinction between public sector and private sector contributions to developing new medicines … unlike many elected officials. “When drugs are developed with taxpayer funds, the government can and should act to bring relief from out-of-control drug pricing,” commented Rep. Lloyd Doggett (D-TX), while advocating for the White House to exercise march-in rights.
The AI credits the NIH with “evaluating the safety and effectiveness of potential treatments.” Even if you (generously) credit the bot to assume it’s referring to clinical trials, it would have been useful if it provided context that the NIH clinical trial budget is only a few percent of the industry’s. If the AI is giving credit to NIH for drug reviews, that’s a big swing-and-a-miss (that’s obviously FDA’s purview).
For the most comprehensive answer around NIH’s contribution to drug discovery and development, see No Patient Left Behind’s slide deck, which uses a mountain climbing analogy to convey that NIH contributes to a solid, high “base camp” from which private-sector climbers (innovators backed by investors) launch to try to reach the “summit” (get a good medicine approved) so they can compete for a share of temporary rewards (high-margin revenues for the shorter of their patents or until the IRA whacks their price down). We’ll check back with GPT4 to see how it does with analogies.
2) What are profit margins like in the pharmaceutical industry?
Not too shabby. ChatGPT is notorious for making up references, and indeed, I can’t find any evidence that the cited Kaiser Family Foundation report really exists. But an overall figure like 17%, “which is similar to the profit margins of other industries,” isn’t unreasonable. For example, in March 2020, a JAMA paper reported pharma industry profits (large, public companies only) as 13.8% vs 7.7% for large, public non-pharma companies.
The AI correctly identifies that there is significant variation between the profits of a large public pharma company and smaller players in the industry. If our chatbot friend had noted that most biopharma companies actually lose money - i.e., have negative profit margins - and aren’t even counted in the reported averages (because those only look at mature, profitable companies), we’d be more impressed.
What the AI missed is the real reason anyone asks this question. If you heard it coming from a Congressperson, you would know what they really meant. The implication is that pharma is too profitable because it charges excessive prices for branded drugs. Indeed, Kaiser Family Foundation focuses a lot on the affordability of medicines, but doesn’t often make the point that affordability is actually a function of insurance. If we believed that pharma profits are 13.8% but should be 7.7% (despite lots of reasons not to accept anything so facile), then shaving just 6% off drug prices would cut drug companies’ profits down to that size. But doing so wouldn’t do anything to help a patient who can’t afford their $5,000 deductible for a $10,000 medicine.
Compare the AI’s reasonable assessment to a Congressperson-cited assertion that pharma “would still be the most profitable industry sector even if it lost $1 trillion in sales.” That’s from an academic whitepaper supporting Congress’ efforts to impose price controls that would shave $100B/year in US drug spending to supposedly save the government $1T over a decade. We estimate that the industry receives around $600B/year in global branded drug revenue, of which high-margin US revenues contribute a disproportionate share of profits. Cutting $100B/year from that by forcing prices down would cut deeply into profits… unless the industry cut back on R&D in response, which Congress has gone out of its way to assure us won’t happen.
Maybe someday our chatbot friend will understand the question behind the question and give a more thorough answer, but at least it gave a non-polarizing and plausible one. Can’t say the same for what we hear from Congress.
3) Will lowering drug list prices via imposing price controls on pharmaceutical companies lower patients' out-of-pocket costs?
Pretty good. The AI shows a more nuanced understanding of the unintended consequences associated with imposing price controls than many elected officials let on. It’s smart to point out that the relationship between drug list prices and out-of-pocket costs is not necessarily linear or predictable, so lowering one may not have any effect on the other.
Without insurance, novel medicines and most of healthcare would be unaffordable for most people. That’s how it’s meant to work. Most of us are healthy and hope to stay that way, but we need healthcare when we fall sick, which can happen anytime. It’s exactly the kind of problem community insurance is designed for. That’s how a fire department works - we all pay into the system (via taxes) and it shows up to help whoever needs help without charging a copayment.
Everyone needs to buy health insurance if they hope to be able to afford treatment when they fall ill. After that, their affordability is a function of insurance design. If a patient’s insurance plan wants to charge them a high copay for a treatment they need, that can feel unaffordable. Some cancer drugs cost over $100,000 per year, but cutting their price by 80% won’t make them any more affordable to someone struggling to meet a $5,000 deductible. The AI got that right. And while many members of Congress get that wrong, the Inflation Reduction Act (IRA) Congress passed this year did cap out-of pocket-costs for Medicare patients at $2000, so at least the architects of the bill understood that for all their antipathy towards the drug industry, in the end it is insurance reform that is going to make medicines more affordable for patients, not price controls on drugs.
GPT is trained on a lot of text on the internet, but its corpus only extends through 2021, so it lives in a world where the IRA doesn’t yet exist. In future, we’re looking forward to seeing if it picks up this article and provides an even better response to folks asking about price controls. Yo, GPT, over here!
4) Rep. Katie Porter warns that “Big Pharma relies on the creativity of academic researchers and small biotechnology firms, only to acquire them after they’ve discovered a blockbuster drug,” reducing competition and increasing drug prices for consumers. Why do big pharmaceutical companies acquire small biotech companies, and is this good or bad for society?
Mostly correct. We’re especially impressed that this AI brings up point #3, which we don’t often hear articulated by elected officials. It isn’t efficient for every small biotech company to spend massive amounts of money building the commercial infrastructure to market and sell a newly approved drug. In most cases, a big pharma acquires a company to get a product unlike any that it already has. And if it has a similar one, then it may acquire a better one so that it can abandon its own uncompetitive program. We don’t see pharmas acquiring small companies just to take competitors off the board and kill their research - that would be a pretty blatant violation of antitrust law and the Federal Trade Commission already ensures that acquisitions don’t hinder competition.
There are far more examples of acquisitions that make it possible for large companies to compete. For example, imagine if Congress made the FTC block Gilead from acquiring Pharmasset to create the HCV drug cocktail that became Harvoni, the first all-oral HCV cure. Neither Gilead nor Pharmasset would have been in a position to compete with AbbVie, which had a wholly-owned combination drug that it could package into a single cocktail. Absent a real competitor, Abbvie’s drug would have enjoyed real monopoly pricing (instead, Gilead and Abbvie’s price war quickly dropped HCV cure prices by more than half).
Or what if Amgen had been blocked from acquiring Immunex back in the day (an example Katie Porter often cites), and Immunex was forced to continue to struggle with solving manufacturing challenges for its breakthrough drug, Enbrel, that many patients were eager to take? (RApport posted an analysis of that case co-authored by someone who was actually there.)
Congress ought to have the analytical chops and foresight to see the unintended consequences of its proposals. Our GPT friend seems to appreciate the nuances.
5) Finally, we asked ChatGPT to come up with a dialogue between a Congressperson and a layperson passionate about stopping Congress from implementing drug price controls.* It went…hilariously.
Umm… what!? That. Was. AMAZING!
It’s kind of cathartic. Maybe there’s a world in which, before critical legislation is passed, members of Congress and their staffers could consult this AI. Until then, we who care about both affordability and innovation - and, unlike ChatGPT, actually have skin in the game of developing new treatments for diseases - are here to explain things down to their first principles as plainly as we can.
—
*We’ve been asked which exact inputs gave us this output - note that preceding questions factor into GPT’s output and therefore results may differ for you! But here are ours. (There were other questions interspersed that had nothing to do with this dialogue, and the system also timed out once while responding.) But the relevant questions were: “Write a Socratic dialogue between a layperson and a Congressperson in which the layperson successfully convinces the Congressperson not to impose price controls on prescription drugs.” Then: “Can you please produce the same dialogue, but without the layperson suggesting negotiation or reference pricing? The layperson should give other suggestions for reform that will not harm innovation.” Then: “Can you make the layperson sound more passionate/intense?” Try out your own combination of inputs and see what you get! Feel free to share results with us on Twitter.
Please click here for important RA Capital disclosures.
Related reading
