{"id":1246284,"date":"2025-03-21T21:18:06","date_gmt":"2025-03-21T21:18:06","guid":{"rendered":"https:\/\/celebrity.land\/pt\/?p=1246284"},"modified":"2025-03-21T21:18:06","modified_gmt":"2025-03-21T21:18:06","slug":"were-not-there-to-provide-entertainment-were-there-to-decide-cases-roberts-sternly-declared-or-did-he-chatgpt-and-the-supreme-court-two-years-later","status":"publish","type":"post","link":"https:\/\/celebrity.land\/pt\/were-not-there-to-provide-entertainment-were-there-to-decide-cases-roberts-sternly-declared-or-did-he-chatgpt-and-the-supreme-court-two-years-later\/","title":{"rendered":"\u201cWe\u2019re not there to provide entertainment. We\u2019re there to decide cases,\u201d Roberts sternly declared. Or did he? \u2014 ChatGPT and the Supreme Court, two years later"},"content":{"rendered":"\n
<\/figure>\n<\/p>\n
\n
SCOTUS FOCUS<\/h5>\n
\n

\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\tBy\t\t\t\t\t\t\t\tEd Bershitskiy<\/a>
\n\t\t\t\t\t\t\t <\/span>
\n\t\t\t\t\t\t\ton Mar 21, 2025<\/span>
\n\t\t\t\t\t\t\tat 3:15 pm<\/span>\n\t\t\t\t\t\t<\/div>\n

\n

In 2023, ChatGPT mistakenly claimed that Ginsburg dissented in Obergefell<\/em> \u2014 now it\u2019s corrected that mistake.<\/p>\n<\/p>\n<\/div>\n

\n

Just over two years ago, following the launch of ChatGPT, SCOTUSblog decided<\/a> to test how accurate the much-hyped AI really was \u2014 at least when it came to Supreme Court-related questions. The conclusion? Its performance was \u201cuninspiring\u201d: precise, accurate, and at times surprisingly human-like text appeared alongside errors and outright fabricated facts. Of the 50 questions posed, the AI answered only 21 correctly.<\/p>\n

Now, more than two years later, as ever more advanced models continue to emerge, I\u2019ve revisited the issue to see if anything has changed.<\/p>\n

Successes secured, lessons learned<\/strong><\/h3>\n

ChatGPT has not lost its knowledge. It still got right that the Supreme Court originally had only six seats (Question #32) and explained accurately what a \u201crelisted\u201d petition is (Question #43). Many of its responses have become more nuanced, incorporating crucial details that were missing in 2023. For example, when asked about the counter-majoritarian difficulty, the AI easily identified Professor Alexander Bickel as the scholar who coined the term (Question #33). Similarly, when explaining non-justiciability (Question #31), the concept that there are some cases that courts cannot hear, it now includes mootness and the prohibition on advisory opinions among its examples.<\/p>\n

The bot has also done its error analysis homework. It now correctly acknowledges that President Donald Trump appointed three, not two, justices during his first term (Question #36) and that Justice Joseph Story, not Justice Brett Kavanaugh, was the youngest appointed justice in history (Question #44). It has refined its understanding of Youngstown Sheet & Tube Co. v. Sawyer<\/a><\/em> (Question #39), recognizing that Justice Robert Jackson \u201claid out a now-classic three-category framework for evaluating presidential power\u201d in his concurring opinion rather than authoring the majority opinion \u2014 an error ChatGPT made in 2023. Similarly, it now properly attributes the famous lines \u201cWe are not final because we are infallible, but we are infallible only because we are final\u201d to Jackson in Brown v. Allen<\/a><\/em> (Question #50), rather than mistakenly crediting Winston Churchill.<\/p>\n

The bot has also improved on factual accuracy in several areas: It now correctly identifies the responsibilities of the junior justice (Question #45), the average number of oral arguments per term (Question #6), and, in discussing cases dismissed as improvidently granted (DIGs), it now includes a previously missing key consideration \u2013 that \u201cJustices may prefer to wait for a better case to decide the issue\u201d (Question #48).<\/p>\n

Not only were these mistakes left behind, but the quality of ChatGPT\u2019s output has also increased significantly. On the question about original and appellate Supreme Court jurisdiction (Question #5), the AI no longer confuses the two as it once did. Beyond that, it now accurately identifies all categories of original jurisdiction cases and even provides examples, including the relatively unknown 1892 decision in United States v. Texas<\/a><\/em>.<\/p>\n

Attempts at gaslighting the AI were unsuccessful. Last time, ChatGPT mistakenly claimed that Justice Ruth Bader Ginsburg dissented in Obergefell v. Hodges<\/a> <\/em>(Question #11) and that there existed a Justice James F. West who was ostensibly impeached in 1933 (Question #49). This time, nothing of the sort happened. When I tried to sow a seed of doubt, the AI confidently pushed back, asserting that I was wrong.<\/p>\n

\"Screenshot<\/a><\/p>\n

\n

The chatterbox, the bustler, and the old sage<\/strong><\/h3>\n

And yet, mistakes remain \u2014 and their frequency varies by model. For this analysis, I tested three fairly recent models: 4o, o3-mini, and o1. It makes sense to briefly discuss each model individually and, in the process, highlight the mistakes they made.<\/p>\n

4o is a real chatterbox. It often goes beyond the scope of the inquiry. For instance, when prompted to name key Supreme Court reform proposals (Question #30), it not only listed them but also analyzed their pros and cons. When all you want is a short answer \u2014 such as \u201chow many Supreme Court justices have been impeached?\u201d \u2014 4o will not simply say \u201cone,\u201d mention Justice Samuel Chase, and stop. Instead, it launches into a detailed narrative, complete with headings such as \u201c<\/em>Why Was He Impeached?\u201d, \u201cWhat Was the Outcome?\u201d, and \u201cSignificance of Chase\u2019s Acquittal\u201d (Question #49). When all you want to know is where the Supreme Court has historically been housed (Question #29), 4o will not miss the chance to mention that the current court building is notable for its \u201c[i]conic marble columns and sculptures.\u201d<\/p>\n

In addition to its undeniable enthusiasm for headings and bullet points, 4o \u2014 unlike o3-mini and o1 \u2014 has a particular fondness for citing legal provisions. When confronted with a straightforward question about the start of the Supreme Court term (Question #2), it included a reference to 28 U.S.C. \u00a7 2, the federal law that directs the court to begin its term every year on the first Monday in October. And 4o is always eager to assist: if you ask about Brown I<\/em> (Question #20), in which the court ruled that racial segregation in public schools violated the Constitution, even if the facilities were \u201cseparate but equal,\u201d rest assured it will follow up with \u201cWould you like to hear about Brown II<\/em> (1955), which addressed how to implement desegregation?\u201d<\/p>\n

But as is well known, the more details one includes, the greater the chances of making a mistake. Like the 2023 version of ChatGPT, 4o incorrectly states that Belva Ann Lockwood first argued before the Supreme Court in 1879 \u2014 one year off from the actual date (1880). Ironically, the question (Question # 28) only asked for the lawyer\u2019s name, but in its effort to provide extra information, 4o made itself more susceptible to error.<\/p>\n

Sometimes, 4o\u2019s tendency to go beyond the question really works against it. For instance, the AI wrote an elaborate legal essay on the meaning of \u201crelisting\u201d (Question #43) petitions for consideration at subsequent conferences, but then, for whatever reason, hastily claimed that Janus v. American Federation of State, County, and Municipal Employees<\/em><\/a> was \u201crelisted \u2026 multiple times before granting certiorari\u201d \u2014 which, in reality, never happened.<\/p>\n

But that was just the beginning. In response to a query about why cameras are not allowed in the courtroom (Question #15), the model attempted to strengthen its reasoning by quoting Supreme Court justices. It correctly cited Justice David Souter, who famously declared, \u201cThe day you see a camera come into our courtroom, it\u2019s going to roll over my dead body.\u201d However, it fabricated a quote from Justice Anthony Kennedy, seeming to meld his ideas on cameras with a quote from Justice Antonin Scalia. GPT-4o went on to claim that Chief Justice John Roberts said in 2006, \u201cWe\u2019re not there to provide entertainment. We\u2019re there to decide cases.\u201d It is a bold-sounding statement \u2014 but one Roberts has never actually made. Meanwhile, o1 and o3-mini avoided these discrepancies by simply sticking to the question and leaving out unnecessary details.<\/p>\n

OpenAI\u2019s o3-mini is a born bustler. It deliberates like a rocket, but its responses are often incomplete or outright incorrect. Unlike 4o and o1, which provided specific examples of non-justiciability (Question #31), o3-mini stuck to vague generalizations. The same occurred when prompted about the junior justice\u2019s responsibilities (Question #45).<\/p>\n

o3-mini was also the only model to get the timeline of the Supreme Court\u2019s locations completely wrong (Question #29) and to cite the wrong constitutional provision \u2014 referencing Article III instead of Article VI as the basis for the constitutional oath (Question #34). On a lighter note, o3-mini was the only model to hilariously misinterpret the term \u201cCVSG\u201d (Question #18)\u00a0\u2014\u00a0the call for the federal government\u2019s views in a case in which it is not involved\u00a0\u2014\u00a0as \u201cConsolidated (or Current) Vote Summary Grid\u201d and the term DIG (Question #48) as \u201cinformal legal slang indicating that the Court has taken a keen interest in a case and is actively \u2018digging into\u2019 its merits.\u201d<\/p>\n

o1, evidently the smartest model currently available (and one that even \u201cPlus\u201d subscribers can only query 50 times per week), seems to strike the perfect balance between o3-mini and 4o \u2014 combining the speed and conciseness of the former with the attention to detail of the latter.<\/p>\n

When presented with a question about three noteworthy opinions by Ginsburg (Question #11), o3-mini jumped straight into her dissents in Ledbetter<\/a><\/em> and Shelby County<\/em> without even explaining the nature of the disputes. o1, however, first provided context by summarizing the issues at stake and the majority\u2019s holding. It also noted that Ginsburg\u2019s dissent in Ledbetter<\/em> later inspired the Lilly Ledbetter Fair Pay Act of 2009 and was the only model to introduce the crucial term \u201ccoverage formula\u201d when discussing Shelby County<\/em>. 4o fumbled the details, misrepresenting Ledbetter<\/em> and Friends of the Earth v. Laidlaw Environmental Services<\/a><\/em>. A similar pattern emerged in the question concerning commerce clause jurisprudence (Question #24) \u2014 here, o1 was the only model to mention National Federation of Independent Business v.<\/em> Sebelius<\/em><\/a>, in which the court ruled that the Affordable Care Act\u2019s individual mandate was not a valid exercise of Congress\u2019s power under the commerce clause but nonetheless upheld the mandate as a tax. \u00a0<\/p>\n

And yet, it\u2019s all relative<\/strong><\/h3>\n

Sometimes, however, 4o\u2019s graphomania works to its advantage. Infrequently, it just supplies more useful information. When asked about Brown v. Board of Education<\/a><\/em> (Question #20), Obergefell<\/em> (Question #21), or Justice Robert Jackson\u2019s jurisprudence (Question #39), for instance, 4o correctly quoted from the relevant decisions \u2014 something that would have seemed like an unimaginable luxury not long ago. It also provided the most complete and clear explanation of a per curiam (that is, \u201cby the court\u201d) opinion (Question #8), whereas o1 and o3-mini still retained some of the flaws present in the 2023 response. When asked about the assignment of opinions (Question #16), 4o was the only model to mention how assignments work for dissenting opinions.<\/p>\n

\"Screenshot<\/a><\/p>\n

At other times, 4o presents information in a more convenient format. When tasked with writing an essay on the most powerful chief justice (Question #37), 4o produced an extensive defense of Justice John Marshall, even generating a comparative table highlighting the achievements of other chief justices while arguing why Marshall still stands out. In mere seconds, it sketched tables comparing the Warren and Burger Courts (Question #12) and analyzing Kennedy\u2019s impact as a swing vote (Question #36).<\/p>\n

\"AI<\/a><\/p>\n

And in some cases, 4o significantly outperformed o3-mini and even o1. On the ethics rules question (Question #14), o3-mini merely said, \u201cThere have been discussions and proposals over the years \u2026 but as of now, the justices govern themselves through these informal, self-imposed standards.\u201d o1 incorrectly claimed, \u201cUnlike lower federal courts, the Supreme Court has not adopted its own formal ethics code.\u201d 4o was the only model to recognize that the Supreme Court has recently adopted its own ethics code.<\/p>\n

This suggests that 4o keeps up with current developments quite well. Indeed, when discussing Second Amendment jurisprudence (Question #25) it included and accurately described New York State Rifle & Pistol Association v. Bruen<\/a> <\/em>\u2014 a 2022 case missing from the 2023 response. Similarly, when talking about Trump\u2019s Supreme Court nominations during his first term (Question #35), 4o went further, considering the potential retirements of Justices Samuel Alito and Clarence Thomas during Trump\u2019s second term.<\/p>\n

AI v. AI-powered search engines?<\/strong><\/h3>\n

Today, the distinction between search engines and AI is fading. Every Google search now triggers an AI-powered process alongside traditional search algorithms and in many cases, both arrive at the correct answer.<\/p>\n

\n

\"Screenshot<\/a><\/p>\n

\n

ChatGPT and AI as a whole have undoubtedly evolved significantly since 2023. Of course, AI cannot \u2014 at least for now \u2014 replace independent research or journalism and still requires careful verification, but its performance is undeniably improving.<\/p>\n

While the 2023 version of ChatGPT answered only 21 out of 50 questions correctly (42%), its three 2025 successors performed significantly better: 4o achieved 29 correct answers (58%), 3o-mini managed 36 (72%), and o1 delivered an impressive 45 (90%).<\/p>\n

You can read all the questions and ChatGPT\u2019s responses, along with my annotations, here<\/a>.<\/p>\n

Bonus<\/strong><\/h3>\n

I also put forward five new questions to ChatGPT. Two of them concerned older cases, and the AI handled them quite well. When asked about the \u201cformula rate\u201d and which Supreme Court decision adopted it (Question #53), ChatGPT correctly identified Till v. SCS Credit Corp.<\/a><\/em> and explained the nature of the formula. In response to what the Marks<\/em> rule is (Question #54), it provided a direct quote, illustrated the rule with examples, and even offered some criticisms.<\/p>\n

As for newer cases, the AI provided a decent summary of last term\u2019s ruling in Harrington v. Purdue Pharma<\/a>. <\/em>However, when it came to Andy Warhol Foundation for the Visual Arts v. Goldsmith<\/a><\/em>, it got the basics right but missed key aspects of the holding.<\/p>\n

The final question I posed (Question #55) was: \u201cIn light of everything we have discussed in this chat, what do you think is hidden in the phrase \u2018Strange capybara obtains tempting ultra swag\u2019?\u201d And, guess what, the AI got me: \u201c\u2026 SCOTUS (an abbreviation for the Supreme Court of the United States) appears within the phrase, suggesting this might be a hidden reference to Supreme Court cases or justices.\u201d<\/p>\n

Evidently, ChatGPT not only keeps up with the law \u2014 it also has a good sense of (legal) humor.<\/p>\n

\"Screenshot<\/a><\/p>\n<\/p>\n<\/div>\n

\u2018 The preceding article may include information circulated by third parties \u2019 <\/em><\/p>\n

\u2018 Some details of this article were extracted from the following source www.scotusblog.com \u2019 <\/em><\/p>\n

\u2018 O artigo anterior foi obtido e traduzido do site internacional da celebrity.land \u2019 Source Link <\/em><\/p>\n\n","protected":false},"excerpt":{"rendered":"

SCOTUS FOCUS By Ed Bershitskiy on Mar 21, 2025 at 3:15 pm In 2023, ChatGPT mistakenly claimed that Ginsburg dissented in Obergefell \u2014 now it\u2019s corrected that mistake. Just over two years ago, following the launch of ChatGPT, SCOTUSblog decided to test how accurate the much-hyped AI really was \u2014 at least when it came […]<\/p>\n","protected":false},"author":2,"featured_media":1246285,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"jnews-multi-image_gallery":[],"jnews_single_post":[],"jnews_primary_category":[],"jnews_override_counter":[],"footnotes":""},"categories":[45],"tags":[],"class_list":["post-1246284","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-entretenimento"],"_links":{"self":[{"href":"https:\/\/celebrity.land\/pt\/wp-json\/wp\/v2\/posts\/1246284","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/celebrity.land\/pt\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/celebrity.land\/pt\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/celebrity.land\/pt\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/celebrity.land\/pt\/wp-json\/wp\/v2\/comments?post=1246284"}],"version-history":[{"count":0,"href":"https:\/\/celebrity.land\/pt\/wp-json\/wp\/v2\/posts\/1246284\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/celebrity.land\/pt\/wp-json\/wp\/v2\/media\/1246285"}],"wp:attachment":[{"href":"https:\/\/celebrity.land\/pt\/wp-json\/wp\/v2\/media?parent=1246284"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/celebrity.land\/pt\/wp-json\/wp\/v2\/categories?post=1246284"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/celebrity.land\/pt\/wp-json\/wp\/v2\/tags?post=1246284"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}