ChatGPT Groupchats, System Prompts, and Daily LLM Use Cases | Sharp Tech with Ben Thompson

And when it comes to group chats or whatever, 1 week in, is Ben actually using the group chat feature on ChatGPT?

This is a mistake that I think is made every time a new product launches: it’s only evaluated under the assumption that it completely replaces what came before. “WhatsApp out, ChatGPT in”—that’s rarely how it works in technology. In the vast majority of cases, something new is layered on top of what came before.

The phone did not replace the PC. It did over time for some people, but it was something layered on top of the PC. The general trend is that we just use technology more and more. We don’t have a tech bucket here where we swap stuff out, and then over here is everything else. So I think there’s a fundamentally wrong assumption underlying this email.

1. AI Joins Collaborative Work

What I want group chats for is when I’m working with someone on something, and it’s just so much more convenient. The whole point is to have the AI there. So we’re going to do something with your podcast setup. We did a little back-and-forth about the specific sort of modern TV that we want to do for you, and it was just so much easier. Instead of me copying and pasting some things from ChatGPT into a chat with you, we could just have the chat right there.

Now, we uncovered some issues, like the fact that you need to be notified if you get mentioned. There’s no easy way to find when you’re being talked to.

I had to keep returning to the ChatGPT app to see whether there were new messages in the group chat. That should be very easy for ChatGPT.

WhatsApp does an amazing job of helping you manage where you are in the chat. If you use the up arrow and down arrow, or tap the quote in a quoted message, it jumps to it. It’s actually very subtle, but it works really, really well.

ChatGPT is missing all that sort of stuff. You need to be able to find your mentions in the middle of a super-long ChatGPT answer, which, by the way, should probably be shorter in a group chat, or it needs to be more collapsed. You can expand it.

I was going to say, if you could expand it in the group chat, it would be a lot easier to use the group chat feature. My issue here is really that I think this is clearly useful to anybody who’s doing knowledge work. Back when I was a lawyer, working with a couple of other associates on a brief, being able to send research back and forth as we were all drafting different pieces of a brief would be tremendously useful.

My question is why other apps can’t easily replicate this feature, and whether this will be a durable moat for ChatGPT, because you don’t actually need history—you wouldn’t need the brief URL.

It doesn’t bring in your memories and stuff like that, which is part of the privacy point.

Yeah, I mean, I don’t think this is going to be a moat that protects ChatGPT forever—the deepest moat of all time. I do think it just depends on how people work. It’s going to develop how people work.

From my perspective, this kind of goes back to a critique I had of Meta. It was Facebook back then, when I started Stratechery back in 2013. That was when Facebook did actually launch a phone. It was a heavily skinned version of Android, and you could also get just that version of Android on other Android phones.

Mark Zuckerberg was going on and on about how, actually, why are we operating in a world organized by apps? People are what matter. That’s how your phone should be organized. And I wrote in an article, using the jobs-to-be-done framework, listing all the things I did on my phone. The largest and most important was communication. If you go to the little battery menu on my phone, WhatsApp is 78%. So it’s definitely important.

But there’s a bunch of stuff I do that is not people-centric. I’m actually trying to accomplish something or do something. In that framework, the app paradigm didn’t win just because Apple was first to the iPhone. It won because it fit the way people work, or maybe it shaped the way people work. Whatever it might be, that was the paradigm, and Facebook was totally mistaken. Mark Zuckerberg was totally wrong about that sort of organization.

2. AI Work Stays Deliberate

Fast-forward to today. The way I think about this is that when I’m interacting with AI, I’m interacting with AI. It’s a very purposeful, explicit thing that I want to do. So it makes sense to me that if I want to work with AI with you, we do it in the AI app, which is OpenAI, as opposed to doing my word processing and thinking, “Oh, there’s my AI right there.”

Now, I might be mistaken in the long run, but for me, right now, personally, that’s how I think about AI. It’s not necessarily always going to be that way. Maybe it’s just because it’s not good enough, or it’s not present everywhere. Certainly, OpenAI has talked about this. They want to have ChatGPT incorporated into everything. That’s one of the justifications they put forward for their whole enterprise and sticking with the API, and they want to have “Login with ChatGPT.” So ChatGPT is in all your apps.

I’m not saying this is the way it’s always going to be. There are people who disagree with me, including Sam Altman. But for me, anyway, I view AI work distinctly. One of the hardest reasons I’ve resisted getting rid of ChatGPT is that there are connectors. ChatGPT is connected to the terminal, and I work in the terminal. I ask ChatGPT something, it looks at the terminal, sees what’s going on, and tells me or gives me a new command, or whatever it might be.

To me, that makes sense, as opposed to having ChatGPT in the terminal, which is already inscrutable, and making it even harder to pay attention to what’s going on. Again, this might be totally wrong, but for me, this makes total sense.

Yeah. Well, I can definitely imagine it making a lot of sense for knowledge workers of the future. Any sort of group project, you would want this feature to exist.

Continuing on here, it says: “Ben and Andrew, you guys seem to talk about AI nonstop these days, pretty much taking over the show. How are you guys actually using AI on a day-to-day basis? Which models are your go-to? Are you using them on the desktop/laptop or on your phone as well? Do you use them mostly for work or personal stuff, too? I want some gory details. Ben, do you have any—”

I teased this question earlier.

Exactly. Professional podcaster there. Old, but still a professional podcaster.

No, I want you to go first. Give me the overview.

Well, I use it exclusively on my phone. I use ChatGPT exclusively. I don’t find the desktop app all that useful, and I use it more for random questions than I do for work.

On that point, one of the big reasons I’m reluctant to lean on it for anything I’m doing professionally is because of the handful of times it has hallucinated and burned me. That’s probably not rational, but with basketball, I have a depth of knowledge, so I don’t really need ChatGPT. I could spot the errors, but I don’t really find myself leaning on ChatGPT for anything related to the NBA.

With China and tech, I don’t have that depth of knowledge, so I want to be more careful about the answers I’m given by these apps. I’d rather not risk getting something horribly wrong. I will use it for understanding various CCP bureaucracies as far as Sharp China is concerned, but for the most part, it’s household stuff and cooking stuff— that sort of thing.

Yeah, I raised the question a little bit ago: Is ChatGPT going to end up being Twitter? That was in the context of it being very text-centric. As we move to multimedia forms being more and more important, it would still be intensely used and appreciated by some number of people, but would not truly reach the mass market, because most people just want images and video. We saw that with social networking. Is that going to be the path?

3. Gemini Opens AI To Everyone

This is where Gemini blows everyone away: in the multimodal stuff. The image generation, obviously, and Veo can do video. That’ll be incorporated at some point. This bit where it will generate dynamic UIs is, again, something I’ve been saying is going to happen. So, of course, that’s my favorite feature.

But the implication of that is the real risk. It’s not that Google replaces ChatGPT for people who use ChatGPT heavily. It’s that Google opens up AI to vastly more people by being better at formats that are more important to more people.

Yeah, so people could have the more compelling GUI sort of interface.

That’s right. It’s just more compelling and interesting—generating videos on demand or UIs on demand. It’s easier to use and more approachable for more people.

The other potential Twitter analogy here is that I remember, in business school, being convinced that I don’t know how you could be interested in tech and want to go into tech and not be on Twitter. I did presentations, as part of the tech club or whatever, on how to use Twitter.

What was impossible then, and is arguably impossible now, is how to actually get a useful feed. How do you figure out who to follow? Everyone would just go and follow the big names, or TechCrunch was the big thing back then, and then you’d get a list of headlines.

But what’s actually valuable is following all these individuals in the space. I talked about the Gemini vibes on Twitter. How do you get the Gemini vibes? You don’t get them by following the official Google account. You get them by following all these randos that you’ve accumulated over time, who are having these super-in-depth discussions about these AI models.

And it's tremendously illuminating. It's information you don't get anywhere else, and it's almost always right, very early, and ahead of everything else. Even today, this is why I was always a huge proponent of Twitter having an algorithmic feed. It's too hard. But even now, it's better than it used to be. The For You page, I think, is much more useful, but it's hard to get really deep.

If you're a normal person opening Twitter, it's just a mess of takes, and it's really hard to navigate the platform. The insinuation of that, which is totally correct, is that if you're already deeply into Twitter, you're not a normal person.

And let me be clear: if you're a normal person confused by Twitter, you're a healthier person. [laughter] So congratulations. We're happy for you.

This kind of applies to chatbots. When you talk about why you don't use ChatGPT, if I can pick on you a little bit—actually, no, we had this conversation a couple of years ago when you got burned by ChatGPT. I'm like, why are you using ChatGPT to ask for statistics? That's a terrible idea. It's going to hallucinate. It's going to miss stuff.

4. LLMs Need The Right Questions

But that comes from me having a mental model of how LLMs work and knowing that asking them for very specific factual information of that type is a mistake. That's a deterministic question. You need to go to a deterministic source where someone actually put numbers into a database, and it's produced in a table. And if it's on ESPN, it's in the wrong order because they screwed with the box score, right? Basketball Reference for the win.

So that is a real barrier. People don't use it because they got burned, or if they use it, they're using it just as a pure Google replacement. I still use Google all the time when I need definitive sources of data. In particular, I want to verify something. I don't trust ChatGPT for verification at all now because of that.

I use it all the time. I use it on my Mac—for me, the Mac app is amazing. Amazing. That's critical to me. I use it particularly if I was doing a lot of stuff about spectrum.

Yeah.

With SpaceX and satellites.

Just understanding that space. It was super useful, super amazing. I thought I was able to do a good description of what was going on with the uplink and downlink sort of thing, thanks to ChatGPT. I also verified all the numbers for the spectrum discussion by going to the original sources, so it gave me the context. Now, that was one I didn't know much about. There's another example.

It's giving you the shape of the industry, and then you can go through and fact-check the specifics as you're writing. Is that right?

That's right. Now, there's another example where it was some sort of antitrust case. I can't remember what it was, but I was comparing my ChatGPT interaction about this case with someone else who didn't really know much about it. Their answer was not very useful. It was very generic, whereas I had this incredibly in-depth discussion citing all these past cases and the implications and precedent because I knew so much about it.

And so this is actually where you mentioned, “Oh, I don't need it for basketball. I know a lot.” Actually, my experience is the more you know about something, the more useful it is, precisely because you end up guiding it to a level of depth and breadth that it's not going to get to on its own.

There are other articles where, of course, I do go and verify. I actually look up the actual cases, and I make sure I read the abstract and understand what it's about, along those lines. But, yeah, I use it all the time.

5. AI Handles Everyday Questions

So today, on the way to school, my son asked me—it was very foggy—“I generally know how fog works, but why was it particularly foggy today?” So—

That's such a perfect application of ChatGPT.

Yeah, we're talking about, “Oh, here's the weather today. Here's XYZ,” and just talking to it about the fog formation and why it was heavier today than other days. I have an ongoing thread about my turkey smoking.

I've done the turkey preparation. I've done it in the oven. I've done it on a grill. I'm doing it on a smoker this year. That is going to be new. But figuring out the timeline for that.

Putting in the paces. Love to see it.

Here's one. NVIDIA was talking about offtake, and I didn't actually know what that meant. I put the quote from the earnings call in and asked, “What does offtake mean?” It's basically: Is this stuff being used up? If you build out GPUs, are they being utilized all the time?

That was a very basic sort of thing I probably should have known, but I didn't. Troubleshooting Christmas tree lighting issues—oh, yeah, a Christmas tree that wasn't lighting up. I was trying to figure out what was going on. TV size recommendation for the podcast. Oh, this is you. This is a group chat.

I now have a TV with a camera in front of it. I'm looking straight at the camera while interacting with you, and you're looking down. I'm like, “I think I look great,” and you look, you know, above and beyond these issues.

Great. I look great. Ben is just very neurotic about the presentation on these videos. And look, I'm going to look even better once we get this new setup installed in my basement. Can't wait for it.

But, yes, you're covering a lot of ground in ChatGPT. That's the operative takeaway here. The other thing that's worth noting is that this may be a personality thing. I'm the sort of person—I've always been the world's most insane Googler. It drives me crazy if there's something I don't know. I have to know what it is.

As a kid, I would have a list of stuff and go to the library and look up all sorts of things, like how stuff worked. I would read the encyclopedia. What a loser. It's unbelievable. [laughter]

No, we've been over this. I read the dictionary as a kid. A couple of nerds here. I love knowing how stuff works all the time. And it's funny because the payoffs are astronomical, right? If you've been looking up how stuff works for 40 years, [laughter] you know how a lot of stuff works, and then you start building connections—how stuff is similar, how stuff is different.

I'm naturally a systemic thinker anyway, but it's accentuated by just building a lot of knowledge. So for this, I just use it for everything I don't know. Actually, the fog thing: I generally knew how fog worked, but I wanted specifics about the weather conditions and so on. So, yeah, I went straight to it.

Again, just a fantastic application. If my son asked me how fog worked, I would have a guess, but I think boning up with ChatGPT would be a better way to answer that question.

6. Inside The System Prompt

Eugene says, “Ben, what is your ChatGPT system prompt?” Bonus prompt, Andrew: record your phone screen. You have 60 seconds to find the system prompt screen on the ChatGPT app. If you fail, you need to read your last 5 ChatGPT prompts on air. So, I—

We're not doing dares on the podcast. [laughter]

Yeah, I did not agree to those terms. Eugene, that is not a binding contract, but I took that bet and lost that bet at about 1:00 a.m., lying in bed.

The settings are a little rough. There's getting to be too many of them. I think this was prompted by the dithering discussion, where John really liked the robot and found it better than I did. The robot was okay; it was not as good as my custom prompt, which I put back in.

I don't want to read through the whole thing. “Don't worry about formalities. Try to answer questions from your own knowledge before relying on search.” This was a problem when they added search: it would just search all the time. I'm like, no, you know this. It's in your depth of knowledge. Just go straight there.

“Please be as terse as possible while still conveying substantially all information relevant. Take however smart you're acting right now and write in the same style, but as if you were 2 standard deviations smarter. Never mention you're an AI. Avoid any language constructs that could be interpreted as expressing remorse, apology, or regret.”

The prompt—yeah, so I have a lot of layers there. Yeah. You want more? [laughter] I keep going.

Maybe I need to work on my prompt. My last 5 ChatGPT prompts are: “Proper temperature for medium-rare ribeye.”

“Disciplining a 2-and-a-half-year-old who keeps pushing his sister.”

You have GPT right here. No, I'm just kidding. [laughter]

Yeah, it's true. I could lean on you for all the answers. [laughter] “Reformatting numbering in a Microsoft Word document.”

And then 2 that I'd forgotten about until losing Eugene's bet at 1:00 a.m. last night: “How do I tell my wife that I'm unhappy in my marriage and want to go to couples therapy?” That was a sarcastic question that I asked ChatGPT and then read the answers aloud to her, and then clarified for your memory.

And then did you end up needing the prompt? [laughter] She was very pissed off and then instructed me to clarify, for ChatGPT's memory purposes, that my comment was 100% joking because she was very concerned that the chat was going to think we had a bad marriage.

So that's the sort of thing I use ChatGPT for. Wonderful app, and I should incorporate it more into my professional workflow here.

Well, it's one of those things. Actually, here's another Twitter analogy. I joined Twitter in December 2006. I think it was the first year, but towards the end I had a couple of messages, and then I didn't use it for about a year.

And then suddenly it clicked, and I've been just absurdly addicted ever since, to the point I should probably go to rehab.

7. Keep AI As Your Assistant

I think there’s something about AI that’s similar. You use it, and it’s very—no, it’s like a novelty, and you think, “Oh, I asked it this random question,” and then at some point it clicks and you’re like, “I should be using this all the time for sort of everything.”

I do think there’s a world where people overdepend on this and just outsource their thinking completely. But particularly if you’re very agentic in your actions and in your thinking, if you make your assistant instead of your boss, it’s tremendously useful.