（预告）理解 AWS 宕机、韧性现实与创新紧迫性，F1 转向 Apple TV

Hello, and welcome to a free preview of Sharp Tech. Hello, and welcome back to another episode of Sharp Tech. I’m Andrew Sharp, and on the other line, Ben Thompson. Ben, how are you doing?

Better than you, Andrew. As we all know, the NBA championship was decided last night. I am wearing my Bucks polo shirt in celebration of obliterating your Washington Wizards. The season’s over. Give us the ring right now.

The Sharp Tech Bowl.

That’s right.

Not a lot of drama in the Sharp Tech Bowl. I was talking to Bill Bishop earlier in the week, and I mentioned that the Wizards were opening with the Milwaukee Bucks. He said, “Are you guys going to have some sort of bet on the game?” And I was like, “Not in a million years will I be betting on the Washington Wizards.”

I’m happy for you. It’s great to see you in the Bucks polo, beaming with joy after Giannis debuts here, turning over a new page after the Brewers’ journey a couple of weeks ago.

The Brewers? What are the Brewers? Not familiar—

It’s all about the Bucks these days.

I’m not familiar with that team.

That’s exactly right. Here we go. Yes, basketball is back. It’s been a very entertaining 48 hours, and now we have a gambling scandal looming over everything.

Now the NBA is truly back.

Indeed.

They tried to run the Clippers as the scandal of the year, forgetting that no one cares about the Clippers.

Child’s play.

Yeah, so they needed to go with something new. So here we are.

Indeed.

Wait, actually, no. I have some comments on the NBA.

Great.

1. NBA on NBC Returns

Number 1, like everyone else, I’m thrilled with NBA on NBC. Imagine trying to make a broadcast look good and make people excited about watching a product. What a concept.

They created a video before the game, an iconic sort of entrance. They didn’t run commercials up until the tip; they showed player introductions. The lowest-hanging fruit in the world, NBC delivered. Thumbs up to them. I’m with everyone raving about it.

Amen. I completely agree.

However—

What’s your next take?

2. Inside NBA Saves ESPN

That was only improvement number 1. In the ranking, it’s actually number 2. Number 1 is—I don’t know how this happened. I mean, I knew it would happen, but I didn’t fully appreciate that the biggest victory in my NBA viewing experience is Inside the NBA basically completely taking over the ESPN presentation. ESPN still does the broadcast themselves.

And salvaging the enterprise by themselves, indeed.

It’s unbelievable how bad the ESPN shows were. It really bugged me when a game would end and go straight to SportsCenter, a property that was massively important and valuable in the 1990s.

Yeah.

What are we doing here? Last night, the games finished. I was working on my update. I had Inside the NBA. They were on fire last night—absolutely hilarious segments—and it was so great.

You know what? It’s an experience. You watch the games, then you go to bed with Kenny, Charles, and Shaq. It was so good. I can’t believe somehow they’re on ESPN.

By the way, there’s something for ESPN to learn from it. The extent to which ESPN is hated is kind of incredible compared to the 1990s, when they were so cool.

Oh, yeah.

There’s a bit here where I’m sad we lost TNT, but TNT was kind of like the rebel upstart. That was the whole thing with the studio show. It didn’t feel highly produced, even though it was the best show on television. It was sort of slapstick. You got the late-night cable feel from Inside the NBA.

It was a couple of guys hanging out for an hour after 4 hours of basketball games. It was never more complicated than that.

Right. What we got from ESPN was like a luxury-brand knockoff, where they’re trying to present themselves as a big, important broadcaster, but they’re cheaping out on everything. Their camera quality is the worst. Their studio shows were always terrible.

The reason they’re pushing SportsCenter is because it’s cheap content that they used to be able to run really expensive commercials against, because everyone watched it. This was a return—actually, even though it felt so weird to have Inside the NBA on ESPN, it felt like a possible opening for ESPN to remember its roots and go back to being the sports channel for people to hang out on, with crazy stuff happening at 11:00 p.m. at night.

Hmm.

And huge thumbs up to me. I’m giving the NBA a lot of grief. It still deserves a lot of grief in many respects, but wow.

Yeah.

I didn’t appreciate how much it really was ESPN. If ESPN is your prime broadcaster, TNT was still great. ESPN as the backup broadcaster, with the slapdash show and whatnot, and then NBA on NBC—

That can work.

Right.

Yeah.

It’s a mullet strategy with a much more attractive mullet. You have business in the front with NBC. You have ESPN in the back, where they belong. Two thumbs up from me.

I am gonna be writing every week on Sharp Text, a new website called Sharp Text. You can reach the site directly at sharptext.net. Last week I wrote about the trade war. This week I wrote about the NBA's new broadcasting landscape for a post that will publish on Friday, and I talked about why de-emphasizing ESPN is so healthy in the grand scheme of things. Having Inside the NBA there to salvage things on the ESPN side makes the new environment that much better. And so I’m very, very excited as a basketball fan.

I didn’t expect to be this excited about the new broadcasting structure, and I was a little bit worried about Inside the NBA on ESPN, because for the last 20 years, ESPN ruins all its programming by overloading these shows with commercials, so people have 60 or 90 seconds to talk.

Well, there’s also such a heavy hand from producers about what to talk about and things like that. We feel this particularly acutely as Bucks fans, when Malika Andrews was pushed by an ESPN producer to ask Giannis, minutes after they lost Game 6 to Toronto in 2019, about his plans to leave for another team.

They’ve pushed this endlessly for the last 6 years. In general, all they focus on is the transaction game, what’s going on there, which is intellectually interesting and such a detriment to actually enjoying the game, which is supposed to be the point.

I have such disdain for ESPN. Basically, the reason Inside the NBA is so great is because it’s so clear that ESPN is not involved at all. They slapped their logos on there, and everything else.

There was even one of the segments last night asking Charles Barkley which TV personality was with which network. One of them was Ellie LeForge, who was the sideline reporter for TNT last year.

Yep.

Charles says, “Oh, she’s with us,” and ESPN. No, actually, she’s still with TNT. But Charles is like, “You tricked us,” because they’re still at the TNT studios, right? It’s just—

Ah, yeah, so he’s seeing her every day.

That’s right. It’s like—

And he’s down there like, “Wait…”

So, yeah. A lesson there.

A plus.

Just like that.

A plus to ESPN—

That’s right.

—for not ruining a good thing, the greatest studio show of all time. They said all along, “We’re going to leave it untouched, hands off, let them do their show.”

But it’s hard to believe them, yeah.

Yeah. Corporate partners say stuff like that all the time. It’s rarely followed through.

By the way, I’m going to defend Inside the NBA before we get emails. There’s a certain contingent of NBA Twitter that complains about Inside the NBA: “They don’t know what they’re talking about. All they want to do is throw in the post. They diss current players,” and so on.

Hmm.

That totally misses the point. This is the same critique that is applied to Bill Simmons. These are entertainment products.

This applies to the NBA too, with the whole replay fiasco, which is still a disaster. It’s an entertainment product. Being perfect and being able to cite an Excel spreadsheet is not entertaining. It’s funny when they’re wrong.

They sometimes get some stuff really, really right that the nerds miss.

Really right. That’s the important point with both Bill Simmons and Charles Barkley. They often have insights that elude some of the nerdier Excel-sheet crowd.

So, A, it’s more entertaining, and B, I would say it’s equally insightful in terms of what you’re getting.

Right about the big things. Your big thing.

There’s a lot of right about the big things coming from the Chuckster. And also, the only sports show that my wife looks forward to is Inside the NBA. So there’s another endorsement for you. But I’m glad that we’re celebrating basketball’s return with a 10-minute intro on the NBA.

Oh, we’re gonna be back to ESPN in a little bit. Just hold on.

Indeed, we will. For now, we will start with a killer article that you wrote this week. I really enjoyed the article on Wednesday.

I just want to jump in. I appreciate the enthusiasm with which you said this, because I saw the rundown: “You write a killer article.” And the problem is, now that we work together, I’m not sure I can trust anything you say.

Okay.

You really turned it up there. You persuaded me, so thank you.

I convinced you it was genuine. Yes. I sat there on Wednesday and was like, “Man, that was really great.” You unified 3 big American stories over the past couple of years, 2 of which dominated discussion over the last 2 weeks. We talked about rare earths on last week’s episode.

So let’s start with the AWS outage, because on Monday, I went to my Starbucks app and found that it was broken. Then I turned my phone on and off 3 times because I thought it must be an iPhone problem. I showed up to Starbucks, and they said, “No, the app’s not working for anyone.”

I finally got my coffee, went to read The Wall Street Journal to prep for a Sharp China episode, and The Wall Street Journal site was down too. That’s how I found out there was an outage that the whole country was dealing with on Monday. So what the hell happened with us-east-1? I had never heard of us-east-1. What’s the proper way to characterize what us-east-1 is? If I’m explaining it to my wife—

Oh, boy—

—is it a data center, a data center cluster? What do we go with?

Yeah, it’s a data center, even though at this point it’s a massive complex. With these huge cloud servers, you say, “I’m gonna serve my business on AWS.” When you do that, you actually pick what’s called a region.

Mm-hmm.

That region is where you put the core of your application. Best practices are that you’re in multiple regions, but there has to be a source of truth somewhere, and ideally you’re doing followers and writes and syncs and all that. This is a massively complex problem. It’s really, really hard.

Okay.

Think about your Starbucks app. You can be there in line and say, “Oh, crap, my Starbucks balance is low.” You press a button, money gets added, you go to the cashier, and you flash the thing. Within a matter of seconds, there is money changing hands, implicating a barista right there taking your order and making it. The level of speed, low latency, and complexity of this is amazing. It’s actually insane that any of this stuff works as well as it does.

Mm-hmm.

But even within that, that’s sort of an internet story broadly. The enemy of everything in computing is latency, and you have this challenge: in some respects, you’re governed by the speed of light.

Yeah.

There are aspects of me being in America where my internet experience is different just because there literally aren’t signals going across the ocean. Stuff like that can certainly matter. But there are also things like the complexity of being on AWS while you have other data on Azure.

Mm-hmm.

You have to cross over the public internet from AWS’s internal network to Microsoft’s. You can run into challenges there. You see this with Netflix. Netflix puts servers in your local ISPs that actually serve the content.

Right.

The thing about the whole Netflix story is that it’s never actually going over the public internet. What’s over the public internet is your authorization and the actual catalog that’s customized to you. But all that is much—

But that’s why Netflix is better at serving content than most of its competitors: because they’ve built out that infrastructure.

There’s a real infrastructure, yeah, for sure. Now, all the other competitors’ content is still on CDNs, or content delivery networks. Say, for example, Cloudflare. If you go to your local ISP office, there’s a Netflix rack there. There’s a Netflix box that has all the Netflix shows right there.

They push out new shows when they come out. They’ve spent the last week seeding them all over the world so that when you hit play, it’s just playing down the road. There’s another box in that server, in that rack. That’s a Cloudflare server that holds all your content-delivery stuff. There’s an Amazon one there.

For example, Passport, like this podcast, is going to be hosted on a CDN so that when you hit download, it downloads super fast. It’s close to you, and it’s not getting clogged up by the internet. One of the huge challenges of Passport is that CDNs are much more straightforward and obvious. If you have one piece of content you want to serve to anyone, we want to have—

Yeah.

—an authentication layer on top of it, so you can only download it if you’re a SysTrackery subscriber. There’s a whole level of complexity: How do you actually manage verifying someone? Do they have the right token, X, Y, Z? Then you have to deliver the content that’s close to them, keep this all in sync, and manage it. It’s devilishly difficult.

Mm-hmm.

So anyhow, within the overall cloud, you have AWS as a whole, and then you have regions. If you’re within a region, having all your stuff in the region is going to operate better and be easier because it’s closer together physically. It’s also closer together from a network-topology standpoint. You’re jumping through fewer hoops. You’re not crossing into—

Even if it’s AWS-to-AWS, they might have dedicated fiber that goes between their things. But every time you go up a level as far as networking goes, you’re introducing latency and complexity. It’s getting harder. As a rule of thumb, the closer things are together, the easier it is to manage, and the fewer variables you’re dealing with, the fewer problems you’re likely to have.

Now, within regions—

Okay, that’s interesting. From your article, my takeaway was that it’s cheaper to have it all in one place, but it also sounds like it’s just more efficient and you’ll have fewer errors if you have it all centralized.

Oh, it’s so much easier. Managing multiple clouds is hard. Even within a cloud, managing multiple regions is hard. Now, within a region, there can then be individual data centers, and that’s sort of abstracted away.

I think Amazon probably has super-high-level plans—I don’t know. This is beyond the realm of my knowledge. They probably have certain even higher-level plans or whatever, where you can guarantee that stuff’s close, even closer together than just—

Mm-hmm.

—being in the same region. But basically, the long and short of it is that by far the easiest and most resilient, to use a word, way to build an application is to have it all in one region. Your databases are there, your compute is there, everything is in one place, and it can sort of work together.

Yeah.

And so us-east-1 is a region of AWS. AWS has tens of, maybe even hundreds of regions. It’s important that you have laws about where data has to be located. This is a big thing in Europe, right? Data has to be in Europe. It has to be in—

Mm-hmm.

But you also want failover regions. You don’t want all your data in one place, because what if that place gets hit by a nuclear bomb, a natural disaster, or whatever it might be?

Yeah.

You need it elsewhere. Or what if the region goes down? What happened here isn’t clear, other than that it’s some sort of DNS issue. If you think networking is hard, this extra-hard part of networking is DNS.

DNS. Here we go.

DNS is basically domain name service. If you type amazon.com, amazon.com is meaningless. What happens is that you go to a DNS server that translates amazon.com to an IP number.

Mm-hmm.

The most-used ones are IPv4. There’s also IPv6, but there’s IPv4, which is that string of 4 numbers, like 192.168.1.0. Now, that’s usually your home, or then we get into NATs and what all that means.

I know. How deep down the rabbit hole do we want to go?

Yeah, but basically, if you have the Amazon IP address, you could just type it into your browser, and it would get there. But there’s a human-translatable aspect here.

But actually, all that NAT stuff is important because there was a huge scare a while ago. The reason why you have IPv6 is that IPv4 is limited; there are only so many numbers. It's 10 to the power of 12.

Yeah.

I think, for every device in the world—every device has its own IP address. My computer has 2 IP addresses: one for the Wi-Fi and one for the network connection. My house alone has hundreds of devices with IP addresses. What happened—

Okay.

It was this crazy hack called NAT. Basically, I have a router that sits facing the internet and gets the incoming connection. Then, within it, it creates a network inside the house and redistributes IP addresses. So your—

Mm.

Your computer's IP address could be the same as my computer's, and it doesn't matter because they're on different NATs. Anyhow, the problem is within—when you think about us-east-1, this huge data center with a gazillion servers, all of which need their own IP addresses.

IP addresses. Yeah.

There are all these tables and translation layers happening, so I can sit here in my office on my laptop and access something that ultimately resides inside a database within a virtual machine in Northern Virginia, or whatever it might be.

Mm-hmm.

Something in that got screwed up.

Okay.

Easy to screw up, by the way. I'm sure Amazon will be looking—

I'm surprised it doesn't happen more often—

—to see what happened and how this happened.

Frankly.

But as soon as they said, “Oh, something went down,” everyone involved with networking probably assumed, “Oh, it might be a DNS issue.” They say it's a DNS issue. Yeah, that makes sense.

It's very hard to recover from because there are all sorts of dependencies. Even with home networking, if you screw up something with an IP address, you can't reach the device. You have to go to the device and connect to it directly to reset it. There's DHCP; it's a whole thing. I'm doing home networking with DHCP, but—

When that happens for me, I call Ben, and Ben walks me through it. I'm not really sophisticated enough to handle a problem like that.

Right.

Yeah, it's really complex. It's really difficult, and you can see how problems like this happen. What's interesting—

Yeah.

—and this is sort of the point of the article—is how broadly impactful a problem like this ended up being.

Totally.

US East 1 is actually really interesting historically. You go back to the early ’90s, and the internet is just becoming a thing. The internet starts as ARPANET, this military project. The myth around the internet is that it was created to avoid a nuclear attack. That's—

Yeah.

—actually not true. It was created because there were very few supercomputers in the world back then. A supercomputer then was slower than your Apple Watch today.

Mm-hmm.

They wanted to give more researchers access to it without having to be physically there. So the initial internet—

So widely dispersed was the goal.

Well, no, it was just to get access to computing, which is actually—

Okay.

—cloud computing today. So you could access it. Now, along the way, where the nuclear story comes in is that it was a motivation for the creation of what's called packet switching. Instead of there being a serial connection, where you have to have a direct connection to a computer and send the data as a stream of bits, the way data—

Mm-hmm.

—is actually distributed on the internet is that it's all chopped up into packets, and all those packets are sent individually. Imagine if I'm mailing you something. Instead of mailing you a big box with the whole thing, I send the box in all its constituent atoms.

Great.

They're very small, and they can all make it. If any one gets blocked—

A billion little boxes on my doorstep.

Yeah. It just—

Yep.

—it routes around automatically. It finds another way there because it's all interconnected. Even if the path is longer, it can still get there. Then, when it gets to your house, it reconstitutes itself into an object. That's how—

Yeah.

—the internet works. A motivation for the creation of packet switching was, in fact, the nuclear threat. That's where the internet's nuclear story comes in. Back in the day, the original ARPANET was sort of serial: send it directly. If one of the nodes got bombed, you couldn't route it.

Mm-hmm.

After packet switching, you could sort of route around. This is one of the many truths and myths about the internet: It is highly resilient. It can survive a nuclear attack. As long as something is connected, stuff can get through.

Yeah.

The theme here, though, and what we saw this week, is that the internet is, in theory, resilient. If you want to be one of the Excel spreadsheet guys who are mad at Chuck and Kenny, it is resilient. But sometimes there's a difference between theory and the actual experienced reality of everyone.

When you were at Starbucks trying to access The Wall Street Journal to prep for Shark Tank China—

Mm-hmm.

—the internet was not feeling very resilient to you.

Life came at me fast. Indeed. Well, my question, reading the article and reading this portion of the article specifically—

Oh, sorry, I didn't finish my history of US East 1. Do you want some more?

Okay. Well, I've driven out there a lot, and it's unbelievable how many data centers have popped up over the last 15 years in Northern Virginia. But this is the big mama data center, so, yeah, give me more US East 1 history.

3. Northern Virginia Became Essential

As they're building out the internet, there is a major exchange that was built in because of Washington, D.C., I think almost certainly. You had these initial laydowns of—I don't even know if it was fiber back then, whatever they were using to sort of—

Mm-hmm.

Because what happens is, when you connect, my laptop connects to my Wi-Fi access point. My Wi-Fi access point is connected to my router. My router is connected to my ISP, and then it makes its way out onto the public internet.

There are basically these big pipes that—

Yeah.

—interconnect. There's a specific name for them. It makes its way to one of the big pipes. It's going to Washington, D.C., probably to this internet exchange, and then it's going to your local ISP and filtering down to where you're at right now. Again, it's a miracle any of this works. And so—

Mm-hmm.

—there was this internet exchange that was built there, number one. Number two, you had land and pretty cheap power, particularly back in the early ’90s. You also had a region that doesn't get that many natural disasters. You guys get a hurricane maybe once every—

We do have that going for us.

Yeah, but—

Some pretty rough summers. The humidity can feel like a natural disaster every now and then.

No, but that doesn't count.

I'm just kidding.

We're talking about hurricanes.

But—

We're talking about earthquakes. We're talking about—

Right.

—tornadoes. And so it was good. That's actually a really important consideration because you don't want your data center to go up in a tornado.

Yep.

But then also, the geographic location: It's right in the middle between Europe and the West Coast.

Yeah.

So you have—remember, distance matters. So if you're going—

That was what was fascinating from your article. You wouldn't want to put it in the middle of the country—

Right.

—if you're also trying to serve Europe.

In the Northeast, it's kind of hard to build there, right? It's so built out. We see this with Amtrak, right? Amtrak got a new train. Congratulations. It's just marginally slower than the old one.

It's slower.

Because that's a track issue, right?

We're doing great in this country.

That's right. Well, it's super dense up there. It's hard to build. Northern Virginia had space. You had this interchange there, and it was close, you know? And so you had—

And I never knew that, just for the record. I always assumed that the reason the data centers were out there was exclusively because of proximity to the federal government, and that's why it was set up that way. But the latency point—that's why I enjoyed that emphasis from you.

Well, so there is a Washington Wizards tie-in.

I'm aware.

What happened—

AOL, baby.

—in the late ’90s is a company called AOL—

We used to be the center of all this shit back in 1996, or earlier.

And now Ted Leonsis owns the Wizards and managed at AOL as well. So, actually, AOL is a great company.

Yeah. They had a nice run.

They had a nice run. That's a good way to put it—a better run than the Wizards ever had.

AOL was established in Northern Virginia. It sort of establishes this space as data center central. They were the biggest internet company in the late '90s. They had a huge build-out, and you get this sort of network density—not just of the data centers and things like that, but power was built up to support this. The grid was built up to support this.

You have all the pieces that go into it, and you start getting people. There are a lot of infrastructure engineers in Northern Virginia. Some of them were there for the federal government, for sure. That was certainly true, so you had a talent base to start. AOL really accelerated this and basically made Northern Virginia the center of the world as far as data centers go.

So when Amazon comes along and says, "We're going to establish this thing called AWS," which no one really understood or appreciated how important it was, there was only one place to put the first data center. That was in—

Yeah.

Northern Virginia. That data center is US East 1. It's where everyone who started at AWS went. Every tutorial that was created about creating a server on Amazon talks about US East 1.

US East 1 got all the new equipment. When Amazon launched new instances—an instance is basically a type of compute you want—you could get a basic instance with an Intel processor. There are the Graviton processors. All these have different memory configurations and different amounts of reserved capacity. There are a gazillion options. All of this—the most options, the newest features, the largest number, the highest availability, and the lowest prices because there's just a bunch there—has always been—

Mm-hmm.

US East 1. So the default place to build anything, particularly anything that was built early on—and remember, back then there was just AWS for a long time before Azure came along, before GCP came along—was US East 1.

And it's funny because Passport did not go down—

Yeah.

because we're on US East 2, specifically because US East 1 has always been this ticking time bomb. There was a US East 1 issue six or seven years ago, right before we started Passport, and I said—

Mm-hmm.

I'm like, "No, we're not going to be on US East 1."

There you go.

Which is probably somewhat delusional. It worked out this week. But, yeah, it's just one of those things: it's cheaper, it's the default, it's what you do. It's just the center of the internet world.

And, yeah, the internet is a bunch of nodes, and any one node can go out. But the actual—

Mm-hmm.

experience, again, to go back to you in line at Starbucks, is that one node is actually more important than the rest of them.

Yeah, and everybody's on US East 1—or it felt like it on Monday morning, certainly. My question is, obviously, this is cheaper, it's more efficient, and everything is set up on US East 1. Your article was about the erosion of resiliency that's resulted from the present-day state of affairs and some of the tough choices that might need to be made across a couple of different industries in order to build that resiliency back in.

With this particular situation, though, we're talking about private companies. How does that dynamic change? Are they basically scared straight after an incident like Monday, which cost God knows how much money to God knows how many companies?