ongoing by Tim Bray

ongoing fragmented essay by Tim Bray

Cloud Traffic 9 Aug 2020, 7:00 pm

I recently watched Build an enterprise-grade service mesh with Traffic Director, featuring Stewart Reichling and Kelsey Hightower of GCP, and of course Google Cloud’s Traffic Director. Coming at this with a brain steeped in 5½ years of AWS technology and culture was surprising in ways that seem worth sharing.

Stewart presents the problem of a retail app’s shopping-cart checkout code. Obviously, first you need to call a payment service. However it’s implemented, this needs to be a synchronous call because you’re not going to start any fulfillment work until you know the payment is OK.

If you’re a big-league operation, your payment processing needs to scale and is quite likely an external service you call out to. Which raises the questions of how you deploy and scale it, and how clients find it. Since this is GCP, both Kubernetes and a service mesh are assumed. I’m not going to explain “service mesh” here; if you need to know go and web-search some combination of Envoy and Istio and Linkerd.

The first thing that surprised me was Stewart talking about the difficulty of scaling the payment service’s load balancer, and it being yet another thing in the service to configure, bearing in mind that you need health checks, and might need to load-balance multiple services. Fair enough, I guess. Their solution was a client-local load balancer, embedded in sidecar code in the service mesh. Wow… in such an environment, everything I think I know about load-balancing issues is probably wrong. There seemed to be an implicit claim that client-side load balancing is a win, but I couldn’t quite parse the argument. Counterintuitive! Need to dig into this.

And the AWS voice in the back of my head is saying “Why don’t you put your payments service behind API Gateway? Or ALB? Or maybe even make direct calls out to a Lambda function? (Or, obviously, their GCP equivalents.) They come with load-balancing and monitoring and error reporting built-in. And anyhow, you’re probably going to need application-level canaries, whichever way you go.” I worry a little bit about hiding the places where the networking happens, just like I worry about ORM hiding the SQL. Because you can’t ignore either networking or SQL.

Google Traffic Director

Traffic Director

It’s an interesting beast. It turns out that there’s a set of APIs called “xDS”, originally from Envoy, nicely introduced in The universal data plane API. They manage the kinds of things a sidecar provides: Endpoint discovery and routing, health checks, secrets, listeners. What Google has done is arrange for gRPC to support xDS for configuration, and it seems Traffic Director can configure and deploy your services using a combination of K8s with a service mesh, gRPC, and even on-prem stuff; plus pretty well anything that supports xDS. Which apparently includes Google Cloud Run.

It does a lot of useful things. Things that are useful, at least, in the world where you build your distributed app by turning potentially any arbitrary API call into a proxied load-balanced monitored logged service, via the Service Mesh.

Is this a good thing? Sometimes, I guess, otherwise people wouldn’t be putting all this work into tooling and facilitation. When would you choose this approach to wiring services together, as opposed to consciously building more or less everything as a service with an endpoint, in the AWS style? I don’t know. Hypothesis: You do this when you’re already bought-in to Kubernetes, because in that context service mesh is the native integration idiom.

I was particularly impressed by how you could set up “global” routing, which means load balancing against resources that run in multiple Google regions (which don’t mean the same things as AWS regions or Azure regions). AWS would encourage you to use multiple AZ’s to achieve this effect.

Also there’s a lot of support for automated-deployment operations, and I don’t know if they extend the current GCP state of the art, but they looked decent.

Finally, I once again taken aback when Stewart pointed out that with Traffic Directors, you don’t have to screw around with iptables to get things working. I had no idea that was something people still had to do; if this makes that go away, that’s gotta be a good thing.

Kelsey makes it go

Kelsey Hightower takes 14 of the video’s 47 minutes to show how you can deploy a simple demo app on your laptop then, with the help of Traffic Director, on various combinations of virts and K8s resources and then Google Cloud Run. It’s impressive, but as with most K8s demos, assumes that you’ve everything up and running and configured because if you didn’t it’d take a galaxy-brain expert like Kelsey a couple of hours (probably?) to pull that together and someone like me who’s mostly a K8s noob, who knows, but days probably.

I dunno, I’m in a minority here but damn, is that stuff ever complicated. The number of moving parts you have to have configured just right to get “Hello world” happening is really super intimidating.

But bear in mind it’s perfectly possible that someone coming into AWS for the first time would find the configuration work there equally scary. To do something like this on on AWS you’d spend (I think) less time doing the service configuration, but then you’d have to get all the IAM roles and permissions wired up so that anything could talk to anything, which can get hairy fast. I noticed the GCP preso entirely omitted access-control issues. So, all in, I don’t have evidence to claim “Wow, this would be simpler on AWS!” — just that the number of knobs and dials was intimidating.

One thing made me gasp then laugh. Kelsey said “for the next step, you just have to put this in your Go imports, you don’t have to use it or anything:

_ "google.golang.org/xds"

I was all “WTF how can that do anything?” but then a few minutes later he started wiring endpoint URIs into config files that began with xdi: and oh, of course. Still, is there, a bit of a code smell happening or is that just me?

Anyhow

If I were already doing a bunch of service-mesh stuff, I think that Traffic Director might meet some needs of today and could become really valuable when my app started getting heterogeneous and needed to talk to various sorts of things that aren’t in the same service mesh.

What I missed

Stewart’s narrative stopped after the payment, and I’d been waiting for the fulfillment part of the puzzle, because for that, synchronous APIs quite likely aren’t what you want, event-driven and message-based asynchronous infrastructure would come into play. Which of course what I spent a lot of time working on recently. I wonder how that fits into the K8s/service-mesh landscape?

Long Links 4 Aug 2020, 7:00 pm

Back in early July I posted ten links to long-form pieces that I’d had a chance to enjoy because of not having one of those nasty “full-time-job” things. I see that the browser tabs are bulking up again, so here we go. Just like last time, people with anything resembling a “life” probably don’t have time for all of them, but if a few pick a juicy-looking essay to enjoy, that’ll have made it worthwhile.

The Truth Is Paywalled But The Lies Are Free by Nathan J. Robinson. I think we’ve all sort of realized that the assertion in the title is true. Robinson doesn’t lay out much in the way of solutions but does a great job of highlighting the problem. I dug into this this myself last year in Subscription Friction. It’s a super-important subject and needs more study.

Like many, I’ve picked up a new quarantine-time activity — in my case, Afro-Cuban drumming lessons. I’ve been studying West-African drumming for many years, but this is a different thing. Afro-Cuban is normally done on a pair of conga drums as opposed to djembé and dunum. At the center of the thing is the almighty Clave rhythm. In The Rhythm that Conquered the World: What Makes a “Good” Rhythm Good? Godfried Toussaint of Harvard dives deep on the “clave son” flavor (which is sort of the Bo Diddly beat with the measures flipped, and with variations) and has fun with math and music! I personally find the “rumba clave” variation just absolutely bewitching, when you can get it right. For more on this infinitely-deep rabbit hole, check out Wikipedia on Songo and Tumbao.

There’s a new insult being flung about in the Left’s internecine polemics: “Class Reductionist”. A class reductionist is one who might argue that while oppression by race and gender and so on are real and must be struggled against, the most important thing to work on right now is arranging that everyone has enough money to live in dignity. There is some conflict with “intersectional” thinking, which argues that the the struggles against poverty and sexism and racism and LGBTQ oppression aren’t distinct, they’re just one struggle. In some respects, though, the two are in harmony: A Black trans woman is way more likely on average, to be broke. In How calling someone a "class reductionist" became a lefty insult, Asad Haider takes on the subject in detail and at length and says what seem to me a lot of really smart things. Confession: I’m sort of a class reductionist.

In The New Yorker, Why America Feels Like a Post-Soviet State offers that rare thing, a new perspective on 2020’s troubles. By Masha Gessen, who knows whereof they speak.

As anti-monopoly energy starts to surge politically around the world, it’s important to drill down on specifics. It’s gotten to the point where I hesitate to point fingers at Amazon because I actually think the Google and Facebook monopolies need more urgent attention, but since details matter, here’s Amazon’s Monopoly Tollbooth, which tries to assemble facts and figures on whether and how the Amazon retail operation has a monopoly smell. And also on a related but distinct subject: The Harmful Impact of Audible Exclusive Audiobooks. The latter is open to criticism as being the whining of a losing competitor. But when you start to hear that a lot, you may be hearing evidence of monopoly.

In “Hurting People At Scale”, Ryan Mac and Craig Silverman dig hard into Facebook employee culture and controversy. I think this is important because the arguments Facebookers are having with each other are ones we need to be having at a wider scale in society.

Related: A possibly new idea: “Technology unions” could be unions of conscience for Big Tech, by Martin Skladany. It’s pretty simple: High-tech knowledge workers are well-paid and well-cared-for and hardly need traditional unions for traditional reasons. But they’re progressive, angry, and would like to work together. Maybe a new kind of “union”?

Most people reading this will know that I’m an environmentalist generally, a radical on the Climate Emergency specifically, and particularly irritated about the TransMountain (“TMX”) pipeline they’re trying to run through my hometown to ship some of the world’s dirtiest fossil fuels and facilitate exhausting their (high) carbon load into the atmosphere. Thus I’m cheered to read about any and all obstacles in TransMountain’s way, most recently having trouble finding insurers.

Also on the energy file, many have probably heard something about the arrest of the Ohio House of Representatives Speaker Larry Householder on corruption charges. An FBI investigation shows Ohio’s abysmal energy law was fueled by corruption, by Leah C. Stokes, has the goods, big-time. This is corruption on such a huge blatant scale that it feels cartoonish, with a highly-directed purpose: Directing public money into stinky bailouts of stinky dirty-energy companies. When the time has come that you have to bribe the governments of entire U.S. States to keep the traditional energy ecosystem ticking over, I’d say the take-away is obvious.

Finally, some good ol’ fashioned leftist theory. Recently, a few people out on the right (e.g. Tucker Carlson) have been saying things that, while they retain that Trumpkin stench, flame away at inequality and monopoly in a way that seems sort of, well, left-wing. It sounds implausible that Fox-head thinking could ever wear the “progressive” label, thus Mike Konczal’s The Populist Right Will Fail to Help Workers or Outflank the Left (Tl;dr: “Pfui!”) is useful.

Not an Amazon Problem 23 Jul 2020, 7:00 pm

The NY Times profile quotes me saying “We don’t really have an Amazon problem. What we have is a deep, societal problem with an unacceptable imbalance of power and wealth.” But the URL contained the string “amazon-critic-tim-bray” and the HTML <title> says “Tim Bray Is Not Done With Amazon”. I feel like I’m being pigeonholed and I don’t like it.

I am totally not some sort of anti-Amazon obsessive. And I am done with Amazon; even though I’m doubtless on a dozen internal enemies lists, I have no real interest in giving the company, specifically, a hard time. I’m mad at the structure of 21st-century capitalism; the fabric of society is in danger of breaking. I’m enjoying the privilege of having an audience for my criticism and can’t see any reason to limit it to one corporation. Because the problem is so big that any of them, even Amazon, is rounding error.

There’s also the fact that I admire many things about Amazon:

  1. It’s by far the best-managed place I ever worked, including the places where I was the CEO.

  2. The people I worked with in AWS were, on average, decent, honest, and smart.

  3. The climate pledge is admirable. (Haven’t seen much action recently, and I guess Covid is a not-terrible excuse, but come on, the climate emergency’s not going away.)

  4. The company is working hard on Diversity & Inclusion. The results are meh (like all over the tech biz) but the energy and funding are there.

  5. Amazon.com has improved the world by showing what well-done online shopping is like. The world likes it. It remains to be seen what the world is like when there are a lot more parties offering really great online shopping.

  6. AWS has improved the world of IT by showing what well-done cloud computing is like. The IT world likes it. Public-cloud competition is stiffening, but AWS is a really good choice for almost anyone’s needs. Its lead over the competition is well-earned. I’m doing a bit of consulting here and there and have found myself recommending AWS more than once already.

Except for that thing

The firing-whistleblowers thing I mean. They shouldn’t have done that and there’s no excuse. I still think quitting was the only thing to do. But I feel no need (nor, apparently, does Amazon) to relitigate that issue.

Well, and one other thing: Amazon’s absurd and cruel down-to-the-last-ditch-and-beyond resistance to unionization. There’s no way that’s reasonable. I’ve gotten to know some fine, inspiring people working that front and I’ll support them going forward, any way I can.

US family wealth distribution

The rest of the things

There’s lots more to complain about but little of it is specific to Amazon, it’s all about 21st-century-capitalism, like so:

  1. Warehouse workers’ lives are often shitty. In fact, life for the US working class these days is shitty, and it’s not by accident, it’s by design. It was called the Reagan-Thatcher neoliberal consensus, and it was wrapped up with lots of high-flying rhetoric about this freedom and that dynamism and those flexibilities, but you don’t have to be that cynical to see it as good old-fashioned class war. It’s obvious who’s winning.

  2. Big Tech sells technology to loathsome customers, for example carbon-vomiting oil extractors and overempowered child-caging immigration officials. Of course, possibly the bug is that these sorts of organizations are allowed to have oceans of money. Fix that and remove the temptation!

  3. Big Tech emits a whole lot of carbon into the atmosphere and facilitates the emission of much, much more. Humanity can’t afford this and it has to stop.

  4. Big Business in general and Big Tech in particular is waaaay too powerful; the millions poured into lobbying have fabulous ROI.

Break ’em Up

I’m strongly convinced that the structure of Big Business in general and Big Tech in particular is too concentrated in ways that are damaging along multiple axes. Check out Anti-Monopoly Thinking and Large Companies Considered Harmful; this is not a new opinion.

Let’s start with Google. It feels the most urgent because the Google/Facebook ad cartel is rapidly destroying publishing business models that have been essential to civilized human dicourse.

Next I’d go after Microsoft who are still, all these decades later, using the cash and incumbency advantages of their steely grip on the business desktop and email server to invade other business sectors. Which is to say, Slack has a gripe. This one is particularly maddening to me because Microsoft has been doing this shit since I was carving code on stone tablets.

By the time you’d launched those campaigns I suspect Amazon would wake up, smell the coffee, and spin out AWS already. Then we’ll see what the retail business is really like given the violent contrast between the galactic revenue growth and the negligible profit margin.

Media Friction

I’ve had a certain amount of friction with the journalists who found me interesting after my May 1st exit from Amazon. Because there was this great, simple, story that they’d like to have told: Plucky engineer leads geek revolt against Jeff Bezos, Richest Man In the World.

That struggle story — Man against company, or Man against Billionaire — is a crowd-pleaser. The actual struggle that interests me is against the current horrifying imbalance in global power and wealth, which is kind of abstract, doesn’t have a chiseled cartoon-villain billionaire in the cast, and is frighteningly large in scale.

Seriously; basically every reporter I’ve talked to has tried to get me to say awful things about Amazon and in particular about Jeff Bezos. But at my last job they taught me to think big and, with all his billions, Jeff is rounding error in the big picture. He’s not the problem; the legal/regulatory power structures that enable him and his peers is.

Amazon is a perfectly OK company, to the extent that planetary-scale sprawling corporate behemoths can be perfectly OK in 2020. Which is to say, not OK at all.

But once again, no one company is the problem. The problem’s the entirely-fixed great global card game illustrated by that graph, above. The problem’s the fact that in one of the world’s wealthiest economies, we have ever-growing homeless camps where they’re dying of opioids faster than Covid. (That’s maybe not the worst part, but it’s the one I see with my own eyes.)

We can do way better as a society than the greed-fueled planet-destroying worker-crushing hamster-wheel we’re spinning right now.

That’s why I’m making noise.

Polishing Lawrence 21 Jul 2020, 7:00 pm

I edit Wikipedia as a hobby, and recommend that hobby. (Fifteen or so years ago I was an early defender in the days when many scoffed at the Wikipedia idea.) I spent much of this past weekend on a long-running editing project, the entry for T.E. Lawrence, better known as Lawrence of Arabia. When I finished up on Monday I felt like a milestone had been passed so I decided to share, because the story behind it seemed worth telling.

If you want the facts about Lawrence, go read the entry, it’s not that long. Briefly: An archeologist, he served in WWI as liaison between British forces in the Middle East and the “Arab revolt”, both of which were fighting the Ottoman Turks, allies of Germany. After the war, he became household-name famous, renounced that fame to serve as an enlisted man in the RAF, wrote a couple of books (Seven Pillars of Wisdom the best-known) and a huge volume of letters (fun to read) and died aged 46 in a motorcycle accident. His fame increased posthumously because of David Lean’s film Lawrence of Arabia, you’ve all seen it.

Editing this entry has consumed probably a few weeks of my life in aggregate, and I’m content with that. It involved one edit war with a crazed Islamist which required that I drive to the nearest university research library and consult obscure journals that nobody should ever have to look at. It has also turned me, I claim, into the world’s greatest living expert on Lawrence’s sexuality.

Library books about T.E. Lawrence

Meta-Lawrence

The entry has long had a reasonable description of the significant things Lawrence did. A related but distinct subject is the truth or falsity of his public image.

Immediately after the Great War ended, Lowell Thomas, an American impresario, toured with a multimedia presentation entitled With Allenby in Palestine and Lawrence in Arabia, including film, slides, dancing girls, and Thomas’ own narration. It was wildly successful and turned Lawrence into perhaps Britain’s most famous war hero of the time.

Two biographies were written during Lawrence’s lifetime, by B. H. Liddell Hart and Robert Graves. They were very rah-rah, attributing nearly godlike powers to their subject.

Then, in the early 1950s Richard Aldington, a well-regarded poet, novelist, and biographer, signed up to produce another tour of Lawrence’s life. His research convinced him that Lawrence was illegitimate, a liar, a sexual pervert, a self-promoter, and a bad writer, whose military contributions were negligible and misguided. Among other things, he discovered that Lawrence had significant input into the Graves and Liddell Hart biographies, apparently having written some sections himself.

Aldington’s draft biography, which said this at length, was leaked before publication. Lawrence’s surviving friends and admirers, horrified, sallied forth ready to rumble; the ensuing kerfuffle is quite a story. Liddell Hart led the “Lawrence Bureau” in an attempt to have the book suppressed. When that failed, he circulated hundreds of copies of an anti-Aldington tract to all the likely reviewers. This worked pretty well; when Lawrence of Arabia: A Biographical Enquiry was published in 1955, the reviews were savage; some showed evidence of having read Liddell Hart’s rebuttal but not Aldington’s book.

To be fair, the book was mean-spirited and full of anger, what today we’d call “trolling”. It also revealed that Aldington was a believer in the virtues of European colonial rule over brown people; exactly what Lawrence’s attempts to establish post-War Arab freedom were designed to thwart. Furthermore, Aldington (who lived in France) was also enraged by what he saw as Lawrence’s Francophobia — in fact, Lawrence’s efforts to block France’s colonial dreams were a function of his desire to see a free Arab state arise in Syria.

Books about T.E. Lawrence

My sincere thanks to the Vancouver Public Library system
for the use of these.

Prejudice aside, Aldington did discover and publish plenty of evidence of Lawrence having embellished the narrative of his life, and likely even invented some of it from whole cloth. He was particularly guilty of inflating numbers: Of bridges he’d blown up, of wounds he’d suffered, of the price on his head, and of the number of books he’d read. Subsequent biographers, most of whom remained admirers of Lawrence (as do I), freely acknowledge the truth of most of Aldington’s claims.

The whole thing is a hell of a story, best told by one Professor Fred D. Crawford in Richard Aldington and Lawrence of Arabia: A Cautionary Tale. It’s not exactly a bestseller; you’ll probably need to hit a big library to find a copy.

The production of Aldington’s book was a financial and emotional disaster for him from which he never fully recovered. Granted, he was clearly a “difficult person”, but still, one feels he deserved better.

Milestone?

Up until last week, the Wikipedia entry, while reasonably good on Lawrence’s military, political, and writing activities, had entirely ignored the Aldington controversy and ensuing revolution in Lawrence studies. No longer. Having whittled away at this thing for over a decade I think the entry is, broadly speaking, complete. Obviously no Wikipedia entry is ever finished because the course of human knowledge advances and better references can always turn up for the facts as outlined. But I still felt good when I posted the new section, and better still when, the next morning, it had picked up a couple of friendly edits and apparently failed to provoke any editorial opposition.

At this point I should emphasize that I don’t get to make the completeness call because I don’t “own” the Lawrence article; nobody owns anything in Wikipedia. Nor do I “lead” the editing in any meaningful sense. Many others have done lots of solid work over the years.

Join in!

This isn’t how I’d recommend everyone spend their weekends, but I find it rewarding sometimes. Who knows, you might too! Because everybody —everybody — is a world-class expert in something, be it only their own hometown, neighborhood, or hobby. Why not share that knowledge?

Wikipedia one of the few places on the Internet where truth is pursued, intently and unironically, for its own sake. It’s not perfect because nothing is, but everyone has the opportunity to make it more so.

Safari Tabs Are Weird 17 Jul 2020, 7:00 pm

Recently I switched from Chrome to Safari because on a 2019 16" MacBook Pro, Chrome has nasty video glitches; apparently Apple and Google are blaming each other for the problem. My first impression of Safari is decent, subjectively it feels faster than Chrome. But browser tabs act in ways that that feel somewhere between “weird” and “badly broken”. Here are a few of the issues I’ve encountered.

[Update: Lots of free wisdom on offer in the comments here, including some that improved my life.]

Tab display

On Chrome, if you have a lot of tabs — I typically have 20 or 30 open — they get narrower and narrower and until completely crushed out of existence show a little glimpse of icon and label, so you can find what you’re looking for.

On Safari, the tabs near the one you’re on are shown pretty well full-width with quite a bit of white-space surrounding labels and icons (where by “white ” I mean “grey”). You can two-finger drag the tab label array right and left and they shuffle around in a graceful way.

I don’t like this because I lost all-my-tabs-at-a-glance. If you have hundreds open I guess that wouldn’t work anyhow.

But I do like it because when I’m looking for a tab that’s not obvious to the eye, it’s easier to find. Hmm, I’ll call this a saw-off.

Tab pinning

All the browsers support tab pinning these days, which is great because it enables the Mighty Tab Trick, where you park the eight tabs you use all the time and insta-hop to them using CMD-1 through CMD-8.

But Safari pins tabs hard. If you follow a link off the site you’re at, new tab. If you go somewhere, pin it, then press Back, new tab. You can’t nuke the tab with CMD-W. I mean it’s not pinned, it’s nailed the fuck down.

This can lead to weirdly buggy behavior, which this blog provokes. If you’re reading this in a web browser, there’s a search field up the top right which does a dumb simple Google search of the blog. Now, I keep ongoing in a pinned tab for obvious reasons. If I a do a search, the blog is replaced by a Google search output page. Fine. But that page now exhibits hard-pinning. If I click a result, that opens in a new tab. If I press back, it opens my blog… in a new tab. In fact there is absolutely no way back to the blog from the search result page. I think I’d call this a bug?

Left and right

[Update: Check the comments for better options. I think I withdraw this complaint.]

You can move from any tab to the next one on the left or right. The key combo is CMD-Shift-left/right, annoyingly different from other browsers. But a lot of the time it just doesn’t work. If you’re in a form, or a text-edit field, or really any mode that you might be able to interact with what’s in the tab, you can’t step left or right.

I often put a couple of tabs next to each deliberately, like when I’m editing a Wiki page and I have supporting evidence I need to copy from, if it’s in the next tab it’s easy to flip back and forth. But impossible on Safari because if you can edit, you can’t flip.

This one really feels like a bug. Or am I weird?

Where do new tabs go?

[Update: “Debug” menu suggestion below: Verified; note that you have to click several things in that sub-sub-menu.]

I mean tabs that are opened because the link requested it, like from a search-results page or, as described above, a pinned tab, not because you hit CMD-T to ask for one. Safari is consistent with Chrome on this one, which is a pity, because they’re both wrong, they both open the new tab as close as possible to where you are. Which, if in you’re in the middle of a bunch of pinned tabs, is not what you’d expect.

I’d actually like it if all new tabs opened over at the right end of the row. At least as a settings option. Because then you’re tabs would naturally arrive at a structure of the oldest ones being at the left and the newest ones to the right. I can see room for argument about this, but I suspect the current policy was arrived at before a lot of people had many pinned tabs, because that’s where the behavior becomes confusing.

Did I miss any?

I wish everything worked the same.

Long Links 7 Jul 2020, 7:00 pm

Having recently quit my job, I have more spare time than I used to. A surprising amount of it has been dedicated to reading longer-form articles, mostly about politics and society, but only mostly. I miss my job but I sure have enjoyed the chance to stretch out my mind in new directions. There are plenty of things in the world that need more than a thousand words to talk about. Anyhow, here is a set of lightly-annotated links that people who still have jobs almost certainly won’t have time to read all of. But maybe one or two will add flavor to your life.

Yavne: A Jewish Case for Equality in Israel-Palestine, by Peter Beinart, assumes the death of the two-state option for Israel/Palestine, and takes its time arguing for the only plausible long-term alternative: Some sort of unitary state in which the citizens are equal whatever their religious heritage, be that state multinational, federal, or contonal. Deep, important, stuff.

Tuning for beginners and (especially) Extra stuff on tunings by, uh, I’m not sure who actually, delivers a horribly-typeset crash course on the mathematics of music. Pretty fun for the (quite a lot of) people who are interested by both.

Trivium is the link-blog of Leah Neukirchen, “Just another random shark-hugging girl”, which gleefully pokes around the deepest, darkest, dustiest corners of software tech.

Battery energy storage is getting cheaper, but how much deployment is too much? by Herman K. Trabish, is an exemplar of my growing fascination with energy economics. In particular storage. The path to an all-reenewables energy ecosystem is pretty straightforwardly visible, except for storage infrastructure. This helps.

In the Covid-19 Economy, You Can Have a Kid or a Job. You Can’t Have Both. Ouch; maybe the most important thing written about the current plague that doesn’t actually instruct on how to save myriads of lives. Covid is shining a harsh, harsh light on lots of things we’ve been ignoring but shouldn’t have.

Richard Rorty’s prescient warnings for the American left. Sean Illing in Vox walks through Rorty’s taxonomy of American progressivism. There’s lots for almost anyone to disagree with here, but the Left needs to think about how it thinks about culture and class and, you know, get its freaking story straight. I think this might be helpful.

What About the Rotten Culture of the Rich? is by Chris Arnade, the guy who wrote the book about the American underclass, as interviewed in the Macdonaldses of the flyover zones. He’s hard to classify politically, and the points here about the moral bankruptcy, and damaging effects, of ruling-class behavior, have been made elsewhere. But they’re made very well here.

End the Globalization Gravy Train is by J.D. Vance. I’m pretty far out left but have mental space for intelligent conservatism (see here), and this is that. I was impressed.

When a tyranny falls, maybe justice will be applied to its collaborators. In History Will Judge the Complicit, Anne Applebaum takes a very indirect route to considering how this might apply after November 2020.

It was gold, by Patricia Lockwood, is about Joan Didion. You might not know who Ms Didion is and still enjoy this, because Ms Lockwood is a fabulous writer; certain sentences should be framed in gold and hung in major galleries and museums. She had a book in one of the recent Booker long or short lists, which is on my must-read list.

Just Too Efficient 5 Jul 2020, 7:00 pm

On a Spring 2019 walk in Beijing I saw two street sweepers at a sunny corner. They were beat-up looking and grizzled but probably younger than me. They’d paused work to smoke and talk. One told a story; the other’s eyes widened and then he laughed so hard he had to bend over, leaning on his broom. I suspect their jobs and pay were lousy and their lives constrained in ways I can’t imagine. But they had time to smoke a cigarette and crack a joke. You know what that’s called? Waste, inefficiency, a suboptimal outcome. Some of the brightest minds in our economy are earnestly engaged in stamping it out. They’re winning, but everyone’s losing.

I’ve felt this for years, and there’s plenty of evidence:
Item: Every successful little store with a personality morphs into a chain because that’s more efficient. The personality becomes part of the brand and thus rote.
Item: I go to a deli fifteen minutes away to buy bacon, rashers cut from the slab while I wait, because they’re better. Except when I can’t, in which case I buy a waterlogged plastic-encased product at the supermarket; no standing or waiting! It’s obvious which is more efficient.
Item: I’ve learned, when I have a problem with a tech vendor, to seek out the online-chat help service; there’s annoying latency between question and answers as the service rep multiplexes me in with lots of other people’s problems, but at least the dialog starts without endless minutes on hold; a really super-efficient process.
Item: Speaking of which, it seems that when you have a problem with a business, the process for solving it each year becomes more and more complex and opaque and irritating and (for the business) efficient.
Item, item, item; as the world grows more efficient it grows less flavorful and less human. Because the more efficient you are, the less humans you need.

The end-game

Efficiency, taken to the max, can get very dark.

I suggest investing a few minutes in reading Behind the Smiles by Will Evans. Summary: Certain (not all) Amazon warehouses seem to have per-employee injury rates that are significantly higher than the industry average, as in twice as high or more. Apparent reason: It’s not they’re actually dangerous places to work, it’s just that they’ve maximized efficiency and reduced waste to the point where people are picking and packing and shipping every minute they’re working, never stopping. And a certain proportion of human bodies simply can’t manage that. They break down under pressure.

Robots matter, but not in the way you might think. The idea was that robotized warehouses should reduce stress and strain because they bring the pick-and-pack to the employees, rather than the people having to walk around to where the items are. But apparently robots correlate with higher injury rates. Behind the Smiles quotes employee Jonathan Meador: “‘Before robots, it was still tough, but it was manageable,’ he said. Afterward, ‘we were in a fight that we just can’t win.’”

It’s important to realize that Amazon isn’t violating any rules, nor even (on the surface) societal norms. Waste is bad, efficiency is good, right? They’re doing what’s taught in every business school; maximizing efficiency is one of the greatest gifts of the free market. Amazon is really extremely good at it.

And it’s good, until it isn’t any more.

Efficiency and weakness

Let’s hand the mike over to Bruce Schneier. In The Security Value of Inefficiency he makes one of those points that isn’t obvious until you hear it. Quoting briefly:

“All of the overcapacity that has been squeezed out of our healthcare system; we now wish we had it. All of the redundancy in our food production that has been consolidated away; we want that, too. We need our old, local supply chains — not the single global ones that are so fragile in this crisis. And we want our local restaurants and businesses to survive, not just the national chains.”

Bruce is pointing out that overoptimizing efficiency doesn’t just burn people out, it also too often requires cutting into what you later realize were prudent safety margins.

How hard should people work?

Today, we assume the forty-hour week without thanking the generations of socialists and unionists in the Eight-hour-day movement, whose struggle started around 1817 and didn’t bear global fruit until the middle of the twentieth century.

But there’s nothing axiomatic about forty hours. Twenty years ago, France introduced a 35-hour workweek. Their economy still functions. And John Maynard Keynes, approximately the most influential economist in the history of the world, predicted his grandchildren would enjoy a 15-hour workweek. It seems he was wrong. But maybe only partly.

And of course Keynes himself worked like a madman. As did I, for most of my career. Because some jobs are just jobs, but others are vocations; people doing what they love, and who’d really rather be working than not. Nothing wrong with that.

Some ideologists of Capitalism think that every business should try to make every job a vocation, that people should be delighted with their work, with the benefit (for the capitalist) that you don’t have to hire that many. One famous example of this thinking is at UPS, the delivery company, whose leaders wanted the delivery people to “bleed brown”. Here’s an interesting take on the UPS story, in which the “bleeding brown” notion didn’t catch on.

And while there’s nothing wrong with vocations — I’m lucky and blessed to have found one — most jobs are just jobs. Whether it be a job or a vacation, work should at least leave time for a smoke break in the sun at the corner (or its 21st-century equivalent). And it’s perfectly possible that Keynes’ prediction could come true, in certain future economic configurations.

But, wealth!

If we all work less, we’ll be poorer, right? Because the total cash output of the economy is a (weird, nonlinear) function of the amount of work that gets put in.

That sounds like it should be important, until you ask basic questions like “how much money is there, and who has it?” The answers, pretty clearly, are “Too much” and “An inefficiently small number of very wealthy people.” Business Insider has a nice take on the problem, highlighting the evidence for and consequences of there being just too much money around.

In practice, interest rates stay low, governments can borrow more or less for free, and all sorts of crazily doomed shit is getting investment funding. There is really no evidence anywhere of a global shortage of money, and plenty for the existence of a surplus.

Stop and think

Specifically, do it when something is annoying you; at work, or in your personal interaction with a big organization. Could this be explained by someone, somewhere, trying to be more efficient? In my life, the answer is almost always “yes”.

Cracks

“There is a crack in everything, that’s how the light gets in” sang Leonard Cohen. And there need to be cracks in the surface of work, in the broader organizational fabric that operates the world. Because that’s where the humanity happens. You can be like the people who optimize warehouses and call the gaps “waste”. But that path, followed far enough, leads to a world that we really don’t want to be living in.

It’s hard to think of a position more radical than being “against efficiency”. And I’m not. Efficiency is a good, and like most good things, has to be bought somehow, and paid for. There is a point where the price is too high, and we’ve passed it.

More Topfew Fun 1 Jul 2020, 7:00 pm

Back in May I wrote a little command-line utility called Topfew (GitHub). It was fun to write, and faster than the shell incantation it replaced. Dirkjan Ochtman dropped in a comment noting that he’d written Topfew along the same lines but in in Rust (GitHub) and that it was 2.86 times faster; at GitHub, the README now says that with a few optimizations it’s now 6.7x faster. I found this puzzling and annoying so I did some optimization too, encountering surprises along the way. You already know whether you’re the kind of person who wants to read the rest of this.

Reminder: What topfew does is replace the sort | uniq -c | sort -rn | head pipeline that you use to do things like find the most popular API call or API caller by plowing through a logfile.

Initial numbers

I ran these tests against a sample of the ongoing Apache access_log, around 2.2GB containing 8,699,657 lines. In each case, I looked at field number 7, which for most entries gives you the URL being fetched. Each number is the average of three runs following a warm-up run. (The variation between runs was very minor.)

ElapsedUserSystemvs Rust
Rust4.964.450.501.0
ls/uniq/sort142.80149.661.2928.80
topfew 1.027.2028.310.545.49

Yep, the Rust code (Surprise #1) measures over five times faster than the Go. In this task, the computation is trivial and the whole thing should be pretty well 100% IO-limited. Granted, the Mac I’m running this on has 32G of RAM so after the first run the code is almost certainly coming straight out of RAM cache.

But still, the data access should dominate the compute. I’ve heard plenty of talk about how fast Rust code runs, so I’d be unsurprised if its user CPU was smaller. But elapsed time!?

What Dirkjan did

I glanced over his code. Gripe: Compared to other modern languages, I find Rust suffers from a significant readability gap, and for me, that’s a big deal. But I understand the appeal of memory-safety and performance.

Anyhow, Dirkjan, with some help from others, had added a few optimizations:

  1. Changed the regex from \s+ to [ \t].

  2. The algorithm has a simple hash-map of keys to occurrence counts. I’d originally done this as just string-to-number. He made it string-to-pointer-to-number, so after you find it in the hash table, you don’t have to update the hash-map, you just increment the pointed-to value.

  3. Broke the file up into segments and read them in separate threads, in an attempt to parallelize the I/O.

I’d already thought of #3 based my intuition that this task is I/O-bound, but not #2, even though it’s obvious.

What I did

In his comment, Dirkjan pointed out correctly that I’d built my binary “naively with ‘go build’”. So I went looking for how to make builds that are super-optimized for production and (Surprise #2) there aren’t any. (There’s a way to turn optimizations off to get more deterministic behavior.) Now that I think about it I guess this is good; make all the optimizations safe and just do them, don’t make the developer ask for them.

(Alternatively, maybe there are techniques for optimizing the build, but some combination of poor documentation or me not being smart enough led to me not finding them.)

Next, I tried switching in the simpler regex, as in Dirkjan’s optimization #1.

ElapsedUserSystemvs Rust
Rust4.964.450.501.0
ls/uniq/sort142.80149.661.2928.80
topfew 1.027.2028.310.545.49
“[ \\t]”25.0326.180.53 5.05

That’s a little better. Hmm, now I’m feeling regex anxiety.

Breakthrough

At this point I fat-fingered one of the runs to select on the first rather than the seventh field and that really made topfew run a lot faster. Which strengthened my growing suspicion that I was spending a lot of my time selecting the field out of the record.

At this point I googled “golang regexp slow” and got lots of hits; there is indeed a widely expressed opinion that Go’s regexp implementation is far from the fastest. Your attitude to this probably depends on your attitude toward using regular expressions at all, particularly in performance-critical code.

So I tossed out the regexp and hand-wrangled a primitive brute-force field finder. Which by the way runs on byte-array slices rather than strings, thus skipping at least one conversion step. The code ain’t pretty but passed the unit tests and look what happened!

ElapsedUserSystemvs Rust
Rust4.964.450.501.0
ls/uniq/sort142.80149.661.2928.80
topfew 1.027.2028.310.545.49
“[ \\t]”25.0326.180.535.05
no regexp4.234.830.660.85

There are a few things to note here. Most obviously (Surprise #3), the Go implementation now has less elapsed time. So yeah, the regexp performance was sufficiently bad to make this process CPU-limited.

Less obviously, in the Rust implementation the user CPU time is less than the elapsed; user+system CPU are almost identical to elapsed, all of which suggests it’s really I/O limited. Whereas in all the different Go permutations, the user CPU exceeds the elapsed. So there’s some concurrency happening in there somewhere even though my code was all single-threaded.

I’m wondering if Go’s sequential file-reading code is doing multi-threaded buffer-shuffling voodoo. It seems highly likely, since that could explain the Go implementation’s smaller elapsed time on what seems like an I/O-limited problem.

[Update: Dirkjan, after reading this, also introduced regex avoidance, but reports that it didn’t appear to speed up the program. Interesting.]

Non-breakthroughs

At this point I was pretty happy, but not ready to give up, so I implemented Dirkjan’s optimization #2 - storing pointers to counts to allow increments without hash-table updates.

ElapsedUserSystemvs Rust
Rust4.964.450.501.0
ls/uniq/sort142.80149.661.2928.80
topfew 1.027.2028.310.545.49
“[ \\t]”25.0326.180.535.05
no regexp4.234.830.660.85
hash pointers4.165.100.650.84

Um well that wasn’t (Surprise #4) what I expected. Avoiding almost nine million hash-table updates had almost no observable effect on latency, while slightly increasing user CPU. At one level, since there’s evidence the code is limited by I/O throughput, the lack of significant change in elapsed time shouldn’t be too surprising. The increase in user CPU is though; possibly it’s just a measurement anomaly?

Well, I thought, if I’m I/O-limited in the filesystem, let’s be a little more modern and instead of reading the file, let’s just pull it into virtual memory with mmap(2). So it should be the VM paging code getting the data, and everyone knows that’s faster, right?

ElapsedUserSystemvs Rust
Rust4.964.450.501.0
ls/uniq/sort142.80149.661.2928.80
topfew 1.027.2028.310.545.49
“[ \\t]”25.0326.180.535.05
no regexp4.234.830.660.85
hash pointers4.165.100.650.84
mmap7.178.330.621.45

So by my math, that’s (Surprise #5) 72% slower. I am actually surprised, because if the paging code just calls through to the filesystem, it ought to be a pretty thin layer and not slow things down that much. I have three hypotheses: First of all, Go’s runtime maybe does some sort of super-intelligent buffer wrangling to make the important special case of sequential filesystem I/O run fast. Second, the Go mmap library I picked more or less at random is not that great. Third, the underlying MacOS mmap(2) implementation might be the choke point. More than one of these could be true.

A future research task is to spin up a muscular EC2 instance and run a few of these comparisons there to see how a more typical server-side Linux box would fare.

Parallel?

Finally, I thought about Dirkjan’s parallel I/O implementation. But I decided not to go there, for two reasons. First of all, I didn’t think it would actually be real-world useful, because it requires having a file to seek around in, and most times I’m running this kind of a pipe I’ve got a grep or something in front of it to pull out the lines I’m interested in. Second, the Rust program that does this is already acting I/O limited, while the Go program that doesn’t is getting there faster. So it seems unlikely there’s much upside.

But hey, Go makes concurrency easy, so I refactored slightly. First, I wrapped a mutex around the hash-table access. Then I put the code that extracts the key and calls the counter in a goroutine, throwing a high-fanout high-TPS concurrency problem at the Go runtime without much prior thought. Here’s what happened.

ElapsedUserSystemvs Rust
Rust4.964.450.501.0
ls/uniq/sort142.80149.661.2928.80
topfew 1.027.2028.310.545.49
“[ \\t]”25.0326.180.535.05
no regexp4.234.830.660.85
hash pointers4.165.100.650.84
goroutines!6.5357.847.681.32

(Note: I turned the mmap off before I turned the goroutines on.)

I’m going to say Surprise #6 at this point even though this was a dumbass brute-force move that I probably wouldn’t consider doing in a normal state of mind. But wow, look at that user time; my Mac claims to be an 8-core machine but I was getting more than 8 times as many CPU-seconds as clock seconds. Uh huh. Pity those poor little mutexes.

I guess a saner approach would be to call runtime.NumCPU() and limit the concurrency to that value, probably by using channels. But once again, given that this smells I/O-limited, the pain and overhead of introducing parallel processing seem unlikely to be cost-effective.

Or, maybe I’m just too lazy.

Lessons

First of all, Rust is not magically and universally faster than Go. Second, using regular expressions can lead to nasty surprises (but we already knew that, didn’t we? DIDN’T WE?!) Third, efficient file handling matters.

Thanks to Dirkjan! I’ll be interested to see if someone else finds a way to push this further forward. And I’ll report back any Linux testing results.

Oboe to Ida 26 Jun 2020, 7:00 pm

What happened was, I was listening to an (at least) fifty-year old LP, a classical collection entitled The Virtuoso Oboe; it’s what the title says. The music was nothing to write home about but led into a maze of twisty little passages that ended with me admiring the life of a glamorous Russian bisexual Jewish ballerina heiress who’s famous for… well, we’ll get to that.

The uncollected Les Brown

Regular readers know that I have a background task — working through an inherited trove of 900 mostly-classical LPs. In that piece I said I was going to blog my way through the collection but I haven’t, which is probably OK. But tonight’s an exception.

Oboe virtuosity

As the picture linked above reveals, the records are in cardboard liquor boxes, and somewhat clustered thematically. The box I’m currently traversing seems to be mostly “weird old shit, mostly bad, mostly scratchy”. It contained some malodorous “easy listening” products of the Fifties and Sixties. (I kept one, 1944-46 recordings by Les Brown’s extremely white jazz band, forgettable except it has Doris Day on vocals and let me tell ya, she sets the place on fire. But I digress.)

The Virtuoso Oboe, Vol. 1

Anyhow, last night’s handful of LPs to wash and dry and try included The Virtuoso Oboe and The Virtuoso Oboe Vol. 4, whose existence suggests the series had legs. It features that famous household name André Lardrot on, well, oboe.

The first record contained music from Cimarosa, Albinoni, Handel, and Haydn and frankly didn’t do anything for me. None of the tunes were that memorable and I didn’t find the oboe-playing compelling. So it’s going off to the Sally Ann.

I questioned whether Vol. 4 was even worth trying, but then I noticed it had a concerto by Antonio Salieri. Who’s got Salieri in their collection, I ask you? Nobody, that’s who! But now I do.

The Virtuoso Oboe, Vol. 4

The concerto is nice; if you told me it was by Mozart I’d believe you. A couple of the melodies are very strong and skip opportunities for convolutions and decorations that W. Amadeus never would have.

On the other side was a Concertino for English Horn by Donizetti. I vaguely remembered that the English Horn (a.k.a. cor anglais) is sort of like an oboe only bigger, and notably neither English nor a horn.

Oboe d’Amore

While I was reading up on that I discovered the Oboe d’amore, which is bigger than an oboe but smaller than a cor anglais. Who could resist learning more about a love oboe? Whereupon I learned that it’s perhaps most famous for leading one of the stanzas in Ravel’s Boléro. At which point I was about to stop, because what more can be said about the Boléro?

Well, lots! I’d somehow internalized it as fact that Ravel hated his most famous work and regretted writing it, but not so. He attended performances and produced a later arrangement for two pianos. He also expressed the perfectly reasonable opinion that once he’d done it, nobody else needed to write any pieces consisting of a single melodic line repeated over and over again for seventeen minutes and successively enhanced by adding more and more instruments playing louder and louder.

Ida Rubinstein

He also thought it should be played at a steady pace and got into a big public pissing match with Toscanini, who wanted to play the piece not only louder and louder but faster and faster.

I was thinking about the Oboe d’Amore and Boléro and speculated that might be a joke to be found along the lines of oBo d’Erek but you’d have to be over fifty to follow the thread and anyhow I was wrong, there isn’t a joke there.

Ida Rubinstein

Wait, there’s more! It turns out that Ravel didn’t write the Boléro just because he felt like it, but because he was paid for it; a commission from one Ida Rubinstein, mentioned at the top: dancer, Russian, heiress, bisexual, etc. The reason I wrote this was to get you to hop over to Wikipedia and read up about her life, which was extraordinary. Having done that, you can actually watch bad silent film of her dancing, someone having supplied a music track. It’s a pity there’s no video of her performing to Boléro.

My work here is done.

Break Up Google 25 Jun 2020, 7:00 pm

It’s easy to say “Break up Big Tech companies!” Depending how politics unfold, the thing might become possible, but figuring out the details will be hard. I spent the last sixteen years of my life working for Big Tech and have educated opinions on the subject. Today: Why and how we should break up Google.

egGolo

Where’s the money?

Google’s published financials are annoyingly opaque. They break out a few segments but (unlike Amazon) only by revenue, there’s nothing about profitability. Still, the distribution is interesting. I collated the percentages back to Q1 2018:

Ads on GoogleAds off GoogleAds on YouTubeCloudOther
2018 Q170.97%14.84%??14.19%
2018 Q271.69%14.77%??13.54%
2018 Q371.73%14.58%??13.69%
2018 Q469.05%14.32%??16.62%
2019 Q170.99%13.81%??14.92%
2019 Q270.36%13.66%??15.98%
2019 Q370.97%13.15%??15.88%
2019 Q459.39%13.10%10.26%5.68%11.57%
2020 Q159.76%12.68%9.76%6.83%10.73%
2020 Q256.05%12.37%10.00%7.89%13.42%

Note that they started breaking out YouTube and Cloud last year; it looks like Cloud was previously hidden in “Other” and YouTube in “Ads on Google”. I wonder what else is hidden in there?

Why break it up?

There are specific problems; I list a few below. But here’s the big one: For many years, the astonishing torrent of money thrown off by Google’s Web-search monopoly has fueled invasions of multiple other segments, enabling Google to bat aside rivals who might have brought better experiences to billions of lives.

  1. Google Apps and Google Maps are both huge presences in the tech economy. Are they paying for themselves, or are they using search advertising revenue as rocket fuel? Nobody outside Google knows.

    In particular, I’m curious about Gmail, which had 1.5B users in 2018. Some of those people see ads, but plenty don’t. So what’s going on there? It can’t be that cheap to run. Where’s the money?!

  2. The maps business, with its reviews and ads, has a built-in monopolization risk that I wrote about in 2017. It needs to be peeled off so we can think about it. We definitely want reviews and ads on maps (“Where’s the nearest walk-in clinic open on Sunday with friendly staff?”), but the potential for destructive corruption is crazy high.

    Used to be, there were multiple competing map producers and some of them were governments. The notion of mapping being a public utility (Perhaps multinational? What a concept!) with competing ad vendors running on it doesn’t sound crazy to me.

  3. The online advertising business has become a Facebook/Google duopoly which is destroying ad-supported publishing and thus intellectually impoverishing the whole population; not to mention putting a lot of fine journalists out of work. The best explanation I’ve ever read of how that works is Data Lords: The Real Story of Big Data, Facebook and the Future of News by Josh Marshall.

  4. The world needs Google Cloud to be viable because it needs more than two public-cloud providers. It’s empirically possible for such a business to make lots of money; AWS does. GCloud needs to be ripped away from its corporate parent, just as AWS does, but for different reasons.

  5. I note the pointed absence of Android in any of the financials. It’s deeply weird that the world’s most popular operating system has costs and revenues that nobody actually knows. Of course, the real reason Android exists is that Google needs mobile advertising not to become an Apple monopoly.

  6. YouTube has become the visual voice of several generations and is too important to leave hidden inside an opaque conglomerate. Is it a money-spinner or strictly a traffic play? Nobody knows. What we do know is that people who try to make a living as YouTubers sure do complain a lot about arbitrary, ham-handed changes of monetization and content policy. Simultaneously, Google’s efforts to avoid promoting creeps and malefactors aren’t working very well.

What to do?

First, spin Advertising off into its own company and then enact aggressive privacy-protection and anti-tracking law. Start by doing away with 100% of third-party cookies, no exceptions. It’d probably be OK to leave the off-site advertising in that company.

But YouTube definitely has to be its own thing; it’s got no real synergy that I can detect with any other Google property.

I’m not even sure Android is a business. Its direct cost is a few buildings full of engineers. Its revenue is (indirectly) mobile ads, plus Play Store commissions, plus Pixel sales plus, uh, well nobody knows what kinds of backroom arrangements are in place with Samsung et al. Absent the mobile ads, I doubt it’s much of a money-maker. Maybe turn it into a foundation funded by a small levy on everyone who ships an Android phone… Other ideas?

What to do with Maps isn’t obvious to me. It’s probably a big-money business (but we don’t know). In combination with the reviews capability it should be a great advertising platform, but the opportunities for corruption are so huge I’m not sure any private business could be trusted to manage them. First step: Force Google to disclose the financials.

I think Google Cloud could probably make a go of it as an indie, if only as a vendor of infrastructure to all the other ex-Google properties. And I think the product is good, although they’re running third in the public-cloud race.

To increase Google Cloud’s chances, throw in the Apps business; Microsoft classifies their equivalent as Cloud and I don’t think that’s crazy. My Microsoft-leaning friends scoff at the G Apps, but they’re just wrong; with competent administration the apps offer a slick, fast, high-productivity office-automation experience.

Finally, as a standalone company, we could hope they’d break Google’s habit of suddenly killing products heavily depended-on by customers. You just can’t do that in the Enterprise space.

The politics

So there should be at least four Google-successor organizations, each with a chance for major success.

I think this would be pretty easy to sell to the public. To start with, what’s left of the world’s press would cheerlead, eager to get out from under the thumb of the Google/Facebook ad cartel. Legions of YouTubers would march in support as well.

Financially, I think Google’s whole is worth less than the sum of its parts. So a breakup might be a win for shareholders. This is a reasonable assumption if only because the fountain of money thrown off by Web-search advertising leaves a lot of room for laziness and mistakes in other sectors of the business.

Also, it’s quite likely the ex-Googles could come out ahead on the leadership front. Larry, Sergey, and the first wave of people they hired made brilliant moves in building Web search and then the advertising business. But the leadership seems to have lost some of that golden touch; fresh blood might really help.

When?

The best time would have been sometime around 2015. The second best…

A-Cloud PR/FAQ 21 Jun 2020, 7:00 pm

I’d like to see AWS split off from the rest of Amazon and I’m pretty sure I’m not alone. So to help that happen, I’ve drafted a PR/FAQ and posted it on GitHub so that it can be improved. People who know what a PR/FAQ is and why this might be helpful can hop on over and critique the doc. For the rest, herewith background and explanation on the what, why, and how.

A project needs a name and so does a company. For the purposes of this draft I’m using “A-Cloud” for both.

Why spin off AWS?

The beating of the antitrust drums is getting pretty loud across a widening swathe of political and economic conversations. The antitrust guns are particularly aimed at Big Tech. Whom I’m not convinced are the most egregious monopolists out there (consider beer, high-speed Internet, and eyeglasses) but they’re maybe the richest and touch more people’s lives more directly.

So Amazon might prefer to spin off AWS proactively, as opposed to under hostile pressure from Washington.

But that’s not the only reason to do it. The cover story in last week’s Economist, Can Amazon keep growing like a youthful startup? spilled plenty of ink on the prospects for an AWS future outside of Amazon. I’ll excerpt one paragraph:

AWS has the resources to defend its market-leading position. But in the cloud wars any handicap could cost it dearly. Its parent may be becoming one such drag. For years being part of Amazon was a huge advantage for AWS, says Heath Terry of Goldman Sachs, a bank. It needed cash from the rest of the group, as well as technology and data. But Mr Bezos’s habit of moving into new industries means that there are now ever more rivals leery of giving their data to it. Potential customers worry that buying services from AWS is tantamount to paying a land-grabber to invade your ranch. Walmart has told its tech suppliers to steer clear of AWS. Boards of firms in industries which Amazon may eye next have directed their it departments “to avoid the use of AWS where possible”, according to Gartner.

The Economist plausibly suggests a valuation of $500B for AWS. Anyhow, the spin-off feels like a complete no-brainer to me.

Now, everyone knows that at Amazon, when you want to drive a serious decision, someone needs to write, polish, and bring forward a six-pager, usually a PR/FAQ. So I started that process.

What’s a PR/FAQ?

It’s a document, six pages plus appendices, that is the most common tool used at Amazon to support making important decisions. There’s no need for me to explain why this is works so well; Brad Porter did a stellar job back in 2015. More recently, Robert Munro went into a little more detail.

On GitHub?

One thing neither of those write-ups really emphasize is that six-pagers in general, and PR/FAQs in particular, are collaborative documents. The ones that matter have input from many people and, by the time you get to the Big Read in the Big Room with the Big Boss, have had a whole lot of revisions. That’s why I put this one on GitHub.

I feel reasonably competent at this, because there are now several successful AWS services in production where I was an initial author or significant contributor to the PR/FAQ. But I know two things: First, there are lots of people out there who are more accomplished than me. A lot of them work at Amazon and have “Product Manager” in their title. Second, these docs get better with input from multiple smart people.

So if you feel qualified, think AWS should be spun out, and would like to improve the document, please fire away. The most obvious way would be with a pull request or new issue, but that requires a GitHub account, which probably has your name on it. If you don’t feel comfortable contributing in public to this project, you could make a burner GitHub account. Or if you really don’t want to, email me diffs or suggestions. But seriously, anyone who’s qualified to improve this should be able to wrangle a pull request or file an issue.

What’s needed?

Since only one human has read this so far, there are guaranteed to be infelicities and generally dumb shit. In particular, the FAQ is not nearly big enough; there need to be more really hard questions with really good answers.

Also, appendices are needed. The most important one would be “Appendix C”, mentioned in the draft but not yet drafted. It covers the projected financial effects of spinning out A-Cloud. I’d love it if someone financially savvy would wrap something around plausible numbers.

Other likely appendices would be a description of the A-Cloud financial transaction, and (especially) a go-to-market plan for the new corporation: How is this presented to the world in a way that makes sense and is compelling?

Document read?

Before these things end up in front of Andy Jassy, there are typically a few preparatory document reads where intermediate-level leaders and related parties get a chance to spend 20-30 minutes reading the document then chime in with comments.

Given enough requests, I’d be happy to organize such a thing and host it on (of course) Amazon Chime, which really isn’t bad. Say so if you think so.

AWS’s Share of Amazon’s Profit 14 Jun 2020, 7:00 pm

I’ve often heard it said that AWS is a major contributor to Amazon’s profitability. How major? Here’s a graph covering the last ten quarters, with a brief methodology discussion:

AWS Proportion of Amazon Profit, Quarterly since 2018 Q1

I think this is a useful number to observe as time goes by, and it would be good if we generally agree on how to compute it. So here’s how I did it.

Start at the Quarterly Results page at ir.amazon.com. It only has data back through 2018 which seems like enough history for this purpose. Let’s take a walk through, for example, the Q1 2020 numbers.

Please note: I have zero inside knowledge about the facts behind these numbers. Every word here is based on publicly available data.

It’s easy to identify the AWS number; search in the quarterly for “Segment Information” and there it is: Net sales $10,219M, Operating expenses $7,144M, Operating Income $3,075M. So if we want to know the proportion of Amazon profits that are due to AWS, that’d be the numerator.

So what’s the denominator? Since the AWS results exclude tax, so should the Amazon number. Search for “Consolidated Statements of Operations” and there are two candidates: “Operating Income” and “Income before income taxes”. The first has the same name as the AWS line, so it’s a plausible candidate. But it excludes net interest charges and $239M of unspecified “Other income”.

This is a little weird, since interest is a regular cost of doing business. It might be the case that the interest expense is mostly due to financing computers and data centers for AWS, or alternatively that it’s mostly financing new fulfillment centers and trucks and planes and bulk buys of toilet paper. And then there’s the “Other income”.

If we assume that the “Other income” and net interest gain/loss is distributed evenly across AWS and the rest of Amazon, then you should use the “Operating” line. I’m not 100% comfy with that assertion, so I made a graph with both lines, and I think it’s likely the best estimate of AWS’s proportion of Amazon’s income falls in the space between them.

So, if this graph is right, then over the last couple of years, somewhere between 50% and 80% of Amazon’s profit has been due to AWS. Glancing at the graph, you might think the trend is up, but that verges on extrapolation, I’d want to wait a couple more quarters before having an opinion.

Since I’m not a finance professional, I may have gone off the rails in some stupidly obvious way. So here’s the .xlsx I used to generate the graph. I used Google Sheets, so I hope it doesn’t break your Excel.

Please let me know if you find any problems.

Anti-Monopoly Thinking 8 Jun 2020, 7:00 pm

I recently read The Myth of Capitalism: Monopolies and the Death of Competition by Jonathan Tepper and Denise Hearn. For a decade or two now I’ve felt the economy is over-concentrated and in recent years I’ve been super-concerned about concentration of market power in the Big-Tech sector where I’ve earned my living. Reading this book has reinforced that concern and been very helpful in introducing new angles on how to think about monopoly. The issue is central to the travails and triumphs of 21st-century Capitalism and if you care about that, you might want to read it too.

The Myth of Capitalism

Where I stand

I believe that de-monopolization is a necessary but not sufficient condition for getting us out of our current socioeconomic turmoil. In particular, I’d go after Big Tech. The functions that are provided by Amazon, Apple, Facebook, Google, and Microsoft should be provided by at least twenty different companies.

I hope to dig into a few of them, with specifics, in a series of blog posts, given time and energy.

A few juicy outtakes

I don’t necessarily agree 100% with all of these, but they are thought-provoking.

“X companies control Y% of the US market in Z.”

  • X=2, Y=90, Z=beer

  • X=4, Y=almost all, Z=airlines

  • X=5, Y=50, Z=banks

  • X=2, Y=90, Z=health insurers in many states

  • X=1, Y=75%, Z=fast Internet, most places in the US

  • X=3, Y=70, Z=pesticides

  • X=3, Y=80, Z=seed corn.

“[Warren Buffet,] when asked at an annual meeting what his ideal business was, he argued it was one that had ‘High pricing power, a monopoly.’”

“Given the lack of any new entrants into most industries, it should be no surprise that companies are getting larger and older. The average age of public companies in the United States is currently 18 years old, up from 12 years old in 1996. In real terms, the average company in the economy has become three times larger during the past two decades.25 Not only do we have fewer, older companies, but they are also capturing almost all the profits. In 1995 the top 100 companies accounted for 53% of all income from publicly traded firms, but by 2015, they captured a whopping 84% of all profits.26 Like Oliver Twist asking for more, there is little left for smaller companies after the big ones eat their fill.”

“The tech giants love startups, but in the same way that lions love feasting on lifeless carcasses of gazelles.”

“Since Reagan, no president has enforced the spirit or the letter of the Sherman and Clayton Acts. It doesn't matter what party has controlled Congress or the presidency, there has been no difference in policy toward industrial concentration. In fact, the budgets for antitrust enforcement have steadily shrunk with each passing president.”

“High markups matter a great deal in the inequality debate because they are tightly correlated with lower wages for workers.”

“Monopolies – not big businesses – are the enemy of competition. Big is neither beautiful nor ugly.”

“The easiest rule of thumb is that industries of fewer than six players should not be allowed to merge.”

Useful: Numbers and stories

In the first part of the book Tepper and Hearn establish that the economy is over-concentrated, demonstrating it with lots of facts and figures. There is a wealth of anecodotal case studies illustrating the practical ill-effects of this concentration.

Useful: Big-tech drill-down

They dive especially deep on concentration in Big Tech: Amazon’s retail, Google’s search, Facebook’s demographic sorting. Little of this was news to me, but for those who haven’t spent decades in the tech trenches, or who harbor any remaining illusions that this business is on the side of the little guy, the stories are valuable.

Useful: The German narrative

After WWII, it turns out that the Allies felt the massive pre-war concentration of the German economy had been a significant factor in the rise of the Nazis. So in the reconstruction of Germany, they made a deliberate attempt to keep that sort of monopolization from happening again. I don’t know if they were right about the rise of the Nazis, but history seems to have shown them right about designing economies; this economic structure has served Germany very well.

Useful: Post-consumerism

In recent times, the US has as a matter of policy gated all antitrust litigation on a single factor: Are consumer prices raised unduly? Tepper and Hearn argue convincingly that this is wrong on many levels. One of them is ethical: Humans are more than just consumers, and undue monopolization damages the human experience of life along many non-consumer axes.

Boring: Patents and trademarks

Yes, the modern IP regime is damaging and dysfunctional, but we already knew that, and this is not really essential to understanding monopolization. This chapter should just be dropped.

Useful: Regulation and lobbying

Most progressives, including me, are big fans of public regulation of businesses. Unfortunately, there is a strong case to be made that monopolies use regulatory arbitrage to crush potential competition and retain their steely grip. This is facilitated by massive lobbying efforts, the de-facto corruption that infests more or less all the capitals of the developed world.

Not only can you lobby for a regulatory structure that favors your company’s position, at some level a high volume of regulation becomes a qualitative advantage for big companies against small ones, because they can afford to hire the lawyers and compliance people to play the game, while startups can’t.

Useful: Cross-shareholding

When a small group of investors are investors in all the leading players in any given industry, this increases everyone’s incentive to carve the space up in a nice little orderly monopoly without any of that nasty profit-destroying competition. Evidence is not scarce.

Wrong: On Piketty

Tepper and Hearn go on a side-trip to explain why Thomas Piketty was wrong, but bizarrely begin by asserting “We have yet to meet anyone who has read the entire book. We're not making that up. Professor Jordan Ellenberg, a mathematics professor, did a study of bookmarks on Kindle e-books and found that almost no one made it past 26 pages in Piketty's book.” Um, wrong, I did. And, with the exception of the final section where Piketty introduces his proposed solutions to the problem, it’s not even heavy going.

I assume Tepper and Hearn’s “anyone” includes Tepper and Hearn, so I think it’s fair to pretty well blow off their discursion on growth and inequality, since to begin with their assertion about what Piketty claims is nowhere near what I encountered when I read it. Anyhow, no biggie, it’s just one little section. And then they go on to agree with Piketty’s finding that inequality has empirically been increasing continuously and strongly, and provide a bunch of useful data graphs.

Useful: Recommendations

Tepper and Hearn finish up with a series of how-do-we-fix-this recommendations. I’m not going to copy that many of them in here because if you care, you should read their damn book already.

But they make a crucial meta-point: Anti-monopoly regulations need to be clear and simple, and based on principles, as opposed to complex rules.

Next steps

Simple: Let’s start by breaking up a few banks and big-techs and agribusiness empires. This book probably isn’t the reason why, but public perception has swung strongly against The BigCos. At this point in history they’re soft targets and the rewards to all of us from doing this are potentially huge. (By the way, this includes their shareholders.)

The best time to have done this would have been ten years ago. The next best time is now.

At CRAB Park 31 May 2020, 7:00 pm

On an errand, we felt the need to be outside near the sea. The closest opportunity was CRAB Park (see here). We hadn’t planned on visiting a homeless encampment but did that too. I seriously recommend the experience. Also I got pleasant pictures to accompany the story.

CRAB is the nearest green waterfront to Vancouver’s Downtown Eastside which means some of the people you see have been considerably damaged by what we call “civilization”, and show it. it’s also real close to the Port of Vancouver, third-largest in North America and biggest in Canada (only 29th in the world). Thus these views:

Vancouver Port from CRAB park

Some parts of the port seem remarkably ad-hoc.

Marine engineering near CRAB park

Homelessness

I saw a cluster of tents in an adjoining parking lot. In Vancouver that means homeless people, which is a little scary to a nice-neighborhood greybeard like me even without Covid-19 maybe lurking where you’re not looking. But this one had a fire burning and signs on display. Feeling an open-for-visitors vibe, I wandered in.

The signs said “reconciliation is dead”. The fire was cedar, an Indigenous woman by it explaining the issues around unceded territory to a typically-Vancouver White/Asian couple.

A tiny dog ran up to me, eager for a sniff at my hand. After we’d made friends a nice lady came up, tucked the dog into her shirt, and told me a whole lot about what was going on. She started by saying I could make an offering at their fire with tobacco, which was provided, so I did. The cedar-and-tobacco smoke smelt great. Here’s the bench by the fire.

At the CRAB park homeless encampment.

I wish I’d asked her name. I make no claim to a deeper understanding than anyone reading this, so I’m just going to enumerate what I heard as I listened.

  1. Sometime in 2019 the BC government transferred the homeless file from a social-services ministry to the law-and-order ministry, and this has not been helpful.

  2. The encampments have inclusion levels depending, among other things, on their tolerance for alcohol or other intoxicants. I didn’t learn the level of the one I was at.

  3. A couple of rope-thin white guys were breaking up skinny pieces of metal. It seemed like hard work but they were cracking jokes. I don’t know what they were doing with them.

  4. People die in these communities, but rarely. An occasional overdose and recently a 51-year old who went to sleep and didn’t wake up. “People without a home have a life expectancy roughly half of people who do”, she pointed out.

  5. There are a lot of indigenous people here, but also a lot who aren’t.

  6. A guy walked up smiling with a bunch of plastic on a dolly and told her “Three tarps, they wanted sixty bucks and I had seventy, so I threw in an extra ten. Now you owe me $10.” She said “I’ll take care of it.” Still smiling: “S’ok, you don’t need to.”

  7. The provincial government is trying to get the homeless out of tents and into hotels. But they want to take the 65-plus and otherwise-vulnerable first. Then, they need an actual legal name; but suppose you have mental-health and/or legal issues (for both reasons, you might not be in a position to offer that kind of name) and/or you aren’t 65 yet? Well, you’re probably in a tent at CRAB park or equivalent.

  8. I couldn’t help but be conscious of the big black SUV labeled “Port of Vancouver Police” (I may have the wording wrong) at the far end of the parking lot, a silhouette visible in the front seat. Since the port is well-known to be gang-infested, you have to worry about what their cops are like.

  9. Some of the people have dementia (not always age-related). They do the best they can.

  10. There were canvas rain-covers set up with cases of fresh-water bottles available and other stuff I didn’t check out. The Portland Hotel Society helps out, I gather, as do other community members. I think I’m going to have to visit and drop some things off. But if it’s like most distressed-community situations, money is most helpful.

Walking away

Suddenly one of Vancouver’s biggest problems had moved out of the “abstract” category in my mind. I found I had trouble speaking because I didn’t know how to process the input.

I could still take pictures though, where the park met the port.

Near CRAB Park in VancouverNear CRAB Park in Vancouver

If you’re a photographer, or just want to get closer to the edge of your comfort zone and meet people who could use your help, I strongly recommend a visit to CRAB park.

Topfew Fun 18 May 2020, 7:00 pm

This was a long weekend in Canada; since I’m unemployed and have no workaday cares, I should have plenty of time to do family stuff. And I did. But I also got interested in a small programming problem and, over the course of the weekend, built a tiny tool called topfew. It does a thing you can already do, only faster, which is what I wanted. But I remain puzzled.

[To those who are interested in Amazon or politics or photography, feel free to skip this. I’ll come back to those things.]

What it does

Back in 2016 I tweeted Hello "sort | uniq -c | sort -rn | head" my old friend, and got a few smiles. If you feed that incantation a large number of lines of text it produces a short list showing which among them appear most often, and how many times each of those appears.

It’s a beautiful expression of the old Unix craft of building simple tools that you can combine to produce surprising and useful results. But when the data gets big, it can also be damn slow. I have a little script on the ongoing Web server that uses this technique to tell me which of my pieces are most popular, and how many times each of those was read.

In the last couple of weeks, my logfiles have gotten really big, and so my script has gotten really slow. As in, it takes the best part of a minute to run and, while it’s running, it sucks the life out that poor little machine.

It turns out to be a pretty common problem. Anyone who’s operating an online service finds themselves processing logfiles to diagnose problems or weirdness, and you regularly need to answer questions like “who are the top requestors?” and “what are the most common API calls?”

What it is

Topfew is a simple-minded little Go program, less than 500 lines of code and almost half of that tests. If you have an Apache server log named access_log, and you wanted to find out what the top twelve IP addresses were hitting your server hardest, you could say:

awk '{print $1}' access_log | sort | uniq -c | sort -rn | head -12

With topfew, you’d say:

tf -fields 1 -few 12 access_log (“tf” is short for “topfew”.)

The two would produce almost the same output, except for topfew would be faster. How much faster? Well, that’s complicated (see below)… but really quite a bit.

I don’t think there’s anything terribly clever about it; My bet is that if you asked a dozen senior developers to write this, they’d all end up with nearly identical approaches, and they’d all be pretty fast. Anyhow, as the links here make obvious, it’s on GitHub. If anyone out there needs such a thing and hasn’t written their own in a weekend project, feel free to use and improve.

Since I haven’t done a Go project from scratch in a long time, I’m not only out of practice but out of touch with current best practices, so it’s probably egregiously unidiomatic in lots of ways. Oh well.

How fast?

Well, that’s interesting. I ran a few tests on my Mac and it was many many times faster than the sort/uniq incantation, so I smiled. I thought “Let’s try bigger data!” and built a Linux version (saying GOOS=linux go build still feels like living in the future to me), hopped over to my Web server, and ran exactly those queries from the last section on a recent Apache logfile (8,699,657 lines, 2.18G). Topfew was faster, but not many times faster like on my Mac. So I ran it on the Mac too. Here’s a summary of the results. All the times are in seconds.

cp
time
sort/uniq
time
topfew
time
speedup
net of I/O
Linux17.2937.6626.752.15x
Mac1.40137.468.8118.36x

The first column (“cp time”) is the number of seconds it takes to copy the big file from one directory to another. The last column is produced by subtracting the “cp” time from each of the other two, leaving something like the amount of extra latency introduced by the computing in excess of the time spent just copying data, then comparing the results.

Anyhow, those are some puzzling numbers. Stray thoughts and observations that I’m not going to dignify with the term “hypotheses”:

  1. The Mac has a damn fast SSD.

  2. The Linux box’s disk is sluggish.

  3. Perhaps the basic tools like sort and uniq are surprisingly fast on Linux or the Go runtime is surprisingly slow. Or, you know, the mirror image of that statement.

  4. The sort/uniq incantation is inherently parallel while topfew is 100% single-threaded. Maybe the Mac is dumb about scheduling processes in a shell pipeline across its eight cores?

Giggling

I broke out the Go profiler (Go is batteries-included in ways that other popular languages just aren’t) and ran that command against the Apache logfile again. I was poking around the output and Lauren said “I hope you’re laughing at the computer, not at me talking to the cat.”

The profiler claimed that 73% of the time was spent in syscall.syscall reading the input data, and the first trace of actual application code was 2.27% of the time being spent in the regexp library doing field splitting (on \s+).

The core code that actually accumulates counts for keys was consuming 1.54% of the elapsed time, and 93% of that was updating the total in a map[string]uint64.

Next steps?

I’m probably curious enough to poke around a little more. And maybe someone reading this will kindly explain to me in words of one syllable what’s going on here. But anyhow, the tool is useful enough as it is to save me a noticeable amount of time.

Meta Blog 13 May 2020, 7:00 pm

My recent Amazon-exit piece got an order of magnitude more traffic then even the post popular outings here normally do. Which turned my mind to thoughts of blogging in 2020, the why and how of the thing. Here they are, along with hit-counts and referer data from last week. Probably skip this unless you’re interested in social-media dynamics and/or publishing technology.

Numbers

In the first week after publication, Bye, Amazon was read somewhat more than 614,669 times by a human (notes on the uncertainty below).

The referers with more than 1% of that total were Twitter (18.13%), Hacker News (17.09%), Facebook (10.55%), Google (4.29%), Vice (3.40%), Reddit(1.66%), and CNBC (1.09%). I think Vice was helped by getting there first. I genuinely honestly have no idea how the piece got mainlined so fast into the media’s arteries.

For comparison, my first Twitter post is up to 1.015M impressions as of now and, last time I checked, Emily’s was quite a ways ahead.

It’s hard for me to know exactly how many people actually read any one ongoing piece because I publish a full-content RSS/Atom feed. Last time I checked, I estimated 20K or so subscribers, but nobody knows how many actually read any given piece. If I cared, I’d put in some kind of a tracker. That 614K number above comes from a script that reads the log and counts the number of fetches executed by a little JavaScript fragment included in each page. Not a perfect measure of human-with-a-browser visits but not terrible.

But aren’t blogs dead?

Um, nope. For every discipline-with-depth that I care about (software/Internet, politics, energy economics, physics), if you want to find out what’s happening and you want to find out from first-person practitioners, you end up reading a blog.

They’re pretty hard to monetize, which means that the people who write them usually aren’t primarily bloggers, they’re primarily professional economists or physicists or oil analysts or Internet geeks. Since most of us don’t even try to monetize ’em, they’re pretty ad-free and thus a snappy reading experience.

Dense information from real experts, delivered fast. Why would you want any other kind?

Static FTW

ongoing ran slow but anyone who was willing to wait fifteen seconds or so got to read that blog. One reason is that the site is “static” which is to say all the payload is in a bunch of ordinary files in ordinary directories on a Linux box, so the web server just has to read ’em out of the disk (where by “disk” I mean in-memory filesystem cache when things are running hot) and push ’em out over the wire. (The bits and pieces are described and linked to from the Colophon.)

It turns out that at the very hottest moments, the Linux box never got much above 10% CPU, but the 100M virt NIC was saturated.

I’ve never regretted writing my own blogging system from the ground up. I’m pretty sure I’ve learned things about the patterns of traffic and attention on the Internet that I couldn’t have learned any other way.

If I were going to rewrite this, since everything’s static, I’d just run it out of an S3 bucket and move the publishing script into a Lambda function. It’d be absurdly cheap and it’d laugh at blog storms like last week’s.

It’s not a top priority.

Responses 6 May 2020, 7:00 pm

Boy, when your I’m-outta-here essay goes viral, do you ever get a lot of input. A few responses came up often enough to be worth sharing. This was via email, Twitter DMs, blog comments, and LinkedIn messages. All of which went completely batshit.

So, some answers. But first…

Thanks for the kind words!

I had no notion how the world might react to a cranky old overpaid geek’s public temper tantrum. The world’s been astonishingly warm and welcoming. Apparently I hit a hot button I didn’t know existed. The crankiest geek on the planet couldn’t fail to have their heart warmed. So in a huge number of cases, simply “Thanks for the kind words” was the right thing to say.

Dear readers: Yes, some of the answers described in this piece were kept handy in editor buffers and delivered by cut-and-paste. But I did read a lot of the messages, and all the ones I actually responded to.

“Can you come on our TV show?”

You name it: ABC, BBC, Bloomberg, CBC, CBS, CTV, CNBC, CNN, NBC, NPR, and a whole lot of cool blogs and indies. Also Anderson Cooper’s people!

“Can we get on the phone so I can ask you some questions?”

A variation on the theme, from non-broadcast organizations.

They all got the same answer: “Hmm, I'm not that interesting, just a grumpy old one-percenter white-guy engineer with a social media presence. If you want to go live with this story you should do it with the actual people who got fired, who are young, fresh-faced, passionate, and really at the center of the news story. I recommend reaching out to Emily Cunningham (contact info redacted) or Maren Costa (same) or Chris Smalls (same).”

“OK, we talked to them. Now will you talk to us live?”

These people were nice and just trying to do their job. I agreed to answer a couple of email questions in a couple of cases, but mostly just said “For complicated reasons, I don’t want to be the public face of this story. Sorry.”

“Complicated reasons?”

Yeah, the story is about firing whistleblowers, not about a random Canadian Distinguished Engineer’s reaction to it. So news organizations should follow the primary sources, not me.

There’s more. I put a lot of thought into what I should say, and then really a whole lot of work into writing that blog piece. I had help with style and fact-checking. (Thanks, Lauren. Thanks, Emily.) It is very, very, unlikely that anything I’d improvise on a phone-call or TV interview would be better. I’ve also had bitter first-hand experience with the Gell-Mann amnesia effect.

I think it worked. The news coverage, lacking alternatives, quoted heavily from the blog, and I thought basically all of it came out fair and accurate. Let’s acknowledge that this tactic is only available to someone who’s near the center of a news story, is a competent writer, and has a good place to publish.

I’ve no interest in becoming some sort of full-time anti-Amazon activist. I just don’t want to work for an organization that fires whistleblowers. I said so. It looks like the message got through.

“What response did you expect?”

I have seventeen years of blogging scars, so I speak from experience in saying: No idea. I’ve had blogs that I considered mightily important and labored over for days sink like a stone with no trace. Then I’ll toss off some three-paragraph squib that I wrote while watching TV and drinking gin, and it goes white-hot. Neither outcome would have surprised me.

“What were you trying to accomplish?”

I’m a blogger. I’ve been writing the story of my life here for seventeen years. Enough people read it and respond to give me hope that it’s at least intermittently interesting, and perhaps even useful. I’m a writer, I can’t not write. This is a major turning point in my life. I totally couldn’t not write it. That’s all. That’s really, really why.

“What about Brad’s piece?”

They’re asking about Response to Tim Bray’s departure by Brad Porter. Since he has the same “VP/Distinguished Engineer” title I did you’d think he’d be a peer. Actually he’s way more important and influential than I used to be, partly because he’s been there since the early days and is directly involved with the retail operation.

I believe that (as Brad says) Amazon retail is working intensely and intelligently to make the warehouses safer. But I also believe the workers. And anyhow, firing whistleblowers is just way, way out of bounds.

While it’s sort of a sideshow to the real issue here (firing whistleblowers), Brad also wrote:

Ultimately though, Tim Bray is simply wrong when he says “It’s that Amazon treats the humans in the warehouses as fungible units of pick-and-pack potential.” I find that deeply offensive to the core.

We’ll have to agree to disagree. If you run an organization with hundreds of thousands of line workers and tens of thousands of managers, and where turnover is typically significant, you need processes where the staff are fungible. Two things can be true at once: You work hard to preserve your employees’ health, and your administrative culture treats them as fungible units.

I actually found the patterns emerging in the responses to Brad’s piece more culturally interesting than his original post.

And hey, bonus: There’s another Amazon voice in the conversation: Anton Okmyanskiy, who’s a “Principal Engineer”, which is to say regarded as a world-class technologist, wrote Tim Bray quit Amazon. My thoughts.... It contains the remarkable sentence “Amazon should stay ahead of anger-driven regulatory enforcement by becoming a leader on social justice issues.”

It’d be great if a few more Amazonian voices weighed in. But I’m not holding my breath.

“Any regrets?”

Yes, I regret intensely that I didn’t link to Emily Cunningham’s original “Amazon fired me” tweet thread, which is exquisite (you have to click “show this thread”).

Favorite response?

Note, header not in quotes because nobody asked, but I’ll answer anyhow. I could drop a dozen portentous media-heavyweight names and yeah, pretty well everyone weighed in. But it’s not close, my fave was Wonkette: Amazon VP VIP Tim Bray Quitfires Self Over 'Chickensh*t' Activist Quitfirings. It says, of yr humble scrivener, “Come the revolution, let's remember not to eat this one” and “Class Traitor of the Day”. These lodged in what I thought was a thoroughly lucid and spirited take on the situation.

Once again

Thanks for the kind words!

Bye, Amazon 29 Apr 2020, 7:00 pm

May 1st was my last day as a VP and Distinguished Engineer at Amazon Web Services, after five years and five months of rewarding fun. I quit in dismay at Amazon firing whistleblowers who were making noise about warehouse employees frightened of Covid-19.

What with big-tech salaries and share vestings, this will probably cost me over a million (pre-tax) dollars, not to mention the best job I’ve ever had, working with awfully good people. So I’m pretty blue.

What happened

Last year, Amazonians on the tech side banded together as Amazon Employees for Climate Justice (AECJ), first coming to the world’s notice with an open letter promoting a shareholders’ resolution calling for dramatic action and leadership from Amazon on the global climate emergency. I was one of its 8,702 signatories.

While the resolution got a lot of votes, it didn’t pass. Four months later, 3,000 Amazon tech workers from around the world joined in the Global Climate Strike walkout. The day before the walkout, Amazon announced a large-scale plan aimed at making the company part of the climate-crisis solution. It’s not as though the activists were acknowledged by their employer for being forward-thinking; in fact, leaders were threatened with dismissal.

Fast-forward to the Covid-19 era. Stories surfaced of unrest in Amazon warehouses, workers raising alarms about being uninformed, unprotected, and frightened. Official statements claimed every possible safety precaution was being taken. Then a worker organizing for better safety conditions was fired, and brutally insensitive remarks appeared in leaked executive meeting notes where the focus was on defending Amazon “talking points”.

Warehouse workers reached out to AECJ for support. They responded by internally promoting a petition and organizing a video call for Thursday April 16 featuring warehouse workers from around the world, with guest activist Naomi Klein. An announcement sent to internal mailing lists on Friday April 10th was apparently the flashpoint. Emily Cunningham and Maren Costa, two visible AECJ leaders, were fired on the spot that day. The justifications were laughable; it was clear to any reasonable observer that they were turfed for whistleblowing.

Management could have objected to the event, or demanded that outsiders be excluded, or that leadership be represented, or any number of other things; there was plenty of time. Instead, they just fired the activists.

Snap!

At that point I snapped. VPs shouldn’t go publicly rogue, so I escalated through the proper channels and by the book. I’m not at liberty to disclose those discussions, but I made many of the arguments appearing in this essay. I think I made them to the appropriate people.

That done, remaining an Amazon VP would have meant, in effect, signing off on actions I despised. So I resigned.

The victims weren’t abstract entities but real people; here are some of their names: Courtney Bowden, Gerald Bryson, Maren Costa, Emily Cunningham, Bashir Mohammed, and Chris Smalls.

I’m sure it’s a coincidence that every one of them is a person of color, a woman, or both. Right?

Let’s give one of those names a voice. Bashir Mohamed said “They fired me to make others scared.” Do you disagree?

[There used to be a list of adjectives here, but voices I respect told me it was mean-spirited and I decided it didn’t add anything so I took it out.]

What about the warehouses?

It’s a matter of fact that workers are saying they’re at risk in the warehouses. I don’t think the media’s done a terribly good job of telling their stories. I went to the video chat that got Maren and Emily fired, and found listening to them moving. You can listen too if you’d like. Up on YouTube is another full-day videochat; it’s nine hours long, but there’s a table of contents, you can decide whether you want to hear people from Poland, Germany, France, or multiple places in the USA. Here’s more reportage from the NY Times.

It’s not just workers who are upset. Here are Attorneys-general from 14 states speaking out. Here’s the New York State Attorney-general with more detailed complaints. Here’s Amazon losing in French courts, twice.

On the other hand, Amazon’s messaging has been urgent that they are prioritizing this issue and putting massive efforts into warehouse safety. I actually believe this: I have heard detailed descriptions from people I trust of the intense work and huge investments. Good for them; and let’s grant that you don’t turn a supertanker on a dime.

But I believe the worker testimony too. And at the end of the day, the big problem isn’t the specifics of Covid-19 response. It’s that Amazon treats the humans in the warehouses as fungible units of pick-and-pack potential. Only that’s not just Amazon, it’s how 21st-century capitalism is done.

Amazon is exceptionally well-managed and has demonstrated great skill at spotting opportunities and building repeatable processes for exploiting them. It has a corresponding lack of vision about the human costs of the relentless growth and accumulation of wealth and power. If we don’t like certain things Amazon is doing, we need to put legal guardrails in place to stop those things. We don’t need to invent anything new; a combination of antitrust and living-wage and worker-empowerment legislation, rigorously enforced, offers a clear path forward.

Don’t say it can’t be done, because France is doing it.

Poison

Firing whistleblowers isn’t just a side-effect of macroeconomic forces, nor is it intrinsic to the function of free markets. It’s evidence of a vein of toxicity running through the company culture. I choose neither to serve nor drink that poison.

What about AWS?

Amazon Web Services (the “Cloud Computing” arm of the company), where I worked, is a different story. It treats its workers humanely, strives for work/life balance, struggles to move the diversity needle (and mostly fails, but so does everyone else), and is by and large an ethical organization. I genuinely admire its leadership.

Of course, its workers have power. The average pay is very high, and anyone who’s unhappy can walk across the street and get another job paying the same or better.

Spot a pattern?

At the end of the day, it’s all about power balances. The warehouse workers are weak and getting weaker, what with mass unemployment and (in the US) job-linked health insurance. So they’re gonna get treated like crap, because capitalism. Any plausible solution has to start with increasing their collective strength.

What’s next?

For me? I don’t know, genuinely haven’t taken time to think about it. I’m sad, but I’m breathing more freely.

Page processed in 2.342 seconds.

Powered by SimplePie 1.3, Build 20150601032828. Run the SimplePie Compatibility Test. SimplePie is © 2004–2020, Ryan Parman and Geoffrey Sneddon, and licensed under the BSD License.