Details Matter, or: Before the Storm

David Smith, ,

I know I’m a few years late on this, but I have a complaint about the ending of “Life is Strange: Before the Storm“. (I was replaying the original, and realized I’d never played this prequel, as part of getting hype for next month’s “LIfe is Strange: Double Exposure.” Yes, I pre-ordered it, including the ridiculous overpriced bonus cat content in the Ultra Deluxe Hyper Fighting Edition.) Needless to say, there will be spoilers…

Continue reading “Details Matter, or: Before the Storm” →
Details Matter, or: Before the Storm

Mini-Rant: TicketMaster and AEW

David Smith,

Look at this graphic. LOOK AT IT. It is a thing of beauty.

AEW advert for Will Ospreay vs Bryan Danielson

I’ll be there, but I’m still slightly grumpy, and that’s because of this picture that is somewhat less attractive:

Ticketmaster Logo

I get more than enough email already, so I wasn’t on any AEW mailing lists. Immediately (like, within five minutes of my boy Avery letting me know this was happening) signed up for several of their spam-o-mats. They sent out presale codes… a few hours after the posted “this is when the presale starts” time. By which time, bots had guessed all the codes and bought out the floor and most of the first tier seats worth having.

(Bought a second-level ticket anyway, because Chaifetz is not that big. Second-level was pretty good for Nine Inch Nails, but to be fair their stage is somewhat larger than a wrestling ring…)

That was Wednesday. Here we are on Saturday, looking for an upgrade, probably from one of the aforementioned bots, and saw this:

First row of the first tier! Still in prime “get your dumb sign on PPV and eventually on Botchamania” territory! And Ticketmaster won’t let me buy it, because I don’t want to buy two seats, and the computer won’t let you leave a single seat all by itself.

Ticketmaster’s continuing inability to run a competent business, yet somehow maintain its near-monopoly position on ticketing, will never not amaze and horrify me.

Mini-Rant: TicketMaster and AEW

You Pull and I Shall Push

David Smith, , ,

Server configuration goes both ways…

From the “new” World of Darkness rulebook.

For most of the past decade, I did my server management primarily with Puppet. Servers being orchestrated by Puppet work on a “pull” model — they periodically contact a central server (colloquially called the puppetmaster), and request info on how they should be configured.

The new employer does things very differently — they primarily use Ansible, which works on a “push” model. When you need to make a configuration change, you actively tell the servers what changes to make and what actions to take.

Is one right and the other wrong? Of course not. But they’re very different, both have ups and downs.

The “pull” model (as Puppet implements it) is pretty much automatic. The Puppet agent phones home on a schedule, and downloads updated configs. You don’t have to think too much about it, just set and forget. That also can be a downside, in that changes will be picked up pretty quickly by your servers. A mistake could propagate pretty quickly. (There are ways to mitigate this, but if you’re new to Puppet these pitfalls and their workarounds may not be obvious.) I’ll admit that I got spoiled by the notion of letting things happen in the background, and just checking in on them later.

Ansible’s “push” model means that changes only go to the specific devices to which you send them. This is done by way of playbooks, files that specify your desired configuration. If you only want to make a change on a subset of servers, it’s a bit easier. But this can lead to configuration drift, because unless you regularly run all your Ansible playbooks against all of your servers, different groups of servers in your environment can have wildly varying versions of things.

Puppet isn’t a good choice for initially building a server, because you have to somehow bootstrap the configuration. (It can’t phone in until it knows where and how to do so.) And Ansible isn’t always the best choice for ongoing validation because of the aforementioned drift, and because YAML is so annoying and picky about spaces. But they’re both valuable tools for ongoing management.

You Pull and I Shall Push

Overwhelmed, in a good way

David Smith, , ,

After ten years at WashU, I’ve moved on to Pagely. It’s been a long time since I was so overwhelmed. Honestly, I feel a bit dumb, in the best possible sense of the word.

I’m proud of the work I did at WashU. They hired me to be “the WordPress guy,” and over time I built up a platform that was efficient, resilient, highly available. The main bottleneck was the “I” in that first sentence. Especially in the first year or two, there wasn’t a lot of not-Windows experience around, and most of the design decisions that I made were inherently limited by my own knowledge and skillset at the time.

Pagely is larger than WashU, in terms of the volume of traffic on their WordPress sites. As I write this, their home page lists clients including Disney, Cisco, and WB. They only started a couple years before I started at WashU, but they had the benefits of being a “they” — a group of smart people are going to generate better ideas than any one smart person can alone.

Obviously, some of the complexity is a matter of scale. The WashU environment I built effectively only had one customer (WashU itself), and only hosts a few hundred sites. Pagely has more customers than that, and many of them are bigger than WashU’s biggest sites. And they have to worry about things like “getting paid,” whereas being internal-only saves a lot of paperwork. But in the past week and a half, I think I’ve learned more about how to host WordPress sites at scale than I did in the ten years previous. I’m being exposed to so many new-to-me technologies and ideas. I feel overwhelmed and a bit stupid, and I’m fighting my fair share of impostor syndrome this week. But it will pass, and I’ll learn All. The. Things. and it will be great.

Overwhelmed, in a good way

Using Splunk as a syslog replacement

David Smith,

Many Splunk users find themselves standing up a “syslog box,” for the purpose of receiving syslog inputs, writing them to disk, running a universal forwarder, monitoring those files that you just created, reading in those files, then sending the data off to the indexer cluster. Why not cut out the middlemen and use Splunk as a direct syslog replacement? It’s not dead-simple, especially in a clustered environment, but it’s not TOO hard…

Continue reading “Using Splunk as a syslog replacement” →
Using Splunk as a syslog replacement

The Job Description, the Whole Job Description, and Nothing But the Job Description

David Smith,

Words mean things. Especially words in job descriptions, and even when those words are missing.

I recently did a couple of interviews for a large web hosting company – big enough that if you know the space, you’d know them. Here’s a description of their requirements and expectations for a recent SRE (site reliability engineer) opportunity they posted:

  • Knowledge of large-scale, high traffic platforms and the design of scalable, robust services in the real world
  • Experience with Infrastructure as code tooling (e.g., Terraform, Chef, Puppet, Ansible, Pulumi, Vault, etc)
  • Experience programming in one or more of the following: Go, Python, Ruby, C, C++, Java, etc
  • Experience with Unix/Linux operating systems internals (e.g., filesystems, system calls, namespaces, containers)
  • Knowledge of large-scale, high traffic platforms and the design of scalable, robust services in the real world
  • Experience with analyzing and troubleshooting systems.
  • Understanding of standard networking protocols and components such as: TCP/IP, HTTP, DNS, IP Subnetting, and Load Balancing
  • Passion, integrity, and humor that makes our team better as a whole

This is a very reasonable-looking job description! I’ve built large, scalable platforms, though nothing quite as large as what they’ve built. I’ve used Puppet and Ansible. I’ve done some programming, though in fairness mostly smaller projects, and I haven’t touched C or Python much in a while. I’ve built big servers, I’ve built big networks, I know a fair bit about *nix internals. I know my way around tcpdump and wireshark, I can troubleshoot network protocols with the best of ’em. I also had most of the nice-to-haves. (One of the nice-to-haves was “polyglot chops,” listing off several programming languages they use. But it was a nice-to-have, so I wasn’t too worried about its absence.)

Based on the above, I felt pretty darned confident going into a Zoom interview with a hiring manger. (I did the initial HR screen a few days previous.)

Almost immediately I was asked about the programming languages I’ve worked with. Like, within maybe five minutes. We were barely past the small-talk stage when my development experience (of which there is precious little, which is why it’s not on my resume) came up. When I said that I know some basic stuff but haven’t done a great deal of programming in the last several years, and most of what I’ve done was fairly domain-specific, you could almost see her heart break a little.

Turns out, the job would be about 50 percent SRE (the thing they advertised), and 50 percent developer. Which I’m not.

Fortunately, all this intervew meant was a couple hours of wasted time on my part (and on theirs), I don’t desperately need another job at this time. But if they had listed all of their requirements, and been a bit more up-front and accurate about the role, I could have saved everyone involved some time.

If you’re writing a job description, please include the whole thing, not just half of it.

The Job Description, the Whole Job Description, and Nothing But the Job Description