No, I’m not starting a podcast, that’s better left to professionals. But I can show up on one, on occasion.
career stuff
You Pull and I Shall Push
Server configuration goes both ways…
For most of the past decade, I did my server management primarily with Puppet. Servers being orchestrated by Puppet work on a “pull” model — they periodically contact a central server (colloquially called the puppetmaster), and request info on how they should be configured.
The new employer does things very differently — they primarily use Ansible, which works on a “push” model. When you need to make a configuration change, you actively tell the servers what changes to make and what actions to take.
Is one right and the other wrong? Of course not. But they’re very different, both have ups and downs.
The “pull” model (as Puppet implements it) is pretty much automatic. The Puppet agent phones home on a schedule, and downloads updated configs. You don’t have to think too much about it, just set and forget. That also can be a downside, in that changes will be picked up pretty quickly by your servers. A mistake could propagate pretty quickly. (There are ways to mitigate this, but if you’re new to Puppet these pitfalls and their workarounds may not be obvious.) I’ll admit that I got spoiled by the notion of letting things happen in the background, and just checking in on them later.
Ansible’s “push” model means that changes only go to the specific devices to which you send them. This is done by way of playbooks, files that specify your desired configuration. If you only want to make a change on a subset of servers, it’s a bit easier. But this can lead to configuration drift, because unless you regularly run all your Ansible playbooks against all of your servers, different groups of servers in your environment can have wildly varying versions of things.
Puppet isn’t a good choice for initially building a server, because you have to somehow bootstrap the configuration. (It can’t phone in until it knows where and how to do so.) And Ansible isn’t always the best choice for ongoing validation because of the aforementioned drift, and because YAML is so annoying and picky about spaces. But they’re both valuable tools for ongoing management.
Overwhelmed, in a good way
After ten years at WashU, I’ve moved on to Pagely. It’s been a long time since I was so overwhelmed. Honestly, I feel a bit dumb, in the best possible sense of the word.
I’m proud of the work I did at WashU. They hired me to be “the WordPress guy,” and over time I built up a platform that was efficient, resilient, highly available. The main bottleneck was the “I” in that first sentence. Especially in the first year or two, there wasn’t a lot of not-Windows experience around, and most of the design decisions that I made were inherently limited by my own knowledge and skillset at the time.
Pagely is larger than WashU, in terms of the volume of traffic on their WordPress sites. As I write this, their home page lists clients including Disney, Cisco, and WB. They only started a couple years before I started at WashU, but they had the benefits of being a “they” — a group of smart people are going to generate better ideas than any one smart person can alone.
Obviously, some of the complexity is a matter of scale. The WashU environment I built effectively only had one customer (WashU itself), and only hosts a few hundred sites. Pagely has more customers than that, and many of them are bigger than WashU’s biggest sites. And they have to worry about things like “getting paid,” whereas being internal-only saves a lot of paperwork. But in the past week and a half, I think I’ve learned more about how to host WordPress sites at scale than I did in the ten years previous. I’m being exposed to so many new-to-me technologies and ideas. I feel overwhelmed and a bit stupid, and I’m fighting my fair share of impostor syndrome this week. But it will pass, and I’ll learn All. The. Things. and it will be great.
The Job Description, the Whole Job Description, and Nothing But the Job Description
Words mean things. Especially words in job descriptions, and even when those words are missing.
I recently did a couple of interviews for a large web hosting company – big enough that if you know the space, you’d know them. Here’s a description of their requirements and expectations for a recent SRE (site reliability engineer) opportunity they posted:
- Knowledge of large-scale, high traffic platforms and the design of scalable, robust services in the real world
- Experience with Infrastructure as code tooling (e.g., Terraform, Chef, Puppet, Ansible, Pulumi, Vault, etc)
- Experience programming in one or more of the following: Go, Python, Ruby, C, C++, Java, etc
- Experience with Unix/Linux operating systems internals (e.g., filesystems, system calls, namespaces, containers)
- Knowledge of large-scale, high traffic platforms and the design of scalable, robust services in the real world
- Experience with analyzing and troubleshooting systems.
- Understanding of standard networking protocols and components such as: TCP/IP, HTTP, DNS, IP Subnetting, and Load Balancing
- Passion, integrity, and humor that makes our team better as a whole
This is a very reasonable-looking job description! I’ve built large, scalable platforms, though nothing quite as large as what they’ve built. I’ve used Puppet and Ansible. I’ve done some programming, though in fairness mostly smaller projects, and I haven’t touched C or Python much in a while. I’ve built big servers, I’ve built big networks, I know a fair bit about *nix internals. I know my way around tcpdump and wireshark, I can troubleshoot network protocols with the best of ’em. I also had most of the nice-to-haves. (One of the nice-to-haves was “polyglot chops,” listing off several programming languages they use. But it was a nice-to-have, so I wasn’t too worried about its absence.)
Based on the above, I felt pretty darned confident going into a Zoom interview with a hiring manger. (I did the initial HR screen a few days previous.)
Almost immediately I was asked about the programming languages I’ve worked with. Like, within maybe five minutes. We were barely past the small-talk stage when my development experience (of which there is precious little, which is why it’s not on my resume) came up. When I said that I know some basic stuff but haven’t done a great deal of programming in the last several years, and most of what I’ve done was fairly domain-specific, you could almost see her heart break a little.
Turns out, the job would be about 50 percent SRE (the thing they advertised), and 50 percent developer. Which I’m not.
Fortunately, all this intervew meant was a couple hours of wasted time on my part (and on theirs), I don’t desperately need another job at this time. But if they had listed all of their requirements, and been a bit more up-front and accurate about the role, I could have saved everyone involved some time.
If you’re writing a job description, please include the whole thing, not just half of it.