Overwhelmed, in a good way

After ten years at WashU, I’ve moved on to Pagely. It’s been a long time since I was so overwhelmed. Honestly, I feel a bit dumb, in the best possible sense of the word.

I’m proud of the work I did at WashU. They hired me to be “the WordPress guy,” and over time I built up a platform that was efficient, resilient, highly available. The main bottleneck was the “I” in that first sentence. Especially in the first year or two, there wasn’t a lot of not-Windows experience around, and most of the design decisions that I made were inherently limited by my own knowledge and skillset at the time.

Pagely is larger than WashU, in terms of the volume of traffic on their WordPress sites. As I write this, their home page lists clients including Disney, Cisco, and WB. They only started a couple years before I started at WashU, but they had the benefits of being a “they” — a group of smart people are going to generate better ideas than any one smart person can alone.

Obviously, some of the complexity is a matter of scale. The WashU environment I built effectively only had one customer (WashU itself), and only hosts a few hundred sites. Pagely has more customers than that, and many of them are bigger than WashU’s biggest sites. And they have to worry about things like “getting paid,” whereas being internal-only saves a lot of paperwork. But in the past week and a half, I think I’ve learned more about how to host WordPress sites at scale than I did in the ten years previous. I’m being exposed to so many new-to-me technologies and ideas. I feel overwhelmed and a bit stupid, and I’m fighting my fair share of impostor syndrome this week. But it will pass, and I’ll learn All. The. Things. and it will be great.

Overwhelmed, in a good way

Using Splunk as a syslog replacement

Many Splunk users find themselves standing up a “syslog box,” for the purpose of receiving syslog inputs, writing them to disk, running a universal forwarder, monitoring those files that you just created, reading in those files, then sending the data off to the indexer cluster. Why not cut out the middlemen and use Splunk as a direct syslog replacement? It’s not dead-simple, especially in a clustered environment, but it’s not TOO hard…

Continue reading “Using Splunk as a syslog replacement”
Using Splunk as a syslog replacement

The Job Description, the Whole Job Description, and Nothing But the Job Description

Words mean things. Especially words in job descriptions, and even when those words are missing.

I recently did a couple of interviews for a large web hosting company – big enough that if you know the space, you’d know them. Here’s a description of their requirements and expectations for a recent SRE (site reliability engineer) opportunity they posted:

  • Knowledge of large-scale, high traffic platforms and the design of scalable, robust services in the real world
  • Experience with Infrastructure as code tooling (e.g., Terraform, Chef, Puppet, Ansible, Pulumi, Vault, etc)
  • Experience programming in one or more of the following: Go, Python, Ruby, C, C++, Java, etc
  • Experience with Unix/Linux operating systems internals (e.g., filesystems, system calls, namespaces, containers)
  • Knowledge of large-scale, high traffic platforms and the design of scalable, robust services in the real world
  • Experience with analyzing and troubleshooting systems.
  • Understanding of standard networking protocols and components such as: TCP/IP, HTTP, DNS, IP Subnetting, and Load Balancing
  • Passion, integrity, and humor that makes our team better as a whole

This is a very reasonable-looking job description! I’ve built large, scalable platforms, though nothing quite as large as what they’ve built. I’ve used Puppet and Ansible. I’ve done some programming, though in fairness mostly smaller projects, and I haven’t touched C or Python much in a while. I’ve built big servers, I’ve built big networks, I know a fair bit about *nix internals. I know my way around tcpdump and wireshark, I can troubleshoot network protocols with the best of ’em. I also had most of the nice-to-haves. (One of the nice-to-haves was “polyglot chops,” listing off several programming languages they use. But it was a nice-to-have, so I wasn’t too worried about its absence.)

Based on the above, I felt pretty darned confident going into a Zoom interview with a hiring manger. (I did the initial HR screen a few days previous.)

Almost immediately I was asked about the programming languages I’ve worked with. Like, within maybe five minutes. We were barely past the small-talk stage when my development experience (of which there is precious little, which is why it’s not on my resume) came up. When I said that I know some basic stuff but haven’t done a great deal of programming in the last several years, and most of what I’ve done was fairly domain-specific, you could almost see her heart break a little.

Turns out, the job would be about 50 percent SRE (the thing they advertised), and 50 percent developer. Which I’m not.

Fortunately, all this intervew meant was a couple hours of wasted time on my part (and on theirs), I don’t desperately need another job at this time. But if they had listed all of their requirements, and been a bit more up-front and accurate about the role, I could have saved everyone involved some time.

If you’re writing a job description, please include the whole thing, not just half of it.

The Job Description, the Whole Job Description, and Nothing But the Job Description

Clear Linux Cheat Sheet

Clear Linux has Big Slackware Energy, in that if you want to use it as a daily driver, you’ll need to know your way around the underlying software. That said, I don’t think that’s how you’re supposed to use it (despite what that one Phoronix post might have you believe). It really feels purpose-built for connecting to cloud-init or Ansible or something like that, for machines you’ll spin up then throw away tomorrow afternoon.

Things I’ve learned while tinkering with Intel’s new Clear Linux distribution. Will probably get updated from time to time, unless I get bored.

  • Checking for updates: sudo swupd update
  • Installing a new bundle (their version of a package group): sudo swupd bundle-add bundlename
    • Bundles you’ll probably want for general tinkering: dev-utils, tmux
    • Bundles you might want if you want a traditional GUI experience: desktop (GNOME) or maybe os-utils-gui (a basic xfce desktop)
  • Growing the hard disk (probably needed if you want to install GNOME on a VM): see this (basically, grow it the VM-specific way, install the storage-utils bundle, use parted and resize2fs to actually use the new space)

 

Clear Linux Cheat Sheet

PAC Files and the Inconstant Constant

Recently, I was working on an update for the PAC files we use at work. Had a clever idea for making clients more resilient in the event of an individual proxy failing – just list all the proxies in the PAC file, and use a bit of MATH to spread the load. This initially failed, but not for any of the reasons you might expect…

Continue reading “PAC Files and the Inconstant Constant”
PAC Files and the Inconstant Constant

Updating your WSL Debian Image to “Buster”

While Windows Subsystem for Linux (WSL) isn’t perfect, it’s coming along nicely as a replacement for many of the Unix-y things I used to do with Cygwin. And since most of my server work involves Linux as opposed to generic *nix, it’s nice to have handy access to something that more closely resembles my work environment.

Just a few days ago, the Debian project announced “Buster,” the first major upgrade in a couple years. (If you want to be boring, it’s also “Debian 10.0”.) As of this writing, the Microsoft app store doesn’t have Buster ready-to-go. And even if it did, there’s not a super-clean way to migrate any custom work, home directories, etc. from an old Debian installation to a new one. So here, I throw caution to the wind and update my existing Debian WSL installation from “Stretch” (9.x) to Buster.

Continue reading “Updating your WSL Debian Image to “Buster””
Updating your WSL Debian Image to “Buster”