• 1 Post
  • 35 Comments
Joined 2 years ago
cake
Cake day: June 19th, 2023

help-circle
  • Certs are a waste of time tbh. If you have 8 years of experience, you should have more than enough to fill out a resume already.

    An AWS cert is almost certainly even more useless for you specifically unless you wanted to get into devops/sre and do systems design. I have been in sre for a very long time and have never even heard of anyone writing tooling in Java. That section of the industry is entirely dominated by go, python, and (more often than anything else) bash for really quick automation.


  • As a counter balance to that though, interviewers need to understand what they are hiring for and tailor the questions asked to those requirements.

    For example, there is genuinely very little coding required of an SRE these days but EVERY job interview wants you to do some leetcode style algorithm design… Since containers took over, the times I have used anything beyond relatively unremarkable bash scripts is exceptionally small. It’s extremely unlikely that I will be responsible for a task that is so dependent on performance that I need to design a perfect O(1) algorithm. On terraform though, I’m a fucking surgeon.

    SRE specifically should HEAVILY focus on system design and almost all other things should have much much less priority… I’ve failed plenty of skill assessments just because of the code though.




  • Most resistance I have seen mostly comes down to a misunderstanding in the benefits that kubernetes offers. The assumption is that kube is used for autoscaling and that, if the inbound traffic is predictable then the added complexity is unnecessary. When that happens the “kube isn’t right for all situations” turns into “kube isn’t right for any situation” whether the person in question would ever admit that or not…

    All of this ignores the MASSIVE reliability enhancement kube delivers and the huge amount of effort currently going into modern tool development surrounding the kube ecosystem.


  • Real talk, you don’t have the luxury of being an idealist right out of university. Your goal is to get a job. When you’re in that job you will likely not have the luxury of being an idealist either.

    When you have enough experience making practical, reasoned decisions, then you can stand on principals.

    For context, I have been in this business for nearly 20 years. The people I have personally worked with who have resisted things on philosophical grounds ALWAYS get left behind. I’ve seen it with systemd, the cloud, and now I’m seeing it again with kubernetes. You cannot escape the collective inertia of an entire industry.

    Obviously there are still thresholds… I would never work for someone like Raytheon. You have to draw lines somewhere but saying you aren’t going to work for a company that does user behavior tracking is short sighted and impractical.



  • We had a service that compiles a dataset once per quarter. The total size is ~30gb. We were starting a container, storing it on an EFS volume, and mounting like any other disk.

    Every time a pod started it would need to read this data into memory so we would get quick initial start-up time but the time to be ready for traffic still took a while.

    Since we didn’t need to update it very often, we decided to just package the compiled dataset into the container and skip the EFS volume. We updated the image pull policy to ifNotPresent so it cut egress traffic pricing from EFS to zero. Now there is a cost to pull the image from ECR but that’s only if the pod is being scheduled onto a node it hasn’t been run on before. There was no noticable change in behavior or performance and we saved a bunch on cost.

    Sometimes the big, dumb option is the right choice.



  • We focus a lot more on production than the average developer. It’s our job to make sure whatever devs build is run quickly, efficiently, safely, and scalably.

    You will work with a lot of kubernetes, Argo, terraform, Prometheus, grafana. You’ll design build pipelines and software rollout strategies. You plan for zero downtime migrations and upgrades, database maintenance… You’ll have your hands in everything from capacity planning to security to cost optimization to developer support… User permissions, infrastructure, networking, observability… You will write RFCs and setup POCs for new tools. You define and track error budgets and figure out how to keep your org under those projections. When there is an outage you will be involved in writing post mortems.

    The days are so varied and unpredictable that it keeps things interesting. The landscape changes so often you’re never really stuck doing the same thing over and over.

    I genuinely love it.

    EDIT: The SRE Podcast from Google is actually really great for learning about this world. The first season talks about what you’ll be doing and why (based around the SRE O’Riley book). The second season talks about what to expect in different stages of your career progression.






  • I still don’t really see the argument. OpenTofu exists because of internal drama about licensing on a tool that you don’t use…

    Someone building a different banana picker that looks just like another banana picker doesn’t need to explain their reasoning in terms a coal miner would understand…

    Also, literally just clicking the intro doc linked from the main page tells you everything you need to know…


  • If you’re asking why aren’t DevOps/SRE mentioned specifically on the OpenTofu front page, I don’t think you understand how common this software is… Like if someone forked Google, you wouldn’t need to describe what Google is. Everyone already knows it. For the people in this industry, terraform is essentially a defacto monopoly. Even if you don’t use it, if you’re working in SRE, you know what it is and what it does.



  • Not who you were responding to but, my company does this in AWS. To be fair, the entire platform is running in EKS so it’s not much more difficult than updating the CI build pipelines to build multi-arch containers, adding additional nodepools, and scaling down the amd64 ones. This was tedious but not difficult to do. I keep a small set of amd64 nodes for off the shelf software that doesn’t support arm… I think the only thing left on those now is newrelic agents. Once we move off of them the x86_64 nodes can be killed entirely.

    This ended up saving us tens of thousands of dollars per month. The next step is to move the bulk of workloads to spot instances. I’ll be preferring arm but if there is only capacity for x86_64, I’ll have that option because of the multi-arch containers. This is going to save even more money and force developers to build applications more tolerant of node failure in the process.


  • I had this just the other day actually. I am in SRE and the overwhelming majority of the code I write is terraform, instructions for a Dockerfile or CI pipeline, or just some random ass bash to compile information… I don’t actually do that much coding at all and what I do end up doing is pretty rudimentary.

    EVERY job interview I go on though wants to do some leetcode style code puzzle. The one I got the other day I just said to the guy, “I honestly don’t know how to do this. The code I write isn’t fancy or clever. It’s mostly just to get things done.” We worked through it together though and I understood the logic by the end but they were mostly holding my hand. What I was doing was throwing out ideas and trying to work out pros/cons with the interviewer. That was enough apparently because he green-lit me for the next round…

    These type of interview questions really annoy me because they are not representative of the job in any way. In addition to work, I also have a life that does not involve computers. After putting in a 40 hour week on engineering stuff, grinding leetcode over a weekend is a hard sell.