I’ve been lurking here for years, and after reading some recent posts, I need to say something that might make me unpopular with the "CV-Driven Development" crowd.

We are engineering our own burnout.

I've sat on hiring panels for the last 6 months, and the state of "Senior" DevOps is terrifying. I’m seeing a generation of engineers who can write complex Helm charts but can’t explain how DNS propagation works or debugging a TCP handshake.

Here is my analysis of why our industry is currently broken:

1. The Abstraction Addiction We are solving problems we don't have. I saw a candidate last week propose a multi-cluster Kubernetes setup with Istio for a simple internal CRUD app. When I asked why not just use a boring EC2 instance or ECS task, they looked at me like I suggested using FTP. We are choosing tools not because they solve a business problem, but because we want to put them on our LinkedIn. We are voluntarily taking on the operational overhead of Netflix without having their scale or their headcount.

2. The Death of Debugging To the user who posted "New DevOps please learn networking": Thank you. We are abstracting away the underlying systems so heavily that we are creating engineers who can "configure" but cannot "fix." When the abstraction leaks (and it always does, usually at 3 AM), these "YAML Engineers" are helpless because they don't understand the Linux primitives underneath.

3. Hiring is a Carnival Game We ask for 8 rounds of interviews to test for trivia on 15 different tools, but we don't test for systems thinking. Real seniority isn't knowing the flags for every CLI tool; it's knowing when not to use a tool. It's about telling management, "No, we don't need to migrate to that shiny new thing."

4. Complexity = Job Security (False) We tell ourselves that building complex systems makes us valuable. It doesn't. It makes us pagers. The best infrared engineers I know build systems so boring that they sleep through the night. If you are currently building a resume-padder architecture: Stop.

If you are a Junior: Stop trying to learn the entire CNCF landscape. Learn Linux. Learn Networking. Learn a scripting language deeply. If you are a Senior: Stop checking boxes. Start deleting code.

The most senior thing you can do is build something so simple it looks like a junior did it, but it never goes down.

/endrant

I'm curious if other DevOps/platform/SRE teams are running into the same thing my team is.

We're a fairly large environment, mostly EKS, and essentially 100% IaC/Terraform. We also support multiple companies/business units, so while I'd argue our infrastructure is fairly well organized, there's inherently a lot of it and a lot of architecture and context to understand.

Over the last year, our devs have sped up dramatically with AI. The company has leaned heavily into AI-assisted development, reduced developer headcount, and is now pushing toward developers being more "full stack with AI," including having them contribute more of their own infrastructure changes.

In theory, I'm completely in favor of that. I've always wanted developers to be able to own more of the infrastructure surrounding their applications.

In practice, though, it has been kind of a disaster.

We're getting flooded with infrastructure PRs largely written by Claude/other AI tools from developers who don't really understand the infrastructure they're modifying. The Terraform might look plausible, but once you understand the larger system there are frequently significant problems with it.

So instead of reducing the workload on DevOps, it feels like AI has massively increased it.

A huge percentage of our time is now spent reviewing AI-generated Terraform, finding problems, explaining why something won't work, explaining how AWS/EKS/networking/IAM/CI/CD/etc. fit together in our environment, and then going through another iteration of an AI-generated PR.

There's an interesting asymmetry I've noticed too. Our DevOps team is mostly made up of former software developers who moved toward infrastructure, automation, and pipelines. Most of us can jump into application code and be productive pretty quickly, especially with AI helping us. Like,...I feel like (and have some evidence to support) that our small DevOps team could largely take over all of the dev's tasks, but they are falling on their faces trying to deal with ours.

AI seems extremely good at helping someone who understands software write more software. It seems much less capable of allowing someone without infrastructure experience to suddenly understand a large production environment.

The complaint we're increasingly hearing is basically: "We can't successfully do full-stack development with AI because the infrastructure is too complicated."

And maybe they're right, but before AI, I would have said that this company is the most organized and best architected I've ever been at. I mean....100% IaC has never been something I've experienced, and it's very rare that we hit a use case brought up by one of the several companies where we don't already have a set of generalized modules that can't support it.

Our environment is complex, but a lot of that complexity isn't accidental. We have a large organization, multiple companies we deploy for, Kubernetes, networking, security requirements, IAM, CI/CD, observability, etc. You can't abstract away the fact that these things exist. And we're already 100% Terraform/IaC, which I would have thought would make this considerably easier for AI to reason about than an environment full of manually configured infrastructure.

The strangest part is the staffing effect.

AI allowed the organization to reduce software engineering headcount because individual developers became more productive. But now those remaining developers can generate changes so quickly that our DevOps team is completely overwhelmed trying to support and review them.

It genuinely feels like we could double the size of the DevOps team right now and still have plenty of work. We are working on an AI assistant that can help the devs deploy to our environment more effectively, but we're having a hard time finding time to work on it because we're constantly helping the devs.

I'm starting to wonder whether this is going to be a broader consequence of AI-assisted development: AI increases the rate at which software can be produced much faster than it increases the rate at which infrastructure/platform teams can safely absorb changes.

For those of you working in DevOps/platform/SRE at companies heavily adopting AI:

Are you seeing this too?

And if you are, how are you handling it?

Have you increased platform/DevOps staffing? Built better abstractions or internal developer platforms? Given developers more direct infrastructure ownership? Put stricter boundaries around what application teams can modify? Found ways of giving AI enough context about your infrastructure that it actually produces good changes?

Or has AI actually reduced your infrastructure workload, and we're doing something wrong?

Why is the management so obsessed with achieving automation through AI agents only.

Management forcing Everyone in the team to come up with an idea to implement/ problem we are currently facing and create an AI agent to solve that problem In 2 weeks. How is it possible.

Basic shell scripts would achieve the same results than these AI agents could. But management is turning its eyes on anyone who doesn't work on AI stuff

Even after all this pushing towards AI, when we really do experiment with it and ask for extra credits or tokens, they are questioning it and asking for explanations. Didn't they realize it before?

Using AI is obviously going to cost you. If you cant afford it, why to even force people into doing this.

I am getting tired of this. They are taking advantage of the bad job market outside and torturing the employees.

Edit:
FYI, I am pro AI. I cant survive without it at this point.
I single handedly migrated most of our codebase to Github and converted all CI/CD pipelines to Github Action workflows. So I do know the impact of the AI.
But we do our due diligence on every new tech we want to implement in our vast enterprise product. But why doesn't AI get the same scrutiny here.
And we are a platform engineering team offering Centralized platform as a SAAS product to our lot of internal clients, If something wrong happens, entire thing goes down.

Hey everyone,
I’m currently a Cloud/DevOps engineer. With AI rapidly automating things like boilerplate YAML, standard CI/CD pipelines, and basic log analysis, I'm trying to be proactive about my next career move.
For those already adapting:
Where do you see traditional DevOps going over the next few years?
What do you think is the most reliable, high-demand career shift adjacent to DevOps right now? (e.g., Platform Engineering, MLOps, DevSecOps?)
Would love to hear your thoughts on where to focus my upskilling. Thanks!

I've been playing around with agents, trying to work on DevOps management tasks (including provisioning, updating, monitoring and troubleshooting across different parts of the infra lifecycle, starting from Terraform/Pulumi to Kubernetes, Prometheus, and app layer tools), and wanted some feedback on what tasks in your usage of these tools you think frontier models do not really achieve what they're set out to do.
I've discovered certain patterns, but they're pretty niche in general. I'm looking for feedback on something more realistic. Curious if folks out here have struggled with some tasks using the frontier models in general.

Disclosure: I'm an academic looking at DevOps automation in general, using agents, and want some community feedback to ground my work. Thanks!

As the title says

We would have lots of different flows, VM updates, cluster rollouts, QA pipelines.

The meeting we had basically was the downsizing of Jenkins and scripts on our part and focus on agents to do this (to me it's a different type of pipeline). Same with Ansible.

Just wondering are other companies seeing the same push, lesser focus on normal tooling.

In my head it's all fun, but there will always be hallucinations that you just won't get with strict scripts and tooling

Software engineering spent thirty years building a predictable culture around Git, CI/CD, reproducible builds, and rollbacks.

You check code in, it gets reviewed, you know exactly what's running in production. If something breaks, you find the commit and roll it back.

With agents that entire safety net disappears at runtime. System prompts, dynamic memory contexts, tool permissions half the state is made in a black box. Trying to audit why an agent made a specific decision on a Tuesday afternoon is nearly impossible(NEARLY)

I don't think we can keep deploying AI this way. Agent behavior needs to be treated like a versioned artifact. Prompts, rules, memory all of it should live in Git just like everything else.

Are other engineering teams moving toward declarative, version controlled agent setups or are most people just praying to the machine gods like me

context: migrating from docker swarm to k8s. small team, needed to move fast. i had some k8s experience but never owned a prod cluster

used cursor to generate configs for our 12 services. honestly saved my ass, would have taken days otherwise. got deployments, services, ingress done in maybe an hour. ran in staging for a few days, did some basic load testing on the api endpoints, looked solid

deployed tuesday afternoon during low traffic window. everything fine for about 6 hours. then around 9pm our monitoring started showing weird patterns - some requests fast, some timing out, no clear pattern

spent the next few hours debugging the most confusing issue. turns out multiple things were breaking simultaneously:

our main api was crashlooping but only 3 out of 8 pods. took forever to realize the ai set liveness probe initialDelaySeconds to 5s. works fine in staging where we have tiny test data. prod loads way more reference data on startup, usually takes 8-10 seconds but varies by node. so some pods would start fast enough, others kept getting killed mid-initialization. probably network latency or node performance differences, never figured out exactly why

while fixing that, noticed our batch processor was getting cpu throttled hard. ai had set pretty conservative limits - 500m cpu for most services. batch job spikes to like 2 cores during processing. didnt catch it in staging because we never run the full batch there, just tested the api layer

then our cache service started oom killing. 256Mi limit looked reasonable in the configs but under real load it needs closer to 1Gi. staging cache is basically empty so never saw this coming

the configs themselves were fine, just completely generic. real problem was my staging environment told me nothing useful:

• test dataset is 1% of prod size
• never run batch jobs in staging
• no real traffic patterns
• didnt know startup probes were even a thing
• zero baseline metrics for what "normal" looks like

basically ai let me move fast but i had no idea what i didnt know. thought i was ready because the yaml looked correct and staging tests passed

took about 2 weeks to get everything stable:

• added startup probes (game changer for slow-starting services)
• actually load tested batch scenarios
• set up prometheus properly, now i have real data
• resource limits based on actual usage not guesses
• tried a few different tools for generating configs after this mess. cursor is fast but pretty generic. copilot similar. someone mentioned verdent which seems to pick up more context from existing services, but honestly at this point i just validate everything manually regardless of what generates it

costs are down about 25% vs swarm which is nice. still probably over-provisioned in places but at least its stable

lesson learned: ai tools are incredible for velocity but they dont teach you what questions to ask. its like having an intern who codes really fast but never tells you when something might be a bad idea

For a small team with a few services, an enterprise checklist can become ceremony without reducing the main risks. My minimum gate would cover a tested rollback, backups with a restore exercise, health and readiness checks, bounded timeouts and retries, an alert owner, log correlation, secret ownership, and a short incident runbook.

I would add load testing or multi-region recovery only when the service’s traffic and recovery target justify them. Which item has prevented a real incident for a small team, and which common checklist item has mostly created busywork?

Is reddit becoming Moltbook.. it feels half of the posta and comments are written by agents. The same syntax, structure, zero mistakes, written like for a robot.

Wtf is happening, its not only this sub but a lot of them. Dead internet theory seems more and more real..

I've been thinking about AI automation lately, and I'm starting to feel like long-term memory might be one of the biggest problems.It's not just about making AI capable of controlling a screen. The AI also needs to remember what it's supposed to do.

There are already quite a few ways for AI to control screens, like OpenAI Computer Use, Claude Computer Use, Gemini Computer Use, as well as various hybrid approaches.The way these systems maintain memory seems to rely more on things like structured actions exposed by apps and the keywords being used in the current interaction. Personally, I don't think this approach works that well.

I also don't find this kind of screen-control approach particularly convenient.If I could use a hardware board to control the entire screen instead, that would make much more sense to me.Basically, you plug a hardware board into the device's USB port, and let the hardware capture the phone's screen and then control the device through USB HID.

I think this approach is pretty interesting because the AI doesn't necessarily need to know what API each app has, and it doesn't need a separate integration for every app.

It just needs to be able to understand what's happening on the screen and remember what it's supposed to do.So I feel like memory is actually the key problem here.

Are there any existing solutions or approaches that I should look into?I'd really like to understand how people are solving this problem.

Hi everyone,

I recently watched a podcast featuring a senior developer from EPAM, where he explained the different phases of the SDLC. He mentioned that he built a pipeline covering the entire software development lifecycle—from planning and design to development, testing, deployment, and maintenance—with each stage flowing into the next.

I really liked that idea, and now I'm wondering how to build a pipeline like that myself. Does anyone have any good resources, guides, or examples to learn from?

Thanks!

Hey everyone,

Quick workflow showcase/idea for anyone who gets enterprise access to AI CLIs at work (Gemini CLI, Claude Code, etc.) and still finds themselves manually jumping between tools during investigations or routine support tasks.

I recently set up a workflow that basically turns my CLI into an automated agent that runs investigations for me. If you have enterprise access and tool integrations available, you’re missing out if you aren't doing this.

The Setup:

1. CLI + Enterprise Access: Using the company-provided enterprise CLI directly in the terminal.
2. MCP (Model Context Protocol) Integration: Connected the CLI to the actual operational and diagnostic tools used daily (monitoring, databases, logs, ticketing APIs, etc.).
3. Custom Subagents & Commands: Built specific subagent prompts and custom CLI commands tailored to my precise, repetitive investigation steps.

How it works in practice:

Instead of manually opening 4 tabs, running 3 queries, and checking status logs when an issue comes up:

1. I trigger a custom slash command or alias in my CLI.
2. The subagent kicks off, uses the connected MCP tools to pull relevant context, inspect logs, run diagnostic checks, and correlate data.
3. It hands back a summarized breakdown of what happened, root cause hypotheses, and recommended next steps.

Why do this?

• Zero Context Switching: You stay in the terminal.
• No Manual Data Copy-Pasting: The MCP server handles fetching real-time data securely.
• Repeatable Process: Standardized subagent prompts mean complex triage takes seconds instead of 20 minutes.

If your company already pays for enterprise CLI access and you're only using it to rewrite functions or explain error messages, hook it up to your tools via MCPs.

Curious if anyone else is running similar subagent setups in their terminal for operational work? What tools/MCPs are you chaining together?

Found this article:

This was an unusually hard post to write, because it flies in the face of everything else going on. I first started noticing a concerning new phenomenon a month ago, just after the new year, where people were overworking due to AI. This week I’m suddenly seeing a bunch of articles about it. I’ve collected a number of data points, and I have a theory. My belief is that this all has a very simple explanation: AI is starting to kill us all, Colin Robinson style.

https://steve-yegge.medium.com/the-ai-vampire-eda6e4f07163

Hey,

May I know if there's any roadmap or a YT playlist to learn Agentic AI in DevOps?

If someone can provide it here, that would be great.

We had Claude rewrite a high-traffic legacy Node.js service in Go as an experiment, and it succeeded thanks to rigorous test coverage and an advanced test harness.

Hey everyone,

I wanted to better understand how LLM inference actually works under the hood, so made a lightweight stack built around llama.cpp - it runs Gemma‑4 E2B model on Azure Container Apps.

Result - a running and ready-to-use LLM available from your browser (https://github.com/groovy-sky/azure/blob/master/local-ai-00/image-1.png)

The goal wasn’t to build anything production‑grade — mostly just to experiment, learn a bit more about the runtime side of LLMs, and document the process along the way.

P.S. For those who wants to run same setup - will leave a link in the first comment

P.P.S. Demo Container Apps are removed (https://gemma-h4ksrlmuz7pfa.ashysky-1e58cf76.westeurope.azurecontainerapps.io/ and https://gemma-lvm2vmhmvkrm6.ashystone-2aad3ea0.westeurope.azurecontainerapps.io/)

I’ve been experimenting with MCP tools at work and ended up building two that have actually stuck:

1) RAG / knowledge search tool

Our knowledge is scattered across wikis, docs, code, and tickets. The RAG tool queries all of it and returns URLs, so it ends up being a better search than anything we had before. My team rarely looks things up manually anymore. We just ask and verify straight at the source.

2) Log retrieval tool

This one’s been a big time saver. Instead of auth’ing into service accounts to pull logs, the tool runs a CloudWatch query and writes results to local JSON files that the agent can read.

These tools work hand-in-hand. We can get AI to analyze the log outputs and then use the knowledge base to reason about what’s going on. Logs + context together has been far more useful than either on its own.

The learning feedback loop

What really made this work for us was creating context docs for common issues: what log groups to look into, what queries to run, and what to look for.

After every investigation we ask: what information would the agent have needed to do this automatically next time? The best way we’ve found to do this is to just ask the agent:

“From what you learned during this investigation, how would you update the investigation context document?”

The agent is already capable of handling common investigations that each used to take us 10+ minutes of manual digging.

How it’s built (high level)

• Lambda parses docs, wikis, code, and tickets and writes them to S3

• Bedrock knowledge bases with OpenSearch Serverless for embeddings from data in S3

• We use Kiro as the assistant orchestrating the MCP tools

MCP tools are intentionally simple:

• The RAG tool just queries the knowledge base and returns the response plus citation URLs

• The log tool runs a CloudWatch query and writes results to local files instead of dumping logs directly into context

One thing I learned quickly is you don’t want MCP tools doing too much. Let the agent do the reasoning. Tools should just fetch.

What MCP tools have you built that you actually find useful day-to-day? I’m looking for ideas on what to build next.

Something I’ve been noticing more and more during reviews is that the bugs we miss usually aren’t about bad syntax or sloppy code.

They’re almost always about flow.

Stuff like an auth check happening after a downstream call. Validation happening too late. Retry logic triggering side effects twice. Error paths not cleaning up properly. A new external API call quietly changing latency or timeout behavior. Or a DB write and queue publish getting reordered in a way that only breaks under failure.

None of this jumps out in a diff. You can read every changed line and still miss it, because the problem isn’t a line of code. It’s how the system behaves when everything is wired together at runtime.

What makes this frustrating is that code review tools and PR diffs are optimized for reading code, not for understanding behavior. To really catch these issues, you have to mentally simulate the execution path across multiple files, branches, and dependencies, which is exhausting and honestly unrealistic to do perfectly every time.

I’m curious how others approach this. Do you review “flow first” before diving into the code? And if you do, how do you actually make the flow visible without drawing diagrams manually for every PR?

EDIT: I found a write-up that talks about making runtime behavior explicit and why diffs alone don’t catch flow issues. Sharing here since it links well with this problem: https://www.codeant.ai/blogs/reproduction-steps-ai-code-review

Disclosure first: I wrote the tool I mention here. It is free, there is no signup, and I am posting because the testing method is the interesting part. Mods, delete it if this is not welcome.

I got tired of IaC tools that validate templates and call it a day. A template can be perfectly valid and still blow up on deploy. So I did it the slow way: 20 scenarios per cloud, each one deployed for real, exercised at runtime, then destroyed. CRUD APIs, queue workers, private databases, Redis, WAF, websockets, fan-out, secrets, containers behind load balancers.

It took months and burned a lot of free tier. Some of what I hit:

A brand new GCP project gives you a compute service account with no roles at all. Your first deploy dies inside Cloud Build with an error that tells you nothing. Took me a while to figure that one out.

Cloud SQL with private IP is fine until you destroy it. The service networking peering can hold for half an hour after the instance is gone. And if your app created tables, the default postgres user cannot be dropped, so the destroy hangs there too.

Cosmos DB speaks the Mongo protocol but rejects sort() on any field without an index. Real MongoDB just sorts. If you port a handler over, your list endpoint breaks and the error message talks about order-by item paths.

Azure Functions on the consumption plan will happily deploy a Service Bus worker that never runs. The zip lands, the function exists, messages pile up in the queue. You have to sync the function triggers explicitly or the scale controller never looks at the queue.

Memorystore Redis with TLS needs the CA cert handed to the client or the handshake fails.

Every one of those turned into a fix in the tool, so a fresh project starts with them already handled. That is the whole point of doing it this way instead of trusting a linter.

Repo if you want to look: https://github.com/Claudio-Fontes/iacmp

It is Fair Source (FSL-1.1, same license Sentry uses). Free for any use including at work, you just cannot resell the tool itself, and each release turns into Apache-2.0 after two years.

Happy to talk about the e2e battery itself. If you have run something similar, I want to hear which scenarios you would add.

started noticing a pattern at a few companies i've talked to recently. instead of just giving devs access to copilot or claude and calling it a day, some teams are building dedicated internal tooling that wraps AI code generation into their existing deployment pipelines.

i'm talking things like: slack bots that trigger AI-assisted code changes, auto-run the test suite, open a PR, and deploy to staging - all without the developer touching their IDE. basically treating the AI model as just another step in the pipeline rather than a developer tool.

spotify apparently went pretty far down this road with something they built internally. but i'm curious if anyone here is seeing similar patterns at smaller companies too.

the devops angle that interests me is that the model itself is becoming table stakes - the actual competitive advantage is in the tooling layer you build around it. guardrails, automated review, deployment gates, rollback triggers. feels like a whole new category of infrastructure.

anyone building something like this? what does your pipeline look like when AI-generated code is involved? are you treating it differently from human-written code in terms of review and deployment gates?

Anyone tried giving their agents a single structured context source instead of wiring it into everything via mcp?

The pattern I’m wondering about: instead of plugging the model into GitHub + PagerDuty + Datadog + Slack and letting it reason its way to an answer, you build a structured context layer that already speaks your org’s language, point the model there first, and only fall back to the raw sources when the answer isn’t in the layer.

The reasoning when a model has to answer something simple like “who owns this service?” by stitching together signals across four tools, two things happen. The reasoning gets expensive (lots of tokens, lots of tool calls), and the answer isn’t always right — not because the model is dumb, but because there are several conflicting versions of truth across the data.

If the same question hits one structured source that already resolves the conflict, the model works way less hard and the answer stays consistent.

Has anyone actually run something like this in production? We’re putting together numbers on the token savings but I’d rather hear what people have seen first.

Hey folks,

I've been learning AWS and kept forgetting to delete test resources.

My last bill had charges for 3 EBS volumes I'd completely forgotten about.

Built a Python script to help catch these before they accumulate:

• Scans all AWS regions
• Finds 6 types of common waste
• Shows exact costs and cleanup commands

It's free/open source. Still learning, it's not perfect but it works and so feedback is welcome!

GitHub: AWS Waste Finder Tool

Specifically checking for:

1. Orphaned EBS volumes
2. Unused Elastic IPs
3. Idle Load Balancers
4. Old snapshots
5. NAT Gateways
6. SageMaker notebooks

Has anyone else dealt with surprise AWS bills? What resources did

you forget about?

I have to admit I am guilty of it. Not in my main tasks but I am overly relying on AI to summarize the whitepapers. That makes me too "lazy" to read the whole thing.

I don't use AI for coding. Not a good idea!

Would you mind to share your story? Have you seen anyone you work with rely on AI and take the "cognitive shortcut"?

Integrating ai-powered code review into ci/cd pipelines sounds good in theory where automated review catches issues before human reviewers even look, which saves time and catches stuff that might slip through manual review, but in practice there's a bunch of gotchas that come up. Speed is one issue where some ai review tools take several minutes to analyze large prs which adds latency to the pipeline and developers end up waiting, and noise is another where tools flag tons of stuff that isn't actually wrong or is subjective style things, so time gets spent filtering false positives. Tuning sensitivity is tricky because reducing it makes the tool miss real issues but leaving it high generates too much noise, and the tools often don't understand specific codebase context well so they flag intentional architectural patterns as "problems" because they lack full picture. Integration with existing tooling can be janky too like getting ai review results to show up inline in gitlab or github pr interface sometimes requires custom scripting, and sending code to external apis makes security teams nervous which limits options. Curious if anyone's found ai code review that actually integrates cleanly and provides more signal than noise, or if this is still an emerging category where the tooling isn't quite mature yet for production use?