So I've been working on this microservices project ( repo ) for a few months. I have it fully running locally using kind, now I want to learn how to deploy it on AWS or GCP following best practices to get hands on experience and record demo videos for my resume.

Here's the stack:

• Microservices: 5 Node.js services (built and pushed to GHCR )
• Stateful Services: 5 Postgres databases (one per service), Redis, and Kafka.
• Observability: Prometheus, Grafana, Loki, and ( Alloy deployed using Helm charts).
• K8s & Networking: I wrote custom manifests using StatefulSets and Headless Services for the stateful components. I am using the Gateway API for ingress, backed by cloud-provider-kind locally.

Being a college student my budget is pretty tight. My plan is to write Terraform scripts so I can spin the whole infrastructure up, record a few demos for my resume, and immediately tear it down to keep costs minimal.

The questions I have:

1. Which cloud provider to choose and how to deploy? Should I use managed K8s like EKS, or just run Kubernetes (or lightweight K8s like k3s) myself on a single EC2/VM instance? What's actually standard practice that I'd use on a real job?
2. How to handle routing in the cloud cheap? Locally I use Gateway API with cloud-provider-kind. What’s the most cost-effective way to handle this on AWS/GCP without getting hit with expensive cloud load balancer fees?
3. Is running DBs/Kafka in the cluster okay? Is keeping Postgres, Kafka, and monitoring inside K8s fine for a portfolio project, or do recruiters/hiring managers specifically look for managed cloud services ?

Any direction, best practices, or potential pitfalls to avoid would be greatly appreciated!

Azure Key Vault Key Deployment – 400 Bad Request
If you’re receiving a vague 400 Bad Request while deploying a key in Azure Key Vault, check how many tags you’re passing to the key.
Azure Key Vault keys support a maximum of 15 tags. I spent almost three hours on Friday evening troubleshooting permissions, networking, and the Terraform configuration before realizing that too many tags were causing the request to fail.
This may be an easy one to spot for some people, but the error message wasn’t very helpful, so I’m leaving this here in case someone runs into the same issue.

Hi, I’m trying to use my Android phone to read a smart card and create a digital signature, but it seems there’s no straightforward way to do this. Does anyone know how to make it work on Android or iOS?

I'm building a simple web app with node/express that uses a local sqlite database. (I'm not experienced in devops, btw)

Although I've already managed to get this app running in production by doing a manual deploy via SSH and cloning my repo from github, I'm having issues understanding how I'm going to deal with the database (.db) file in production.

Since my database is running on the same debian VPS as the application, I'm starting to realize I need to be a little bit more thoughtful and stabilish some sort of process if I ever need to restore a db backup or do migrations on this database.

I though about leaving the db file outside the project directory, which should allow me to safely pull new code that I push to production while leaving the db file intact, (as long as I'm not changing db structure/doing migrations) but I'm not quite sure how to deal with backups, migrations and whatever scenarios might come down the line.

Do any of you have suggestions on how to deal with this sort of stateful local database scenario?

Hello, I’m having an issue with GitHub Actions in one of my private repositories. I have a workflow configured to run daily, but it is not being triggered.

I tested the same thing in another repository by configuring a workflow to run every 5 minutes, but same thing happens there as well.

has anyone experienced this recently? Could this be an issue on GitHub’s side?

Workflow: https://github.com/ahmedhesham301/autoscaling-hetzner/blob/main/.github/workflows/snyk-security.yml

Hi, I'm trying to setup a basic ha postgres cluster using the Spilos images in a docker swarm setup, but every few days the thing crumbles down with DNS resolution issues, timeouts, wal and etcd records corruption, I don't have the money to rely on an hosted solution rn, so has anybody run into these problems and solved them?. This makes me understand why the DBA role exists, but it is so frustrating and absurd that it is not a solved problem for something that feels so relatively trivial to setup in mariadb with galera.

I have recently migrated the production environment on the new cluster version 1.36. And i want to install the cluster autoscaler in it. But the cluster autoscaler has latest version of 1.35. what should i do now ?

Should i install the cluster autoscaler with version 1.35 inside the eks 1.36 ??

So let me tell the context first, I have an kubernetes environment for production, and for like shared storage solution, I have installed nfs-utils and use the nfs-server service. In my scenario I cannot use cloud as an option. And for long I have been using the nfs-server but I have been facing alot of issues with it.
1. On multiple occasions I found dead or queued nfs processes in worker nodes which made the worker node totally unresponsive sys load was very high but low cpu and ram usage.
2. Something with the nfs4+ client where it uses session. On of the worker node again stopped responding, df -h was not responding rebooted the node still the same issue. And while viewing logs in the nfs node it stated nfsd was blocked and looking it up stated the thread on that worker node was locked due to unknown reason and all the requests from the worker node had just queued up. Restarting the service also hanged and needed to reboot the node itself to resolve the issue.

These are some major events there are also other small issues I have faced. I wanted to understand am I the only one facing these issues or this happens with others too, and what resolutions did you used to fix it? Like any alternative utilities and stuff

Posting this both to ask if anyone else in **me-riyadh-1** is hitting it, and so the next person googling "OCI curl timeout Riyadh" finds something useful.

**Setup:** Ubuntu compute instance, public subnet, default route → Internet Gateway, stateful egress allow-all, VCN resolver for DNS. Running a Laravel app in Docker. Nothing exotic.

**Symptoms**

- `curl https://api.anthropic.com` → times out every single time (TCP connect, ~10s). Claude Code installer obviously fails too.

- `curl https://deb.debian.org` → works maybe half the time, times out the other half. So `apt update` inside Docker builds randomly fails, then `install-php-extensions` blows up with "Unable to locate package libfreetype6 / libpq5 / libzip" (not actually missing, just failed index fetch).

- `curl https://github.com` and `https://pecl.php.net\` → fine, every time.

- DNS is fine. Resolution works for everything. This is not a DNS problem.

- IPv6 is a separate thing: DNS returns AAAA but the instance has no global v6 (never enabled at VCN/subnet/VNIC level), so anything preferring v6 stalls. Fixed that on my side with `precedence ::ffff:0:0/96 100` in `/etc/gai.conf`. Not the main issue.

**The mtr evidence** (TCP mode, port 443, 100 probes each)

Anthropic — path just dies after the upstream handoff:

```

1. 84.8.76.x (OCI edge) 0.0%

2. 87.109.47.100.0%

3. ??? 100.0%

```

That's it. Nothing at any TTL beyond hop 2, no SYN-ACK from the destination, ever.

Debian (Fastly) — real loss starting at hop 3, carried all the way to the destination:

```

1. 84.8.76.x (OCI edge) 0.0%

2. 87.109.47.100.0%

3. 87.101.252.6154.0%

4. 10.0.28.1347.0%

5. fra1.decixfra.fastly.net51.0% ~80ms

6. 146.75.118.132 (destination) 54.0%

```

(Also getting routed to Fastly in **Frankfurt** from Riyadh, which seems wrong on its own.)

GitHub control trace completes fine and real traffic works, so the path itself isn't globally broken — it's destination-specific.

**Why I'm confident it's not my config:** hops 1 and 2 (OCI's edge + the upstream handoff) are 0% on every trace. My route table / security lists / IGW are passing everything. The loss and the blackhole both begin at the same boundary (87.109.47.10 → 87.101.252.61), i.e. regional transit/peering. I've opened an SR with all three reports.

**Questions**

1. Anyone else in me-riyadh-1 seeing Anthropic (AS399358, 160.79.104.0/21) unreachable, or Fastly flaky?

2. Has anyone gotten OCI to actually escalate a transit/peering issue in a newer region? How long did it take?

3. Any recommended Debian mirror that has a clean path from Riyadh? I'm currently shopping for one with curl to keep Docker builds working while this is open.

Interim plan: apt mirror swap for builds, and if the Anthropic blackhole drags on, a WireGuard egress through another region for Claude Code only. Would much rather OCI just fix the route.

Hi everyone,

first off, if this isn't the perfect subreddit for this issue,please let me know if I should ask this elsewhere! Thanks in advance :)

We are hosting a custom R package repository and getting a 404 Not Found error in our browser when trying to access this URL:

http://our-ip:8090/__api__/filter/packages

Our Setup:

1)CI/CD: GitLab Pipeline (Runner on Windows, PowerShell).

2)Target: Docker container on a Linux server (Port 8090).

What we know:

1)The deployment job copies everything to a public folder and saves it as an artifact.

2)Checking the artifacts in GitLab UI, the file perfectly exists under public/__api__/filter/packages.

3)The pipeline works correctly.

The Problem:

When requesting the URL directly in the browser, the webserver returns a 404. Our hypothesis is that either the deployment transfer from the GitLab artifact to the Linux/Docker server silently drops the __api__ folder, or the webserver (Nginx/Apache) blocks access to folders starting with double underscores by default.

Has anyone experienced files/folders with __ being ignored during GitLab artifact transfers or blocked by default Linux webserver configs? Any hints on how to debug this?

Hey everyone,

I'm looking into tightening security on our Puppet infrastructure. Currently, our environment relies on autosign = true to handle ephemeral instances and autoscaling groups seamlessly.

Obviously, leaving naive auto-signing on is a massive security risk if someone requests a cert from an unauthorized node. However, setting autosign = false completely breaks our automated provisioning pipelines since we can't manually sign every instance.

For those running Puppet in AWS/Azure/GCP with dynamic infrastructure:

How are you handling secure auto-signing? Do you use policy-based validation (autosign.rb) with a challenge password, or have you migrated to something like JWT/OIDC tokens?

If you use a pre-shared secret/challenge password in your cloud-init scripts, how do you handle secret rotation securely without leaking it?

Are there any good open-source wrapper scripts or standard patterns you recommend for validating CSRs before the Puppet CA signs them?

Appreciate any advice or architectural patterns you can share!

Hello fellow colleagues.

First time configuring Load Balancing in an ECS service. Before I used to manually register targets to a Target Group and add it to a Load Balancer rule.

However I've been experiencing random issues ever since, especially when restarting/re-deploying such services. The targets appear healthy at first, but then the load balancer starts draining them, even though they appear to be healthy. It can repeat a couple of time until either it settles on its own, or I manually de-register and register the targets manually. Then it stays healthy.

I wonder if anyone experienced this before, or is more experienced on this than I am who can hint about what's going on?

Thanks.

EDIT: amazed by all the helpful responses, thanks a bunch!!

Not looking for complaints, genuinely curious about the specific moments where something about Jenkins behaviour surprised you and cost real time to debug.

Mine: discovering that a plugin update silently changed default timeout behaviour and nobody noticed until builds started randomly hanging.

What's yours?

Hi everyone,

​I have recently taken over the administration of an Azure DevOps Server 2020 (on-premises) environment. The previous administrator is no longer with the company, and unfortunately, there is no existing documentation regarding the backup strategy. It appears that no automated backups have been configured via the Administration Console so far.

​Environment Details:

​Version: Azure DevOps Server 2020.

​Scope: Single server instance containing one Collection with two active projects.

​Content: Includes source code (TFVC/Git) and several CI/CD YAML/Classic pipelines.

​Status: The environment is live and business-critical.

​My Goal:

I want to use the built-in Scheduled Backups tool within the Azure DevOps Administration Console to create a backup plan, including an initial full backup and subsequent scheduled increments.

​My Concerns:

Since I am new to this specific instance, I want to ensure that enabling the backup plan won't inadvertently disrupt the production services or lock any databases in a way that affects the pipelines or developer access.

​Specific Questions:

​Impact on Live Environment: Does the initial full backup via the Admin Console trigger any significant downtime or "Read-Only" states for the collections?

​Permissions: Besides the service account having sysadmin rights on SQL Server, are there any easily overlooked folder permissions required for the backup network share?

​TFS Integration: As there is still legacy source code on the instance, are there specific metadata files outside of the SQL databases that I need to manually include, or does the wizard cover all necessary components (databases + reporting + encryption keys)?

​Common Pitfalls: Are there any known issues when running the backup wizard for the first time on a "neglected" 2020 instance?

​I want to avoid breaking anything while securing the data. Any advice or checklists from experienced Azure DevOps admins would be greatly appreciated.

​Thanks in advance!

I'm trying to enable MQTT over TLS on port 8883 on a self-hosted ThingsBoard created on Ubuntu and running on Amazon Lightsail. As soon as I enable the below given commands..it shows this error: "Caused by: java.lang.RuntimeException:
MQTT SSL Credentials: Invalid SSL credentials configuration.
None of the PEM or KEYSTORE configurations can be used!"
but when these commands are turned off, everything works fine. I'm not able to enable 8883. MQTT port 1883 works fine when these commands are turned off.. otherwise the website goes down.
where am i going wrong?? I would love insights :(

MQTT_SSL_ENABLED=true
MQTT_SSL_BIND_PORT=8883
MQTT_SSL_PROTOCOL=TLSv1.2
MQTT_SSL_CREDENTIALS_TYPE=PEM
MQTT_SSL_PEM_CERT=/config/server_chain.pem
MQTT_SSL_PEM_KEY=/config/server.key

I’m facing a persistent ALB issue and need help isolating the root cause.

Setup

• AWS EC2 (Ubuntu)
• Docker Compose (3 services: frontend (nginx), backend (Node/Express), DB)
• Application Load Balancer (ALB)
• Target group → EC2 instance on port 80
• Health check path: / → Healthy

Architecture

Client → ALB → EC2:80 → Nginx (frontend container)
                       └── /api → backend:5000

What works

• curl http://localhost → 200 OK
• curl http://<private-ip> → 200 OK
• curl http://<public-ip> → 200 OK
• Browser via EC2 public IP → frontend loads correctly

What does NOT work

• curl http://<ALB-DNS> → 503 Service Unavailable
• Browser via ALB → same 503

Verified (not guesses)

• Target group has 1 healthy instance
• Listener: HTTP:80 → forwarding to correct target group
• No extra listener rules (only default)
• Security groups:
  • ALB SG → allows 80 from 0.0.0.0/0
  • EC2 SG → allows 80 from ALB SG
• EC2 and ALB are in same VPC + AZs
• Docker containers are running correctly

Important observation

Using tcpdump, I can see:

ALB → EC2 → GET /
EC2 → ALB → HTTP/1.1 200 OK

So:

• ALB reaches EC2
• EC2 responds correctly

Yet ALB still returns 503 to client.

Nginx config (frontend container)

server {
    listen 80;

    location / {
        root /usr/share/nginx/html;
        index index.html;
        try_files $uri $uri/ /index.html;
    }

    location /api {
        proxy_pass http://backend:5000;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
    }
}

My current suspicion

This seems like:

• ALB receives response but rejects it
• Possibly HTTP behavior / connection handling / headers issue

Question

What are the exact conditions where ALB:

• marks target as healthy
• successfully receives 200
• but still returns 503 to client?

What should I inspect next:

• ALB access logs?
• Nginx response headers / connection behavior?
• Something subtle in Docker networking?

Looking for precise debugging direction, not generic setup steps.

Thanks.

Hi everyone,

I’m trying to clarify whether TikTok’s Business Messaging API is currently available for businesses registered in Spain / the European Union.

My goal is to automate replies to TikTok DMs and integrate the messaging flow directly with our own internal system. We already have a verified TikTok Business Center, but I’m not looking to use a third-party inbox or chatbot provider if it can be avoided. Ideally, I’d like to request access to the API and build the implementation myself.

From what I’ve found, TikTok does have a Business Messaging API, but I’ve also seen some mentions that it may not be available for EEA / EU accounts. Does anyone know if this is currently possible for a Spanish business account?

Also, just in case: has anyone found a third-party provider that can legally and reliably offer TikTok DM automation in Spain / the EU? I assume access is restricted, but I’d like to confirm.

Thanks in advance.

A few things this covers that tripped me up,may be useful:

• The listen ... http2 directive is deprecated as of Nginx 1.25.1
• HTTP/3/QUIC is native in mainline now, no more compiling from source.
• If you're on Let's Encrypt, OCSP stapling is basicallly dead, they shut off their responders in August 2025, so ssl_stapling on; just throws a warning.

Curious what protocol split everyone's seeing and using in production?

Guys, I'm kind of at a loss here. My team wants to implement embedded analytics into our app using Sisense and I cannot make heads or tails of how you're actually supposed to.

Yes, development is all done inside the environment and it's handled locally, and it connects to git... but the constraints are so weird.

I've worked with other technologies like Databricks where everything done inside boils down to yaml files, and you can connect it to git, and develop on a feature branch and merge your config changes into main, and cut release branches to roll out the config into environments, but it seems like Sisense kind of doesn't understand this mentality.

First of all, everything is an "asset", and you have to create a "project" in order to add your assets to a repository. That's all fine an good. I can connect to my remote main branch and assets are automatically converted to json. But a project can only operate on one branch at a time. so if I cut a feature branch and check it out, everyone in that project is now on a feature branch!

That's fine, I'll just make a new project, connect it to the same repo... pull down main then cut a branch and work there, push the branch and PR it to main and ... NOPE! you can't pull down main in a separate project because assets can ONLY BE IN ONE PROJECT AT A TIME.

How about I create a new project, connect to the same repo, but add DIFFERENT assets, and just have that project track THOSE files? Nope, divergent history!

What about revoking a users access to a project after they've left the company? nah, that user OWNED those assets. poof, they're gone now from your project!

Any advice would be helpful: https://docs.sisense.com/main/SisenseLinux/introduction-to-sisense-git-integration.htm?tocpath=Git%20Integration%7C_____0

I’m choosing between Qwen3.6 27B and Qwen3.6 35B A3B for an AI agent that helps users solve everyday household tasks.

Right now I’m using Qwen3.6 27B via OpenRouter, but sometimes it takes around 10 seconds just to start responding to a simple "Hello!", even with streaming enabled.

My servers are hosted in the US, so I was thinking about switching to DeepInfra, but the traceroute to DeepInfra looks pretty long from my server.

Does anyone know a fast API provider for servers in the US where inference starts quickly! Ideally within 1–2 seconds for the first streamed token?

Also, which model would you choose for this type of household AI agent: Qwen3.6 27B or Qwen3.6 35B A3B?

Over the past week or so my Fargate Spot tasks are getting interrupted way more often in ap-south-1 than they used to. It was rare before, now it's near-constant. Nothing on the public status page and nothing in my personal Health Dashboard.

Is anyone else running Fargate in Mumbai seeing this right now, or is it just my account/AZ? Mainly trying to figure out if it's a transient capacity crunch or the new normal.

We were testing internal service-to-service communication via ECS Service Connect, but one thing I noticed was that after updating it in the ECS Service, the time it takes to decommission the ECS Task has increased significantly. Before, it used to take approximately 2-3 minutes, and now it's taking approximately 10 minutes.

Has anybody else faced a similar issue? How can I fix this? This has increased the overall pipeline time, which looks bad from the outside, and every deployment takes longer to get deployed.

I am trying to create a pipeline, I have a sql file inside db/migrations but when I execute my script I keep getting " schema "system" is up to date. No migrations applied". Anyone can help with this?

Universal 502 Bad Gateway responses across production API. Nginx ingress controller log: [error] 45#45: *1890201 no live upstreams while connecting to upstream, client: 10.1.4.55, server: api.prod.internal, request: "GET /v2/metrics HTTP/1.1", upstream: "http://api-backend".

K8s endpoints showed api-backend pods as Running and 100% Ready.

Assumed Nginx failed routing packets to pod IPs. Exec'd into Nginx pod, ran curl -I <pod-ip>:8080. TCP connection succeeded but returned 400 Bad Request.

Suspected asymmetric routing drops or MTU truncation on the Calico VXLAN tunnel corrupting the payload. Deep-dived iptables -t filter -L and tcpdump for 90 minutes looking for dropped fragments.

Looked at Envoy sidecars rejecting plain HTTP originating from Nginx due to PeerAuthentication strict mTLS policy. Verified destination rules. Applied PERMISSIVE mode. Upstream pool remained empty.

Nginx active health checks (health_check directive) were failing due to strict HTTP Host-header validation at the Go Fiber layer. Standard Nginx health checks request the URI using the upstream Pod IP as the default Host header (Host: 10.244.1.55).

Go application rejected requests lacking exact FQDN (Host: api.prod.internal) with 400 Bad Request. Nginx received 400, failed match condition expecting 2xx/3xx.

Incremented failure count to max_fails=3. Evicted every backend pod from the upstream pool. Round-robin found zero valid targets.

Fixed by defining a raw HTTP request string in the Nginx match block via the send directive. Forces health-check daemon to inject correct Host header expected by backend validation.

upstream api-backend {
    zone api-backend 64k;
    server 10.244.1.55:8080 max_fails=3 fail_timeout=10s;
    server 10.244.2.12:8080 max_fails=3 fail_timeout=10s;
}

match server_ok {
    # Send explicit HTTP literal to bypass IP-based Host header defaults
    send "GET /healthz HTTP/1.1\r\nHost: api.prod.internal\r\nConnection: close\r\n\r\n";
    expect ~* "200 OK";
}

server {
    listen 80;
    server_name api.prod.internal;

    location / {
        proxy_pass http://api-backend;
        proxy_set_header Host $host;
        health_check uri=/healthz match=server_ok mandatory;
    }
}

I apologize if this questions is not related to the group.

Hi everyone, I am a begineer in AWS and was following some courses in youtube. In this process, I noticed that I have $300+ dues to be paid although I made sure to close all the instances found out it was due to EKS clusters. It was an honest mistake and I want to see what my options are. Unfortunately, this is a very huge amount for me at this time. Futhermore, the cost this month (February) is projected to be $400+ but I have already deleted all the EKS cluster, volumes and instances.

I have opened a case in aws support but haven't heard back from them so that is why I am posting here to see if I have any other options. Your help will be greatly appreciated. Thank you!