r/sysadmin 11h ago

ai threat intelligence going predictive in 2026 sounds like nightmare fuel for small teams like mine, thoughts?

Upvotes

everywhere i look now articles on cti evolving into agentic ai that predicts attacks from ttp patterns and blends it with our vuln data. sounds great if you’ve got a full security team, but im basically solo it at a 30 person shop and barely keeping up with the basics as it is. what worries me is not just missing something, but the opposite too. what if these systems start flagging the wrong stuff and i end up chasing noise while something real slips through. or worse, ai powered impersonation gets through looking normal because it mimics behavior well enough. we dont have a soc, just an msp and they mostly stay reactive unless something is already broken. im trying to figure out what actually makes sense at our size. do i even think about collective defense stuff like isac sharing or is that overkill?

it kind of feels like the early k8s days where solving one problem just unlocked ten more you now had to manage. i havent had a major incident yet but this shift feels like it could create one if i get it wrong. for those in small teams, what are you actually doing right now that feels realistic and not over engineered and what would you prioritize before 2026 really hits?


r/sysadmin 8h ago

Question How do you deal with physical fatigue after long on-call stretches at your desk?

Upvotes

After a rough on-call week, I'm realizing my setup is not built for marathon sessions. Wrists, neck, lower back — all taking a beating. What have you added to your workstation that actually helps? Wrist rests, monitor risers, anti-fatigue mats, lumbar cushions?


r/sysadmin 20h ago

Beware when using existing meeting ownership transfer scripts

Upvotes

I noticed something that is easy to miss when transferring bulk meetings ownership. Several meeting transfer scripts available online retrieve meetings based on a specific date range. That can cause problems with recurring meetings.

A recurring meeting may have started months or even years ago, but still have future occurrences. If the script only retrieves meetings within the specified date range, it may not pick up that recurring meeting.

The transfer can then appear successful while some recurring meetings remain with the original organizer.

One thing to check is whether the script uses the Microsoft Graph Get-MgUserCalendarView cmdlet to retrieve upcoming calendar occurrences. This helps identify recurring meetings that are still active, even when the series originally started much earlier.

So, before using a meeting transfer script, check which cmdlet it uses to retrieve meeting details.


r/sysadmin 15h ago

Question System Admin Interview, What Should I Expect?

Upvotes

Hi

I have an interview tomorrow, and someone who is currently part of their team gave me a few hints about what to focus on. He mentioned Exchange, VMs, AD, deployment and patching.

I have around four years of experience, mainly focused on technical support with some exposure to networking. The role I’m interviewing for is a System Administrator, and this transition is veery important to me. I’d like to know what kind of technical questions I might expect in the interview??


r/sysadmin 18h ago

Question F3 users need to edit files on an on-prem SMB share. SharePoint migration not an option yet. any ideas?

Upvotes

Setting up a shared (multi-user) Entra-joined Windows laptop for healthcare staff (nurses mainly, no doctors) using Autopilot Self-Deploying mode.

No hybrid join, no AD computer object, but on-prem SSO to our file server works fine (hybrid identities, Kerberos/NTLM via the standard "SSO to on-premises resources" mechanism).

Licensing is a mix of F3 and E3 users on the same device. We're using Shared Computer Activation for Microsoft 365 Apps so desktop Office works fine for E3 users, and pointing F3 users to Office for the web (Outlook/Word/Excel Online) via forced Edge webapp installs.

NOW, our issue: our healthcare content isn't in SharePoint/OneDrive yet (separate migration project, not happening soon). F3 users can browse and copy files from the on-prem share fine through Explorer, but Office for the web obviously can't open/edit anything directly from a local/network drive only from SharePoint/OneDrive.

Options I'm aware of so far:

  1. Migrate the content to SharePoint (the "real" fix, but a separate project, not happening in the short term)
  2. Device-activated Office LTSC via KMS, so it's independent of the signed-in user's license (also on hold also no KMS infra ready yet)
  3. Third-party gateway products (e.g. MyWorkDrive) that let Office Online edit files that physically stay on an on-prem file share

Has anyone actually solved this in production? I'm open for suggestions.


r/sysadmin 17h ago

General Discussion Small Rant: Windows Activation

Upvotes

TLDR: Offline activation via https://aka.ms/aoh requires a captcha, MS account login and 2FA once per client, with ~80 to go. Any tips on speeding this up?

I'm just kinda curious about the opinion of other Sysadmins that have to work with Windows clients.

I work as a sysadmin at the factory of a fairly large company, where we mainly use Windows for our production floor clients. To extend the amount of support we get for each client, we have a standardized image, which gets updated every few years for a new release of IoT Enterprise LTSB / LTSC.

To get an image of what I am doing right now: (spoilering this part as it isn't too important to my rant) I was tasked with executing our project of updating out-of-date 2016 LTSB clients to 21H2 LTSC, as the new version will allow security updates for us until 2032. Some of our clients are former Windows 7 clients that aren't even officially compatible with Windows 10 according to the manufacturer, but Windows 10 might run on them, so to save money we keep the clients as long as they will run (meaning they will be phased out with Windows 11). We are also jumping from 1607 directly to 21H2, which Microsoft has explicitly advised us is not the official procedure or supported by them. But, updating between each version and needing a license would be a way higher financial impact than how we're currently going about it.

So, since our clients aren't connected to the internet, they cannot connect to Windows servers for activation. As such, I have to use slmgr.exe and SLUI 4 to activate the clients (formerly using the hotline, nowadays just the website https://aka.ms/aoh).

This activation requires you to put in a long string of numbers that the UI will show you into a field on the website and to reach that website you must always 1. Solve a Captcha and 2. Log into your Microsoft account, always forcing 2FA confirmation. And it doesn't remember your Microsoft Account, ever. You always have to log in again, from the start, solving the captcha. There is no button to activate another client once you finish activating the one you're currently working on, you need to re-load the link and start again. And again. And again. I've already optimized my current workflow as well as I can, using a python script to generate the QR codes I need for given commands so that I can avoid dealing with typing in the same commands over and over. But every time, I have to spend around 2-5 minutes dealing with the online Microsoft activation process.

Btw, if your connection is lost, you connect to a different AP or such, somehow the website doesn't require reauthentication. I don't know how exactly the process here works, but I can log into a tab for the site in one browser, open another in an incognito tab, open the next in a third browser, etc. And it'll usually last all day, so if I logged in and didn't use the session 'til the evening, it won't require reauthentication. My guess is that the endpoint / API request in the background actually works without authentication, and the whole login process is just a security circus akin to the TSA, and that if I knew a bit better about how web development works, I could probably just find out how the API request to Microsoft servers work to skip authentication. But I don't wanna risk getting in legal trouble just because I am not following the officially mandated license activation procedure.

Does anyone have some advice for me on how to save some time doing this? I got around 80 clients left to go and am pretty tired of the whole process.


r/sysadmin 23h ago

Young part time worker came in with Arch Linux on his laptop. He needs extra help with ton of things. I grow impatient.

Upvotes

I am kind of glad that he is trying to learn something. Don't take me wrong I would be happy to help with Ubuntu or Debian setup given that it is going to take some little extra time and in exchange we all learn something. I myself use Mint and Mac OS. I see the benefit. But I can't with clear consience tell the customer that we use the time meaningfully if we are spending 30 minutes trying to set up his vpn client.

We are here to work not to spend hours trying to learn core linux features on our paid time.

What would you guys do? I told him that I would be much happier if he used some more user firendly distro. Doesn't have to be debian based. Just something kind of works.

Edit: The laptop is a work laptop. I offer new guys option to use whatever os they like. If they reinstall their machine and choose to use Linux, MacOS or Windows is up to them. This is the first time someone came in with arch.


r/sysadmin 19h ago

Another M365 Tenant blocked (TenantAccessBlockedException, MCA Billing Account "Under Review", All Licenses Disabled) Microsoft 365 Tenant blocked, this time a non-profit that provides needs to at risk individuals and families.

Upvotes

In similar veins to the following two recent Reddit posts, I'm posting here because it appears they gained traction with Microsoft and ultimately reached someone who could help. I'm hoping someone from Microsoft or someone who has experienced this exact issue will see this.

https://www.reddit.com/r/sysadmin/comments/1vfbvvs/our_entire_m365_tenant_has_been_deauthenticated/

https://www.reddit.com/r/sysadmin/comments/1w1qc0i/microsoft_strikes_again_entire_m365_tenant_has/

Our tenant has not been deauthenticated, but it has effectively been rendered unusable. This tenant has been active since: 5/28/2013, so it's not a new tenant.

This is a frontline nonprofit social safety net organization providing food assistance, healthcare access, emergency financial assistance, and other critical services. This outage is impacting real people with urgent needs, including eviction prevention, time-sensitive clinic appointments, and emergency assistance cases. Email, SharePoint, and OneDrive are core operational systems for this organization.

We currently have open cases with:

  • Microsoft Technical Support
  • Microsoft Billing Support
  • Microsoft Nonprofit Support

So far we remain stuck at Tier 1 support. The representatives have been professional and are trying to help, but nobody we've reached has had the authority or access needed to resolve the issue.

9/3/2026

Users began reporting that they could not send or receive external email.

Internal email continued to work.

Microsoft had ongoing Exchange Online incidents at the time (EX1464935 and later EX1467029), so initially we believed it might be related.

9/4/2026

Users could still successfully authenticate through Microsoft 365 and SSO.

However, attempting to launch services such as:

  • Outlook Online
  • SharePoint
  • OneDrive
  • Other Microsoft 365 workloads

results in errors.

Any inbound email sent to the tenant bounces back after approximately 24 hours.

The error when trying to access Outlook:

Microsoft.Exchange.Data.Storage.TenantAccessBlockedException

Additional error details:

Client Version: 20260821009.11

BootResult: configuration

Back Filled Errors:

Unhandled Rejection: Error: 500:undefined

Unhandled Rejection: SyntaxError: JSON.parse: unexpected character at line 2 column 1 of the JSON data

err: Microsoft.Exchange.Data.Storage.TenantAccessBlockedException

esrc: StartupData

et: ServerError

st: 500

ehk: X-OWA-Error

ewsver: 15.21.382.9

9/5/2026

One Microsoft manager responded to an escalation email and reviewed the issue with us via remote session.

Screenshots and information were provided.

We were told an escalation attempt would be made.

I sincerely appreciate that effort, but as of today the tenant remains inaccessible.

---

While troubleshooting, we discovered that the organization's Microsoft Customer Agreement (MCA) billing account shows:

Status: Under Review

All billing profiles underneath the MCA also show:

Under Review

The billing portal displays:

Your account is under review. We're checking to make sure we can offer you Microsoft products and services.

The confusing part is that Microsoft also states:

This review won't affect your current services.

Unfortunately, that is not what we are experiencing.

------

Every subscription in the tenant appears to have been marked as Disabled on 8/31/2026.

Examples include:

  • Microsoft 365 Business Premium (Nonprofit Staff Pricing)
  • Office 365 E3 (Nonprofit Pricing)
  • Microsoft Teams Premium (Nonprofit Pricing)

All show:

Status: Disabled

Effective Date: 8/31/2026

The licenses still exist.

They are still assigned.

However, they are disabled and cannot be re-enabled.

As a result:

  • User access is broken
  • Exchange Online is inaccessible
  • SharePoint is inaccessible
  • OneDrive is inaccessible
  • Email delivery has stopped
  • Data is being reported as pending deletion because there are no active licenses

------

The tenant is fully paid.

There are no outstanding invoices.

There have never been any payment issues.

The MCA account contains the organization's EIN

We attempted to:

  • Add a new payment method
  • Add a new billing profile
  • Purchase replacement licenses
  • Purchase commercial licenses

All attempts are blocked.

The portal returns:

We can't authorize your billing account right now.

Actions will be blocked during this time.

Check back later.

Additional Information

We also see notifications regarding:

  • Account Under Review
  • Email Verification Required

However, all relevant controls are greyed out and cannot be modified.

------

At this point, the evidence suggests this is not:

  • A payment issue
  • A license assignment issue
  • A DNS issue
  • An authentication issue
  • An Exchange configuration issue

The combination of:

  • TenantAccessBlockedException
  • MCA Billing Account showing Under Review
  • Every subscription becoming Disabled on the same date
  • Users being unable to access any Microsoft 365 services

makes this appear to be a Microsoft-side account verification or tenant restriction issue.

If anyone has experienced this before, knows the correct Microsoft escalation path, or can help get this in front of the appropriate engineering or commerce team, I would be extremely grateful.

This organization provides critical services to vulnerable individuals and families, and every additional day of downtime has real-world consequences.

(Again, we have 3 tickets open, and I've attempted to reach out to Support at: 1-800-865-9408 (yesterday I did speak to someone from the Data Safety Team, they said they were unable to help but would transfer me to the Team that could do it but they were probably not available due to the weekend, no one answered after being on hold for just under 8 hours, I'm calling again this morning).

Thank you in advance.

(MFA and Conditional Access is on the tenant, we have Cloud [3rd Party] backups of the tenant, we have ITDR, there are no signs of compromise, this appears to be something that got flagged for review by Microsoft's back end and has put this non profit to a screeching halt).

Edit: Edited to add 2 messages that didn't show up once the post went live, and removing 2 bold words.


r/sysadmin 19h ago

General Discussion Best practices, tips, and tools for racking heavy-ass network gear (when working solo)?

Upvotes

Racking a lightweight 24-port PoE switch or a patch panel is easy enough, but trying to mount a 50lb core switch, a heavy enterprise router, or a massive UPS by yourself can be a nightmare and turns into a two-person job quickly.

Any weird DIY rigs, specific rack shelf models, or mounting tricks you use when you're stuck doing an install alone in a remote location?


r/sysadmin 22h ago

Question What laptops are you standardizing on in 2026 with prices where they are?

Upvotes

We're a medium sized NGO enviroment and I'm having a hard time finding something in price range. We used to budget around ~800$ for each laptop, this seems highly unlikely the average specs nowadays.

What is everyone going with? I used to love good ol Lenono ThinkPads. =/


r/sysadmin 15h ago

Better Auth 1.7 issuer change just broke every login on my Saas and it was already late when I found out

Upvotes

Out of nowhere this popped up like the support tckets rolling in and people complaining they cant log in, when i tried for myself, i couldn't log in to it either as it showed no accounts for this name was found. Nothing was at stake from my side like no errors in the logs no alert or anything, everything seemed ok

SO when I dug in manually it was better auth 1.7 which they changed on how accounts are keyed from providerId to issuer, accountID so the account table needs a required issuer column now, irritating! my existing rows didn't have it so nothing matched anymore and no exception was thrown just an empty match and how am i supposed to find out what caused it . then added the column nullable, backfill it and enforce not null then add the unique index-

ALTER TABLE "Account" ADD COLUMN "issuer" TEXT;

-- local:credential for email/password, local:oauth:<provider> for oauth
UPDATE "Account"
SET "issuer" = CASE
WHEN "providerId" = 'credential' THEN 'local:credential'
ELSE 'local:oauth:' || "providerId"
END
WHERE "issuer" IS NULL;

ALTER TABLE "Account" ALTER COLUMN "issuer" SET NOT NULL;

CREATE UNIQUE INDEX "Account_issuer_accountId_key" ON "Account"("issuer", "accountId");

Im not looking into dropping better auth over this since already fixed it but pretty annoyed at them but this gave me a thought tho that there is still a blind spot left open here , an external dependency quietly changed something and the failure was silent while my end didn't catch it but the customers did

What do you guys approach on to hear on from your side first

  • does normal error or runtime monitoring even catch a no error empty match, checking hud and other runtime tools but im not sure passive monitoring flags a silent one like this
  • or is the real answer just a synthetic login canary like something that logs in as a test user every few minutes and alerts when it fails
  • also does anyone alert on dependency shipping a breaking change before it hits prod??

r/sysadmin 7h ago

What is your SSL certificate provider doing now that 200 day certs are starting to expire? My provider (Namecheap) is being sketchy.

Upvotes

We have 50 or so WatchGuard firewalls at sites that have certs installed. Getting WatchGuard to implement cert automation has been the most user requested feature for about 2 years now. Until they come through we continue to purchase and manually install certs.

Last week I started getting notifications from our SSL source, Namecheap, that the first batch of 200-day certs was about to expire. I had cert request files at the ready and logged into Namecheap. Each expiring cert showed the upcoming expiration date and an icon to purchase a new cert.

But...wait a minute. Back in March, didn't I pay for 365 days even if the issued cert was for 200 days? Why doesn't the page clearly indicate that I can get a re-issue for the remaining 165 days that I paid for?

I opened a support ticket with Namecheap. They confirmed that I can get a new cert with a 165 day life rather than pay for a new cert. I asked why they don't make this open option more clear. They asked me to send them screen shots showing the expiration date and the little "buy new cert" icon. Like they don't know what their web site looks like?

I sent the screen shot and got a reply along the lines of "Well, you can just request a rekey, ya know. You don't need to buy a new cert."

Namecheap certs are cheap enough but all in that's an extra $500 or so that I don't need to spend. I find their behavior to be...distasteful at best.

What are the larger/more expensive providers such as DigiCert doing about this? Maybe even Godaddy is being more up front about the 365 day purchase vs. the 200 day cert.


r/sysadmin 17h ago

Question Cowork sandbox fails to provision — "useradd failed: exit status 12", every bash call broken

Upvotes

Anyone else hitting this? Every bash call in my Cowork session fails at the "ensure user" step before any of my code even runs.

What I'm seeing:

  • Error: RPC error -1: ensure user: useradd failed: exit status 12: useradd: cannot create directory /sessions/...
  • Happens on resume, on create, and on re-resume — all three attempts fail identically.
  • Started a brand new conversation to get a fresh sandbox, same error persists.
  • No shell access at all as a result, so nothing that depends on the Linux workspace (scripts, file processing, scheduled task setup) works.

Exit status 12 from useradd usually means "can't create home directory," which points to something like disk-full or a permissions/quota issue on the provisioning side, but I have no way to confirm that since I can't get a shell to check df -h or anything else.

Has anyone else run into this? Any known fix, or is this just an outage on Anthropic's end right now? Happy to share more error output if it helps.


r/sysadmin 20h ago

Outlook issues again?

Upvotes

Hi - anybody facing outlook issue again in exchange online, specifically not being able to search in either owa or the outlook client?

We have a number of users reporting.

Same issue as last week.

Thanks


r/sysadmin 5h ago

Workplace Conditions Relocating tech department

Upvotes

Our admin has decided to move the tech department so they can use our offices as classrooms.

The likely location is between a literal shed, a small room in central office, or a rotting building that's more rats, black mold, and abestos than building.

None of these options are appropriate for a tech department that actually does work. I'm also older and not in the best physical shape, so the shed is likely my best option.

Everyone else has decided to quit as soon as they announce it.. I don't have the savings to do that immediately.

Anyone else run into this kind of mess?


r/sysadmin 1h ago

Question MySQL ODBC stopped working overnight

Upvotes

You guys will love this.

This company has an in-house project management system. It's the core of their business, and they are lost without it. They are aware it needs to be migrated to something more modern, but after 5 years, that project still hasn't started.

I was asked to look into a network issue, but this isn't network but SSL I think. Let's first show the architecture:

  • The server is a CentOS 7 running MySQL Community Edition 5.7.16
  • Clients connect from Windows 11 with a 32-bit MS Access, using a 32-bit MySQL ODBC driver v5.3.13

Since yesterday, they get a "protocol version mismatch". The server wasn't accessed since 18 October 2016 (haha), so I presumed a Windows update might have disabled some SSL version. But: I see no relevant Windows update, and if I manually allow every possible SSL version and encryption algorithm, it still doesn't work. What does work however, is downgrading the ODBC driver from version 5.3.13 (from 2019) to version 5.1.13 (from 2013), further adding to my confusion.

The cherry on top: the single guy responsible for this application is on a one year sabbatical.

Edit: Found it, but leaving this here for anyone stumbling on the same issue. The MySQL_Server_5.7.15_Auto_Generated_CA_Certificate had expired after 10 years


r/sysadmin 7h ago

Auth0 alternatives?

Upvotes

Wondering what people are using instead of Auth0 for service account or within program api authentication? Their billing is killing us


r/sysadmin 12h ago

General Discussion Need advice on new job in a manufacturing facility. Going from MSP to a private company.

Upvotes

I was approached out of the blue on Linkedin by a recruiter hiring for a cyber/infrastructure role. The base pay was about 15k more than I make now plus a bonus of 30%. It would put me about 85k above what I make now.

I work remote currently so no commute. The base pay and bonus would be enough to cover a commute and put me over what I am making now.

Here is where it gets interesting and where I need the advice. They have NO active directory or ticketing system. They have about 450~ employees and a few hundred endpoints. They have ESXi for a hypervisor on Dell VX rail servers. A few linux servers run the ERP application and Synolgy NAS is used for Backup/storage. I get to pick new firewall/switches/APs/servers for infra upgrades and implement a migration to O365 for Defender, email, DLP, etc. Currtly they are using google workspace for email. I would be in charge of the migration from Google to O365. They also have no patching system in place other than general windows updates.

Network is cisco switches and a Palo firewall with some older Cisco APs. They currently have a 4ish member IT team with 2 juniors who handle helpdesk. I am being brought on to help modernize infrastructure and to implement some kind of cybersec/security compliance and standards. I get to build the solution and implement it along with a ticketing system and either on prem DC or Entra, while also implementing SCCM or Intune for windows patches and updates.

My background is networking, wireless, switching & routing, with sysadmin experience with ESXi, M365, server deployment and desktop support. I spent a few years in a service desk role and also in a sysadmin role. I currently work for an MSP doing mostly firewall installs.

During our discussions, the IT manager said that since he is busy and can't do things immediately there has developed a shadow IT situation where some departments are just doing things in a way they want without involving IT. 

My question, is this a bad idea? Leaving a remote role at an MSP with an increasing focus on KPIs or move over to this new and exciting adventure with a company with older gear that I get to help upgrade and be part of.

I like my current team and manager but the MSP lifestyle has gotten a little old after 5 years.

ETA:One big thing worth mentioning is that this role was created after PE purchased the company but the company is still run locally.

PE is providing the budget and I get to pick the direction and the hardware