r/ProgrammerHumor 2h ago

Meme [ Removed by moderator ]

Post image

[removed] — view removed post

Upvotes

281 comments sorted by

u/RedditDumpAcc 2h ago

is it an edge case if it crops up every night for three years?

u/jllauser 2h ago

It could be a one in a billion chance, but if you do a billion transactions a week...

u/Crafty_Independence 2h ago

This. One in a million happens multiple times a day at these scales

u/mxforest 2h ago

Good systems have contained blast radius. Edge cases are expected. Them taking down entire systems is not. The only visibility those edge cases should have should be a raised customer support ticket.

u/Crafty_Independence 2h ago

The post says "randomly failing" rather than taken down. That's pretty standard fare edge case symptoms right there

u/mxforest 2h ago

Payment service is failing. Not a single API request. These numbnuts wouldn't even notice a few failed API requests.

u/Crafty_Independence 2h ago

There are definitely severe organizational issues with this company

→ More replies (1)

u/SnugglyCoderGuy 1h ago

Emphasis on 'good'. Most systems are made by idiots and are not good.

u/Bardez 1h ago

Or are made under idiotic conditions. Every service, application, backbone I've ever worked on has been fairly well-designed, and then forced to go live before it was even remotely ready to do so. These engineers out here are making the best out of bad circumstances.

u/SnugglyCoderGuy 1h ago

That can also be the case. Mileage varies.

u/Bardez 1h ago

"Made by" also does some lifting here, as top-down factors and personnel are also part of the "team" that "made" the software.

u/SnugglyCoderGuy 1h ago

Doesn't seem to change the statement much. If top-down personnel are part of the "team" that "made" the software and they are idiots, then it still fits with what I said (I've been there under the thrall of the HIPPO that thinks they know things).

u/haywire-ES 1h ago

Most systems aren’t good they’re just good enough

→ More replies (1)

u/tgwombat 1h ago

I’ve got some bad news for you if you think the world is mostly running on good systems.

→ More replies (1)

u/Landoof-Ladi 2h ago

At that it’s not an edge case. It’s a scheduled feature.

u/Crafty_Independence 2h ago

Lol technically true. Edge case is whatever people want it to be, quite often.

u/Z4REN 1h ago

"Scientists have calculated that the chances of something so patently absurd actually existing are millions to one. But magicians have calculated that million-to-one chances crop up nine times out of ten."
— Terry Pratchett

u/namezam 2h ago

IMO “edge case” doesn’t mean the percent chance it’s how often it occurs. If something is happening every day it doesn’t seem too edgy.

u/sth128 1h ago

You're assuming these edge cases are all identical in their edgy conditions.

It could be edge cases of different nature most of which are unique in an unfixable/too-expensive-fix or a can't-be-fixed-without-breaking-10-million-things way.

u/1ii1i 1h ago

That and/or this issue has been raised over and over again but they cannot fix it due to downtime or cost requirements xyz. It should NOT be surprising to a lot here at this point how often this happens.

u/anto2554 1h ago

I regularly fix stuff manually because we didn't have the time to automate the fixes (simple things like outdated docker image references) because we haven't had time to automate a fix

u/1ii1i 42m ago edited 39m ago

I've seen worse from F500s as well. Had a lead engineer bring up an issue SIX months out. Needed some form of scheduled downtime to fix the issue before it gets worse. Emails sent for tracability and accountability, and promptly ignored or downplayed. Repeat email 3 months out, 2 months out, 1 month out, 2 weeks out, same story. Then it finally breaks, and guess what? All of a sudden they need him to work on it ASAP with no overtime becuase policy was OT needed to be requested two weeks ahead of time (salary worker). Lots of bickering, he said she said (when there is literally an email papertrail). The sad part is it caused significantly MORE downtime than just letting him fix the issue as he had originally stated. The afteraction was "neither here nor there" and of course leadership admits zero fault.

Edit: The person requested a transfer after half a year more of this BS. A manager he worked for (but was not supervised by) sent out nastygrams to the places he requested to transfer to. Still got the transfer, HR still gave him a "note" in his file for the nonsensical stuff that was made up against him in his file. They left and are doing much better now.

u/toastybred 1h ago

I think people are getting hung up on prevention (closing the edge case) when there was clearly an working solution for remediation. The big miss was not understanding what he was doing and taking the steps to automate his process to deal with the situation. The work he was doing was already baked into the operational costs and the best ROI would have been just tightening that up.

Not every engineering problem is worth solving.

→ More replies (4)
→ More replies (6)

u/MrKingsport 1h ago

We have a process like this. Certain records are being assigned to a former senior leader. It's built into a process nobody can find much less change. We run 1 report every week, click 2-3 buttons and it resolves the 1-2 records each week. It's in that awkward space between "can't ignore but not critical enough to dedicate more resources to researching." This was after spending a week trying to track it down and hitting wall after wall of "not my system not my problem" when it's very clearly somebody's system they just won't look at it to avoid taking responsibility...which I also understand lol.

u/BucketsAndBrackets 1h ago

There is no way one guy could do that many PRs withouth anyone accepting them if there is so many transactions.

I work in a company with 30ish devs and we have 20 millions of transactions by day, I remember it started bugging once in the 3 and the half years I'm here.

So if you fix same shit every day, it is on you.

u/Eisbaer811 1h ago

you assume it was infrastructure as code, which is tracked in PRs.

chances are, a stack of this "quality" gets manual changes directly via SSH, and nobody notices

u/ronlugge 54m ago

I'm on a project where the automated workflows don't handle some edge cases; we're working on closing it out, but a dev spent a major fraction of their time manually correcting individual items by investigating what error buckets they fell into and then applying the appropriate correction -- complete with coordinating with another team to fix their data each time it got pulled incorrectly. None of which is done by PR; we SSH and update DB values and reschedule workflows once the data is fixed.

Correcting the underlying workflows is on the board, but it just isn't a high enough of a priority yet. Other required features are just more important than fixing some of the edge cases we're seeing.

→ More replies (4)

u/tomvorlostriddle 2h ago

"User with your birthday already exists"

u/Happy-Fun-Ball 43m ago

password already taken

→ More replies (2)
→ More replies (3)

u/Jolly-joe 2h ago

Well you call it edge case if anyone finds out about it but it is effectively a dead man's switch in case you get fired.

u/bot403 1h ago

Didn't seem to help him not get fired. What's the purpose then? Revenge?

u/ScoobyPwnsOnU 19m ago

To be fair the next step could be "oh, u want me to come back? how much is it worth to you?..."

→ More replies (1)

u/Roth_Pond 2h ago

Edge of the capabilities of the process

u/Stummi 2h ago

maybe its always another edge case that pops up

u/Hmm_would_bang 2h ago

You wouldn’t understand. It’s a secret

u/AutonomousOrganism 1h ago

A permanent edge case.

u/Jello_Penguin_2956 1h ago

Everything is an edge case in the hands of manager/ceo who don't understand how it works.

u/SupahSpankeh 1h ago

Bro should've raised a ticket and/or fixed at source smh

→ More replies (1)

u/Gran_Duma 21m ago

With all those edges I think the system can be approximated as a circle.

u/vanzir 12m ago

Right? my first question was how the hell does an edge case that requires daily manual intervention an edge case?, and how the fuck does NOBODY ELSE on the project know about it?

u/Wizywig 5m ago

Yes. If someone is silently fixing something every night for 3 years... That's a failure. This should be documented and addressed. Even if not addressed a procedure should be known and in place.

There should be backup oncall rotation for vacations, if something happens and he disappears. You just can't run a functional eng org with procedures thst are voodoo done by individuals. 

u/dukeofgonzo 1h ago

Their monolith is a heptagon. The edge happens seven times a week.

u/Defie22 1h ago

I'm full time Edge case manager, so yes

u/Same_Instruction_100 1h ago

If he was fixing them, in the eyes of management, yes.

Source: I fix billing error 'edge cases'.

u/dougmcclean 50m ago

Is it engineering if you just redo the data entry and don't document the process or automate a fix?

→ More replies (3)

u/Eternal_Alooboi 2h ago

This is the same post from 3 months ago with the exact same title. Goddamn clankers.

https://www.reddit.com/r/ProgrammerHumor/s/EhfWxtyuAw

u/nabrok 2h ago

Pretty sure I first saw this long before 3 months ago.

u/gentlemandinosaur 2h ago

Yep this is a couple years for sure.

u/aVarangian 27m ago

my mother saw this post just before I was born

she complained it was a repost

→ More replies (1)

u/El_Mojo42 57m ago

Yeah, it's an old meme.

I'm fine when something gets reposted after a year or so. Theres always new people that haven't seen it.

Nowadays, the repost time is down to hours though.

→ More replies (1)

u/Amar0k171 1h ago

The post itself even reads like chatgpt wrote it.

u/Darhhaall 1h ago

No, this is far older than ChatGPT. Basically copypasta circulating net for decade at least.

→ More replies (1)

u/who_you_are 2h ago

We will repost it for at least 3 years!

→ More replies (1)

u/seedless0 1h ago

Mod should ban such obvious repost karma bots.

→ More replies (7)

u/MyFeetLookLikeHands 57m ago

yeap anytime they hide the date of the original tweet, you know it’s a repost of a repost

→ More replies (1)

u/King_Chochacho 43m ago

Reddit: the site by bots, for bots

It all looks like engagement so admins don't care!

→ More replies (1)

u/Fach-All-Religions 45m ago

yep i remember it

u/brick_gnarlson 29m ago

Reddit would be dead without bots anyway.

u/Jon-Robb 25m ago

Yeah but new edge cases showed up

→ More replies (5)

u/oldsecondhand 2h ago

Sounds like "insurance". Normally you wouldn't just fix that without telling anyone.

u/HomsarWasRight 2h ago

I’m honestly guessing he was explicitly asked to do this, but then the people who asked him left.

u/Shifter25 2h ago

Or, he'd told them about it, they decided it wasn't worth the effort to fix, and promptly forgot about it

u/lonestar-rasbryjamco 1h ago

Or that it wasn’t “product facing”.

u/Ed_Trucks_Head 1h ago

Oh man this is my job all day long. I don't know why even bother telling people about problems. Nothing happens. It's not even a difficult fix. They're just space cadets.

u/Shifter25 1h ago

That's part of why I don't see the point of AI code. Developers have never been the bottleneck for fixes or new features. It's the higher ups who want nothing done until they suddenly want everything done yesterday.

→ More replies (3)

u/yusrandpasswdisbad 13m ago

Can say with confidence this is it. Also - automating the fix with Task Scheduler only works if his computer is on.

u/TypicalOrca 2h ago

As soon as they left I would have been in someone's face telling them the bullshit work I've had to do for that period of time and tell them it is time to change this process.

u/ihvnnm 1h ago

I am feeling like that, but going on 20 years. Small company of +/- 25 people when I started, company got sold 5 years ago, now roughly 50 onsite, thousands elsewhere. There is now only 2 people who have been here longer than I, and they only started a few months before me. I know "where the bodies are buried" as being the sole IT/software developer for most of the time.

→ More replies (5)

u/Jolly-joe 2h ago

If I have to do a manual data fix more than a couple times, I'm bitching to my manager and hard swapping to a fix that sprint. Doing this for 3 years is absurd.

u/Ninth_ghost 2h ago

Even if it's a scanning+OCR error?

u/OnceMoreAndAgain 1h ago

Seriously though what is the solve for doctor's handwriting on paperwork for insurance claims? You can't OCR that shit lol. And if you force them into electric then they'll protest by leaving the PPO network since only the old, stubborn ones submit claims by paper.

Our solve is hiring low salary workers to do that work instead of having a developer do it.

u/vivaaprimavera 2h ago

All fixes are final /s

u/TypicalOrca 2h ago

No matter what it was. If nothing else you have to know when to push data back to the business and tell them to fix it.

u/Astramancer_ 1h ago

Even if it's a scanning+OCR error?

Absolutely. Even if the fix is "officially add this to someone's job responsibilities." A recurring ad-hoc fix is just asking for trouble. A formalized fix is harder to break accidentally.

u/TypicalOrca 2h ago

That's why I can't believe this or think that engineer must have been a junior and just never got past junior thinking.

u/kingslayerer 2h ago

"every night for 3 years"?

dont attribute to malice what can be attributed to incompetence

u/Sweaty-Willingness27 2h ago

Or made up for Internet clout

u/kingslayerer 1h ago

Idk. I have seen incompetence.

→ More replies (1)

u/GoBuffaloes 1h ago

I mean it's not insurance if you don't get anything out of it when they fire you. 

u/OkPosition4563 1h ago

I wouldnt try this here in Switzerland. This would definitely bite you in the ass once someone finds out and you could even be found liable for damages if you never reported this and especially if you didnt report this before leaving.

u/AdamAnderson320 17m ago

Nah, if anything it was a dead man's switch

u/Legitimate-Yard-9464 2h ago

Was he a good engineer is debatable.

u/SardScroll 2h ago

Agreed. If you're doing that level of manual fixing constantly, with no one knowing, as a staff engineer...something is wrong. Pull a junior engineer to fix the day to day and redesign the system or at least figure out how to automate the fixes.

u/laplongejr 1h ago

At my job we do that way since a decade because we aren't the ones tasked with redesigning the system. Joys of government and multipme independant groups.  

We had multiple proposals submitted in 2020 or so, all turned down because "the system runs fines for users"  

u/NamityName 36m ago

That's when you follow the bureaucratic process exactly in order to slow things down so it is now longer "fine for the user". A nightly fix like in the post would require a new service ticket every morning to fix. They clearly do not want anyone going above and beyond. So don't.

u/akatherder 6m ago

Not exactly the situation here, but the only time I would log in to fix something at night is when I know it will prevent me from having a shitty day tomorrow. If you stop the payment tonight, it will prevent you from having to run a bunch of cleanup junk, talk to miserable people, and explain why the software is bad.

u/Furyful_Fawful 22m ago

Odds are he had a local job set up to do it as nobody gave him the time of day to fix it the correct way

u/akatherder 13m ago

redesign the system

Oh, that's all.

We have legacy systems that we mostly understand, but to fully redesign would take hundreds of hours just to document all the requirements and hardcoded junk. And we'd still miss some.

"Hey boss we want to spent 14 months fully understanding and documenting how this works so we can redesign it and possibly miss stuff so it doesn't work right. It will save Tommy 5-10 mins a few evenings per week!"

You have been invited to a 15 minute meeting at 4:45pm this Friday, Accept?

u/Crossfire124 1m ago

Yea lmao. Just stop developing all new features and spend all your time on redoing the majority of the work people already done before you.

Rewriting is a noob trap that's rarely justified. Plenty of examples of companies heading down this route and realizing it's always much bigger work than initially thought

u/Final_Alps 2h ago

Precisely. We had this situation. (I was a peer not his manager, btw)

Plenty juniors felt “he’s indispensable. He manually fixes things”. At his level he should have fixed the need to manually fix things.

u/laplongejr 1h ago

He probably wanted to fix that need, but was overruled by his  manager?  

Part of why my boss (and his manual fixes) is leaving is because he's now convinced he'll never have a chance to remove that workload.  

u/KitchenFullOfCake 2h ago

Whether good or bad they were apparently load-bearing.

u/Grey_Raven 2h ago

Good is debatable, worth paying (or at least off boarding and replacing in a smarter way) is inarguable

→ More replies (1)

u/Fadamaka 2h ago

He was not. Nothing to debate.

u/Stopkilling0 42m ago

Tell me youve never worked with data as a cog in a legacy corporate machine without telling me. Often people in these roles do not have the ability to fix the underlying issue because it lies upstream or with a differient team. You think this guy wants to spend all this time manually correcting data? Probably not, I can almost gaurentee they raised the issue multiple times over the years but its just never prioritized by decision makers.

→ More replies (2)

u/SuperSquirrel13 2h ago

A dev that manually fixes a problem for three years? 

u/JeremyTwiggs 2h ago

They usually get a promotion at their next job.

I worked with someone who was fixing the CI build machine every morning before we got in.

u/SuperSquirrel13 2h ago

Needs a visit from the inquisition. Sounds a lot like heresy. 

u/laplongejr 1h ago

Sounds believable if the manager was awful and only cared about changing their own job.   Source : Gov job. We have this situation but more like a decade...  

u/thereallgr 14m ago

We had one customer like that. They had an ETL import that would fail every night because it couldn't resolve some keys because the import order was wrong. We couldn't fix the underlying issue, because the customer would only pay for fixes and features that they ordered us to do. They would happily pay us for the manual correction once the imported data got so stale that the users noticed, but they wouldn't pay us to fix the underlying issue that we already had diagnosed. Why? Apparently because it was a different budget for the 1st and second level support requests from their users and for our production hotfix.

→ More replies (1)

u/JPhi1618 2h ago

Since when do people getting laid off have farewell meetings, let alone a meeting for 12 whole minutes?

u/Nuggyfresh 1h ago

Literally never

u/0xBAFF0 36m ago

Our farewell meetings in my previous company were 2 hours at least, with coffee sweets and alcohol if it was a friday

→ More replies (1)
→ More replies (1)

u/CuckModerator69420 2h ago

In my experience, he told his boss about it 12 times over 6 months and it was never a priority. So he stopped asking and just maintained

u/MojeDrugieKonto 9m ago

It even might have been a checkout, run local script, push to some pipe, kind of a deal. Never important enought to warrant a global change, easy enough to do with simple bash script.

u/Goalie3533 9m ago

Possibly, but just as likely that was a developer who always completed ~95% of the task, but manually ran 1 thing behind the scenes, just to complete the project by the deadline. Always saying to himself while he's manually running it at 10pm, "I really need to get this automated."

u/elelyon3 9m ago

This is the more reasonable answer. Can't devote proper time to fixing it if it's not prioritized, so the less time consuming / maintaining fix is done regularly instead.

u/code_archeologist 1h ago

ROFL... that happened at a company I worked at years ago. I had spent over five years building automation to do my job, the scripts required small tweaks from time to time but they mostly worked without any action on my part.

But I hated my new manager, I was a core part of the client support and I hadn't received a raise in years. I tried transferring to another division for more career mobility and discovered he had sabotaged my application.

So I found another job (with a 120% raise, yeah I was being abused). When I put in my two weeks the shitty manager had me escorted out the same day... no knowledge transfer, just a short exit interview and "heave ho", because he thought that anybody should be able to run some scripts without a problem.

Four months later he had to hire two people to fill in the work I was doing. Each of them were demanding way more than what I was being paid at the time. Senior managers investigated what was happening and why costs suddenly exploded and the clients were angry. They fired the shitty manager 18 months after I left, with my exit being the primary cause for his termination.

u/Ponbe 1h ago

People reading stuff like this and then still being "nah we don't need unions" are beyond me

u/code_archeologist 1h ago edited 1h ago

Yeah, the only employment protection we have is the fact that senior developers tend to have a lot of mobility and can just leave if they aren't treated well. But yeah, there should probably be a union, guild, or professional organization (like IEEE, ASME, and ASCE). But the profession is still a little too "new".

→ More replies (3)

u/MissiveFinding6111 2h ago

I was this engineer for awhile.

My boss hated me because "whenever we talked all I said was negative"

So I just stopped. Stopped bringing up things that were about to fail. Stopped pointing out the problems with implementing his "ideas". I just blandly agreed. Waited for things to break to fix them instead of bumping other priorities to prevent outages.

My review 6 month later went from "needs improvement" to "exemplary".

Still no raise. I moved jobs about a week after that.

An instructive lesson that some bosses (bad ones IMHO) care more about what you say then what you actually do.

u/Gontarius 1h ago

The key moment when things are about to go to shit is when people who keep bringing up problems start agreeing with the management on all points.

It's my life's work to keep them complaining.

u/laplongejr 1h ago

One of my bosses's hardest challenge was getting me to FINALLY voice my disapprovals.  

→ More replies (1)

u/PiezoelectricityOne 1h ago

Same reason everybody hated Gandalf despite being the only one literally preventing an Apocalipse.

u/deusmetallum 2h ago

Turns out he deserved to be fired because he should have talked to the team about these issues he's been fixing every night for three years and getting them permanently resolved. Not talking to anybody probably means he's trying to cover his ass for other mistakes. Definitely a liability.

u/laplongejr 1h ago

 because he should have talked to the team about these issues he's been fixing every night for three years and getting them permanently resolved.  

I do a manual fix on every update of our software. I tell my boss every year. He forgets every year.  

u/LutimoDancer3459 1h ago

In one project the customer always fucked up the db because he wanted to "fix" some cases directly instead of telling us how to avoid it in the first place, or giving him the tools to fix it...

Its possible that the dev told others but got overruled and people forgot about it. 3 years isnt a short time. Management may also have changed in that period

u/wknight8111 2h ago

A "staff" level engineer who is manually doing something every night for years and doesn't take a few minutes to script up an automation is expendable.

u/laplongejr 1h ago

How do we know he could've scripted said automation? Possibly each case is different, or maybe he was told only manual fixes were allowed.   Ask me why I think it's possible...  

u/Troph_A 31m ago

Why do you think it's possible ?

u/laplongejr 22m ago

Because my boss fixes edge-case tickets since almost a decade because their dev team can't fix the issues properly :(  

We can't automate changes on legal data for... well, legal reasons.  

→ More replies (5)

u/l0wskilled 1h ago

If everything was fixed in a few minutes the world would be perfect.

u/Swarrlly 1h ago

It’s probably an issue that needs a major rework. Or it would require a downtime to implement. We’ve all had similar issues that never get fixed because management always says no. Eventually you stop asking and just keep things running.

u/wally-sage 45m ago

They're not always able to.

I worked on a large scale data engineering project for a FAANG company that essentially drove all the financials for the company. The system ran hundreds of data pipelines daily. It was so locked down and so issue prone that making any real changes was basically impossible since they required corporate sign-off. Most of the dev work had to be spent on fixing issues that pop up in the system, on-call was basically guaranteed to be completely miserable.

It was actually so shitty that I left the company and took a pay cut. I have no doubt that plenty of situations like the one in OP's post are caused by buearocracy, not bad engineers.

u/OldMcTaylor 11m ago

Two scenarios I've seen are:
1. They did script it but ran it locally and it was never prioritized as a real fix.
2. The data was unique enough that scripting wasn't really an option.

u/ClipboardCopyPaste 2h ago

New lesson learnt: WOL (Work Out Loud)

u/secularDruid 1h ago

bro had automated the thing 2 years 11 months and 3 weeks ago but sneakily deleted the script upon leaving lmao

u/Callec254 2h ago

There's probably a dozen things like this at my job, where I've said repeatedly that I wanted to fix them but it just never gets prioritized.

u/FightOnForUsc 1h ago

So this is why you force people to take vacations

→ More replies (1)

u/Outrageous-Machine-5 2h ago

A staff engineer that couldn't automate this edge case handling?

u/laplongejr 1h ago

Automation may require different permissions. Source : gov job, I manually run a script since 10 years because no server can access the shared folders where the output file must be put... But our dev workstations can write there.  

u/vivaaprimavera 2h ago

Are all edge cases equal?

u/YANGxGANG 2h ago

Does P = NP?

u/rakklle 1h ago

Leadership likes sexy projects, and operational stuff isn't sexy. Day to day operational stuff doesn't get fixed or updated. Then organizations always have some people constantly patching holes so the ship don't sink.

u/NeuroEats 1h ago

three years of quietly fixing it every single night and nobody noticed?? i’d be leaving that farewell meeting with the most evil little smile

u/Shinigami_kills 43m ago

He wasn't fixing anything. It was a dead man's switch😂

u/triggered__being 1h ago

I can guarantee he must have informed the issue multiple times either to the reporting manager or tl, I had one small issue related to normalising of the filename before uploading in s3 like NFC or NFD Which could have become a really big issue later on and it did and i reported this for like 2 months , and this fucker didn't listen to me

u/MisinformedGenius 1h ago

I really doubt a manager heard "I'm making changes to payment data every night" and just passed over it.

u/FatUglyUseless 1h ago

There are tons of managers that equate “I don’t have to tell people no, or get complaints” to “I’m doing an amazing job of being a manager” with out asking any questions.

u/Zav0d 2h ago

i do same but with expired api tokens, i silently renew it in a few month period.

u/Sooowasthinking 2h ago

Someone is about to find they are worth more than they thought previously.

u/Hungry-Wash-194 1h ago

Guy sounds like an idiot

u/Minute_Attempt3063 1h ago

He likely also reported it, was shrugged of as not important, and he remained fixing it for the fun of it.

Then he was removed and now that company is under fire, and the bug can't easily be fixed

u/ow_meer 1h ago

I am literally helping fix a similar problem right now. At least on our case it happens once a year.

Can we fix it forever? Of course, but the manager prefers having us chasing the next shiny feature that nobody will use rather than fixing 7 years old tech debt

u/bartekltg 1h ago

Paraphrasing Dr Strangelove: The whole point of the Doomsday Machine job security solution is lost, if you keep it a secret.

u/EuenovAyabayya 1h ago

nobody even knew

Except it was in his weekly status reports that his boss never opened.

u/Initial-Reading-2775 1h ago

Was he monkey-fixing that all three years even on vacations?

u/Lemortheureux 1h ago

Lol why not make a permanent fix? It's probably a good thing he's gone

→ More replies (1)

u/Orio_n 1h ago

Then everyone clapped ahh story

u/quantum-fitness 49m ago

Probably a good cost cut. The job of a staff engineer is not to solve those errors manually.

Ots either to solve the problem or get someone who isnt the most expensive employee to do it

u/drnullpointer 36m ago

I had this happen to me (ie. I was the guy being laid off who was propping everything up).

Our accounting application used a very strange synchronization system based on Novell Netware shared drives. This synchronization system had a bug which caused the datafiles to be corrupted.

This was fine for a typical use of this accounting application (single, maybe two users), but we had about 20 busy accountants furiously typing in documents. On average, once a week the database files would get corrupted and I would just silently fix the corruption.

I petitioned to buy a better accounting software, but they said that it is fine and won't spend money just to fix a bug that I can resolve myself.

So fast forward some time, they laid me off and escorted me out of the building. No chance to train a new guy, no chance to pass any information.

A week later they were trying to get in contact with me because the new guy failed to fix the corruption and instead basically killed all of the accounting data and then managed to break the backup too.

I refused, I already had a new job (that was 2000, even mentioning that you know anything landed you a job instantly).

u/AMissionFromDog 24m ago

That guy is sitting at home saying "Thank god I don't have to babysit their data ever again!"

u/quantumcoke 2h ago

Sounds like a shitty engineer.

u/jyajay2 1h ago

Sounds like an engineer who made sure they'd regret firing them

u/rover_G 2h ago

Why pay someone a staff engineer salary if they can’t correctly surface this kind of issue and get it fixed?

u/laplongejr 1h ago

Answer : because the manager is overpaid and doesn't understand why the engineer wants to put changes to a process that worked at fixing things since years...  

u/MisinformedGenius 1h ago

Not to mention expose the company to massive legal liability by editing payment data manually every night.

u/skadoodlee 2h ago

And then Codex fixed it within 3 minutes. Sleep well children!

u/CinematicUniversity 2h ago

As someone who fixes edge case issues that crop up regularly... He and I are both bad engineers

u/Sad-Woodpecker-7416 1h ago

That is a terrible engineer. If you’re manually fixing data corruption something in the system is buggy and obviously must be causing it. Rather than fixing the actual issue or at least making the team aware, it was just swept under the rug for 3 years. Terrible engineer.

u/createthiscom 2h ago

That's a bad engineer. He should have found a way to unit test the failing parts so the system could gain stability over time.

u/Cesalv 2h ago

"nobody is irreplaceable" try me...

u/TypicalOrca 2h ago

That's just dumb. Must have been a junior.

u/potatodioxide 1h ago

a well known recurring edge case = vendor lock

u/axisdork 1h ago

every laid off SWE's 2nd most wettest dream

u/redviper7579 1h ago

I'd argue that a good engineer would document the fix somewhere with all the fixes covered in it and store it in a common website (jira/confluence wiki) so that a simple search would find it.

A better engineer would debug the root issue and push the fix or at the very least have an automated way to apply the fix every time the issue surfaces.

u/stlcdr 1h ago

He wasn’t an engineer he was a janitor.

u/srb4 1h ago

Been there. I worked on a system just like this. It required manually data updates to the production database daily to fix a variety of issues. I raised plenty of noise about it, but management wanted to replace the entire system so they didn't want me wasting time fixing it. The replacement system never arrived and I was let go to cut costs. I really feel bad for the off-shore guy that got handled this thing with no clue how to keep it running.

u/KileiFedaykin 1h ago

Sounds like something an engineer who doesn't want to get laid off would post.

u/Conroman16 1h ago

Sadly corporations would rather this happen and it get fixed permanently than to keep that employee, no matter whether they knew he was doing it or not

u/Glum_Cheesecake9859 1h ago

We had a similar guy at our job, got laid off, found out he was running SQL queries for clients and non-technical users manually to fix stuff. No one knew about that until he was gone.

u/bluemountaintrees 1h ago

Somebody needs to fire the fucking CTO if this is true

u/tomcat2285 1h ago

Prevention never gets praised.

u/Kalandros 1h ago

The lunatics on LinkedIn have taken to calling this guy “Dave” and how the AI platforms will save you from having a “Dave” at your company.

u/Hot_Swan_1657 1h ago

Companies stupid enough to don't know what their employees do deserve to disappear.

u/CLOWNFLOGGER 1h ago

He was a horrible engineer and probably should have been fired sooner.

u/Every-Abroad-847 1h ago

I would argue if that’s what he was doing, he wasn’t good at his job …

u/awpt1mus 1h ago

This reads like LinkedIn slop

u/clauEB 1h ago

Not very staff if his solution was to manually fix data.

u/psychicesp 1h ago

This is exactly what people are trying to get rid of with AI. Companies are okay with having software engineers. What execs HATE is needing one particular engineer. Large enough companies have thousands of that "one particular engineer."

They have just enough fuck-you money to stand their ground and prevent from being starved out, and therefore ALL of the leverage.

Not saying AI is or isn't up to the job, but this is the hope they are shovelling so much money towards.

u/emcoffey3 1h ago

I've had a few coworkers like this. One was a developer who inherited a bunch of legacy stuff when others either left or were let go. She was an incredibly hard worker, but she was also responsible for way too much. When she left, all hell broke loose. Amongst the many surprises she left for us was a myriad of undocumented manual processes she was just taking care of everyday.

u/Pfhoenix 1h ago

A real Edge of Tomorrow situation.

u/Juanbolastristes 1h ago

You should always leave a logic bomb lying dormant... you'll keep postponing it until you get fired... dead man's switch...

u/UpvoteCircleJerk 58m ago

his name ? albrecht ramstein

u/Ronnoc527 57m ago

He was securing nightly OT.

u/Quailman_z 55m ago

I know this is funny, but also...maybe he wouldn't have gotten laid off if he told the company that he was solving endless problems for them lol

u/Lazy-Restaurant-5520 52m ago

Did they hire him back with a raise?

u/bounceswoosh 48m ago

Often, someone "irreplaceable" quits; everyone panics, and then it turns out that what the system really needed was fresh eyes. This can happen even within a team, because everyone defers to the initial developer and considers parts of the system voodoo.

u/arstin 43m ago

Glad to see there is still some rage in this ol' bait.

u/smokeabowlofbud 39m ago

That's what happens when firing decisions are made by people who don't actually understand what employees do.

My employer had to learn that the hard way. We had a round of firings and refusal to backfill positions of people who quit, managed to hold it together for most of a year, then had one final essential engineer quit and suddenly entire basic processes were impossible--nobody knew how to do certain things or use certain tools, corporate realized that without certain certifications we had lost we'd lose relationships with some of our vendors at the next fiscal year...we ended up literally paying a competitor to do our own work whitelabeled under our name for about 6 months before we were able to get people hired and trained to be able to actually provide the services we sell again.

u/phoenix5irre 36m ago

Why not just fix the edge case🤔

u/WhoSc3w3dDaP00ch 33m ago

So... does he now make bank consulting for them?

u/AwkwardWaltz3996 21m ago

This is why you should replace staff every 3 weeks. /j

u/snipsuper415 19m ago

Funny in the 90s, but bad management and bad development team today. That system should have been overhauled so data wasn’t corrupted every day for 3 years. Like bro, sounds like we’re just trying to look useful instead of being a good engineer. Not to mention there should have been a manager low enough to identify this issues from their devs to help unblock them and put them into more meaningful tasks.

u/Acceptable-Drive3781 13m ago

I'd fire him too if he has to go in and correct the problems every night he should have fixed in the first place.

u/Ok_Net_1674 9m ago

Gschichten ausm Paulanergarten

u/maybeinoregon 2m ago

I bet donuts to dollars that company ran on silos. Department silos, employee silos, etc.

Poor leadership such as in the Peter Principle, easily leads to employees or departments filling in the gaps.

It happens almost naturally as there is no one to report to that can actually address the problem - this is proven and learned employee wisdom that may or even may not be true - so someone comes up with a fix.

The fix works. The fix forever hides the problem that existed. The fixer gets recognition from within the company, as the only employee that can run / fix / make this thing work, etc etc. Worse case scenario, no one knows how / who fixed the problem that rears its head after retirement / illness.

It’s a problem as old as time itself lol