Avoid adding new library to project: 10-year retrospective

[u/Xaneris47]

https://pvs-studio.com/en/blog/posts/1408/

[u/m_adduci]

Interesting, but isn't the usage of already made bricks one of the rules of Engineering?

Keeping a project Dependency-free spawns another class of problems and leads to the "not invented here" syndrome. You don't want to do your own cryptography, you don't want to create a new XML or Json Parser, you don't want to implement Database ORM yourself.

In an era where business is what at the end of the day matters, sinking too much time in getting things correctly and safely done might be counterproductive, especially in a time where AI-fans and enthusiasts (mostly from management levels) bring the argument that "AI is faster", because AI uses exactly those already made blocks.

[u/Due_Cap_7720 → u/m_adduci]

To keep with the analogy I think some libraries and frameworks are less like bricks and more like prebuilt homes. Too many decisions have already been made and trying to reconstruct them for your needs becomes counterproductive.

[u/turtle_dragonfly → u/Due_Cap_7720]

To me, this is one of the key distinctions between a "library" and "framework." The library is used by your code. Your code is driving. The framework uses your code. The framework is driving.

I do much prefer library-style 3rd-party code than framework-style. You can use lots of libraries, but using more than 1 framework is trouble town.

[u/Finickyflame → u/turtle_dragonfly]

That's also the way we go with our internal packages. Provide libraries/tools and don't try to control the way people work via frameworks. It's also easier to switch between libraries that offer similar features than to rewrite your application when you swap framework.

[u/amroamroamro → u/m_adduci]

Keeping a project Dependency-free spawns another class of problems and leads to the "not invented here" syndrome. You don't want to do your own cryptography, you don't want to create a new XML or Json Parser, you don't want to implement Database ORM yourself.

original post addressed this, with the PNG example:

Again, I should emphasize; I don't say that we should stop using third-party libraries at all. If we have to work with images in PNG format in the program, we'll take the LibPNG library, and not reinvent the wheel.

But even working with PNG we need to stop and think. Do we really need a library? What do we want to do with the images? If the task is just to save an image in *.png file, you can get by with system functions. For example, if you have a Windows application, you could use WIC. And if you're already using an MFC library, there is no need to make the code more sophisticated, because there's a Cimage class (see the discussion on Stack Overflow). Minus one library—great!

they are saying stop for a moment and consider the existing system api, if it it provides the functionality you need, do you really need another external dependency

[u/gordonmessmer → u/m_adduci]

The article isn't saying "don't use shared libraries," it's laying out specific characteristics you should evaluate before you decide to use a shared library and reminding developers to make sure they trust the shared library's developers.

Remember that using a shared library effectively extends your application's commit access to the developers of the shared library, and to anyone they give commit access to in the future.

[u/SnugglyCoderGuy → u/m_adduci]

With all things engineering the answer is "It depends".

The left pad incident comes to mind.

It's not just about reinventing the wheel, its about the implications of having external dependencies. There are rugs pulls to eorry about like left-pad, and supply chain attacks, and dependencies dictating how you structure your business logic instead of your business logic dictating your dependencies.

[u/george-silva → u/m_adduci]

I work in a place with heavy nih syndrome. My team uses the tools available to get the job done. The other teams rely heavily on reinvented squareish wheels.

Guess which team never missed a deadline?

While the work is similar (web apis, data processing, etc) our team is more productive by miles. Being more productive also leads to more money being made.

These people were against docker and chose guix for example. We had a migration to be made to gcp. Guess which team migrated it fast and cleanly ? Guess which team still struggles with this (not done yet last Ive heard).

We're paid to deliver value, not to chase our own engineering dreams. Sometimes those intersect. Sometimes it doesnt. Stick to what works , build only when you're sure that it does not fit your use case.

[u/lolimouto_enjoyer → u/george-silva]

The one with the lead that can negotiate their deadlines better.

[u/lelanthran → u/m_adduci]

If you don't reinvent wheels occasionally, you will never have better wheels.

you don't want to implement Database ORM yourself.

I did that to great effect. Out of all the languages and ORMs I've used over my ~28 year development career, the one I have is by far better than any of them. Used it in production now for over 4 years, and no regrets.

Closed source, meant to open it, but now in the age of LLMs, I probably won't.

[u/mpyne → u/m_adduci]

Interesting, but isn't the usage of already made bricks one of the rules of Engineering?

The iron rule is “It Depends” and that’s the case here.

Each new dependency carries costs (in terms of integration maintenance and later upkeep and version tracking) and, hopefully, reduces other costs (development burden, testing, bug fixing) to make it worth it.

It is by no means a rule that anything that can be replaced by a dependency should be.

In an era where business is what at the end of the day matters, sinking too much time in getting things correctly and safely done might be counterproductive, especially in a time where AI-fans and enthusiasts (mostly from management levels) bring the argument that "AI is faster", because AI uses exactly those already made blocks.

The rise of AI also changes the game the other way. It’s quite literally never been easier, if you have a very simple need that is 5% of some popular dependency, to just have the AI correctly implement that 5% directly in your repo, without pulling in the extra dependency.

AI can also make it easier to bring in new dependencies, my point is that the costs on both sides of the equation have moved, not just on the side of ease of integration.

[u/Full-Spectral → u/m_adduci]

There are other variations as well. For instance, in my Rust project, it's just for brand new, 64 bit Linux, and for a particular variant of it even. I could have used Rust crates for a lot of stuff, but instead I just provided safe wrappers around the equivalent OS calls.

So I avoided probably 50 transitive dependencies, and I don't think that qualifies as NIH syndrome (though I'm a proud member of the NIH committee either way.) And my project which would probably be taking 2 minutes to rebuild from scratch (even at this early stage) rebuilds in 2 seconds.

Yeh, it's possible one of these OS calls has a memory issue. But the odds are so low that that's very way down my list of concerns, particularly given that much of the Rust standard library is also just wrappers around the OS.

In other cases, I write my own because I've been doing it for decades and am very comfy doing so. So, so far, I have a completely SOUP-free project that covers a lot of functionality. And my implementations only have to meet my needs. No portability concerns, no huge performance requirements, no accessibility concerns, etc... And it all is completely integrated, consistent, throughout, with no duct taping together bits and pieces of 3rd party code.

Obviously everyone can't go that route, but there are a lot of benefits if you can.

[u/TOGoS → u/m_adduci]

You don't want to do your own cryptography, you don't want to create a new XML or Json Parser, you don't want to implement Database ORM yourself.

Speak for yourself. I've avoided cryptography for the most part, but otherwise there have been times when I absolutely did want to each of those myself (as a new library) rather than use some third-party library.

JSON parsers are trivial enough that rolling your own is often cheaper than pulling in a dependency.

ORMs in Java tend to be a bit of a wrongly-engineered nightmare full of mutable state and spooky action at a distance. So rolling your own of those that fits the needs of your application and not what somebody else thought they wanted is also understandable.

[u/CharmingComputer3844 → u/TOGoS]

It's a fine line between wanting control and ending up reinventing the wheel. Sometimes, the simplicity of a well-established library can be a lifesaver.

[u/m_adduci → u/CharmingComputer3844]

Absolutely. There are so.many edge cases lurking out there and many libraries are meanwhile so hard battle tested that inventing something by yourself is much more painful.

Examples: curl, openssl, Jackson, opencv

[u/-Y0- → u/m_adduci]

Forget those, having a decent XML library that handles ALL the edge cases is a godsend.

[u/b0w3n → u/TOGoS]

ORMs in particular can be kludgy to interface with and bloated out the ass in features most people are never going to use.

Also serialization is another one, "just use xml or json" okay that's cool and all but it's slow. Binary can be better! At least I don't have to worry if some russian spybot has snuck some horseshit into this library I'm using for some of those simpler things that might not have a lot of eyes on it.

[u/-Y0- → u/b0w3n]

Binary can be better!

Good luck debugging Binary. And ensuring it works correctly if source and target systems are different Endians.

[u/-Y0- → u/TOGoS]

JSON parsers are trivial enough that rolling your own is often cheaper than pulling in a dependency.

HAHAHAHAHAHAHAHHAHAHAHAHAAHA.

No.

Now realize that JSON is the simplest possible of these challenges. And it's a fucking nightmare.

---

Reminds me how one particularly smort individual managed to implement a "rough" XML parser as output verifier. It even supported namespaces! ... Except it didn't check what XMLNS expanded names, but checked only namespace by prefix.

Took me 12hrs to debug that exceptionally smart piece of NIH-engineering. I didn't have the code, and was convinced for the first 8hrs that there was an error in my XML somehow. Next 2hrs was spent decompiling and cursing the idiot in question.

[u/hippydipster → u/m_adduci]

Taste, discernment, risk understanding, and experience are used to know when its best to use whats out there vs when its best to write your own.

[u/CoVegGirl]

I think it’s difficult to write even a moderately complex front-end project without taking on a copious amount of dependencies, or at least paying a lot of engineers a lot of money to reinvent the wheel.

[u/Skellicious]

I've noticed LLM vibe coding agents love to poop out their own implementation of stuff that would be very valid to use a library for. Admittedly I've only seen it for small file parsers and similar, but I've seen others notice this tendency too.

[u/max123246 → u/Skellicious]

There's a clear reason for this. LLMs are trained to implement it themselves because it decreases the risk of hallucinating. Can't hallucinate in a call to JQuery if you've been trained to only use the standard library of the language

[u/ouvreboite → u/max123246]

I don’t think that’s the reason.

I think that because "I built my own custom implementation" is by essence more verbose than "I added ta lib and solved it in 2 lines".

Even if the examples in the dataset were 50/50 for "custom implementation" vs "using a lib", the custom implementation examples would represent more lines in the total, so more emphasis during training.

[u/Flimsy_Complaint490 → u/ouvreboite]

Agree. And they dont seem to be trained to do any trade-off reasoning during the implementation either. There is never "is this or this better", but "do that" and if that doesnt work, immediatly move to plan b and stay within original constraints. Like, if you are doing a CLI app, it will never do research, pick a flag parsing library and such, it will poop out 1000 LoC file from its training data.

But ask it to do exactly that and it will do a very admirable job picking a flag parsing library, hook up the machinery and remove those 1000 LoC.

[u/gordonmessmer]

A shared library is a part of your application that's maintained by someone else.

Pro: Someone else maintains part of your application so you don't have to!

Con: You've given someone else commit access to your application, but in a way that gives you little to no insight or oversight of their changes.

Remember: Once your application becomes a process, there are no security boundaries. Every part of your application has full control of the application and can change the application any way it wants to. (Someone will surely point out that code pages are read-only, but remember that references to code pages usually aren't. It is usually possible to hijack/monkey patch functions.)

[u/ExtremePermit3242]

I don’t know who the target of this article is.

All the points read like “this is obvious and software development 101”.

They grow project size, repo size, build time, you don’t use 100% of the library, a non-multiplatform library makes your project non-multiplatform?

In other news: water is wet

[u/winian → u/ExtremePermit3242]

I once had to fix an incompatibility issue between two versions of a library after updating an in-house module, because some other guy had decided that he needed a third-party library for throwing an IllegalArgumentException in a slightly more convenient way. Some people need the obvious spelled out for them.

[u/sacheie → u/ExtremePermit3242]

"I don’t know who the target of this article is"

Javascript/Node developers

[u/turkoid → u/sacheie]

Someone who got burned by the left-pad incident.

[u/AryanPandey]

Ok

[u/brunogadaleta]

Do we need function level libraries? Like in Unison lang.

[u/rechaptca]

is not about librariea that helps you print coloured text output, but please tell your manager you prefer to biild from scratch scheduling system like celery or cron rather to use solid maintained library. good luck. also 10years time frame is quite some time in software development that mayne wven your programming language or interpeter will be outdated (vide python 2.7 vs 3.6 vs current line). so yeah - if you have time and team to write everything from scratch and ok to tell your client you deliver in 6months instead of 1 then go for it.

[u/Yay295 → u/rechaptca]

The current project I'm working on is more than 20 years old, and the main issue I'm working on is that the libraries haven't been updated in 10 years and they have security vulnerabilities. So I need to figure out how to either remove the dependencies, or integrate 10 years worth of updates to the dependencies.

[u/AndyKJMehta]

Jokes on him because every piece of code you write today is effectively a shared library of assembly calls