Rendered at 14:51:27 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
__MatrixMan__ 1 days ago [-]
I'm all for pushing back against "let's not make perfect the enemy of good." I hear that all the time in reference to software that is usually quite bad, and often a little evil to boot--nothing good anywhere in sight except some poor schmuck trying to find a way to make their job something worth taking pride in.
I'm not sure I agree that systems are products though. The product mindset is toxic. It means that you've got goals which are independent of the user's goals (typically to make money, which sometimes means doing something dastardly to the users on behalf of the shareholders).
All the best software is more in the "tool" category and less in the "product" category. Usually it's made by the users, only bothers with solving problems they have, and has no ulterior motives.
KaiserPro 6 hours ago [-]
Its about pragmatism. The problem is that a lot of software engineering tends to be not about building a product to service the buisness need.
Lets be honest, a lot of the time engineering decisions are made because "it would be cool to use x" rather than "this is the fastest path to achieving business goal z"
"over engineering" in a lot of cases is just either engineering gate keeping "oh you don't know how to do x properly, its soooooo much more 'performant' than y" or design by CV "lets use kafka, k8s, mobile app, AI, RAG" etc etc etc.
in 99% of cases, the architecture really doesn't matter that much, you product is never going to need what ever fancy system you put in. Mainly because people over estimate scale.
The thing that matters is that the customer is able to use it, and know how to use it. Everything else is almost irrelevant.
elmer2 3 hours ago [-]
Most engimeering decisioms aren't made my engineers. Many times, the goalpost keeps moving while the deadline stays the same.
The end result is cutting corners and perfection is never even a thought.
bch 17 hours ago [-]
> “let's not make perfect the enemy of good."
Nuanced, but I’ve heard “don’t let perfection be the enemy of completion”, which is real in the cases of not shipping because one is constantly polishing/perfecting, or, as I heard sitting beside phk[0] at BSDCan commenting on rejecting some environmental sensors into FreeBSD, something to the effect of “We need ‘good’, not ‘good enough’.”, which draws a line at accepting janky “solutions” that are architecturally weak, or otherwise look obviously problematic.
I always thought of that as “yak-shaving,” or “bikeshedding.”
Quality takes longer. Sometimes, significantly longer, for many reasons. That’s why something top-shelf, costs so much more than commodity, when it doesn’t seem to have much more, in the way of features.
Many orgs aren’t interested in spending that much extra time on it (which is their prerogative). The customer base tends to be smaller, and a lot more fickle. It’s really difficult to make a living, doing top-Quality work.
Source: I worked for many years, for a company known for top-shelf stuff. I’m quite familiar with Quality as an everyday feature.
bch 10 hours ago [-]
I love the social/cultural aspects of our engineering discipline.
> I always thought of that as “yak-shaving,” or “bikeshedding.”
I think of yak-shaving as ending up (by lack of discipline, or perhaps out of necessity) working on some supporting aspect of a supporting aspect of a supporting aspect... in service of the original task.
So, a potential time suck, for sure, but "perfection is the enemy of completion" doesn't have to be that deep - if we're throwing out more clever phrases and war stories - another phrase that comes to mind about just "polishing" and insisting it's not good enough to ship yet is it's (slightly NSFW)
"just like masturbation - it feels good, but you're only f*cking yourself". You need to ship.
Bike Shedding I think of as social - nearly Peter Principle in practice - everybody thinks they've got something to contribute to the construction of the bike shed because it's a thing that's easy to manage, and people want to be helpful and/or make a mark.
ChrisMarshallNY 1 hours ago [-]
Sorry if your reply got dinged (looks all right to me, now). It was spot-on.
I think of "Yak-Shaving" as working on peripheral aspects of the project, to the detriment of the main path. Easy to justify, as they are "working on the project," but maybe not helping to significantly advance it.
I think of "Bikeshedding" as working on another aspect of the project (or a completely different project), because we want to (often, because it is more approachable or pleasant). This often becomes a problem, when the project is getting near shipping, as there's a ton of really boring stuff that needs to be done.
"Perfecting," on the other hand, can look like either of the above, but is, indeed, working on the main path, and advancing the project; just maybe not quickly.
fingerlocks 2 hours ago [-]
Not sure why you were flagged to death, I had all the same thoughts. Vouched.
These phrases have a different meanings and it’s important to highlight them
axus 22 hours ago [-]
Let's not make fixing bugs and backwards compatibility the enemy of shoveling out more features
close04 20 hours ago [-]
The expression is “the enemy of good” not “the enemy if something” so in the end it’s about defining “good”. It varies with context. What’s good, move fast and break things or go slow and considerate? Is risking bugs worth it to launch a feature? How important is that feature? How important is it to have it now vs. later? And how do you reconcile it when what’s good for you is bad for me?
moritzwarhier 20 hours ago [-]
It's just the same about the definition of "perfect".
Fixing critical bugs is not perfectionism. Fixing low-impact bugs by making high-risk, well-intentioned changes might be.
stouset 19 hours ago [-]
The number of times I've heard that phrase used to justify shipping absolute crap utterly dwarfs the times it's been used to prevent over-engineering something that's already in a good state.
In a thirty year career of software engineering, I haven't once come across one of these mythical perfectionists that everyone is constantly retelling cautionary tales of, who endlessly rewrites perfectly serviceable software and never releases it. I have, however, worked at multiple companies that have collapsed under the weight of their unmaintainable spaghetti tech stacks.
patrakov 16 hours ago [-]
I was one of those initiators of a rewrite. And I was facing a real problem: the codebase was in C, all C developers except me left, I was going to leave too, and both our HR department and an external consulting firm failed to find a competent C developer to replace me.
So I said: screw performance, we are rewriting this thing into Go, as that's what our existing developers were willing to work with - with the intention of leaving when this would be done.
And the day I finally submitted my resignation letter, a competent C developer was found, and later I heard that the Go rewrite was scrapped.
malux85 19 hours ago [-]
> I haven't once come across one of these mythical perfectionists that everyone is constantly retelling cautionary tales of, who endlessly rewrites perfectly serviceable software and never releases
Oh, I have. A LOT! In my friend group theres at least 5 of these right now. They work day jobs (that they dislike), they try to build a side hustle but never get it off the ground because they endlessly rewrite perfectly serviceable software and never release, the reason is:
- they believe that running a startup only requires writing a good product
- they avoid releasing because thats "judgement day" and might flop
- they think if they code it "just right" a money Waterfall will open up magically like a lottery ticket win and be instant success
- they believe that because theres the occasional exception to these rules above, it will happen to them too (everyone thinks they are the special exception,) because thats easier than accepting building a successful business takes discipline, hard work, and doing the tasks you dont want to do.
You havent come across them because by definition they never release and they dont share (they are embarrassed it might fail) so theres thousands and thousands of them, grinding away under false assumptions. They enjoy the act of building (which is totally cool!) but think thats all there is to it. Convinced that just one more rewrite is the thing holding them back from success - because they'd rather rewrite than talk to users. Millions of people doing this, millions of dreams that will die, because /just one more rewrite/
Just because you havent seen them, doesnt mean they aren't there
stouset 11 hours ago [-]
I couldn’t care less about solo developers who never release their hobby projects. The cautionary tales are always about potential coworkers, and as far as I can tell that stereotype simple doesn’t exist in the business world.
lazyasciiart 9 hours ago [-]
I have not just worked with but had code rejected by these people. “Don’t fix that bug, we are going to write an entire new structure that will make it irrelevant”, “don’t upgrade that library, we are going to rewrite our code to not use it”, “don’t improve exception handling, fix all the bugs that throw bad exceptions”. Not at all surprisingly, all of those planned upgrades took at least two years after my attempted improvement was rejected (the library I didn’t upgrade stuck around for six years).
ryandrake 17 hours ago [-]
I'm just like that, but minus the expectation of release/money/success. To me, the building and programming itself is the reward, and I have no motivation or desire to release anything I work on--or any belief that it would/should make money for me. To me, the hours spent fine sanding down and polishing my code, always approaching but never achieving perfection, is its own reward.
malux85 8 hours ago [-]
I would say thats a different thing entirely, an amateur in the original sense of the word
Latin amator, French amateur, means lover.
In this sense, it means somebody who is doing something because he/she loves it, not as his/her profession. It is not a large semantic shift to infer that he/she is not as skilled.
imtringued 8 hours ago [-]
I'm that perfectionist and it is constantly preventing me from shipping. Basically everything I'm working on requires me to cut losses at some point because it would take twice or three times as long if done perfectly and there is no guarantee that it is actually perfect in the end.
I don't really feel it collapsing under the weight of unmaintainable spaghetti tech stacks, I'm mostly thinking that writing the missing features of the tech stack would require months of work and hard thinking that in the end might simply get rejected and now I would be stuck maintaining it myself so nothing has changed other than that there are even more things to maintain.
All of this for the sake of clean architecture that seemingly has no impact on users in the end... Like the users complain about completely different things.
socketcluster 15 hours ago [-]
Yes, agreed, that quote is not as relevant to software development. Some would make the case that software isn't good unless it's perfect. Engineers will use the word 'correct' (e.g. correctness); and by that, they mean; is it perfect... And it's not wrong to demand it because the smallest gap can be a critical security vulnerability.
mtweak 24 hours ago [-]
I'm also wondering if "perfect" and "good enough" are not really as important now vs. when a team of software engineers had to spend sprints implementing features. The rate of iteration is faster now, the rate of regenerating entire code bases is days. We can perhaps over engineer /more/ today than before.
jaggederest 23 hours ago [-]
The real challenge is reigning in the size and complexity of the codebase if you're using AI to generate it. I have a bunch of skills (YAGNI / KISS inspired) but it still requires a significant amount of human effort. Knocked down about 9000 lines of cruft this month and I know there's another 10k in there sitting around.
gjvc 10 hours ago [-]
"reining"
BloodyIron 13 hours ago [-]
So is a screwdriver a tool or a product?
I think you're getting caught up on nomenclature and limited sample sizes.
__MatrixMan__ 12 hours ago [-]
What it is to you depends on if you want to sell a screwdriver or if you want to drive a screw.
If you're looking for somebody to design a screwdriver for you, you're better off with the guy who wants to drive a screw.
lazyasciiart 9 hours ago [-]
That’s a nice thought, but who are the users who plan to write their own air traffic control tool?
dasil003 24 hours ago [-]
I get what you mean about the conflict of interest for consumer software, but I don’t think a product mindset is inherently toxic. It’s more a function of where you are in the enshittification cycle.
smaudet 20 hours ago [-]
Products tend to be someone else's tool.
Like you hint at, though, a tool is useful immediately and without end (at a particular task - windows operating systems remain useful despite whatever current version is being sold), the product is often an attempt to continuously derive value from the tool (attach a time limited license, offer "upgrades", new features, etc.).
A service is more like a product than a tool - selling data is not something that a tool can (by itself) provide (unless you accept virtual or test data).
Attaching a product mindset to a tool then becomes a matter of complication/value reduction, vs a product mindset to a consumable, like a game, or dataset, where the value is at least constant, if not growing over time.
I.e., I'm not sure I agree that the "enshittification cycle" is the variable here - even in the age of AI the value e.g. of a stock image service is constant or growing, even if the LLM generation process dilutes the market.
__MatrixMan__ 20 hours ago [-]
> Products tend to be someone else's tool
I like this. The question is really whether you're facing the "business end of the stick" or not (although stick isn't quite right here, because not all software has both ends).
sublinear 24 hours ago [-]
You've definitely used non-toxic and thoughtful products or you wouldn't be making such a distinction.
You are throwing the baby out with the bathwater by retreating to "tools". If you want to stop the enshittification, this is not the way. You're just opening yourself up to new scams. i.e. AI tools, political agendas, etc.
Better products are just better. Demand them.
__MatrixMan__ 23 hours ago [-]
When I think of software that is minimally tainted by productism, these come to mind:
- linux
- nix
- nushell
- helix
Are these products... at all? Do the open me up to new scams? I don't think so.
I'm not in a position to demand anything from the people who make these things. If they were subject to demands, their craft would be tainted by compromises made in acquiescence to those demands, and I'd probably be less enthusiastic about their software (because presumably, my tastes don't align with whichever others are also in a position to be making demands).
Supply and demand are well and good if what you're after is barley. But when you compare what there's demand for with what's being supplied re: software, there appears to be no correlation.
We gotta stop selling picks and shovels and start learning to be miners who have good taste in picks and shovels and the ability to make and remake our tools as needed. The disconnect is creating a hell for our users.
qsort 1 days ago [-]
I wouldn't say that "over-engineering means solving the wrong problem". It's possible that the idea is basically correct, but people are directing effort towards optimizing for constraints that don't really exist or can be dealt with once a better picture is in place, whether it's the mythical PMF or just "we now understand what the users want, let's build that".
The worst clusterfuck I've ever worked on was a web application that was actually solving a real problem fairly well, but the team was spending time building an absurd Rube-Goldberg contraption of microservices when the entire platform had less MAU than my hobby website. It wasn't the wrong problem, but it certainly was over-engineered!
var0xyz 1 days ago [-]
> but the team was spending time building an absurd Rube-Goldberg contraption of microservices
This is literally the example that I use, the most common case of over-engineering, having more microservices than team members. Microservices are the right solution for certain problems, but those were not problems they had.
Some anecdotal evidence. I worked in many of these places, and the most common tell of over-engineering is that when you ask "what problem were we solving when we decided to have all these many microservices?" the answers you will get is problems they either didn't have (for example, high availability) or they state a problem they actually had but could have been solved in the monolith.
In other words, they "overshoot" and - as I write in the post - end up with "a system that solves multiple problems partially, none of them completely, while introducing a bunch of problems you wouldn't have had otherwise."
xnx 24 hours ago [-]
> more microservices than team members
Or more microservices than customers
zephen 1 hours ago [-]
You need to be careful with this one, or reductio ad absurdum might lead you to not build microservices, or even a monolithic app, before you have acquired your first customer.
xnx 1 minutes ago [-]
True, but also "Do Things that Don't Scale" (https://www.paulgraham.com/ds.html). There are many examples of successful companies that had customers before they had products.
win311fwg 21 hours ago [-]
> Microservices are the right solution for certain problems
Just one problem, actually: Conway's law.
When your organization needs to operate like the macro economy, where independent groups provide services for each other, then you are going to see the same structure found in the macro economy mirrored in the microcosm of that single organization. Hence the name microservices. Same as services, except not across business lines.
At Google scale you have no choice but to have an internal economy, but trying to build an entire economy inside a small business with less activity than a personal blog is crazy.
estetlinus 24 hours ago [-]
> absurd Rube-Goldberg contraption of microservices
Such a lovely analogy. FYI it’s mine now.
godelski 23 hours ago [-]
While I agree with you, a large number of people use the term to mean that something was engineered to a complexity they have difficulty understanding. Abstraction is also treated as a dirty word, with people focusing on only one interpretation of the word.
While I don't actually agree with OP, I do agree with their sentiment. I've seen people say something is "over engineered" when there's an elegant design. Elegance isn't over engineering, it is solving problems effectively. It's something we should chase! Elegance is solving the right problem, which usually people are having a hard time seeing. (It's not always easy)
If we constantly let people drag quality down then we get into this frustrating world where everything is constantly half broken.
kccqzy 18 hours ago [-]
And sometimes optimizing for constraints happen because someone from above (perhaps someone with a title of Director who doesn’t actually write code) imposed their constraints on the team without understanding the ramification of these constraints. It takes a manager with very good people skills and/or very good writing skills to persuade that someone to drop their constraints.
bluefirebrand 1 days ago [-]
> optimizing for constraints that don't really exist or can be dealt with once a better picture is in place
I think these both fall comfortable under the umbrella of "solving the wrong problem"
nickelpro 1 days ago [-]
"We're not trying to build a perfect solution here" is not something said to assuage over-engineering or encourage sloppy work.
It's said to head off a specific complaint from a specific kind engineer who will object that the proposed solution won't work because it doesn't cover some obscure edge case which rarely comes up in production.
"We're not trying to build a perfect solution here" is saying "We acknowledge not everything will be covered, we're setting the requirements at the 90th percentile use case".
calebkaiser 22 hours ago [-]
This was my initial reaction to reading this post as well.
Additionally, as I get older, I find the sentiment of "we're not trying to build a perfect system here" is less about "let's just go fast vroooom" and more akin to saying "I've been humbled before by thinking I had the perfect mental model of the universe before a single user touched the product."
esikich 20 hours ago [-]
My grandpa used to say "we're not building a piano" when doing things like building a quick shelf or something in the basement and be done after a quick measure and a few cuts. My other grandpa would spend an entire day measuring, sanding, pulling out a router etc. Both valid, but if you just need something to put some paint cans on, it doesn't require a day of work.
1-more 17 hours ago [-]
hahah I picked that up from a youtube woodworker when I was a CALENDAR YEAR into building my son's toy chest. A lot of measurements I had sweated over didn't matter; they are inside of the channels on shaker panels. A bunch of things I didn't sweat are apparent; not ever pencil mark got sanded off before I oiled it and I didn't notice.
ozim 19 hours ago [-]
I find it that in wild it is quite often used by product people to push crap ASAP because they don’t want to spend time finding out even basic constraints in the system they supposed to know like the back of their hand.
Then engineers are on the hook because they run into those constraints while building and everyone always blames „those lazy software developers” ;)
hectdev 24 hours ago [-]
The bane of my engineering career is working under engineers like this. It's like we forget we are doing a very analog thing (collaboration and building) under the guise of something digital. We should accept that there will be edge cases, there will be crashes. And unless you're actually in a life-saving industry, that is ok. (I say this with the idea in mind of a 10+ year old code bases spanning many new coding patterns that achieves over 99.5% crash-free)
tetha 22 hours ago [-]
Agreed. The much better question is: Are we locking ourselves out of a feature, or do we just leave a gap for a rare case?
At work we agreed that some use cases are very niche. These have guards in place to log an alerted-upon marker + return HTTP/500. They have not tripped for years by customers. So, it's fine to not support some rare cases and to deliberately leave known gaps in some contexts. As long as you don't close these paths forward if you need them.
On the other hand, we have contexts like our PostgreSQL instances. Those have a very well defined scope and rooting out all known problems has been the right choice. Most issues we have ignored in that scope have bitten us in the butt sooner rather than later. Very hard in some cases, I may add.
Realizing this about a domain is very important.
okwhateverdude 22 hours ago [-]
I've been that engineer and with good reason. That obscure edge case which rarely comes up in production is very disruptive when it does come up. The product person waving it off is also not the one that will get paged at 2am to address the issue when it occurs. Accepting that 90th percentile use case is infuriating because it is tacit consent for an unfinished solution with the rest being made up later with additional toil, now constrained by load bearing things you cannot change. Thanks for rushing the thinking and making my life harder later for no reason.
ozim 19 hours ago [-]
Don’t forget months or years later when shit hits the fan, developers will be blamed for being lazy and not the product person. No one will remember Joe said something on one or two meetings we build for 90th percentile - code in git repository will point to a developer who wrote and GitHub will point to the one that approved merge request.
In online discussions people always blame „lazy developers „ like there would be no product owners, testers, business analysts, scrum masters etc.
skydhash 15 hours ago [-]
This is one of the reason I like a ticket tracker. I make my commit as descriptive as possible and I tag them with the ticket number. Also I add comments when Product says to cut corners. It's nice to spread responsibility around when things take a bad turn.
nickelpro 8 hours ago [-]
> it is tacit consent for an unfinished solution
No. It is cutting unused features to make a manageable product.
* "That won't work on GCC 5.5." We don't support GCC 5.5, or any compiler which wasn't shipped this decade.
* "What about FreeBSD?" What about it? We only have Linux servers
* "This only works on systemd." Good, we're a systemd shop.
jurgenburgen 4 hours ago [-]
In my experience when a manager says that it means “let’s just ignore the architecture and use a short cut if we can get the security team to rubber stamp it”.
Ekaros 20 hours ago [-]
What is the qualifier of "rarely". One in ten, one in hundred, one in thousand? What is the failure in these cases? I could accept rarely if that is one in trillion to one in quadrillion range say. Truly rare cases. But not if it might be for seen reasonably.
cj 19 hours ago [-]
Very few businesses care about accommodating "one in a trillion" edge cases. (Most would call that over-engineering IMO)
nickelpro 19 hours ago [-]
"None of the deployments under consideration for this new initiative use that mechanism, that other deployments might is irrelevant to the internal product we are building"
In my space this is usually something like, "X won't work on POSIX make", being a reason not to add X to the build system. Well, it works on Ninja, and on GNU make, so it just won't work on POSIX make and the handful of deployments still using POSIX make just won't use X.
sesm 17 hours ago [-]
> We acknowledge not everything will be covered, we're setting the requirements at the 90th percentile use case
I'm ok with this as long as dropped requirements are documented and nobody wakes you up at night when those edge cases show up in production. Also, when the next feature needs to build on top of those dropped edge cases, you are given enough resources to redo the previous solution.
datakan 24 hours ago [-]
Correct. That is one of the worst types of engineers too. Constant paralysis by analysis. It's why Agile became a thing because those people just refuse to ship anything until it's utterly perfect and spotless. Fortunately they are a minority though.
laszlojamf 7 hours ago [-]
The caveat here is, as TFA states, is that you need to have "every constraint on the table". And that is almost never the case. Unless you create something in a mature domain, where everything has already been explored, you _won't know_ what the constraints are.
You can spend three months in whiteboard sessions planning out the perfect solution, and then your plan falls apart the second you start implementing, because the real world imposes constraints on you that were practically impossible to know a priori. Not theoretically impossible, mind you. If you'd been smart enough and spent enough time thinking about it, you would have thought of them. But you aren't smart enough, and you don't have the time. Those are some other constraints that the universe puts on you.
BiteCode_dev 6 hours ago [-]
We spent 20 years talking about agile precisely because of this, and people took it, turned it into a checklist based processed, forgot about the short feedback loop it's precisely supposed to bring to the table to figure our your unknown unknown quickly, and declared it useless.
And now we are back to square one, where we argue again "but just make good specs".
The world is a silly place.
s08148692 4 hours ago [-]
"you need a very clear set of requirements. Every constraint on the table"
How nice it would be to live in this fantasy world
MantisShrimp90 22 hours ago [-]
I believe perfection is a dirty word because it has a pernicious effect on the human mind.
Striving towards perfection too often leads to yes over-engineering but I think more importantly it also: leads to way too much bike shedding due to the need to be perfect and allot of even emotional baggage when we don't make that perfect things.
Even the author admits that their definition of perfect can only arise with stringent requirements and I'll take it a step further, maybe it was perfect for that problem at that time but guess what change happens all the time, and as architects we have to think about future problems as much as current ones. With that in mind I think that having something that is good in many scenarios is better than the thing that is perfect in the scenario you start in.
mlinsey 24 hours ago [-]
"With one big caveat: you need a very clear set of requirements"
Most new product launches are an exercise in figuring out what the product requirements should be through trial and error (really: through ongoing dialogue with your users). Even mature products can have requirements change over time as the market changes.
I think internalizing this reality is why most senior engineers who work in domains that touch the messy real world will reflexively push back against perfectionism.
bad_username 18 hours ago [-]
> Most new product launches are an exercise in figuring out what the product requirements should be through trial and error
Sure, but these have to be well-defined trials. In other words, yes, you will test several hypotheses, but your hypotheses have to be hypotheses, not hand waving.
smallnix 24 hours ago [-]
But this article allows framing this decision in a nice way. It's not a push back against over engineering, it's deliberate trade-off for speed by lowering quality, because quality is relative to the targeted outcome, which is yet fuzzy.
titzer 23 hours ago [-]
I think over-complicated and over-engineered are not the same thing.
Over-complicated is adding too many features, too much mechanism, too many moving parts. Over-engineering is far exceeding the requirements in an unhelpful way. For example: build me a treehouse. Said treehouse could probably be made of wood. If you made it out of concrete and steel, it might be a heck of a lot stronger and last a heck of a lot longer, without being more complicated (just more expensive). Too strong is over-engineered. If you made a treehouse with 13 bedrooms and an elevator, glass windows, solar power, and running water, that's overcomplicated.
edg5000 22 hours ago [-]
I generally use:
- Overbuilt: Designed to handle more of the same. Too strong.
- Overengineered: Complexity not justified by the requirements.
Sometimes overbuilding (e.g. using really thick wood on a treehouse) means you can lower complexity (less bracing needed) at the cost of being more wasteful with material. That is often a great tradeoff.
nulltrace 15 hours ago [-]
I think your concrete treehouse counts too though. Add a shower and suddenly you need plumbing, drainage, the structure has to hold more weight. The overcomplicated part drags the overengineered part along with it. They pretty much collapse into the same thing.
marknutter 19 hours ago [-]
Reminds me of the Seinfeld episode where he hires a contractor to work on his kitchen and ends up with twice as many cabinets as he had before.
Talpur1 5 hours ago [-]
Agree with over all post but one thing that always haunts me while engineering is perfection means what? a perfection that i want to engineer or a perfection that is based on response from users/ market, would love to have your thoughts on this angle
khurs 3 hours ago [-]
It depends on the stage of the company and resources available.
For example if a startup running out of money, don't waste time on perfection, hack it out. Future revenue will lead to larger teams and re-write
grogers 3 hours ago [-]
The budget is just another constraint that helps define the perfect solution. Sometimes quick and dirty is the perfect solution.
nhggb 3 hours ago [-]
Great article. So many engineers forget that requirements exist even when you don't write or don't search for them. It's kinda like mixing the map with the reality. And worse, engineers will push all the understanding of requirements to PM's that have zero technical judgement.
I really think this problem is likely within the top 3 causes that slows down our industry.
Wish organizations had training programs to make people learn this.
W-Stool 1 days ago [-]
"Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away
- Antoine de Saint-Exupéry
GodelNumbering 24 hours ago [-]
This was my first thought too. I actually use this in my Agent prompts
Naulian 3 hours ago [-]
The problem is that sometimes while trying to make it perfect, you end up over-engineering it.
pilgrim0 13 hours ago [-]
> The shape of the solution only becomes obvious once you treat the system as a product and define the requirements honestly.
The “honest” part is the important one. It’s also the hardest. It’s so tempting to project preferences onto a new development, which tends to confuse the distinction between desire and necessity. There’s so many things we want to try. Maybe having lots of personal projects to vent the need for novelty helps one to make good choices when it’s critical that they must be right.
lincoln20xx 24 hours ago [-]
I concur. Most of the most painfully over-engineered swamp projects I've dealt with have been a result of not understanding the requirements.
Of course, this is not to imply that it's necessarily possible to know every requirement perfectly before you begin - I find that I frequently don't really understand the requirements until I've built the damn thing and it's made first contact with the real world.
So I usually toss that first version out, and build the second one. And that one is usually good enough that I don't immediately want to throw it out the window.
And by the time I get through the third version, usually a fair bit after the second is done, it's pretty damn good.
Because now I understand the problem and the requirements so much better than I did at the start, and I better understand what the system needs to be to solve that problem.
Will it be perfect? Never, except in a static and unchanging universe. But we can approach optimality for a given moment in time, which is functionally the same thing.
bluefirebrand 24 hours ago [-]
> Most of the most painfully over-engineered swamp projects I've dealt with have been a result of not understanding the requirements.
For me it's been a complete lack of requirements. Absolutely no one in the company seeming to want to make a firm decision on what we're building right at the moment. Every decision coming with a hedge "but maybe we might want to do this sometime later so we should keep our options open"
So any time there was a tradeoff we would take the option that leaves our options open, which takes way more time and effort to get right
teodorlu 22 hours ago [-]
Beautifully written! I want to live in a world where we strive towards perfection, rather than strive to destroy it.
IshKebab 4 hours ago [-]
> Beautifully written!
This is written by AI. Maybe whoever prompted it has a point but I wouldn't say AI can reach the heights of beautiful writing yet, and this definitely doesn't.
oooyay 18 hours ago [-]
> Set clearer requirements, set stricter constraints, and the solution follows. That solution is the perfect one for you, for that case.
I think this is a bit of a generous reframing of what seeking perfection is. Generally I think perfection seeking is best described as over-focusing on the details and pre-planning instead of laying a general blueprint that leaves room for pivots, future decisions, and iteration along the way. There's a whole breed of engineers that are really great thinkers but get stuck in the mud trying to pre-think the best way to do something instead of being adaptable.
charles_f 24 hours ago [-]
> Over-engineering is solving the wrong problem
Is it though? Engineering is providing the cheapest solution that corresponds to what your customer asked. As it goes: anyone can build a bridge with infinite budget, only an engineer can do it for cheap.
A system that's un-necessarily complex for what it does is over built, and under engineered.
I think it's important beyond semantics, because sometimes you have to propose designs that wouldn't be in books, yet they're the cheap and fit with all requirements, and are thus well engineered.
Now the difficulty is to figure the requirements that are not being expressed. "Allowing an occasional resurfacing" and "don't explode when there's icing" are obvious ones; the hard one for our job are "allow for adding another lane" or "allow support for trains as well"... because you know these might come.
RandomLensman 24 hours ago [-]
> Every constraint on the table. Tighten those enough and something interesting happens, you end up with only one possible solution.
Is there always a unique solution? If constraints A is tightened and then determines the outcome, could that not also happen with a tightened constraint B that determines a different outcome?
leugim 1 days ago [-]
I disagree, requirements are only a part of the engineering process. Perfection will be redefined after the product, tech, team knowledge changes, which will constantly.
Spend too much time thinking into perfection, trying to achieve it and you will get a convoluted system with lava-layer patterns full of complexity astronauts.
lalitmaganti 1 days ago [-]
I agree with the actual principle I think this article is trying to put across (don't give up on technical quality too early by dismissing it as "perfection") but I strongly disagree with the framing.
My biggest problem comes from:
> Every constraint on the table. Tighten those enough and something interesting happens, you end up with only one possible solution. And that solution is, somewhat ironically, the perfect one.
Constraints are not rigid and unchangable [1] but flexible things which can bend to solve the problem at hand. Constraints can be: deadlines you impose, frameworks you are working within, the shape of the output schema, performance characteristics etc. And because they trade off against each other, tightening them leaves you with several solutions, each fitting a different trade-off.
Constraints also change over time as you and your team's collective understanding of the problem, the environment or priority of the problem change. This needs to be considered when building the initial system too: baking in some flexibility means giving up some of today's "perfection" to deal with future variance.
Understanding what constraints matter, which can be shaped and trading them off against each other is, to a very large degree, the job of software engineering. So no, I don't think you enumerate the constraints and find the only solution which fits; this is a very simplistic take on the complexities of building software.
[1] unless you work in a dictatorial company in which case my condolences!
ablob 24 hours ago [-]
I'd like to add an anecdote about machining.
Some designers impose incredibly tight tolerances on part shapes that hardly change the overall behavior of the final machine, but are so difficult to produce that sometimes special tools have to be bought and instead of being able to produce 4 parts an hour it might just be one instead.
Maybe this sentiment stems from being software engineers, but at least for hardware it becomes abundantly clear that "good enough" is a desirable state unless you want to spend years in development. The only direct comparison I can find in software is computer graphics, where the endeavor often halts as soon as the average person can't perceive a difference anymore.
qiqitori 14 hours ago [-]
If you want to make all developers working on a project happy about the way the project is being designed/implemented, it's best to reduce the number of developers to 1 or less.
HPsquared 24 hours ago [-]
If something can be defined in rigorous mathematical terms and proven correct, that can save a lot of trouble downstream dealing with exceptions. Not all problems can be solved so neatly, but it's often worth putting in the effort to get from 99% to 100% correct if it means you never have to read or touch it again.
Mikhail_Edoshin 8 hours ago [-]
Perfection is a search for truth, the true nature of things.
epolanski 24 hours ago [-]
I don't share the definition of over engineering.
As engineering is the act of solving technical problems, over engineering is about putting too much engineering effort on aspects, features or products that don't have a linear payoff to the budget spent.
E.g. I worked in a company that was obsessed with unit test coverage metrics and the effort of maintaining the test suite was considerably biting in the ability to move the product. The ratio was 25% of product and 75% of unit tests.
And the payoff was small if not even negative, impacted morale, productivity and actively pushed back against refactors, because any large refactor was met with a disproportionate effort in unit test rewriting. Let alone the fact that as you were mocking external dependencies, and the mocks costed engineering effort and internals digging, it also suppressed work to keep it up to date.
I never got to convince the org to shift focus on E2E testing, which answered the real questions: does the product make money and works as expected. Uni test had to be used when writing a parser, not to validate some react scroll component as browser apis and the triggers mimicking costed days of work.
In fact we often had gigantic all green test suites for broken products.
That's to me over engineering an aspect of the engineering cycle.
starky 23 hours ago [-]
This is a good way of putting it. People who advocate for over-engineering don't consider cost-benefit vs not doing it. This is where FMEA analysis comes in when developing products, you assign values for severity, occurrence, and detection to determine which potential issues need extra effort implemented, but just as important is identifying the potential issues that aren't worth putting effort into to prevent over-engineering.
I've recently had to deal with a problem where the parent company has dictated 100% production testing for a spec that we meet for the spec sheet but customers practically don't ever need. But since the parent company test their products this way they say we have to. I've been very vocally against this because it has delayed us shipping products that are otherwise ready, prevented starting new products, and added a lot of production cost when our RMA rate without the test is something on the order of 0.01% and most of those are due to installation errors, not true failures.
Night_Thastus 14 hours ago [-]
Knowing your constraints perfectly is impossible, and takes more and more effort the closer you get.
And frankly, they change on a regular basis. By the time you figure them out that perfectly, they're invalidated. A new feature is suddenly needed by the customer, a bottleneck that wasn't there before appears now, scale changes, consumer preferences change, new hardware and software come out, everything changes.
IMO, the best engineering is 80/20. Get 80% of the result you want for relatively little effort and complexity, and don't sweat the remaining 20%. Embrace the imperfection. Get it done in the simplest, most obvious way - even if you know it has limitations, edge cases, and some things missing.
Knowing how to get that 80% is extremely hard (and how to do it without causing lots of problems down the road), but I think senior engineers get pretty good at it. And if you absolutely need to, get another 80% of that remaining 20%.
solatic 22 hours ago [-]
> For someone else it's the wrong choice, they don't know Python
The author ruins their entire argument with this one claim.
If "we picked this tool because we know it best" is a legitimate requirement, then every tool choice is justified as a "perfect" choice because it's what the architect was most comfortable with. If your emotions and current knowledge levels are considered reasonable justifications for a "perfect" solution, then all solutions are perfect solutions; they simply haven't had enough emotional justification yet. If all solutions are perfect solutions, then none are.
There are, ultimately, two kinds of software - those that need to ship by a deadline, and those that don't. A deadline forces you to eject dead weight that you don't need - requirements have a habit of getting clarified real fast when you need to build to a deadline. If you had time to over-engineer despite a deadline, you should consider working for a more productive organization. Meanwhile, the concept of "over-engineering" is a little vague for software that doesn't have a deadline. If you don't have a deadline, you don't have to compromise on quality. "Over-engineering" is then just a value judgement that you made poor use of your infinite timescale and built the wrong things with it. But who is making that judgement? Not the person who built it, not the person who funded it (i.e. usually self-funded as a hobby project), and not the person who uses it (since over-engineering is an implementation detail, rather than a product choice), so who cares?
edit: to clarify: "pick a stack you know already since we don't have time to learn a new one" is a totally valid requirement. But I disagree that it means that you built a "perfect" solution with it. I also disagree that you usually need to build perfect software - getting comfortable with adequate is how most people ship most software.
alasano 18 hours ago [-]
"Start a new project. Every language, every tool, every hosting model available. You pick serverless."
Picking serverless as the leading example to show off perfection.
dondraper36 23 hours ago [-]
The idea sounds related to the distinction between essential and accidental complexity.
I really like the article above in that complexity does have to live somewhere. If you have a hard problem to solve, there is a minimum level of complexity (essential complexity) that can't be further reduced, only moved from one place to another.
And yet we often add accidental complexity by making bad choices (thinking of you, microservices). If your choice is bad enough, you get all the disadvantages (challenges with distributed systems, complicated debugging), but without any of its advantages (a distributed monolith doesn't get you any deployment or team independence).
Telling one from the other, well, is something I'd like to learn and that seems to require both experience and some sort of intuition.
4lx87 20 hours ago [-]
I agree. Too many use "perfectionism" as an excuse to ship unremarkable crap that doesn't meet spec.
yuliyp 1 days ago [-]
There is not a perfect solution. There are many bad solutions. There are a few solutions that are OK but with different tradeoffs.
datakan 24 hours ago [-]
Fast. Cheap. Good.
You can pick two of those, can never have all three.
0x20cowboy 21 hours ago [-]
> I believe a perfect solution exists.
That is incorrect when dealing in the real world with many different humans using a complex system. I’ve heard other people say this, and it always confuses me how it is not self evident. Human interaction is not a maths equation. There are too many random variables.
esher 24 hours ago [-]
Hang on. Let me overthink this.
OutOfHere 23 hours ago [-]
Let me do that for you. I think perfection is subjectively met at the end of one's consciousness about a problem. A higher being will see the problem more deeply, and its perfect solution could therefore be more perfect than that of a lesser being.
Imagine that one has to implement a function to multiply two large integers. A junior programmer will likely use the default multiplication operator. A mid-level programmer will add a couple of strategies with a weak heuristic for routing, and perhaps with basic caching. A senior will sample the incoming requests to identify which strategy might empirically be good, and ensure that it is being hit, with optimal fallbacks. A superintelligent AI might design custom kernels, and design a custom model for routing. A god will... we don't know what a god will do. Perfection, like beauty, is in the eyes of the beholder.
tombert 18 hours ago [-]
It's rare that I so thoroughly disagree with the thesis of a post here.
> Over-engineering is solving the wrong problem. That's the whole definition.
Wholeheartedly disagree. Over-engineering already seems pretty well defined just based on the words alone without tacking on a strange arbitrary definition to suit your blog post, but I think most people most people would define it more as "spending too much time and/or resources to solve a problem more-thoroughly than necessary".
Suppose I was given the task to convert all the JPEG images in a folder to PNG.
Solution A) I could write a highly optimized program writing custom CUDA decoders and encoders and using Open MPI for CPU-based optimization, and create an elaborate plugin framework on top of this to make this modular and cool.
Solution B) I could just run `find -name "*.jpg" | parallel magick {} {.}.png`.
Both solve the "right" problem; the ask was to convert these images, and both approaches can absolutely do that, so they didn't solve the "wrong problem", but I would argue that for most people Solution A would be "over engineered". You could argue that by building custom encoders and decoders and the like we're solving the "wrong problem" by going beyond what the problem asked for, but I feel like if you're criticizing the engineering methodology then that kind of proves my point.
We generally think someone is "over engineering" primarily when we are happy with the low-effort result.
There are times when Solution A is the right way to go about it. Maybe you want something that takes advantage of custom hardware, or maybe you can make an assumption about these images that Imagemagick can't, or maybe you really just understand image encoding significantly better than the Imagemagick people. in those cases, we generally don't consider it "over-engineering".
prmph 22 hours ago [-]
It's not over-engineering if you are committed to what you are building for the long term, and know you'd very likely need stronger architecture at a certain time.
No one starts a project for perfectly objective reasons. Your choice of careers was not perfectly objective, you decided you liked a field, and it seemed you had enough aptitude for it, and then you did the work to get into it, learning as you went along and adjusting.
So it is with all kinds of decisions that go along with starting or changing a software project. Of course are your architectural and engineering acumen increases, you get a goo feel for how to choose good constraints and requirements, and then within the those constraints it is now up to you to find optimal solution for them, not the most over-engineered one.
Like I said before, sometimes going from A-B-C-D is much less efficient (and can actually be counter-productive) in the aggregate than going form A-D directly, even though YAGNI would dictate A-B-C-D. So long as you generally know that D would be needed sooner or later, A-B-C-D can be counter-productive because it causes to you to miss the forest for the trees. Even if you are not doing D immediately, some thought given to D can improve greatly your understanding of the context of A and the roadmap, leading to better design.
Note that string design instincts honed over time make this process work, not building out stuff just because of fads (e.g., microservices) or for some functionality you do not actually need. So here is my point of agreement with the article: the reason teams might waste time over-engineering is not because they are exploring a design space and objectives that is larger than what is strictly required for now; it is because they lack good architectural skill and experience and instincts. Again, considering D upfront can actually simplify the A you build now.
sandeepkd 19 hours ago [-]
In the absence of concrete examples its hard to say what is OP trying to optimize for. Optimism is good, however at the end of the day everything is a trade off, the cost has to be paid for all decisions. A good design is all about optimizing when, where and how you push those costs.
hakunin 24 hours ago [-]
Another point that goes along with this: good engineering is not premature optimization[1].
Perfection is always relative to a purpose or telos. So in the case of artifacts - which are the products of engineering, arts, and techne in general - the purpose isn't inherent to the object in the manner it is in oak trees and human beings, but projected onto the thing. It lives in the mind of the designer and the user. That's why we can say that technology is an "accidental" extension of human beings by which human purposes are realized and mediated in ways that inherent human powers cannot directly accomplish.
So, yes, if you have a purpose in mind, you can build something that can be used for that purpose, more or less effectively. The more general the purpose, the less specific the implementation; the more specific the purpose, the less general the implementation. Over-engineering here is something that indeed goes beyond the purpose of something, with feature creep as an example of moving goalposts and a slippery grasp of the limit.
However, production is only one layer of concern in engineering. You can talk about ideal artifacts whose forms perfectly enable the realization of some purpose, but there is the practicality of constructing one that must also be considered. Think of diminishing returns. Does it makes sense to invest a certain amount of time and resources into producing a more perfect artifact? If not, then pursing such perfection is itself an imperfection on the part of human judgement. That is, a more perfect engineer might know the limit of perfection for an idealized artifact, but he also knows what the perfect practical limit of justifiable effort is. Perfectionism is a sign of an imperfect engineer.
Returning to feature creep, the author might argue that the moving goalposts are a sign that the purpose is being refined, but feature creep can also mean that the shifting purpose is detached from real needs. So, in that case, the purpose served is no longer the purpose that ought to be served, and in that sense, while one may realize an end perfectly in a relative sense, the end pursued is itself the result of imperfect understanding, which is effectively what the author is gesturing at when discussing requirements.
godelski 23 hours ago [-]
There is no such thing as perfect.
Anyone saying "don't make perfection the enemy of good" is using a thought terminating cliché, avoiding the conversation of what is good enough. Worse, it is often used by people to drag quality down. If you're creating the "minimum viable product" you usually create a product that isn't working.
But, perfection doesn't exist. Most solutions in the world have no global optima. There are always tradeoffs. You must choose. You must argue with your peers to figure out that tradeoff. Perfection has infinite depth in detail. You must optimize. You just learn the unknown unknowns in an every moving landscape.
But you should still chase perfection. Like you chase a utopia. Like you chase your dreams. There's always something to improve on. Chasing perfection while knowing it can't exist means you will continue to search for the flaws. It means you will continue to improve. "Over engineer", because that's just engineering. Make things actually work, while recognizing they're always broken somewhere. Don't get offended when someone points out a flaw, you already know it's not perfect, so figure out if it is a tradeoff or can be fixed. Just keep improving things, because otherwise they keep getting worse
croisillon 18 hours ago [-]
next time just post the prompt
groby_b 17 hours ago [-]
Sure, but did it need to be written by AI? Does that solve for the right constraint?
shevy-java 18 hours ago [-]
> I believe a perfect solution exists.
The article seems too focused on over-engineering.
Achieving 100% is often much much harder than, say,
95% or so. Just good enough. This is the old "Worse
is better" problem. Why did Linux outcompete, say,
Minix or something else? Worse is better is a more
practical focus. You don't need to achieve 100%
perfect per se. Momentum is often more important.
You can still have a lot of efficiency if the momentum
is the main driver.
Perfection is nice in theory; in practice it will be
whacked into submission by the worse-is-better speedy
wizards. The hobo programmers built for speed. You make
fun of them. Then the top 500 supercomputers all use what
they wrote ... like Gandhi they were winning.
zzzeek 20 hours ago [-]
great post but also the real thing they are talking about here is YAGNI
helge9210 23 hours ago [-]
Perfection is not "perfectionism". Former is provable given set of constraint. Later is anxiety due to fear of criticism.
"We don't want to build the perfect solution." Can both mean "no need to cover use cases we don't care about" and "no need to deal with divizion by zero in a product, where a number is divided by the user entered number".
0xbadcafebee 15 hours ago [-]
Perfection is the enemy because the very notion of it assumes that something cannot be any better. Everything can always be better. That's the problem. Whatever solution you make, will eventually be "not good enough". Follow that path long enough and you will never be done with anything. So not only do you need to realize perfection is impossible, but you also need to accept "the bad" and live with it, or become Sisyphus.
mateuszingano 2 hours ago [-]
[flagged]
JesseHowell 21 hours ago [-]
[flagged]
shoggi 24 hours ago [-]
[flagged]
m463 22 hours ago [-]
[dead]
szundi 19 hours ago [-]
[dead]
IanCal 1 days ago [-]
I disagree, you’re optimising many different things and there is not a simple way to compare them. The very concept of Pareto optimal is about this! There is not one uniquely perfect solution to a problem most of the time.
Spending too much time gathering requirements can be a bad business decision too. Those requirements are often not even accurate.
Building a perfect system may not be over engineering to solve the problem perfectly… but what if you don’t need to solve it perfectly? What if it’s better to fail on some requirements to deliver sooner or cheaper or deliver this and some other project?
I'm not sure I agree that systems are products though. The product mindset is toxic. It means that you've got goals which are independent of the user's goals (typically to make money, which sometimes means doing something dastardly to the users on behalf of the shareholders).
All the best software is more in the "tool" category and less in the "product" category. Usually it's made by the users, only bothers with solving problems they have, and has no ulterior motives.
Lets be honest, a lot of the time engineering decisions are made because "it would be cool to use x" rather than "this is the fastest path to achieving business goal z"
"over engineering" in a lot of cases is just either engineering gate keeping "oh you don't know how to do x properly, its soooooo much more 'performant' than y" or design by CV "lets use kafka, k8s, mobile app, AI, RAG" etc etc etc.
in 99% of cases, the architecture really doesn't matter that much, you product is never going to need what ever fancy system you put in. Mainly because people over estimate scale.
The thing that matters is that the customer is able to use it, and know how to use it. Everything else is almost irrelevant.
The end result is cutting corners and perfection is never even a thought.
Nuanced, but I’ve heard “don’t let perfection be the enemy of completion”, which is real in the cases of not shipping because one is constantly polishing/perfecting, or, as I heard sitting beside phk[0] at BSDCan commenting on rejecting some environmental sensors into FreeBSD, something to the effect of “We need ‘good’, not ‘good enough’.”, which draws a line at accepting janky “solutions” that are architecturally weak, or otherwise look obviously problematic.
[0] https://people.freebsd.org/~phk/
Quality takes longer. Sometimes, significantly longer, for many reasons. That’s why something top-shelf, costs so much more than commodity, when it doesn’t seem to have much more, in the way of features.
Many orgs aren’t interested in spending that much extra time on it (which is their prerogative). The customer base tends to be smaller, and a lot more fickle. It’s really difficult to make a living, doing top-Quality work.
Source: I worked for many years, for a company known for top-shelf stuff. I’m quite familiar with Quality as an everyday feature.
> I always thought of that as “yak-shaving,” or “bikeshedding.”
I think of yak-shaving as ending up (by lack of discipline, or perhaps out of necessity) working on some supporting aspect of a supporting aspect of a supporting aspect... in service of the original task.
So, a potential time suck, for sure, but "perfection is the enemy of completion" doesn't have to be that deep - if we're throwing out more clever phrases and war stories - another phrase that comes to mind about just "polishing" and insisting it's not good enough to ship yet is it's (slightly NSFW)
"just like masturbation - it feels good, but you're only f*cking yourself". You need to ship.
Bike Shedding I think of as social - nearly Peter Principle in practice - everybody thinks they've got something to contribute to the construction of the bike shed because it's a thing that's easy to manage, and people want to be helpful and/or make a mark.
I think of "Yak-Shaving" as working on peripheral aspects of the project, to the detriment of the main path. Easy to justify, as they are "working on the project," but maybe not helping to significantly advance it.
I think of "Bikeshedding" as working on another aspect of the project (or a completely different project), because we want to (often, because it is more approachable or pleasant). This often becomes a problem, when the project is getting near shipping, as there's a ton of really boring stuff that needs to be done.
"Perfecting," on the other hand, can look like either of the above, but is, indeed, working on the main path, and advancing the project; just maybe not quickly.
These phrases have a different meanings and it’s important to highlight them
Fixing critical bugs is not perfectionism. Fixing low-impact bugs by making high-risk, well-intentioned changes might be.
In a thirty year career of software engineering, I haven't once come across one of these mythical perfectionists that everyone is constantly retelling cautionary tales of, who endlessly rewrites perfectly serviceable software and never releases it. I have, however, worked at multiple companies that have collapsed under the weight of their unmaintainable spaghetti tech stacks.
So I said: screw performance, we are rewriting this thing into Go, as that's what our existing developers were willing to work with - with the intention of leaving when this would be done.
And the day I finally submitted my resignation letter, a competent C developer was found, and later I heard that the Go rewrite was scrapped.
Oh, I have. A LOT! In my friend group theres at least 5 of these right now. They work day jobs (that they dislike), they try to build a side hustle but never get it off the ground because they endlessly rewrite perfectly serviceable software and never release, the reason is:
- they believe that running a startup only requires writing a good product
- they avoid releasing because thats "judgement day" and might flop
- they think if they code it "just right" a money Waterfall will open up magically like a lottery ticket win and be instant success
- they believe that because theres the occasional exception to these rules above, it will happen to them too (everyone thinks they are the special exception,) because thats easier than accepting building a successful business takes discipline, hard work, and doing the tasks you dont want to do.
You havent come across them because by definition they never release and they dont share (they are embarrassed it might fail) so theres thousands and thousands of them, grinding away under false assumptions. They enjoy the act of building (which is totally cool!) but think thats all there is to it. Convinced that just one more rewrite is the thing holding them back from success - because they'd rather rewrite than talk to users. Millions of people doing this, millions of dreams that will die, because /just one more rewrite/
Just because you havent seen them, doesnt mean they aren't there
Latin amator, French amateur, means lover.
In this sense, it means somebody who is doing something because he/she loves it, not as his/her profession. It is not a large semantic shift to infer that he/she is not as skilled.
I don't really feel it collapsing under the weight of unmaintainable spaghetti tech stacks, I'm mostly thinking that writing the missing features of the tech stack would require months of work and hard thinking that in the end might simply get rejected and now I would be stuck maintaining it myself so nothing has changed other than that there are even more things to maintain.
All of this for the sake of clean architecture that seemingly has no impact on users in the end... Like the users complain about completely different things.
I think you're getting caught up on nomenclature and limited sample sizes.
If you're looking for somebody to design a screwdriver for you, you're better off with the guy who wants to drive a screw.
Like you hint at, though, a tool is useful immediately and without end (at a particular task - windows operating systems remain useful despite whatever current version is being sold), the product is often an attempt to continuously derive value from the tool (attach a time limited license, offer "upgrades", new features, etc.).
A service is more like a product than a tool - selling data is not something that a tool can (by itself) provide (unless you accept virtual or test data).
Attaching a product mindset to a tool then becomes a matter of complication/value reduction, vs a product mindset to a consumable, like a game, or dataset, where the value is at least constant, if not growing over time.
I.e., I'm not sure I agree that the "enshittification cycle" is the variable here - even in the age of AI the value e.g. of a stock image service is constant or growing, even if the LLM generation process dilutes the market.
I like this. The question is really whether you're facing the "business end of the stick" or not (although stick isn't quite right here, because not all software has both ends).
You are throwing the baby out with the bathwater by retreating to "tools". If you want to stop the enshittification, this is not the way. You're just opening yourself up to new scams. i.e. AI tools, political agendas, etc.
Better products are just better. Demand them.
- linux
- nix
- nushell
- helix
Are these products... at all? Do the open me up to new scams? I don't think so.
I'm not in a position to demand anything from the people who make these things. If they were subject to demands, their craft would be tainted by compromises made in acquiescence to those demands, and I'd probably be less enthusiastic about their software (because presumably, my tastes don't align with whichever others are also in a position to be making demands).
Supply and demand are well and good if what you're after is barley. But when you compare what there's demand for with what's being supplied re: software, there appears to be no correlation.
We gotta stop selling picks and shovels and start learning to be miners who have good taste in picks and shovels and the ability to make and remake our tools as needed. The disconnect is creating a hell for our users.
The worst clusterfuck I've ever worked on was a web application that was actually solving a real problem fairly well, but the team was spending time building an absurd Rube-Goldberg contraption of microservices when the entire platform had less MAU than my hobby website. It wasn't the wrong problem, but it certainly was over-engineered!
This is literally the example that I use, the most common case of over-engineering, having more microservices than team members. Microservices are the right solution for certain problems, but those were not problems they had.
Some anecdotal evidence. I worked in many of these places, and the most common tell of over-engineering is that when you ask "what problem were we solving when we decided to have all these many microservices?" the answers you will get is problems they either didn't have (for example, high availability) or they state a problem they actually had but could have been solved in the monolith.
In other words, they "overshoot" and - as I write in the post - end up with "a system that solves multiple problems partially, none of them completely, while introducing a bunch of problems you wouldn't have had otherwise."
Or more microservices than customers
Just one problem, actually: Conway's law.
When your organization needs to operate like the macro economy, where independent groups provide services for each other, then you are going to see the same structure found in the macro economy mirrored in the microcosm of that single organization. Hence the name microservices. Same as services, except not across business lines.
At Google scale you have no choice but to have an internal economy, but trying to build an entire economy inside a small business with less activity than a personal blog is crazy.
Such a lovely analogy. FYI it’s mine now.
While I don't actually agree with OP, I do agree with their sentiment. I've seen people say something is "over engineered" when there's an elegant design. Elegance isn't over engineering, it is solving problems effectively. It's something we should chase! Elegance is solving the right problem, which usually people are having a hard time seeing. (It's not always easy)
If we constantly let people drag quality down then we get into this frustrating world where everything is constantly half broken.
I think these both fall comfortable under the umbrella of "solving the wrong problem"
It's said to head off a specific complaint from a specific kind engineer who will object that the proposed solution won't work because it doesn't cover some obscure edge case which rarely comes up in production.
"We're not trying to build a perfect solution here" is saying "We acknowledge not everything will be covered, we're setting the requirements at the 90th percentile use case".
Additionally, as I get older, I find the sentiment of "we're not trying to build a perfect system here" is less about "let's just go fast vroooom" and more akin to saying "I've been humbled before by thinking I had the perfect mental model of the universe before a single user touched the product."
Then engineers are on the hook because they run into those constraints while building and everyone always blames „those lazy software developers” ;)
At work we agreed that some use cases are very niche. These have guards in place to log an alerted-upon marker + return HTTP/500. They have not tripped for years by customers. So, it's fine to not support some rare cases and to deliberately leave known gaps in some contexts. As long as you don't close these paths forward if you need them.
On the other hand, we have contexts like our PostgreSQL instances. Those have a very well defined scope and rooting out all known problems has been the right choice. Most issues we have ignored in that scope have bitten us in the butt sooner rather than later. Very hard in some cases, I may add.
Realizing this about a domain is very important.
In online discussions people always blame „lazy developers „ like there would be no product owners, testers, business analysts, scrum masters etc.
No. It is cutting unused features to make a manageable product.
* "That won't work on GCC 5.5." We don't support GCC 5.5, or any compiler which wasn't shipped this decade.
* "What about FreeBSD?" What about it? We only have Linux servers
* "This only works on systemd." Good, we're a systemd shop.
In my space this is usually something like, "X won't work on POSIX make", being a reason not to add X to the build system. Well, it works on Ninja, and on GNU make, so it just won't work on POSIX make and the handful of deployments still using POSIX make just won't use X.
I'm ok with this as long as dropped requirements are documented and nobody wakes you up at night when those edge cases show up in production. Also, when the next feature needs to build on top of those dropped edge cases, you are given enough resources to redo the previous solution.
You can spend three months in whiteboard sessions planning out the perfect solution, and then your plan falls apart the second you start implementing, because the real world imposes constraints on you that were practically impossible to know a priori. Not theoretically impossible, mind you. If you'd been smart enough and spent enough time thinking about it, you would have thought of them. But you aren't smart enough, and you don't have the time. Those are some other constraints that the universe puts on you.
And now we are back to square one, where we argue again "but just make good specs".
The world is a silly place.
How nice it would be to live in this fantasy world
Striving towards perfection too often leads to yes over-engineering but I think more importantly it also: leads to way too much bike shedding due to the need to be perfect and allot of even emotional baggage when we don't make that perfect things.
Even the author admits that their definition of perfect can only arise with stringent requirements and I'll take it a step further, maybe it was perfect for that problem at that time but guess what change happens all the time, and as architects we have to think about future problems as much as current ones. With that in mind I think that having something that is good in many scenarios is better than the thing that is perfect in the scenario you start in.
Most new product launches are an exercise in figuring out what the product requirements should be through trial and error (really: through ongoing dialogue with your users). Even mature products can have requirements change over time as the market changes.
I think internalizing this reality is why most senior engineers who work in domains that touch the messy real world will reflexively push back against perfectionism.
Sure, but these have to be well-defined trials. In other words, yes, you will test several hypotheses, but your hypotheses have to be hypotheses, not hand waving.
Over-complicated is adding too many features, too much mechanism, too many moving parts. Over-engineering is far exceeding the requirements in an unhelpful way. For example: build me a treehouse. Said treehouse could probably be made of wood. If you made it out of concrete and steel, it might be a heck of a lot stronger and last a heck of a lot longer, without being more complicated (just more expensive). Too strong is over-engineered. If you made a treehouse with 13 bedrooms and an elevator, glass windows, solar power, and running water, that's overcomplicated.
Sometimes overbuilding (e.g. using really thick wood on a treehouse) means you can lower complexity (less bracing needed) at the cost of being more wasteful with material. That is often a great tradeoff.
For example if a startup running out of money, don't waste time on perfection, hack it out. Future revenue will lead to larger teams and re-write
The “honest” part is the important one. It’s also the hardest. It’s so tempting to project preferences onto a new development, which tends to confuse the distinction between desire and necessity. There’s so many things we want to try. Maybe having lots of personal projects to vent the need for novelty helps one to make good choices when it’s critical that they must be right.
Of course, this is not to imply that it's necessarily possible to know every requirement perfectly before you begin - I find that I frequently don't really understand the requirements until I've built the damn thing and it's made first contact with the real world.
So I usually toss that first version out, and build the second one. And that one is usually good enough that I don't immediately want to throw it out the window.
And by the time I get through the third version, usually a fair bit after the second is done, it's pretty damn good.
Because now I understand the problem and the requirements so much better than I did at the start, and I better understand what the system needs to be to solve that problem.
Will it be perfect? Never, except in a static and unchanging universe. But we can approach optimality for a given moment in time, which is functionally the same thing.
For me it's been a complete lack of requirements. Absolutely no one in the company seeming to want to make a firm decision on what we're building right at the moment. Every decision coming with a hedge "but maybe we might want to do this sometime later so we should keep our options open"
So any time there was a tradeoff we would take the option that leaves our options open, which takes way more time and effort to get right
This is written by AI. Maybe whoever prompted it has a point but I wouldn't say AI can reach the heights of beautiful writing yet, and this definitely doesn't.
I think this is a bit of a generous reframing of what seeking perfection is. Generally I think perfection seeking is best described as over-focusing on the details and pre-planning instead of laying a general blueprint that leaves room for pivots, future decisions, and iteration along the way. There's a whole breed of engineers that are really great thinkers but get stuck in the mud trying to pre-think the best way to do something instead of being adaptable.
Is it though? Engineering is providing the cheapest solution that corresponds to what your customer asked. As it goes: anyone can build a bridge with infinite budget, only an engineer can do it for cheap.
A system that's un-necessarily complex for what it does is over built, and under engineered.
I think it's important beyond semantics, because sometimes you have to propose designs that wouldn't be in books, yet they're the cheap and fit with all requirements, and are thus well engineered.
Now the difficulty is to figure the requirements that are not being expressed. "Allowing an occasional resurfacing" and "don't explode when there's icing" are obvious ones; the hard one for our job are "allow for adding another lane" or "allow support for trains as well"... because you know these might come.
Is there always a unique solution? If constraints A is tightened and then determines the outcome, could that not also happen with a tightened constraint B that determines a different outcome?
Spend too much time thinking into perfection, trying to achieve it and you will get a convoluted system with lava-layer patterns full of complexity astronauts.
My biggest problem comes from:
> Every constraint on the table. Tighten those enough and something interesting happens, you end up with only one possible solution. And that solution is, somewhat ironically, the perfect one.
Constraints are not rigid and unchangable [1] but flexible things which can bend to solve the problem at hand. Constraints can be: deadlines you impose, frameworks you are working within, the shape of the output schema, performance characteristics etc. And because they trade off against each other, tightening them leaves you with several solutions, each fitting a different trade-off.
Constraints also change over time as you and your team's collective understanding of the problem, the environment or priority of the problem change. This needs to be considered when building the initial system too: baking in some flexibility means giving up some of today's "perfection" to deal with future variance.
Understanding what constraints matter, which can be shaped and trading them off against each other is, to a very large degree, the job of software engineering. So no, I don't think you enumerate the constraints and find the only solution which fits; this is a very simplistic take on the complexities of building software.
[1] unless you work in a dictatorial company in which case my condolences!
Maybe this sentiment stems from being software engineers, but at least for hardware it becomes abundantly clear that "good enough" is a desirable state unless you want to spend years in development. The only direct comparison I can find in software is computer graphics, where the endeavor often halts as soon as the average person can't perceive a difference anymore.
As engineering is the act of solving technical problems, over engineering is about putting too much engineering effort on aspects, features or products that don't have a linear payoff to the budget spent.
E.g. I worked in a company that was obsessed with unit test coverage metrics and the effort of maintaining the test suite was considerably biting in the ability to move the product. The ratio was 25% of product and 75% of unit tests.
And the payoff was small if not even negative, impacted morale, productivity and actively pushed back against refactors, because any large refactor was met with a disproportionate effort in unit test rewriting. Let alone the fact that as you were mocking external dependencies, and the mocks costed engineering effort and internals digging, it also suppressed work to keep it up to date.
I never got to convince the org to shift focus on E2E testing, which answered the real questions: does the product make money and works as expected. Uni test had to be used when writing a parser, not to validate some react scroll component as browser apis and the triggers mimicking costed days of work.
In fact we often had gigantic all green test suites for broken products.
That's to me over engineering an aspect of the engineering cycle.
I've recently had to deal with a problem where the parent company has dictated 100% production testing for a spec that we meet for the spec sheet but customers practically don't ever need. But since the parent company test their products this way they say we have to. I've been very vocally against this because it has delayed us shipping products that are otherwise ready, prevented starting new products, and added a lot of production cost when our RMA rate without the test is something on the order of 0.01% and most of those are due to installation errors, not true failures.
And frankly, they change on a regular basis. By the time you figure them out that perfectly, they're invalidated. A new feature is suddenly needed by the customer, a bottleneck that wasn't there before appears now, scale changes, consumer preferences change, new hardware and software come out, everything changes.
IMO, the best engineering is 80/20. Get 80% of the result you want for relatively little effort and complexity, and don't sweat the remaining 20%. Embrace the imperfection. Get it done in the simplest, most obvious way - even if you know it has limitations, edge cases, and some things missing.
Knowing how to get that 80% is extremely hard (and how to do it without causing lots of problems down the road), but I think senior engineers get pretty good at it. And if you absolutely need to, get another 80% of that remaining 20%.
The author ruins their entire argument with this one claim.
If "we picked this tool because we know it best" is a legitimate requirement, then every tool choice is justified as a "perfect" choice because it's what the architect was most comfortable with. If your emotions and current knowledge levels are considered reasonable justifications for a "perfect" solution, then all solutions are perfect solutions; they simply haven't had enough emotional justification yet. If all solutions are perfect solutions, then none are.
There are, ultimately, two kinds of software - those that need to ship by a deadline, and those that don't. A deadline forces you to eject dead weight that you don't need - requirements have a habit of getting clarified real fast when you need to build to a deadline. If you had time to over-engineer despite a deadline, you should consider working for a more productive organization. Meanwhile, the concept of "over-engineering" is a little vague for software that doesn't have a deadline. If you don't have a deadline, you don't have to compromise on quality. "Over-engineering" is then just a value judgement that you made poor use of your infinite timescale and built the wrong things with it. But who is making that judgement? Not the person who built it, not the person who funded it (i.e. usually self-funded as a hobby project), and not the person who uses it (since over-engineering is an implementation detail, rather than a product choice), so who cares?
edit: to clarify: "pick a stack you know already since we don't have time to learn a new one" is a totally valid requirement. But I disagree that it means that you built a "perfect" solution with it. I also disagree that you usually need to build perfect software - getting comfortable with adequate is how most people ship most software.
Picking serverless as the leading example to show off perfection.
https://ferd.ca/complexity-has-to-live-somewhere.html
I really like the article above in that complexity does have to live somewhere. If you have a hard problem to solve, there is a minimum level of complexity (essential complexity) that can't be further reduced, only moved from one place to another.
And yet we often add accidental complexity by making bad choices (thinking of you, microservices). If your choice is bad enough, you get all the disadvantages (challenges with distributed systems, complicated debugging), but without any of its advantages (a distributed monolith doesn't get you any deployment or team independence).
Telling one from the other, well, is something I'd like to learn and that seems to require both experience and some sort of intuition.
That is incorrect when dealing in the real world with many different humans using a complex system. I’ve heard other people say this, and it always confuses me how it is not self evident. Human interaction is not a maths equation. There are too many random variables.
Imagine that one has to implement a function to multiply two large integers. A junior programmer will likely use the default multiplication operator. A mid-level programmer will add a couple of strategies with a weak heuristic for routing, and perhaps with basic caching. A senior will sample the incoming requests to identify which strategy might empirically be good, and ensure that it is being hit, with optimal fallbacks. A superintelligent AI might design custom kernels, and design a custom model for routing. A god will... we don't know what a god will do. Perfection, like beauty, is in the eyes of the beholder.
> Over-engineering is solving the wrong problem. That's the whole definition.
Wholeheartedly disagree. Over-engineering already seems pretty well defined just based on the words alone without tacking on a strange arbitrary definition to suit your blog post, but I think most people most people would define it more as "spending too much time and/or resources to solve a problem more-thoroughly than necessary".
Suppose I was given the task to convert all the JPEG images in a folder to PNG.
Solution A) I could write a highly optimized program writing custom CUDA decoders and encoders and using Open MPI for CPU-based optimization, and create an elaborate plugin framework on top of this to make this modular and cool.
Solution B) I could just run `find -name "*.jpg" | parallel magick {} {.}.png`.
Both solve the "right" problem; the ask was to convert these images, and both approaches can absolutely do that, so they didn't solve the "wrong problem", but I would argue that for most people Solution A would be "over engineered". You could argue that by building custom encoders and decoders and the like we're solving the "wrong problem" by going beyond what the problem asked for, but I feel like if you're criticizing the engineering methodology then that kind of proves my point.
We generally think someone is "over engineering" primarily when we are happy with the low-effort result.
There are times when Solution A is the right way to go about it. Maybe you want something that takes advantage of custom hardware, or maybe you can make an assumption about these images that Imagemagick can't, or maybe you really just understand image encoding significantly better than the Imagemagick people. in those cases, we generally don't consider it "over-engineering".
No one starts a project for perfectly objective reasons. Your choice of careers was not perfectly objective, you decided you liked a field, and it seemed you had enough aptitude for it, and then you did the work to get into it, learning as you went along and adjusting.
So it is with all kinds of decisions that go along with starting or changing a software project. Of course are your architectural and engineering acumen increases, you get a goo feel for how to choose good constraints and requirements, and then within the those constraints it is now up to you to find optimal solution for them, not the most over-engineered one.
Like I said before, sometimes going from A-B-C-D is much less efficient (and can actually be counter-productive) in the aggregate than going form A-D directly, even though YAGNI would dictate A-B-C-D. So long as you generally know that D would be needed sooner or later, A-B-C-D can be counter-productive because it causes to you to miss the forest for the trees. Even if you are not doing D immediately, some thought given to D can improve greatly your understanding of the context of A and the roadmap, leading to better design.
Note that string design instincts honed over time make this process work, not building out stuff just because of fads (e.g., microservices) or for some functionality you do not actually need. So here is my point of agreement with the article: the reason teams might waste time over-engineering is not because they are exploring a design space and objectives that is larger than what is strictly required for now; it is because they lack good architectural skill and experience and instincts. Again, considering D upfront can actually simplify the A you build now.
[1]: https://max.engineer/premature-optimization
So, yes, if you have a purpose in mind, you can build something that can be used for that purpose, more or less effectively. The more general the purpose, the less specific the implementation; the more specific the purpose, the less general the implementation. Over-engineering here is something that indeed goes beyond the purpose of something, with feature creep as an example of moving goalposts and a slippery grasp of the limit.
However, production is only one layer of concern in engineering. You can talk about ideal artifacts whose forms perfectly enable the realization of some purpose, but there is the practicality of constructing one that must also be considered. Think of diminishing returns. Does it makes sense to invest a certain amount of time and resources into producing a more perfect artifact? If not, then pursing such perfection is itself an imperfection on the part of human judgement. That is, a more perfect engineer might know the limit of perfection for an idealized artifact, but he also knows what the perfect practical limit of justifiable effort is. Perfectionism is a sign of an imperfect engineer.
Returning to feature creep, the author might argue that the moving goalposts are a sign that the purpose is being refined, but feature creep can also mean that the shifting purpose is detached from real needs. So, in that case, the purpose served is no longer the purpose that ought to be served, and in that sense, while one may realize an end perfectly in a relative sense, the end pursued is itself the result of imperfect understanding, which is effectively what the author is gesturing at when discussing requirements.
Anyone saying "don't make perfection the enemy of good" is using a thought terminating cliché, avoiding the conversation of what is good enough. Worse, it is often used by people to drag quality down. If you're creating the "minimum viable product" you usually create a product that isn't working.
But, perfection doesn't exist. Most solutions in the world have no global optima. There are always tradeoffs. You must choose. You must argue with your peers to figure out that tradeoff. Perfection has infinite depth in detail. You must optimize. You just learn the unknown unknowns in an every moving landscape.
But you should still chase perfection. Like you chase a utopia. Like you chase your dreams. There's always something to improve on. Chasing perfection while knowing it can't exist means you will continue to search for the flaws. It means you will continue to improve. "Over engineer", because that's just engineering. Make things actually work, while recognizing they're always broken somewhere. Don't get offended when someone points out a flaw, you already know it's not perfect, so figure out if it is a tradeoff or can be fixed. Just keep improving things, because otherwise they keep getting worse
The article seems too focused on over-engineering.
Achieving 100% is often much much harder than, say, 95% or so. Just good enough. This is the old "Worse is better" problem. Why did Linux outcompete, say, Minix or something else? Worse is better is a more practical focus. You don't need to achieve 100% perfect per se. Momentum is often more important. You can still have a lot of efficiency if the momentum is the main driver.
https://en.wikipedia.org/wiki/Worse_is_better
Perfection is nice in theory; in practice it will be whacked into submission by the worse-is-better speedy wizards. The hobo programmers built for speed. You make fun of them. Then the top 500 supercomputers all use what they wrote ... like Gandhi they were winning.
"We don't want to build the perfect solution." Can both mean "no need to cover use cases we don't care about" and "no need to deal with divizion by zero in a product, where a number is divided by the user entered number".
Spending too much time gathering requirements can be a bad business decision too. Those requirements are often not even accurate.
Building a perfect system may not be over engineering to solve the problem perfectly… but what if you don’t need to solve it perfectly? What if it’s better to fail on some requirements to deliver sooner or cheaper or deliver this and some other project?