Maybe people succumb to the hype. Maybe they want the latest shiny running their own shiny new thing. I'm sure these things contribute greatly.
But I see another aspect to this whole "We Use Shiny Cogs" movement: high-level vs. low-level. As we make tools that abstract us away from the metal, we are able to spend less time thinking about the electrons flowing across silicon and more time thinking about building something John Q. Public will pay for.
We architect higher and higher abstractions for exactly this reason. And it comes with a price: at some point you stop running things as efficiently as possible and there's waste. If we were all studied CS students and could write kernels and compilers from scratch, we might spend five years building a very tight, efficient stack for Twitter that could run on a single box (maybe with a hot failover). But we're not. We're a collection of humans with differing levels of understanding and will power, many of whom just want to Get Shit Done and stop thinking about kernel task queues.
So lets turn his "rich man's problem" around a bit: you build your idea on top of a stack that you understand and keeps you happy, and when you bring in the capital (through revenue or investment - whatever) you put money into deeper, lower-level engineering. Until then, build your idea the way you know how.
"As we make tools that abstract us away from the metal, we are able to spend less time thinking about the electrons flowing across silicon and more time thinking about building something John Q. Public will pay for."
That makes it sound like there's a complete dichotomy here. It's not like the mailinator guys are doing assembly whereas the rest of the startup crowd is doing Prolog AI. Never mind that I doubt that most backend programmers employed are using the time saved on improving the sheer monetization output of the company (mileage obviously varies for single-person shops).
I do think you hit the nail on your head with your final paragraph. Quite often it's not that the people programming this choose to pick the cogs, quite often they don't know
any other way. In which case this isn't really a decision. But it's a sad trend in my opinion. Remember the old saying "Nobody ever got fired for choosing IBM"? I think the level where this kind of thinking is applied has dropped from non-technical upper management to some actual builders. Smart business decision - maybe. But I do think that a lot of us here aren't just in it for the money.
The new people are not in it to perfect their engineering craftsmanship. Overall product craftsmanship maybe, but then half their attention is taken up by visual/UX design, they have less time for beautiful plumbing.
But at the same time engineer craftsmen can specialize in making better cogs. That would be the ideal, better cogs like Stripe, Heroku, Redis, DynamoDB, whatever. Someday most will stop thinking about those problems just as we stopped thinking about design patterns of making procedure calls in assemblers.
In your story "better cogs" sound like "more cowbell". It's still a fundamental decision which will not go away: do you solve a problem in process or do you communicate with anything external: Redis vs. HasMap. It's not comparable to levels in programming languages.
What about a programming language that integrates fancy cogs so they're conceptually in process? Do I want to know the difference between sorting in the native data-structure and Redis? Hopefully not, the language/compiler could make that call.
Programmer could specify how much data he's sorting, how fast he needs it sorted, how much data he'll get after each sort and the language will decide which cog to use for best performance. You don't need to know what Redis, Riak, or native data-structures are.
But the question was: can I afford to go through a tcp socket (and that's a long way, possibly even to another machine) or not. A language cannot cover this up.
Yes they're probably libraries, but in the language of the future that covers everything up you wouldn't know such distinctions.
Why can't a lib/lang cover it up? You tell it how much data will be stored and transferred. It will benchmark various approaches, having separate DB nodes or not, determine how many users each setup can support. The developer would need to be aware that 5000 users can fit into a $10 per month box. When he wants more users, he gets more boxes. If he wants more users per box he must cut expensive features.
Amazon DynamoDB does something along those lines by asking the dev to specify how much data to store and how many transfers to expect. The details of setting up nodes, how many nodes, how they're sharded, hashed, are hidden.
It's not that smart for these cases. No AI required. Run the app with various cogs, benchmark each run, pick the cog than ran fastest. Compiling for cog picks will take more time. The programmer would be responsible for sending the right dummy data by linking to a data gen cog and writing tests.
Cogs come with lots of metadata for use cases, performance profiles, data sizes, etc. They don't need to be marketed, tried by hundreds of hopeful devs and become resume buzzwords. The programmer no longer needs to read HN and care about the shiniest new cogs. Mental overhead is greatly reduced.
There are many people, such as the OP probably, who take pride in the engineering aspect of what they do. It's my understanding that the term "hacker" as used on HN actually does not include this.
Also, not everybody is working on a startup. Most programmers actually work in the "put money into deeper, lower-level engineering" phase of software development, and it's important that companies have a stream of good engineers to hire.
But I see another aspect to this whole "We Use Shiny Cogs" movement: high-level vs. low-level. As we make tools that abstract us away from the metal, we are able to spend less time thinking about the electrons flowing across silicon and more time thinking about building something John Q. Public will pay for.
We architect higher and higher abstractions for exactly this reason. And it comes with a price: at some point you stop running things as efficiently as possible and there's waste. If we were all studied CS students and could write kernels and compilers from scratch, we might spend five years building a very tight, efficient stack for Twitter that could run on a single box (maybe with a hot failover). But we're not. We're a collection of humans with differing levels of understanding and will power, many of whom just want to Get Shit Done and stop thinking about kernel task queues.
So lets turn his "rich man's problem" around a bit: you build your idea on top of a stack that you understand and keeps you happy, and when you bring in the capital (through revenue or investment - whatever) you put money into deeper, lower-level engineering. Until then, build your idea the way you know how.