I mostly work as a consultant, and in a recent engagement with a software development team I encountered – in addition to the usual "disorganization" and "lone wolfs" – a phenomenon I have since dubbed, DCS – Developer Computer Syndrome.

Invariably, software developers write and test (sic ...) their brainchildren on computers that are equivalent to "a Lamborghini on a deserted open road." An abundance of CPU horsepower, RAM, and empty disk space. But production environments are never like that, and this is one of the things that was tripping this group up.

Categorically, the code that this group was writing, or in many cases "inheriting," and then [failing to ...] use in the wee hours of the morning on the production machines, was very sloppy. Particularly with regard to both "error conditions" (as in: correctly handling the possibility that a file doesn't exist), and resource consumption. "Testing" was done with unrealistically-small amounts of input data, on unrealistically-capacious machines. Sometimes, extreme "memory leak" problems simply fell through the cracks, undetected.

I helped this group to set up a (containerized) testing environment which realistically represented the production situation with its physical constraints. A clever developer wrote a script which would "rsync" realistically-sized (slices of ...) input files from the production queues and then sanitize the "PII = Personally-Identifiable Information" fields out of them for legal compliance. Together, we reviewed production scripts (many of them left over from a company that had been acquired and written by people who had left with that company) to identify and to categorize the types of errors – such as "failing to test for file-open failed" – which characterized these scripts. A pro-active project was then started to clean up these problems before they had a chance to actually occur in the wee hours.

In my experience, many companies have the problem of having to rely on "utility script code" that they did not write and therefore do not really know. The "front line" software that physically operates daily operation is of course lavished with proper attention, but these supporting scripts – although equally or perhaps more essential – are not. And, "one o'clock in the morning" is not the time that you want to see an error message that you've never seen before ... nor to realize that the software is malfunctioning and you didn't see an error message.

Interesting times. I love my job. Because I get to leave things better off when I leave.

I like "Developer Computer Syndrome". Web designers in particular all have fantastic monitors and graphics cards. And they run a web server on their own system to test on. Then they wonder why people have trouble in the real world deployment.

"Testing on your own system" is not "real testing." I teach people about "mock deployments." Set up a parallel environment which matches as closely as possible the production environment along with its constraints. Go through the exact same steps at 11:00 AM that you intend to go through at 1:30 AM, and do this in a shared environment.

"Virtual machines" were an old standby for doing this, but these days "containers" make it even easier. (Developers can easily set up "constrained environments" on their Lamborghinis.) Nonetheless, "pre-flight testing does not just happen 'on your box.'"

There is also a bit of an "I'm From Missouri" thing: "The mere fact that the program 'appeared to run without error' does not mean that it actually 'ran without error!'"

A few years ago, after a major deployment which moved a company's entire web presence from a dying blade-server (and ancient-version PHP code) to "the web," and just in time, I happened to meet the owner of the company(!) the next morning at a local coffee shop. We were both "all smiles," and I wasn't nervous at all. Because nothing had gone wrong, and thereafter nothing ever did.

The company's customers never had the slightest idea that everything had changed overnight ... except possibly to notice that it was a whole lot faster!

"Priceless ..."

I don't see how this is developers fault. Blame your product manager or w/e, they are responsible for gathering requirements and if they missed "must run on 10 year old laptop with 3G internet speed" - it's their fault.

If we support older devices, like if we support iPhone 6S - then most tests are done on this lowest denominator, and iPhone 13 or w/e will get only smoke tests.
If we support crap internet connections, like software for truck drivers, shipping, etc, where internet connection can be choppy - we will test with agreed slow internet speed, agreed percentage of package drops, etc, and once again - perfect conditions are only with smoke tests.

Sounds like you went and "write me X as fast as possible" - yea no shit developers will write X as fast as possible ignoring limiting factors.