Working for a company in Germany which is planing production 3 months in advance using printed Excel sheets. The migration of ERP system gone wrong and nobody knows how to fix it. Production management tries to hide this fact and does not talk to the engineering department. This will go for years, consultants will gather their fees for non functional system. Obviously IT infrastructure is not needed for manufacturing. It is just nice to have.
In the late 90s, early 2000, the Danish department of defence decided that they needed a new procurement system, DeMars, built on SAP. I know a sergent that worked in procurement at the time, he made insanely large purchases of everything he was responsible for in the months leading up to the launch. It came to the point where he was pulled in for questioning, on the suspicion of fraud. He explained that he had no faith in the launch of DeMars and wanted to ensure that stock would not run out. Everything was accounted for, if anyone believe that he as committing fraud, they where welcome to do a complete inventory.
DeMars launched, and procurement basically stopped for a year. Only the items my friend was in charge of remained in stock, through out the launch/roll-out process.
There's lots of pushes to add software to more of the military, but I don't think these kinds of resilience questions are really taken seriously. A system intended for wartime use will be running in non-optimal conditions while under constant attack. But many of these "enterprise" systems barely work better than paper to start with.
I think that's changed a bit - witness how reluctant the US gov is to be locked into Anduril Lattice. I think we'll see this start to change but certainly a lot of bad past decisions to make up for.
Switching to SAP ERP was already an in-joke level of well-known catastrophe in IT consulting circles 20 years ago. I’m glad to see nothing has changed in that respect.
I remember reading an article about a hospital switching computerized medical record system and the CIO had a killer quote along the lines of "every extra day I take on this migration, people die"
(Disclosure: I work at SAP, though not on the ERP stuff or anything of that sort.)
From what I've been told, this is actually supposed to be a selling point of SAP: They have built tools to fit the processes of the industry leaders, so by buying into SAP, you're buying into the winning way of doing business.
I am not endorsing this opinion or making a claim regarding its veracity, just stating that that is what I have heard.
In the late-90s I worked for a manufacturing company in a firmware dev capacity. They did everything on paper still. They migrated successfully to an in-house built ERP system sitting on top of Oracle. Big celebration, everyone happy. Six months later someone drove a forklift through the wall of the machine room into the UPS which caught fire and destroyed three racks of kit including the Oracle node. Turns out no one really trusted the system and was running paper on the side. When I left 6 years later they were still doing it on paper and reporting on Excel. It works and is considerably more forklift proof.
I know a manufacturing plant that used an Excel spreadsheet to do all its production planning. There was only one person who understood the spreadsheet and could modify it, a consultant who made more than the plant manager.
"Understandable and fixable" depends more on the complexity of the application rather than the fact it's in Excel.
Agree. IT has forgotten that computing should enable more people to be producers instead of mere consumers. IT management cares about control, audit, permissions and expense - no focus on achieving productivity in the workplace and in many cases are anti-user.
If you try running a business where several workers get involved with fixing and extending information systems (e.g. spreadsheets), you'll soon understand why successful IT management cares about controls, audits and permissions.
I would argue that excel being "fixable" by regular office workers is half the reason why these projects fail in the first place. I've worked on migrating people's reporting to BI platforms before and what looks like a simple spreadsheet produced monthly is often really 12 different sets of formulas, special cases, kluges, hard-coded data and long-gone sources etc. etc. Because instead of correcting the source of data used for the report, it's all "done in post" in the excel sheet itself by a regular office worker.
Yes, important point. I wonder if a system could be made that made things properly fixable. As in, a regular office worker spots an error. Can fix it in post. But in doing so they will also create a change suggestion for the source data that can be approved and committed by master data admins.
The fact that every office worker understands excel, does not mean that every office worker understands every excel sheet.
Most of the projects we did in consultancy dev, was turning that one critical excel sheet nobody but 'the excel guy' understands into a simple to use web application, so that everybody could use it and the business won't explode when mr. excel would leave the shop.
Most of those office workers were not capable of fixing anything on the first day they used Excel. Many didn't understand it at all. The main difference isn't that Excel is super accessible and easy to use for non-technical people; it's its ubiquity, and especially that of training on its usage.
Ubiquity is important but it's not the only important factor. An excel sheet can typically be downloaded and experimented with. You can't download an ERP system and try stuff with it.
This is so huge in finance. Lots of smaller shops will hire data scientists or even SWEs hoping to up productivity and replace slow Excel sheets, and end up disilusioned when the team just glues together some Python scripts with no UI and no way for stakeholders to tinker without talking to someone else first.
On the other hand, once you have a well-established IT automation around your production, and people aren't trained in pre-automation production, it's actually quite hard to go back to manual.
Probably also depends on the complexity of the orders and workflows.
Without software, drones are useless. I suppose they can still assemble manually operated quadcopters if they know their inventory by heart, but they will be unable to produce more parts by 3D printing or drones capable of stable flight, autonomous operations, surveillance or any more advanced use cases. Even remote control is probably out of the picture.
They can continue to run the same thing they had before.
As an old software engineer, I can say with certainty that software engineering is a very, VERY wasteful practice. We could all be running Windows 3 right now, DOS, or some old Unix. The overhead involved in making actual advancements shows our slow progress as a species, and that we’re in a thread discussing a drone manufacturing facility being blown up in a war and how much that matters.
I think the natives had it right to live off of the land peacefully, and if anything to devote full time on science to determining what we do to help life survive in the universe.
I can't agree with you. People have got their human mind as a result of ever increasing and self-inflicted costs driven by a competition among males. They developed minds to play politics and they came to a point when 20% of metabolism of human body was devoted to its brain.
The result of such a wasteful way to spend their energy resources? Humans colonized all the Earth, drove to the extinction almost all big animals, and now there are as much humans on the Earth as mosquitos. Looks like a win, doesn't it?
These things go off the rails sometimes. Just today I've found a new example to it:
highlanders who had practiced brutal initiation ceremonies “in which they were forced to drink only partly slaked lime that blistered their mouths and throats, were beaten with stinging nettles, were denied water, had barbed grass pushed up their urethras to cause bleeding, were compelled to swallow bent lengths of cane until vomiting was induced, and were required to fellate older men, who also had anal intercourse with them” gave them up after only minimal contact with outside disapproval. Some later told anthropologists they felt “deeply shamed” by their treatment of their own sons and were relieved to stop.[1]
The waste of resources into useless things doesn't lead to good outcomes each time, but I believe that software engineering will lead to something. I'm not Jesus, I can't predict exactly what the beneficial results will be, but at least I can point to a growing ability of engineers of handling complexity. It lags behind their ability to create complexity, but still it grows.
No, they’re right, manufacturing machines like these are independent. We’re so used to interconnected software systems for everything, but even though these things may run old versions of Windows in airgapped or isolated networks, that’s just to run the machines. You give it a thumbdrive, save a part file on it, and as long as it’s got power, materials, and whatever is necessary for basic safety like noble gas for sintering safety, you’re set.
Even accounting systems are able to usually run fairly independently.
It’s not that IT and business and manufacturing support software engineers don’t help, but they aren’t necessary, especially if they’re just making the same thing over and over.
Russia is also behind in modern technology by over a decade. I'm pretty sure if the CIA wanted they could destroy a lot of Russian software infrastructure, but it suits them to be in and out of Russian systems collecting information instead.
This seems a bit of a stretch of a claim to make. In what ways would you say that Russia is a decade behind?
I visited Russia a few years ago. Commercially, they have all the same technology we have (for me, in the UK). Like us, they outsource most of their manufacturing to China, but internally they produce software equivalent to (or to be honest greater than) what we produce. The difference seems to be that a lot of Russian software, websites and apps are more local, which gives the illusion that it's not as good. Google is multinational, whereas the equivalence Yandex sticks to Russian and Slavic language countries. I was actually quite surprised to see in some areas they are ahead in digitising things (government services, payments). I expected the opposite.
Whatever software you can think of originating from the US, UK, or wherever, Russia has an equivalent. The major difference isn't the technical ability, but the commercial and cultural reach of that technology. Most of the world is happy to use Facebook, except Russia (and some others) who uses VK. We don't use VK, because it's Russian and we already use Facebook. Google, Facebook, Twitter, Uber (all artificially high value commercial products) have Russian equivalents. Sometimes they are even combined into one (Yandex has an Uber-like service within it). And when it comes to hardware, none of us are particularly strong with that. We all designate that to China, who sells it to all of us equally.
Whenever we hear about cyberwarfare, cybercrime and exploits, we usually pin it on Russian/Chinese speaking hackers. Russia seems to have better primary, secondary and tertiary education in computing, and, like the rest of Eastern Europe, produces many of the better programmers (something you can see in open source communities). From discussions with Russians, the level of maths, science and computing education is higher at a younger age than it was for me in the UK. Quite a lot of what would be A-level (18) Maths in my country was taught at Russian secondary school level (16).
In warfare, Russia has made fools of themselves in Ukraine, but on the other hand war is (sadly) the greatest contributor to military evolution. We see that with the introduction and evolution of drone warfare. Our UK Challenger tanks have been disabled and destroyed by far lower cost drones. All the technology associated with that (comms, jamming, avoiding jamming, self-targeting) is being rapidly developed by both Ukraine and Russia on the battlefield right now.
Where exactly would a decade back put them, technologically speaking?
> This seems a bit of a stretch of a claim to make. In what ways would you say that Russia is a decade behind?
Every country had it's own facebook. The difference was not features but scale.
Russia scales to million of users. Facebook/Google etc. scale to billions of users.
Everybody use Office, Chrome, commercial CADs, etc. Russia has no alternatives in most of these categories, and where it has alternatives - it's usually global (i.e. mostly made by programmers paid by western corporations) open source project they fork and add a russian skin over it.
> And when it comes to hardware, none of us are particularly strong with that.
USA and EU design the top-end chips and make crucial parts of the machines that produce chips (see ASML).
Russia was left behind in 90s and tries to catch up using some open-source alternatives around RISC-V. But they have no capability of designing nor producing chips anywhere near modern desktop CPUs.
Russian Lancet drones use smuggled Nvidia AI chips for example. We do not use smuggled Russian chips :)
I have no love (or even reason) to support modern Russia, but this is just wrong.
Russia has multiple home-grown office suites. Besides MS Office, the market leader still is full of bugs that harken back decades.
They also have multiple commercial CAD programs (KOMPAS, T-FLEX) that scale up to everything including airliners.
As for those 'western' top programmers, especially good ones, you'd be surprised how many of them are from post-USSR countries, including Russia (and Ukraine, Belarus, Kazakhstan etc.).
As for chips they are behind (for reasons beyond the scope of my post), but the post mainly extended to software, in which, many of the supposed crown jewels of the West (aka US) have been replicated quite successfully in other parts of the world including Russia.
> I was actually quite surprised to see in some areas they are ahead in digitising things (government services, payments). I expected the opposite.
Why were you surprised by this? Russia is a totalitarian dictatorship, it is quite expected that systems of total control will be actively implemented there. And what is better for total control than digitalized things?
> Russia seems to have better primary, secondary and tertiary education in computing,
I've talked to many Russians and this is complete bs. The quality of education is quite low, but due to the competition created by remote work, programmers were easily paid 5x times more than people with comparable qualifications in other fields. So a lot of youth with a good work ethic put a lot of efforts into self-education in this fields even if they have no access to any systemic education in computing at all.
In other words, in Russia, as in other Eastern European countries, you either do programming, or you are screwed. And the advantage of mathematics is that you don't need a teacher for it (for school level), everything is in the book, one thing after another. All you need is work ethic and motivation.
>Russia is a totalitarian dictatorship, it is quite expected that systems of total control will be actively implemented there. And what is better for total control than digitalized things?
Ah yes, the reason the US is years behind most other country's payments systems is because they have too much freedom.
What do you mean by modern technology? Surely not the software. Russian engineering culture is strong and their IT strategy is far ahead of what you can find in Europe. I doubt it’s easy to hack into their systems - this breach illustrates it quite well, actually (it’s rare and required focused effort).
Fwiw I actually agree with you. My point is that early in the war, it was commonly thought that just the western sanctions alone would totally cripple the Russian economy. Or that they'd soon run out of arms, or anything like that. None of that happened. It's not pro-Russian to establish that they were more resilient than what many people anticipated/hoped. This doesn't take anything away from Ukraine's resilience in the face of years of obscene unwarranted aggression which is easily 10x more impressive to me.
You can help instead of waiting for politicians to "do something about it". It's not that hard to find a reputable organization that helps Ukraine and send it some money.
> In 2024, charitable giving in the US was $592 billion. $392 billion of that was from individual donations.
That's a single-digit percentage of the US Federal budget.
Some of that goes to "family foundation" sinecures. Some of it goes to 10% church tithes. Quite a bit of it is spent on… raising funds. (https://www.cbsnews.com/news/when-giving-to-charity-ask-wher... - "Of the more than $1.3 billion raised by charities in the [New York] in 2018, about $369 million — or 27% — went to pay professional fundraisers' fees")
> If 1% of our donations went to Ukraine, that's not a number to casually dismiss.
I think that's wildly optimistic, but that'd be somewhere between $3-5B. The US alone has earmarked something like $200B thus far. The EU has given a similar amount.
Every bit undoubtedly counts, but a single Patriot battery costs $1B.
I would be surprised if they could manage to keep refilling their squirt guns and deal with all the logistics required to keep an army available to use them
To be fair, it is quite difficult to support a regime where random people are grabbed off the streets and sent to their deaths. Where for expressing oppositional opinions your male relatives will have their door kicked down and will be sent to an assault on enemy position with a 90% mortality rate. And if they survive that - to another one just like it. To support a regime that has no long-term plan and goals for waging a senseless war and which openly promises to commit genocide and ethnic cleansing in the reclaimed territories.
So the support from Western countries is enormous, considering all these aspects.
I'm a big supporter of Ukraine, but let's be honest
People aren't being dragged off the streets in Russia. This was briefly true in mid-late 2022 when they flirted with a partial mobilization, but hasn't been true for a while.
This is (sadly) actually more true in Ukraine. But there's also some nuance there - they can stop and question but supposedly they technically can't use physical force anymore.
What Russia is doing is increasing the bonuses and salary for signing a contract. And they don't have manpower problems for the most part - Ukraine is the one having that problem.
Now the Russian military is doing alot of shady shit, like promising recruits they won't be sent to Ukraine or would serve only in rear areas (even the US military recruiters were frequently guilty of this tactic). Classifying certain infantry units as "disposable" (especially foreign recruits and those from less politically unimportant regions), basically to be used as bait in assaults. And I'm sure the pressure for the required conscripts every year to sign a regular contract so they can be deployed is quite great, but its nothing like what some would have us believe.
Reminds me of the Louvois[1] disaster in the French armed forces, they fucked the payment system so bad they had to roll back to manual accounting for a moment. Yes, for the whole French Army, which was at the time involved in multiple foreign operations…
Working for a manufacturing company, you may be making drone parts, and you don’t really know which side of the war you’re making them for, because they can buy individual parts through different reputable-looking companies.
You could also be making surgical parts that help save lives.
Overall though, I think I’d rather be making nice practical furniture that hopefully people would never throw away. While I support people that want to protect, war is horrible.
