The topic of my research right now is a subset of this; it essentially researches the quality of the outputs of LLMs, when they're writing tight-fitting DSL code, for very context-specific areas of knowledge.
One example could be a low-level programming language for a given PLC manufacturer, where the prompt comes from a context-aware domain expert, and the LLM is able to output proper DSL code for that PLC. Think of "make sure this motor spins at 300rpm while this other task takes place"-type prompts.
The LLM essentially needs to juggle between understanding those highly-contextual clues, and writing DSL code that very tightly fits the DSL definition.
We're still years away from this being thoroughly reliable for all contexts, but it's interesting research nonetheless. Happy to see that someone also agrees with my sentiment ;-)
> I do most of my work over SSH on big metal machines, maybe that's the disconnect?
Yeah, I believe that's where the disconnect is. I moved from a Thinkpad to the 16in Macbook Pro with the M3 Pro chip, and I am able to reliably build and write code that runs locally on 5 different Docker containers, for at least 10 hours. I once did a 48hr hackathon with this laptop and I only had to charge it I think 4 or 5 times. I need to be very mobile as I'm going to different locations to attend meetings or write code, and it's able to do everything reliably for a (very extended) workday.
I would have to move from wall socket to wall socket on my old Thinkpad, but something to note is that I was using Windows 10 at the time. The Macbook's best-in-class (in performance-per-watt and per-kg) hardware combined with the software was something that became unbeatable for my workflow.
That being said, my next laptop will be a reliable, non-Apple, but Apple-like performance, ARM64 laptop, and I'll be using some Linux distribution on it.
It's not that I actually write code over SSH (I usually work with wifi disabled), more that I just push things to big machines for building and testing them, rather than doing that locally. My pair of 32-thread Ryzen machines are worth their weight in gold for the amount they juice my productivity. No laptop can ever touch that, for obvious physics reasons.
I can do everything you're describing with my XPS13. I regularly go days without plugging it in.
Some years ago I researched the whole credential stuffing ecosystem for a course paper at uni.
Credential Stuffing is (or at least was) a gigantic market, and it is one of the biggest headaches for the biggest pay-walled services, like Netflix, HBO, Prime, etc.
The people that made a living out of it were stuffing millions or billions of credentials (sourced from database leaks) in the most popular services, hoping to then sell the accounts for small amounts of money, like a dollar for a Netflix account with a 10-day warranty. It's a numbers game at heart with a substantial technical aspect, where you need to optimize your checker code to essentially send properly formatted requests that can't be intercepted and don't arouse suspicion, and then you had an ecosystem of "methods" that are certain request-response chains that make your login request look like it's from a real person. People needed to figure out advanced methods to not invoke a CAPTCHA check, which is cost-prohibitive, but not impossible to solve automatically (AI wasn't a thing back then). You then have to buy millions of proxies that are able to route the requests from different IPs so that you're not sending millions of requests from a single IP. Checkers had reached a point where, depending on your proxies, were performing 10,000 or even 20,000 checks per minute. Multithreading was the cornerstone of these technologies, as a simple 2vCPU VM was already bottlenecked by proxy speeds.
Back when I looked into it, it was the wild west, as SSO and other technologies just weren't a thing yet. Companies would become fads of this credential stuffing scene, and it would take a dev team an entire sprint just for them to make a login page that was able to at least force a CAPTCHA check for each single request, and that's IF they had the proper monitoring tools to notice the gigantic spike in login requests. Having a valid account to a service like Ebay where you can then order whatever you want with the linked credit-card, you can understand how big of a security issue this is.
I haven't looked at it recently, but I assume that this has become vastly more difficult for the common-place services like streaming providers and digital goods marketplaces. SSO, IAM platforms like Keycloak, and advanced request scanning techniques have evolved. I'm guessing things have become substantially better, but it's always going to be a big issue for those smaller websites without a dedicated dev team or without at least someone maintaining them.
Password managers and 2FA mostly solve the problem of credential stuffing but unfortunately millions of people still don't use it. 2FA is actually pretty good unless your account is a high profile account then bad actors might try to SIM swap[0] you and hijack your mobile phone number and essentially bypass 2FA. But there are others ways of 2FA too.
I recently worked at a big home lighting company, working on the OS of the router device that communicates with the light bulbs themselves and the internet/user.
Our OTAU architecture uses A/B system updates [1]. Core idea is that both the kernel and the rootfs (read-only) partitions had 2 different bootslots in storage, and the OTAU would only write to the bootslot that is unused. Hence, if something went wrong, the system would automatically fallback to the previous version by just switching the bootslot used. Over the numerous years that that architecture was used, I couldn't find a single post-mortem that resulted in devices being bricked. Something to note is that the rootfs partition was overlaid with a writable partition for persisting state data etc.
Now that was a $two-figure USD device, not a $5/6-figure USD electric SUV. Is this a cost-cutting measure? At those price levels, doubling your NAND size is not even half of a percent of the total cost of the vehicle.
Unless there was a serious issue that the used bootslot corrupted the unused bootslot, then I don't see how this could have happened.
It's saddening that car manufacturers are so unserious about the code they're deploying.
I've worked in both IoT lighting and automotive, so I'm comfortable comparing the two. This also isn't offered as a defense.
The big auto OEMs are just as sensitive to absolute BOM cost optimization, regardless of the percentage increases. I don't think this was a bootslot issue though, regardless of the word "bricked". Even as backwards and ill-advised as auto software can be, generally accepted practice is that updates are impossible while the vehicle is in motion. This is usually enforced by systems shared across multiple OEMs through the tier system.
The situation sounds more like a disastrously buggy new firmware.
I wouldn't put either past stellantis though. The auto industry already scrapes the bottom of the proverbial barrel sometimes, and stellantis isn't exactly known for their top of market compensation.
Until this year, my Hyundai let me install updates while driving. Apparently the nhtsa or someone made them stop because they weren't meeting the standard of being able to activate the backup cam within 2 seconds
I quite like the new Hyundais, but, knowing the horrific state the Korean software industry is in, and the general horrific mentality around software in Korea, I cannot help but wonder how that impacts the quality, longevity and user experience of their cars.
I've had my car for 2 years and I still love it, although a bit early to discuss longevity.
There are some minor annoyances with the software, but their infotainment system is better than most. I was surprised when I test drove some other brands and the UIs in NEW cars were visibly dropping frames.
The only bummer is that they're more oldschool than brands like Tesla/Rivian when it comes to software updates. When a new generation of the vehicle comes out, older cars don't get feature parity with the new software, just maintenance updates. There's a few inexplicable bugs that have never been fixed in my car and most likely never will. None of them are show stoppers, just irritating.
This is generally how other devices work as well - for example all Android devices and Android-derivatives (which many of these cars are!) use a similar A/B partition to prevent bricking.
It definitely reduces the risk of updates, but it absolutely doesn't eliminate it.
The A/B updater itself is a surface area - especially if the logic is complex and there are other child devices that are updated at the same time (likely for cars). In that case you're not just coordinating between two independent partitions, you're coordinating between 2 * N partitions, half of which have dependencies on each other.
Also, the key bit of the mechanism is that upon successful boot the new partition is flagged as "good", which causes flags to be set to assert that the update was successful and the backup partition is no longer needed. That logic can (and does) fail - if your failure point occurs after this checkpoint you're hosed still because you're past the point of no return.
Making that worse is that in most systems you want the "it's all good" checkpoint to occur early - you don't want to, for example, wait multiple minutes for all user services to come up. But that also means that if a critical failure happens in said services, you're past the checkpoint.
> Now that was a $two-figure USD device, not a $5/6-figure USD electric SUV. Is this a cost-cutting measure? At those price levels, doubling your NAND size is not even half of a percent of the total cost of the vehicle.
Could just be a competence and priorities problem. If it's cost cutting, it feels way more likely that some PM cut some story from a sprint to hit a deadline (and objections were either not raised or ignored), than they did some engineering analysis and explicitly decided to save $3 per vehicle by cutting the NAND size.
Edit: Actually, I don't think that technique would have helped, the problem wasn't a botched update, but a seriously buggy one. From the OP:
> The buggy update doesn't appear to brick the car immediately. Instead, the failure appears to occur while driving—a far more serious problem.
> Edit: Actually, I don't think that technique would have helped, the problem wasn't a botched update, but a seriously buggy one. From the OP:
That and combined with general refusal of new automotive bootloaders to downgrade. You can go only up in versioning. So even that you could have working version on second partition, it will never get loaded because it has lower version than currently one you are running.
1) Total cost of the vehicle does not matter. What does matter is the operating margin. Half a percent of the total cost of the vehicle will move them from 2% margin to 1.5% margin. (Ford has operating margin of 2% as an example)
In other words an increase in 0.5% cost of total vehicle will reduce their profits by 25%.
That’s a huge number now! Note also that car manufacturers are in a bad spot because their volumes are fairly low (smartphone = 1M/yr, car = 40k/yr) and have harsher requirements for chips, driving the cost way up.
2)AB updates are great, but they can still fail or get soft locked. Especially important around code when you configure the slot to be good and when bad.
> the system would automatically fallback to the previous version by just switching the bootslot used.
That's the hard part though.
It's shockingly common in my experience to have an A/B boot setup, but no actual logic in the userspace application to switch back to the other partition if something goes wrong. It's just a defense against somebody pulling the plug during the OTA, it doesn't protect against software bugs at all.
I once managed to brick a PC motherboard that advertised "dual BIOS". It didn't fallback to the previous version after a botched BIOS update.
It's totally possible that the update corrupted the other bootslot as well. If those blocks aren't off-limits to the updater program, it's just an off-by-one error waiting to happen. Slot 0 or slot 1?
Another possibility is that the updated version booted up just enough not to trigger the automatic fallback, and then got stuck in a loop.
I have heard anecdotally that auto manufacturers are sensitive to a price change less than $5/vehicle. This is better than some industries that are sensitive to $1.
What could easily have happened is that the negotiators didn't include A/B updates in their spec, or they only specced A/B updates at 1GB OTA size.
They do their usual hammering on price, and the head unit or ECU manufacturer gave them some savings by cutting storage space to the bone.
Maybe it was still enough for A/B updates, until the usual software bloat took the updates past the critical limit.
They could still do a safe update by doing an A/B/A update (where B is a shrunken, update-only OS), but that requires development time, and the engineers should already be working on the next vehicle.
Worked for them. Corporations with many brands in their portfolio might discuss for weeks over price differences of components of 0.20 Euro. That‘s twenty Euro cents difference for e.g. a USB connector. If you expect that a vehicle platform sells in the 10s of millions over its lifetime, you‘re talking real money very quick!
yet another example of the flawed logic where "we don't have time/money to do it right now, yet we always seem to find the time/money to redo it later after the shit hits the fan"
We used to do that with device that where in difficult to reach places with harsh uptime requirement! Think industrial routers and protocol converters. I think it pays for itself very quickly. Sending someone for such a device can get expensive.
Brick is now slang for a lot of fail conditions that aren't classically 'bricked.' This has become really common I've noticed. Honestly, this ship has sailed and isn't even worth fighting anymore. Its like Xerox asking people to stop calling copies Xeroxes.
We just never bothered to develop a new term. Maybe 'soft-bricked?' 'Semi-bricked?' I would like journalists at least to start using more accurate terms, but 'bricked' I imagine gets a lot more engagement and ad impressions, so here we are.
Wikipedia has a section about this. They call it soft bricked vs hard bricked, according to the difficulty of restoring device function and how the broken state presents. Even hard bricked is usually recoverable with appropriate tools, so it is a spectrum.
I’m sorry, but you’re incorrect the vehicle completely shutting down while driving and not working again until you put it into park and then it’s shutting down five minutes later is effectively bricked and extremely dangerous. Myself and my family almost died just trying to get home from dinner. It was a complete loss of propulsion and power steering.
Being temporarily unusable is not how I've seen "bricked" used, bricked means unrecoverable and the item is completely unusable except for as a brick/paperweight/door stop.
Then it would better be described as a life-threatening event rather than a bricking - especially as, in the hierarchy of concerns, the former is more serious than the latter.
And then it was fixed with another OTA, so it was not bricked. Why bring up this pedantic point you may ask? Because the grandparent raises a scenario that doesn't apply here. A/B updates or not were not at all the issue here.
I for one am always grateful when things are engineered thoughtfully and with redundancy as it is symbolic of respect for the people who are your customers. Especially in something as important as a car, "can be bootstrapped from a central computer" - when? how easily? how reliably? - is not good enough because things will inevitably go wrong for some portion of the user base.
Well, on the positive side, at least they were stationary unlike these vehicles. Don't get me started on botched OTA updates, there are so many ways companies get those wrong it's not even funny.
All those words you are saying, it's quite possible the sub-contractor to the sub-contractor to the sub-contractor in a foreign low-cost country that actually did the work has absolutely no idea what any of that means, and they are doing the bare minimum to deliver
Problem with that statement is some of these “foreign low-cost” countries are building great cars while American vehicles have a global reputation for being garbage.
The only American-made vehicle that sold in any volume outside the US was Tesla and that is already over.
Why wouldnt a foreigner know what this means? This seems very xenophobic. And if US/Euro management is hiring these groups and not giving them requirements for redundancy then guess who is at fault? Not the contractor.
If I had no problem with devoting the time and money, contributing to the kernel (especially in a topic as obscure as making the extra buttons work on a 20-year-old laptop) is at the top of my bucket list, and I am definitely going to be doing it in the near future when my calendar clears up a bit.
Exquisite write-up and OP's simple writing has a motivating ring to it, and I'm now on the local used marketplace looking for pieces of tech like this :-)
What I did was buy a wifi router with a chipset supported by OpenWrt but with no build target yet. One of the OpenWrt maintainers then guided me in what I had to do to get the info he needed for a patch. I followed his instructions, he wrote the patch and there it was, a new build target. Later I wrote a patch myself to fix the LED control. It was a rewarding and educational experience. Later I even found someone using OpenWrt on the router I helped get supported which put a smile on my face.
If you want to find devices that still need hardware support under Linux, I highly recommend trying to get a mobile Linux distribution to work on an old smartphone or tablet.
postmarketOS in particular has a really good devices page [1] showing missing feature support at a glance, as well as guides for porting to new devices [2] and porting features from an outdated vendor-provided Linux fork to the upstream kernel [3].
I genuinely want to work on postmarketOS but I don't have the technical know-how right now but one day.
I would prefer a sort of dual-boot or just a simple ability to run linux in "android phones"
Like, if we were to build a linux phone somehow hacking it through a raspberry pi or the alikes, they would be so much more costly and subpar.
Android phones have whole globalization working on it and the only reason why they don't work is lack of drivers support/software side, something which can be worked on.
I think if society rewards something like PostMarketOS more/make it easier to install it, then things can be so great as well.
I know I can run a terminal inside android but till now it was only possible through qemu which had its issues...
I am not sure but I have never really appreciated having linux run inside a vm, I'd much rather run something like waydroid in a postmarketOS phone than linux inside android in an ideal world technically speaking from strictly performance but we don't live in one and installing waydroid on postmarketos or even installing postmarketos can be a very huge issue whereas installing linux on android can be a single step with termux or userLand.
It's possible to dual-boot. My favorite method is to have a device with a SD-card slot and have android on the eMMC but PostmarketOS on the SD-card.
Currently only do that on one of my older devices, would love to do it on my main device but when I bought it I was in a hurry to get it going and did not have time to unlock the bootloader. Unlocking the bootloader now means having to factory default the device but I simply have too much important stuff set up that it's not worth it. Apparently there's also no great way to backup all partitions of the Android device which I find to be quite strange.
You could also look into running mobile Linux on top of libhybris - it's a proprietary compatibility layer, but some people use it to get support for mobile Linux runtimes on more recent devices.
I'm actually trying this right now, but it is becoming much harder than I expected. The fact that I keep needing to recompile kernel and rebooting device doesn't help. I'm trying to make the display and touchscreen work consistently on a qualcomm soc.
We might have a different definition of issue.. I think 100% compatibile working would be launching bloatware installed by the manufacturer. I'm happy not to have the pavlovian training that may some day cause me to click one of these things on someone's windows machine.
I think it's more of the buttons perform specialized tricks to launch bloatware in Windows.
Some of the issue here is the keys themselves have almost no standardization, even across models. Hell, possibly in the same model sometimes. Some backend windows driver captures these signals via a 50 mile long series of if statements that make grown men weep when viewed. This later can mean your totally working fix for the kernel doesn't actually work on a 1/3rd of that fleet of laptops.
> I think it's more of the buttons perform specialized tricks to launch bloatware in Windows.
The linked article is discussing play/pause buttons as well as a "mode-switch" button that allows the play/pause button to have a second function. I do not understand how any of these regular functions become bloatware in your estimation.
> Some of the issue here is the keys themselves have almost no standardization, even across models.
There is actually widespread standardization, which is why many important keys work by default. Laptops sometimes have buttons to disable the internal wifi or adjust the keyboard brightness. These keys are less universal, but still hard to categorize as bloatware.
> ome backend windows driver captures these signals via a 50 mile long series of if statements that make grown men weep when viewed.
I don't know any grown men who would weep when viewing this. I'm confused that you do not like a simple solution (if statements, which a computer has zero problems following precisely even if it is complex to you) nor the complex solution ("bloatware")
> This later can mean your totally working fix for the kernel doesn't actually work on a 1/3rd of that fleet of laptops.
Most devices used in fleets are well-supported in linux after a few years, specifically because of users like the linked article who spend time making buttons worked when pressed.
Yep! It's basically just a button that tells the OS "open the system calculator"
~~
I looked it up, the Human Interface Devices usage "Consumer Control" code assigned to "Application Launch - Calculator" is 0x0C0192 or 0x192
This keypress is sent as a scancode/keycode, not an ASCII character. On Windows, this opens calc.exe by default, but you can change which app opens in response to the calculator key by editing the media key mappings in the Registry
You can obviously map arbitrary key codes however you want on a custom OS and have extremely little fear of someone having embedded nonsense down to the bios.
On windows many of these laptop buttons were added like the Yahoo browser bar to specifically work with bloatware that might go on to make a meaningful action for non malicious software as well as what it is really for.
I prefer not to be in the habit of pressing footguns given that I might occasionally be placed in front of a consumers windows laptop that no one cleaned.
> I prefer not to be in the habit of pressing footguns given that I might occasionally be placed in front of a consumers windows laptop that no one cleaned.
If you're this anxious about security, you might not want to be anywhere near a Windows machine.
I think they're trying to say avoiding a Windows computer is about as difficult as avoiding an automobile, and potentially just as fatal.
I think if they're honestly not being hyperbolic, they should find a less technical career or hobby. If you're afraid of flying, don't join the Air Force.
I'm not sure if there is a word for people who choose high risk activities because they have less fear, because I'm not a coroner.. Perhaps because I'm not sure I have a healthy enough fear of death.
If you examine the stats with the assumption that you could be one who dies early and probably won't take up opioids then there is some logical reasoning to do about your relationship with cars and death. "Mistaking" that for fear is a defense mechanism.
The situation of the worse is better crowd and computer security increasingly looks like one of these things that I don't want to apply such a defense mechanism to.
Linux has not really gotten us closer to eliminating Windows yet it has eliminated a lot of the other approaches to eliminating Windows. Maybe it is only on par with the escooters that can now run me down on the sidewalk but don't seem to have reduced my risk from cars.
> I'm not sure I have a healthy enough fear of death.
You seem to have a very healthy fear of other things, like drivers or scooters running you over, and media buttons on computers launching life-altering software. I'm betting your fear of death might not be healthy, but possibly unhealthy in the sense you're spending your life worrying about things that... well, likely aren't gonna happen to you.
No external compass, something the group views as (or probably unavoidably knows to be) dangerous. Overcoming the fear of it and then belittling those who haven't completed the same process. Psychologists call that the normalization of deviance. A commercial airline pilot has the external compass in your questionable aviation example.
You don't want a group with initial success with the left, right left and so much enjoyment from the sense of fulfillment in social deviance process that they think they can take over everything.
When did every mundane choice became a life-or-death calculation for you? Media buttons aren't footguns. Scooters aren't assassins.
The gap between reasonable caution and this kind of hypervigilance isn't wisdom, it's self-imposed paralysis.
Running pedestrian safety and keyboard shortcuts through the same catastrophic framework is a system that's eating itself - every interaction becomes a referendum on survival.
When people point out that you aren't serving yourself with this worldview, you seem to take the disagreement as proof that others just don't see what's coming.
That's not living... that's white-knuckling through existence, mistaking the grip for control.
By the way, delving into obscure and hardware-specific kernel code, sometimes yields quite interesting generally-applicable problems. For example, @dougg3 did an (excellent!) series of articles about his work on bringing mainline kernel support to "Chumby 8", a somewhat obscure, but historically significant piece of hardware, and as a side-quest he stumbled into this:
One example could be a low-level programming language for a given PLC manufacturer, where the prompt comes from a context-aware domain expert, and the LLM is able to output proper DSL code for that PLC. Think of "make sure this motor spins at 300rpm while this other task takes place"-type prompts.
The LLM essentially needs to juggle between understanding those highly-contextual clues, and writing DSL code that very tightly fits the DSL definition.
We're still years away from this being thoroughly reliable for all contexts, but it's interesting research nonetheless. Happy to see that someone also agrees with my sentiment ;-)