TL;DR
If you are in the infosec / cyber security community or even in technology, you have probably heard of the near miss with the xz supply chain attack.
If not, the brief summary is as follows: likely a nation state, likely Russian, mounted a campaign for over 2 years to build a set of personas that ultimately got Maintainer access to an open source Linux compression suite that is widely used (xz) and used that to backdoor SSH.
The backdoor would have ultimately given full remote code execution with controlled authentication (CVSS 10) to the controller of a private key. There is a high likelihood this would've made it to basically every Linux and Unix install worldwide.
They would've got away with it too if it wasn't for those dam
kids… nerd that benchmarks his SSH login performance and doesn't clearly have enough paid work from Microsoft :)Even though this specific attack didn't work, they likely have multiple such campaigns running currently, in the past and definitely the future. When the 5 eyes does it or has done it in the past, it maybe even better / to this date undetected.
Open source is also not the only risk. It is not that hard to get a rogue developer into a commercial company, especially where outsourced development is done. Maybe there is a bit more scrutiny there but there are also a lot fewer eyes.
What won't work against this type of attack is the traditional supply chain security nor even Software Composition Analysis (SCA).
What will work IMHO is the boring stuff: benchmarks of what processes are doing on your compute and network traffic to identify anomalies and actually using your firewalls effectively. This not cheap nor easy to do.
More details on the xz attack
There has been plenty of ink and podcasts on this.
Some of my favorites:
The humint side of the xz supply chain hack:
https://pca.st/episode/792390c8-c389-4673-9cd8-9611d18d76c9
Riskybiz interviewing the guy that found the bug:
Top ranking posts on Hacker News on the topic with discussion:
https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=XZ&sort=byPopularity&type=sto
What won't work as a defense
Standard supply chain security i.e a questionnaire IRAP assessment etc.
Obviously if it is open source there there is no one to ask that will answer. Do you do background checks including criminal record checks on your maintainers? Strong identity verification? Obviously not in majority of cases.
Even if you don't use open source in your crown jewel systems, obligatory XKCD:
It's not like closed source is immune to this type of attack. Any employee or outsourced developer can be bribed or extorted. Closed source also likely to have a single / few peer reviews of the code and if they don't pick it up and neither do any tests, off it goes into your production environment. This is pretty likely if a team of nation state threat actors is being paid to develop and hide the back door.
Software Composition Analysis (SCA)
This is the modern approach to technically mitigating vulnerabilities in artifacts including libraries and dependencies. Even then how many companies have it working effectively? People process and technology. Do you scan all your artifacts before deployment? Both custom developed and commercial off the shelf software? Do you have people and processes to test and fix vulnerabilities your SCA identifies?
Even if you do SCA effectively, it is less useful against this type of attack because:
It was signed code from the official repo, by a persona that had access for over 2 years, been peer reviewed.
Do you even compile something like RedHat Linux from source and scan all its dependencies?
This would not have had a CVEE entry or signature.
It had significant obfuscation. Maybe too much in this case that led to it's detection.
Allow listing
I.e. only allow executables and child processes to run from binaries that are in the approved and tested image.
Normally a highly effective control. In this type of attack the backdoor is in your approved image. It was SystemD spawning the SSH process and users logging in with a legit key.
Every process that has to be allowed in most Linux installs.
EDR / anti-malware
Again just a regular SSH user logging in.
What Would Work
Baselining compute and network traffic
Basically how this attack got discovered, SSH should not spike your CPU 100%. Do you know what the CPU, memory, disk read write profile of at least your crown jewel compute? Do you have alerts and automated actions for things outside of this baseline? Obviously this is much harder in PaaS and SaaS.
Similarly do you know what network profile of all the processes running on at least your crown jewel compute? You can do this with an agent on the compute or by capturing network traffic and analyzing it.
If you must have AI in everything to get funding or approval, this type of clustering or peer group analysis is what AI or machine learning algorithms are great at.
Network filtering still has a role
The death of the firewall has been greatly exaggerated.
Do you need to login to your compute at all?
Is your management access to compute like SSH or RDP Internet accessible?
Can any of your systems be accessed before authentication to your identity provider? Even your SaaS systems can have a simple rule where the WAF only allows access to the URL after authentication.
If you can implement this pattern, even for SaaS systems, it means an attacker has to authenticate to your Internet accessible Identity Provider e.g. Microsoft, Okta, Ping before being able to connect to the target system. The access can only be for the specific time needed and the specific service/port. This makes it a lot harder to exploit even a backdoor vulnerability to your system.