The Bug That Hit 70% of the Web, and Why Your Site Never Went Down
Last Tuesday night, October 20, the PHP project put out a notice. Emergency release coming the next day. Two critical bugs. No details.
If you don't know what PHP is, here's the short version. It's the language that runs most of the websites on the internet. WordPress runs on it. Most of the sites I build and host run on it. Roughly 70% of the web does. When PHP has a serious problem, everybody has a serious problem.
This one was serious. The bugs, now called PHP2Shell, let an attacker run their own code on a web server. No password. No login. Before your website's own code even starts. Within hours of the details leaking, exploitation attempts were in the millions. Amazon later found someone trying it against one of their decoy servers on October 17. Four days before a fix existed.
By the morning of October 23, every server I run was patched. No customer site went down. I found no sign that anyone got in.
I want to walk through what happens behind the scenes when something like this hits. Not because it was heroic. Because most people never see this part of the job, and it is worth knowing who is standing between your website and the people trying to break into it.
What happens when a bug like this is found
Somebody finds the flaw. Sometimes that is a researcher who reports it quietly. Sometimes it is a criminal who starts using it without telling anyone. This time it was both. Amazon runs decoy servers to catch attackers in the act, and one of those decoys logged an attempt on October 17. Four days before anyone outside a small circle knew the bug existed.
The people who maintain PHP build a fix in private. Then they announce a release date without saying what the bug is, so the companies that run websites can get ready without handing criminals a map.
That is the plan, anyway. This time the details leaked about twelve hours early. Once that happens it is a race. Automated tools start hitting every website on the internet, looking for one that has not been fixed yet. Within hours, attempts were in the millions.
Then the fix ships. And here is the part people miss: a fix sitting on a download page protects nobody. Someone has to install it on every server in the world. One at a time. Without breaking anything. That is where hosting companies and people like me come in.
What I did, in plain English
I started the night the warning went out, before the fix existed. You don't wait. I put up temporary barriers based on what was known at the time, to slow an attacker down while the real fix was on its way.
One of those barriers knocked a server offline for a few minutes at 8 in the morning. It was the one step I didn't test on a copy first. I had it back within minutes, and I mention it because the honest version of this story is more useful than the clean one.
When the official fix and the official details came out, it turned out the early reports were only half right. So I threw out my guesses and rebuilt my defenses around what the PHP project actually published. I also set up alarms that would tell me if anyone managed to get in.
Then the real work: installing the fix. I never apply a fix to a live server first. I test it on a copy. That test caught a compatibility problem that would have knocked every website on that server offline the moment I restarted. I solved that first, then rolled the fix out server by server over two days, checking after every step. Forms kept submitting. Phones kept ringing.
Finally, I went back through the records, all the way to before the public announcement, looking for any sign that someone had gotten in before I closed the door. I found none.
The reading is half the job
Here is something nobody tells you about a bug like this: the hardest part is not installing the fix. It is figuring out what to believe.
Within a day there were official advisories, vendor bulletins, security research write-ups, news articles, forum threads, and a hundred blog posts from people who had read the other blog posts. Some of it was right. Some of it was half right. Some of it was flat wrong and very confident. And a fair amount of it was written by companies with something to sell you.
The first explanation to leak was the one everybody repeated. It was wrong about half the problem. If I had stopped reading there, I would have patched the wrong thing and felt good about it.
So you read. You check the official source against the researchers against the people who actually got hit. You figure out which three documents matter and set aside the other three hundred. On this one, the reading took longer than the patching. That is normal. Finding the right information is the job. The install is just what you do once you have found it.
Who is protecting the web
This is the part almost nobody sees. There is no company in charge of the internet. Nobody sends out the fix and makes sure it lands. When something like PHP2Shell hits, protection comes from a chain of people who mostly have never met:
- The PHP maintainers, a small group who keep the language running for most of the web, who built and shipped the fix on an emergency timeline.
- Security researchers who took the bug apart and published exactly how the attacks work, so defenders could check their own systems instead of guessing.
- The companies and volunteers who package the fix so it installs cleanly on each kind of server.
- Hosting providers and technology teams, thousands of them, who install it on millions of servers, one at a time.
- The people watching the logs afterward, checking that the door really closed.
Every link in that chain hands off to the next. If any one of them stops, the fix never reaches your website. The whole thing runs on people doing unglamorous work, fast, at all hours, for businesses that will never know their names.
What I want you to take from this
Test before you touch. Every risky step I tested on a copy first, and that is what caught the problem that would have taken every site on that server down. The one step we skipped it on gave me my one outage. That is not a coincidence.
Go by the facts, not the first headline. The early reports were half right. If I had built everything around them, I would have felt protected and been half exposed.
Knowing your systems is the job. The difference between a scare and a crisis is whether someone already knows what is on each server, what it talks to, and how to fix it on a Tuesday night with no warning. That knowledge is not glamorous. It is the whole job.
Someone is always trying the door. There are people whose whole job is getting into websites like yours. They work nights, weekends, and holidays, and they do not announce themselves. Staying ahead of them is a 24/7 job with no finish line, and when it goes well, nobody notices. Nobody sends a thank-you note for the break-in that didn't happen. I do it anyway.
If you're a contractor reading this
You didn't get an email from me last week asking you to do anything. No DNS changes. No "please log in and update your plugin." No downtime notice. That's the point.
You hired me so you can run an HVAC company, or a plumbing company, or a roofing company. Not so you can learn what a multipart POST is. A good partner absorbs this kind of risk so you never see it. If your vendor hands it back to you, that tells you something.
Every company that runs a website had to deal with PHP2Shell. Every one. So it is fair to ask the people who host anything for you a few questions:
- Were you affected? (If they say no, ask again. Nearly everyone was.)
- When did you start working on it, and when were you done?
- Did anything go down?
- Did you check for signs of a break-in, going back to before the public announcement?
- Do I need to do anything?
You don't need to understand the answers in detail. You need to notice whether they have answers. Vague is a red flag. "We're looking into it" a week later is a bigger one.
Keep it simple, make it look beautiful, have it work
That's the line I use for everything I build. Most weeks, the first two get all the attention.
Last week was about the third one. Nobody saw it. Nothing changed on your site. The forms kept submitting and the phone kept ringing.
That's what working looks like.