Over the weekend, I got an urgent email from our website. It was in trouble and needed my help. We were under attack and while our security systems and firewall were able to stop the attack, the situation still required my immediate attention.

Just to be clear, during the attack at no time was our website compromised nor was anyone who visited our website during or after the attack at risk in any way.

Here’s what happened and how our preventative measures stopped it.

On Saturday 2,053 comments were made on our website’s blog to recent posts. Our spam filter immediately identified the comments as spam meaning that they came from an automated system not a real person. While our spam filter could not detect that many of the comments included malicious code, it was smart enough to quarantine the comments to ensure that they didn’t cause any harm.

The malicious code, or malware, embedded in the comments was a SQL injection attack. This type of attack tries to insert code into a form field on your website. For example instead of entering a real comment into the comment box at the end of the post the attacker enters code that if successfully executed on the server will try to do something nasty. SQL injection attacks are designed to do their work in databases and when you enter a comment or any text into an input box on a website that information (real or malicious) is stored directly into your website database. This is where it get’s the name SQL injection attack, because it is attempting to “inject” SQL into the database by means of a field on your website.

When the code was entered into the comment box on our website, it was identified as spam and so it was immediately quarantined. While it couldn’t do anything while quarantined it was still in our website database. In fact there were over 200 successful SQL injections that were identified and quarantined as spam but not necessarily as malware. This may seem like a non-issue as all of the attacks were caught and quarantined by our spam filter there are two important takeaways here:

First, while the spam filter caught all of the attempts; it only realized that it was spam not that it was a malware attack.

Second, while it caught over 200 distinct instances of spam if it had missed only one and we had no further security measures in place we could have been compromised.

So, what happened next?

Our firewall kicked in and identified the malicious code but not when it first appeared on the site. At that point it had been quarantined by our spam filter meaning that it was sort of put in code jail. When the database backed up however the firewall immediately scanned the backup file and recognized the malware. Once the firewall knew there was malicious code located on our site it immediately sent out a critical security alert to me.

How I removed the threat from our site.

I scanned the firewall security logs and it took a little investigation for me to figure out what had happened. Once I knew how we had been attacked and what our systems had done to prevent it the remediation process was quite simple:

  1. I manually deleted all the spam removing the malicious code from our database.
  2. I ran a repair script and optimized the database to ensure there were no further errors or performance issues resulting from the attack. The process ran successfully.
  3. I ran a manual backup of our database to create a new clean file to test.
  4. I ran a complete manual scan on the firewall to ensure that all traces of the malware were gone.
  5. I ran manual backups on our 2 additional independent backup services and scanned these files to ensure there were no further traces of the malware.
  6. I added the suspected source IPs for the attack to our firewall backlist. This in effect says that anyone trying to contact our site from these specific computer addresses should be completely denied access to our website.
  7. I did a manual inspection of the site to ensure that everything looked and ran like it should.

Lessons Learned

A security breach is inevitable. It’s going to happen and there is no absolute way to prevent a determined hacker. Most security professionals have abandoned the idea that you can build bulletproof security to stop all attacks. The best defence is the deployment of systems in what is called a “kill chain” defence strategy. This involves having successive systems and processes in place that identify attempts and breaches immediately, take preventative actions to stop or mitigate the damage automatically and then call for help. In the worst case scenario having good backups and a restore plan are a good idea too.

How do we protect the Smashing Pixels website?

This was not a highly sophisticated or focused attack and was not successful because we had prepared our site or “hardened” it for occasions just like this. Here are the basics of what we did to lockdown Smashing Pixels:

  1. For Wordpress and all of our plugins we automate all updates and patches to ensure the site is always up to date.
  2. We have developed a proprietary 29 point security hardening process for Wordpress which includes things like deleting all plugins and themes that are not in use, deleting the default “admin” account, setting the number of unsuccessful login attempts to lock out attackers etc. This list is constantly being reviewed and revised with the latest information and ideas from the security community being adopted and integrated.
  3. We use strong passwords. All passwords on the system must contain completely random strings of letters, numbers and symbols that are computer generated and are so long that it would be almost impossible for even a computer to crack them.
  4. We run an application firewall that monitors all incoming and outgoing traffic and scans all files on the site at regular intervals to ensure that there are no issues.
  5. We have automated response rules built into the system so that the site can take preventative measures on it’s own immediately if it detects a potential attack is in progress.
  6. We have an alert system that sends messages to the system administrators when something happens that we should be aware of or if there is trouble. Most of these alerts are simply FYI type of messages but when there is a critical situation we receive not only the message that something has failed, but also a summary of the preventative actions taken already.
  7. Most importantly we run 3 independent backup systems to ensure that if 1 fails or is compromised there are 2 other clean backups at different locations that we can use to completely restore from.
  8. Finally we train our employees on how best to ensure optimal security. This means not only technical best practices but also how to prevent social engineering which is by far the fastest and easiest way to breach any security system.

Questions to ask your website guy(s)

Whether you host your own website or have a development and design company like us host it for you, here are some questions you should be asking to ensure your online business continues to run uninterrupted:

  1. What security measures do you take to protect our site and systems?
  2. Would you even know about it if we were under attack or had our systems hacked?
  3. What would be the damage to our business if we were hacked? What would be the effect on our customers and our ability to serve them?
  4. If our site was hacked and destroyed, how fast could we recover and get back up and running?
  5. Do we have off site backups in case our regular backup is hacked or destroyed along with the website?
  6. And most importantly… have you tested all of this or are you just guessing at the answers?

In conclusion…

While it’s unfortunate that these things happen if you prepare properly for then inevitable then you will significantly reduce the chances that your site will be hacked. More importantly you’ll know about it if it does happen and have a plan in place to identify and remove the threat and then mitigate the damage.

And of course if you need some help or advice with how best to lock down your website, please give Smashing Pixels a call!