Attacked!

Saturday morning I was in the ideal programming groove.  I got up early, grabbed a coffee from Panera’s, and faced the screen of my main computer, SuperSluggo, with its Ubuntu desktop and a nice wallpaper showing a snowy scene.  I had cleaned the condo (oops, I mean EP Studios Eastern HQ) a few days before.  I really had no other distracting tasks I needed to do.  The weekend, full of promise and seemingly endless at this hour of 8 in the morning, stretched before me.  I had just read that the FDA had approved the anticoagulant apixaban for use in non-valvular atrial fibrillation.  I downloaded the monograph from Pfizer on dosing of the drug and found that it actually had a fairly complex dosing scheme, depending not just on creatinine clearance, but also on a combination of age, weight, and serum creatinine.  It was clear that EP Mobile, my mobile app for electrophysiology that includes several drug dose calculators, needed an apixaban calculator.

This was fairly easy to implement, and before long I had the Android version ready to go.  I clicked on the EP Studios software site (epstudiossoftware.com) to add the apixaban reference to the References page of the site.  And there appeared, instead of the site, a garish red ATTACK SITE — EXTREME DANGER — GET THE #@% OUT OF HERE! page.

My sweet and innocent web site, residing in the innards of my web server computer, TomServo, on the floor of the guest room, had been hacked!  But how?  I had a tough password.  My router only lets stuff through the HTTP port 80 to TomServo.  There must have been some mistake.

I clicked through the GET ME OUT OF HERE warnings to the actual web page.  Looked normal, except for a red bar at the top with the GET ME OUT OF HERE button and a THIS IS NOT A MALICIOUS SITE button.  I clicked through the latter button, got to the Google Web Tools site and read the details.  Since December 27, my web server had been serving malicious pages to people who visited the site.  My site was infected with malware.  My site was blacklisted by Google.  It would no longer show up on web searches.  Anyone clicking on the site would be greeted by the same red warning page I had seen.  This was not good.

I had to fix this and get un-blacklisted.  I read the steps I needed to take.  They were extreme.  They were the sort of thing that would probably take most of the day to accomplish.  So much for my infinite weekend.

I got to work.  I took TomServo offline by unplugging the ethernet cable.  There, take that, hackers!  Next I hooked up a keyboard and monitor and tried to see what was going on.  I ran a utility I had previously installed that checks for rootkits.  A rootkit is a program that takes over your computer and then hides itself so that it is very hard to detect.  An example of how it hides itself would be to substitute its own ls (which lists files and directories) for the original ls program.  The new ls would act exactly like the old, but wouldn’t ever list programs that were part of the rootkit.  Well I ran chkrootkit and, lo and behold, it found that my server was infected with the SUCKIT rootkit.

How to get rid of a rootkit?  Unfortunately the only sure way is to erase the hard drive and reinstall the operating system.  Fortunately my WordPress blog (and the entire site) was backed up by a WordPress service named VaultPress, so my data was secure.  Taking a deep breath, I crossed the Rubicon.  I erased the drive and reinstalled Ubuntu Server 12.10 from a CD.

I went back online.  I then download WordPress and installed it.  There was a bunch of tweaking that needed to be redone, and, unfortunately, I had forgotten a lot of stuff, like how to set up my mail server.  After fumbling around I seemed to have everything working, including a new, fresh, blank WordPress installation.

Now to restore a backup from VaultPress.  I tried to do it automatically.  It didn’t work.  I ended up downloading the files manually and copying them into the proper folders.  Worked!  I had a running version of my web site that was indistinguishable from the old.

I thought I was done.  I went to the Google Web Tools site, declared “This house is clean!” and hoped they would quickly review my site and un-blacklist it.  I went to bed that night hoping the crisis had passed.

The next day I was still getting the red screen of shame when I accessed my site.  I checked the Google Web Tools.  They had rechecked my site, and it was still infected!

I reran the chkrootkit program.  Everything was clean.  Maybe Google was full of it.  Everything was new on the server, and I had changed all the passwords.  Well, not everything was new.

I realized the backup copy of the blog itself might be infected.  I logged into the administration page and searched for some plugins that could detect malware on a WordPress site.  I downloaded one and ran it.  Bingo!  Some files residing in the theme directory of WordPress contained some JavaScript that shouldn’t have been there.  I copied these files over using the fresh version of WordPress I had downloaded the day before.  I reran the malware detector.  Everything was clean!

I appealed again to Google to recheck my site.  Within a few hours, the site was declared clean by Google.  The red page was gone.  I’m not sure how long it will be before my site’s pages will show up on the Google search engine without the malware warnings appended to them.

Breathing a sigh of relief, I went on to finish my upgrading of EP Mobile for both the Android and Apple versions.  I am now much more paranoid about the possibility of exploitation of my web site, and will keep a closer eye on it.  This was not a pleasant experience for me.