Frequent visitors to terragalleria.com may have noticed unusual,
repeated periods of downtime over the past month, as well as a gap in updates. The quick
explanation is that the server hosting the site was hacked. Sure, this has
happened several times in the past, but each time I had been able to
figure out which vulnerability the hackers had used, which (criminal)
activities they had planned, and shut them out. Not this time, which
opened to door to multiple, escalating attacks which eventually forced me
to decommission the server and rebuild a new one. To get a sense of the
aggravation I faced, read on the long explanation.
Let’s begin by addressing the most obvious concern.
There isn’t much to steal on the server. For safety reasons, we do not
store credit card numbers, simply passing them to our gateway.
High-resolution digital files are valuable, by only if you can license them, a task which is nowadays quite difficult even for legitimate image creators. To the hacker’s credit,
they did not vandalize the site contents. They seemed to be interested in
the server only as a platform to try and compromise other systems and get
access to them.
It started about a month ago. A reader reported to me that his
anti-virus software had flagged the terragalleria.com website. An
investigation based on time-stamps (to look at recently modified
files) quickly showed that in one of the files containing Javascript
utilities, the following code was appended (I’ve included it as an image because as text it is harmful !):
If this sounds like Hebrew to you, rest assured that it was the same for me
until someone on G+ pointed me to the following page, which explains everything. The
code was obfuscated to make it difficult to understand what it does:
it creates a iframe – a special invisible frame around the webpage – which points
to a website that tries to infect visitors with malware, which I can only assume (I didn’t download it, and nor should you ever download software with no clear purpose) allows them to take control of the victim’s computer.
Although getting rid of this particular code was easy, finding out how the hackers were able to plant it remained an unsolved challenge, as there are so many different security flaws which can be exploited to gain control over a computer. Even systems from the US Department of Defense are not immune to hacking. Since I didn’t know the extent to which the server was compromised, the safest thing to do was to
rebuild the site. However, I was going to travel the following week, so
I didn’t have time to do much. I also hoped that the intrusion was limited – as had been the case in the past. I left for Alaska.
In search of the Aurora, we drove north of Fairbanks on the Dalton
Highway, passing the Arctic Circle. Although there were absolutely no
services of any kind on the 240 mile stretch of road until Coldfoot -making it the most remote in North America – the
nearby village of Wiseman (pop 13), has two B&Bs open
year-round. Because our rental car was marginally equipped to deal
with -30F weather, instead of camping further north, we stayed
there. Unexpectedly, they provided for a few hours per day
satellite-based internet.
That’s when I learned that my server woes were getting worse. The
hackers were trying to use the compromised server to hack into other
machines on the network, using what is called a “brute force attack”
which consists of automatically trying to log on by trying thousands
of different passwords (the reason you shouldn’t use simple passwords). As a result, the hosting company took the
server offline after I had failed to respond to their warnings. Back
in Fairbanks, we huddled in the public library to benefit from a reliable
internet connection. At first we found themselves locked out from the
system, but after assistance from the data center technicians, we had
the main (root) password reset so we could do administrative tasks. My friend
then found one of the backdoors they left and removed it.
All was (almost) quiet for a few days, but the following week they
tried again to do brute force attacks against other computers. I was
not able to locate the script nor the security hole, but at least I
was able to stop the activity by disabling some system utilities.
A few days later, I discovered that they were back to trying to
propagate malware using an iframe. But this time, they used more
sophisticated methods. They had covered more carefully their tracks,
since a time-stamp based search didn’t yield anything
interesting. Instead of just modifying a single text file, they
managed to get the webserver to append a malicious iframe for all web
pages and all the domains hosted on the server. Protecting our
visitors is the priority, so upon discovering that the server was
trying to spread malware, I immediately shut down all web services.
I looked all day long at various configuration files, yet remained
stumped. Because I hold a PhD in Computer Science, it is sometimes
assumed that I am a wizard with computers. But in fact, I was
mostly a mathematician studying the geometry of visual perception of
three-dimensional space. I learned very little about system
administration in grad school – there was no world wide web yet.
I decided to simply abandon the server. This was the right decision, as
in the following week, the situation degrated further. Not content
with changing all the server passwords, they reconfigured the server so
that even after reseting the root password it was not possible for me nor
the data center technicians to log on. In the while, I received complaints
from system administrators of large, well-known companies, into which they
tried to hack using the server. I eventually put an end to this activity by wiping out
all server contents with an OS reload.
Upon returning home from Alaska, I had already begun to plan for a
new replacement server – kind of like burning your house to the ground and
rebuilding it to get rid of vermin. I was looking at more complex solutions involving virtualization and cloud computing.
However the server was now down, and every such day costs us quite a bit in lost business, so I decided
to rebuid again a simple and cost-effective dedicated server. I rent a box running Linux on
a brand new Quad Core Xeon 1270 – 3.40GHz (Sandy Bridge), 1 x 8MB cache w/HT, 2 GB DDR3 1333 RAM, 2x 750GB SATA II HDD with a 100 Mbps uplink port and a 3TB monthly data transfer allowance.
Besides the hardware upgrade – which greatly accelerate searches and dynamic loads – the main difference is that unlike on my
previous servers, decided to forego a control panel (I used
Plesk before), mostly for security reasons. The data center technicians had pointed out
to specific Plesk-related vulnerabilities. A control panel provides an easy-to-use
graphical interface to all services on your server. Without one, you
administer your server by editing text files and typing commands.
I found out during installation that while a control panel makes
configuration simpler and more foolproof (no messing all your web
services because of a single missing directive or a mistyped command),
its main benefit is that it provides web-ready software. In my case,
all the installation went smoothly until I noticed that PHP (an active component of web pages) would not
communicate with MySQL (the database). Since they are such basic and popular building
blocks, you’d think that they would work right out of the box in a
distribution such as Red Hat Enterprise Linux (RHEL) 5, but this wasn’t the case at all.
After trying many things, I re-installed PHP from source, but in the process
this happened to break httpd. At this point I got fed up with fiddling
around without real understanding. I hired the services of a server
administration company.
They fixed the problem promptly, and I was happy to get the site up again, first in an emergency instance without many functions. However, as
I began to set-up my backup software, I found out that the hard drives were not
configured properly. I had ordered the server with two hard drives, with the intention to use
the second one for backup. However, the hosting company set up the drives as an LVM (Logical Volume Manager) volume
group, which defeated the purpose. They recommended that I reload the OS and restart from scratch, although
there exists a risky procedure to remove a drive from the LVM. As it had taken me more than a day to get to that point,
I wasn’t too keen on doing it over again, so I asked my system administrator to try the procedure. Sure enough, it failed, leaving the server in a state where it wouldn’t even boot.
It quickly became clear that I had no choice but to reload the OS. I thought that I might as well
upgrade to version 6 of RHEL as older versions sometimes lack critical security updates. It turned out to be even worse than RHEL 5. PHP didn’t even work properly. After trying to fix it without success, the system administrator recommended that
instead I reload CentOS 5. I was surprised, since CentOS is the free – and supposedly functionally identical – version of the licensed RHEL, but indeed PHP and MySQL worked out of the box. This doesn’t mean that all my scripts worked, since software upgrades forced me to update my scripts, but at least the basics were there.
Acting as system administrator for two weeks was more than I cared for, but thanks to backups I didn’t loose any data. I just wished hackers would be more considerate in choosing their targets. Many companies who run websites with less traffic than terragalleria.com have full-time system administrators, but this is essentially a one-person operation. I have found that one of the main difficulties – and sometimes source of frustration – in running a small photography business is that one has to wear so many hats. Over the years, terragalleria.com has grown to become, amongst other things, a fully-featured stock photography website and digital wallpaper subscription site. That it needed to run on its own server, which was at the root of last month’s troubles, is something I suppose I should be grateful for, as it had provided me my own path to full-time photography.