Server down – PANIC!!!

I thought I would look at my website since I view it so seldom, and lo and behold, IT’S DOWN!!!

PANIC!

First I tried restarting the AWS EC2 instance – NO SITE!

PANIC!

Next, I tried to SSH in, but the IP address has changed since i rebooted.

forehead slap

Got the new IP, started SSH, got in. httpd and mysqld aren’t running (Dufuq?)

Tried restarting them  but couldn’t!!!  Have I been hacked?

After I calmed down, I saw that I needed to sudo.

Silly me!

Once I did that, I was able to restart httpd and mysqld

Phew!

But now I can’t see the site in my browser. Clearly this is a Route 53 / DNS issue.

Updated DNS entry with new, correct IP address. Now comes the waiting game, as pinging the domain name reveals the old IP address:

First ping test failed

(My OS is in Japanese. Trust me, the pings were timing out).

After a few minutes, the IP address is correct and the site comes up:

Site's up!

Phew! It’s nice that the right IP address shows up, but I can’t test because it’s not responding to pings:

Ping test 2 success, but no reply

I decided to add a security policy so I can ping just for testing.

Instead of adding it to the main policy, I’m going to do the “Photoshop layer” thing and create a new policy that I can add/remove at will and apply it to the same EC2 instance.

I created a Echo reply rule and allowed it from anywhere (this is for testing).

Custom rules for ping

Okay, it’s set for both incoming and outgoing. Great! Let’s test it:

Ping Still failing

Hmmm. Failure.  Oh,  snap! I need to set the instance itself to allow pings. Duh!

I had to look it up, but I found the iptables command that would work best. I normally wouldn’t set this to respond to pings from everywhere but this is for testing.

iptables to the rescue!

Okay, the security policy is in place, the DNS entry is correct and the iptables/local firewall is set. Let’s test again:

Ping success!!

Success!! I can rest easy now. And since the ping policy is separate from the main security policy, I can turn it off/on at will, or delete it outright.

I love AWS!!