{"id":41,"date":"2019-09-17T03:58:13","date_gmt":"2019-09-17T03:58:13","guid":{"rendered":"http:\/\/tensigh.de\/?p=41"},"modified":"2020-08-27T16:18:41","modified_gmt":"2020-08-27T16:18:41","slug":"server-down-panic","status":"publish","type":"post","link":"https:\/\/japanobserver.net\/index.php\/2019\/09\/17\/server-down-panic\/","title":{"rendered":"Server down – PANIC!!!"},"content":{"rendered":"

I thought I would look at my website since I view it so seldom, and lo and behold, IT’S DOWN!!!<\/p>\n

\"PANIC!\"<\/p>\n

First I tried restarting the AWS EC2 instance – NO SITE!<\/p>\n

\"PANIC!\"<\/p>\n

Next, I tried to SSH in, but the IP address has changed since i rebooted.<\/p>\n

\"forehead<\/p>\n

Got the new IP, started SSH, got in. httpd and mysqld aren’t running (Dufuq?)<\/p>\n

Tried restarting them\u00a0 but couldn’t!!!\u00a0 Have I been hacked?<\/p>\n

After I calmed down, I saw that I needed to sudo.<\/p>\n

\"Silly<\/p>\n

Once I did that, I was able to restart httpd and mysqld<\/p>\n

\"Phew!\"<\/p>\n

But now I can’t see the site in my browser. Clearly this is a Route 53 \/ DNS issue.<\/p>\n

Updated DNS entry with new, correct IP address. Now comes the waiting game, as pinging the domain name reveals the old IP address:<\/p>\n

\"First<\/p>\n

(My OS is in Japanese. Trust me, the pings were timing out).<\/p>\n

After a few minutes, the IP address is correct and the site comes up:<\/p>\n

\"Site's<\/p>\n

Phew! It’s nice that the right IP address shows up, but I can’t test because it’s not responding to pings:<\/p>\n

\"Ping<\/p>\n

I decided to add a security policy so I can ping just for testing.<\/p>\n

Instead of adding it to the main policy, I’m going to do the “Photoshop layer” thing and create a new policy that I can add\/remove at will and apply it to the same EC2 instance.<\/p>\n

I created a Echo reply rule and allowed it from anywhere (this is for testing).<\/p>\n

\"Custom<\/p>\n

Okay, it’s set for both incoming and outgoing. Great! Let’s test it:<\/p>\n

\"Ping<\/p>\n

Hmmm. Failure.\u00a0 Oh,\u00a0 snap! I need to set the instance itself to allow pings. Duh!<\/p>\n

I had to look it up, but I found the iptables command that would work best. I normally wouldn’t set this to respond to pings from everywhere but this is for testing.<\/p>\n

\"iptables<\/p>\n

Okay, the security policy is in place, the DNS entry is correct and the iptables\/local firewall is set. Let’s test again:<\/p>\n

\"Ping<\/p>\n

Success!! I can rest easy now. And since the ping policy is separate from the main security policy, I can turn it off\/on at will, or delete it outright.<\/p>\n

I love AWS!!<\/p>\n","protected":false},"excerpt":{"rendered":"

I thought I would look at my website since I view it so seldom, and lo and behold, IT’S DOWN!!! First I tried restarting the AWS EC2 instance – NO SITE! Next, I tried to SSH in, but the IP address has changed since i rebooted. Got the new IP, started SSH, got in. httpd … <\/p>\n

Continue reading “Server down – PANIC!!!”<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[4,10,16],"tags":[17,13],"class_list":["post-41","post","type-post","status-publish","format-standard","hentry","category-aws","category-system-administration","category-system-down-emergency","tag-aws","tag-system-administration"],"_links":{"self":[{"href":"https:\/\/japanobserver.net\/index.php\/wp-json\/wp\/v2\/posts\/41","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/japanobserver.net\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/japanobserver.net\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/japanobserver.net\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/japanobserver.net\/index.php\/wp-json\/wp\/v2\/comments?post=41"}],"version-history":[{"count":11,"href":"https:\/\/japanobserver.net\/index.php\/wp-json\/wp\/v2\/posts\/41\/revisions"}],"predecessor-version":[{"id":85,"href":"https:\/\/japanobserver.net\/index.php\/wp-json\/wp\/v2\/posts\/41\/revisions\/85"}],"wp:attachment":[{"href":"https:\/\/japanobserver.net\/index.php\/wp-json\/wp\/v2\/media?parent=41"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/japanobserver.net\/index.php\/wp-json\/wp\/v2\/categories?post=41"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/japanobserver.net\/index.php\/wp-json\/wp\/v2\/tags?post=41"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}