Edit: I’ve come to believe that one premise of this article is wrong. You definitely can send email from an EC2 instance; Django will send error emails from a production server. So instead of using SNS, it would probably be easier to trigger some kind of simple email sending Python (or whatever) script when the termination event happens.
I make pretty heavy use of EC2 spot instances, which as you know can terminate at any time with no warning.
In order to get my spots back up ASAP, I’d like be notified when they terminate.
This turned out to be much harder than I expected. I thought I’d be able to add a simple script that would send me an email when the instance shuts down (Amazon is nice enough to send a shutdown command on termination instead of just pulling the plug.)
But that approach has a couple of problems.
First, you can’t easily send email from EC2 instances (because of spammers) and have to manually get instances whitelisted by elastic IP which is a pain.
Second, it’s not the easiest thing to write a shutdown hook in Linux.
So, here’s a solution for both of those problems.
The email problem
Amazon has a service called CloudWatch that seems great for this, except that it can only monitor metrics emitted by running instances. So you can’t set it to alert you on a system shutdown, nor on a metric polling failure because it randomly misses packets all the time.
So the solution is a different Amazon service called SNS (simple notification service) that will let you trigger an event that can be configured to send you an email. So we’re going to write a script that tells SNS to send us an email.
(SNS is free for ~200k requests/month, so unless you’re planning on doing something nuts this approach should have no marginal costs.)
To do that, you first need to set up a “topic” in SNS. Go to the SNS dashboard and
1) click “create topic”.
2) Create a topic called “instance_down” or whatever you like.
3) Click the topic and click “create subscription”
4) Choose protocol “email” and enter your email address as the endpoint
To use SNS in a script, we need the AWS command line tools.
If you already have pip installed, just
pip install awscli
Then:
touch ~/.awsconfig
emacs ~/.awsconfig
Make it say:
[default]
AWS_ACCESS_KEY_ID=<< YOUR AWS ID>>
AWS_SECRET_ACCESS_KEY=<< YOUR AWS SECRET KEY>>
region=<< YOUR REGION >>
Create an init.d script:
sudo emacs /etc/init.d/ec2-shutdown
Make it say:
#! /bin/sh
### BEGIN INIT INFO
# Provides: ec2-terminate
# Required-Start: $network $syslog
# Required-Stop:
# Default-Start:
# Default-Stop:
# Short-Description: restart
# Description: send termination email
### END INIT INFO
#export AWS_CONFIG_FILE={{ YOUR CONFIG FILE }}
export AWS_DEFAULT_REGION={{ YOUR REGION }} # config file not picking up region for some reason
sudo -E aws sns publish –topic-arn {{ YOUR SNS ARN }} –message “ec2 ser
ver {{ YOUR IDENTIFIER }} went down at $(date)”
sleep 3 # make sure the message has time to sendexit 0
That script will tell SNS to send you a email saying your server is down and giving the time. You can customize the message however you like.
The Shutdown Script Problem
So how do we make this run on shutdown?
We’re going to use init.d scripts. It’s taken me a little while to get my head around how this works, but in a nutshell…
Linux has a concept of “runstates”, which include things like “shutdown” and “logged in”. You can tell Ubuntu to run shell scripts when it changes into a runstate by placing scripts in certain folders in side of /etc. The two states that concern us are rc0 and rc6, i.e. “shutdown” and “reboot”, which correspond to folders /etc/rc0.d and /etc/rc6.d respectively.
Ubuntu has a command line tool, update-rc.d that will automatically symlink scripts into the appropriate folders depending on the CL paramaters you pass it. It’s all a bit complicated, but all you need to do here is:
sudo update-rc.d ec2-shutdown start 10 0 6 .
(This says run the script on entering run states 0 and 6, shutdown and reboot. And put it 10th in the list of things to do)
And that should do it! Now your instance should send you an email whenever it shuts down, terminates or reboots. I’m not sure how limit to just EC2 termination, but please comment if you know!
If you have any problems, leave a comment.
And if you find this useful, consider following me on Twitter.