Author: George London

Edit: I’ve come to believe that one premise of this article is wrong. You definitely can send email from an EC2 instance; Django will send error emails from a production server. So instead of using SNS, it would probably be easier to trigger some kind of simple email sending Python (or whatever) script when the termination event happens.

I make pretty heavy use of EC2 spot instances, which as you know can terminate at any time with no warning.

In order to get my spots back up ASAP, I’d like be notified when they terminate.

This turned out to be much harder than I expected. I thought I’d be able to add a simple script that would send me an email when the instance shuts down (Amazon is nice enough to send a shutdown command on termination instead of just pulling the plug.)

But that approach has a couple of problems.

First, you can’t easily send email from EC2 instances (because of spammers) and have to manually get instances whitelisted by elastic IP which is a pain.

Second, it’s not the easiest thing to write a shutdown hook in Linux.

So, here’s a solution for both of those problems.

The email problem

Amazon has a service called CloudWatch that seems great for this, except that it can only monitor metrics emitted by running instances. So you can’t set it to alert you on a system shutdown, nor on a metric polling failure because it randomly misses packets all the time.

So the solution is a different Amazon service called SNS (simple notification service) that will let you trigger an event that can be configured to send you an email. So we’re going to write a script that tells SNS to send us an email.

(SNS is free for ~200k requests/month, so unless you’re planning on doing something nuts this approach should have no marginal costs.)

To do that, you first need to set up a “topic” in SNS. Go to the SNS dashboard and

1) click “create topic”.
2) Create a topic called “instance_down” or whatever you like.
3) Click the topic and click “create subscription”
4) Choose protocol “email” and enter your email address as the endpoint

To use SNS in a script, we need the AWS command line tools.

If you already have pip installed, just

pip install awscli

Then:

touch ~/.awsconfig
emacs ~/.awsconfig

Make it say:

[default]
AWS_ACCESS_KEY_ID=<< YOUR AWS ID>>
AWS_SECRET_ACCESS_KEY=<< YOUR AWS SECRET KEY>>
region=<< YOUR REGION >>

Create an init.d script:

sudo emacs /etc/init.d/ec2-shutdown

Make it say:

#! /bin/sh
### BEGIN INIT INFO
# Provides: ec2-terminate
# Required-Start: $network $syslog
# Required-Stop:
# Default-Start:
# Default-Stop:
# Short-Description: restart
# Description: send termination email
### END INIT INFO
#

export AWS_CONFIG_FILE={{ YOUR CONFIG FILE }}
export AWS_DEFAULT_REGION={{ YOUR REGION }} # config file not picking up region for some reason
sudo -E aws sns publish –topic-arn {{ YOUR SNS ARN }} –message “ec2 ser
ver {{ YOUR IDENTIFIER }} went down at $(date)”
sleep 3 # make sure the message has time to send

exit 0

That script will tell SNS to send you a email saying your server is down and giving the time. You can customize the message however you like.

The Shutdown Script Problem

So how do we make this run on shutdown?

We’re going to use init.d scripts. It’s taken me a little while to get my head around how this works, but in a nutshell…

Linux has a concept of “runstates”, which include things like “shutdown” and “logged in”. You can tell Ubuntu to run shell scripts when it changes into a runstate by placing scripts in certain folders in side of /etc. The two states that concern us are rc0 and rc6, i.e. “shutdown” and “reboot”, which correspond to folders /etc/rc0.d and /etc/rc6.d respectively.

Ubuntu has a command line tool, update-rc.d that will automatically symlink scripts into the appropriate folders depending on the CL paramaters you pass it. It’s all a bit complicated, but all you need to do here is:

sudo update-rc.d ec2-shutdown start 10 0 6 .

(This says run the script on entering run states 0 and 6, shutdown and reboot. And put it 10th in the list of things to do)

And that should do it! Now your instance should send you an email whenever it shuts down, terminates or reboots. I’m not sure how limit to just EC2 termination, but please comment if you know!

If you have any problems, leave a comment.

And if you find this useful, consider following me on Twitter.

Back linking has gotten unwieldy, so I’m going to just update links on this page to all the parts from now on:

Intro: The future of the economy, in bite-sized chunks.

Part 1: What is An Economy?

Part 2(a): Rich Hypothetical Society, Poor Hypothetical Society

Part 2(b): Productivity is Prosperity

Part 3: How Productivity Grows

Part 4: Computers and Productivity

Part 5: Lateral Productivity Growth

A couple of weeks ago, I left a comment on Albert Wenger’s excellent blog “Continuations” about Free Will. The comment was rather long, so I reposed it here on my blog. Somewhat amazingly, Albert read my comment and left a thoughtful reply.

I’m not sure of the etiquette of reposting Albert’s comment, but to summarize he said that my argument was off-base because the “information layer” I described is just an abstraction and doesn’t interact with the physical world. No interaction means no influence, and that means that the stimulus->thought->action sequence remains a deterministically closed loop.

I, logorrheic that I am, responded with another rather long comment. This is that comment.

Well, actually this is:

If it’s true that the “information layer is entirely abstract” (i.e. completely non-interacting with the tangible world) then your whole argument is clearly right.

But I’m not sure that premise is true. It seems that the informational world is constantly interacting with the physical world in the form of “physical laws being followed.”

Metaphorically, particles in the world are “data” and the laws of physics (i.e. the information layer) are “source code.” Every particle interaction is (metaphorically) a computation. Now all of our observation to date suggests that the “source code” of the universe is stable and fixed. But any honest physicist will tell you that our (extremely elegant) mathematically descriptions of how objects have behaved in the past tell us nothing about why they behaved that way.

I raised GEB to show that “laws” aren’t necessarily well-behaved in edge cases.[1] And so similarly, the fact that we’ve observed that some parts of the “Ur-program” work one way does NOT demonstrate conclusively that there aren’t other parts of the program that introspectively alter the Ur-program’s source code and lead to measurable, experimentally verifiable physical effects.

You raise one potential “point of strangeness” in your PS – quantum waveform collapse. My knowledge of quantum mechanics and neuroanatomy is limited, but my best understanding is that neural interactions are sensitive enough that quantum effects can make the difference between whether a neuron fires or not. And we know from computers (cf. my Caesar’s cipher example above) that changing one bit in a computation chain can radically alter the downstream result, so any slight (non-random!) bias in the quantum coin-flip could be enough to completely alter behavior. (And as far as I know would not violate the conservation of energy or any other physical laws except our observation that quantum collapse is usually random.)

Could patterns of thought directly alter the wave-form collapse? Well, we already know that the quantum world is altered by observation per se. And conscious introspection is fundamentally self-observation. So it is possible that when the wetware finds itself in physical states that mirror certain “information patterns” (e.g. “contemplating two choices”), the presence of those patterns triggers an alteration in the source code that directly influences the way quantum collapse happens in connected synapses, leading a neuron to fire that otherwise wouldn’t. And from the GEB problem, we might expect the “meta program” to be especially poorly behaved (perhaps even fundamentally indeterminate, i.e. “free”) in this circumstance.

So is this explanation likely? That’s hard to say. It’s neither confirmed nor contradicted by existing data. It’s counter-intuitive, but so is every other part of quantum mechanics.

But best of all, this theory is testable. We just need to carefully measure the statistical in vivo quantum behavior of synapses during decision-making and I can be proven wrong once and for all. 🙂

[1] I admit I’m not sufficiently demonstrating that “logic laws” work the same ways as physical laws but there is certainly a remarkable symmetry. It doesn’t seem at all fundamentally necessary that physical laws should be mathematically expressible and elegant. I’m happy to write more on this point if anyone is curious.

[Once again, I’ve been inspired by a post on Albert Wenger’s blog, this time about whether or not free will exists.]

I’ve spent a long time thinking about this one as well. I think you’re probably right, but here’s the best alternative I’ve been able to come up with. These ideas are heavily inspired by having finally finished Gödel, Escher, Bach.

Basically, it’s clearly true that determinism holds over the space of physical objects if you assume that a continuous set of physical laws governs all physical interactions. If my brain is just a physical object, then the arrangement of its atoms at present is exclusively a function of their arrangement in the past.

But how justified are we in making that assumption about the continuity and coverage of physical laws?

Let’s start with coverage: it’s actually not hard to think of something that doesn’t interact with the physical world in quite the same way as an apple – a computer program.

Yes, a program is represented by marks on a disk, and yes it is executed by a set of magnets that write and erase those marks using electricity routed through a CPU. But ultimately those purely physical components produce an informational state projected through my screen. And as I write I am interacting not only with the keys of my keyboard but also but with the abstract state of the system. And the state per se has causal power: perform a Caesar cipher on your blog post and it becomes virtually meaningless to me despite having extremely similar statistical properties and on-disk representation.

So then where does the meaningfulness of the program state reside? Can we point to a memory address that distinguishes “meaningful” from not meaningful (even in principle)? Can we point to an atom in my brain that makes the text meaningful to me, even though it would remain meaningful to you even if my brain we destroyed (and vice versa) but become meaningless if all brains were destroyed?

The point here is that even in closed physical systems, there is an information “layer” that has both its own rules (i.e. “logic”) and yet tangibly influences the physical world. This is the layer where the program-ness of a program resides, or the law-ness of physical laws, or the meaningful-ness of a sentence.

Normally this layer is well behaved – I encode my experiences into words that you understand; when fundamental particles interact they “check” which laws to follow and then they follow them.

But the layer has known flaws, specifically when self-reference enters the picture. Gödel’s incompleteness theorem establishes that there are questions which can be precisely formulated but not decided, e.g whether the statement “this statement is a lie” is true or not (or the somewhat related question of whether an arbitrary computer program will ever terminate.) These questions aren’t undecidable because of a deficit in our ability to measure (a la Heisenberg uncertainty) but rather because of an irresolvable quirk in the laws of the information layer.

The interesting thing about consciousness (the domain within which free will would presumably operate) is that it is deeply and inherently self-referential. The very exercise of writing this comment is an example of an information state reflecting on its own statefulness.

Now let’s circle back to the question of consistency in physical laws. Usually it seems like the surest bet: no one has ever been fired for predicting that an apple will obey gravity.

But when we start talking about the way that particles in our brain interact with the information layer (in ways we understand only loosely, at best) and we focus in on an area (self-reference) where the information layer is already know to misbehave, how confident can we be that the rules we’ve observed elsewhere operate the same way here?

Or more colorfully, how certain can we free will isn’t some sort of buffer overflow in the operating system of the universe?

[Edit: if you like this post, please upvote on Hacker News! Look for the post titled “Is Free Will a Buffer Overflow?”]

Things in Austin got weird. There was carousing, there was trespassing,  and yes, there was even laser tag. But after seven crazy days in Austin, my SXSW adventure is over. Instead of sharing a small Austin house with ten people (including two different bands), I’m now just sharing a house with my parents as I visit the Bay Area for some meetings.

SXSW was definitely fun, but it was also probably a waste of time.[1] I probably could have seen that coming – one of the clearer lessons of my time working on LinerNotes is to be very skeptical of conferences. I’ve been to several of the bigger ones (mostly because I managed to snag free tickets), and none of them have delivered enough value to clearly justify the time it took to attend, much less the the jaw-dropping ticket prices (two thousand dollars for TC Disrupt? Seriously?)

Still, out of all the hundreds of tech conferences SXSW retains a somewhat mystical reputation for “serendipity,” i.e. that simply by walking around and talking to people one might encounter the investor or business partner one needs to “kick it up a notch.” That’s a tempting offer, since I do believe in serendipity (or rather the darker flip-side that many endeavors will be destroyed by the absence of serendipity.)

My skepticism almost made me follow Danielle Morrill’s advice and skip SXSW, but a few things fell in place at the last minute (i.e. I found a place to stay for free) and so I decided to take the plunge.

And…I was disappointed.

Serendipity did not strike. I’m sure that it did for some people, but I didn’t meet even one new person with whom I plan to stay in close contact. To be fair, I didn’t do much to maximize my luck surface – I only went for the last two days, I didn’t buy a conference badge ($800?!) and I only struck up conversations with 20% of the people I randomly stood next to in lines instead of 100%. Still, the conversations I did have didn’t give me the sense that my digital soul mate was just one forced conversation away.

The basic problem is with the odds. When SXSWi was smaller, it may have been viable to talk to a large percentage of attendees and either meet someone useful or meet someone who could introduce you to someone useful they just met. But now there are 30,000 people at SXSWi. Let’s assume that’s 1.5% of the entire startup world, and let’s assume that the population is representative of the general startup population.  [2]

What are my (very rough) odds of meeting someone useful? Well, I’d estimate that there are probably less than 500 viable co-founders and 500 viable investors for me in the whole world, and LinerNotes is not really at a point where I’d benefit much from any business development relationships or marketing hires. If 1.5% of those people are at SXSW, that’s approximately 30 people at SXSW worth meeting.

If I’m there for two days, I probably come into physical proximity with 300 people in a day or 600 in total. That’s 4% of the conference. 4% of 30 is 1.2, meaning that in the whole two days, I was probably near someone useful about twice.

Now how many conversations with strangers can I reasonably have in a day? If I’m really forcing it, maybe 20. So that’s (approximately) 40 dice rolls with a 599/600 odds per roll of not meeting someone useful. That’s an 93% chance of failure, which implies a 7% chance of meeting someone useful, if I’m aggressively social and if the population is really representative (which it probably isn’t).

So after all that expense, travel, and foregone working time, the odds of serendipity striking are about 1 in 15. Compared to everything else I could do with those resources, going to SXSW for networking is pretty clearly not worth it.

So why does SXSW continue to have a reputation for magical serendipity?

Simple, because only the 1 in 15 people who benefit bother to write about it or tell their friends. No one likes a party pooper and so other than me and Danielle Morrill, few people want to loudly admit to wasting their time in Austin.

To be clear, I don’t regret going. The main purpose of my trip was actually to go to SXSW Music and do market research for LinerNotes, and it was worth sacrificing a weekend of work to have some fun and confirm my suspicions about SXSW. And I did meet up with a few old friends at SXSWi and got to strengthen those relationships.

There are some people for whom SXSW still does make sense (specifically startups that sell to marketers or app makers, or VC’s and later stage startup employees who just want to relax and party.) But now I can say from experience that there are better ways to make new friends than shouting at strangers at a loud, crowded Taco Bell sponsored party.

 [1]  This post only applies to SXSW Interactive. The dynamics of SXSW Music were much different and probably need their own blog post.

[2] This is obviously a rough calculation. These numbers are mostly made up with the goal of being ballpark correct and having errors cancel out. And even very generous assumptions don’t change the overall picture much.