'The Martian' and Software Development

Charlie Wood
22 April 2014

I’ve been coding the server side of Numerous for the last 165 days. I know that because I keep a work journal in Evernote—sort of a running commentary of the thoughts going through my head. The first Numerous-related journal entry is dated November 8, 2013, which is 165 days ago.

Why do I do this? Simple: because this stuff is too complicated for me to keep in my head all at once. That, plus it’s really helpful to be able to go back and answer the inevitable question, “What was I thinking?!”

It’s not meant to be a story. It’s a journal. But it describes in fairly minute detail the problem-solving process that lies at the heart of software development. Here’s a representative example:

The 1MB problem in scanMetricSubscriptions
2014-04-21 08:11:10

OK it’s really 2 problems:

1. This should be a query, not a scan
2. It needs to handle paging so it can deal with more than 1MB of matches

The second problem is the pressing one, since until I fix it, we’re in danger of not delivering notifications (once the amount of subscription data scanned by Amazon exceeds 1MB, which is probably something like 5,000 subscriptions, so call it 500 users). After that, notifications won’t get sent to some random set of people, which would be a bad thing.

The first problem is an efficiency thing. I’ll fix it while I fix #2.

I can model this on how I handle paging for Events and Interactions. That means the (new) getMetricSubscriptions function needs to accept values that will make up an ExclusiveStartKey for the subscriptions table. Which brings up another important point.

Right now the subscriptions table has a single index: userId as the primary hash key and metricId as the primary range key. This is great for the getting a single subscription (the one for a certain userId and a certain metricId), and for the query “give me all subscriptions for this user”. But it doesn’t help with the query “give me all subscriptions for this metric." What it needs is a global secondary index with a primary hash key of metricId.

And yes, that means rebuilding this table from scratch. Right now there are 517 items in it. Hmm. Looks like I’m going to be writing some code.

OK, so in order, here’s what I need to do:

1. Write the new getMetricSubscriptions code (/v2/!), including paging.
   DONE!
2. Create a temporary table (subscriptions-2) with the existing structure plus the new GSI.
   table name: subscriptions-2
   primary hash key: userId (N)
   primary range key: metricId (N)
   secondary hash key: metricId (N)
   secondary range key: userId (N)
   provisioned reads: 6
   provisioned writes: 1
   DONE!
3. Test the new code against the new table.
   DONE!
   
4. Write and run a utility to copy each item from subscriptions to subscriptions-2.
   DONE!
   
5. Update subscriptions-db.go with the new table name. (And anywhere else it’s referenced.)
   DONE!
   5.1. Push to production
   DONE!
   
6. Update the functions in notifications.go to handle multiple pages of results.
   DONE!
   
7. Update deleteSubscriptionsForMetric() to use the query and to handle paging
   DONE!
   
8. Update deleteSubscriptionsForUser() to use the query and to handle paging
   DONE!
   
9. Update all other instances of scanUserSubscriptions() and scanMetricSubscriptions()
   DONE!
   9.1. Push to production
   DONE!
   

Whew.

When I read back over my journal I see the same pattern repeated again and again: identify a problem, consider it in the whole, break it down into its constituent parts, figure out the solution, and implement it. Sometimes the pattern is recursive as I find sub-problems. Sometimes I backtrack when I reach a dead end. But often it turns into a list of “Do this… DONE!”, “Do this… DONE!”. It’s satisfying.

There’s an amazing variety of problems you encounter when developing any non-trivial software. (And often stuff that seems like it’s trivial isn’t.) Broaden your scope from simply writing software to launching a whole product, or a whole company, and the variety gets even more amazing. Which is what makes it fun to do what I do.

You’re probably wondering when I’m going to get around to tying this into The Martian, a most awesome book I recently read. Well, that book describes exactly the process I’m talking about. But instead of developing software, it’s about surviving alone on Mars.

You should buy it and read it. You might find it completely boring. You might find it completely compelling. But either way, you and I won’t be at a loss for something to talk about if we ever run into each other. Which, like failing to send notifications to a random set of users, would be a bad thing.