Sam Merritt

torgomatic

TempURL Tricks: Write-Only Access for Backups

Imagine that you have a small fleet of machines sitting in the cloud somewhere serving up a small herd of webapps. Of course, you need to make sure your users’ data is durable even in the event of failures. For a lot of cases, it makes sense to store data directly in Swift and let Swift take care of durability, but you can’t do that with everything. In particular, you can’t do that with RDBMS data, so you have to keep backups.

Now, the naïve way of doing this is to distribute your Swift credentials to every machine, then have a cron job that calls pg_dumpall and sends the result to Swift. This works okay, but has some problems.

First, you’re distributing your Swift credentials to every machine. If an attacker breaks in to any one of your machines, they have full access to all the data in your object store. They can download all your database dumps and sift them for credit card info, they can replace your users’ uploaded videos with porn, or they can be destructive and delete everything.

Second, credential rotation becomes a real pain in the butt. When you rotate your passwords and API keys, you now have to distribute the new Swift credentials to every single box you have. If you miss one, backups just stop working. If that happens, you’d better hope your cron emails are going somewhere useful or else you may never find out.

Fortunately, there’s a way to solve both those problems. The basic idea is to take the Swift credentials off all the machines that don’t need them, and instead have one service that’s responsible for handing out signed PUT tempurls.

Let’s take a look at a simple little webapp that hands back signed URLs. Now, of course you can write this in whatever language and framework you like. I like Python and Flask, so here’s what you get.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
from flask import Flask, request, abort
import hashlib
import hmac
import time


def get_tempurl_secret():
    # You'll probably want to do something smart here
    return "correcthorsebatterystaple"


def get_swift_account():
    # You'll probably want to do something smart here
    return "AUTH_test"


def is_authorized(request):
    # You'll probably want to do something smart here
    return True


def sign(method, expires, container_name, object_name):
    url = "/v1/{acc}/{con}/{obj}".format(
        acc=get_swift_account(),
        con=container_name,
        obj=object_name)
    sig = hmac.new(
        get_tempurl_secret(),
        "\n".join([method, str(expires), url]),
        hashlib.sha1).hexdigest()
    return url + "?temp_url=sig={sig}&temp_url_expires={exp}".format(
        sig=sig,
        exp=expires)


app = Flask(__name__)


@app.route("/")
def sign_urls():
    if not is_authorized(request):
        abort(403)

    segments = int(request.args.get('segments', '1'))
    duration = int(request.args.get('duration', '3600'))
    hostname = request.args['hostname']
    container = "backups_" + hostname
    now = time.time()
    base_object = str(now)
    expires = now + duration
    urls = [sign('PUT', duration, container, "%s_%d" % (base_object, i))
            for i in xrange(segments)]
    return "\n".join(urls)


if __name__ == '__main__':
    app.run()

The app is also on Github with a proper license, a README, and all that jazz. You can fork it and add in some actual authorization as well as a secret that isn’t “correcthorsebatterystaple”. :)

Now, let’s look at a simple shell script that uses this to perform a backup:

1
2
3
4
5
6
7
#!/bin/bash
HOSTNAME="`hostname -f`"
UPLOAD_PATH="`curl https://signer.example.com/?host=$HOSTNAME`"
UPLOAD_URL="https://swift.example.com$UPLOAD_PATH"

# do_database_backup is shorthand for "dump databases, gzip, encrypt, etc."
do_database_backup | curl -X PUT --data-binary @- "$UPLOAD_URL"

This has some definite advantages over the old way. Now, if an attacker does manage to root this machine, they don’t get access to the rest of your Swift cluster. In fact, they can’t even read the old backups from this machine; the only thing this machine ever got was some signed URLs that work with PUT requests. If you try to use one with the GET verb, it won’t work. Also, when you need to change your TempURL shared secret, you only have to do it in the signer application.

There you have it: easier administration and better security through applied cryptography.

An Introduction to TempURL in OpenStack Swift

TempURL is a middleware for Swift that lets you grant temporary access to particular objects. This post is intended to show how TempURL works and get you started using it.

Set Up Your Account

TempURL works by adding a signature to the URL based on a shared secret. So, to use TempURL, you need to tell Swift what the value of that secret is. That’s easy; it’s just a particular piece of metadata on your Swift account. Make up a secret to use (any string will do), and set it as the value of X-Account-Meta-Temp-Url-Key. Here’s some examples:

python-swiftclient
1
2
# (Assuming that $ST_AUTH, $ST_USER, and $ST_KEY are set in the environment)
$ swift post -m Temp-Url-Key:correcthorsebatterystaple
curl
1
2
3
$ curl -X POST -H "X-Auth-Token: $AUTH_TOKEN" \
    -H "X-Account-Meta-Temp-Url-Key: correcthorsebatterystaple" \
    https://swiftcluster/v1/myaccount
pyrax
1
2
3
import pyrax
pyrax.set_credential_file("~/.rackspace-pyrax-creds")   # or however you like to do auth
pyrax.cloudfiles.set_account_metadata('temp-url-key', 'correcthorsebatterystaple')

Set Up Your Swift Cluster

If you’re not operating a Swift cluster, skip to the next section, as this doesn’t apply to you.

TempURL is part of Swift, so the code is already on your cluster. To enable it, make sure that it’s in your proxy server’s middleware pipeline.

/etc/swift/proxy-server.conf
1
2
3
4
5
6
[pipeline:main]
pipeline = proxy-logging healthcheck cache tempurl tempauth proxy-logging proxy-server

[filter:tempurl]
use = egg:swift#tempurl
methods = GET HEAD PUT DELETE POST

This configuration enables all the HTTP verbs. For historical reasons, the default set of allowed methods is only GET, HEAD, and PUT. However, I can’t think of any reason not to let your users use whatever HTTP verbs they want.

Signing URLs

A URL’s signature is derived from four things:

  • the URL’s path component (e.g. “/v1/AUTH_myusername/gutenberg/beowulf.epub”)
  • the expiration time as a Unix timestamp (e.g. 1368056658 for 2013-05-08T23:44:18Z)
  • the HTTP verb (e.g. GET, POST)
  • the shared secret

The signature is the HMAC-SHA1 of the method, the expiration time, and the path, separated by newlines. The shared secret is the HMAC key.

Since that’s technically correct but completely impenetrable, let’s see an example.

Signing a URL
1
2
3
4
5
6
7
8
9
10
11
12
13
import hashlib
import hmac
import time

method, path = 'GET', '/v1/AUTH_myusername/gutenberg/beowulf.epub'
expiration = int(time.time() + 1800)  # valid until half an hour from now
shared_secret = 'correcthorsebatterystaple'

hmac_body = "\n".join([method, expiration, path])
sig = hmac.new(shared_secret, hmac_body, hashlib.sha1).hexdigest()

signed_url = "https://swift/{path}?temp_url_sig={sig}&temp_url_expires={exp}".format(
    path=path, sig=sig, exp=expiration)

If you don’t feel like writing your own URL signer, Swift comes with a utility called swift-temp-url that does just what the name implies. It’s really simple to use:

Signing with swift-temp-url
1
2
3
$ swift-temp-url GET 1800 /v1/AUTH_myusername/gutenberg/beowulf.epub correcthorsebatterystaple
/v1/AUTH_myusername/gutenberg/beowulf.epub?temp_url_sig=cf62879f772eaf73c86c2f3bc8b6a4cfd94d4125&temp_url_expires=1368059214
$ 

That’s all there is to using TempURL. In future posts, I’ll explore just a few of the awesome things you can do with signed URLs.

Programming Languages Don’t Count

The Count

Remember the Count from Sesame Street? The Count loves nothing more than to count, all day long. Unfortunately for him, he’s limited in how fast he can speak the numbers aloud. Imagine if the Count could program! Why, there’d be no limit to the numbers he could count to! Right?

Nope. Most programming languages don’t count. That’s not to say that they don’t matter, but that they don’t count: 1, 2, 3, 4, and so on.

C makes the Count sad

Let’s count in C!

Counting in C
1
2
3
4
5
6
7
8
9
#include <stdio.h>

int main(int argc, char* argv[]) {
  int x = 1;
  while (1) {
    printf("%d\n", x);
    x++;
  }
}

Look at all those numbers. 1, 2, 100, 1000! 6,827,251! They just keep going up and up and up! 2,000,000,000! 2,147,483,647! -2,147,483,648!

“Wait!” says the Count. “My program was counting up, up, up! Where’d this negative number come from? Adding one to a number should always make it bigger, not smaller.”

The Count has fallen victim to integer overflow. In C, the declaration “int x” causes the compiler to allocate a small chunk of memory for the variable x. Ints are quite small; usually 32 bits (it’s system-dependent; YMMV). That means that an int can only have 232 different values. The possible values are split in half between negative and positive numbers, so there are only 231 possible positive numbers and 231 possible negative numbers that an int can hold.

To see why 2,147,483,647 + 1 = -2,147,483,648, the Count needs to understand how counting in binary works. Let’s look at a few 32-bit binary numbers.

1 = 00000000000000000000000000000001

2 = 00000000000000000000000000000010

3 = 00000000000000000000000000000011

4 = 00000000000000000000000000000100

(skip ahead a bit)

127,553,821 = 00000111100110100101000100011101

Now, C ints are typically represented in two’s complement format. In two’s complement, the most significant (i.e. leftmost) bit of a number is the sign bit. If it’s 0, then the number is positive, and if it’s 1, then the number is negative.

So, what happens when we get to 2,147,483,647 + 1? Well, let’s look at it in binary.

2,147,483,647 = 01111111111111111111111111111111

Add 1:

2,147,483,648 = 10000000000000000000000000000000

But the leftmost bit of that number is 1, making it negative. That’s the kicker: the leftmost bit is special, so as soon as you count high enough to change it, the value that the bits represent becomes negative.

Java, maybe?

Count.java
1
2
3
4
5
6
7
8
9
class Count {
    public static void main(String argv[]) {
        int x = 1;
        while (true) {
            System.out.println(x);
            x++;
        }
    }
}

And we get (skipping a few at the start)

2,147,483,645,

2,147,483,646,

2,147,483,647…

-2,147,483,648.

Same problem. C and Java aren’t alone here; other languages with this problem include C++, C#, Objective C, PHP, Visual Basic, Delphi, COBOL, and Fortran. 1

Javascript makes the Count sad in an entirely different way

“Well,” says the Count, “Java has many different kinds of numbers. There’s byte, short, int, long, float, and double. That’s six! Six kinds of numbers! Ah, ah, ah. But I digress. What about Javascript? It only has one kind of number. Surely it can do the job.”

Let’s count in Javascript!

Counting in Javascript
1
2
3
4
5
var x = 1;
while (true) {
    console.log(x);
    x = x + 1;
}

“Looks pretty good,” says the Count. “1, 2, 3, 4, …, 9007199254740991, 9007199254740992, 9007199254740992, 9007199254740992… Oh, no! The numbers have stopped counting up!”

Sadly, the Count’s dream of counting forever won’t work in Javascript either. Javascript numbers are 64-bit IEEE floating-point numbers. Basically, that means that Javascript numbers are stored in binary scientific notation. That’s where you write numbers as a mantissa m and an exponent e, and then the value is m × 2e. The mantissa is stored using 53 bits, the exponent is stored using 10 bits, and the remaining bit is a sign bit. (There’s those pesky fixed sizes again. The Count had a hunch they’d make an appearance.)

In binary scientific notation, 1100101 would be written as mantissa = 1.100101, e = 110 (decimal 6).

Let’s take a look at that number where Javascript stopped counting. 9007199254740992 is 253, which means it’s the smallest number that can’t be represented with 53 bits (you can get from 0 to 253 – 1).

So we’ve got this number 253, represented with mantissa 1.0000…000 and exponent 110101 (decimal 53). The next biggest number we can represent is the one with mantissa 1.0000…001 and the same exponent. That number is 253 + 2, or in decimal, 9007199254740994.

“What happened to 253 + 1?” asked the Count. “Shouldn’t that come before 253 + 2?”

Well, yes, it should, but it was swallowed up by rounding error. With an exponent of 53, there’s no way to represent the number 1. Adding 1 to the mantissa adds 2 to the value of the number. Essentially, the exponent got big enough that now there’s no more ones place. When we added 253 and 1, Javascript had to round it to either 253 or 253 + 2, and the rules for floating-point numbers say that it must be rounded down.

Counting on Clojure

“Is there no language out there that knows how to count?” laments the Count. “Surely one of them must get it right! What about Clojure? Perhaps it has learned from the mistakes of others.”

Counting in Clojure
1
(dorun (map println (iterate inc 1)))

This starts off as you’d expect. 1, 2, 3…

Then comes 2147483647 (231 – 1) and 2147483648 (232) and 2147483649 (232 + 1); no problems with 32-bit integer overflow here.

A while later, here comes 9007199254740992, and 9007199254740993, and 9007199254740994; no problems with floating-point rounding.

And if this program runs long enough, it’ll count to 9223372036854775808 (263 – 1), 340282366920938463463374607431768211455 (2128 – 1), and on and on as long as the Count’s computer still works.

The way Clojure achieves this is by using a thing called a big (arbitrary-precision) integer 2. It expands to be as big as it needs to be. Think of this: when the Count adds 1 to 9999, he can’t write it using just 4 places, so he makes a fifth number place and writes 10,000. Big integers work the same way; 231 + 1 won’t fit into 32 bits, so more memory gets allocated so that 231 + 1 will fit.

This means Clojure can count until the Count’s computer runs out of memory to allocate, and that’ll take a very, very long time. With an amount of memory that’s infinitesimally small, like 1 KB (modern computers tend to have 2 GB of memory, or about 2.1 million times that much), the Count can count until the heat death of the universe.

“Finally!” says the Count.

Conclusion

Choose your programming language carefully, or you may find that your project doesn’t count.


  1. Of course there’s java.lang.BigInteger; but x = x.add(1) just isn’t the same. Also, unless your code and every library you interface with accepts, returns, and uses BigIntegers internally, you’re still susceptible to integer overflow.

  2. Clojure doesn’t start off with BigIntegers; it waits until the numbers are big enough to warrant their use. That way, operations on small numbers stay fast.