Sam Merritt

torgomatic

TempURL Tricks: Write-Only Access for Backups

Imagine that you have a small fleet of machines sitting in the cloud somewhere serving up a small herd of webapps. Of course, you need to make sure your users’ data is durable even in the event of failures. For a lot of cases, it makes sense to store data directly in Swift and let Swift take care of durability, but you can’t do that with everything. In particular, you can’t do that with RDBMS data, so you have to keep backups.

Now, the naïve way of doing this is to distribute your Swift credentials to every machine, then have a cron job that calls pg_dumpall and sends the result to Swift. This works okay, but has some problems.

First, you’re distributing your Swift credentials to every machine. If an attacker breaks in to any one of your machines, they have full access to all the data in your object store. They can download all your database dumps and sift them for credit card info, they can replace your users’ uploaded videos with porn, or they can be destructive and delete everything.

Second, credential rotation becomes a real pain in the butt. When you rotate your passwords and API keys, you now have to distribute the new Swift credentials to every single box you have. If you miss one, backups just stop working. If that happens, you’d better hope your cron emails are going somewhere useful or else you may never find out.

Fortunately, there’s a way to solve both those problems. The basic idea is to take the Swift credentials off all the machines that don’t need them, and instead have one service that’s responsible for handing out signed PUT tempurls.

Let’s take a look at a simple little webapp that hands back signed URLs. Now, of course you can write this in whatever language and framework you like. I like Python and Flask, so here’s what you get.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
from flask import Flask, request, abort
import hashlib
import hmac
import time


def get_tempurl_secret():
    # You'll probably want to do something smart here
    return "correcthorsebatterystaple"


def get_swift_account():
    # You'll probably want to do something smart here
    return "AUTH_test"


def is_authorized(request):
    # You'll probably want to do something smart here
    return True


def sign(method, expires, container_name, object_name):
    url = "/v1/{acc}/{con}/{obj}".format(
        acc=get_swift_account(),
        con=container_name,
        obj=object_name)
    sig = hmac.new(
        get_tempurl_secret(),
        "\n".join([method, str(expires), url]),
        hashlib.sha1).hexdigest()
    return url + "?temp_url=sig={sig}&temp_url_expires={exp}".format(
        sig=sig,
        exp=expires)


app = Flask(__name__)


@app.route("/")
def sign_urls():
    if not is_authorized(request):
        abort(403)

    segments = int(request.args.get('segments', '1'))
    duration = int(request.args.get('duration', '3600'))
    hostname = request.args['hostname']
    container = "backups_" + hostname
    now = time.time()
    base_object = str(now)
    expires = now + duration
    urls = [sign('PUT', duration, container, "%s_%d" % (base_object, i))
            for i in xrange(segments)]
    return "\n".join(urls)


if __name__ == '__main__':
    app.run()

The app is also on Github with a proper license, a README, and all that jazz. You can fork it and add in some actual authorization as well as a secret that isn’t “correcthorsebatterystaple”. :)

Now, let’s look at a simple shell script that uses this to perform a backup:

1
2
3
4
5
6
7
#!/bin/bash
HOSTNAME="`hostname -f`"
UPLOAD_PATH="`curl https://signer.example.com/?host=$HOSTNAME`"
UPLOAD_URL="https://swift.example.com$UPLOAD_PATH"

# do_database_backup is shorthand for "dump databases, gzip, encrypt, etc."
do_database_backup | curl -X PUT --data-binary @- "$UPLOAD_URL"

This has some definite advantages over the old way. Now, if an attacker does manage to root this machine, they don’t get access to the rest of your Swift cluster. In fact, they can’t even read the old backups from this machine; the only thing this machine ever got was some signed URLs that work with PUT requests. If you try to use one with the GET verb, it won’t work. Also, when you need to change your TempURL shared secret, you only have to do it in the signer application.

There you have it: easier administration and better security through applied cryptography.