In this chapter, I’ll be discussing the methods I use to authenticate against SoftLayer’s Object Storage regions, and how I developed my own local cache of the authentication token that’s used when uploading and download Object Storage objects.
South Curl Curl: Whenever I use the Linux curl command, I always think about this beach town in the suburbs of Sydney, Australia, where I stayed during my first visit there in 1987.
During planning, I realized that the curl command would be central to most of my Object Storage library. Curl is a Linux command that lets you transfer data to or from a server using a variety of supported protocols. My use would be strictly using https (which is how SoftLayer’s Object Storage is setup). Curl allows you to setup http headers, and send those along with this request.
As stated in the my previous blog post, the objective was really to glue all the pieces of the puzzle together using Bash, and to ultimately call curl correctly to either upload or download objects. Any call to my library would spend 99.99% of the time in curl.
Authentication: Before you can transact any real data with Object Storage, you have to authenticate against the SoftLayer authentication service on either of their internal or external networks, using a well-known authentication URL with the username and password that was generated (by them) for you.
The authentication URL on the private network, can be summed up in this Bash statement:
auth_url=”https://${region}.objectstorage.service.networklayer.com/auth/v1.0″
…where ${region} is the label for the region name assigned by SoftLayer. For example, the Dallas region (which serves all of their Dallas data centers) is dal05. By return, after you have successfully authenticated, the auth server will provide you with some http headers, and the http body will be a JSON string which contains all the supported auth URLs.
From now on, to read or write any objects stored in this region, you will have to supply the authentication token that was supplied (as one of the headers) with any further requests. Along with the auth token will be the date and time that the token was issued, and the number of seconds from that date/time that it will expire.
X-Auth-Token-Expires: 84157
Date: Thu, 04 Jan 2018 17:49:14 GMT
X-Auth-Token: AUTH_tkf9040bde767540b09531acfcac46d5b4
But be careful: SoftLayer caches this token on their side, and a token cache will typically expire after 23 to 24-hours. So the token you acquire may have been cached in SoftLayer by some other authentication request, on some other server somewhere. Don’t assume that the date/time supplied is the date/time of your most recent auth request.
I made this mistake early on, and it tested out OK – basically because I was the only one testing. But in production, I would occasionally find that the curl command I was using to fetch an object would fail because of permissions. That’s because for optimization reasons, I would only authenticate once at the start of provisioning the virtual machines for a demo server. But in reality I may have been grabbing an auth token that had almost expired, which would fail at some point during a long build. This would have been due to another server authenticating much earlier, and because its auth token was still valid it was also handed to me (but expired mid-way through my build).
So, for my authentication routines I had:
- SLObjectStorageGetAuth: This is the top-level routine that should always be called to authenticate against a region. I pass it the region, and it returns an auth token. In my code, the credentials are kept in memory, in exported variables.
- SLObjectStorageGetAuthFile: This is a routine that’s called by the above that will download and cache the headers and payload after a successful authentication for a region. It caches this information on the server’s local disk. This way the locally cached data (specifically the X-Auth-Token-Expires and the Date headers), can be used to calculate if the token has expired. If the token is within 5-seconds of expiring, I wait 10-seconds then go get a new token from the auth server.
The SLObjectStorageGetAuth routine is called before every curl command that’s used to upload or download an object.
The Linux date command: This command has evolved into a date calculator over the years. As shown previously the following will display a date on the terminal:
date
Wed Jan 17 11:16:22 CST 2018
But it can do much more than that, for example the following displays the number of seconds since 1970-01-01 00:00:00 UTC:
date ‘+%s’
1516209607
You can also supply a date, and get it to turn that date into a number of seconds:
date -d ‘1971-01-01 00:00:00 UTC’ ‘ ‘+%s’
31536000
With Bash, there are several ways to get the output of a command into a shell variable. This is the one I favor:
seconds=$(date -d ‘1971-01-01 00:00:00 UTC’ ‘+%s’)
echo $seconds
31536000
With use of the date command, it’s easy enough to:
- Isolate the auth token’s creation date, and time-to-live (in seconds). If these are not cached locally, then re-auth and cache.
- Calculate the number of seconds for the auth creation date, and add the time-to-live value.
- Calculate the number of seconds for the current date/time (“now”).
- See if the expiration seconds is greater than now; if so re-auth and cache.
- See if the expiration seconds is within 5-seconds of now; if so, wait 10-seconds, and then re-auth and cache.
Remember that this logic is traversed before every curl command that is uploading or downloading an object, and because the authentication token (and associated data) is cached locally, saves a round-trip to the auth server for every curl.
The local disk cache is nothing more than a directory named swauth_cache, under which are text files named after the region, for example dal05. These files contain the last output of a call to the auth URL.
As an example, the code to call the auth URL may look like:
if ! curl –retry 10 -s -S -i \
-H “X-Auth-User: ${username}” \
-H “X-Auth-Key: ${password}” “${auth_url}” \
> “${swauth_file}”
then
‘ Some error code
fi
In this case any output (headers and payload) would go to our cache file, and any error output would go to the standard error.
