Today I hit an issue which completely stumped me (for a while). Running an aws s3 cp from a Docker container — that had never failed before in the same scenario — was failing every time with ‘A client error (403) occurred when calling the HeadObject operation: Forbidden’. When run from my Mac directly with the same AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY it worked. WTF!

Some Googling lead me to this question, which contained this answer of particular interest:

“can also be caused if your system clock is too far off. I was 12 hours in the past and got this error. Set the clock to within a minute of the true time, and error went away.”

So I checked the time of the container by running date an sure enough, the container’s date was 3 days ago:

$ date
Fri Jul 29 15:18:17 UTC 2016

I was a bit perplexed how the container — or Docker — got in such a state, because I’d never had an issue like this before. I found this answer which indicated that when using boot2docker the Docker Engine VM’s time can drift when OS X is asleep:

“Time synch becomes an issue because the boot2docker host has its time drift while your OS is asleep.”

I’d recently updated to Docker for Mac, so maybe that had something to do with it? To correct the issue, I initially tried to set the container’s date, before I learned you can’t do that without using --privileged. Doing so can cause bad things to happen:

“Using the –privileged switch here will cause the command from the container to affect the docker host. Also other bad things can happen if you use the –privileged without knowing the effects.”

So I thought the quickest way to see if things would resolve themselves would be to restart my machine (something I don’t do often enough). Sure enough, after a restart running the same image produced the correct date and time:

$ date
Mon Aug  1 15:18:17 UTC 2016

But I wasn’t happy with just getting things working again; I wanted to know what the issue was with the aws cli when your system date/time isn’t correct. So I dug a little deeper into the AWS Command Line Interface User Guide and found the answer on page 10:

The AWS CLI signs requests on your behalf, and includes a date in the signature. Ensure that your computer’s date and time are set correctly; if not, the date in the signature may not match the date of the request, and AWS rejects the request.

And there you have it. A mistake not to be made again in the future.

Your thoughts? I'd love to hear them. Please get in contact.