Integrating BITS with Amazon S3 (Part 2)
In part 1 of this post series we talked about the pain points of integrating BITS with Amazon S3. In this part of the series we will look at how to overcome the issue described in part 1.
So basically, at this point, we cannot change the way BITS works obviously. Also, we cannot change the way Amazon S3 works either when security is added to resources. The one thing we can do, however, is be a middle man in the process. In other words, instead of downloading a file from Amazon S3, why can’t we download it from another service (i.e. a proxy service that acts on behalf of Amazon S3 for us) that knows how to deal with these intricacies? The following illustrates all the components taking the proxy service into account.
So now, instead of giving a bits a file url that points to S3, we give a url that points to our proxy service.
This is the approach we took and it worked out really well. You still have to take care of generating both a presigned url for a HEAD and a GET request but when BITS makes either request to our proxy service, we can respond appropriately.
- When BITS make a HEAD request to the proxy service, we make a GET request to Amazon S3 and return the obtained response back to BITS.
- When a GET request comes in, we just do a redirect (302) to the actual Amazon S3 presigned url for a GET and it will respond to BITS directly.
Initially, we hosted our proxy service within the process of the client application. This presented some drawbacks like what happens if the user decided to kill the client application while we were downloading? BITS would encounter a transient error and continue to try to reestablish communications but it tries to do this for a limited amount of time only. What if the application was down for a longer period of time? We decided to host the proxy service in a Windows Service application on the local machine. That way, it was always up and it would automatically start on system reboot.
I hope you found this post helpful. When we encountered this issue, we searched for a solution and there was very little information that was available except for a post here or there that eventually led us in the right direction. Although making a basic wcf service that implements this functionality is rather simple, I will be posting the code for the proxy service in the near future so please check back if not already posted.