Working out with Amazon S3

Recently, I was bit playing around with amazon S3. This is said to be the cheapest storage for people who don't want to spend much of their brain cells thinking about DB issues. Now, Amazon S3 primarily supports Web service (SOAP) to store and query data out of the S3 server. The concept behind S3 is that you have to create a bucket and under it you can store as many objects you want. Generally for a particular account 100 is the threshold value for buckets. In this post, I will basically focus on configuring your proxy for uploading large files to amazon using WSE and optionally I will point out a cool browser extension for analyzing your S3 space and third party library alternative.

Amazon depends on WSE 2.0 if you want to upload files larger than 1MB. The large file is transferred in the form of DIME (Direct Internet Message Encapsulation) attachment. Microsoft has released WSE 3.0 that uses MTOM which is a W3C recommendation for replacing DIME. But the time of writing this post it is not yet supported by Amazon.

How to prepare your proxy to support DIME  ?  Well, first you have to get the WSE 2.0 SP3 from Microsoft downloads. Once the step is completed, you can copy the Microsoft.Web.Services2.dll to your local folder. Basically, once the WSE 2.0 pack is installed the DLL is added to the GAC so it is not a required step, but if you want to ship it along with your distribution then you can consider doing that.

Now once you have created the proxy from http://s3.amazonaws.com/doc/2006-03-01/AmazonS3.wsdl. You need to do some manual tweaks before your project is ready to go with WSE.

From the Diff for Reference.cs we can see that we have to add a reference to Microsoft.Web.Services2 and inherit from WebServicesClientProtocol instead of SoapHttpClientProtocol

Pretty easy but hang on a bit. While you are instantiating the proxy class you have to add/remove some standard WS filters. Therefore, if we have a method called CreateProxyInstance , with the OutputFilters it will look like

private AmazonS3 CreateProxyInstance()
{
    AmazonS3 proxy = new AmazonS3();

    /// Remove the standard WSE soap headers.
    proxy.Pipeline.OutputFilters.Remove(typeof(Microsoft.Web.Services2.Security.SecurityOutputFilter));
    proxy.Pipeline.OutputFilters.Remove(typeof(Microsoft.Web.Services2.Referral.ReferralOutputFilter));
    proxy.Pipeline.OutputFilters.Remove(typeof(Microsoft.Web.Services2.Policy.PolicyEnforcementOutputFilter));

    /// Add our custom filter to remove the unwanted WSE soap headers.
    proxy.Pipeline.OutputFilters.Add(new HeaderOutputFilter("wsa:"));

    return proxy;
}

Here the HeaderOuputFilter is a inherited from SoapOutputFilter , where we build the SoapEnvolop header without wsa node. Pretty much that's it for setting up the proxy , now let's upload something to the S3 server.

MetadataEntry[] metadataEntries = new MetadataEntry[2];

metadataEntries[0] = mContentType;
metadataEntries[1] = contentLength;

DateTime timestamp = GetCurrentTimeInMilliseconds();
string signature = GenerateSignature("PutObject", timestamp);

using (AmazonS3 proxy = CreateProxyInstance())
{
    MemoryStream stream = new MemoryStream(content);
    Microsoft.Web.Services2.Dime.DimeAttachment dimeAttachment = new 
    Microsoft.Web.Services2.Dime.DimeAttachment("S3Object",   
    Microsoft.Web.Services2.Dime.TypeFormat.Unknown, stream);
    dimeAttachment.ContentType = contentType; 

    /// add the object in wire.
    proxy.RequestSoapContext.Attachments.Add(dimeAttachment);
    proxy.PutObject(_bucketName, key, metadataEntries, content.LongLength, grants, 
    StorageClass.STANDARD, true, _accessKey, timestamp, true, signature, null);
}

From the fragmented code block we can see that "PutObject" is the method for large content not "PutObjectInLine" and along passing the stream, we have to pass the content type as well or less it wont recognized properly even you specify it in the MimeDataEntry array.

So far that is all. I have joined in a tiny class library project that will be helpful if you are getting started with S3 using SOAP. You can download it HERE. About the tool there is a cool firefox addon called S3Fox that gives you an explorer look for analyzing and even let you to add/delete items directly from S3.

Finally, WSE 2.0 does not work in medium trust mode which is an absolute show stopper for CMS and blog solutions. There is a nice S3 library at codeplex can be found at www.codeplex.com/ThreeSharp that don't requires any of these and can be a good alternative for medium trust problem.

Enjoy !!!