S3Proxy

S3Proxy implements the S3 API and proxies requests, enabling several use cases:

• translation from S3 to Backblaze B2, Google Cloud, Microsoft Azure, or OpenStack Swift
• testing without Amazon by using the local filesystem
• extension via middlewares
• embedding into Java applications

Usage with Docker

Docker Hub hosts a Docker image and has instructions on how to run it.

Usage without Docker

Users can download releases from GitHub. Developers can build the project by running mvn package which produces a binary at target/s3proxy. S3Proxy requires Java 7 to run.

Configure S3Proxy via a properties file. An example using the local file system as the storage backend with anonymous access:

s3proxy.authorization=none
s3proxy.endpoint=http://127.0.0.1:8080
jclouds.provider=filesystem
jclouds.filesystem.basedir=/tmp/s3proxy

First create the filesystem basedir:

mkdir /tmp/s3proxy

Next run S3Proxy. Linux and Mac OS X users can run the executable jar:

chmod +x s3proxy
s3proxy --properties s3proxy.conf

Windows users must explicitly invoke java:

java -jar s3proxy --properties s3proxy.conf

Finally test by creating a bucket then listing all the buckets:

$ curl --request PUT http://localhost:8080/testbucket

$ curl http://localhost:8080/
<?xml version="1.0" ?><ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>75aa57f09aa0c8caeab4f8c24e99d10f8e7faeebf76c078efc7c6caea54ba06a</ID><DisplayName>CustomersName@amazon.com</DisplayName></Owner><Buckets><Bucket><Name>testbucket</Name><CreationDate>2015-08-05T22:16:24.000Z</CreationDate></Bucket></Buckets></ListAllMyBucketsResult>

Usage with Java

Maven Central hosts S3Proxy artifacts and the wiki has instructions on Java use.

Supported storage backends

• atmos
• aws-s3 (Amazon-only)
• azureblob
• b2
• filesystem (on-disk storage)
• google-cloud-storage
• openstack-swift
• rackspace-cloudfiles-uk and rackspace-cloudfiles-us
• s3 (all implementations)
• transient (in-memory storage)

See the wiki for examples of configurations.

Middlewares

S3Proxy can modify its behavior based on middlewares:

• eventual consistency modeling
• large object mocking

Limitations

S3Proxy has broad compatibility with the S3 API, however, it does not support:

• ACLs other than private and public-read
• BitTorrent hosting
• bucket logging
• cross-origin resource sharing, see #142
• POST upload policies, see #73
• list objects v2, see #168
• object server-side encryption
• object tagging
• object versioning, see #74
• requester pays buckets

S3Proxy emulates the following operations:

• copy multi-part objects, see #76

The wiki collects compatability notes for specific storage backends.

Support

GitHub issues provide community support. Andrew Gaul provides commercial support via andrew@gaul.org.

References

• Apache jclouds provides storage backend support for S3Proxy
• Ceph s3-tests help maintain and improve compatibility with the S3 API
• fake-s3, gofakes3, S3 ninja, s3rver, and Scality S3 Server provide functionality similar to S3Proxy when using the filesystem backend
• GlacierProxy and SwiftProxy provide similar functionality for the Amazon Glacier and OpenStack Swift APIs
• minio provides filesystem and Azure storage backends.
• sbt-s3 runs S3Proxy via the Scala Build Tool
• swift3 provides an S3 middleware for OpenStack Swift

License

Copyright (C) 2014-2017 Andrew Gaul

Licensed under the Apache License, Version 2.0