S3Proxy implements the S3 API and proxies requests

atmos aws-s3 azure backblaze-b2 google-cloud-storage openstack-swift proxy s3

Go to file

oxalis-gps 09d744763d Added Docker environment variable "S3PROXY_CORS_ALLOW_ALL" refs issues #142 refs Pull #144 The default value is false		2017-02-16 13:49:48 -08:00
s3-tests@3b8eca26c2	Update s3-tests to fix Python six.moves error	2017-01-25 13:44:28 -08:00
src	Added Docker environment variable "S3PROXY_CORS_ALLOW_ALL"	2017-02-16 13:49:48 -08:00
.gitignore	more gitignore	2015-03-26 17:17:29 -07:00
.gitmodules	Add s3-tests submodule	2015-01-22 17:05:04 -08:00
.travis.yml	Provide Dockerfile	2016-07-01 19:31:05 -07:00
Dockerfile	Added Docker environment variable "S3PROXY_CORS_ALLOW_ALL"	2017-02-16 13:49:48 -08:00
LICENSE	Initial commit of S3Proxy	2014-07-27 13:32:04 -07:00
README.md	Update copyright year	2017-02-15 23:15:16 -08:00
pom.xml	Upgrade to modernizer-maven-plugin 1.5.0	2017-02-15 23:41:14 -08:00

README.md

S3Proxy

S3Proxy implements the S3 API and proxies requests, enabling several use cases:

translation from S3 to Backblaze B2, Google Cloud, Microsoft Azure, or OpenStack Swift
testing without Amazon by using the local filesystem
extension via middlewares, e.g., eventual consistency simulation
embedding into Java applications

Usage with Docker

Docker Hub hosts a Docker image and has instructions on how to run it.

Usage without Docker

Users can download releases from GitHub. Developers can build the project by running mvn package which produces a binary at target/s3proxy. S3Proxy requires Java 7 to run.

Configure S3Proxy via a properties file. An example using the local file system as the storage backend with anonymous access:

s3proxy.authorization=none
s3proxy.endpoint=http://127.0.0.1:8080
jclouds.provider=filesystem
jclouds.filesystem.basedir=/tmp/s3proxy

First create the filesystem basedir:

mkdir /tmp/s3proxy

Next run S3Proxy. Linux and Mac OS X users can run the executable jar:

chmod +x s3proxy
s3proxy --properties s3proxy.conf

Windows users must explicitly invoke java:

java -jar s3proxy --properties s3proxy.conf

Finally test by creating a bucket then listing all the buckets:

$ curl --request PUT http://localhost:8080/testbucket

$ curl http://localhost:8080/
<?xml version="1.0" ?><ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>75aa57f09aa0c8caeab4f8c24e99d10f8e7faeebf76c078efc7c6caea54ba06a</ID><DisplayName>CustomersName@amazon.com</DisplayName></Owner><Buckets><Bucket><Name>testbucket</Name><CreationDate>2015-08-05T22:16:24.000Z</CreationDate></Bucket></Buckets></ListAllMyBucketsResult>

Supported storage backends

atmos
aws-s3
azureblob
b2
filesystem (on-disk storage)
google-cloud-storage
openstack-swift
rackspace-cloudfiles-uk and rackspace-cloudfiles-us
s3
transient (in-memory storage)

See the wiki for examples of configurations.

Limitations

S3Proxy has broad compatibility with the S3 API, however, it does not support:

ACLs other than private and public-read
BitTorrent hosting
bucket logging
cross-origin resource sharing, see #142
POST upload policies, see #73
list objects v2, see #168
object server-side encryption
object tagging
object versioning, see #74
requester pays buckets

S3Proxy emulates the following operations:

copy multi-part objects, see #76

The wiki collects compatability notes for specific storage backends.

References

Apache jclouds provides storage backend support for S3Proxy
Ceph s3-tests help maintain and improve compatibility with the S3 API
fake-s3, gofakes3, S3 ninja, and s3rver provide functionality similar to S3Proxy when using the filesystem backend
GlacierProxy and SwiftProxy provide similar functionality for the Amazon Glacier and OpenStack Swift APIs
sbt-s3 run S3Proxy via the Scala Build Tool
swift3 provides an S3 middleware for OpenStack Swift

License

Licensed under the Apache License, Version 2.0