2014-07-22 03:25:31 +00:00
S3Proxy
=======
2016-11-17 03:25:44 +00:00
[](https://github.com/andrewgaul/s3proxy/releases/)
[](https://hub.docker.com/r/andrewgaul/s3proxy/)
[](https://search.maven.org/#search%7Cga%7C1%7Ca%3A%22s3proxy%22)
[](https://twitter.com/S3Proxy)
2016-11-17 03:48:18 +00:00
S3Proxy implements the
2014-08-18 07:17:48 +00:00
[S3 API ](https://en.wikipedia.org/wiki/Amazon_S3#S3_API_and_competing_services )
2016-11-17 03:48:18 +00:00
and *proxies* requests, enabling several use cases:
* translation from S3 to Backblaze B2, Google Cloud, Microsoft Azure, or OpenStack Swift
* testing without Amazon by using the local filesystem
* extension via middlewares, e.g., eventual consistency simulation
* embedding into Java applications
2014-07-22 03:25:31 +00:00
2016-07-02 07:34:12 +00:00
Usage with Docker
-----------------
[Docker Hub ](https://hub.docker.com/r/andrewgaul/s3proxy/ ) hosts a Docker image
and has instructions on how to run it.
Usage without Docker
--------------------
2014-08-12 08:30:17 +00:00
Users can [download releases ](https://github.com/andrewgaul/s3proxy/releases )
2015-12-17 19:03:22 +00:00
from GitHub. Developers can build the project by running `mvn package` which
2014-08-12 08:30:17 +00:00
produces a binary at `target/s3proxy` . S3Proxy requires Java 7 to run.
2014-07-22 03:25:31 +00:00
2015-11-17 00:14:38 +00:00
Configure S3Proxy via a properties file. An example using the local
file system as the storage backend with anonymous access:
2014-07-22 03:25:31 +00:00
```
2014-08-12 08:30:17 +00:00
s3proxy.authorization=none
s3proxy.endpoint=http://127.0.0.1:8080
jclouds.provider=filesystem
2015-11-17 00:14:38 +00:00
jclouds.filesystem.basedir=/tmp/s3proxy
2014-07-22 03:25:31 +00:00
```
2015-11-17 00:14:38 +00:00
First create the filesystem basedir:
2014-07-26 20:19:43 +00:00
```
2015-11-17 00:14:38 +00:00
mkdir /tmp/s3proxy
2014-07-26 20:19:43 +00:00
```
2015-11-17 00:14:38 +00:00
Next run S3Proxy. Linux and Mac OS X users can run the executable jar:
2015-05-09 01:13:44 +00:00
```
2015-11-17 00:14:38 +00:00
chmod +x s3proxy
s3proxy --properties s3proxy.conf
2015-05-09 01:13:44 +00:00
```
2015-11-17 00:14:38 +00:00
Windows users must explicitly invoke java:
2015-05-09 01:12:26 +00:00
```
2015-11-17 00:14:38 +00:00
java -jar s3proxy --properties s3proxy.conf
```
Finally test by creating a bucket then listing all the buckets:
```
$ curl --request PUT http://localhost:8080/testbucket
$ curl http://localhost:8080/
<?xml version="1.0" ?> < ListAllMyBucketsResult xmlns = "http://s3.amazonaws.com/doc/2006-03-01/" > < Owner > < ID > 75aa57f09aa0c8caeab4f8c24e99d10f8e7faeebf76c078efc7c6caea54ba06a< / ID > < DisplayName > CustomersName@amazon.com< / DisplayName > < / Owner > < Buckets > < Bucket > < Name > testbucket< / Name > < CreationDate > 2015-08-05T22:16:24.000Z< / CreationDate > < / Bucket > < / Buckets > < / ListAllMyBucketsResult >
2015-05-09 01:12:26 +00:00
```
2015-11-11 01:12:16 +00:00
Supported storage backends
--------------------------
* atmos
* aws-s3
* azureblob
2015-04-09 01:29:41 +00:00
* b2
2015-11-11 01:12:16 +00:00
* filesystem (on-disk storage)
* google-cloud-storage
* openstack-swift
* rackspace-cloudfiles-uk and rackspace-cloudfiles-us
* s3
* transient (in-memory storage)
2016-07-07 11:21:35 +00:00
See the wiki for [examples of configurations ](https://github.com/andrewgaul/s3proxy/wiki/Storage-backend-examples ).
2014-07-22 03:25:31 +00:00
Limitations
-----------
2016-02-05 08:19:44 +00:00
S3Proxy has broad compatibility with the S3 API, however, it does not support:
2016-01-21 20:13:01 +00:00
* ACLs other than private and public-read
2016-01-08 23:30:56 +00:00
* BitTorrent hosting
* bucket logging
2016-06-22 18:58:16 +00:00
* cross-origin resource sharing, see [#142 ](https://github.com/andrewgaul/s3proxy/issues/142 )
2015-04-09 01:29:41 +00:00
* POST upload policies, see [#73 ](https://github.com/andrewgaul/s3proxy/issues/73 )
2016-11-16 18:55:39 +00:00
* list objects v2, see [#168 ](https://github.com/andrewgaul/s3proxy/issues/168 )
2014-08-28 18:41:56 +00:00
* object server-side encryption
2015-08-03 00:11:31 +00:00
* object versioning, see [#74 ](https://github.com/andrewgaul/s3proxy/issues/74 )
2016-01-08 23:30:56 +00:00
* requester pays buckets
2014-07-22 03:25:31 +00:00
2015-11-11 01:12:16 +00:00
S3Proxy emulates the following operations:
2015-04-09 01:29:41 +00:00
* copy multi-part objects, see [#76 ](https://github.com/andrewgaul/s3proxy/issues/76 )
2015-11-11 01:12:16 +00:00
2016-01-21 20:13:01 +00:00
The wiki collects
[compatability notes ](https://github.com/andrewgaul/s3proxy/wiki/Storage-backend-compatibility )
for specific storage backends.
2015-11-17 01:45:50 +00:00
2014-07-22 03:25:31 +00:00
References
----------
2015-05-13 18:44:49 +00:00
2015-12-17 19:03:22 +00:00
* Apache [jclouds ](http://jclouds.apache.org/ ) provides storage backend support for S3Proxy
2015-05-13 18:44:49 +00:00
* Ceph [s3-tests ](https://github.com/ceph/s3-tests ) help maintain and improve compatibility with the S3 API
2016-03-01 17:51:12 +00:00
* [fake-s3 ](https://github.com/jubos/fake-s3 ), [gofakes3 ](https://github.com/johannesboyne/gofakes3 ), [S3 ninja ](https://github.com/scireum/s3ninja ), and [s3rver ](https://github.com/jamhall/s3rver ) provide functionality similar to S3Proxy when using the filesystem backend
2015-11-17 00:14:38 +00:00
* [GlacierProxy ](https://github.com/bouncestorage/glacier-proxy ) and [SwiftProxy ](https://github.com/bouncestorage/swiftproxy ) provide similar functionality for the Amazon Glacier and OpenStack Swift APIs
2016-02-05 08:14:06 +00:00
* [sbt-s3 ](https://github.com/localytics/sbt-s3 ) run S3Proxy via the Scala Build Tool
2015-12-09 07:51:21 +00:00
* [swift3 ](https://github.com/openstack/swift3 ) provides an S3 middleware for OpenStack Swift
2014-07-22 03:25:31 +00:00
License
-------
2016-01-10 05:14:59 +00:00
Copyright (C) 2014-2016 Andrew Gaul
2014-07-22 03:25:31 +00:00
Licensed under the Apache License, Version 2.0
