2014-07-22 03:25:31 +00:00
S3Proxy
=======
2016-11-17 03:25:44 +00:00
[![Github All Releases ](https://img.shields.io/github/downloads/andrewgaul/s3proxy/total.svg )](https://github.com/andrewgaul/s3proxy/releases/)
[![Docker Pulls ](https://img.shields.io/docker/pulls/andrewgaul/s3proxy.svg )](https://hub.docker.com/r/andrewgaul/s3proxy/)
[![Maven Central ](https://img.shields.io/maven-central/v/org.gaul/s3proxy.svg )](https://search.maven.org/#search%7Cga%7C1%7Ca%3A%22s3proxy%22)
[![Twitter Follow ](https://img.shields.io/twitter/follow/S3Proxy.svg?style=social&label=Follow )](https://twitter.com/S3Proxy)
2016-11-17 03:48:18 +00:00
S3Proxy implements the
2014-08-18 07:17:48 +00:00
[S3 API ](https://en.wikipedia.org/wiki/Amazon_S3#S3_API_and_competing_services )
2016-11-17 03:48:18 +00:00
and *proxies* requests, enabling several use cases:
* translation from S3 to Backblaze B2, Google Cloud, Microsoft Azure, or OpenStack Swift
* testing without Amazon by using the local filesystem
2017-03-09 18:53:08 +00:00
* extension via middlewares
2016-11-17 03:48:18 +00:00
* embedding into Java applications
2014-07-22 03:25:31 +00:00
2016-07-02 07:34:12 +00:00
Usage with Docker
-----------------
[Docker Hub ](https://hub.docker.com/r/andrewgaul/s3proxy/ ) hosts a Docker image
and has instructions on how to run it.
Usage without Docker
--------------------
2014-08-12 08:30:17 +00:00
Users can [download releases ](https://github.com/andrewgaul/s3proxy/releases )
2015-12-17 19:03:22 +00:00
from GitHub. Developers can build the project by running `mvn package` which
2014-08-12 08:30:17 +00:00
produces a binary at `target/s3proxy` . S3Proxy requires Java 7 to run.
2014-07-22 03:25:31 +00:00
2015-11-17 00:14:38 +00:00
Configure S3Proxy via a properties file. An example using the local
file system as the storage backend with anonymous access:
2014-07-22 03:25:31 +00:00
```
2014-08-12 08:30:17 +00:00
s3proxy.authorization=none
s3proxy.endpoint=http://127.0.0.1:8080
jclouds.provider=filesystem
2015-11-17 00:14:38 +00:00
jclouds.filesystem.basedir=/tmp/s3proxy
2014-07-22 03:25:31 +00:00
```
2015-11-17 00:14:38 +00:00
First create the filesystem basedir:
2014-07-26 20:19:43 +00:00
```
2015-11-17 00:14:38 +00:00
mkdir /tmp/s3proxy
2014-07-26 20:19:43 +00:00
```
2015-11-17 00:14:38 +00:00
Next run S3Proxy. Linux and Mac OS X users can run the executable jar:
2015-05-09 01:13:44 +00:00
```
2015-11-17 00:14:38 +00:00
chmod +x s3proxy
s3proxy --properties s3proxy.conf
2015-05-09 01:13:44 +00:00
```
2015-11-17 00:14:38 +00:00
Windows users must explicitly invoke java:
2015-05-09 01:12:26 +00:00
```
2015-11-17 00:14:38 +00:00
java -jar s3proxy --properties s3proxy.conf
```
Finally test by creating a bucket then listing all the buckets:
```
$ curl --request PUT http://localhost:8080/testbucket
$ curl http://localhost:8080/
<?xml version="1.0" ?> < ListAllMyBucketsResult xmlns = "http://s3.amazonaws.com/doc/2006-03-01/" > < Owner > < ID > 75aa57f09aa0c8caeab4f8c24e99d10f8e7faeebf76c078efc7c6caea54ba06a< / ID > < DisplayName > CustomersName@amazon.com< / DisplayName > < / Owner > < Buckets > < Bucket > < Name > testbucket< / Name > < CreationDate > 2015-08-05T22:16:24.000Z< / CreationDate > < / Bucket > < / Buckets > < / ListAllMyBucketsResult >
2015-05-09 01:12:26 +00:00
```
2016-11-23 18:22:48 +00:00
Usage with Java
---------------
Maven Central hosts S3Proxy artifacts and the wiki has
[instructions on Java use ](https://github.com/andrewgaul/s3proxy/wiki/Using-S3Proxy-in-Java-projects ).
2015-11-11 01:12:16 +00:00
Supported storage backends
--------------------------
2017-03-17 03:27:29 +00:00
2015-11-11 01:12:16 +00:00
* atmos
2017-03-17 03:27:29 +00:00
* aws-s3 (Amazon-only)
2015-11-11 01:12:16 +00:00
* azureblob
2015-04-09 01:29:41 +00:00
* b2
2015-11-11 01:12:16 +00:00
* filesystem (on-disk storage)
* google-cloud-storage
* openstack-swift
* rackspace-cloudfiles-uk and rackspace-cloudfiles-us
2017-03-17 03:27:29 +00:00
* s3 (all implementations)
2015-11-11 01:12:16 +00:00
* transient (in-memory storage)
2016-07-07 11:21:35 +00:00
See the wiki for [examples of configurations ](https://github.com/andrewgaul/s3proxy/wiki/Storage-backend-examples ).
2017-03-09 18:53:08 +00:00
Middlewares
-----------
S3Proxy can modify its behavior based on middlewares:
* [eventual consistency modeling ](https://github.com/andrewgaul/s3proxy/wiki/Middleware---eventual-consistency )
* [large object mocking ](https://github.com/andrewgaul/s3proxy/wiki/Middleware-large-object-mocking )
2014-07-22 03:25:31 +00:00
Limitations
-----------
2016-02-05 08:19:44 +00:00
S3Proxy has broad compatibility with the S3 API, however, it does not support:
2016-01-21 20:13:01 +00:00
* ACLs other than private and public-read
2016-01-08 23:30:56 +00:00
* BitTorrent hosting
* bucket logging
2016-06-22 18:58:16 +00:00
* cross-origin resource sharing, see [#142 ](https://github.com/andrewgaul/s3proxy/issues/142 )
2015-04-09 01:29:41 +00:00
* POST upload policies, see [#73 ](https://github.com/andrewgaul/s3proxy/issues/73 )
2016-11-16 18:55:39 +00:00
* list objects v2, see [#168 ](https://github.com/andrewgaul/s3proxy/issues/168 )
2014-08-28 18:41:56 +00:00
* object server-side encryption
2016-12-05 10:32:10 +00:00
* object tagging
2015-08-03 00:11:31 +00:00
* object versioning, see [#74 ](https://github.com/andrewgaul/s3proxy/issues/74 )
2016-01-08 23:30:56 +00:00
* requester pays buckets
2014-07-22 03:25:31 +00:00
2015-11-11 01:12:16 +00:00
S3Proxy emulates the following operations:
2015-04-09 01:29:41 +00:00
* copy multi-part objects, see [#76 ](https://github.com/andrewgaul/s3proxy/issues/76 )
2015-11-11 01:12:16 +00:00
2016-01-21 20:13:01 +00:00
The wiki collects
[compatability notes ](https://github.com/andrewgaul/s3proxy/wiki/Storage-backend-compatibility )
for specific storage backends.
2015-11-17 01:45:50 +00:00
2017-04-12 20:20:49 +00:00
Support
-------
[GitHub issues ](https://github.com/andrewgaul/s3proxy/issues ) provide community
support. Andrew Gaul provides commercial support via < andrew @ gaul . org > .
2014-07-22 03:25:31 +00:00
References
----------
2015-05-13 18:44:49 +00:00
2017-04-26 18:07:10 +00:00
* [Apache jclouds ](http://jclouds.apache.org/ ) provides storage backend support for S3Proxy
* [Ceph s3-tests ](https://github.com/ceph/s3-tests ) help maintain and improve compatibility with the S3 API
2017-04-26 18:05:26 +00:00
* [fake-s3 ](https://github.com/jubos/fake-s3 ), [gofakes3 ](https://github.com/johannesboyne/gofakes3 ), [S3 ninja ](https://github.com/scireum/s3ninja ), [s3rver ](https://github.com/jamhall/s3rver ), and [Scality S3 Server ](https://s3.scality.com/ ) provide functionality similar to S3Proxy when using the filesystem backend
2015-11-17 00:14:38 +00:00
* [GlacierProxy ](https://github.com/bouncestorage/glacier-proxy ) and [SwiftProxy ](https://github.com/bouncestorage/swiftproxy ) provide similar functionality for the Amazon Glacier and OpenStack Swift APIs
2017-03-17 03:26:13 +00:00
* [minio ](https://github.com/minio/minio ) provides filesystem and Azure storage backends.
2017-04-26 18:07:10 +00:00
* [sbt-s3 ](https://github.com/localytics/sbt-s3 ) runs S3Proxy via the Scala Build Tool
2015-12-09 07:51:21 +00:00
* [swift3 ](https://github.com/openstack/swift3 ) provides an S3 middleware for OpenStack Swift
2014-07-22 03:25:31 +00:00
License
-------
2017-02-16 07:14:47 +00:00
Copyright (C) 2014-2017 Andrew Gaul
2014-07-22 03:25:31 +00:00
Licensed under the Apache License, Version 2.0