s3proxy/README.md

# S3Proxy

[![Github All Releases](https://img.shields.io/github/downloads/gaul/s3proxy/total.svg)](https://github.com/gaul/s3proxy/releases/)
[![Docker Pulls](https://img.shields.io/docker/pulls/andrewgaul/s3proxy.svg)](https://hub.docker.com/r/andrewgaul/s3proxy/)
[![Maven Central](https://img.shields.io/maven-central/v/org.gaul/s3proxy.svg)](https://search.maven.org/#search%7Cga%7C1%7Ca%3A%22s3proxy%22)
[![Twitter Follow](https://img.shields.io/twitter/follow/S3Proxy.svg?style=social&label=Follow)](https://twitter.com/S3Proxy)

S3Proxy implements the
[S3 API](https://en.wikipedia.org/wiki/Amazon_S3#S3_API_and_competing_services)
and *proxies* requests, enabling several use cases:

* translation from S3 to Backblaze B2, EMC Atmos, Google Cloud, Microsoft Azure, and OpenStack Swift
* testing without Amazon by using the local filesystem
* extension via middlewares
* embedding into Java applications

## Usage with Docker

[Docker Hub](https://hub.docker.com/r/andrewgaul/s3proxy/) hosts a Docker image
and has instructions on how to run it.

## Usage without Docker

Users can [download releases](https://github.com/gaul/s3proxy/releases)
from GitHub.  Developers can build the project by running `mvn package` which
produces a binary at `target/s3proxy`.  S3Proxy requires Java 7 to run.

Configure S3Proxy via a properties file.  An example using the local
file system as the storage backend with anonymous access:

```
s3proxy.authorization=none
s3proxy.endpoint=http://127.0.0.1:8080
jclouds.provider=filesystem
jclouds.filesystem.basedir=/tmp/s3proxy
```

First create the filesystem basedir:

```
mkdir /tmp/s3proxy
```

Next run S3Proxy.  Linux and Mac OS X users can run the executable jar:

```
chmod +x s3proxy
s3proxy --properties s3proxy.conf
```

Windows users must explicitly invoke java:

```
java -jar s3proxy --properties s3proxy.conf
```

Finally test by creating a bucket then listing all the buckets:

```
$ curl --request PUT http://localhost:8080/testbucket

$ curl http://localhost:8080/
<?xml version="1.0" ?><ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>75aa57f09aa0c8caeab4f8c24e99d10f8e7faeebf76c078efc7c6caea54ba06a</ID><DisplayName>CustomersName@amazon.com</DisplayName></Owner><Buckets><Bucket><Name>testbucket</Name><CreationDate>2015-08-05T22:16:24.000Z</CreationDate></Bucket></Buckets></ListAllMyBucketsResult>
```

## Usage with Java

Maven Central hosts S3Proxy artifacts and the wiki has
[instructions on Java use](https://github.com/gaul/s3proxy/wiki/Using-S3Proxy-in-Java-projects).

## Supported storage backends

* atmos
* aws-s3 (Amazon-only)
* azureblob
* b2
* filesystem (on-disk storage)
* google-cloud-storage
* openstack-swift
* rackspace-cloudfiles-uk and rackspace-cloudfiles-us
* s3 (all implementations)
* transient (in-memory storage)

See the wiki for [examples of configurations](https://github.com/gaul/s3proxy/wiki/Storage-backend-examples).

## Middlewares

S3Proxy can modify its behavior based on middlewares:

* [eventual consistency modeling](https://github.com/gaul/s3proxy/wiki/Middleware---eventual-consistency)
* [large object mocking](https://github.com/gaul/s3proxy/wiki/Middleware-large-object-mocking)
* [read-only](https://github.com/gaul/s3proxy/wiki/Middleware-read-only)

## Limitations

S3Proxy has broad compatibility with the S3 API, however, it does not support:

* ACLs other than private and public-read
* BitTorrent hosting
* bucket logging
* bucket policies
* [CORS bucket operations](https://docs.aws.amazon.com/AmazonS3/latest/dev/cors.html#how-do-i-enable-cors) like getting or setting the CORS configuration for a bucket. S3Proxy only supports a static configuration (see below).
* hosting static websites
* object server-side encryption
* object tagging
* object versioning, see [#74](https://github.com/gaul/s3proxy/issues/74)
* POST upload policies, see [#73](https://github.com/gaul/s3proxy/issues/73)
* requester pays buckets
* [select object content](https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectSELECTContent.html)

S3Proxy emulates the following operations:

* copy multi-part objects, see [#76](https://github.com/gaul/s3proxy/issues/76)

S3Proxy has basic CORS preflight and actual request/response handling. It can be configured within the properties
file (and corresponding ENV variables for Docker):

```
s3proxy.cors-allow-origins=https://example\.com https://.+\.example\.com https://example\.cloud
s3proxy.cors-allow-methods=GET PUT
s3proxy.cors-allow-headers=Accept Content-Type
```

CORS cannot be configured per bucket. `s3proxy.cors-allow-all=true` will accept any origin and header.
Actual CORS requests are supported for GET, PUT and POST methods.

The wiki collects
[compatibility notes](https://github.com/gaul/s3proxy/wiki/Storage-backend-compatibility)
for specific storage backends.

## Support

* [GitHub issues](https://github.com/gaul/s3proxy/issues)
* [Stack Overflow](https://stackoverflow.com/questions/tagged/s3proxy)
* [commercial support](mailto:andrew@gaul.org)

## References

* [Apache jclouds](http://jclouds.apache.org/) provides storage backend support for S3Proxy
* [Ceph s3-tests](https://github.com/ceph/s3-tests) help maintain and improve compatibility with the S3 API
* [fake-s3](https://github.com/jubos/fake-s3), [gofakes3](https://github.com/johannesboyne/gofakes3), [S3 ninja](https://github.com/scireum/s3ninja), and [s3rver](https://github.com/jamhall/s3rver) provide functionality similar to S3Proxy when using the filesystem backend
* [GlacierProxy](https://github.com/bouncestorage/glacier-proxy) and [SwiftProxy](https://github.com/bouncestorage/swiftproxy) provide similar functionality for the Amazon Glacier and OpenStack Swift APIs
* [minio](https://github.com/minio/minio) and [Zenko](https://www.zenko.io/) provide similar multi-cloud functionality
* [s3mock](https://github.com/findify/s3mock) mocks the S3 API for Java/Scala projects
* [sbt-s3](https://github.com/localytics/sbt-s3) runs S3Proxy via the Scala Build Tool
* [swift3](https://github.com/openstack/swift3) provides an S3 middleware for OpenStack Swift

## License

Copyright (C) 2014-2020 Andrew Gaul

Licensed under the Apache License, Version 2.0
Prefer modern markdown heading style 2020-04-29 05:43:39 +00:00			`# S3Proxy`
Add badges to README 2016-11-17 03:25:44 +00:00
Update andrewgaul references to gaul 2017-10-06 17:35:44 +00:00			`[![Github All Releases](https://img.shields.io/github/downloads/gaul/s3proxy/total.svg)](https://github.com/gaul/s3proxy/releases/)`
Add badges to README 2016-11-17 03:25:44 +00:00			`[![Docker Pulls](https://img.shields.io/docker/pulls/andrewgaul/s3proxy.svg)](https://hub.docker.com/r/andrewgaul/s3proxy/)`
			`[![Maven Central](https://img.shields.io/maven-central/v/org.gaul/s3proxy.svg)](https://search.maven.org/#search%7Cga%7C1%7Ca%3A%22s3proxy%22)`
			`[![Twitter Follow](https://img.shields.io/twitter/follow/S3Proxy.svg?style=social&label=Follow)](https://twitter.com/S3Proxy)`

Reword README summary 2016-11-17 03:48:18 +00:00			`S3Proxy implements the`
Update README 2014-08-18 07:17:48 +00:00			`[S3 API](https://en.wikipedia.org/wiki/Amazon_S3#S3_API_and_competing_services)`
Reword README summary 2016-11-17 03:48:18 +00:00			`and proxies requests, enabling several use cases:`

Add EMC Atmos to translation bullet Also use and instead of or. 2017-11-04 17:11:49 +00:00			`* translation from S3 to Backblaze B2, EMC Atmos, Google Cloud, Microsoft Azure, and OpenStack Swift`
Reword README summary 2016-11-17 03:48:18 +00:00			`* testing without Amazon by using the local filesystem`
Add section about middlewares linking to wiki 2017-03-09 18:53:08 +00:00			`* extension via middlewares`
Reword README summary 2016-11-17 03:48:18 +00:00			`* embedding into Java applications`
Initial commit of S3Proxy 2014-07-22 03:25:31 +00:00
Prefer modern markdown heading style 2020-04-29 05:43:39 +00:00			`## Usage with Docker`

Add Docker usage 2016-07-02 07:34:12 +00:00			`[Docker Hub](https://hub.docker.com/r/andrewgaul/s3proxy/) hosts a Docker image`
			`and has instructions on how to run it.`

Prefer modern markdown heading style 2020-04-29 05:43:39 +00:00			`## Usage without Docker`

Update andrewgaul references to gaul 2017-10-06 17:35:44 +00:00			`Users can [download releases](https://github.com/gaul/s3proxy/releases)`
Consistently refer to storage backends 2015-12-17 19:03:22 +00:00			from GitHub. Developers can build the project by running `mvn package` which
Add more features and limitations to README Also use more accurate links for upstream issues. 2014-08-12 08:30:17 +00:00			produces a binary at `target/s3proxy`. S3Proxy requires Java 7 to run.
Initial commit of S3Proxy 2014-07-22 03:25:31 +00:00
Simplify usage and add curl example Link to wiki for more complicated examples. 2015-11-17 00:14:38 +00:00			`Configure S3Proxy via a properties file. An example using the local`
			`file system as the storage backend with anonymous access:`
Initial commit of S3Proxy 2014-07-22 03:25:31 +00:00
			```
Add more features and limitations to README Also use more accurate links for upstream issues. 2014-08-12 08:30:17 +00:00			`s3proxy.authorization=none`
			`s3proxy.endpoint=http://127.0.0.1:8080`
			`jclouds.provider=filesystem`
Simplify usage and add curl example Link to wiki for more complicated examples. 2015-11-17 00:14:38 +00:00			`jclouds.filesystem.basedir=/tmp/s3proxy`
Initial commit of S3Proxy 2014-07-22 03:25:31 +00:00			```

Simplify usage and add curl example Link to wiki for more complicated examples. 2015-11-17 00:14:38 +00:00			`First create the filesystem basedir:`
Add support for listening on HTTPS Fixes #3. 2014-07-26 20:19:43 +00:00
			```
Simplify usage and add curl example Link to wiki for more complicated examples. 2015-11-17 00:14:38 +00:00			`mkdir /tmp/s3proxy`
Add support for listening on HTTPS Fixes #3. 2014-07-26 20:19:43 +00:00			```

Simplify usage and add curl example Link to wiki for more complicated examples. 2015-11-17 00:14:38 +00:00			`Next run S3Proxy. Linux and Mac OS X users can run the executable jar:`
breakup the line for readability 2015-05-09 01:13:44 +00:00
			```
Simplify usage and add curl example Link to wiki for more complicated examples. 2015-11-17 00:14:38 +00:00			`chmod +x s3proxy`
			`s3proxy --properties s3proxy.conf`
breakup the line for readability 2015-05-09 01:13:44 +00:00			```

Simplify usage and add curl example Link to wiki for more complicated examples. 2015-11-17 00:14:38 +00:00			`Windows users must explicitly invoke java:`
add a blurb about SSL setup 2015-05-09 01:12:26 +00:00
			```
Simplify usage and add curl example Link to wiki for more complicated examples. 2015-11-17 00:14:38 +00:00			`java -jar s3proxy --properties s3proxy.conf`
			```

			`Finally test by creating a bucket then listing all the buckets:`

			```
			`$ curl --request PUT http://localhost:8080/testbucket`

			`$ curl http://localhost:8080/`
			`<?xml version="1.0" ?><ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>75aa57f09aa0c8caeab4f8c24e99d10f8e7faeebf76c078efc7c6caea54ba06a</ID><DisplayName>CustomersName@amazon.com</DisplayName></Owner><Buckets><Bucket><Name>testbucket</Name><CreationDate>2015-08-05T22:16:24.000Z</CreationDate></Bucket></Buckets></ListAllMyBucketsResult>`
add a blurb about SSL setup 2015-05-09 01:12:26 +00:00			```

Prefer modern markdown heading style 2020-04-29 05:43:39 +00:00			`## Usage with Java`
Add usage with Java 2016-11-23 18:22:48 +00:00
			`Maven Central hosts S3Proxy artifacts and the wiki has`
Update andrewgaul references to gaul 2017-10-06 17:35:44 +00:00			`[instructions on Java use](https://github.com/gaul/s3proxy/wiki/Using-S3Proxy-in-Java-projects).`
Add usage with Java 2016-11-23 18:22:48 +00:00
Prefer modern markdown heading style 2020-04-29 05:43:39 +00:00			`## Supported storage backends`
Clarify aws-s3 vs. s3 storage backend The former is Amazon-only. 2017-03-17 03:27:29 +00:00
Rewrite summary and rework features Users should assume these features work; instead document the limitations. 2015-11-11 01:12:16 +00:00			`* atmos`
Clarify aws-s3 vs. s3 storage backend The former is Amazon-only. 2017-03-17 03:27:29 +00:00			`* aws-s3 (Amazon-only)`
Rewrite summary and rework features Users should assume these features work; instead document the limitations. 2015-11-11 01:12:16 +00:00			`* azureblob`
Update README for 1.5.0 release 2015-04-09 01:29:41 +00:00			`* b2`
Rewrite summary and rework features Users should assume these features work; instead document the limitations. 2015-11-11 01:12:16 +00:00			`* filesystem (on-disk storage)`
			`* google-cloud-storage`
			`* openstack-swift`
			`* rackspace-cloudfiles-uk and rackspace-cloudfiles-us`
Clarify aws-s3 vs. s3 storage backend The former is Amazon-only. 2017-03-17 03:27:29 +00:00			`* s3 (all implementations)`
Rewrite summary and rework features Users should assume these features work; instead document the limitations. 2015-11-11 01:12:16 +00:00			`* transient (in-memory storage)`

Update andrewgaul references to gaul 2017-10-06 17:35:44 +00:00			`See the wiki for [examples of configurations](https://github.com/gaul/s3proxy/wiki/Storage-backend-examples).`
Give storage examples a more specific wiki link 2016-07-07 11:21:35 +00:00
Prefer modern markdown heading style 2020-04-29 05:43:39 +00:00			`## Middlewares`
Add section about middlewares linking to wiki 2017-03-09 18:53:08 +00:00
			`S3Proxy can modify its behavior based on middlewares:`

Update andrewgaul references to gaul 2017-10-06 17:35:44 +00:00			`* [eventual consistency modeling](https://github.com/gaul/s3proxy/wiki/Middleware---eventual-consistency)`
			`* [large object mocking](https://github.com/gaul/s3proxy/wiki/Middleware-large-object-mocking)`
Read-only middleware Fixes #217. 2017-10-09 01:06:07 +00:00			`* [read-only](https://github.com/gaul/s3proxy/wiki/Middleware-read-only)`
Add section about middlewares linking to wiki 2017-03-09 18:53:08 +00:00
Prefer modern markdown heading style 2020-04-29 05:43:39 +00:00			`## Limitations`
Initial commit of S3Proxy 2014-07-22 03:25:31 +00:00
Reword limitations 2016-02-05 08:19:44 +00:00			`S3Proxy has broad compatibility with the S3 API, however, it does not support:`
Update limitations 2016-01-21 20:13:01 +00:00
			`* ACLs other than private and public-read`
Add more limitations Also update wiki link. 2016-01-08 23:30:56 +00:00			`* BitTorrent hosting`
			`* bucket logging`
Add bucket policies to limitations References #301. 2019-05-21 02:45:04 +00:00			`* bucket policies`
Basic CORS support This commit adds a globally configurable CORS support. Note that this differs from AWS per-bucket support. 2018-12-21 22:19:50 +00:00			`* [CORS bucket operations](https://docs.aws.amazon.com/AmazonS3/latest/dev/cors.html#how-do-i-enable-cors) like getting or setting the CORS configuration for a bucket. S3Proxy only supports a static configuration (see below).`
Expand limitations and references 2017-05-17 18:22:46 +00:00			`* hosting static websites`
Add references to fake-s3 and abustany s3proxy 2014-08-28 18:41:56 +00:00			`* object server-side encryption`
Add object tagging to limitations 2016-12-05 10:32:10 +00:00			`* object tagging`
Update andrewgaul references to gaul 2017-10-06 17:35:44 +00:00			`* object versioning, see [#74](https://github.com/gaul/s3proxy/issues/74)`
Alphabetize limitations 2018-01-19 19:27:23 +00:00			`* POST upload policies, see [#73](https://github.com/gaul/s3proxy/issues/73)`
Add more limitations Also update wiki link. 2016-01-08 23:30:56 +00:00			`* requester pays buckets`
Add select object content to limitations 2018-01-03 01:33:14 +00:00			`* [select object content](https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectSELECTContent.html)`
Initial commit of S3Proxy 2014-07-22 03:25:31 +00:00
Rewrite summary and rework features Users should assume these features work; instead document the limitations. 2015-11-11 01:12:16 +00:00			`S3Proxy emulates the following operations:`

Update andrewgaul references to gaul 2017-10-06 17:35:44 +00:00			`* copy multi-part objects, see [#76](https://github.com/gaul/s3proxy/issues/76)`
Rewrite summary and rework features Users should assume these features work; instead document the limitations. 2015-11-11 01:12:16 +00:00
Basic CORS support This commit adds a globally configurable CORS support. Note that this differs from AWS per-bucket support. 2018-12-21 22:19:50 +00:00			`S3Proxy has basic CORS preflight and actual request/response handling. It can be configured within the properties`
			`file (and corresponding ENV variables for Docker):`

			```
			`s3proxy.cors-allow-origins=https://example\.com https://.+\.example\.com https://example\.cloud`
			`s3proxy.cors-allow-methods=GET PUT`
			`s3proxy.cors-allow-headers=Accept Content-Type`
			```

			CORS cannot be configured per bucket. `s3proxy.cors-allow-all=true` will accept any origin and header.
			`Actual CORS requests are supported for GET, PUT and POST methods.`

Update limitations 2016-01-21 20:13:01 +00:00			`The wiki collects`
Correct typos Found via codespell. 2018-02-08 04:20:28 +00:00			`[compatibility notes](https://github.com/gaul/s3proxy/wiki/Storage-backend-compatibility)`
Update limitations 2016-01-21 20:13:01 +00:00			`for specific storage backends.`
Add compatibility notes 2015-11-17 01:45:50 +00:00
Prefer modern markdown heading style 2020-04-29 05:43:39 +00:00			`## Support`
Add support heading 2017-04-12 20:20:49 +00:00
Reorganize support heading 2020-04-29 05:48:23 +00:00			`* [GitHub issues](https://github.com/gaul/s3proxy/issues)`
			`* [Stack Overflow](https://stackoverflow.com/questions/tagged/s3proxy)`
			`* [commercial support](mailto:andrew@gaul.org)`
Add support heading 2017-04-12 20:20:49 +00:00
Prefer modern markdown heading style 2020-04-29 05:43:39 +00:00			`## References`
Format references with bullet points 2015-05-13 18:44:49 +00:00
Fix references nits 2017-04-26 18:07:10 +00:00			`* [Apache jclouds](http://jclouds.apache.org/) provides storage backend support for S3Proxy`
			`* [Ceph s3-tests](https://github.com/ceph/s3-tests) help maintain and improve compatibility with the S3 API`
Rename Scality to Zenko and reparent reference 2018-04-17 04:25:14 +00:00			`* [fake-s3](https://github.com/jubos/fake-s3), [gofakes3](https://github.com/johannesboyne/gofakes3), [S3 ninja](https://github.com/scireum/s3ninja), and [s3rver](https://github.com/jamhall/s3rver) provide functionality similar to S3Proxy when using the filesystem backend`
Simplify usage and add curl example Link to wiki for more complicated examples. 2015-11-17 00:14:38 +00:00			`* [GlacierProxy](https://github.com/bouncestorage/glacier-proxy) and [SwiftProxy](https://github.com/bouncestorage/swiftproxy) provide similar functionality for the Amazon Glacier and OpenStack Swift APIs`
Rename Scality to Zenko and reparent reference 2018-04-17 04:25:14 +00:00			`* [minio](https://github.com/minio/minio) and [Zenko](https://www.zenko.io/) provide similar multi-cloud functionality`
Expand limitations and references 2017-05-17 18:22:46 +00:00			`* [s3mock](https://github.com/findify/s3mock) mocks the S3 API for Java/Scala projects`
Fix references nits 2017-04-26 18:07:10 +00:00			`* [sbt-s3](https://github.com/localytics/sbt-s3) runs S3Proxy via the Scala Build Tool`
Add swift3 to references 2015-12-09 07:51:21 +00:00			`* [swift3](https://github.com/openstack/swift3) provides an S3 middleware for OpenStack Swift`
Initial commit of S3Proxy 2014-07-22 03:25:31 +00:00
Prefer modern markdown heading style 2020-04-29 05:43:39 +00:00			`## License`

Update copyright year 2020-05-02 10:50:19 +00:00			`Copyright (C) 2014-2020 Andrew Gaul`
Initial commit of S3Proxy 2014-07-22 03:25:31 +00:00
			`Licensed under the Apache License, Version 2.0`