2014-07-22 03:25:31 +00:00
|
|
|
S3Proxy
|
|
|
|
|
=======
|
2014-08-18 07:17:48 +00:00
|
|
|
S3Proxy allows applications using the
|
|
|
|
|
[S3 API](https://en.wikipedia.org/wiki/Amazon_S3#S3_API_and_competing_services)
|
|
|
|
|
to access other object stores,
|
2014-08-15 19:41:28 +00:00
|
|
|
e.g., EMC Atmos, Microsoft Azure, OpenStack Swift. It also allows local
|
|
|
|
|
testing of S3 without the cost or latency associated with using AWS.
|
2014-07-22 03:25:31 +00:00
|
|
|
|
|
|
|
|
Features
|
|
|
|
|
--------
|
2014-08-12 01:19:26 +00:00
|
|
|
* create, remove, and list buckets (including user-specified regions)
|
|
|
|
|
* put, get, delete, and list objects
|
2014-08-22 04:18:08 +00:00
|
|
|
* copy objects and delete multiple objects (emulated operations)
|
2014-08-12 01:19:26 +00:00
|
|
|
* store and retrieve object metadata, including user metadata
|
2014-08-12 08:30:17 +00:00
|
|
|
* authorization via AWS signature v2 (including pre-signed URLs) or anonymous access
|
2014-07-26 20:19:43 +00:00
|
|
|
* listen on HTTP or HTTPS
|
2014-07-22 03:25:31 +00:00
|
|
|
|
|
|
|
|
Supported object stores:
|
|
|
|
|
|
|
|
|
|
* atmos
|
|
|
|
|
* aws-s3
|
|
|
|
|
* azureblob
|
|
|
|
|
* cloudfiles-uk and cloudfiles-us
|
|
|
|
|
* filesystem (on-disk storage)
|
2014-09-05 02:55:19 +00:00
|
|
|
* google-cloud-storage
|
2014-07-22 03:25:31 +00:00
|
|
|
* hpcloud-objectstorage
|
|
|
|
|
* s3
|
|
|
|
|
* swift and swift-keystone
|
|
|
|
|
* transient (in-memory storage)
|
|
|
|
|
|
|
|
|
|
Installation
|
|
|
|
|
------------
|
2014-08-12 08:30:17 +00:00
|
|
|
Users can [download releases](https://github.com/andrewgaul/s3proxy/releases)
|
|
|
|
|
from GitHub. One can also build the project by running `mvn package` which
|
|
|
|
|
produces a binary at `target/s3proxy`. S3Proxy requires Java 7 to run.
|
2014-07-22 03:25:31 +00:00
|
|
|
|
|
|
|
|
Examples
|
|
|
|
|
--------
|
2014-08-15 19:41:28 +00:00
|
|
|
Linux and Mac OS X users can run S3Proxy via the executable jar:
|
2014-07-22 03:25:31 +00:00
|
|
|
|
|
|
|
|
```
|
2014-08-12 08:30:17 +00:00
|
|
|
chmod +x s3proxy
|
2014-07-22 03:25:31 +00:00
|
|
|
s3proxy --properties s3proxy.conf
|
2014-08-12 08:30:17 +00:00
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Windows users must explicitly invoke java:
|
|
|
|
|
|
|
|
|
|
```
|
2014-07-22 03:25:31 +00:00
|
|
|
java -jar s3proxy --properties s3proxy.conf
|
|
|
|
|
```
|
|
|
|
|
|
2014-08-12 08:30:17 +00:00
|
|
|
Users can configure S3Proxy via a properties file. An example using Rackspace
|
|
|
|
|
CloudFiles (based on OpenStack Swift) as the backing store:
|
2014-07-22 03:25:31 +00:00
|
|
|
|
2014-08-12 08:30:17 +00:00
|
|
|
```
|
|
|
|
|
s3proxy.endpoint=http://127.0.0.1:8080
|
|
|
|
|
s3proxy.authorization=aws-v2
|
|
|
|
|
s3proxy.identity=local-identity
|
|
|
|
|
s3proxy.credential=local-credential
|
|
|
|
|
jclouds.provider=cloudfiles-us
|
|
|
|
|
jclouds.identity=remote-identity
|
|
|
|
|
jclouds.credential=remote-credential
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Another example using the local file system as the backing store with anonymous
|
|
|
|
|
access:
|
2014-07-22 03:25:31 +00:00
|
|
|
|
|
|
|
|
```
|
2014-08-12 08:30:17 +00:00
|
|
|
s3proxy.authorization=none
|
|
|
|
|
s3proxy.endpoint=http://127.0.0.1:8080
|
|
|
|
|
jclouds.provider=filesystem
|
2014-07-22 03:25:31 +00:00
|
|
|
jclouds.identity=identity
|
|
|
|
|
jclouds.credential=credential
|
2014-08-12 08:30:17 +00:00
|
|
|
jclouds.filesystem.basedir=/tmp
|
2014-07-22 03:25:31 +00:00
|
|
|
```
|
|
|
|
|
|
2014-08-18 07:17:48 +00:00
|
|
|
S3Proxy can listen on HTTPS by setting the endpoint and
|
2014-07-26 20:19:43 +00:00
|
|
|
[configuring a keystore](http://wiki.eclipse.org/Jetty/Howto/Configure_SSL#Generating_Keys_and_Certificates_with_JDK_keytool).
|
|
|
|
|
An example:
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
s3proxy.endpoint=https://127.0.0.1:8080
|
|
|
|
|
s3proxy.keystore-path=keystore.jks
|
|
|
|
|
s3proxy.keystore-password=password
|
|
|
|
|
```
|
|
|
|
|
|
2014-08-18 06:37:10 +00:00
|
|
|
Users can also set other Java,
|
|
|
|
|
[jclouds](https://github.com/jclouds/jclouds/blob/master/core/src/main/java/org/jclouds/Constants.java),
|
|
|
|
|
and [S3Proxy](https://github.com/andrewgaul/s3proxy/blob/master/src/main/java/org/gaul/s3proxy/S3ProxyConstants.java)
|
|
|
|
|
properties.
|
2014-07-22 03:25:31 +00:00
|
|
|
|
|
|
|
|
Limitations
|
|
|
|
|
-----------
|
|
|
|
|
S3Proxy does not support:
|
|
|
|
|
|
2014-08-12 08:30:17 +00:00
|
|
|
* single-part uploads larger than 2 GB ([jclouds issue](https://issues.apache.org/jira/browse/JCLOUDS-264))
|
2014-07-22 03:25:31 +00:00
|
|
|
* multi-part uploads
|
2014-08-09 23:12:45 +00:00
|
|
|
* POST uploads
|
2014-08-12 08:30:17 +00:00
|
|
|
* bucket and object ACLs ([jclouds issue](https://issues.apache.org/jira/browse/JCLOUDS-660))
|
|
|
|
|
* object metadata with filesystem provider ([jclouds issue](https://issues.apache.org/jira/browse/JCLOUDS-658))
|
|
|
|
|
* object versioning
|
2014-07-22 03:25:31 +00:00
|
|
|
|
|
|
|
|
References
|
|
|
|
|
----------
|
2014-08-09 23:12:45 +00:00
|
|
|
Apache [jclouds](http://jclouds.apache.org/) provides object store support for
|
2014-07-29 23:31:36 +00:00
|
|
|
S3Proxy. Ceph [s3-tests](https://github.com/ceph/s3-tests) help maintain
|
|
|
|
|
and improve compatibility with the S3 API.
|
2014-07-22 03:25:31 +00:00
|
|
|
|
|
|
|
|
License
|
|
|
|
|
-------
|
|
|
|
|
Copyright (C) 2014 Andrew Gaul
|
|
|
|
|
|
|
|
|
|
Licensed under the Apache License, Version 2.0
|
