# S3Proxy [![Github All Releases](https://img.shields.io/github/downloads/gaul/s3proxy/total.svg)](https://github.com/gaul/s3proxy/releases/) [![Docker Pulls](https://img.shields.io/docker/pulls/andrewgaul/s3proxy.svg)](https://hub.docker.com/r/andrewgaul/s3proxy/) [![Maven Central](https://img.shields.io/maven-central/v/org.gaul/s3proxy.svg)](https://search.maven.org/#search%7Cga%7C1%7Ca%3A%22s3proxy%22) [![Twitter Follow](https://img.shields.io/twitter/follow/S3Proxy.svg?style=social&label=Follow)](https://twitter.com/S3Proxy) S3Proxy implements the [S3 API](https://en.wikipedia.org/wiki/Amazon_S3#S3_API_and_competing_services) and *proxies* requests, enabling several use cases: * translation from S3 to Backblaze B2, EMC Atmos, Google Cloud, Microsoft Azure, and OpenStack Swift * testing without Amazon by using the local filesystem * extension via middlewares * embedding into Java applications ## Usage with Docker [Docker Hub](https://hub.docker.com/r/andrewgaul/s3proxy/) hosts a Docker image and has instructions on how to run it. ## Usage without Docker Users can [download releases](https://github.com/gaul/s3proxy/releases) from GitHub. Developers can build the project by running `mvn package` which produces a binary at `target/s3proxy`. S3Proxy requires Java 11 or newer to run. Configure S3Proxy via a properties file. An example using the local file system as the storage backend with anonymous access: ``` s3proxy.authorization=none s3proxy.endpoint=http://127.0.0.1:8080 jclouds.provider=filesystem jclouds.filesystem.basedir=/tmp/s3proxy ``` First create the filesystem basedir: ``` mkdir /tmp/s3proxy ``` Next run S3Proxy. Linux and Mac OS X users can run the executable jar: ``` chmod +x s3proxy s3proxy --properties s3proxy.conf ``` Windows users must explicitly invoke java: ``` java -jar s3proxy --properties s3proxy.conf ``` Finally test by creating a bucket then listing all the buckets: ``` $ curl --request PUT http://localhost:8080/testbucket $ curl http://localhost:8080/ 75aa57f09aa0c8caeab4f8c24e99d10f8e7faeebf76c078efc7c6caea54ba06aCustomersName@amazon.comtestbucket2015-08-05T22:16:24.000Z ``` ## Usage with Java Maven Central hosts S3Proxy artifacts and the wiki has [instructions on Java use](https://github.com/gaul/s3proxy/wiki/Using-S3Proxy-in-Java-projects). ## Supported storage backends * atmos * aws-s3 (Amazon-only) * azureblob * azureblob-sdk (newer but lacks multi-part upload, see [Azure/azure-sdk-for-java#42603](https://github.com/Azure/azure-sdk-for-java/issues/42603)) * b2 * filesystem (on-disk storage) * filesystem-nio2 (on-disk storage, preview) * google-cloud-storage * openstack-swift * rackspace-cloudfiles-uk and rackspace-cloudfiles-us * s3 (all implementations) * transient (in-memory storage, deprecated) * transient-nio2 (in-memory storage, recommended) See the wiki for [examples of configurations](https://github.com/gaul/s3proxy/wiki/Storage-backend-examples). ## Assigning buckets to backends S3Proxy can be configured to assign buckets to different backends with the same credentials. The configuration in the properties file is as follows: ``` s3proxy.bucket-locator.1=bucket s3proxy.bucket-locator.2=another-bucket ``` In addition to the explicit names, [glob syntax](https://docs.oracle.com/javase/tutorial/essential/io/fileOps.html#glob) can be used to configure many buckets for a given backend. A bucket (or a glob) cannot be assigned to multiple backends. ## Middlewares S3Proxy can modify its behavior based on middlewares: * [bucket aliasing](https://github.com/gaul/s3proxy/wiki/Middleware-alias-blobstore) * [bucket locator](https://github.com/gaul/s3proxy/wiki/Middleware-bucket-locator) * [eventual consistency modeling](https://github.com/gaul/s3proxy/wiki/Middleware---eventual-consistency) * [large object mocking](https://github.com/gaul/s3proxy/wiki/Middleware-large-object-mocking) * [read-only](https://github.com/gaul/s3proxy/wiki/Middleware-read-only) * [regex rename blobs](https://github.com/gaul/s3proxy/wiki/Middleware-regex) * [sharded backend containers](https://github.com/gaul/s3proxy/wiki/Middleware-sharded-backend) * [storage class override](https://github.com/gaul/s3proxy/wiki/Middleware-storage-class-override) * [user metadata replacer](https://github.com/gaul/s3proxy/wiki/Middleware-user-metadata-replacer) ## SSL Support S3Proxy can listen on HTTPS by setting the `secure-endpoint` and [configuring a keystore](http://wiki.eclipse.org/Jetty/Howto/Configure_SSL#Generating_Keys_and_Certificates_with_JDK_keytool). You can read more about how configure S3Proxy for SSL Support in [the dedicated wiki page](https://github.com/gaul/s3proxy/wiki/SSL-support) with Docker, Kubernetes or simply Java. ## Limitations S3Proxy has broad compatibility with the S3 API, however, it does not support: * ACLs other than private and public-read * BitTorrent hosting * bucket logging * bucket policies * [CORS bucket operations](https://docs.aws.amazon.com/AmazonS3/latest/dev/cors.html#how-do-i-enable-cors) like getting or setting the CORS configuration for a bucket. S3Proxy only supports a static configuration (see below). * hosting static websites * object server-side encryption * object tagging * object versioning, see [#74](https://github.com/gaul/s3proxy/issues/74) * POST upload policies, see [#73](https://github.com/gaul/s3proxy/issues/73) * requester pays buckets * [select object content](https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectSELECTContent.html) S3Proxy emulates the following operations: * copy multi-part objects, see [#76](https://github.com/gaul/s3proxy/issues/76) S3Proxy has basic CORS preflight and actual request/response handling. It can be configured within the properties file (and corresponding ENV variables for Docker): ``` s3proxy.cors-allow-origins=https://example\.com https://.+\.example\.com https://example\.cloud s3proxy.cors-allow-methods=GET PUT s3proxy.cors-allow-headers=Accept Content-Type s3proxy.cors-allow-credential=true ``` CORS cannot be configured per bucket. `s3proxy.cors-allow-all=true` will accept any origin and header. Actual CORS requests are supported for GET, PUT, POST, HEAD and DELETE methods. The wiki collects [compatibility notes](https://github.com/gaul/s3proxy/wiki/Storage-backend-compatibility) for specific storage backends. ## Support * [GitHub issues](https://github.com/gaul/s3proxy/issues) * [Stack Overflow](https://stackoverflow.com/questions/tagged/s3proxy) * [commercial support](mailto:andrew@gaul.org) ## References * [Apache jclouds](https://jclouds.apache.org/) provides storage backend support for S3Proxy * [Ceph s3-tests](https://github.com/ceph/s3-tests) help maintain and improve compatibility with the S3 API * [fake-s3](https://github.com/jubos/fake-s3), [gofakes3](https://github.com/johannesboyne/gofakes3), [minio](https://github.com/minio/minio), [S3 ninja](https://github.com/scireum/s3ninja), and [s3rver](https://github.com/jamhall/s3rver) provide functionality similar to S3Proxy when using the filesystem backend * [GlacierProxy](https://github.com/bouncestorage/glacier-proxy) and [SwiftProxy](https://github.com/bouncestorage/swiftproxy) provide similar functionality for the Amazon Glacier and OpenStack Swift APIs * [s3mock](https://github.com/adobe/S3Mock) - Adobe's s3 mock implementation * [sbt-s3](https://github.com/localytics/sbt-s3) runs S3Proxy via the Scala Build Tool * [swift3](https://github.com/openstack/swift3) provides an S3 middleware for OpenStack Swift * [Zenko](https://www.zenko.io/) provide similar multi-cloud functionality ## License Copyright (C) 2014-2024 Andrew Gaul Licensed under the Apache License, Version 2.0