558efb7b29 introduced a bug where if anonymous access is
configured, the bucket locator hits a null pointer exception. There is
an additional bug where if an incorrect identity is used with the
blobstore locator (as in, the request identity does not match the
configured bucket's blobstore identity), another null pointer exception
can be triggered.
The patch creates a new GlobBlobStoreLocator class and adds tests to it,
as it is difficult to test Main.java directly. The blobstore locator
logic is also reworked to fix the above bugs.
Allows differentiating the backend blob stores by configured buckets.
The commit allows using the same s3proxy credentials across the
different backends and relies on the bucket names to disambiguate them.
If no buckets are configured, the prior blob store selection algorithm
is used, which returns the first configured blob store for the specified
identity.
The patch supports the glob syntax, which allows specifying groups of
buckets more easily. However, there is no checking for overlapping
globs.
Fixes: #371
Alias middleware implements a way to remap backend buckets to a
configurable front-end name. The mappings are configured as:
s3proxy.alias-blobstore.<s3proxy bucket> = <backend bucket>
A single bucket cannot be mapped to multiple names.
Adds the sharded bucket middleware, which allows for splitting objects
across multiple backend buckets for a given virtual bucket. The
middleware should be configured as:
s3proxy.sharded-blobstore.<bucket name>.shards=<number of shards>
s3proxy.sharded-blobstore.<bucket name>.prefix=<prefix>.
All shards are named <prefix>-<index>, where index is an
integer from 0 to <number of shards> - 1. If the <prefix> is not
supplied, the <bucket name> is used as the prefix.
Listing the virtual bucket and multipart uploads are not supported. When
listing all containers, the shards are elided from the result.
Fixes#325Fixes#351
This prevents OpenStack blobstores from matching regions incorrectly, basing on a value of the endpoint variable. OVH has different URLs for each storage endpoint, which does not match the auth endpoint.
Sometimes the service is not the only one under a domain.
This commit adds s3proxy.service-path to allow s3proxy to be deployed
with a context path. Fixes#48.
This throws away all data during object creation and reconstructs it
with NUL bytes. Clients can use this middleware for performance
testing or to simulate very large objects. Fixes#131.
Providing both credential and the credential property used only the
latter, which was not expanded when credential is a file. Also expand
credential in tests. References #128.
This models eventually-consistent behavior. This implementation uses
two buckets and client writes going to the first bucket and reads to
the second. Operations later replicate from the first to the second
with a variable delays. A more complete implementation could flap
between the strongly- and eventually-consistent buckets and arbitrary
reorder operations which conclude with last-writer wins. Fixes#65.
jclouds has inconsistent support for regions due to provider
differences. AWS providers a single global namespace while Swift
scopes per-endpoint. Tested with Rackspace Cloud Files in HKG and ORD
regions. Fixes#81.
Andrew Gaul