mirror
/
s3proxy
kopia lustrzana https://github.com/gaul/s3proxy
1
0
Forkuj 0
Kod Zgłoszenia Packages Projekty Wydania Wiki Aktywność
694 Commity
12 Gałęzie
23 Tagi
3,7 MiB
Wykres commitów

369 Commity (45e4d6348f189a316748b74ee2c7effaf29c8e00)

Autor SHA1 Wiadomość Data
Andrew Gaul 72138e1278 Handle null AccessControlPolicy.aclList 
Also add toString methods.
 2016-06-06 17:01:48 -07:00
Andrew Gaul 519cd9313c Remove stub blob during multi-part upload 
References #135.
 2016-06-02 07:42:16 -07:00
Andrew Gaul 2eaf7de68f Handle null ETag during multipart upload 
Fixes #135.
 2016-06-01 12:51:41 -07:00
Andrew Gaul d6c81c9b64 Correct issues with encoding of plus character 
Also remove some questionable decoding.
 2016-06-01 11:22:16 -07:00
Andrew Gaul e7bb6ddb47 Avoid NullPointerException when quoting ETags 2016-05-27 18:07:41 -07:00
Andrew Gaul a4b96924eb Skip time-limited signed requests 
jclouds aws-s3 provider supports these but s3 does not.
 2016-05-12 12:06:03 -07:00
Andrew Gaul 2296b8228b Add configuration to ignore unknown headers 
Fixes #130.
 2016-03-12 23:15:45 -07:00
Andrew Gaul d878642b5d Add quirks for google-cloud-storage 2016-03-09 13:35:02 -08:00
Andrew Gaul eece05f02e Fix path-based GCS credentials 
Providing both credential and the credential property used only the
latter, which was not expanded when credential is a file.  Also expand
credential in tests.  References #128.
 2016-03-09 13:35:02 -08:00
Andrew Gaul 6292d28c9c Add EventualBlobStore 
This models eventually-consistent behavior.  This implementation uses
two buckets and client writes going to the first bucket and reads to
the second.  Operations later replicate from the first to the second
with a variable delays.  A more complete implementation could flap
between the strongly- and eventually-consistent buckets and arbitrary
reorder operations which conclude with last-writer wins.  Fixes #65.
 2016-03-01 17:13:39 -08:00
Andrew Gaul 0351ee73dc Expand GCS credential files into PEMs 2016-02-26 17:45:33 -08:00
Andrew Gaul e670537822 Handle Swift quirks in jclouds tests 2016-02-20 00:49:16 -08:00
Andrew Gaul 2062aadfeb Handle conditional HEAD object requests 2016-02-18 11:48:48 -08:00
Andrew Gaul 52ba2c843f Add support for conditional copy object 
Fixes #113.
 2016-02-16 20:31:12 -08:00
Andrew Gaul d66b539f2c Reduce visibility of methods 2016-02-16 16:16:51 -08:00
Andrew Gaul a4385daa45 Fix v4 signing of queries with special characters 
Reference:

https://docs.aws.amazon.com/general/latest/gr/sigv4-create-canonical-request.html
 2016-02-09 11:52:04 -08:00
Andrew Gaul 9c3d9409ac Read from PushBackInputStream not InputStream 
Previously S3Proxy truncated create bucket requests.
 2016-02-08 22:54:39 -08:00
Andrew Gaul 7198c9088c Do not close InputStream for each uploaded subpart 2016-02-04 00:09:41 -08:00
Andrew Gaul 997dc9707c Skip expires tests when backend lacks support 2016-02-03 23:12:26 -08:00
Andrew Gaul 6d7a46296e Skip zero keys tests when backend lacks support 2016-02-01 12:29:43 -08:00
Andrew Gaul 1ae596e870 Support list multipart upload parts with Azure 
Map the Azure subparts back into S3 parts.
 2016-01-31 13:57:52 -08:00
Andrew Gaul cea4a42922 Use decoded content length when provided 
Previously S3Proxy provided an incorrect Content-Length header when
using V4 signing which providers rejected.
 2016-01-30 19:13:04 -08:00
Andrew Gaul e292ebaccb Consistently parse XML with Jackson 2016-01-29 17:50:22 -08:00
Andrew Gaul bc1833ed2e Remove BlobAccess TODO 
jclouds cannot provide BlobAccess since providers do not return these
ACLs.
 2016-01-26 19:28:50 -08:00
Andrew Gaul f202207c63 Map a subset of XML ACLs to canned ACLs 
Fixes #116.
 2016-01-23 15:34:03 -08:00
Andrew Gaul 0e3a7ef843 Add missing EmailAddress field 
This allows S3Proxy to return the correct NotImplemented error code
instead of InternalError.  References #116.
 2016-01-23 15:16:38 -08:00
Andrew Gaul d557af774d Replace query parameters in anonymous requests 2016-01-23 15:06:04 -08:00
Andrew Gaul 19e2426220 Map a subset of XML ACLs to canned ACLs 
Fixes #116.
 2016-01-21 18:24:57 -08:00
Andrew Gaul 587a4a6670 Centralize blobstore quirks 2016-01-17 23:10:28 -08:00
Andrew Gaul 744d126331 Add support for Cache-Control header 
Fixes #115.
 2016-01-17 12:08:52 -08:00
Andrew Gaul 542c9b8112 Add Accept-Ranges header to partial get requests 
Fixes #66.
 2016-01-15 22:37:19 -08:00
Andrew Gaul 05bd2c1158 Use uploadId for stub blob name 
This makes MPU overwrites of an existing blob atomic.  Fixes #108.
 2016-01-14 15:11:03 -08:00
Andrew Gaul 9c5499a05e Add JcloudsS3ClientLiveTest 
Uncovers several missing features:

Cache-Control header
conditional copies
public-read-write canned ACLs
XML ACLs

References #62.
 2016-01-12 16:20:02 -08:00
Andrew Gaul 91429586dc Reuse encodingType local 2016-01-11 20:01:50 -08:00
Andrew Gaul 3c8f68feb2 Update copyright year to 2016 2016-01-09 21:14:59 -08:00
Andrew Gaul f86bd162ea Call startsWithIgnoreCase helper 2016-01-09 21:03:46 -08:00
Andrew Gaul a5c9f28e0a Make V4_MAX_NON_CHUNKED_REQUEST_SIZE configurable 
Also default to 32 MB since goofys via aws-sdk-go uses 8 MB requests.
 2016-01-09 20:33:28 -08:00
Andrew Gaul 9c25dc557c Fix off-by-one in V4_MAX_NON_CHUNKED_REQUEST_SIZE 2016-01-09 19:22:28 -08:00
Andrew Gaul 3a042ff6e8 Mask high bits in ChunkedInputStream.read 
Previously read could erroneously return -1 end of stream when
returning ASCII 128.
 2016-01-09 18:48:26 -08:00
Andrew Gaul 9976f3c02f Consume trailing newline in ChunkedInputStream 
References #24.
 2016-01-09 18:48:15 -08:00
Andrew Gaul cc516e7ede Improve ChunkedInputStream exception message 2016-01-09 17:21:47 -08:00
Andrew Gaul 00dc2a94e0 Use BaseEncoding.lowerCase where possible 
This avoid extra allocations from String.toLowerCase.
 2016-01-09 17:06:38 -08:00
Andrew Gaul 5c3333d4d9 Parse chunk length in hexadecimal 
Resolves an issue when uploading large objects and parts.
References #24.
 2016-01-09 16:56:12 -08:00
Andrew Gaul e9baeb9c76 Always re-encode parameters 
Assume UTF-8 encoding if not provided.  Reference #24.
 2016-01-09 14:37:32 -08:00
Andrew Gaul c0d7cce353 Fix reversed arguments to String.indexOf 
Found via goofys tests.  References #24.
 2016-01-09 13:06:12 -08:00
Timur Alperovich 0f872c0a2c Support AWS v4 signatures 
Fixes #24.
 2016-01-08 15:31:06 -08:00
Andrew Gaul f24b5647fa Provide requests with explicit payload 
Some requests require payload transformations, e.g., chunked encoding,
so explicitly provide the payload instead of using the request
payload.  References #24.
 2016-01-07 14:02:20 -08:00
Andrew Gaul e93529bfeb Use native putBlob ACLs 2015-12-23 12:56:28 -08:00
Andrew Gaul 118b656f8e Allow anonymous HEAD object on public-read objects 2015-12-22 18:41:15 -08:00
Andrew Gaul 024c7a3fe9 Encode list blobs HTTP responses in UTF-8 
Previously responses incorrectly encoded in ISO-8859-1.
 2015-12-22 18:31:32 -08:00
Andrew Gaul ee55828375 Emit NotImplemented when setting XML ACLs 2015-12-16 19:19:44 -08:00
Derrek Leute d8c73bc86d URL encode blob names when client requests it 
This allows support for characters which XML 1.0 cannot represent and
improves compatibility with newer versions of the AWS CLI.
Fixes #104.
 2015-12-14 17:10:45 -08:00
Andrew Gaul 23cff3a61d Do not require credentials for local blobstores 2015-12-14 16:01:35 -08:00
Andrew Gaul 9e033c677f Require credentials for aws-v2 authentication 2015-12-14 15:48:19 -08:00
Andrew Gaul 177d9b7566 Handle possibly-null HttpResponse 2015-12-13 23:43:22 -08:00
Andrew Gaul c45dde3cbe Skip testMaximumMultipartUpload with large sizes 2015-12-12 13:00:48 +08:00
Andrew Gaul 9210cd679e Map IllegalArgumentException to HTTP 400 2015-11-20 22:31:06 -08:00
Andrew Gaul 67839acef1 Map UnsupportedOperationException to HTTP 501 2015-11-20 22:09:24 -08:00
Andrew Gaul 22ed1c542c Map KeyNotFoundException to NoSuchKey 
Unfortunately jclouds inconsistently throws ContainerNotFoundException
and KeyNotFoundException.
 2015-11-20 22:09:24 -08:00
Andrew Gaul bc2b18ee2b Add support for conditional get 
Fixes #77.
 2015-11-19 18:34:56 -08:00
Rita Zhang 71e2d61377 Allow anonymous access mode to use private buckets 
Fixes #94.
 2015-11-16 21:20:22 -08:00
Andrew Gaul 2207dfbd21 Basic support for POST uploads 
This commit lacks support for policies.  References #73.
 2015-11-14 23:22:05 -08:00
Andrew Gaul 1797992e02 Do not emit response body for HEAD requests 2015-11-11 16:03:42 -08:00
Andrew Gaul 22d57b1f18 Remove redundant private modifier from enum 2015-10-26 13:11:14 -07:00
Andrew Gaul 97d467a698 Enforce no part number during complete MPU 
References #80.
 2015-10-20 14:41:42 -07:00
Andrew Gaul 274bf8a32e Handle listing a public-read container 
Also handle checking existence.
 2015-10-15 20:14:41 -07:00
Andrew Gaul 0faa68ccfc Honor jclouds.region with Swift blobstores 
jclouds has inconsistent support for regions due to provider
differences.  AWS providers a single global namespace while Swift
scopes per-endpoint.  Tested with Rackspace Cloud Files in HKG and ORD
regions.  Fixes #81.
 2015-10-14 13:08:29 -07:00
Andrew Gaul 618c35054c Do not check authorization when in anonymous mode 2015-10-11 11:09:04 -07:00
Andrew Gaul cad436fb95 Do not set Content-MD5 during copy part 
The previously used MD5 covered the entire object, not just the range
request.  Uncovered by ceph/s3-tests#75.
 2015-09-02 11:31:46 -07:00
Andrew Gaul 1d8dfea9c7 Use args4j for argument parsing 2015-09-01 11:12:34 -07:00
Nate Rosenblum f40693f214 Terminate on s3proxy start failure 
E.g. when bind(2) fails.
 2015-08-22 16:28:02 -07:00
Andrew Gaul 095d9aa94d Check anonymous access configuration first 
Previously, when configured for anonymous access and a client tried to
authenticate, S3Proxy would create a signature with a non-existent
credential.  Instead check anonymous access configuration first to
avoid a NullPointerException.
 2015-08-05 17:14:10 -07:00
Andrew Gaul 0daa04a3ed Emit NotImplemented for unknown x-amz headers 2015-07-30 21:49:44 -07:00
Andrew Gaul b85dc31d48 Call HttpServletRequest.getWriter 
This avoids explicit wrapping with OutputStreamWriter.
 2015-07-30 21:00:54 -07:00
Andrew Gaul cc5f1e259f Issue BlobStore calls before opening Writer 
This allows Jetty to propagate errors from the calls correctly.
Follow on to dba252bb99.
 2015-07-30 20:50:48 -07:00
Andrew Gaul 96a6d188a3 Set Content-MD5 during emulated copy part 
Dead store found with FindBugs.  References #56.
 2015-07-30 20:32:03 -07:00
Andrew Gaul cdbe139234 Remove munging of Google Cloud Storage ETag 
Clients need the original value for conditional GETs.
 2015-07-30 14:29:23 -07:00
Andrew Gaul 8e5fecaa91 Emulate multipart copy 
Fixes #56.
 2015-07-30 14:19:09 -07:00
Andrew Gaul 81c3fe9fc7 Consistently compared unquoted ETags 
Also use the ETag from listing parts which resolves an issue with
Swift multipart upload due to use of quoted ETags in the client S3
request and unquoted in underlying Swift request.
 2015-07-30 14:01:43 -07:00
Andrew Gaul dba252bb99 Call completeMultipartUpload before opening writer 
This allows Jetty to propagate errors from completeMultipartUpload
correctly.
 2015-07-29 16:22:18 -07:00
Andrew Gaul 50871a55ec Partially emulate arbitrary markers with GCS 
S3 supports arbitrary keys for the marker while GCS only supports its
opaque marker.  Emulate the common case for GCS by mapping the last
key from a listing to the corresponding previously returned marker.
 2015-07-28 15:49:32 -07:00
Andrew Gaul 19aa01c090 Partially emulate arbitrary markers with Azure 
S3 supports arbitrary keys for the marker while Azure only supports
its opaque marker.  Emulate the common case for Azure by mapping the
last key from a listing to the corresponding previously returned
marker.
 2015-07-28 05:15:07 -07:00
Andrew Gaul aed1a3ab3c Emit NotImplemented for multipart copy 
References #56.
 2015-07-23 23:14:36 -07:00
Andrew Gaul 566552a1e3 Split larger S3 MPU parts into smaller Azure parts 
Fixes #58.
 2015-07-17 16:06:48 -07:00
Timur Alperovich 763a004177 Enable delimiter support. 
Enables support for the delimiter option.

Amends a recursive test to expect that the delimiter is included in
the common prefix name.
 2015-07-14 13:43:48 -07:00
Andrew Gaul be3f38b937 Allow anonymous access via bucket and object ACLs 
Fixes #44.
 2015-07-13 13:58:03 -07:00
Andrew Gaul 757ed5acbe Add missing final modifier to classes 2015-07-12 23:27:35 -07:00
Andrew Gaul 2fc3d9b665 Remove unneeded builder assignments 2015-07-12 23:06:57 -07:00
Ka-Hing Cheung 16bb2082f4 get rid of default blob store 2015-07-07 14:22:35 -07:00
Andrew Gaul 58c4600d3f Use native prefix support instead of emulation 2015-06-30 09:52:38 -07:00
Ka-Hing Cheung a1c1129b53 run all tests in parallel 
need to strip expect headers from jclouds to workaround an issue,
which maybe https://issues.apache.org/jira/browse/JCLOUDS-905
 2015-06-24 18:26:36 -07:00
Andrew Gaul 0ed4d6c568 Include Content-Range header in range requests 
Fixes #63.
 2015-06-17 11:28:31 -07:00
Andrew Gaul 4b01ffbba2 Handle illegal ranges in range requests 2015-06-16 21:42:46 -07:00
Ka-Hing Cheung f17d968a33 allow endpoint to be missing when secure endpoint is set 2015-05-13 17:45:51 -07:00
Ka-Hing Cheung 6e4ebad1c8 add a way to listen to both http and https 2015-05-13 17:45:51 -07:00
Ka-Hing Cheung c26693490c correctly follow the host header documentation 
https://docs.aws.amazon.com/AmazonS3/latest/dev/VirtualHosting.html#VirtualHostingSpecifyBucket
 2015-05-11 17:53:58 -07:00
Andrew Gaul 96e1a65968 Use correct BlobStore for fake metadata 
Previously we could NPE when using the BlobStoreLocator.
 2015-05-08 16:11:07 -07:00
Ka-Hing Cheung c1b8234596 decode before splitting 2015-05-08 15:46:55 -07:00
Andrew Gaul 009a71d06c Remove unintentional log message 2015-05-05 11:08:39 -07:00
Andrew Gaul d47f607544 Enable more Checkstyle checks 
Also remove useless RedundantThrows check.
 2015-05-02 14:50:06 -07:00
Andrew Gaul fe4e36a0ea Emit InvalidPart when MPU part has invalid ETag 2015-05-02 11:32:24 -07:00
Andrew Gaul ceb805ef0a Emit InvalidPart when missing an MPU part 2015-05-02 11:08:09 -07:00
Andrew Gaul e947942f82 Quote ETag during uploadPart 2015-05-01 14:36:38 -07:00
Andrew Gaul 4a49f32f17 Native multipart upload support 
Fixes #2.
 2015-05-01 13:49:37 -07:00
Andrew Gaul 89279f5c58 Factor out calls to setHandled 2015-05-01 11:17:02 -07:00
Andrew Gaul 7fde437a7b Add XML elements to S3Exception 
This allows more descriptive errors, e.g., giving argument name when
returning InvalidArgument.
 2015-05-01 05:29:36 -07:00
Ka-Hing Cheung abc1bc4910 parse authorization header backwards 
this is to accomodate using s3proxy with swift backend, which has
a colon in the access key
 2015-04-28 18:07:03 -07:00
Ka-Hing Cheung 8fa555d093 handle empty prefix as if it's missing 2015-04-27 17:38:01 -07:00
Andrew Gaul 4a74a29604 Prefer ThrowingCallable over ExpectedException 2015-04-22 13:30:32 -07:00
Andrew Gaul 5f1c4979ef Remove multipart manifest during complete upload 2015-04-20 18:21:18 -07:00
Andrew Gaul 43b21829c7 Set ACLs during copy object 2015-04-18 04:38:03 -07:00
Andrew Gaul 38210cc498 Emit LastModified and ETag during copy object 
Regression from ea1a42dcd8.
 2015-04-18 04:37:20 -07:00
Andrew Gaul 9d072e4e11 Handle non-existent source blob during copy 2015-04-18 03:58:15 -07:00
Andrew Gaul 6350f6d26f Validate Content-Length before reading entity body 2015-04-17 19:30:36 -07:00
Andrew Gaul ea1a42dcd8 Add writeSimpleElement helper 2015-04-17 15:26:55 -07:00
Andrew Gaul 9a5d17fd62 Ignore case when matching header values 2015-04-17 15:09:52 -07:00
Andrew Gaul 5f9a1b519d Enable ACLs for local blobstores 2015-04-09 13:38:51 -07:00
Andrew Gaul 7b937c1571 Add native object copy support 
Fixes #46.
 2015-04-09 10:28:11 -07:00
Ka-Hing Cheung 35b37c34a9 allow S3Proxy to start without a blobstore 
because the mapping maybe provided by the callback
 2015-04-06 11:01:02 -07:00
Andrew Gaul 7271a5ace6 Compare identity in blobstore locator 
This allows S3Proxy to return the more specific InvalidAccessKeyId
instead of SignatureDoesNotMatch.  Addresses a s3-tests regression in
s3tests.functional.test_s3:test_list_buckets_invalid_auth.
 2015-04-04 12:36:24 -07:00
Andrew Gaul 4929573df9 Pass builder to constructor 2015-03-30 22:34:07 -07:00
Andrew Gaul e0a1560cb2 Use ExpectedException to handle test failures 
Also prefer matching AWS error code over the message.
 2015-03-29 13:02:55 -07:00
Andrew Gaul a3043c1e6f Suppress serialization warning 2015-03-25 17:03:57 -07:00
Ka-Hing Cheung 4450c6aebb add a way to lookup blobstore with a callback 
remove the old map based lookup
 2015-03-25 16:59:02 -07:00
Ka-Hing Cheung 4ca219470e reordered blobstore parameter 2015-03-25 16:59:02 -07:00
Ka-Hing Cheung 87bed283cb Allow multiple account providers 
This allows different backend blobstores to be configured at
runtime, based on the request access key. As a result different
frontend auth can also be used. Currently this can only be done
programmatically and not through properties config file.
 2015-03-24 13:23:50 -07:00
Andrew Gaul b036318ec6 Update copyright year to 2015 2015-03-21 23:51:24 -07:00
Andrew Gaul 02c1e8f77f Move main from S3Proxy to Main class 2015-03-21 23:51:23 -07:00
Andrew Gaul 103f949627 Centralize ContainerNotFoundException handling 2015-03-18 19:26:25 -07:00
Andrew Gaul 58800de7bf Expect AWSResponseException 2015-03-17 22:57:27 -07:00
Andrew Gaul 70a7a6ecf6 Upgrade to Modernizer Maven Plugin 1.3.0 
Changelog:

https://github.com/andrewgaul/modernizer-maven-plugin/releases/tag/modernizer-maven-plugin-1.3.0
 2015-03-17 17:46:25 -07:00
Andrew Gaul 209f3e60e4 Narrow exception handling scope 2015-03-13 18:58:27 -07:00
Andrew Gaul 27900bf4de Allow setting object ACL during PUT 
References #44.
 2015-03-12 17:42:40 -07:00
Andrew Gaul 5f117c15a6 Add SeparatorWrap check 
This ensures that chained method calls have the dot operator at the
beginning of the line.
 2015-03-12 17:22:55 -07:00
Andrew Gaul 7dbdc4faf8 Handle default private ACL 
References:

http://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketPUTacl.html
http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectPUTacl.html
 2015-03-06 18:02:00 -08:00
Andrew Gaul abe6b9ddf1 Automatically use MPU for Azure when needed 
All providers except for Azure support at least S3's 5 GB single-part
upload limit.  The previous configuration knob was suboptimal since it
used multi-part upload for small blobs.
 2015-03-06 17:28:20 -08:00
Andrew Gaul c2724b4399 Introduce S3Exception 
This elides some boilerplate and ensures that we consistently handle
errors.
 2015-03-04 12:49:09 -08:00
Andrew Gaul 7e0c817660 Add NotImplemented error code 2015-03-04 10:38:57 -08:00
Andrew Gaul cbac36efdd Avoid unnecessary String concatenation 2015-03-03 16:50:38 -08:00
Ka-Hing Cheung 335e735c5b provide reason for access denied 2015-03-03 12:47:33 -08:00
Andrew Gaul 71d3a67b95 Explicitly add Content-Length 
Previously InitiateMultipartUpload used the incorrect length.  Found
with s3fs integration tests.  References #2.
 2015-03-02 17:00:41 -08:00
Andrew Gaul e90074afc0 Add helper to emit Owner stanza 2015-03-02 16:08:09 -08:00
Andrew Gaul 0337a4e4bc Handle zero-length multi-part upload 2015-03-02 16:08:09 -08:00
Andrew Gaul b0b1f4e9fa Emulate multipart upload with single-part uploads 
This approach requires three times as many operations as the optimal
approach.  Implementing this correctly requires exposing the
underlying multipart operations in jclouds.  Most s3-tests pass but
test_list_multipart_upload still fails:

References #2.
 2015-03-02 14:37:16 -08:00
Andrew Gaul 11c93a43a8 Use correct content length during copy 
Previously replacing the metadata would zero out the object contents.
 2015-03-02 14:01:00 -08:00
Andrew Gaul a65221091f Use maven-assembly-plugin instead of Maven Shade 
This works around MSHADE-183 which prevents use with Maven 3.2.5 and
has caused recent Travis CI failures.  Fixes #38.
 2015-02-22 23:22:05 -08:00
Andrew Gaul 52118ced78 Add support to set and get canned object ACLs 
Currently only support private and public-read access.
 2015-02-22 18:48:08 -08:00
Andrew Gaul 613da2f3ec Use builder to create S3Proxy 
Also hoist sanity checks out of main.
 2015-02-20 11:01:37 -08:00
Andrew Gaul 2e82478f0e Prefer static imports for Preconditions 2015-02-20 10:50:37 -08:00
Timur Alperovich 51e7f3c22a Expose the Jetty state in the S3Proxy interface. 
The commit adds a getState() method, which exposes the Jetty state
to the caller. This is useful to check whether the server is running
or is in some other state.
 2015-02-18 16:13:45 -08:00