mirror
/
s3proxy
kopia lustrzana https://github.com/gaul/s3proxy
1
0
Forkuj 0
Kod Zgłoszenia Packages Projekty Wydania Wiki Aktywność
650 Commity
10 Gałęzie
27 Tagi
5,8 MiB
Wykres commitów

434 Commity (blobstore-locator)

Autor SHA1 Wiadomość Data
Andrew Gaul 3c8f68feb2 Update copyright year to 2016 2016-01-09 21:14:59 -08:00
Andrew Gaul f86bd162ea Call startsWithIgnoreCase helper 2016-01-09 21:03:46 -08:00
Andrew Gaul a5c9f28e0a Make V4_MAX_NON_CHUNKED_REQUEST_SIZE configurable 
Also default to 32 MB since goofys via aws-sdk-go uses 8 MB requests.
 2016-01-09 20:33:28 -08:00
Andrew Gaul 2a03aa137d Use v4 signing with S3AwsSdkTest 2016-01-09 19:54:51 -08:00
Andrew Gaul 9c25dc557c Fix off-by-one in V4_MAX_NON_CHUNKED_REQUEST_SIZE 2016-01-09 19:22:28 -08:00
Andrew Gaul 3a042ff6e8 Mask high bits in ChunkedInputStream.read 
Previously read could erroneously return -1 end of stream when
returning ASCII 128.
 2016-01-09 18:48:26 -08:00
Andrew Gaul 9976f3c02f Consume trailing newline in ChunkedInputStream 
References #24.
 2016-01-09 18:48:15 -08:00
Andrew Gaul 564463c0e0 Factor out MPU test ByteSource offsets 2016-01-09 17:24:02 -08:00
Andrew Gaul cc516e7ede Improve ChunkedInputStream exception message 2016-01-09 17:21:47 -08:00
Andrew Gaul 00dc2a94e0 Use BaseEncoding.lowerCase where possible 
This avoid extra allocations from String.toLowerCase.
 2016-01-09 17:06:38 -08:00
Andrew Gaul 5c3333d4d9 Parse chunk length in hexadecimal 
Resolves an issue when uploading large objects and parts.
References #24.
 2016-01-09 16:56:12 -08:00
Andrew Gaul e7ba7e5c0d Factor out S3Client initialization 2016-01-09 16:19:23 -08:00
Andrew Gaul e9baeb9c76 Always re-encode parameters 
Assume UTF-8 encoding if not provided.  Reference #24.
 2016-01-09 14:37:32 -08:00
Andrew Gaul c0d7cce353 Fix reversed arguments to String.indexOf 
Found via goofys tests.  References #24.
 2016-01-09 13:06:12 -08:00
Timur Alperovich 0f872c0a2c Support AWS v4 signatures 
Fixes #24.
 2016-01-08 15:31:06 -08:00
Andrew Gaul f24b5647fa Provide requests with explicit payload 
Some requests require payload transformations, e.g., chunked encoding,
so explicitly provide the payload instead of using the request
payload.  References #24.
 2016-01-07 14:02:20 -08:00
Andrew Gaul 0ab0b55f31 Remove duplicate call to get blobStoreType 2015-12-28 12:00:41 -08:00
Andrew Gaul e93529bfeb Use native putBlob ACLs 2015-12-23 12:56:28 -08:00
Andrew Gaul 118b656f8e Allow anonymous HEAD object on public-read objects 2015-12-22 18:41:15 -08:00
Andrew Gaul 024c7a3fe9 Encode list blobs HTTP responses in UTF-8 
Previously responses incorrectly encoded in ISO-8859-1.
 2015-12-22 18:31:32 -08:00
Andrew Gaul 0332a2107d Enabled testSetBlobAccess 
jclouds now uses canned ACLs which S3Proxy supports.
 2015-12-19 18:19:37 -08:00
Andrew Gaul 04f50b7363 Annotate JcloudsIntegrationTest with @Test 
For consistency with jclouds tests.
 2015-12-19 13:09:07 -08:00
Andrew Gaul d1dcc1abee Use testng annotations for JcloudsIntegrationTest 
This matches what the underlying test does and resolves a spurious
issue with testPutObjectStream.
 2015-12-19 00:41:27 -08:00
Andrew Gaul bc993cc3cb Skip testMaximumMultipartUpload for remote backend 
This test dominates run-time due to issuing 10,000 RPCs.
 2015-12-16 19:21:59 -08:00
Andrew Gaul ee55828375 Emit NotImplemented when setting XML ACLs 2015-12-16 19:19:44 -08:00
Andrew Gaul d9b3753f49 Do not set expires when creating blobs 
No tests currently check expires and setting it causes problems for
Swift-based backends.
 2015-12-15 17:22:25 -08:00
Andrew Gaul 3151754ebc Use container public-read for Swift test 
Swift does not support blob-level ACLs.
 2015-12-15 07:09:22 -08:00
Derrek Leute d8c73bc86d URL encode blob names when client requests it 
This allows support for characters which XML 1.0 cannot represent and
improves compatibility with newer versions of the AWS CLI.
Fixes #104.
 2015-12-14 17:10:45 -08:00
Andrew Gaul 23cff3a61d Do not require credentials for local blobstores 2015-12-14 16:01:35 -08:00
Andrew Gaul 9e033c677f Require credentials for aws-v2 authentication 2015-12-14 15:48:19 -08:00
Andrew Gaul 9af1d14dcc Provide content length to AmazonS3.putObject 
Prevents warnings of the form:

WARNING: No content length specified for stream data.  Stream contents
will be buffered in memory and could result in out of memory errors.
 2015-12-14 00:31:19 -08:00
Andrew Gaul 177d9b7566 Handle possibly-null HttpResponse 2015-12-13 23:43:22 -08:00
Andrew Gaul fa215ecb8a Factor out calls to get blobStoreType 2015-12-13 22:58:35 -08:00
Andrew Gaul e237ee9ebb Disable ETag checking for Google Cloud Storage 
Google Cloud Storage ETag is opaque and not an MD5 hash.
 2015-12-12 20:40:26 -08:00
Andrew Gaul c45dde3cbe Skip testMaximumMultipartUpload with large sizes 2015-12-12 13:00:48 +08:00
Andrew Gaul 0581c747bd Add missing @Override annotation 
Found via error-prone.
 2015-12-12 11:24:59 +08:00
Andrew Gaul 33fac374a6 Use container public-read for Azure test 
Azure does not support blob-level ACLs.
 2015-12-12 11:24:58 +08:00
Andrew Gaul eccf6d1b1f Disable ETag checking for Azure 
Azure ETag is opaque and not an MD5 hash.  References #96.
 2015-12-12 11:24:58 +08:00
Rita Zhang a5221f5f8e Upload two parts in testBigMultipartUpload 2015-12-03 16:31:42 +08:00
Andrew Gaul ed29debdf9 Upgrade to aws-java-sdk-s3 1.10.38 
This addresses an issue with the SDK providing an incorrect
Content-Type when uploading a multipart part.  Fixes #80.
 2015-12-02 22:24:47 +08:00
Andrew Gaul 01c43c9f1b Assert expected content length 
This makes some errors more obvious than comparing the content data.
 2015-12-02 22:18:28 +08:00
Andrew Gaul 9210cd679e Map IllegalArgumentException to HTTP 400 2015-11-20 22:31:06 -08:00
Andrew Gaul 67839acef1 Map UnsupportedOperationException to HTTP 501 2015-11-20 22:09:24 -08:00
Andrew Gaul 22ed1c542c Map KeyNotFoundException to NoSuchKey 
Unfortunately jclouds inconsistently throws ContainerNotFoundException
and KeyNotFoundException.
 2015-11-20 22:09:24 -08:00
Andrew Gaul bc2b18ee2b Add support for conditional get 
Fixes #77.
 2015-11-19 18:34:56 -08:00
Rita Zhang 71e2d61377 Allow anonymous access mode to use private buckets 
Fixes #94.
 2015-11-16 21:20:22 -08:00
Andrew Gaul 2207dfbd21 Basic support for POST uploads 
This commit lacks support for policies.  References #73.
 2015-11-14 23:22:05 -08:00
Andrew Gaul 1797992e02 Do not emit response body for HEAD requests 2015-11-11 16:03:42 -08:00
Andrew Gaul 22d57b1f18 Remove redundant private modifier from enum 2015-10-26 13:11:14 -07:00
Andrew Gaul 732ef5d260 Correct RandomByteSource.read return value 
Previously read returned a value between -128 and 127.  -1 indicates
end of stream, causing issues for callers.  Instead return values
between 0 and 255 as intended.
 2015-10-24 20:53:11 -07:00
Andrew Gaul 50a3024362 Work around incorrect Content-Type 
References #80.
 2015-10-20 22:17:56 -07:00
Alex Hall b7e98d1ce3 Test uploading large parts in multipart upload 
This commit exposes a AWS Java SDK bug.  References #80.
 2015-10-20 22:17:56 -07:00
Andrew Gaul d4f20646b4 Add test for 10,000 part MPU 
References #80.
 2015-10-20 14:44:17 -07:00
Andrew Gaul 97d467a698 Enforce no part number during complete MPU 
References #80.
 2015-10-20 14:41:42 -07:00
Andrew Gaul 274bf8a32e Handle listing a public-read container 
Also handle checking existence.
 2015-10-15 20:14:41 -07:00
Andrew Gaul 0faa68ccfc Honor jclouds.region with Swift blobstores 
jclouds has inconsistent support for regions due to provider
differences.  AWS providers a single global namespace while Swift
scopes per-endpoint.  Tested with Rackspace Cloud Files in HKG and ORD
regions.  Fixes #81.
 2015-10-14 13:08:29 -07:00
Andrew Gaul 618c35054c Do not check authorization when in anonymous mode 2015-10-11 11:09:04 -07:00
Andrew Gaul cad436fb95 Do not set Content-MD5 during copy part 
The previously used MD5 covered the entire object, not just the range
request.  Uncovered by ceph/s3-tests#75.
 2015-09-02 11:31:46 -07:00
Andrew Gaul 1d8dfea9c7 Use args4j for argument parsing 2015-09-01 11:12:34 -07:00
Nate Rosenblum f40693f214 Terminate on s3proxy start failure 
E.g. when bind(2) fails.
 2015-08-22 16:28:02 -07:00
Andrew Gaul 095d9aa94d Check anonymous access configuration first 
Previously, when configured for anonymous access and a client tried to
authenticate, S3Proxy would create a signature with a non-existent
credential.  Instead check anonymous access configuration first to
avoid a NullPointerException.
 2015-08-05 17:14:10 -07:00
Andrew Gaul 0daa04a3ed Emit NotImplemented for unknown x-amz headers 2015-07-30 21:49:44 -07:00
Andrew Gaul b85dc31d48 Call HttpServletRequest.getWriter 
This avoids explicit wrapping with OutputStreamWriter.
 2015-07-30 21:00:54 -07:00
Andrew Gaul cc5f1e259f Issue BlobStore calls before opening Writer 
This allows Jetty to propagate errors from the calls correctly.
Follow on to dba252bb99.
 2015-07-30 20:50:48 -07:00
Andrew Gaul 96a6d188a3 Set Content-MD5 during emulated copy part 
Dead store found with FindBugs.  References #56.
 2015-07-30 20:32:03 -07:00
Andrew Gaul cdbe139234 Remove munging of Google Cloud Storage ETag 
Clients need the original value for conditional GETs.
 2015-07-30 14:29:23 -07:00
Andrew Gaul 8e5fecaa91 Emulate multipart copy 
Fixes #56.
 2015-07-30 14:19:09 -07:00
Andrew Gaul 81c3fe9fc7 Consistently compared unquoted ETags 
Also use the ETag from listing parts which resolves an issue with
Swift multipart upload due to use of quoted ETags in the client S3
request and unquoted in underlying Swift request.
 2015-07-30 14:01:43 -07:00
Andrew Gaul 4684c1efe0 Do not test Content-Language with Swift 
Swift does not support this header.
 2015-07-29 16:59:18 -07:00
Andrew Gaul dba252bb99 Call completeMultipartUpload before opening writer 
This allows Jetty to propagate errors from completeMultipartUpload
correctly.
 2015-07-29 16:22:18 -07:00
Andrew Gaul 50871a55ec Partially emulate arbitrary markers with GCS 
S3 supports arbitrary keys for the marker while GCS only supports its
opaque marker.  Emulate the common case for GCS by mapping the last
key from a listing to the corresponding previously returned marker.
 2015-07-28 15:49:32 -07:00
Andrew Gaul 19aa01c090 Partially emulate arbitrary markers with Azure 
S3 supports arbitrary keys for the marker while Azure only supports
its opaque marker.  Emulate the common case for Azure by mapping the
last key from a listing to the corresponding previously returned
marker.
 2015-07-28 05:15:07 -07:00
Andrew Gaul f94d1834c9 Add test for multipart copy 
S3Proxy does not yet support this so we expect NotImplemented.
References #56.
 2015-07-25 13:04:05 -07:00
Andrew Gaul aed1a3ab3c Emit NotImplemented for multipart copy 
References #56.
 2015-07-23 23:14:36 -07:00
Andrew Gaul 566552a1e3 Split larger S3 MPU parts into smaller Azure parts 
Fixes #58.
 2015-07-17 16:06:48 -07:00
Timur Alperovich 763a004177 Enable delimiter support. 
Enables support for the delimiter option.

Amends a recursive test to expect that the delimiter is included in
the common prefix name.
 2015-07-14 13:43:48 -07:00
Andrew Gaul 2a403aa494 Temporarily disable testBlobListRecursive 
Regression due to jclouds delimiter changes.
 2015-07-14 11:33:39 -07:00
Andrew Gaul be3f38b937 Allow anonymous access via bucket and object ACLs 
Fixes #44.
 2015-07-13 13:58:03 -07:00
Andrew Gaul 757ed5acbe Add missing final modifier to classes 2015-07-12 23:27:35 -07:00
Andrew Gaul 2fc3d9b665 Remove unneeded builder assignments 2015-07-12 23:06:57 -07:00
Andrew Gaul 2a1e7b5da1 Add test for single-part upload metadata 
Previously we only checked metadata in multi-part upload.
 2015-07-10 16:06:38 -07:00
Ka-Hing Cheung 16bb2082f4 get rid of default blob store 2015-07-07 14:22:35 -07:00
Andrew Gaul 58c4600d3f Use native prefix support instead of emulation 2015-06-30 09:52:38 -07:00
Andrew Gaul 46d1fc21e9 Check actual content metadata during MPU test 2015-06-26 18:15:28 -07:00
Ka-Hing Cheung a1c1129b53 run all tests in parallel 
need to strip expect headers from jclouds to workaround an issue,
which maybe https://issues.apache.org/jira/browse/JCLOUDS-905
 2015-06-24 18:26:36 -07:00
Ka-Hing Cheung 5098722b8b run test with http 2015-06-24 18:25:34 -07:00
Ka-Hing Cheung 8f60c1e076 add jclouds test 2015-06-24 14:12:15 -07:00
Ka-Hing Cheung 9b61608352 re-factor out s3proxy launch function 2015-06-24 14:04:57 -07:00
Ka-Hing Cheung 67e4ff80b2 downgrade to only use assertj 1.7.0 features 
also fixed the incorrect equals assertion
 2015-06-24 13:25:50 -07:00
Andrew Gaul 0ed4d6c568 Include Content-Range header in range requests 
Fixes #63.
 2015-06-17 11:28:31 -07:00
Andrew Gaul 4b01ffbba2 Handle illegal ranges in range requests 2015-06-16 21:42:46 -07:00
Andrew Gaul 3690079f09 Correct actual and expected variable names 2015-06-05 19:51:48 -07:00
Andrew Gaul bff39d143e Use random input for multipart payload test 2015-06-05 19:48:06 -07:00
Ka-Hing Cheung f17d968a33 allow endpoint to be missing when secure endpoint is set 2015-05-13 17:45:51 -07:00
Ka-Hing Cheung 6e4ebad1c8 add a way to listen to both http and https 2015-05-13 17:45:51 -07:00
Ka-Hing Cheung c26693490c correctly follow the host header documentation 
https://docs.aws.amazon.com/AmazonS3/latest/dev/VirtualHosting.html#VirtualHostingSpecifyBucket
 2015-05-11 17:53:58 -07:00
Andrew Gaul 96e1a65968 Use correct BlobStore for fake metadata 
Previously we could NPE when using the BlobStoreLocator.
 2015-05-08 16:11:07 -07:00
Ka-Hing Cheung c1b8234596 decode before splitting 2015-05-08 15:46:55 -07:00
Andrew Gaul 009a71d06c Remove unintentional log message 2015-05-05 11:08:39 -07:00
Andrew Gaul d47f607544 Enable more Checkstyle checks 
Also remove useless RedundantThrows check.
 2015-05-02 14:50:06 -07:00
Andrew Gaul fe4e36a0ea Emit InvalidPart when MPU part has invalid ETag 2015-05-02 11:32:24 -07:00
Andrew Gaul ceb805ef0a Emit InvalidPart when missing an MPU part 2015-05-02 11:08:09 -07:00
Andrew Gaul e947942f82 Quote ETag during uploadPart 2015-05-01 14:36:38 -07:00
Andrew Gaul 4a49f32f17 Native multipart upload support 
Fixes #2.
 2015-05-01 13:49:37 -07:00
Andrew Gaul 89279f5c58 Factor out calls to setHandled 2015-05-01 11:17:02 -07:00
Andrew Gaul 7fde437a7b Add XML elements to S3Exception 
This allows more descriptive errors, e.g., giving argument name when
returning InvalidArgument.
 2015-05-01 05:29:36 -07:00
Ka-Hing Cheung abc1bc4910 parse authorization header backwards 
this is to accomodate using s3proxy with swift backend, which has
a colon in the access key
 2015-04-28 18:07:03 -07:00
Ka-Hing Cheung 8fa555d093 handle empty prefix as if it's missing 2015-04-27 17:38:01 -07:00
Andrew Gaul 4a74a29604 Prefer ThrowingCallable over ExpectedException 2015-04-22 13:30:32 -07:00
Andrew Gaul ea5f6a9363 Test URL signing using AWS SDK 
jclouds generic S3 provider does not implement the correct logic for
this.
 2015-04-21 19:56:49 -07:00
Andrew Gaul 5f1c4979ef Remove multipart manifest during complete upload 2015-04-20 18:21:18 -07:00
Andrew Gaul 43b21829c7 Set ACLs during copy object 2015-04-18 04:38:03 -07:00
Andrew Gaul 38210cc498 Emit LastModified and ETag during copy object 
Regression from ea1a42dcd8.
 2015-04-18 04:37:20 -07:00
Andrew Gaul 9d072e4e11 Handle non-existent source blob during copy 2015-04-18 03:58:15 -07:00
Andrew Gaul 6350f6d26f Validate Content-Length before reading entity body 2015-04-17 19:30:36 -07:00
Andrew Gaul 17e9f71bb4 Factor out ByteSource payload 2015-04-17 18:24:56 -07:00
Andrew Gaul ea1a42dcd8 Add writeSimpleElement helper 2015-04-17 15:26:55 -07:00
Andrew Gaul 9a5d17fd62 Ignore case when matching header values 2015-04-17 15:09:52 -07:00
Andrew Gaul f1dec1e3fc Test AWS SDK with v2 signature 2015-04-17 14:41:57 -07:00
Andrew Gaul 5f9a1b519d Enable ACLs for local blobstores 2015-04-09 13:38:51 -07:00
Andrew Gaul 7b937c1571 Add native object copy support 
Fixes #46.
 2015-04-09 10:28:11 -07:00
Ka-Hing Cheung 35b37c34a9 allow S3Proxy to start without a blobstore 
because the mapping maybe provided by the callback
 2015-04-06 11:01:02 -07:00
Andrew Gaul 7271a5ace6 Compare identity in blobstore locator 
This allows S3Proxy to return the more specific InvalidAccessKeyId
instead of SignatureDoesNotMatch.  Addresses a s3-tests regression in
s3tests.functional.test_s3:test_list_buckets_invalid_auth.
 2015-04-04 12:36:24 -07:00
Andrew Gaul 4929573df9 Pass builder to constructor 2015-03-30 22:34:07 -07:00
Andrew Gaul e0a1560cb2 Use ExpectedException to handle test failures 
Also prefer matching AWS error code over the message.
 2015-03-29 13:02:55 -07:00
Andrew Gaul a3043c1e6f Suppress serialization warning 2015-03-25 17:03:57 -07:00
Ka-Hing Cheung 4450c6aebb add a way to lookup blobstore with a callback 
remove the old map based lookup
 2015-03-25 16:59:02 -07:00
Ka-Hing Cheung 4ca219470e reordered blobstore parameter 2015-03-25 16:59:02 -07:00
Ka-Hing Cheung 87bed283cb Allow multiple account providers 
This allows different backend blobstores to be configured at
runtime, based on the request access key. As a result different
frontend auth can also be used. Currently this can only be done
programmatically and not through properties config file.
 2015-03-24 13:23:50 -07:00
Andrew Gaul b036318ec6 Update copyright year to 2015 2015-03-21 23:51:24 -07:00
Andrew Gaul 02c1e8f77f Move main from S3Proxy to Main class 2015-03-21 23:51:23 -07:00
Andrew Gaul 103f949627 Centralize ContainerNotFoundException handling 2015-03-18 19:26:25 -07:00
Andrew Gaul 81a5024bd3 Include jclouds properties in overrides 2015-03-18 19:21:31 -07:00
Andrew Gaul 6b5f9fd7f3 Remove unused s3Context 
This test uses AmazonS3Client instead of jclouds client.
 2015-03-17 23:55:48 -07:00
Andrew Gaul 58800de7bf Expect AWSResponseException 2015-03-17 22:57:27 -07:00
Andrew Gaul 70a7a6ecf6 Upgrade to Modernizer Maven Plugin 1.3.0 
Changelog:

https://github.com/andrewgaul/modernizer-maven-plugin/releases/tag/modernizer-maven-plugin-1.3.0
 2015-03-17 17:46:25 -07:00
Andrew Gaul 209f3e60e4 Narrow exception handling scope 2015-03-13 18:58:27 -07:00
Andrew Gaul 27900bf4de Allow setting object ACL during PUT 
References #44.
 2015-03-12 17:42:40 -07:00
Andrew Gaul 2b29bcd588 Disable AWS SDK SSL verification 2015-03-12 17:38:51 -07:00
Andrew Gaul 5f117c15a6 Add SeparatorWrap check 
This ensures that chained method calls have the dot operator at the
beginning of the line.
 2015-03-12 17:22:55 -07:00
Andrew Gaul 7dbdc4faf8 Handle default private ACL 
References:

http://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketPUTacl.html
http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectPUTacl.html
 2015-03-06 18:02:00 -08:00
Andrew Gaul abe6b9ddf1 Automatically use MPU for Azure when needed 
All providers except for Azure support at least S3's 5 GB single-part
upload limit.  The previous configuration knob was suboptimal since it
used multi-part upload for small blobs.
 2015-03-06 17:28:20 -08:00
Timur Alperovich 1d7af6acae Add a test for failure when using v4 signatures. 
Adds a unit test with the Amazon Java SDK for failures when submitting
v4-signed requests. We use the Amazon Java SDK while waiting for the
issues JCLOUDS-480 to be resolved (implementing v4 signatures).
 2015-03-05 23:37:25 -08:00
Andrew Gaul c2724b4399 Introduce S3Exception 
This elides some boilerplate and ensures that we consistently handle
errors.
 2015-03-04 12:49:09 -08:00
Andrew Gaul 7e0c817660 Add NotImplemented error code 2015-03-04 10:38:57 -08:00
Andrew Gaul cbac36efdd Avoid unnecessary String concatenation 2015-03-03 16:50:38 -08:00
Ka-Hing Cheung 335e735c5b provide reason for access denied 2015-03-03 12:47:33 -08:00
Andrew Gaul 71d3a67b95 Explicitly add Content-Length 
Previously InitiateMultipartUpload used the incorrect length.  Found
with s3fs integration tests.  References #2.
 2015-03-02 17:00:41 -08:00
Andrew Gaul e90074afc0 Add helper to emit Owner stanza 2015-03-02 16:08:09 -08:00
Andrew Gaul 0337a4e4bc Handle zero-length multi-part upload 2015-03-02 16:08:09 -08:00
Andrew Gaul b0b1f4e9fa Emulate multipart upload with single-part uploads 
This approach requires three times as many operations as the optimal
approach.  Implementing this correctly requires exposing the
underlying multipart operations in jclouds.  Most s3-tests pass but
test_list_multipart_upload still fails:

References #2.
 2015-03-02 14:37:16 -08:00
Andrew Gaul 11c93a43a8 Use correct content length during copy 
Previously replacing the metadata would zero out the object contents.
 2015-03-02 14:01:00 -08:00
Andrew Gaul a65221091f Use maven-assembly-plugin instead of Maven Shade 
This works around MSHADE-183 which prevents use with Maven 3.2.5 and
has caused recent Travis CI failures.  Fixes #38.
 2015-02-22 23:22:05 -08:00
Andrew Gaul 52118ced78 Add support to set and get canned object ACLs 
Currently only support private and public-read access.
 2015-02-22 18:48:08 -08:00
Andrew Gaul c8e4279853 Propagate Exception if not HttpResponseException 2015-02-20 17:22:47 -08:00
Andrew Gaul 613da2f3ec Use builder to create S3Proxy 
Also hoist sanity checks out of main.
 2015-02-20 11:01:37 -08:00
Andrew Gaul 2e82478f0e Prefer static imports for Preconditions 2015-02-20 10:50:37 -08:00
Timur Alperovich 51e7f3c22a Expose the Jetty state in the S3Proxy interface. 
The commit adds a getState() method, which exposes the Jetty state
to the caller. This is useful to check whether the server is running
or is in some other state.
 2015-02-18 16:13:45 -08:00
Andrew Gaul b85ab17316 Correct typo 2015-02-18 11:34:28 -08:00
Andrew Gaul 7e61856db1 Support native multi-object delete 2015-02-15 01:07:10 -08:00
Andrew Gaul 9a261a62b0 Add support to set and get canned container ACLs 
Currently only support private and public-read access.
 2015-02-13 15:43:58 -08:00
Andrew Gaul 4ebbce7956 Run unknown parameter test 
JCLOUDS-258 allows use of MPU with the generic S3 provider.
 2015-02-12 18:08:29 -08:00
Andrew Gaul 074c36baab Add test harness to execute s3-tests 
Also add Travis configuration.
 2015-01-22 17:14:45 -08:00
Andrew Gaul 23f8e772da Upgrade to jclouds 2.0.0-SNAPSHOT 
This release allows larger than 2 GB single-part blobs and supports
object metadata with the filesystem provider on Linux and Windows.  It
also includes support for object size in container lists.  Fixes #1.
Fixes #6.
 2015-01-22 14:03:18 -08:00
Andrew Gaul fb0f321303 Improve CommonPrefixes support 
References #5.
 2015-01-22 13:30:28 -08:00
Andrew Gaul 7815a79548 Emit NotImplemented with unknown parameters 
This ensures that requests like multi-part upload and version
operations emit the intended responses.
 2015-01-19 20:06:58 -08:00
Andrew Gaul bae51b500a Reduce exception handling scope 2015-01-16 15:56:08 -08:00
Andrew Gaul e1c7b22b7d Handle NoSuchBucket during copy object 2015-01-16 15:14:53 -08:00
Andrew Gaul d81ee81a86 Handle null from HttpResponseException.getResponse 2015-01-15 18:36:14 -08:00
Andrew Gaul 05b63365bf Emit bogus CreationDate when not present 2015-01-15 18:23:22 -08:00
Andrew Gaul 7dff664c17 Add prefix to CommonPrefixes 
Addresses s3cmd tests failures.
 2015-01-14 12:24:23 -08:00
Andrew Gaul 5b5e9877b9 Improve support for x-amz-metadata-directive 
Replace content metadata in addition to user metadata.  Addresses
s3cmd tests failures.
 2015-01-13 23:32:46 -08:00
Andrew Gaul a6133a60a6 Honor force multipart in emulated copy object 2015-01-13 23:28:09 -08:00
Andrew Gaul 86665450ca URI decode copy source path 
Fixes s3cmd tests failures seen with Unicode object names.  Follow on
to b91b8e25d7.
 2015-01-13 18:44:59 -08:00
Andrew Gaul e3c49331ac Emit UTF-8 in object listing and multi-delete 
Fixes s3cmd tests failures seen with Unicode object names.
 2015-01-13 17:50:47 -08:00
Andrew Gaul 946d34b03b Parse XML with StAX 
This addresses issues with object names containing special characters.
Fixes #25.
 2015-01-11 20:45:56 -08:00
Andrew Gaul f3221cb136 Emit XML via StAX 
This addresses issues with object names containing special characters.
References #25.
 2015-01-11 18:56:39 -08:00
Ka-Hing Cheung e955b998fb use static function in URLDecoder 2015-01-09 15:31:45 -08:00
Ka-Hing Cheung b91b8e25d7 uri decode path before passing them to lower layers 2015-01-06 15:54:36 -08:00
Andrew Gaul 15e11975a2 Allow kernel to pick S3Proxy port in tests 2015-01-06 15:05:21 -08:00
Ka-Hing Cheung 2dd3dac970 add test for % in blob name 2015-01-06 14:48:03 -08:00
Andrew Gaul ff551248a1 Fail V4 signature requests with InvalidArgument 
This allows V4 clients like s3cmd to retry with V2 signatures.
References #24.
 2015-01-06 14:42:29 -08:00
Andrew Gaul 0eac109276 Add logback.xml only to with jar-with-dependencies 2015-01-05 18:51:56 -08:00
Ka-Hing Cheung dbd7304e65 make S3Proxy contructor public 2015-01-05 11:39:05 -08:00
Andrew Gaul 1f9fd7fa21 Consistently emit XML prologue and AWS namespace 2014-12-29 23:18:49 -08:00
Andrew Gaul a2c6bfc4cc Use correct container name with testHttpClient 
This test still fails due to authorization issues.
 2014-12-29 17:45:43 -08:00
Andrew Gaul 807f6ef47c Explicitly handle get location 2014-12-29 16:29:30 -08:00
Andrew Gaul d33324797a Consistently flush before closing Writer 2014-12-28 14:52:36 -08:00
Andrew Gaul 711c0821d1 Sort subresources 
This ensures deterministic signatures when requests have multiple
subresources, e.g., multipart uploads.
 2014-12-21 23:11:59 -08:00
Andrew Gaul 161fd3d847 Include delete as a signed resource 
This allows multi-object delete to work and addresses a regression
introduced by e5c12c1777.
 2014-12-03 22:59:36 -08:00
Andrew Gaul ae3598dbfb Emit InvalidAccessKeyId on access key mismatch 
References #5.
 2014-12-03 22:39:41 -08:00
Andrew Gaul 40c4d099d9 Handle invalid Content-Length during create bucket 
References #5.
 2014-12-03 18:36:15 -08:00
Andrew Gaul 3b2bd615a0 Simplify parsing of Content-Length and Content-MD5 
HttpServletRequest uses null to indicate a header value which is
present but has no value.  Use empty String instead.
 2014-12-03 04:30:54 -08:00
Ka-Hing Cheung e5c12c1777 more required parameters needed for signing 
allows s3cmd info s3://bucket/ to work
 2014-11-20 04:31:23 +01:00
Ka-Hing Cheung 617b3ae3e9 more compatibility fixes for s3cmd 
s3cmd uses virtual hostname for bucket name, with the host header
including the port. need to strip out the port to find out the
bucket name
 2014-11-18 10:14:16 +01:00
Ka-Hing Cheung 004d12761e http header name is case insensitive 
fixes compatibility with s3cmd
 2014-11-18 03:45:58 +01:00
Andrew Gaul 559b1d36dd Clean up signer 2014-10-27 15:29:53 -07:00
Andrew Gaul 5742323a70 Remove Azure ETag hack 
Previously jclouds required ETag to be a hex-encoded Content-MD5.
 2014-10-27 15:21:40 -07:00
Andrew Gaul 9a81b73324 Add more Checkstyle rules 2014-10-24 00:28:58 -07:00
Andrew Gaul 4a58f00885 Add Google Cloud Storage provider 
Also convert ETag from GCS-style base64 to AWS-style base16.
 2014-10-24 00:23:51 -07:00