mirror
/
s3proxy
kopia lustrzana https://github.com/gaul/s3proxy
1
0
Forkuj 0
Kod Zgłoszenia Packages Projekty Wydania Wiki Aktywność
650 Commity
10 Gałęzie
27 Tagi
5,7 MiB
Wykres commitów

434 Commity (blobstore-locator)

Autor SHA1 Wiadomość Data
Andrew Gaul 99d8e5e8d7 Add missing @Override annotations 
Found via error-prone.
 2016-06-03 10:41:05 -07:00
Andrew Gaul 0053102df5 Add test for aborting multi-part uploads 2016-06-02 07:42:17 -07:00
Andrew Gaul 519cd9313c Remove stub blob during multi-part upload 
References #135.
 2016-06-02 07:42:16 -07:00
Andrew Gaul 2eaf7de68f Handle null ETag during multipart upload 
Fixes #135.
 2016-06-01 12:51:41 -07:00
Andrew Gaul d6c81c9b64 Correct issues with encoding of plus character 
Also remove some questionable decoding.
 2016-06-01 11:22:16 -07:00
Andrew Gaul e7bb6ddb47 Avoid NullPointerException when quoting ETags 2016-05-27 18:07:41 -07:00
Andrew Gaul a4b96924eb Skip time-limited signed requests 
jclouds aws-s3 provider supports these but s3 does not.
 2016-05-12 12:06:03 -07:00
Andrew Gaul 2cfcc33bb8 Upgrade to aws-java-sdk-s3 1.10.76 
Also address deprecations.
 2016-05-05 22:22:04 -07:00
Andrew Gaul 2296b8228b Add configuration to ignore unknown headers 
Fixes #130.
 2016-03-12 23:15:45 -07:00
Andrew Gaul d878642b5d Add quirks for google-cloud-storage 2016-03-09 13:35:02 -08:00
Andrew Gaul eece05f02e Fix path-based GCS credentials 
Providing both credential and the credential property used only the
latter, which was not expanded when credential is a file.  Also expand
credential in tests.  References #128.
 2016-03-09 13:35:02 -08:00
Andrew Gaul 6292d28c9c Add EventualBlobStore 
This models eventually-consistent behavior.  This implementation uses
two buckets and client writes going to the first bucket and reads to
the second.  Operations later replicate from the first to the second
with a variable delays.  A more complete implementation could flap
between the strongly- and eventually-consistent buckets and arbitrary
reorder operations which conclude with last-writer wins.  Fixes #65.
 2016-03-01 17:13:39 -08:00
Andrew Gaul 0351ee73dc Expand GCS credential files into PEMs 2016-02-26 17:45:33 -08:00
Andrew Gaul e670537822 Handle Swift quirks in jclouds tests 2016-02-20 00:49:16 -08:00
Andrew Gaul 2062aadfeb Handle conditional HEAD object requests 2016-02-18 11:48:48 -08:00
Andrew Gaul 52ba2c843f Add support for conditional copy object 
Fixes #113.
 2016-02-16 20:31:12 -08:00
Andrew Gaul d66b539f2c Reduce visibility of methods 2016-02-16 16:16:51 -08:00
Andrew Gaul 884660ea05 Correct typo 2016-02-11 15:40:45 -08:00
Andrew Gaul a4385daa45 Fix v4 signing of queries with special characters 
Reference:

https://docs.aws.amazon.com/general/latest/gr/sigv4-create-canonical-request.html
 2016-02-09 11:52:04 -08:00
Andrew Gaul 9c3d9409ac Read from PushBackInputStream not InputStream 
Previously S3Proxy truncated create bucket requests.
 2016-02-08 22:54:39 -08:00
Andrew Gaul 7198c9088c Do not close InputStream for each uploaded subpart 2016-02-04 00:09:41 -08:00
Andrew Gaul 997dc9707c Skip expires tests when backend lacks support 2016-02-03 23:12:26 -08:00
Andrew Gaul 6d7a46296e Skip zero keys tests when backend lacks support 2016-02-01 12:29:43 -08:00
Andrew Gaul 1ae596e870 Support list multipart upload parts with Azure 
Map the Azure subparts back into S3 parts.
 2016-01-31 13:57:52 -08:00
Andrew Gaul bc68e2cb2e Do not run marker tests when backend lacks support 2016-01-31 12:50:04 -08:00
Andrew Gaul 82d95a2746 Use getObject and generatePresignedUrl short forms 2016-01-30 21:26:17 -08:00
Andrew Gaul 940286b022 Correct cut and paste errors in copy tests 2016-01-30 21:26:17 -08:00
Andrew Gaul 65fd80036f Do not run ACL tests when backend lacks support 2016-01-30 21:26:17 -08:00
Andrew Gaul cea4a42922 Use decoded content length when provided 
Previously S3Proxy provided an incorrect Content-Length header when
using V4 signing which providers rejected.
 2016-01-30 19:13:04 -08:00
Andrew Gaul 983c2b003e Remove stale TODO 2016-01-30 17:11:47 -08:00
Andrew Gaul e292ebaccb Consistently parse XML with Jackson 2016-01-29 17:50:22 -08:00
Andrew Gaul 511c968ce7 Add test for V4 URL signing 
Currently fails.
 2016-01-29 17:30:13 -08:00
Andrew Gaul 8d8e8ff290 Upgrade to aws-java-sdk-s3 1.10.50 
This release includes an implementation of AccessControlList.equals.
References #116.

Release notes:

https://aws.amazon.com/releasenotes/8286007886477675
 2016-01-29 09:50:14 -08:00
Andrew Gaul bc1833ed2e Remove BlobAccess TODO 
jclouds cannot provide BlobAccess since providers do not return these
ACLs.
 2016-01-26 19:28:50 -08:00
Andrew Gaul 6b0a6d8cdc Ensure failing tests raise NotImplemented error 2016-01-24 13:52:56 -08:00
Andrew Gaul f202207c63 Map a subset of XML ACLs to canned ACLs 
Fixes #116.
 2016-01-23 15:34:03 -08:00
Andrew Gaul 0e3a7ef843 Add missing EmailAddress field 
This allows S3Proxy to return the correct NotImplemented error code
instead of InternalError.  References #116.
 2016-01-23 15:16:38 -08:00
Andrew Gaul d557af774d Replace query parameters in anonymous requests 2016-01-23 15:06:04 -08:00
Andrew Gaul 19e2426220 Map a subset of XML ACLs to canned ACLs 
Fixes #116.
 2016-01-21 18:24:57 -08:00
Andrew Gaul 587a4a6670 Centralize blobstore quirks 2016-01-17 23:10:28 -08:00
Andrew Gaul 744d126331 Add support for Cache-Control header 
Fixes #115.
 2016-01-17 12:08:52 -08:00
Andrew Gaul 542c9b8112 Add Accept-Ranges header to partial get requests 
Fixes #66.
 2016-01-15 22:37:19 -08:00
Andrew Gaul 05bd2c1158 Use uploadId for stub blob name 
This makes MPU overwrites of an existing blob atomic.  Fixes #108.
 2016-01-14 15:11:03 -08:00
Andrew Gaul 3aed140a15 Avoid mutable ArrayList 
Latest AWS SDK handles immutable List.  Also tidy up line wrapping.
 2016-01-13 16:12:26 -08:00
Andrew Gaul ef3a7a6adf Add JcloudsS3BlobSignerLiveTest 
References #62.
 2016-01-12 16:20:03 -08:00
Andrew Gaul fb3bd591d6 Add JcloudsBucketsLiveTest 
Uncovers several missing features:

Bucket logging
Bucket payer
XML ACLs

References #62.
 2016-01-12 16:20:02 -08:00
Andrew Gaul 9c5499a05e Add JcloudsS3ClientLiveTest 
Uncovers several missing features:

Cache-Control header
conditional copies
public-read-write canned ACLs
XML ACLs

References #62.
 2016-01-12 16:20:02 -08:00
Andrew Gaul ae95538647 Add JcloudsS3ContainerIntegrationLiveTest 
References #62.
 2016-01-12 16:19:56 -08:00
Andrew Gaul f4212571b0 Rename suite to JcloudsS3BlobIntegrationLiveTest 2016-01-11 20:23:13 -08:00
Andrew Gaul 91429586dc Reuse encodingType local 2016-01-11 20:01:50 -08:00
Andrew Gaul 3c8f68feb2 Update copyright year to 2016 2016-01-09 21:14:59 -08:00
Andrew Gaul f86bd162ea Call startsWithIgnoreCase helper 2016-01-09 21:03:46 -08:00
Andrew Gaul a5c9f28e0a Make V4_MAX_NON_CHUNKED_REQUEST_SIZE configurable 
Also default to 32 MB since goofys via aws-sdk-go uses 8 MB requests.
 2016-01-09 20:33:28 -08:00
Andrew Gaul 2a03aa137d Use v4 signing with S3AwsSdkTest 2016-01-09 19:54:51 -08:00
Andrew Gaul 9c25dc557c Fix off-by-one in V4_MAX_NON_CHUNKED_REQUEST_SIZE 2016-01-09 19:22:28 -08:00
Andrew Gaul 3a042ff6e8 Mask high bits in ChunkedInputStream.read 
Previously read could erroneously return -1 end of stream when
returning ASCII 128.
 2016-01-09 18:48:26 -08:00
Andrew Gaul 9976f3c02f Consume trailing newline in ChunkedInputStream 
References #24.
 2016-01-09 18:48:15 -08:00
Andrew Gaul 564463c0e0 Factor out MPU test ByteSource offsets 2016-01-09 17:24:02 -08:00
Andrew Gaul cc516e7ede Improve ChunkedInputStream exception message 2016-01-09 17:21:47 -08:00
Andrew Gaul 00dc2a94e0 Use BaseEncoding.lowerCase where possible 
This avoid extra allocations from String.toLowerCase.
 2016-01-09 17:06:38 -08:00
Andrew Gaul 5c3333d4d9 Parse chunk length in hexadecimal 
Resolves an issue when uploading large objects and parts.
References #24.
 2016-01-09 16:56:12 -08:00
Andrew Gaul e7ba7e5c0d Factor out S3Client initialization 2016-01-09 16:19:23 -08:00
Andrew Gaul e9baeb9c76 Always re-encode parameters 
Assume UTF-8 encoding if not provided.  Reference #24.
 2016-01-09 14:37:32 -08:00
Andrew Gaul c0d7cce353 Fix reversed arguments to String.indexOf 
Found via goofys tests.  References #24.
 2016-01-09 13:06:12 -08:00
Timur Alperovich 0f872c0a2c Support AWS v4 signatures 
Fixes #24.
 2016-01-08 15:31:06 -08:00
Andrew Gaul f24b5647fa Provide requests with explicit payload 
Some requests require payload transformations, e.g., chunked encoding,
so explicitly provide the payload instead of using the request
payload.  References #24.
 2016-01-07 14:02:20 -08:00
Andrew Gaul 0ab0b55f31 Remove duplicate call to get blobStoreType 2015-12-28 12:00:41 -08:00
Andrew Gaul e93529bfeb Use native putBlob ACLs 2015-12-23 12:56:28 -08:00
Andrew Gaul 118b656f8e Allow anonymous HEAD object on public-read objects 2015-12-22 18:41:15 -08:00
Andrew Gaul 024c7a3fe9 Encode list blobs HTTP responses in UTF-8 
Previously responses incorrectly encoded in ISO-8859-1.
 2015-12-22 18:31:32 -08:00
Andrew Gaul 0332a2107d Enabled testSetBlobAccess 
jclouds now uses canned ACLs which S3Proxy supports.
 2015-12-19 18:19:37 -08:00
Andrew Gaul 04f50b7363 Annotate JcloudsIntegrationTest with @Test 
For consistency with jclouds tests.
 2015-12-19 13:09:07 -08:00
Andrew Gaul d1dcc1abee Use testng annotations for JcloudsIntegrationTest 
This matches what the underlying test does and resolves a spurious
issue with testPutObjectStream.
 2015-12-19 00:41:27 -08:00
Andrew Gaul bc993cc3cb Skip testMaximumMultipartUpload for remote backend 
This test dominates run-time due to issuing 10,000 RPCs.
 2015-12-16 19:21:59 -08:00
Andrew Gaul ee55828375 Emit NotImplemented when setting XML ACLs 2015-12-16 19:19:44 -08:00
Andrew Gaul d9b3753f49 Do not set expires when creating blobs 
No tests currently check expires and setting it causes problems for
Swift-based backends.
 2015-12-15 17:22:25 -08:00
Andrew Gaul 3151754ebc Use container public-read for Swift test 
Swift does not support blob-level ACLs.
 2015-12-15 07:09:22 -08:00
Derrek Leute d8c73bc86d URL encode blob names when client requests it 
This allows support for characters which XML 1.0 cannot represent and
improves compatibility with newer versions of the AWS CLI.
Fixes #104.
 2015-12-14 17:10:45 -08:00
Andrew Gaul 23cff3a61d Do not require credentials for local blobstores 2015-12-14 16:01:35 -08:00
Andrew Gaul 9e033c677f Require credentials for aws-v2 authentication 2015-12-14 15:48:19 -08:00
Andrew Gaul 9af1d14dcc Provide content length to AmazonS3.putObject 
Prevents warnings of the form:

WARNING: No content length specified for stream data.  Stream contents
will be buffered in memory and could result in out of memory errors.
 2015-12-14 00:31:19 -08:00
Andrew Gaul 177d9b7566 Handle possibly-null HttpResponse 2015-12-13 23:43:22 -08:00
Andrew Gaul fa215ecb8a Factor out calls to get blobStoreType 2015-12-13 22:58:35 -08:00
Andrew Gaul e237ee9ebb Disable ETag checking for Google Cloud Storage 
Google Cloud Storage ETag is opaque and not an MD5 hash.
 2015-12-12 20:40:26 -08:00
Andrew Gaul c45dde3cbe Skip testMaximumMultipartUpload with large sizes 2015-12-12 13:00:48 +08:00
Andrew Gaul 0581c747bd Add missing @Override annotation 
Found via error-prone.
 2015-12-12 11:24:59 +08:00
Andrew Gaul 33fac374a6 Use container public-read for Azure test 
Azure does not support blob-level ACLs.
 2015-12-12 11:24:58 +08:00
Andrew Gaul eccf6d1b1f Disable ETag checking for Azure 
Azure ETag is opaque and not an MD5 hash.  References #96.
 2015-12-12 11:24:58 +08:00
Rita Zhang a5221f5f8e Upload two parts in testBigMultipartUpload 2015-12-03 16:31:42 +08:00
Andrew Gaul ed29debdf9 Upgrade to aws-java-sdk-s3 1.10.38 
This addresses an issue with the SDK providing an incorrect
Content-Type when uploading a multipart part.  Fixes #80.
 2015-12-02 22:24:47 +08:00
Andrew Gaul 01c43c9f1b Assert expected content length 
This makes some errors more obvious than comparing the content data.
 2015-12-02 22:18:28 +08:00
Andrew Gaul 9210cd679e Map IllegalArgumentException to HTTP 400 2015-11-20 22:31:06 -08:00
Andrew Gaul 67839acef1 Map UnsupportedOperationException to HTTP 501 2015-11-20 22:09:24 -08:00
Andrew Gaul 22ed1c542c Map KeyNotFoundException to NoSuchKey 
Unfortunately jclouds inconsistently throws ContainerNotFoundException
and KeyNotFoundException.
 2015-11-20 22:09:24 -08:00
Andrew Gaul bc2b18ee2b Add support for conditional get 
Fixes #77.
 2015-11-19 18:34:56 -08:00
Rita Zhang 71e2d61377 Allow anonymous access mode to use private buckets 
Fixes #94.
 2015-11-16 21:20:22 -08:00
Andrew Gaul 2207dfbd21 Basic support for POST uploads 
This commit lacks support for policies.  References #73.
 2015-11-14 23:22:05 -08:00
Andrew Gaul 1797992e02 Do not emit response body for HEAD requests 2015-11-11 16:03:42 -08:00
Andrew Gaul 22d57b1f18 Remove redundant private modifier from enum 2015-10-26 13:11:14 -07:00
Andrew Gaul 732ef5d260 Correct RandomByteSource.read return value 
Previously read returned a value between -128 and 127.  -1 indicates
end of stream, causing issues for callers.  Instead return values
between 0 and 255 as intended.
 2015-10-24 20:53:11 -07:00
Andrew Gaul 50a3024362 Work around incorrect Content-Type 
References #80.
 2015-10-20 22:17:56 -07:00
Alex Hall b7e98d1ce3 Test uploading large parts in multipart upload 
This commit exposes a AWS Java SDK bug.  References #80.
 2015-10-20 22:17:56 -07:00
Andrew Gaul d4f20646b4 Add test for 10,000 part MPU 
References #80.
 2015-10-20 14:44:17 -07:00
Andrew Gaul 97d467a698 Enforce no part number during complete MPU 
References #80.
 2015-10-20 14:41:42 -07:00
Andrew Gaul 274bf8a32e Handle listing a public-read container 
Also handle checking existence.
 2015-10-15 20:14:41 -07:00
Andrew Gaul 0faa68ccfc Honor jclouds.region with Swift blobstores 
jclouds has inconsistent support for regions due to provider
differences.  AWS providers a single global namespace while Swift
scopes per-endpoint.  Tested with Rackspace Cloud Files in HKG and ORD
regions.  Fixes #81.
 2015-10-14 13:08:29 -07:00
Andrew Gaul 618c35054c Do not check authorization when in anonymous mode 2015-10-11 11:09:04 -07:00
Andrew Gaul cad436fb95 Do not set Content-MD5 during copy part 
The previously used MD5 covered the entire object, not just the range
request.  Uncovered by ceph/s3-tests#75.
 2015-09-02 11:31:46 -07:00
Andrew Gaul 1d8dfea9c7 Use args4j for argument parsing 2015-09-01 11:12:34 -07:00
Nate Rosenblum f40693f214 Terminate on s3proxy start failure 
E.g. when bind(2) fails.
 2015-08-22 16:28:02 -07:00
Andrew Gaul 095d9aa94d Check anonymous access configuration first 
Previously, when configured for anonymous access and a client tried to
authenticate, S3Proxy would create a signature with a non-existent
credential.  Instead check anonymous access configuration first to
avoid a NullPointerException.
 2015-08-05 17:14:10 -07:00
Andrew Gaul 0daa04a3ed Emit NotImplemented for unknown x-amz headers 2015-07-30 21:49:44 -07:00
Andrew Gaul b85dc31d48 Call HttpServletRequest.getWriter 
This avoids explicit wrapping with OutputStreamWriter.
 2015-07-30 21:00:54 -07:00
Andrew Gaul cc5f1e259f Issue BlobStore calls before opening Writer 
This allows Jetty to propagate errors from the calls correctly.
Follow on to dba252bb99.
 2015-07-30 20:50:48 -07:00
Andrew Gaul 96a6d188a3 Set Content-MD5 during emulated copy part 
Dead store found with FindBugs.  References #56.
 2015-07-30 20:32:03 -07:00
Andrew Gaul cdbe139234 Remove munging of Google Cloud Storage ETag 
Clients need the original value for conditional GETs.
 2015-07-30 14:29:23 -07:00
Andrew Gaul 8e5fecaa91 Emulate multipart copy 
Fixes #56.
 2015-07-30 14:19:09 -07:00
Andrew Gaul 81c3fe9fc7 Consistently compared unquoted ETags 
Also use the ETag from listing parts which resolves an issue with
Swift multipart upload due to use of quoted ETags in the client S3
request and unquoted in underlying Swift request.
 2015-07-30 14:01:43 -07:00
Andrew Gaul 4684c1efe0 Do not test Content-Language with Swift 
Swift does not support this header.
 2015-07-29 16:59:18 -07:00
Andrew Gaul dba252bb99 Call completeMultipartUpload before opening writer 
This allows Jetty to propagate errors from completeMultipartUpload
correctly.
 2015-07-29 16:22:18 -07:00
Andrew Gaul 50871a55ec Partially emulate arbitrary markers with GCS 
S3 supports arbitrary keys for the marker while GCS only supports its
opaque marker.  Emulate the common case for GCS by mapping the last
key from a listing to the corresponding previously returned marker.
 2015-07-28 15:49:32 -07:00
Andrew Gaul 19aa01c090 Partially emulate arbitrary markers with Azure 
S3 supports arbitrary keys for the marker while Azure only supports
its opaque marker.  Emulate the common case for Azure by mapping the
last key from a listing to the corresponding previously returned
marker.
 2015-07-28 05:15:07 -07:00
Andrew Gaul f94d1834c9 Add test for multipart copy 
S3Proxy does not yet support this so we expect NotImplemented.
References #56.
 2015-07-25 13:04:05 -07:00
Andrew Gaul aed1a3ab3c Emit NotImplemented for multipart copy 
References #56.
 2015-07-23 23:14:36 -07:00
Andrew Gaul 566552a1e3 Split larger S3 MPU parts into smaller Azure parts 
Fixes #58.
 2015-07-17 16:06:48 -07:00
Timur Alperovich 763a004177 Enable delimiter support. 
Enables support for the delimiter option.

Amends a recursive test to expect that the delimiter is included in
the common prefix name.
 2015-07-14 13:43:48 -07:00
Andrew Gaul 2a403aa494 Temporarily disable testBlobListRecursive 
Regression due to jclouds delimiter changes.
 2015-07-14 11:33:39 -07:00
Andrew Gaul be3f38b937 Allow anonymous access via bucket and object ACLs 
Fixes #44.
 2015-07-13 13:58:03 -07:00
Andrew Gaul 757ed5acbe Add missing final modifier to classes 2015-07-12 23:27:35 -07:00
Andrew Gaul 2fc3d9b665 Remove unneeded builder assignments 2015-07-12 23:06:57 -07:00
Andrew Gaul 2a1e7b5da1 Add test for single-part upload metadata 
Previously we only checked metadata in multi-part upload.
 2015-07-10 16:06:38 -07:00
Ka-Hing Cheung 16bb2082f4 get rid of default blob store 2015-07-07 14:22:35 -07:00
Andrew Gaul 58c4600d3f Use native prefix support instead of emulation 2015-06-30 09:52:38 -07:00
Andrew Gaul 46d1fc21e9 Check actual content metadata during MPU test 2015-06-26 18:15:28 -07:00
Ka-Hing Cheung a1c1129b53 run all tests in parallel 
need to strip expect headers from jclouds to workaround an issue,
which maybe https://issues.apache.org/jira/browse/JCLOUDS-905
 2015-06-24 18:26:36 -07:00
Ka-Hing Cheung 5098722b8b run test with http 2015-06-24 18:25:34 -07:00
Ka-Hing Cheung 8f60c1e076 add jclouds test 2015-06-24 14:12:15 -07:00
Ka-Hing Cheung 9b61608352 re-factor out s3proxy launch function 2015-06-24 14:04:57 -07:00
Ka-Hing Cheung 67e4ff80b2 downgrade to only use assertj 1.7.0 features 
also fixed the incorrect equals assertion
 2015-06-24 13:25:50 -07:00
Andrew Gaul 0ed4d6c568 Include Content-Range header in range requests 
Fixes #63.
 2015-06-17 11:28:31 -07:00
Andrew Gaul 4b01ffbba2 Handle illegal ranges in range requests 2015-06-16 21:42:46 -07:00
Andrew Gaul 3690079f09 Correct actual and expected variable names 2015-06-05 19:51:48 -07:00
Andrew Gaul bff39d143e Use random input for multipart payload test 2015-06-05 19:48:06 -07:00
Ka-Hing Cheung f17d968a33 allow endpoint to be missing when secure endpoint is set 2015-05-13 17:45:51 -07:00
Ka-Hing Cheung 6e4ebad1c8 add a way to listen to both http and https 2015-05-13 17:45:51 -07:00
Ka-Hing Cheung c26693490c correctly follow the host header documentation 
https://docs.aws.amazon.com/AmazonS3/latest/dev/VirtualHosting.html#VirtualHostingSpecifyBucket
 2015-05-11 17:53:58 -07:00
Andrew Gaul 96e1a65968 Use correct BlobStore for fake metadata 
Previously we could NPE when using the BlobStoreLocator.
 2015-05-08 16:11:07 -07:00
Ka-Hing Cheung c1b8234596 decode before splitting 2015-05-08 15:46:55 -07:00
Andrew Gaul 009a71d06c Remove unintentional log message 2015-05-05 11:08:39 -07:00
Andrew Gaul d47f607544 Enable more Checkstyle checks 
Also remove useless RedundantThrows check.
 2015-05-02 14:50:06 -07:00