mirror
/
s3proxy
kopia lustrzana https://github.com/gaul/s3proxy
1
0
Forkuj 0
Kod Zgłoszenia Packages Projekty Wydania Wiki Aktywność
650 Commity
10 Gałęzie
27 Tagi
5,8 MiB
Wykres commitów

434 Commity (blobstore-locator)

Autor SHA1 Wiadomość Data
Andrew Gaul 7116f29199 Add NullBlobStore 
This throws away all data during object creation and reconstructs it
with NUL bytes.  Clients can use this middleware for performance
testing or to simulate very large objects.  Fixes #131.
 2017-03-08 22:53:16 -08:00
Andrew Gaul 92ef32f860 Address deprecated MultipartPart.create calls 2017-03-08 22:35:55 -08:00
Andrew Gaul 9af67702ad Always emit LastModified header in list parts 
Some clients like jclouds expect this to be present.  Fixes #195.
 2017-03-08 14:10:09 -08:00
shenghu b265fed46f Add verification of override parameters for presigned url. 2017-03-04 17:16:39 -08:00
Andrew Gaul 2ddce6af86 Get Azure maximum part size dynamically 
This size will increase from 4 to 100 MB in jclouds 2.1.0.
References #191.
 2017-03-01 12:52:08 -08:00
Andrew Gaul 64c288ef43 Skip MD5 checking on GET for Azure 
Also use constant name instead of string.
 2017-03-01 12:52:07 -08:00
Andrew Gaul 5ee5cf6c53 Prepend S3Proxy version in User-Agent header 
This only includes the version in the main package and not the unit
test.  Fixes #136.
 2017-02-27 09:48:37 -08:00
Philipp Marx d6ca2fb280 Add support for ignoring unknown parameters 
Presigned URLs may contain additional parameters in the URL which will
be used “as is” by the client uploading/downloading content from S3.

CLOSES #138
 2017-02-22 10:39:37 -08:00
Andrew Gaul f5d2fa9a0b Put each subresource on its own line 2017-02-21 16:34:13 -08:00
shenghu 3b4e0c8fdf This change makes S3Proxy able to validate presigned url which 
has override parameters, i.e. "content-disposition",
"response-content-encoding".
 2017-02-22 00:32:21 +08:00
Andrew Gaul 66c3c73fc7 Remove stale TODOs 2017-02-20 13:27:51 -08:00
oxalis-gps 09d744763d Added Docker environment variable "S3PROXY_CORS_ALLOW_ALL" 
refs issues #142
refs Pull #144

The default value is false
 2017-02-16 13:49:48 -08:00
Andrew Gaul e581d99517 Rename S3AwsSdkTest to AwsSdkTest 2017-02-15 23:33:19 -08:00
Andrew Gaul 05c5643681 Update copyright year 2017-02-15 23:15:16 -08:00
Andrew Gaul 4295f72be5 Move tests from S3ProxyTest to S3AwsSdkTest 
This makes testing more consistent and gives better coverage of the
more widely-used AWS SDK.  Fixes #147.
 2017-02-15 23:11:11 -08:00
Andrew Gaul bbcdf4391f Ignore long-running test 2017-02-15 18:17:57 -08:00
Andrew Gaul 9329b161f2 Allow unsigned payloads 
Fixes #184.
 2017-02-15 18:17:57 -08:00
Andrew Gaul acf125d8e9 Use secure endpoint in tests 2017-02-15 18:17:56 -08:00
Andrew Gaul dc0c8663b8 Uprade to aws-java-sdk-s3 1.11.84 
Address various deprecations.
 2017-01-25 13:06:01 -08:00
Andrew Gaul 48f7fe2742 Remove calls to Throwables.propagate 
These are not needed and Guava 21 removes this API.
 2017-01-19 22:37:40 -08:00
Andrew Gaul 13b10997c8 Make private methods static where possible 
Found via error-prone.
 2016-12-20 13:28:18 -08:00
Andrew Gaul 8052e67eac Remove unused parameters 
Found via PMD.
 2016-11-21 13:09:01 -08:00
Andrew Gaul 4336000409 Allow Quiet element in DeleteMultipleObjects 
Fixes #173.
 2016-11-20 16:33:44 -08:00
Andrew Gaul 0003d011f3 Correct test exception message 2016-11-16 10:54:46 -08:00
Andrew Gaul 23bfecd6be Emit AccessDenied if client sends wrong auth type 
This allows fallback from v4 to v2.  Also handle anonymous mode
explicitly.
 2016-11-14 22:24:01 -08:00
Andrew Gaul 3d9a7120ce Ensure S3Exception has message 
Also add missing toString implementation.
 2016-11-14 15:39:44 -08:00
Andrew Gaul 04657867b8 Add configuration for AWS authentication V4 
Supports aws-v2, aws-v4, aws-v2-or-v4, and none.  Fixes #156.
 2016-11-14 00:59:46 -08:00
Andrew Gaul 80e256435e Fix indentation and work around wrong visibility 
Found via Checkstyle 7.2.
 2016-11-14 00:58:22 -08:00
Andrew Gaul 778002e7f5 Format LastModified without millisecond precision 
Found via s3verify.
 2016-11-13 18:52:56 -08:00
Andrew Gaul 3188605ea7 More strictly check bucket names 
Found via s3verify.
 2016-11-13 18:42:12 -08:00
Andrew Gaul 85bd7432a7 Limit GET Bucket maxKeys to 1000 
Found via s3verify.
 2016-11-13 17:45:25 -08:00
Andrew Gaul c0c01c2e7f Add Location header to PutBucket response 
Found via s3verify.
 2016-11-11 18:42:37 -08:00
Andrew Gaul c707ed4a6d Correct some S3 error responses 
Found via s3verify.
 2016-11-11 16:17:00 -08:00
Andrew Gaul 0dc133036b Ignore part-number-marker when zero 
S3Proxy does not support pagination but it can support this initial
no-op value.  Fixes #162.
 2016-11-07 22:13:22 -08:00
shenghu 4fb4a718b9 Refactor S3ProxyHandler to be agnostic to Jetty. Added a new file, 
S3ProxyHandlerJetty, to have codes related to Jetty.
 2016-11-07 22:07:55 -08:00
Andrew Gaul 164ef077b3 Emit correct EncodingType stanza 
Previously this contained a spurious hyphen which confused clients
that checked the XML response instead of relying on their HTTP request
parameters.  Fixes #158.
 2016-10-05 04:45:32 -07:00
Andrew Gaul bb64884f09 Return ETag as ETag instead of Content-MD5 
Previously S3Proxy only returned ETag if jclouds populated the
possibly-null Content-MD5 field.  Providers like Azure do not store
MD5 in ETag and thus S3Proxy would not return any ETag.  Also do not
return unneeded Content-MD5 header.  Fixes #151.
 2016-09-12 22:53:54 -07:00
Andrew Gaul 9f0bc8a17b Honor SIGINT in Docker container 
Use intermediate shell script to forward environment variables then
exec java.
 2016-09-04 22:21:54 -07:00
Andrew Gaul 224bb8b2f6 Unwrap AuthorizationException to ACCESS_DENIED 2016-08-29 23:02:15 -07:00
Andrew Gaul 5c3b4494d2 Use generic get blob behavior for anonymous access 
Previously S3Proxy tried to be too clever about pushing ACL
enforcement into the provider.
 2016-07-19 03:20:07 -07:00
Andrew Gaul 01f956052e Add property to include permissive CORS response 
References #142.
 2016-07-19 03:20:07 -07:00
Zvika Gart fcbc14780b Added support for S3 response header override options 2016-07-15 23:13:54 -07:00
Andrew Gaul 6ecd8d789f Upgrade to aws-java-sdk-s3 1.11.18 
Remove deprecated V4 configuration which is now default.
 2016-07-14 14:14:06 -07:00
Andrew Gaul 1513606992 Inspect all Throwables for TimeoutException 2016-06-26 23:23:13 -07:00
Andrew Gaul b7eb53b7c7 Hoist timeout handling to top-level 2016-06-26 13:56:30 -07:00
Andrew Gaul 17f8fae37b Add support for B2 provider 
Fixes #98.
 2016-06-23 11:56:52 -07:00
Andrew Gaul 0658df372f Remove empty CommonPrefixes stanza 
This is not needed and confuses s3-tests.
 2016-06-21 11:21:21 -07:00
Andrew Gaul cd3a208243 Add missing quirks checks in tests 
Missing Cache-Control and Content-Language.
 2016-06-17 18:29:53 -07:00
Andrew Gaul 187eb065aa Add support for listing multipart uploads 
Only supported in Azure, B2, and S3.  Fixes #118.
 2016-06-14 15:22:45 -07:00
Andrew Gaul 72138e1278 Handle null AccessControlPolicy.aclList 
Also add toString methods.
 2016-06-06 17:01:48 -07:00
Andrew Gaul 99d8e5e8d7 Add missing @Override annotations 
Found via error-prone.
 2016-06-03 10:41:05 -07:00
Andrew Gaul 0053102df5 Add test for aborting multi-part uploads 2016-06-02 07:42:17 -07:00
Andrew Gaul 519cd9313c Remove stub blob during multi-part upload 
References #135.
 2016-06-02 07:42:16 -07:00
Andrew Gaul 2eaf7de68f Handle null ETag during multipart upload 
Fixes #135.
 2016-06-01 12:51:41 -07:00
Andrew Gaul d6c81c9b64 Correct issues with encoding of plus character 
Also remove some questionable decoding.
 2016-06-01 11:22:16 -07:00
Andrew Gaul e7bb6ddb47 Avoid NullPointerException when quoting ETags 2016-05-27 18:07:41 -07:00
Andrew Gaul a4b96924eb Skip time-limited signed requests 
jclouds aws-s3 provider supports these but s3 does not.
 2016-05-12 12:06:03 -07:00
Andrew Gaul 2cfcc33bb8 Upgrade to aws-java-sdk-s3 1.10.76 
Also address deprecations.
 2016-05-05 22:22:04 -07:00
Andrew Gaul 2296b8228b Add configuration to ignore unknown headers 
Fixes #130.
 2016-03-12 23:15:45 -07:00
Andrew Gaul d878642b5d Add quirks for google-cloud-storage 2016-03-09 13:35:02 -08:00
Andrew Gaul eece05f02e Fix path-based GCS credentials 
Providing both credential and the credential property used only the
latter, which was not expanded when credential is a file.  Also expand
credential in tests.  References #128.
 2016-03-09 13:35:02 -08:00
Andrew Gaul 6292d28c9c Add EventualBlobStore 
This models eventually-consistent behavior.  This implementation uses
two buckets and client writes going to the first bucket and reads to
the second.  Operations later replicate from the first to the second
with a variable delays.  A more complete implementation could flap
between the strongly- and eventually-consistent buckets and arbitrary
reorder operations which conclude with last-writer wins.  Fixes #65.
 2016-03-01 17:13:39 -08:00
Andrew Gaul 0351ee73dc Expand GCS credential files into PEMs 2016-02-26 17:45:33 -08:00
Andrew Gaul e670537822 Handle Swift quirks in jclouds tests 2016-02-20 00:49:16 -08:00
Andrew Gaul 2062aadfeb Handle conditional HEAD object requests 2016-02-18 11:48:48 -08:00
Andrew Gaul 52ba2c843f Add support for conditional copy object 
Fixes #113.
 2016-02-16 20:31:12 -08:00
Andrew Gaul d66b539f2c Reduce visibility of methods 2016-02-16 16:16:51 -08:00
Andrew Gaul 884660ea05 Correct typo 2016-02-11 15:40:45 -08:00
Andrew Gaul a4385daa45 Fix v4 signing of queries with special characters 
Reference:

https://docs.aws.amazon.com/general/latest/gr/sigv4-create-canonical-request.html
 2016-02-09 11:52:04 -08:00
Andrew Gaul 9c3d9409ac Read from PushBackInputStream not InputStream 
Previously S3Proxy truncated create bucket requests.
 2016-02-08 22:54:39 -08:00
Andrew Gaul 7198c9088c Do not close InputStream for each uploaded subpart 2016-02-04 00:09:41 -08:00
Andrew Gaul 997dc9707c Skip expires tests when backend lacks support 2016-02-03 23:12:26 -08:00
Andrew Gaul 6d7a46296e Skip zero keys tests when backend lacks support 2016-02-01 12:29:43 -08:00
Andrew Gaul 1ae596e870 Support list multipart upload parts with Azure 
Map the Azure subparts back into S3 parts.
 2016-01-31 13:57:52 -08:00
Andrew Gaul bc68e2cb2e Do not run marker tests when backend lacks support 2016-01-31 12:50:04 -08:00
Andrew Gaul 82d95a2746 Use getObject and generatePresignedUrl short forms 2016-01-30 21:26:17 -08:00
Andrew Gaul 940286b022 Correct cut and paste errors in copy tests 2016-01-30 21:26:17 -08:00
Andrew Gaul 65fd80036f Do not run ACL tests when backend lacks support 2016-01-30 21:26:17 -08:00
Andrew Gaul cea4a42922 Use decoded content length when provided 
Previously S3Proxy provided an incorrect Content-Length header when
using V4 signing which providers rejected.
 2016-01-30 19:13:04 -08:00
Andrew Gaul 983c2b003e Remove stale TODO 2016-01-30 17:11:47 -08:00
Andrew Gaul e292ebaccb Consistently parse XML with Jackson 2016-01-29 17:50:22 -08:00
Andrew Gaul 511c968ce7 Add test for V4 URL signing 
Currently fails.
 2016-01-29 17:30:13 -08:00
Andrew Gaul 8d8e8ff290 Upgrade to aws-java-sdk-s3 1.10.50 
This release includes an implementation of AccessControlList.equals.
References #116.

Release notes:

https://aws.amazon.com/releasenotes/8286007886477675
 2016-01-29 09:50:14 -08:00
Andrew Gaul bc1833ed2e Remove BlobAccess TODO 
jclouds cannot provide BlobAccess since providers do not return these
ACLs.
 2016-01-26 19:28:50 -08:00
Andrew Gaul 6b0a6d8cdc Ensure failing tests raise NotImplemented error 2016-01-24 13:52:56 -08:00
Andrew Gaul f202207c63 Map a subset of XML ACLs to canned ACLs 
Fixes #116.
 2016-01-23 15:34:03 -08:00
Andrew Gaul 0e3a7ef843 Add missing EmailAddress field 
This allows S3Proxy to return the correct NotImplemented error code
instead of InternalError.  References #116.
 2016-01-23 15:16:38 -08:00
Andrew Gaul d557af774d Replace query parameters in anonymous requests 2016-01-23 15:06:04 -08:00
Andrew Gaul 19e2426220 Map a subset of XML ACLs to canned ACLs 
Fixes #116.
 2016-01-21 18:24:57 -08:00
Andrew Gaul 587a4a6670 Centralize blobstore quirks 2016-01-17 23:10:28 -08:00
Andrew Gaul 744d126331 Add support for Cache-Control header 
Fixes #115.
 2016-01-17 12:08:52 -08:00
Andrew Gaul 542c9b8112 Add Accept-Ranges header to partial get requests 
Fixes #66.
 2016-01-15 22:37:19 -08:00
Andrew Gaul 05bd2c1158 Use uploadId for stub blob name 
This makes MPU overwrites of an existing blob atomic.  Fixes #108.
 2016-01-14 15:11:03 -08:00
Andrew Gaul 3aed140a15 Avoid mutable ArrayList 
Latest AWS SDK handles immutable List.  Also tidy up line wrapping.
 2016-01-13 16:12:26 -08:00
Andrew Gaul ef3a7a6adf Add JcloudsS3BlobSignerLiveTest 
References #62.
 2016-01-12 16:20:03 -08:00
Andrew Gaul fb3bd591d6 Add JcloudsBucketsLiveTest 
Uncovers several missing features:

Bucket logging
Bucket payer
XML ACLs

References #62.
 2016-01-12 16:20:02 -08:00
Andrew Gaul 9c5499a05e Add JcloudsS3ClientLiveTest 
Uncovers several missing features:

Cache-Control header
conditional copies
public-read-write canned ACLs
XML ACLs

References #62.
 2016-01-12 16:20:02 -08:00
Andrew Gaul ae95538647 Add JcloudsS3ContainerIntegrationLiveTest 
References #62.
 2016-01-12 16:19:56 -08:00
Andrew Gaul f4212571b0 Rename suite to JcloudsS3BlobIntegrationLiveTest 2016-01-11 20:23:13 -08:00
Andrew Gaul 91429586dc Reuse encodingType local 2016-01-11 20:01:50 -08:00