mirror
/
s3proxy
kopia lustrzana https://github.com/gaul/s3proxy
1
0
Forkuj 0
Kod Zgłoszenia Packages Projekty Wydania Wiki Aktywność
1 029 Commity
10 Gałęzie
22 Tagi
2,5 MiB
Wykres commitów

596 Commity (master)

Autor SHA1 Wiadomość Data
Andrew Gaul a80e75a20f Add test for anonymous authentication 
Fixes #200.
 2018-01-08 22:56:28 -08:00
Andrew Gaul c410df593d Parameterize TestUtils.startS3Proxy 2018-01-08 22:56:28 -08:00
Random Coder 0f26d65846 Add check for non anonymous identity before extracting auth header 
Fixes #251.
 2018-01-08 22:56:18 -08:00
Andrew Gaul 2905b720b8 Update copyright year 2018-01-04 14:57:47 -08:00
Andrew Gaul aee169e403 Check stream closed in read 
This matches TestUtils.randomByteSource.
 2018-01-04 14:54:54 -08:00
baul ddfba4e7a4 v2/v4 auth improve 
* 15 minitues timeskew
* Add x-amz-date header or query parameter check
* Change the timeskew logic to first get client req auth type
* When v2,x-amz-date header format is rfc2616,when v4,is iso8601
* If have both x-amz-date header and date header in v2 auth,date value in
  stringtosign is x-amz-date header value,CanonicalizedAmzHeaders
  have no x-amz-header.
  Ref Delete example in site:
  http://docs.aws.amazon.com/AmazonS3/latest/dev/RESTAuthentication.html
* Fix v2 query auth:
  If expires is nil ,does not mean that the auth type is not query
  auth type.
  Ref http://docs.aws.amazon.com/AmazonS3/latest/dev/RESTAuthentication.html#RESTAuthenticationQueryStringAuth
  It says that 'Additionally, you can limit a pre-signed request by specifying an expiration time.'
 2018-01-02 17:32:49 -08:00
Andrew Gaul 94bf8ca88e Add missing @Nullable 2017-12-23 08:03:29 -08:00
Andrew Gaul d380661f76 Use a single jclouds thread pool for all backends 
S3Proxy does not seem to use the jclouds thread pools but limiting
this to one makes sense for multiple configurations.  References #180.
 2017-11-08 18:11:17 -08:00
Andrew Gaul 11f82ec997 Factor out properties parsing into Builder 2017-11-08 17:34:09 -08:00
Andrew Gaul 732a24d990 Allow limiting the number of Jetty threads 
Also name Jetty threads more specifically.  References #180.
 2017-11-08 15:25:57 -08:00
Andrew Gaul f48bd352c4 Consume S3Object InputStream 
Previously these tests leaked HTTP connections from the AWS SDK pool.
 2017-11-08 11:41:35 -08:00
Andrew Gaul 251082f7e7 Support multiple configurations 
Fixes #11.
 2017-11-07 17:06:57 -08:00
Andrew Gaul 6ee33b89f9 Refactor S3Proxy properties parsing 
Also convert a few calls to System.exit into IllegalArgumentException.
References #11.
 2017-11-07 14:27:24 -08:00
Andrew Gaul a09c9e1998 Treat directories as relative paths 
When listing blobs foo/1 and foo/2 with prefix foo and delimiter /,
jclouds returns foo as a relative path, except for Atmos which returns
foo as a directory.  Treat the latter like the former to allow listing
pseudo-directories with Atmos.
fb0f321303 introduced this regression.
Fixes #244.
 2017-10-31 11:43:22 -07:00
Andrew Gaul 35d7c48475 Read-only middleware 
Fixes #217.
 2017-10-08 18:11:14 -07:00
Andrew Gaul 3dd7cfcd42 Address long narrowing issues 
Found via error-prone 2.1.0.
 2017-08-21 19:46:34 -07:00
Andrew Gaul f6c3ab9e75 Add @Nullable annotations where appropriate 
Found via error-prone 2.1.0.
 2017-08-21 19:46:33 -07:00
Andrew Gaul 51aa2e7fcd Address modern Guava deprecations 2017-08-21 15:04:41 -07:00
Andrew Gaul 39c289d649 Allow Docker environment to ignore unknown headers 2017-08-16 18:04:52 -07:00
Andrew Gaul 233559bb2b Do not ignore x-amz-storage-class if not STANDARD 
Clients can set the ignore unknown headers parameter to preserve the
existing behavior.  References #234.
 2017-08-16 18:04:52 -07:00
Andrew Gaul 1a639d1062 Upgrade s3-tests 2017-08-16 14:11:40 -07:00
Andrew Gaul d4802ec54f Emit InvalidArgument on CopyPart invalid range 
References kahing/goofys#212.
 2017-08-06 23:24:58 -07:00
Andrew Gaul 93b80f6373 Do not escape / when URL encoding 
This allows clients to list pseudo-directories with slashes in them.
References kahing/goofys#213.
 2017-08-02 23:06:13 -07:00
Andrew Gaul 9e406531bb Set response Content-Type to application/xml 2017-07-24 18:40:03 -07:00
Andrew Gaul 1e5ebadee4 Do not allow anonymous access to list buckets 
Fixes #236.
 2017-07-22 16:11:29 -07:00
Andrew Gaul 17294be152 Do not set endpoint if it is an empty string 
Also do not pass along endpoint as an override if we set it via the
builder.  Fixes #229.
 2017-07-08 19:45:22 -07:00
Chaithanya Ganta 23705a879b Moving signature creation logic to a separate class 
Signed-off-by: Chaithanya Ganta <ganta@adobe.com>
 2017-07-07 09:06:04 -07:00
Chaithanya Ganta 84885fa8cd Throw proper S3 error code for invalid buck name 
INVALID_BUCKET_NAME for container creation request, else NO_SUCH_BUCKET
 2017-06-02 13:47:51 +02:00
Chaithanya Ganta 6b92cc06f4 Validate bucket name before processing any request 2017-06-02 13:47:51 +02:00
Andrew Gaul 742820a93d Allow capitals as first character in bucket name 
Curiously Amazon permits mixed case but documents that only lowercase
letters are allowed:

https://docs.aws.amazon.com/AmazonS3/latest/dev/BucketRestrictions.html

Fixes #226.
 2017-05-24 22:02:48 -07:00
Rached Ben Mustapha e23e9b78e4 Handle HttpResponseException in S3ProxyHandlerJetty 2017-05-12 12:33:40 -07:00
Andrew Gaul 525c6e2275 Override PrintStream.write as well as print 
args4j inconsistently uses both methods for output.  Workaround for
kohsuke/args4j#149.
 2017-05-08 11:33:12 -07:00
Kishore battula 84ec8e22da added whether console is attached or not 2017-05-08 11:33:12 -07:00
Andrew Gaul 9c74851c43 Address error-prone 2.0.19 issues 2017-05-06 17:00:38 -07:00
Andrew Gaul e21b467110 Improve IP address validation 2017-05-06 12:27:46 -07:00
Michael Wittig 4f14b8607a HTTP POST Requests using AWS Signature Version 4 2017-05-05 12:11:54 -07:00
kishore d3c77384f1 Allow anonymous access to public-read containers 2017-05-03 00:02:05 -07:00
Andrew Gaul 8db6ce2a92 Replace parameters whitelist with blacklist 
Mimic Amazon behavior of ignoring unknown parameters and only emit
NotImplemented when present in blacklist.  Fixes #189.
 2017-04-25 14:03:43 -07:00
Andrew Gaul 6b30e66127 Do not set virtual host when it is an empty string 
References #210.
 2017-04-25 12:52:24 -07:00
Eric Urban 16af493cad Update docker script to allow for configuration of virtualhost 2017-04-25 10:40:01 -07:00
Andrew Gaul d427c8f7df Get Azure maximum part size dynamically 
This size will increase from 4 to 100 MB in jclouds 2.1.0.
References #191.  Follow on to 2ddce6af86.
 2017-04-24 12:16:38 -07:00
Andrew Gaul 0105fd66df Handle X-Amz-Expires header during authentication 
Found via s3verify.
 2017-04-24 04:00:48 -07:00
Andrew Gaul d005f452de Prefer failBecauseExceptionWasNotThrown over fail 2017-04-18 11:53:09 -07:00
Andrew Gaul 9ddcc5b2e6 Remove stale TODO 2017-04-16 21:08:45 -07:00
Andrew Gaul dea4b3bae3 Handle testMultipartUploadAbort azureblob quirks 2017-04-16 19:50:18 -07:00
Andrew Gaul fc204c0d49 Only create multipart stub blob when required 2017-04-16 19:04:29 -07:00
Andrew Gaul b9730a4b2f Make testSpecialCharacters compatible with Azure 
Also sort prefix chars by hex values and remove unneeded delimiter.
 2017-04-12 00:59:10 -07:00
Andrew Gaul d8546c00ce URL encode CommonPrefix when requested 2017-04-12 00:58:43 -07:00
Andrew Gaul 7d53dead20 Expand MPU test to check MPU and part listings 2017-04-11 19:30:12 -07:00
Andrew Gaul d1cc178f77 Skip tests which B2 does not support 2017-04-09 13:04:38 -07:00
Andrew Gaul 76d7bd911d Include cause when throwing S3Exception 2017-04-09 13:04:38 -07:00
Andrew Gaul 437ae2068d Stream B2 writes 
Previously S3Proxy buffered B2 writes on disk to create a repeatable
payload for jclouds which calculated the SHA-1 checksum.  The B2
service no longer requires a checksum so we can remove the buffering.
 2017-04-09 13:04:38 -07:00
Andrew Gaul 65e65bc214 Ensure that testBigMultipartUpload meets min size 
Previously this failed on Backblaze B2.
 2017-04-08 12:24:40 -07:00
Andrew Gaul 802b8510cf Set content bits only when provider supports them 
Previously the AWS SDK choked on these null values.
 2017-04-08 12:24:39 -07:00
Raphaël Pinson d3023e1695 Use JCLOUDS_ENDPOINT and JCLOUDS_REGION in run-docker-container.sh 2017-03-30 09:01:16 -07:00
Andrew Gaul 014f558c51 Support V4 Signed URLs 
Fixes #160.  Fixes #164.  Fixes #204.
 2017-03-16 19:29:03 -07:00
shenghu 8b2d056d40 Allow endpoint of s3proxy to have service path 
Sometimes the service is not the only one under a domain.
This commit adds s3proxy.service-path to allow s3proxy to be deployed
with a context path. Fixes #48.
 2017-03-12 16:55:49 -07:00
Andrew Gaul 8224f6fe6e Interpret chunked encoding with anonymous access 
Also check authentication type correctly in the constructor to handle
configurations with anonymous access but set identity and credential.
References #145.
 2017-03-10 19:46:15 -08:00
Kishore battula 842eaaf719 Added log prefix in test logback 2017-03-10 02:50:20 -08:00
Kishore battula bf281edebc Added log prefix in logback 2017-03-10 02:50:20 -08:00
Andrew Gaul a5692b986b Return zero when listing blob size 
This makes more sense than returning the 8 byte stub blob size.
References #131.
 2017-03-09 10:48:56 -08:00
Andrew Gaul 26f4232756 Emit LastModified during List Parts 2017-03-08 22:58:40 -08:00
Andrew Gaul 7116f29199 Add NullBlobStore 
This throws away all data during object creation and reconstructs it
with NUL bytes.  Clients can use this middleware for performance
testing or to simulate very large objects.  Fixes #131.
 2017-03-08 22:53:16 -08:00
Andrew Gaul 92ef32f860 Address deprecated MultipartPart.create calls 2017-03-08 22:35:55 -08:00
Andrew Gaul 9af67702ad Always emit LastModified header in list parts 
Some clients like jclouds expect this to be present.  Fixes #195.
 2017-03-08 14:10:09 -08:00
shenghu b265fed46f Add verification of override parameters for presigned url. 2017-03-04 17:16:39 -08:00
Andrew Gaul 2ddce6af86 Get Azure maximum part size dynamically 
This size will increase from 4 to 100 MB in jclouds 2.1.0.
References #191.
 2017-03-01 12:52:08 -08:00
Andrew Gaul 64c288ef43 Skip MD5 checking on GET for Azure 
Also use constant name instead of string.
 2017-03-01 12:52:07 -08:00
Andrew Gaul 5ee5cf6c53 Prepend S3Proxy version in User-Agent header 
This only includes the version in the main package and not the unit
test.  Fixes #136.
 2017-02-27 09:48:37 -08:00
Philipp Marx d6ca2fb280 Add support for ignoring unknown parameters 
Presigned URLs may contain additional parameters in the URL which will
be used “as is” by the client uploading/downloading content from S3.

CLOSES #138
 2017-02-22 10:39:37 -08:00
Andrew Gaul f5d2fa9a0b Put each subresource on its own line 2017-02-21 16:34:13 -08:00
shenghu 3b4e0c8fdf This change makes S3Proxy able to validate presigned url which 
has override parameters, i.e. "content-disposition",
"response-content-encoding".
 2017-02-22 00:32:21 +08:00
Andrew Gaul 66c3c73fc7 Remove stale TODOs 2017-02-20 13:27:51 -08:00
oxalis-gps 09d744763d Added Docker environment variable "S3PROXY_CORS_ALLOW_ALL" 
refs issues #142
refs Pull #144

The default value is false
 2017-02-16 13:49:48 -08:00
Andrew Gaul e581d99517 Rename S3AwsSdkTest to AwsSdkTest 2017-02-15 23:33:19 -08:00
Andrew Gaul 05c5643681 Update copyright year 2017-02-15 23:15:16 -08:00
Andrew Gaul 4295f72be5 Move tests from S3ProxyTest to S3AwsSdkTest 
This makes testing more consistent and gives better coverage of the
more widely-used AWS SDK.  Fixes #147.
 2017-02-15 23:11:11 -08:00
Andrew Gaul bbcdf4391f Ignore long-running test 2017-02-15 18:17:57 -08:00
Andrew Gaul 9329b161f2 Allow unsigned payloads 
Fixes #184.
 2017-02-15 18:17:57 -08:00
Andrew Gaul acf125d8e9 Use secure endpoint in tests 2017-02-15 18:17:56 -08:00
Andrew Gaul dc0c8663b8 Uprade to aws-java-sdk-s3 1.11.84 
Address various deprecations.
 2017-01-25 13:06:01 -08:00
Andrew Gaul 48f7fe2742 Remove calls to Throwables.propagate 
These are not needed and Guava 21 removes this API.
 2017-01-19 22:37:40 -08:00
Andrew Gaul 13b10997c8 Make private methods static where possible 
Found via error-prone.
 2016-12-20 13:28:18 -08:00
Andrew Gaul 8052e67eac Remove unused parameters 
Found via PMD.
 2016-11-21 13:09:01 -08:00
Andrew Gaul 4336000409 Allow Quiet element in DeleteMultipleObjects 
Fixes #173.
 2016-11-20 16:33:44 -08:00
Andrew Gaul 0003d011f3 Correct test exception message 2016-11-16 10:54:46 -08:00
Andrew Gaul 23bfecd6be Emit AccessDenied if client sends wrong auth type 
This allows fallback from v4 to v2.  Also handle anonymous mode
explicitly.
 2016-11-14 22:24:01 -08:00
Andrew Gaul 3d9a7120ce Ensure S3Exception has message 
Also add missing toString implementation.
 2016-11-14 15:39:44 -08:00
Andrew Gaul 04657867b8 Add configuration for AWS authentication V4 
Supports aws-v2, aws-v4, aws-v2-or-v4, and none.  Fixes #156.
 2016-11-14 00:59:46 -08:00
Andrew Gaul 80e256435e Fix indentation and work around wrong visibility 
Found via Checkstyle 7.2.
 2016-11-14 00:58:22 -08:00
Andrew Gaul 778002e7f5 Format LastModified without millisecond precision 
Found via s3verify.
 2016-11-13 18:52:56 -08:00
Andrew Gaul 3188605ea7 More strictly check bucket names 
Found via s3verify.
 2016-11-13 18:42:12 -08:00
Andrew Gaul 85bd7432a7 Limit GET Bucket maxKeys to 1000 
Found via s3verify.
 2016-11-13 17:45:25 -08:00
Andrew Gaul c0c01c2e7f Add Location header to PutBucket response 
Found via s3verify.
 2016-11-11 18:42:37 -08:00
Andrew Gaul c707ed4a6d Correct some S3 error responses 
Found via s3verify.
 2016-11-11 16:17:00 -08:00
Andrew Gaul 0dc133036b Ignore part-number-marker when zero 
S3Proxy does not support pagination but it can support this initial
no-op value.  Fixes #162.
 2016-11-07 22:13:22 -08:00
shenghu 4fb4a718b9 Refactor S3ProxyHandler to be agnostic to Jetty. Added a new file, 
S3ProxyHandlerJetty, to have codes related to Jetty.
 2016-11-07 22:07:55 -08:00
Andrew Gaul 164ef077b3 Emit correct EncodingType stanza 
Previously this contained a spurious hyphen which confused clients
that checked the XML response instead of relying on their HTTP request
parameters.  Fixes #158.
 2016-10-05 04:45:32 -07:00
Andrew Gaul bb64884f09 Return ETag as ETag instead of Content-MD5 
Previously S3Proxy only returned ETag if jclouds populated the
possibly-null Content-MD5 field.  Providers like Azure do not store
MD5 in ETag and thus S3Proxy would not return any ETag.  Also do not
return unneeded Content-MD5 header.  Fixes #151.
 2016-09-12 22:53:54 -07:00
Andrew Gaul 9f0bc8a17b Honor SIGINT in Docker container 
Use intermediate shell script to forward environment variables then
exec java.
 2016-09-04 22:21:54 -07:00
Andrew Gaul 224bb8b2f6 Unwrap AuthorizationException to ACCESS_DENIED 2016-08-29 23:02:15 -07:00
Andrew Gaul 5c3b4494d2 Use generic get blob behavior for anonymous access 
Previously S3Proxy tried to be too clever about pushing ACL
enforcement into the provider.
 2016-07-19 03:20:07 -07:00
Andrew Gaul 01f956052e Add property to include permissive CORS response 
References #142.
 2016-07-19 03:20:07 -07:00
Zvika Gart fcbc14780b Added support for S3 response header override options 2016-07-15 23:13:54 -07:00
Andrew Gaul 6ecd8d789f Upgrade to aws-java-sdk-s3 1.11.18 
Remove deprecated V4 configuration which is now default.
 2016-07-14 14:14:06 -07:00
Andrew Gaul 1513606992 Inspect all Throwables for TimeoutException 2016-06-26 23:23:13 -07:00
Andrew Gaul b7eb53b7c7 Hoist timeout handling to top-level 2016-06-26 13:56:30 -07:00
Andrew Gaul 17f8fae37b Add support for B2 provider 
Fixes #98.
 2016-06-23 11:56:52 -07:00
Andrew Gaul 0658df372f Remove empty CommonPrefixes stanza 
This is not needed and confuses s3-tests.
 2016-06-21 11:21:21 -07:00
Andrew Gaul cd3a208243 Add missing quirks checks in tests 
Missing Cache-Control and Content-Language.
 2016-06-17 18:29:53 -07:00
Andrew Gaul 187eb065aa Add support for listing multipart uploads 
Only supported in Azure, B2, and S3.  Fixes #118.
 2016-06-14 15:22:45 -07:00
Andrew Gaul 72138e1278 Handle null AccessControlPolicy.aclList 
Also add toString methods.
 2016-06-06 17:01:48 -07:00
Andrew Gaul 99d8e5e8d7 Add missing @Override annotations 
Found via error-prone.
 2016-06-03 10:41:05 -07:00
Andrew Gaul 0053102df5 Add test for aborting multi-part uploads 2016-06-02 07:42:17 -07:00
Andrew Gaul 519cd9313c Remove stub blob during multi-part upload 
References #135.
 2016-06-02 07:42:16 -07:00
Andrew Gaul 2eaf7de68f Handle null ETag during multipart upload 
Fixes #135.
 2016-06-01 12:51:41 -07:00
Andrew Gaul d6c81c9b64 Correct issues with encoding of plus character 
Also remove some questionable decoding.
 2016-06-01 11:22:16 -07:00
Andrew Gaul e7bb6ddb47 Avoid NullPointerException when quoting ETags 2016-05-27 18:07:41 -07:00
Andrew Gaul a4b96924eb Skip time-limited signed requests 
jclouds aws-s3 provider supports these but s3 does not.
 2016-05-12 12:06:03 -07:00
Andrew Gaul 2cfcc33bb8 Upgrade to aws-java-sdk-s3 1.10.76 
Also address deprecations.
 2016-05-05 22:22:04 -07:00
Andrew Gaul 2296b8228b Add configuration to ignore unknown headers 
Fixes #130.
 2016-03-12 23:15:45 -07:00
Andrew Gaul d878642b5d Add quirks for google-cloud-storage 2016-03-09 13:35:02 -08:00
Andrew Gaul eece05f02e Fix path-based GCS credentials 
Providing both credential and the credential property used only the
latter, which was not expanded when credential is a file.  Also expand
credential in tests.  References #128.
 2016-03-09 13:35:02 -08:00
Andrew Gaul 6292d28c9c Add EventualBlobStore 
This models eventually-consistent behavior.  This implementation uses
two buckets and client writes going to the first bucket and reads to
the second.  Operations later replicate from the first to the second
with a variable delays.  A more complete implementation could flap
between the strongly- and eventually-consistent buckets and arbitrary
reorder operations which conclude with last-writer wins.  Fixes #65.
 2016-03-01 17:13:39 -08:00
Andrew Gaul 0351ee73dc Expand GCS credential files into PEMs 2016-02-26 17:45:33 -08:00
Andrew Gaul e670537822 Handle Swift quirks in jclouds tests 2016-02-20 00:49:16 -08:00
Andrew Gaul 2062aadfeb Handle conditional HEAD object requests 2016-02-18 11:48:48 -08:00
Andrew Gaul 52ba2c843f Add support for conditional copy object 
Fixes #113.
 2016-02-16 20:31:12 -08:00
Andrew Gaul d66b539f2c Reduce visibility of methods 2016-02-16 16:16:51 -08:00
Andrew Gaul 884660ea05 Correct typo 2016-02-11 15:40:45 -08:00
Andrew Gaul a4385daa45 Fix v4 signing of queries with special characters 
Reference:

https://docs.aws.amazon.com/general/latest/gr/sigv4-create-canonical-request.html
 2016-02-09 11:52:04 -08:00
Andrew Gaul 9c3d9409ac Read from PushBackInputStream not InputStream 
Previously S3Proxy truncated create bucket requests.
 2016-02-08 22:54:39 -08:00
Andrew Gaul 7198c9088c Do not close InputStream for each uploaded subpart 2016-02-04 00:09:41 -08:00
Andrew Gaul 997dc9707c Skip expires tests when backend lacks support 2016-02-03 23:12:26 -08:00
Andrew Gaul 6d7a46296e Skip zero keys tests when backend lacks support 2016-02-01 12:29:43 -08:00
Andrew Gaul 1ae596e870 Support list multipart upload parts with Azure 
Map the Azure subparts back into S3 parts.
 2016-01-31 13:57:52 -08:00
Andrew Gaul bc68e2cb2e Do not run marker tests when backend lacks support 2016-01-31 12:50:04 -08:00
Andrew Gaul 82d95a2746 Use getObject and generatePresignedUrl short forms 2016-01-30 21:26:17 -08:00
Andrew Gaul 940286b022 Correct cut and paste errors in copy tests 2016-01-30 21:26:17 -08:00
Andrew Gaul 65fd80036f Do not run ACL tests when backend lacks support 2016-01-30 21:26:17 -08:00
Andrew Gaul cea4a42922 Use decoded content length when provided 
Previously S3Proxy provided an incorrect Content-Length header when
using V4 signing which providers rejected.
 2016-01-30 19:13:04 -08:00
Andrew Gaul 983c2b003e Remove stale TODO 2016-01-30 17:11:47 -08:00
Andrew Gaul e292ebaccb Consistently parse XML with Jackson 2016-01-29 17:50:22 -08:00
Andrew Gaul 511c968ce7 Add test for V4 URL signing 
Currently fails.
 2016-01-29 17:30:13 -08:00
Andrew Gaul 8d8e8ff290 Upgrade to aws-java-sdk-s3 1.10.50 
This release includes an implementation of AccessControlList.equals.
References #116.

Release notes:

https://aws.amazon.com/releasenotes/8286007886477675
 2016-01-29 09:50:14 -08:00
Andrew Gaul bc1833ed2e Remove BlobAccess TODO 
jclouds cannot provide BlobAccess since providers do not return these
ACLs.
 2016-01-26 19:28:50 -08:00
Andrew Gaul 6b0a6d8cdc Ensure failing tests raise NotImplemented error 2016-01-24 13:52:56 -08:00
Andrew Gaul f202207c63 Map a subset of XML ACLs to canned ACLs 
Fixes #116.
 2016-01-23 15:34:03 -08:00
Andrew Gaul 0e3a7ef843 Add missing EmailAddress field 
This allows S3Proxy to return the correct NotImplemented error code
instead of InternalError.  References #116.
 2016-01-23 15:16:38 -08:00
Andrew Gaul d557af774d Replace query parameters in anonymous requests 2016-01-23 15:06:04 -08:00
Andrew Gaul 19e2426220 Map a subset of XML ACLs to canned ACLs 
Fixes #116.
 2016-01-21 18:24:57 -08:00
Andrew Gaul 587a4a6670 Centralize blobstore quirks 2016-01-17 23:10:28 -08:00
Andrew Gaul 744d126331 Add support for Cache-Control header 
Fixes #115.
 2016-01-17 12:08:52 -08:00
Andrew Gaul 542c9b8112 Add Accept-Ranges header to partial get requests 
Fixes #66.
 2016-01-15 22:37:19 -08:00
Andrew Gaul 05bd2c1158 Use uploadId for stub blob name 
This makes MPU overwrites of an existing blob atomic.  Fixes #108.
 2016-01-14 15:11:03 -08:00
Andrew Gaul 3aed140a15 Avoid mutable ArrayList 
Latest AWS SDK handles immutable List.  Also tidy up line wrapping.
 2016-01-13 16:12:26 -08:00
Andrew Gaul ef3a7a6adf Add JcloudsS3BlobSignerLiveTest 
References #62.
 2016-01-12 16:20:03 -08:00
Andrew Gaul fb3bd591d6 Add JcloudsBucketsLiveTest 
Uncovers several missing features:

Bucket logging
Bucket payer
XML ACLs

References #62.
 2016-01-12 16:20:02 -08:00
Andrew Gaul 9c5499a05e Add JcloudsS3ClientLiveTest 
Uncovers several missing features:

Cache-Control header
conditional copies
public-read-write canned ACLs
XML ACLs

References #62.
 2016-01-12 16:20:02 -08:00
Andrew Gaul ae95538647 Add JcloudsS3ContainerIntegrationLiveTest 
References #62.
 2016-01-12 16:19:56 -08:00
Andrew Gaul f4212571b0 Rename suite to JcloudsS3BlobIntegrationLiveTest 2016-01-11 20:23:13 -08:00
Andrew Gaul 91429586dc Reuse encodingType local 2016-01-11 20:01:50 -08:00
Andrew Gaul 3c8f68feb2 Update copyright year to 2016 2016-01-09 21:14:59 -08:00
Andrew Gaul f86bd162ea Call startsWithIgnoreCase helper 2016-01-09 21:03:46 -08:00
Andrew Gaul a5c9f28e0a Make V4_MAX_NON_CHUNKED_REQUEST_SIZE configurable 
Also default to 32 MB since goofys via aws-sdk-go uses 8 MB requests.
 2016-01-09 20:33:28 -08:00
Andrew Gaul 2a03aa137d Use v4 signing with S3AwsSdkTest 2016-01-09 19:54:51 -08:00
Andrew Gaul 9c25dc557c Fix off-by-one in V4_MAX_NON_CHUNKED_REQUEST_SIZE 2016-01-09 19:22:28 -08:00
Andrew Gaul 3a042ff6e8 Mask high bits in ChunkedInputStream.read 
Previously read could erroneously return -1 end of stream when
returning ASCII 128.
 2016-01-09 18:48:26 -08:00
Andrew Gaul 9976f3c02f Consume trailing newline in ChunkedInputStream 
References #24.
 2016-01-09 18:48:15 -08:00
Andrew Gaul 564463c0e0 Factor out MPU test ByteSource offsets 2016-01-09 17:24:02 -08:00
Andrew Gaul cc516e7ede Improve ChunkedInputStream exception message 2016-01-09 17:21:47 -08:00
Andrew Gaul 00dc2a94e0 Use BaseEncoding.lowerCase where possible 
This avoid extra allocations from String.toLowerCase.
 2016-01-09 17:06:38 -08:00
Andrew Gaul 5c3333d4d9 Parse chunk length in hexadecimal 
Resolves an issue when uploading large objects and parts.
References #24.
 2016-01-09 16:56:12 -08:00
Andrew Gaul e7ba7e5c0d Factor out S3Client initialization 2016-01-09 16:19:23 -08:00
Andrew Gaul e9baeb9c76 Always re-encode parameters 
Assume UTF-8 encoding if not provided.  Reference #24.
 2016-01-09 14:37:32 -08:00
Andrew Gaul c0d7cce353 Fix reversed arguments to String.indexOf 
Found via goofys tests.  References #24.
 2016-01-09 13:06:12 -08:00
Timur Alperovich 0f872c0a2c Support AWS v4 signatures 
Fixes #24.
 2016-01-08 15:31:06 -08:00
Andrew Gaul f24b5647fa Provide requests with explicit payload 
Some requests require payload transformations, e.g., chunked encoding,
so explicitly provide the payload instead of using the request
payload.  References #24.
 2016-01-07 14:02:20 -08:00
Andrew Gaul 0ab0b55f31 Remove duplicate call to get blobStoreType 2015-12-28 12:00:41 -08:00
Andrew Gaul e93529bfeb Use native putBlob ACLs 2015-12-23 12:56:28 -08:00
Andrew Gaul 118b656f8e Allow anonymous HEAD object on public-read objects 2015-12-22 18:41:15 -08:00
Andrew Gaul 024c7a3fe9 Encode list blobs HTTP responses in UTF-8 
Previously responses incorrectly encoded in ISO-8859-1.
 2015-12-22 18:31:32 -08:00
Andrew Gaul 0332a2107d Enabled testSetBlobAccess 
jclouds now uses canned ACLs which S3Proxy supports.
 2015-12-19 18:19:37 -08:00
Andrew Gaul 04f50b7363 Annotate JcloudsIntegrationTest with @Test 
For consistency with jclouds tests.
 2015-12-19 13:09:07 -08:00
Andrew Gaul d1dcc1abee Use testng annotations for JcloudsIntegrationTest 
This matches what the underlying test does and resolves a spurious
issue with testPutObjectStream.
 2015-12-19 00:41:27 -08:00
Andrew Gaul bc993cc3cb Skip testMaximumMultipartUpload for remote backend 
This test dominates run-time due to issuing 10,000 RPCs.
 2015-12-16 19:21:59 -08:00
Andrew Gaul ee55828375 Emit NotImplemented when setting XML ACLs 2015-12-16 19:19:44 -08:00
Andrew Gaul d9b3753f49 Do not set expires when creating blobs 
No tests currently check expires and setting it causes problems for
Swift-based backends.
 2015-12-15 17:22:25 -08:00
Andrew Gaul 3151754ebc Use container public-read for Swift test 
Swift does not support blob-level ACLs.
 2015-12-15 07:09:22 -08:00
Derrek Leute d8c73bc86d URL encode blob names when client requests it 
This allows support for characters which XML 1.0 cannot represent and
improves compatibility with newer versions of the AWS CLI.
Fixes #104.
 2015-12-14 17:10:45 -08:00
Andrew Gaul 23cff3a61d Do not require credentials for local blobstores 2015-12-14 16:01:35 -08:00
Andrew Gaul 9e033c677f Require credentials for aws-v2 authentication 2015-12-14 15:48:19 -08:00
Andrew Gaul 9af1d14dcc Provide content length to AmazonS3.putObject 
Prevents warnings of the form:

WARNING: No content length specified for stream data.  Stream contents
will be buffered in memory and could result in out of memory errors.
 2015-12-14 00:31:19 -08:00
Andrew Gaul 177d9b7566 Handle possibly-null HttpResponse 2015-12-13 23:43:22 -08:00
Andrew Gaul fa215ecb8a Factor out calls to get blobStoreType 2015-12-13 22:58:35 -08:00
Andrew Gaul e237ee9ebb Disable ETag checking for Google Cloud Storage 
Google Cloud Storage ETag is opaque and not an MD5 hash.
 2015-12-12 20:40:26 -08:00
Andrew Gaul c45dde3cbe Skip testMaximumMultipartUpload with large sizes 2015-12-12 13:00:48 +08:00
Andrew Gaul 0581c747bd Add missing @Override annotation 
Found via error-prone.
 2015-12-12 11:24:59 +08:00
Andrew Gaul 33fac374a6 Use container public-read for Azure test 
Azure does not support blob-level ACLs.
 2015-12-12 11:24:58 +08:00
Andrew Gaul eccf6d1b1f Disable ETag checking for Azure 
Azure ETag is opaque and not an MD5 hash.  References #96.
 2015-12-12 11:24:58 +08:00