mirror
/
s3proxy
kopia lustrzana https://github.com/gaul/s3proxy
1
0
Forkuj 0
Kod Zgłoszenia Packages Projekty Wydania Wiki Aktywność
650 Commity
10 Gałęzie
27 Tagi
5,7 MiB
Wykres commitów

434 Commity (blobstore-locator)

Autor SHA1 Wiadomość Data
Andrew Gaul b0b1f4e9fa Emulate multipart upload with single-part uploads 
This approach requires three times as many operations as the optimal
approach.  Implementing this correctly requires exposing the
underlying multipart operations in jclouds.  Most s3-tests pass but
test_list_multipart_upload still fails:

References #2.
 2015-03-02 14:37:16 -08:00
Andrew Gaul 11c93a43a8 Use correct content length during copy 
Previously replacing the metadata would zero out the object contents.
 2015-03-02 14:01:00 -08:00
Andrew Gaul a65221091f Use maven-assembly-plugin instead of Maven Shade 
This works around MSHADE-183 which prevents use with Maven 3.2.5 and
has caused recent Travis CI failures.  Fixes #38.
 2015-02-22 23:22:05 -08:00
Andrew Gaul 52118ced78 Add support to set and get canned object ACLs 
Currently only support private and public-read access.
 2015-02-22 18:48:08 -08:00
Andrew Gaul c8e4279853 Propagate Exception if not HttpResponseException 2015-02-20 17:22:47 -08:00
Andrew Gaul 613da2f3ec Use builder to create S3Proxy 
Also hoist sanity checks out of main.
 2015-02-20 11:01:37 -08:00
Andrew Gaul 2e82478f0e Prefer static imports for Preconditions 2015-02-20 10:50:37 -08:00
Timur Alperovich 51e7f3c22a Expose the Jetty state in the S3Proxy interface. 
The commit adds a getState() method, which exposes the Jetty state
to the caller. This is useful to check whether the server is running
or is in some other state.
 2015-02-18 16:13:45 -08:00
Andrew Gaul b85ab17316 Correct typo 2015-02-18 11:34:28 -08:00
Andrew Gaul 7e61856db1 Support native multi-object delete 2015-02-15 01:07:10 -08:00
Andrew Gaul 9a261a62b0 Add support to set and get canned container ACLs 
Currently only support private and public-read access.
 2015-02-13 15:43:58 -08:00
Andrew Gaul 4ebbce7956 Run unknown parameter test 
JCLOUDS-258 allows use of MPU with the generic S3 provider.
 2015-02-12 18:08:29 -08:00
Andrew Gaul 074c36baab Add test harness to execute s3-tests 
Also add Travis configuration.
 2015-01-22 17:14:45 -08:00
Andrew Gaul 23f8e772da Upgrade to jclouds 2.0.0-SNAPSHOT 
This release allows larger than 2 GB single-part blobs and supports
object metadata with the filesystem provider on Linux and Windows.  It
also includes support for object size in container lists.  Fixes #1.
Fixes #6.
 2015-01-22 14:03:18 -08:00
Andrew Gaul fb0f321303 Improve CommonPrefixes support 
References #5.
 2015-01-22 13:30:28 -08:00
Andrew Gaul 7815a79548 Emit NotImplemented with unknown parameters 
This ensures that requests like multi-part upload and version
operations emit the intended responses.
 2015-01-19 20:06:58 -08:00
Andrew Gaul bae51b500a Reduce exception handling scope 2015-01-16 15:56:08 -08:00
Andrew Gaul e1c7b22b7d Handle NoSuchBucket during copy object 2015-01-16 15:14:53 -08:00
Andrew Gaul d81ee81a86 Handle null from HttpResponseException.getResponse 2015-01-15 18:36:14 -08:00
Andrew Gaul 05b63365bf Emit bogus CreationDate when not present 2015-01-15 18:23:22 -08:00
Andrew Gaul 7dff664c17 Add prefix to CommonPrefixes 
Addresses s3cmd tests failures.
 2015-01-14 12:24:23 -08:00
Andrew Gaul 5b5e9877b9 Improve support for x-amz-metadata-directive 
Replace content metadata in addition to user metadata.  Addresses
s3cmd tests failures.
 2015-01-13 23:32:46 -08:00
Andrew Gaul a6133a60a6 Honor force multipart in emulated copy object 2015-01-13 23:28:09 -08:00
Andrew Gaul 86665450ca URI decode copy source path 
Fixes s3cmd tests failures seen with Unicode object names.  Follow on
to b91b8e25d7.
 2015-01-13 18:44:59 -08:00
Andrew Gaul e3c49331ac Emit UTF-8 in object listing and multi-delete 
Fixes s3cmd tests failures seen with Unicode object names.
 2015-01-13 17:50:47 -08:00
Andrew Gaul 946d34b03b Parse XML with StAX 
This addresses issues with object names containing special characters.
Fixes #25.
 2015-01-11 20:45:56 -08:00
Andrew Gaul f3221cb136 Emit XML via StAX 
This addresses issues with object names containing special characters.
References #25.
 2015-01-11 18:56:39 -08:00
Ka-Hing Cheung e955b998fb use static function in URLDecoder 2015-01-09 15:31:45 -08:00
Ka-Hing Cheung b91b8e25d7 uri decode path before passing them to lower layers 2015-01-06 15:54:36 -08:00
Andrew Gaul 15e11975a2 Allow kernel to pick S3Proxy port in tests 2015-01-06 15:05:21 -08:00
Ka-Hing Cheung 2dd3dac970 add test for % in blob name 2015-01-06 14:48:03 -08:00
Andrew Gaul ff551248a1 Fail V4 signature requests with InvalidArgument 
This allows V4 clients like s3cmd to retry with V2 signatures.
References #24.
 2015-01-06 14:42:29 -08:00
Andrew Gaul 0eac109276 Add logback.xml only to with jar-with-dependencies 2015-01-05 18:51:56 -08:00
Ka-Hing Cheung dbd7304e65 make S3Proxy contructor public 2015-01-05 11:39:05 -08:00
Andrew Gaul 1f9fd7fa21 Consistently emit XML prologue and AWS namespace 2014-12-29 23:18:49 -08:00
Andrew Gaul a2c6bfc4cc Use correct container name with testHttpClient 
This test still fails due to authorization issues.
 2014-12-29 17:45:43 -08:00
Andrew Gaul 807f6ef47c Explicitly handle get location 2014-12-29 16:29:30 -08:00
Andrew Gaul d33324797a Consistently flush before closing Writer 2014-12-28 14:52:36 -08:00
Andrew Gaul 711c0821d1 Sort subresources 
This ensures deterministic signatures when requests have multiple
subresources, e.g., multipart uploads.
 2014-12-21 23:11:59 -08:00
Andrew Gaul 161fd3d847 Include delete as a signed resource 
This allows multi-object delete to work and addresses a regression
introduced by e5c12c1777.
 2014-12-03 22:59:36 -08:00
Andrew Gaul ae3598dbfb Emit InvalidAccessKeyId on access key mismatch 
References #5.
 2014-12-03 22:39:41 -08:00
Andrew Gaul 40c4d099d9 Handle invalid Content-Length during create bucket 
References #5.
 2014-12-03 18:36:15 -08:00
Andrew Gaul 3b2bd615a0 Simplify parsing of Content-Length and Content-MD5 
HttpServletRequest uses null to indicate a header value which is
present but has no value.  Use empty String instead.
 2014-12-03 04:30:54 -08:00
Ka-Hing Cheung e5c12c1777 more required parameters needed for signing 
allows s3cmd info s3://bucket/ to work
 2014-11-20 04:31:23 +01:00
Ka-Hing Cheung 617b3ae3e9 more compatibility fixes for s3cmd 
s3cmd uses virtual hostname for bucket name, with the host header
including the port. need to strip out the port to find out the
bucket name
 2014-11-18 10:14:16 +01:00
Ka-Hing Cheung 004d12761e http header name is case insensitive 
fixes compatibility with s3cmd
 2014-11-18 03:45:58 +01:00
Andrew Gaul 559b1d36dd Clean up signer 2014-10-27 15:29:53 -07:00
Andrew Gaul 5742323a70 Remove Azure ETag hack 
Previously jclouds required ETag to be a hex-encoded Content-MD5.
 2014-10-27 15:21:40 -07:00
Andrew Gaul 9a81b73324 Add more Checkstyle rules 2014-10-24 00:28:58 -07:00
Andrew Gaul 4a58f00885 Add Google Cloud Storage provider 
Also convert ETag from GCS-style base64 to AWS-style base16.
 2014-10-24 00:23:51 -07:00
Andrew Gaul 6792798347 Do not require date header w/ none authentication 
AWS S3 uses a date header to prevent replay attacks on signed
requests.  None authentication should not require this header.
Regression from 8c623c179c.  Fixes #17.
 2014-10-14 17:18:02 -07:00
Andrew Gaul 4baaca86f9 Allow System properties to override configuration 
Fixes #15.
 2014-10-10 07:09:01 -07:00
Andrew Gaul 99a00499ca Set Content-Type only when provided by client 2014-09-05 12:51:07 -07:00
Andrew Gaul c6115546c4 Configure jclouds SLF4J module 2014-09-05 12:26:32 -07:00
Andrew Gaul 136dd679c4 Enforce consistent operator wrapping style 2014-08-31 13:59:45 -07:00
Andrew Gaul 7883dd0574 Remove unneeded file, method and parameter checks 2014-08-31 13:59:26 -07:00
Andrew Gaul 016170f83f Clean up get handling 2014-08-28 19:27:40 -07:00
Andrew Gaul e2816190aa Emit fake key ACLs 
References #5.
 2014-08-28 18:50:27 -07:00
Andrew Gaul 15d885d370 Emit ETag with key metadata 
References #5.
 2014-08-28 18:50:08 -07:00
Andrew Gaul f5ab87241c Emit InvalidBucketName on non-alphanumeric name 
References #5.
 2014-08-28 18:30:17 -07:00
Andrew Gaul f8b5dde61f Enforce indentation via Checkstyle 2014-08-28 17:42:38 -07:00
Andrew Gaul f39ea39d93 Enforce 80 character line length via Checkstyle 2014-08-28 17:40:46 -07:00
Andrew Gaul 452d8d366c Add support for virtual host buckets 
Fixes #8.
 2014-08-26 19:51:15 -07:00
Andrew Gaul 34db5cae9a Emit InvalidArgument for malformed Authorization 
Also keep identity and signature as distinct values.  References #5.
 2014-08-25 19:18:29 -07:00
Andrew Gaul 7b6f86b3fe Ensure ordering for CommonPrefixes 2014-08-25 19:18:29 -07:00
Andrew Gaul 6dde404e88 Emit not implemented for put and get object ACL 2014-08-24 06:29:09 -07:00
Andrew Gaul 8c623c179c Require time reference point for requests 
Requests must have a Date header, X-Amz-Date header, or Expires
parameter.
 2014-08-24 05:53:12 -07:00
Andrew Gaul ce3d7edad3 Add expiration to query string authorization 
Also repair support for query string authorization, regression from
d74a409.
 2014-08-23 19:09:05 -07:00
Andrew Gaul 84bfb27d66 Prefer Collections.list over Enumeration 2014-08-23 16:58:48 -07:00
Andrew Gaul 820b884169 Emit BadDigest for mismatched MD5 hash 
S3 uses InvalidDigest for a malformed MD5 hash.
 2014-08-23 15:40:16 -07:00
Andrew Gaul d74a409f87 Emit AccessDenied for empty Authorization header 2014-08-23 15:29:06 -07:00
Andrew Gaul 8042e9b490 Ensure that ETag has quotes 2014-08-22 14:38:19 -07:00
Andrew Gaul b8e732c794 Map Swift HTTP 422 to InvalidDigest 
References #5.
 2014-08-22 10:01:55 -07:00
Andrew Gaul 21a708a38d Remove newlines which the AWS SDK rejects 2014-08-21 22:49:36 -07:00
Andrew Gaul 65074b6338 Consistently propagate IOException 
Jetty automatically returns 500 internal error to clients and logs the
exception stack trace.
 2014-08-21 22:39:55 -07:00
Andrew Gaul c44cfdfbd4 Honor --version 2014-08-21 22:21:09 -07:00
Andrew Gaul f346e703c7 Implement object multi-delete 
Currently emulated with single object delete since jclouds does not
expose a portable multi-delete method.  References #5.
 2014-08-21 22:01:12 -07:00
Andrew Gaul d7cd5e107f Add String constant for AWS xmlns 2014-08-21 21:41:05 -07:00
Andrew Gaul 90261c1ce7 Sanity check Date header 
References #5.
 2014-08-21 13:39:12 -07:00
Andrew Gaul 886c27e63a Emit BucketAlreadyExists 
References #5.
 2014-08-20 23:38:51 -07:00
Andrew Gaul 1f8d9f47a8 Consistently emit NoSuchBucket and NoSuchKey 
Unfortunately jclouds does not consistently throw
ContainerNotFoundException.  References #5.
 2014-08-20 23:38:10 -07:00
Andrew Gaul 0cc7a1390a Factor out S3 error codes into enum class 2014-08-20 18:45:06 -07:00
Andrew Gaul fc8613b4f8 Convert Azure-style ETag into S3-style ETag 2014-08-18 18:28:51 -07:00
Andrew Gaul da512876de Use unique container names in tests 
This commit improves test compatibility with real blobstores.
 2014-08-18 18:28:51 -07:00
Andrew Gaul 02f68372e0 Remove bucket after tests complete 2014-08-18 15:23:24 -07:00
Andrew Gaul 694c033848 Only send Expires header when present in request 
Previously S3Proxy incorrectly compared against 0 instead of -1,
emitting bogus Expires headers.  Fixes #13.
 2014-08-18 15:23:20 -07:00
Andrew Gaul ad1c01057a Configure jclouds SLF4J module 
Also use jclouds-slf4j instead of slf4j-simple to avoid multiple
bindings.
 2014-08-18 15:23:18 -07:00
Andrew Gaul ed971cdcbe Do not set endpoint in overrides 
Previously tests used the same endpoint for both client-s3proxy and
s3proxy-server.
 2014-08-18 12:00:12 -07:00
Andrew Gaul c6a87d6816 Handle null Content-MD5 in response 2014-08-18 11:51:50 -07:00
Andrew Gaul ba3171a8e7 Add findbugs-maven-plugin 
Address one bug and ignore XSS bugs.
 2014-08-18 11:08:44 -07:00
Andrew Gaul bd2b055f1b Add option to force proxy-server multi-part upload 
This allows use of S3Proxy with object stores with small single-part
object sizes, e.g., Azure.  Fixes #12.
 2014-08-17 23:42:37 -07:00
Andrew Gaul 64ce6a1fa8 Move constants to utility class 2014-08-17 23:21:58 -07:00
Andrew Gaul c042fa6165 Add support for listening on HTTPS 
Fixes #3.
 2014-08-14 01:28:52 -07:00
Andrew Gaul 5bca38ca24 Configure tests via s3proxy.conf 2014-08-14 01:02:51 -07:00
Andrew Gaul 0b18a976f3 Log duplicate headers 2014-08-13 18:16:10 -07:00
Andrew Gaul b47e8387fd Enable Jetty STRICT mode to get true header values 
Previously Jetty rewrote header values such as the Content-Type, which
broke signing with the AWS SDK.  Reference:

https://bugs.eclipse.org/bugs/show_bug.cgi?id=414449
 2014-08-13 18:14:06 -07:00
Andrew Gaul 874766eade Remove newlines which the AWS SDK rejects 
Reference:

http://stackoverflow.com/questions/5138696/org-xml-sax-saxparseexception-content-is-not-allowed-in-prolog
 2014-08-13 17:13:05 -07:00
Andrew Gaul 73d647f46e Strip leading / from x-amz-copy-source header 
Previously S3Proxy did not update copySourceHeader.  Fixes #10.
 2014-08-13 15:24:31 -07:00
Andrew Gaul e59e2660dd Allow creation of public-read buckets 
jclouds portable abstraction does not allow finer-grained permissions
than public-read and does not allow changing this policy later.
 2014-08-13 01:29:23 -07:00
Andrew Gaul 1ee05fed62 Use x-amz-copy-source header without leading / 
Fixes #10.
 2014-08-13 00:51:41 -07:00
Andrew Gaul 63bd5a95cf Add headers at trace logging level 2014-08-12 17:23:36 -07:00
Andrew Gaul 0de7d2aa8f Support half-open range requests. 
Also emit the correct status code 206.  Fixes #9.
 2014-08-12 12:26:53 -07:00
Andrew Gaul 5346e5e181 Address Checkstyle violations 2014-08-11 23:08:59 -07:00
Andrew Gaul 84c33c9b42 Emit NoSuchBucket on getBlob 2014-08-11 19:22:03 -07:00
Andrew Gaul b65d6e2ce5 Correct ListBucket indentation 2014-08-11 17:12:05 -07:00
Andrew Gaul 87edea10e9 Support creating buckets in user-specified regions 2014-08-11 17:11:08 -07:00
Andrew Gaul 41046c966a Reduce log level of signer 2014-08-11 17:10:56 -07:00
Andrew Gaul f3392d13a2 Simplify ListBucket parsing 2014-08-10 17:54:33 -07:00
Andrew Gaul e35d059e1d Remove special-case for listing with max-keys=0 
The existing list logic handles this; the underlying jclouds request
confused me and will be corrected in jclouds/jclouds#475.
 2014-08-10 13:12:08 -07:00
Andrew Gaul 8e63a6c93e Support query string request authentication 2014-08-05 13:07:05 -07:00
Andrew Gaul 8f24e8cb7e Implement server-side copy support 
S3Proxy implements this via proxy-side copying since jclouds does not
expose server-side copy in its portable abstraction.
 2014-08-05 10:51:30 -07:00
Andrew Gaul 17b3d6c0ba Emit HTTP status code inside handler methods 
Also remove bogus HTTP 500 internal errors which were sent after the
response body.
 2014-08-01 22:20:26 -07:00
Andrew Gaul df13f8247c Emit RequestTimeout when putBlob times out 
Found with Ceph s3-tests.  References #5.
 2014-08-01 21:49:16 -07:00
Andrew Gaul 5d069b26b7 Add delimiter to CommonPrefixes 
Found with Ceph s3-tests.  References #5.
 2014-08-01 17:53:35 -07:00
Andrew Gaul a94ded5443 Emit 501 not implemented for PUT Bucket ACL 2014-08-01 17:53:33 -07:00
Andrew Gaul 49489430d5 Handle trailing slash during HEAD bucket 
Found with Ceph s3-tests.  References #5.
 2014-08-01 16:06:38 -07:00
Andrew Gaul 94f8582a67 Emit HTTP 204 on successful bucket delete 
Found with Ceph s3-tests.  References #5.
 2014-07-31 15:56:23 -07:00
Andrew Gaul 4c99f6870f Emit NoSuchBucket/BucketNotEmpty on bucket delete 
Found with Ceph s3-tests.  References #5.
 2014-07-31 00:54:25 -07:00
Andrew Gaul 1e7a4a2270 Handle empty header values in Authorization 
Found with Ceph s3-tests.  References #5.
 2014-07-30 01:01:03 -07:00
Andrew Gaul f7db3ee4fb Implement AWS Authorization 
Fixes #4.
 2014-07-30 00:18:03 -07:00
Andrew Gaul 3f52992a25 Add variant of sendSimpleErrorResponse w/o extra 2014-07-29 17:07:20 -07:00
Andrew Gaul 88d521b11d Emit NoSuchBucket on head to non-existent bucket 
Found with Ceph s3-tests.  References #5.
 2014-07-29 15:58:28 -07:00
Andrew Gaul 1284c91e37 Emit NoSuchBucket on write to non-existent bucket 
Found with Ceph s3-tests.  References #5.
 2014-07-29 12:06:54 -07:00
Andrew Gaul b603abefc8 Emit NoSuchKey when reading non-existent object 
Found with Ceph s3-tests.  References #5.
 2014-07-29 08:16:41 -07:00
Andrew Gaul 16dfa70aec Handle empty user metadata values 
Found with Ceph s3-tests.  References #5.
 2014-07-29 08:03:55 -07:00
Andrew Gaul 7c5f219535 Return HTTP 204 on successful delete object 
Found with Ceph s3-tests.  References #5.
 2014-07-29 08:03:55 -07:00
Andrew Gaul cf1376387c Improve list bucket max-keys error handling 
Found with Ceph s3-tests.  References #5.
 2014-07-29 08:03:52 -07:00
Andrew Gaul f698bd68e8 Improve create bucket error handling 
Found with Ceph s3-tests.  References #5.
 2014-07-29 07:35:48 -07:00
Andrew Gaul 5c843eb272 Improve Content-Length error handling 
Found with Ceph s3-tests.  References #5.
 2014-07-29 07:35:48 -07:00
Andrew Gaul a6c997d71c Improve Content-MD5 error handling 
Found with Ceph s3-tests.  References #5.
 2014-07-29 07:35:48 -07:00
Andrew Gaul e53d8c929c Add sendSimpleErrorResponse helper 
Will reduce duplicated code in future commits.
 2014-07-28 22:26:46 -07:00
Andrew Gaul 55484a4f6b Quote eTag header 
Found with Ceph s3-tests.  References #5.
 2014-07-28 22:04:56 -07:00
Andrew Gaul 3edd5712aa Allow trailing slash when creating bucket 
Found with Ceph s3-tests.  References #5.
 2014-07-28 00:38:55 -07:00
Andrew Gaul ff84ccf19e Initial commit of S3Proxy 2014-07-27 13:32:04 -07:00