eb5c59a371
Enforce payload checksums
...
CRC32, CRC32C, SHA1, and SHA256 are support but CRC64NVME is not.
Fixes #806 . References #830 .
2025-09-04 14:20:59 -07:00
d503c07655
Disable retries in AwsSdk2Test
2025-09-03 10:23:26 -07:00
d069ac1370
Ignore all checksum stanzas
2025-08-28 22:52:19 -07:00
02fbdc149b
Parse but do not enforce CRC32
...
This suffices to allow AWS SDK for Java v2 to work without additional
configuration. A more complete solution would allow for additional
checksum algorithms and actually enforce the checksums. Fixes #830 .
2025-08-28 20:18:51 -07:00
6e5fbf5a5f
Add test for encrypted conditional get
2025-08-29 08:34:03 +09:00
9c587c7f38
Fix EncryptedBlobStore error handling
2025-08-29 08:34:03 +09:00
69297fedd1
Add NoCacheBlobStore
2025-08-27 15:08:52 +09:00
8d963702aa
Update s3-tests to run conditional get tests
2025-08-27 14:05:54 +09:00
f8f4359ab0
Include --fail with curl to propagate exit code
2025-08-26 20:57:11 -07:00
664a94dbd5
Update README.md to include latency middleware
2025-08-24 02:02:37 +09:00
2acdb9629b
Bump to 2.7.1-SNAPSHOT
2025-08-21 14:17:00 -07:00
c13a49d1d7
S3Proxy 2.7.0 release
2025-08-21 14:12:45 -07:00
f8ebf43474
Prepare for Central Portal releasing
...
As OSS service is being sunset.
2025-08-21 14:10:54 -07:00
944d19d975
Add latency blobstore middleware
2025-08-22 05:56:26 +09:00
cc4cbfe4cf
Munge / key to container name
...
Filesystems cannot support this path but we can support s3fs's use
case for setting different user metadata. References
s3fs-fuse/s3fs-fuse#2656 .
2025-08-21 13:38:29 -07:00
aadba1f438
Upgrade Surefire to 3.5.2
...
3.5.3 seems to have a junit incompatibility. References #819 .
2025-08-05 05:10:03 +09:00
53c1d37dd8
Bump ch.qos.logback:logback-classic from 1.5.17 to 1.5.18
...
Bumps [ch.qos.logback:logback-classic](https://github.com/qos-ch/logback ) from 1.5.17 to 1.5.18.
- [Release notes](https://github.com/qos-ch/logback/releases )
- [Commits](https://github.com/qos-ch/logback/compare/v_1.5.17...v_1.5.18 )
---
updated-dependencies:
- dependency-name: ch.qos.logback:logback-classic
dependency-version: 1.5.18
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2025-08-05 03:37:41 +09:00
8540e4f63b
Bump com.github.spotbugs:spotbugs-annotations from 4.9.2 to 4.9.3
...
Bumps [com.github.spotbugs:spotbugs-annotations](https://github.com/spotbugs/spotbugs ) from 4.9.2 to 4.9.3.
- [Release notes](https://github.com/spotbugs/spotbugs/releases )
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.9.2...4.9.3 )
---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
dependency-version: 4.9.3
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2025-08-05 03:33:05 +09:00
5b5d151dda
Bump docker/metadata-action from 5.7.0 to 5.8.0
...
Bumps [docker/metadata-action](https://github.com/docker/metadata-action ) from 5.7.0 to 5.8.0.
- [Release notes](https://github.com/docker/metadata-action/releases )
- [Commits](https://github.com/docker/metadata-action/compare/v5.7.0...v5.8.0 )
---
updated-dependencies:
- dependency-name: docker/metadata-action
dependency-version: 5.8.0
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2025-08-05 03:32:55 +09:00
45c9402133
Bump com.google.jimfs:jimfs from 1.3.0 to 1.3.1
...
Bumps [com.google.jimfs:jimfs](https://github.com/google/jimfs ) from 1.3.0 to 1.3.1.
- [Release notes](https://github.com/google/jimfs/releases )
- [Commits](https://github.com/google/jimfs/compare/v1.3.0...v1.3.1 )
---
updated-dependencies:
- dependency-name: com.google.jimfs:jimfs
dependency-version: 1.3.1
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2025-08-05 03:18:12 +09:00
24e869ca6a
Update s3-tests
2025-08-04 11:04:10 -07:00
5ff5896ac6
Force Jetty header cache to be case-sensitive, to avoid signature mismatch (e.g. upper case charset used in S3Proxy, but lowercase used in original signature calculation)
2025-07-17 02:46:16 +09:00
027cd5fdd7
Bump com.puppycrawl.tools:checkstyle from 10.23.1 to 10.26.1
...
Bumps [com.puppycrawl.tools:checkstyle](https://github.com/checkstyle/checkstyle ) from 10.23.1 to 10.26.1.
- [Release notes](https://github.com/checkstyle/checkstyle/releases )
- [Commits](https://github.com/checkstyle/checkstyle/compare/checkstyle-10.23.1...checkstyle-10.26.1 )
---
updated-dependencies:
- dependency-name: com.puppycrawl.tools:checkstyle
dependency-version: 10.26.1
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2025-07-02 15:03:30 +09:00
a8ac96cdbc
Bump com.fasterxml.jackson.dataformat:jackson-dataformat-xml
...
Bumps [com.fasterxml.jackson.dataformat:jackson-dataformat-xml](https://github.com/FasterXML/jackson-dataformat-xml ) from 2.18.2 to 2.19.1.
- [Commits](https://github.com/FasterXML/jackson-dataformat-xml/compare/jackson-dataformat-xml-2.18.2...jackson-dataformat-xml-2.19.1 )
---
updated-dependencies:
- dependency-name: com.fasterxml.jackson.dataformat:jackson-dataformat-xml
dependency-version: 2.19.1
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2025-07-02 15:03:09 +09:00
ab68547993
Bump aws-sdk.version from 1.12.783 to 1.12.787
...
Bumps `aws-sdk.version` from 1.12.783 to 1.12.787.
Updates `com.amazonaws:aws-java-sdk-s3` from 1.12.783 to 1.12.787
- [Changelog](https://github.com/aws/aws-sdk-java/blob/master/CHANGELOG.md )
- [Commits](https://github.com/aws/aws-sdk-java/compare/1.12.783...1.12.787 )
Updates `com.amazonaws:aws-java-sdk-sts` from 1.12.783 to 1.12.787
- [Changelog](https://github.com/aws/aws-sdk-java/blob/master/CHANGELOG.md )
- [Commits](https://github.com/aws/aws-sdk-java/compare/1.12.783...1.12.787 )
---
updated-dependencies:
- dependency-name: com.amazonaws:aws-java-sdk-s3
dependency-version: 1.12.787
dependency-type: direct:development
update-type: version-update:semver-patch
- dependency-name: com.amazonaws:aws-java-sdk-sts
dependency-version: 1.12.787
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2025-07-02 15:02:57 +09:00
9b0f4cc84a
Remove linux/arm/v7 container
2025-06-19 05:04:16 +09:00
5f7e597b5e
Run on Eclipse Temurin OpenJDK 21 instead of 17
2025-06-19 05:04:16 +09:00
1003265cd9
Fix EncryptedBlobStore support for s3 compatible backends without eTag suffix
...
Some mostly S3 REST API compatible storage backends do not return the number of multipart upload parts as a suffix to the eTag as Amazon does and as the previous code expects. An example for this is NetApp StorageGRID S3 REST API.
The old code had a fallback to just assume one encrypted part, but this is actually wrong in the multipart upload case for these backends, thus this is replace in this commit.
Example eTag structure from real AWS S3 for multipart uploads with 2 parts:`xyzabc-2`
In this case the number of parts is also not present in the object metadata, as metadata is set on S3 API when starting the multipart upload, and not when completing it, thus the number of parts is not yet known at this earlier point in time.
This change adds a third fallback option to just read the number of parts from the final part padding, which is the only place guaranteed to always be present for encrypted blobs.
Only for these backends this adds an additional GET request to the backend, but only for the actual 64 bytes of the padding.
2025-06-12 22:53:35 +09:00
9ddc581285
Add .dockerignore
...
Exclude everything from Docker build context that is not used by COPY steps in the Dockerfile.
This speeds up the container build on macOS, Windows or remote builders as less files and directories are copied over to the container builder.
https://docs.docker.com/build/concepts/context/#dockerignore-files
2025-06-12 22:52:23 +09:00
17e490e775
Fix Python yield error in s3-tests
...
'yield' keyword is allowed in fixtures, but not in tests
(test_routing_generator)
2025-06-11 13:16:06 -04:00
7b9e8602cd
Bump org.apache.maven.plugins:maven-resources-plugin from 3.3.0 to 3.3.1
...
Bumps [org.apache.maven.plugins:maven-resources-plugin](https://github.com/apache/maven-resources-plugin ) from 3.3.0 to 3.3.1.
- [Release notes](https://github.com/apache/maven-resources-plugin/releases )
- [Commits](https://github.com/apache/maven-resources-plugin/compare/maven-resources-plugin-3.3.0...maven-resources-plugin-3.3.1 )
---
updated-dependencies:
- dependency-name: org.apache.maven.plugins:maven-resources-plugin
dependency-version: 3.3.1
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2025-06-02 13:50:29 +09:00
e5c20b6c2e
Bump org.apache.maven.plugins:maven-assembly-plugin from 3.5.0 to 3.7.1
...
Bumps [org.apache.maven.plugins:maven-assembly-plugin](https://github.com/apache/maven-assembly-plugin ) from 3.5.0 to 3.7.1.
- [Release notes](https://github.com/apache/maven-assembly-plugin/releases )
- [Commits](https://github.com/apache/maven-assembly-plugin/compare/maven-assembly-plugin-3.5.0...maven-assembly-plugin-3.7.1 )
---
updated-dependencies:
- dependency-name: org.apache.maven.plugins:maven-assembly-plugin
dependency-version: 3.7.1
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2025-06-02 13:50:12 +09:00
276465feeb
Bump com.google.guava:guava from 33.4.0-jre to 33.4.8-jre
...
Bumps [com.google.guava:guava](https://github.com/google/guava ) from 33.4.0-jre to 33.4.8-jre.
- [Release notes](https://github.com/google/guava/releases )
- [Commits](https://github.com/google/guava/commits )
---
updated-dependencies:
- dependency-name: com.google.guava:guava
dependency-version: 33.4.8-jre
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2025-06-02 13:42:39 +09:00
144f56a562
Handle UnsupportedOperationException for getPosixFilePermissions and use writeCommonMetadataAttr in all relevant places
2025-05-30 03:06:08 +09:00
bf2d50670c
Catch UnsupportedOperationException for getPosixFilePermissions and default to PRIVATE on non-POSIX systems
2025-05-30 03:06:08 +09:00
3d67b5657a
Bump com.puppycrawl.tools:checkstyle from 10.22.0 to 10.23.1
...
Bumps [com.puppycrawl.tools:checkstyle](https://github.com/checkstyle/checkstyle ) from 10.22.0 to 10.23.1.
- [Release notes](https://github.com/checkstyle/checkstyle/releases )
- [Commits](https://github.com/checkstyle/checkstyle/compare/checkstyle-10.22.0...checkstyle-10.23.1 )
---
updated-dependencies:
- dependency-name: com.puppycrawl.tools:checkstyle
dependency-version: 10.23.1
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2025-05-02 12:30:07 +09:00
9aa8ea5af2
Bump org.eclipse.jetty:jetty-servlet from 11.0.24 to 11.0.25
...
Bumps org.eclipse.jetty:jetty-servlet from 11.0.24 to 11.0.25.
---
updated-dependencies:
- dependency-name: org.eclipse.jetty:jetty-servlet
dependency-version: 11.0.25
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2025-05-02 12:28:59 +09:00
73e7d14079
Bump aws-sdk.version from 1.12.782 to 1.12.783
...
Bumps `aws-sdk.version` from 1.12.782 to 1.12.783.
Updates `com.amazonaws:aws-java-sdk-s3` from 1.12.782 to 1.12.783
- [Changelog](https://github.com/aws/aws-sdk-java/blob/master/CHANGELOG.md )
- [Commits](https://github.com/aws/aws-sdk-java/compare/1.12.782...1.12.783 )
Updates `com.amazonaws:aws-java-sdk-sts` from 1.12.782 to 1.12.783
- [Changelog](https://github.com/aws/aws-sdk-java/blob/master/CHANGELOG.md )
- [Commits](https://github.com/aws/aws-sdk-java/compare/1.12.782...1.12.783 )
---
updated-dependencies:
- dependency-name: com.amazonaws:aws-java-sdk-s3
dependency-version: 1.12.783
dependency-type: direct:development
update-type: version-update:semver-patch
- dependency-name: com.amazonaws:aws-java-sdk-sts
dependency-version: 1.12.783
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2025-05-02 12:28:51 +09:00
bfcfd9df4e
Bump com.azure:azure-storage-blob from 12.29.0 to 12.30.0
...
Bumps [com.azure:azure-storage-blob](https://github.com/Azure/azure-sdk-for-java ) from 12.29.0 to 12.30.0.
- [Release notes](https://github.com/Azure/azure-sdk-for-java/releases )
- [Commits](https://github.com/Azure/azure-sdk-for-java/compare/azure-storage-blob_12.29.0...azure-storage-blob_12.30.0 )
---
updated-dependencies:
- dependency-name: com.azure:azure-storage-blob
dependency-version: 12.30.0
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2025-05-02 12:28:33 +09:00
73e5f3da1d
Check supported headers before signature
...
This gives better errors. Also use case-insensitive comparisons.
2025-04-09 14:09:45 +09:00
8417af1f6e
Ignore more new AWS checksum headers
...
Previously newer versions of aws-cli could not call
CreateMultipartUpload. References #760 . References #765 .
2025-04-03 08:00:12 -07:00
e2afed1fd9
Address deprecation and rawtypes warnings
2025-04-02 21:50:56 -07:00
27e51711a4
Bump slf4j.version from 2.0.16 to 2.0.17
...
Bumps `slf4j.version` from 2.0.16 to 2.0.17.
Updates `org.slf4j:slf4j-api` from 2.0.16 to 2.0.17
Updates `org.slf4j:jcl-over-slf4j` from 2.0.16 to 2.0.17
---
updated-dependencies:
- dependency-name: org.slf4j:slf4j-api
dependency-type: direct:production
update-type: version-update:semver-patch
- dependency-name: org.slf4j:jcl-over-slf4j
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2025-04-01 17:41:35 -07:00
134c6c5443
Bump aws-sdk.version from 1.12.780 to 1.12.782
...
Bumps `aws-sdk.version` from 1.12.780 to 1.12.782.
Updates `com.amazonaws:aws-java-sdk-s3` from 1.12.780 to 1.12.782
- [Changelog](https://github.com/aws/aws-sdk-java/blob/master/CHANGELOG.md )
- [Commits](https://github.com/aws/aws-sdk-java/compare/1.12.780...1.12.782 )
Updates `com.amazonaws:aws-java-sdk-sts` from 1.12.780 to 1.12.782
- [Changelog](https://github.com/aws/aws-sdk-java/blob/master/CHANGELOG.md )
- [Commits](https://github.com/aws/aws-sdk-java/compare/1.12.780...1.12.782 )
---
updated-dependencies:
- dependency-name: com.amazonaws:aws-java-sdk-s3
dependency-type: direct:development
update-type: version-update:semver-patch
- dependency-name: com.amazonaws:aws-java-sdk-sts
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2025-04-01 17:41:28 -07:00
5c88a4a388
Bump com.puppycrawl.tools:checkstyle from 10.21.2 to 10.22.0
...
Bumps [com.puppycrawl.tools:checkstyle](https://github.com/checkstyle/checkstyle ) from 10.21.2 to 10.22.0.
- [Release notes](https://github.com/checkstyle/checkstyle/releases )
- [Commits](https://github.com/checkstyle/checkstyle/compare/checkstyle-10.21.2...checkstyle-10.22.0 )
---
updated-dependencies:
- dependency-name: com.puppycrawl.tools:checkstyle
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2025-04-01 17:41:23 -07:00
d191d58dab
Bump com.github.spotbugs:spotbugs-maven-plugin from 4.9.1.0 to 4.9.3.0
...
Bumps [com.github.spotbugs:spotbugs-maven-plugin](https://github.com/spotbugs/spotbugs-maven-plugin ) from 4.9.1.0 to 4.9.3.0.
- [Release notes](https://github.com/spotbugs/spotbugs-maven-plugin/releases )
- [Commits](https://github.com/spotbugs/spotbugs-maven-plugin/compare/spotbugs-maven-plugin-4.9.1.0...spotbugs-maven-plugin-4.9.3.0 )
---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-maven-plugin
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2025-04-01 17:41:15 -07:00
f5d14ba58b
Docker: Use dumb-init as init system
...
PID 1 is special in Linux kernels; do not start the `java` process as
PID 1. Instead, use dumb-init [1] as the init system and start it at
PID 1.
This is generally done for two main reasons [1]:
- Signal handling
When k8s terminates a Pod, it sends a SIGTERM to PID 1 in the
containers. When a non-PID-1 process receives SIGTERM and it does not
register a SIGTERM handler, the kernel will fall back to the default
behavior for a SIGTERM signal: killing the process.
However, this is not the case for PID 1; the kernel does not fallback to
killing the PID 1 process if no SIGTERM handler is registered. This
means that the SIGTERM will have no effect on the process. k8s will wait
for `terminationGracePeriodSeconds` (default: 30 seconds) and only then
SIGKILL the process (having wasted 30 seconds where nothing useful was
happening).
With `Java` as PID 1 though, I don't run into this issue. Even though
s3Proxy does not register a SIGTERM handler, it seems like the JVM does
react to the SIGTERM and immediately kills s3Proxy.
Still, let's use dumb-init as a best-practice. Init-systems like
dumb-init take the responsibliity for properly registering signal
handlers and passing signals to children processes correctly.
- Orphaned processes
Any running process that becomes an orphan (parent process dies) is
adopted by PID 1. This means PID 1 is responsible for cleaning up
(reaping) the orphan process after it has terminated (become zombie
/defunct) However, Java as PID 1 will not know about these zombie
processes and will not reap them. Using dumb-init will reap such
processes.
This is a non-breaking change; users of the docker image do not need any
action on their part.
[1] https://engineeringblog.yelp.com/2016/01/dumb-init-an-init-for-docker.html
[2] https://daveiscoding.hashnode.dev/why-do-you-need-an-init-process-inside-your-docker-container-pid-1
2025-03-06 22:30:54 -08:00
dff88b39f4
Pin Azurite version and update s3-tests
...
References #793 .
2025-03-06 22:28:35 -08:00
a8ebb3999d
Bump docker/metadata-action from 5.6.1 to 5.7.0
...
Bumps [docker/metadata-action](https://github.com/docker/metadata-action ) from 5.6.1 to 5.7.0.
- [Release notes](https://github.com/docker/metadata-action/releases )
- [Commits](https://github.com/docker/metadata-action/compare/v5.6.1...v5.7.0 )
---
updated-dependencies:
- dependency-name: docker/metadata-action
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2025-03-01 22:56:22 -08:00
61459ace6c
Bump com.github.spotbugs:spotbugs-annotations from 4.9.0 to 4.9.2
...
Bumps [com.github.spotbugs:spotbugs-annotations](https://github.com/spotbugs/spotbugs ) from 4.9.0 to 4.9.2.
- [Release notes](https://github.com/spotbugs/spotbugs/releases )
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.9.0...4.9.2 )
---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2025-03-01 22:56:16 -08:00
Andrew Gaul