S3 default storage class breaks Minio compatibility #5984

New Issue

GiteaMirror · 2026-03-07T20:44:29-06:00

GiteaMirror commented

2026-03-07 20:44:29 -06:00

Originally created by @tycho on GitHub (Jul 28, 2025).

This line:

a0198d8d7c/src/config.rs (L1243)

Causes problems if using Minio as the S3 endpoint:

[REQUEST s3.PutObject] [2025-07-28T10:34:52.211] [Client IP: redacted]
PUT /rsa_key.pem
Proto: HTTP/1.1
Host: [redacted]
X-Amz-Content-Sha256: UNSIGNED-PAYLOAD
X-Forwarded-For: [redacted]
X-Forwarded-Proto: https
Accept: */*
Authorization: AWS4-HMAC-SHA256 Credential=vaultwarden/20250728/us-east-1/s3/aws4_request, SignedHeaders=content-length;host;x-amz-content-sha256;x-amz-date;x-amz-storage-class, Signature=4d257b5c8807a7cddf1c2c49749af1f96dce936a0cdf11f96ed363fce5323f81
X-Amz-Date: 20250728T173452Z
X-Amz-Storage-Class: INTELLIGENT_TIERING
X-Real-Ip: [redacted]
Accept-Encoding: gzip, br, zstd, deflate
Content-Length: 1675
<BLOB>
[RESPONSE] [2025-07-28T10:34:52.211] [ Duration 58µs TTFB 50.045µs ↑ 153 B  ↓ 338 B ]
400 Bad Request
Content-Length: 338
Content-Type: application/xml
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin,Accept-Encoding
X-Amz-Id-2: 37b62f02de360c6949659e1d2fb72852efb5d74e5dcbc7d4e665920a1c72a29e
X-Ratelimit-Limit: 20413
X-Xss-Protection: 1; mode=block
Server: MinIO
X-Amz-Request-Id: 18567AC52E4FC471
X-Content-Type-Options: nosniff
X-Ratelimit-Remaining: 20413
Accept-Ranges: bytes
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>InvalidStorageClass</Code><Message>Invalid storage class.</Message><Key>rsa_key.pem</Key><BucketName>vaultwarden</BucketName><Resource>/rsa_key.pem</Resource><RequestId>18567AC52E4FC471</RequestId><HostId>37b62f02de360c6949659e1d2fb72852efb5d74e5dcbc7d4e665920a1c72a29e</HostId></Error>

Can that storage class configuration be made user-configurable or even optional, please?

Originally created by @tycho on GitHub (Jul 28, 2025). This line: https://github.com/dani-garcia/vaultwarden/blob/a0198d8d7cadc666643a4e32a2a8d4217038ea3a/src/config.rs#L1243 Causes problems if using [Minio](https://github.com/minio) as the S3 endpoint: ``` [REQUEST s3.PutObject] [2025-07-28T10:34:52.211] [Client IP: redacted] PUT /rsa_key.pem Proto: HTTP/1.1 Host: [redacted] X-Amz-Content-Sha256: UNSIGNED-PAYLOAD X-Forwarded-For: [redacted] X-Forwarded-Proto: https Accept: */* Authorization: AWS4-HMAC-SHA256 Credential=vaultwarden/20250728/us-east-1/s3/aws4_request, SignedHeaders=content-length;host;x-amz-content-sha256;x-amz-date;x-amz-storage-class, Signature=4d257b5c8807a7cddf1c2c49749af1f96dce936a0cdf11f96ed363fce5323f81 X-Amz-Date: 20250728T173452Z X-Amz-Storage-Class: INTELLIGENT_TIERING X-Real-Ip: [redacted] Accept-Encoding: gzip, br, zstd, deflate Content-Length: 1675 <BLOB> [RESPONSE] [2025-07-28T10:34:52.211] [ Duration 58µs TTFB 50.045µs ↑ 153 B ↓ 338 B ] 400 Bad Request Content-Length: 338 Content-Type: application/xml Strict-Transport-Security: max-age=31536000; includeSubDomains Vary: Origin,Accept-Encoding X-Amz-Id-2: 37b62f02de360c6949659e1d2fb72852efb5d74e5dcbc7d4e665920a1c72a29e X-Ratelimit-Limit: 20413 X-Xss-Protection: 1; mode=block Server: MinIO X-Amz-Request-Id: 18567AC52E4FC471 X-Content-Type-Options: nosniff X-Ratelimit-Remaining: 20413 Accept-Ranges: bytes <?xml version="1.0" encoding="UTF-8"?> <Error><Code>InvalidStorageClass</Code><Message>Invalid storage class.</Message><Key>rsa_key.pem</Key><BucketName>vaultwarden</BucketName><Resource>/rsa_key.pem</Resource><RequestId>18567AC52E4FC471</RequestId><HostId>37b62f02de360c6949659e1d2fb72852efb5d74e5dcbc7d4e665920a1c72a29e</HostId></Error> ``` Can that storage class configuration be made user-configurable or even optional, please?

GiteaMirror added the enhancement low priority labels 2026-03-07 20:44:29 -06:00

GiteaMirror commented

2026-03-07 20:44:30 -06:00

@tycho commented on GitHub (Jul 28, 2025):

Additionally, this line makes it hard to use other S3-compatible implementations like Garage:
a0198d8d7c/src/config.rs (L1240)

@tycho commented on GitHub (Jul 28, 2025): Additionally, this line makes it hard to use other S3-compatible implementations like Garage: https://github.com/dani-garcia/vaultwarden/blob/a0198d8d7cadc666643a4e32a2a8d4217038ea3a/src/config.rs#L1240

GiteaMirror commented

2026-03-07 20:44:31 -06:00

@BlackDex commented on GitHub (Jul 28, 2025):

@txase, you might have some ideas here?

@BlackDex commented on GitHub (Jul 28, 2025): @txase, you might have some ideas here?

GiteaMirror commented

2026-03-07 20:44:31 -06:00

@txase commented on GitHub (Jul 28, 2025):

I wonder if we can look at the bucket config to determine if we should call these methods or not. We definitely want these options if it’s a real aws s3 bucket. If it’s not, then we can skip calling these methods.

I can look into this within the next few days and report back here on what I think we should do.

@txase commented on GitHub (Jul 28, 2025): I wonder if we can look at the bucket config to determine if we should call these methods or not. We definitely want these options if it’s a real aws s3 bucket. If it’s not, then we can skip calling these methods. I can look into this within the next few days and report back here on what I think we should do.

GiteaMirror commented

2026-03-07 20:44:32 -06:00

@rezzorix commented on GitHub (Jul 29, 2025):

Just wondering... isnt MiniO something to avoid or move away from anyway considering what the developers recently pulled?

Just one example; https://www.reddit.com/r/selfhosted/comments/1kva3pw/avoid_minio_developers_introduce_trojan_horse/

So maybe not worth to spend too many resources and effort on a software that will be not so popular overall anymore?

@rezzorix commented on GitHub (Jul 29, 2025): Just wondering... isnt MiniO something to avoid or move away from anyway considering what the developers recently pulled? Just one example; https://www.reddit.com/r/selfhosted/comments/1kva3pw/avoid_minio_developers_introduce_trojan_horse/ So maybe not worth to spend too many resources and effort on a software that will be not so popular overall anymore?

GiteaMirror commented

2026-03-07 20:44:32 -06:00

@Asutorufa commented on GitHub (Jul 29, 2025):

The aws s3 also support Use Path Style, so consider setting Use Path Style to true.
There are many S3-compatible implementations, such as Cloudflare R2, Cephfs, SeaweedFS, Garage, and minio.

@Asutorufa commented on GitHub (Jul 29, 2025): The `aws s3` also support `Use Path Style`, so consider setting `Use Path Style` to true. There are many S3-compatible implementations, such as `Cloudflare R2`, `Cephfs`, `SeaweedFS`, `Garage`, and `minio`.

GiteaMirror commented

2026-03-07 20:44:33 -06:00

@txase commented on GitHub (Jul 29, 2025):

AWS has said they are deprecating path style hosting. That said, it’s been so long now that it’s unclear if they really will do it. But it is clear that they suggest virtual host style hosting is the preferred way to operate.

@txase commented on GitHub (Jul 29, 2025): AWS has said they are [deprecating path style hosting](https://aws.amazon.com/blogs/storage/update-to-amazon-s3-path-deprecation-plan/). That said, it’s been so long now that it’s unclear if they really will do it. But it is clear that they suggest virtual host style hosting is the preferred way to operate.

GiteaMirror commented

2026-03-07 20:44:34 -06:00

@txase commented on GitHub (Jul 29, 2025):

I have now reviewed s3 and a few compatible services, including minio and garage. I think we need an extensible solution for enabling s3 features, as each service supports different sets of capabilities. @tycho @BlackDex: Please respond with agreement or concerns with the following proposal. 🙏

Proposal

We currently support a URI for file paths for s3-compatible stores where the URI starts with s3://:

5d84f17600/src/config.rs (L1173)

We can add support for querystring parameters for s3 features, which would be similar to how many database connection strings support parameters. For example, we could have the following for minio:

s3://my-minio-bucket?storage-class=STANDARD

And for garage:

s3://my-garage-bucket?path-style-requests

An example of this parameterization approach can be found in the Go Cloud Development Kit.

Alternative Proposal 1: Provider Parameters

We could set a provider parameter: s3://my-minio-bucket?provider=minio to determine which s3 features to enable. However, I think it is likely easier to maintain per-feature params over time (e.g. if minio began supporting intelligent tiering tomorrow, what should the Vaultwarden provider=minio functionality do?). It also means each service provider must have its own hardcoded set of features, and/or users must map new service providers to other existing service providers already implemented in Vaultwarden that enable a compatible set of features.

Alternative Proposal 2: Provider Protocols

We could use the URI protocol to specify the provider type: minio://my-minio-bucket. But this has the same long-term maintenance issue as the above provider parameter alternative proposal.

Alternative Proposal 3: Dynamic Checks

When we first access an s3-compatible service we could check whether it supports various features. However, this may require multiple requests and the possibility of mutating the underlying s3 store. For example, there is no way to check whether intelligent tiering is available other than to attempt to create an object. This will also increase startup latency, which is a problem when running Vaultwarden in a serverless FaaS (Functions-as-a-Service) service like AWS Lambda.

@txase commented on GitHub (Jul 29, 2025): I have now reviewed s3 and a few compatible services, including minio and garage. I think we need an extensible solution for enabling s3 features, as each service supports different sets of capabilities. @tycho @BlackDex: Please respond with agreement or concerns with the following proposal. 🙏 ## Proposal We currently support a URI for file paths for s3-compatible stores where the URI starts with `s3://`: https://github.com/dani-garcia/vaultwarden/blob/5d84f17600e179280e44c391b92ee9eecc2b7cdc/src/config.rs#L1173 We can add support for querystring parameters for s3 features, which would be similar to how many database connection strings support parameters. For example, we could have the following for minio: `s3://my-minio-bucket?storage-class=STANDARD` And for garage: `s3://my-garage-bucket?path-style-requests` An example of this parameterization approach can be found in the [Go Cloud Development Kit](https://gocloud.dev/howto/blob/#s3). ### Alternative Proposal 1: Provider Parameters We could set a provider parameter: `s3://my-minio-bucket?provider=minio` to determine which s3 features to enable. However, I think it is likely easier to maintain per-feature params over time (e.g. if minio began supporting intelligent tiering tomorrow, what should the Vaultwarden `provider=minio` functionality do?). It also means each service provider must have its own hardcoded set of features, and/or users must map new service providers to other existing service providers already implemented in Vaultwarden that enable a compatible set of features. ### Alternative Proposal 2: Provider Protocols We could use the URI protocol to specify the provider type: `minio://my-minio-bucket`. But this has the same long-term maintenance issue as the above provider parameter alternative proposal. ### Alternative Proposal 3: Dynamic Checks When we first access an s3-compatible service we could check whether it supports various features. However, this may require multiple requests and the possibility of mutating the underlying s3 store. For example, there is no way to check whether intelligent tiering is available other than to attempt to create an object. This will also increase startup latency, which is a problem when running Vaultwarden in a serverless FaaS (Functions-as-a-Service) service like AWS Lambda.

GiteaMirror commented

2026-03-07 20:44:35 -06:00

@dani-garcia commented on GitHub (Jul 29, 2025):

I think your first proposal looks pretty good and it also mirrors something that OpenDAL itself wants to implement in the future: https://github.com/apache/opendal/issues/5445. That seems like it would resolve this issue for us, but it's not implemented yet. I'd recommend to try to keep it similar to the OpenDAL config naming, so we can migrate over once that's released, so something like might work best:

s3://my-minio-bucket?default_storage_class=STANDARD
s3://my-garage-bucket?enable_virtual_host_style=false

@dani-garcia commented on GitHub (Jul 29, 2025): I think your first proposal looks pretty good and it also mirrors something that OpenDAL itself wants to implement in the future: https://github.com/apache/opendal/issues/5445. That seems like it would resolve this issue for us, but it's not implemented yet. I'd recommend to try to keep it similar to the [OpenDAL config](https://github.com/apache/opendal/blob/main/core/src/services/s3/config.rs) naming, so we can migrate over once that's released, so something like might work best: ``` s3://my-minio-bucket?default_storage_class=STANDARD s3://my-garage-bucket?enable_virtual_host_style=false ```

GiteaMirror commented

2026-03-07 20:44:35 -06:00

@txase commented on GitHub (Jul 29, 2025):

@dani-garcia Oh, good find!

This reminds me that @Xuanwo chimed in to offer guidance on integrating OpenDAL in #5626. @Xuanwo: Can you confirm that this is the best approach, and that the specific params and values @dani-garcia provided above would be correct for future OpenDAL compatibility?

@txase commented on GitHub (Jul 29, 2025): @dani-garcia Oh, good find! This reminds me that @Xuanwo chimed in to offer guidance on integrating OpenDAL in #5626. @Xuanwo: Can you confirm that this is the best approach, and that the specific params and values @dani-garcia provided above would be correct for future OpenDAL compatibility?

GiteaMirror commented

2026-03-07 20:44:37 -06:00

@Xuanwo commented on GitHub (Jul 29, 2025):

Hi, I think the parameters proposed by @dani-garcia are correct and compatible.

@Xuanwo commented on GitHub (Jul 29, 2025): Hi, I think the parameters proposed by @dani-garcia are correct and compatible.

GiteaMirror commented

2026-03-07 20:44:38 -06:00

@BlackDex commented on GitHub (Jul 29, 2025):

Looks good to me too. And, it also mimics the same way as is done for Diesel's Database connections.

@BlackDex commented on GitHub (Jul 29, 2025): Looks good to me too. And, it also mimics the same way as is done for Diesel's Database connections.

GiteaMirror commented

2026-03-07 20:44:39 -06:00

@txase commented on GitHub (Jul 29, 2025):

Great!

@tycho: Is this something you would have an interest in drafting a PR for to fix your issue? If you're not able (e.g. you're not familiar enough with Rust), I can probably get to it sometime in the next week. But I don't want to steal away an opportunity for you to have fun getting it working :).

@txase commented on GitHub (Jul 29, 2025): Great! @tycho: Is this something you would have an interest in drafting a PR for to fix your issue? If you're not able (e.g. you're not familiar enough with Rust), I can probably get to it sometime in the next week. But I don't want to steal away an opportunity for you to have fun getting it working :).

GiteaMirror commented

2026-03-07 20:44:40 -06:00

@tycho commented on GitHub (Jul 29, 2025):

I might be able to, however I've just started a new job so I might not have time for a bit. If someone can get something done sooner, I'd really appreciate if they could work on it!

I think the best path forward is the initial proposal by @txase, with specific functionality overrides in query strings. We could customize things like:

virtual_host_style, 1 or 0, to enable use of virtual host-based vs. path-based respectively
storage_class, supporting any option described in the OpenDAL docs, though we can probably just pass it through directly as a string and let OpenDAL decide whether it's valid
access_key_id and secret_access_key, it might make sense to allow explicitly defining credentials in the connection string, instead of relying on the AWS CLI being configured, especially in a service account like Vaultwarden would typically use (that was a pain point when setting this up for me actually)
endpoint_uri, another useful one for people that don't have the AWS CLI configured -- when using an S3-compatible self-hosted option, you don't want to use the default https://s3.amazonaws.com endpoint, you want to use a custom one. Right now to do that you have to plug it into the AWS CLI config, so having an explicit option would be good!

So a complete S3 storage URI would probably look something like:

s3://bucket-name?endpoint_uri=https%3A%2F%2Fminio.example.com&access_key_id=SOME_ACCESS_KEY_ID&secret_access_key=SOME_SECRET_ACCESS_KEY&storage_class=STANDARD&virtual_host_style=0

@tycho commented on GitHub (Jul 29, 2025): I might be able to, **however** I've just started a new job so I might not have time for a bit. If someone can get something done sooner, I'd really appreciate if they could work on it! I think the best path forward is the initial proposal by @txase, with specific functionality overrides in query strings. We could customize things like: - `virtual_host_style`, 1 or 0, to enable use of virtual host-based vs. path-based respectively - `storage_class`, supporting any option described in the [OpenDAL docs](https://opendal.apache.org/docs/rust/opendal/services/struct.S3.html#method.default_storage_class), though we can probably just pass it through directly as a string and let OpenDAL decide whether it's valid - `access_key_id` and `secret_access_key`, it might make sense to allow explicitly defining credentials in the connection string, instead of relying on the AWS CLI being configured, especially in a service account like Vaultwarden would typically use (that was a pain point when setting this up for me actually) - `endpoint_uri`, another useful one for people that don't have the AWS CLI configured -- when using an S3-compatible self-hosted option, you don't want to use the default `https://s3.amazonaws.com` endpoint, you want to use a custom one. Right now to do that you have to plug it into the AWS CLI config, so having an explicit option would be good! So a complete S3 storage URI would probably look something like: ``` s3://bucket-name?endpoint_uri=https%3A%2F%2Fminio.example.com&access_key_id=SOME_ACCESS_KEY_ID&secret_access_key=SOME_SECRET_ACCESS_KEY&storage_class=STANDARD&virtual_host_style=0 ```

GiteaMirror commented

2026-03-07 20:44:40 -06:00

@Xuanwo commented on GitHub (Jul 30, 2025):

access_key_id and secret_access_key, it might make sense to allow explicitly defining credentials in the connection string, instead of relying on the AWS CLI being configured, especially in a service account like Vaultwarden would typically use (that was a pain point when setting this up for me actually)

Hi, OpenDAL supports ENV, IMDSv2, and many other credential loaders, so usually you can just use IAM to configure them instead of static keys, which are insecure.

@Xuanwo commented on GitHub (Jul 30, 2025): > * `access_key_id` and `secret_access_key`, it might make sense to allow explicitly defining credentials in the connection string, instead of relying on the AWS CLI being configured, especially in a service account like Vaultwarden would typically use (that was a pain point when setting this up for me actually) Hi, OpenDAL supports ENV, IMDSv2, and many other credential loaders, so usually you can just use IAM to configure them instead of static keys, which are insecure.

GiteaMirror commented

2026-03-07 20:44:41 -06:00

@txase commented on GitHub (Aug 3, 2025):

PR is posted: #6127. Testing from others using non-AWS S3 services would be greatly appreciated :).

@txase commented on GitHub (Aug 3, 2025): PR is posted: #6127. Testing from others using non-AWS S3 services would be greatly appreciated :).

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/vaultwarden#5984