Missing X-Amz-Version-Id in s3 gw response #544

New issue

Open

opened 2025-12-28 17:37:45 +00:00 by sami · 3 comments

sami commented

2025-12-28 17:37:45 +00:00

Owner

Originally created by @EESergey on GitHub (Dec 19, 2025).

Current Behavior

Approximately 3% of requests do not have X-Amz-Version-Id

---------START-HTTP---------
PUT /bucket001/Hwu2oI2o/835.158vUl%28G4TCuZSgu.rnd HTTP/1.1
Host: #REMOVED
User-Agent: MinIO (linux; amd64) minio-go/v7.0.95 warp/1.3.1
Content-Length: 1198
Authorization: AWS4-HMAC-SHA256 Credential=#REMOVED/20251219/default/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length,Signature=**REDACTED**
Content-Type: application/octet-stream
X-Amz-Content-Sha256: STREAMING-AWS4-HMAC-SHA256-PAYLOAD
X-Amz-Date: 20251219T130133Z
X-Amz-Decoded-Content-Length: 1024
Accept-Encoding: gzip

HTTP/1.1 200 OK
Content-Length: 0
Accept-Ranges: bytes
Date: Fri, 19 Dec 2025 13:01:33 GMT
Etag: 431ecafd92e2b862ed43edddfc4fbe2fc412875009d7c244a0891f3a2880c41b
Server: NeoFS-S3-GW/0.41.5
X-Amz-Request-Id: eb04f063-812d-4c65-8c22-6af1393d3ca3
X-Amz-Version-Id: 9Aa5KAEnkpYPYqg17bwYc6nRLwGFvXwDy5tqdUcGwsts
---------END-HTTP---------
---------START-HTTP---------
PUT /bucket001/Hwu2oI2o/835.158vUl%28G4TCuZSgu.rnd HTTP/1.1
Host: #REMOVED
User-Agent: MinIO (linux; amd64) minio-go/v7.0.95 warp/1.3.1
Content-Length: 1198
Authorization: AWS4-HMAC-SHA256 Credential=#REMOVED/20251219/default/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length,Signature=**REDACTED**
Content-Type: application/octet-stream
X-Amz-Content-Sha256: STREAMING-AWS4-HMAC-SHA256-PAYLOAD
X-Amz-Date: 20251219T130133Z
X-Amz-Decoded-Content-Length: 1024
Accept-Encoding: gzip

HTTP/1.1 200 OK
Content-Length: 0
Accept-Ranges: bytes
Date: Fri, 19 Dec 2025 13:01:33 GMT
Etag: 431ecafd92e2b862ed43edddfc4fbe2fc412875009d7c244a0891f3a2880c41b
Server: NeoFS-S3-GW/0.41.5
X-Amz-Request-Id: 5f905281-8cb5-41ba-aa60-261f30568cf4
---------END-HTTP---------


$ aws s3api list-object-versions --bucket bucket001 --prefix 'Hwu2oI2o/835.158vUl(G4TCuZSgu.rnd'
{
    "Versions": [
        {
            "ETag": "431ecafd92e2b862ed43edddfc4fbe2fc412875009d7c244a0891f3a2880c41b",
            "Size": 1024,
            "Key": "Hwu2oI2o/835.158vUl(G4TCuZSgu.rnd",
            "VersionId": "9Aa5KAEnkpYPYqg17bwYc6nRLwGFvXwDy5tqdUcGwsts",
            "IsLatest": true,
            "LastModified": "2025-12-19T13:01:33+00:00",
            "Owner": {
                "DisplayName": "NVoFb6JUzBFKNYjMvYziBSxvJa29vQPr5c",
                "ID": "NVoFb6JUzBFKNYjMvYziBSxvJa29vQPr5c"
            }
        },
        {
            "ETag": "431ecafd92e2b862ed43edddfc4fbe2fc412875009d7c244a0891f3a2880c41b",
            "Size": 1024,
            "Key": "Hwu2oI2o/835.158vUl(G4TCuZSgu.rnd",
            "VersionId": "null",
            "IsLatest": false,
            "LastModified": "2025-12-19T13:01:33+00:00",
            "Owner": {
                "DisplayName": "NPTQwmvhofhVLNodL9r7eZE2uZcjjgPDts",
                "ID": "NPTQwmvhofhVLNodL9r7eZE2uZcjjgPDts"
            }
        }
    ],
    "RequestCharged": null,
    "Prefix": "Hwu2oI2o/835.158vUl(G4TCuZSgu.rnd"
}

Steps to Reproduce

Run the warp list test with the following parameters:

  params:
    duration: 30s
    concurrent: 12
    objects: 5000
    metadata: false
    max-keys: 1000
    obj:
      size: 1KiB
      versions: 2
      rand-size: false
      part-size:

Originally created by @EESergey on GitHub (Dec 19, 2025). ## Current Behavior Approximately 3% of requests do not have X-Amz-Version-Id ``` ---------START-HTTP--------- PUT /bucket001/Hwu2oI2o/835.158vUl%28G4TCuZSgu.rnd HTTP/1.1 Host: #REMOVED User-Agent: MinIO (linux; amd64) minio-go/v7.0.95 warp/1.3.1 Content-Length: 1198 Authorization: AWS4-HMAC-SHA256 Credential=#REMOVED/20251219/default/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length,Signature=**REDACTED** Content-Type: application/octet-stream X-Amz-Content-Sha256: STREAMING-AWS4-HMAC-SHA256-PAYLOAD X-Amz-Date: 20251219T130133Z X-Amz-Decoded-Content-Length: 1024 Accept-Encoding: gzip HTTP/1.1 200 OK Content-Length: 0 Accept-Ranges: bytes Date: Fri, 19 Dec 2025 13:01:33 GMT Etag: 431ecafd92e2b862ed43edddfc4fbe2fc412875009d7c244a0891f3a2880c41b Server: NeoFS-S3-GW/0.41.5 X-Amz-Request-Id: eb04f063-812d-4c65-8c22-6af1393d3ca3 X-Amz-Version-Id: 9Aa5KAEnkpYPYqg17bwYc6nRLwGFvXwDy5tqdUcGwsts ---------END-HTTP--------- ---------START-HTTP--------- PUT /bucket001/Hwu2oI2o/835.158vUl%28G4TCuZSgu.rnd HTTP/1.1 Host: #REMOVED User-Agent: MinIO (linux; amd64) minio-go/v7.0.95 warp/1.3.1 Content-Length: 1198 Authorization: AWS4-HMAC-SHA256 Credential=#REMOVED/20251219/default/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length,Signature=**REDACTED** Content-Type: application/octet-stream X-Amz-Content-Sha256: STREAMING-AWS4-HMAC-SHA256-PAYLOAD X-Amz-Date: 20251219T130133Z X-Amz-Decoded-Content-Length: 1024 Accept-Encoding: gzip HTTP/1.1 200 OK Content-Length: 0 Accept-Ranges: bytes Date: Fri, 19 Dec 2025 13:01:33 GMT Etag: 431ecafd92e2b862ed43edddfc4fbe2fc412875009d7c244a0891f3a2880c41b Server: NeoFS-S3-GW/0.41.5 X-Amz-Request-Id: 5f905281-8cb5-41ba-aa60-261f30568cf4 ---------END-HTTP--------- $ aws s3api list-object-versions --bucket bucket001 --prefix 'Hwu2oI2o/835.158vUl(G4TCuZSgu.rnd' { "Versions": [ { "ETag": "431ecafd92e2b862ed43edddfc4fbe2fc412875009d7c244a0891f3a2880c41b", "Size": 1024, "Key": "Hwu2oI2o/835.158vUl(G4TCuZSgu.rnd", "VersionId": "9Aa5KAEnkpYPYqg17bwYc6nRLwGFvXwDy5tqdUcGwsts", "IsLatest": true, "LastModified": "2025-12-19T13:01:33+00:00", "Owner": { "DisplayName": "NVoFb6JUzBFKNYjMvYziBSxvJa29vQPr5c", "ID": "NVoFb6JUzBFKNYjMvYziBSxvJa29vQPr5c" } }, { "ETag": "431ecafd92e2b862ed43edddfc4fbe2fc412875009d7c244a0891f3a2880c41b", "Size": 1024, "Key": "Hwu2oI2o/835.158vUl(G4TCuZSgu.rnd", "VersionId": "null", "IsLatest": false, "LastModified": "2025-12-19T13:01:33+00:00", "Owner": { "DisplayName": "NPTQwmvhofhVLNodL9r7eZE2uZcjjgPDts", "ID": "NPTQwmvhofhVLNodL9r7eZE2uZcjjgPDts" } } ], "RequestCharged": null, "Prefix": "Hwu2oI2o/835.158vUl(G4TCuZSgu.rnd" } ``` ## Steps to Reproduce Run the warp list test with the following parameters: ``` params: duration: 30s concurrent: 12 objects: 5000 metadata: false max-keys: 1000 obj: size: 1KiB versions: 2 rand-size: false part-size: ```

sami added the

bug

labels

2025-12-28 17:37:45 +00:00

sami commented

2025-12-28 17:37:45 +00:00

Author

Owner

@roman-khimov commented on GitHub (Dec 19, 2025):

Bucket config, container setup?

@roman-khimov commented on GitHub (Dec 19, 2025): Bucket config, container setup?

sami commented

2025-12-28 17:37:45 +00:00

Author

Owner

@roman-khimov commented on GitHub (Dec 19, 2025):

Can be the same as https://rest.fs.neo.org/Aku38wpn2xEqfEPxmwyAa7N3VnAkMV8PEVkpLqwKM1z3/4530-1766014379/index.html#suites/c859c7aaaab4c135ad3947c79c74a722/bd2c0807b905d36f/

@roman-khimov commented on GitHub (Dec 19, 2025): Can be the same as https://rest.fs.neo.org/Aku38wpn2xEqfEPxmwyAa7N3VnAkMV8PEVkpLqwKM1z3/4530-1766014379/index.html#suites/c859c7aaaab4c135ad3947c79c74a722/bd2c0807b905d36f/

sami commented

2025-12-28 17:37:46 +00:00

Author

Owner

@EESergey commented on GitHub (Dec 22, 2025):

The issue was caused by the bucket being created through one node, but versioning was enabled through another. The bucket settings cache on the first node had an incorrect versioning parameter, causing objects uploaded through that node to lack the X-Amz-Version-Id attribute.

@EESergey commented on GitHub (Dec 22, 2025): The issue was caused by the bucket being created through one node, but versioning was enabled through another. The bucket settings cache on the first node had an incorrect versioning parameter, causing objects uploaded through that node to lack the X-Amz-Version-Id attribute.