nspcc-dev/neofs-s3-gw
1
0
Fork 0
mirror of https://github.com/nspcc-dev/neofs-s3-gw.git synced 2026-03-01 04:29:15 +00:00
Code Issues 24 Projects Packages Wiki Activity

Cannot delete bucket after restoring versioned object #286

New issue
Closed
opened 2025-12-28 17:36:52 +00:00 by sami · 10 comments
sami commented 2025-12-28 17:36:52 +00:00
Owner
Copy link

Originally created by @vdomnich-yadro on GitHub (Jul 20, 2022).

Originally assigned to: @KirillovDenis on GitHub.

Steps to Reproduce (for bugs)

The issue can be reproduced both with AWS CLI and with boto3 client. Steps are:

  • Create new S3 bucket
  • Enable versioning on the bucket
  • Put object to the bucket
  • Put a new version of the object to the bucket
  • Delete object and capture the resulting VersionId
  • Restore object (by deleting the VersionId from previous step)
  • List objects in the bucket, it will return the restored object
  • Delete the object
  • Delete the bucket

Please, find attached logs for automated test run:
aws-cli-log.txt
boto3-log.txt

Expected Behavior

The bucket is deleted.

Current Behavior

The delete command fails with error:

An error occurred (BucketNotEmpty) when calling the DeleteBucket operation: The bucket you tried to delete is not empty

Possible Solution

Maybe the bucket deletion code doesn't handle deletion marker properly when it checks that the bucket is empty.

Your Environment

  • Version used:
COMMAND: 'neo-go --version'
OUTPUT:
 neo-go version 0.98.5

OMMAND: 'neofs-cli --version'
OUTPUT:
 Version: v0.28.1 
Build: 2022-05-05T09:07:45+00:00 
Debug: false

COMMAND: 'neofs-authmate --version'
OUTPUT:
 NeoFS gate authentication manager version v0.21.1

COMMAND: 'aws --version'
OUTPUT:
 2.2.0
AWS CLI v2 command: /root/.awscliv2/binaries/aws
aws-cli/2.7.11 Python/3.9.11 Linux/5.10.0-15-amd64 exe/x86_64.debian.11 prompt/off
  • Server setup and configuration:
<project name="morph-crest.git" path="morph-crest" revision="a915cb945b7571c6098a21185bba620f7e0516eb" upstream="master" dest-branch="master" />
<project name="neo-go.git" path="neo-go" revision="b21a3c992de510c0598828fd9b6db9b35af33f3f" upstream="add-dbft-watchdog" dest-branch="add-dbft-watchdog" />
<project name="neo-preodolenie.git" path="neo-preodolenie" revision="a6fb56b26392cf837940483d09dcc6db272d9427" upstream="master" dest-branch="master" />
<project name="neofs-contract.git" path="neofs-contract" revision="a92ad8b8f0e7f5265bced249949ea7cbc9fdad75" upstream="refs/tags/v0.15.2" dest-branch="refs/tags/v0.15.2" />
<project name="neofs-dev-env.git" path="neofs-dev-env" revision="4f53093b186e8f4f5f8dd6d2946a9ca699235907" upstream="internal_exp_b" dest-branch="internal_exp_b" />
<project name="neofs-http-gw.git" path="neofs-http-gw" revision="6ce5fe80255ac2085031c57ddd88fbd1fd5b1219" upstream="refs/tags/v0.21.0" dest-branch="refs/tags/v0.21.0" />
<project name="neofs-keywords.git" path="neofs-testcases/neofs-keywords" revision="98b7f2601e5cc4f67184d4321da56df5531b68d5" upstream="internal_tmp_b" dest-branch="internal_tmp_b" />
<project name="neofs-keywords.git" path="neofs-keywords" revision="58650099362f05b51b97f04e2244a4ba20f88351" upstream="master" dest-branch="master" />
<project name="neofs-locode-db.git" path="neofs-locode-db" revision="ffd8516fccb63c9955f1d4202fb8f5b858188aa1" upstream="refs/tags/v0.2.1" dest-branch="refs/tags/v0.2.1" />
<project name="neofs-node.git" path="neofs-node" revision="34d20fd592f24d7b16fc4b119ad7675d6e01baf7" upstream="experimental" dest-branch="experimental" />
<project name="neofs-s3-gw.git" path="neofs-s3-gw" revision="03a68d0f9fc2ee020596c55445e86e493a5da5dd" upstream="tree-service" dest-branch="tree-service" />
<project name="neofs-testcases.git" path="neofs-testcases" revision="bed41402eedaed234349bfab2e22331c061a415b" upstream="internal_tmp_b" dest-branch="internal_tmp_b" />
Originally created by @vdomnich-yadro on GitHub (Jul 20, 2022). Originally assigned to: @KirillovDenis on GitHub. ## Steps to Reproduce (for bugs) The issue can be reproduced both with AWS CLI and with boto3 client. Steps are: * Create new S3 bucket * Enable versioning on the bucket * Put object to the bucket * Put a new version of the object to the bucket * Delete object and capture the resulting `VersionId` * Restore object (by deleting the `VersionId` from previous step) * List objects in the bucket, it will return the restored object * Delete the object * Delete the bucket Please, find attached logs for automated test run: [aws-cli-log.txt](https://github.com/nspcc-dev/neofs-s3-gw/files/9150777/aws-cli-log.txt) [boto3-log.txt](https://github.com/nspcc-dev/neofs-s3-gw/files/9150778/boto3-log.txt) ## Expected Behavior The bucket is deleted. ## Current Behavior The delete command fails with error: ``` An error occurred (BucketNotEmpty) when calling the DeleteBucket operation: The bucket you tried to delete is not empty ``` ## Possible Solution Maybe the bucket deletion code doesn't handle deletion marker properly when it checks that the bucket is empty. ## Your Environment * Version used: ``` COMMAND: 'neo-go --version' OUTPUT: neo-go version 0.98.5 OMMAND: 'neofs-cli --version' OUTPUT: Version: v0.28.1 Build: 2022-05-05T09:07:45+00:00 Debug: false COMMAND: 'neofs-authmate --version' OUTPUT: NeoFS gate authentication manager version v0.21.1 COMMAND: 'aws --version' OUTPUT: 2.2.0 AWS CLI v2 command: /root/.awscliv2/binaries/aws aws-cli/2.7.11 Python/3.9.11 Linux/5.10.0-15-amd64 exe/x86_64.debian.11 prompt/off ``` * Server setup and configuration: ``` <project name="morph-crest.git" path="morph-crest" revision="a915cb945b7571c6098a21185bba620f7e0516eb" upstream="master" dest-branch="master" /> <project name="neo-go.git" path="neo-go" revision="b21a3c992de510c0598828fd9b6db9b35af33f3f" upstream="add-dbft-watchdog" dest-branch="add-dbft-watchdog" /> <project name="neo-preodolenie.git" path="neo-preodolenie" revision="a6fb56b26392cf837940483d09dcc6db272d9427" upstream="master" dest-branch="master" /> <project name="neofs-contract.git" path="neofs-contract" revision="a92ad8b8f0e7f5265bced249949ea7cbc9fdad75" upstream="refs/tags/v0.15.2" dest-branch="refs/tags/v0.15.2" /> <project name="neofs-dev-env.git" path="neofs-dev-env" revision="4f53093b186e8f4f5f8dd6d2946a9ca699235907" upstream="internal_exp_b" dest-branch="internal_exp_b" /> <project name="neofs-http-gw.git" path="neofs-http-gw" revision="6ce5fe80255ac2085031c57ddd88fbd1fd5b1219" upstream="refs/tags/v0.21.0" dest-branch="refs/tags/v0.21.0" /> <project name="neofs-keywords.git" path="neofs-testcases/neofs-keywords" revision="98b7f2601e5cc4f67184d4321da56df5531b68d5" upstream="internal_tmp_b" dest-branch="internal_tmp_b" /> <project name="neofs-keywords.git" path="neofs-keywords" revision="58650099362f05b51b97f04e2244a4ba20f88351" upstream="master" dest-branch="master" /> <project name="neofs-locode-db.git" path="neofs-locode-db" revision="ffd8516fccb63c9955f1d4202fb8f5b858188aa1" upstream="refs/tags/v0.2.1" dest-branch="refs/tags/v0.2.1" /> <project name="neofs-node.git" path="neofs-node" revision="34d20fd592f24d7b16fc4b119ad7675d6e01baf7" upstream="experimental" dest-branch="experimental" /> <project name="neofs-s3-gw.git" path="neofs-s3-gw" revision="03a68d0f9fc2ee020596c55445e86e493a5da5dd" upstream="tree-service" dest-branch="tree-service" /> <project name="neofs-testcases.git" path="neofs-testcases" revision="bed41402eedaed234349bfab2e22331c061a415b" upstream="internal_tmp_b" dest-branch="internal_tmp_b" /> ```
sami 2025-12-28 17:36:52 +00:00
sami commented 2025-12-28 17:36:52 +00:00
Author
Owner
Copy link

@KirillovDenis commented on GitHub (Jul 20, 2022):

What version of neofs-s3-gw do you use? The same as neofs-authmate - v0.21.1?

@KirillovDenis commented on GitHub (Jul 20, 2022): What version of `neofs-s3-gw` do you use? The same as `neofs-authmate` - `v0.21.1`?
sami commented 2025-12-28 17:36:52 +00:00
Author
Owner
Copy link

@vdomnich-yadro commented on GitHub (Jul 20, 2022):

@KirillovDenis updated the description and included this info

@vdomnich-yadro commented on GitHub (Jul 20, 2022): @KirillovDenis updated the description and included this info
sami commented 2025-12-28 17:36:52 +00:00
Author
Owner
Copy link

@vdomnich-yadro commented on GitHub (Jul 20, 2022):

@anatoly-bogatyrev this is the issue that we discussed yesterday, it was not a sporadic problem and reproduces quite stably.
I can't assign labels to issues, but this should go under label P0, thanks!

@vdomnich-yadro commented on GitHub (Jul 20, 2022): @anatoly-bogatyrev this is the issue that we discussed yesterday, it was not a sporadic problem and reproduces quite stably. I can't assign labels to issues, but this should go under label `P0`, thanks!
sami commented 2025-12-28 17:36:52 +00:00
Author
Owner
Copy link

@alexvanin commented on GitHub (Jul 20, 2022):

#!/bin/bash -x

BUCK=bucket01
ENDPOINT="http://127.0.0.1:8080"

echo "File" > data.txt
aws s3api create-bucket --bucket $BUCK --endpoint-url $ENDPOINT
aws s3api put-bucket-versioning --bucket $BUCK --versioning-configuration Status=Enabled --endpoint-url $ENDPOINT
aws s3api put-object --bucket $BUCK --key versioned-object --body data.txt --endpoint-url $ENDPOINT
echo "NewFile" > data.txt
aws s3api put-object --bucket $BUCK --key versioned-object --body data.txt --endpoint-url $ENDPOINT
del=`aws s3api delete-object --bucket $BUCK --key versioned-object --endpoint-url $ENDPOINT | jq .VersionId | sed 's/\"//g'`
aws s3api delete-object --bucket $BUCK --version-id $del --key versioned-object --endpoint-url $ENDPOINT
aws s3api list-object-versions --bucket $BUCK --endpoint-url $ENDPOINT
aws s3api delete-object --bucket $BUCK --key versioned-object --endpoint-url $ENDPOINT
aws s3api delete-bucket --bucket $BUCK --endpoint-url $ENDPOINT

Reproduced in tree-service (v0.21.1-119-ga8d6916)

@alexvanin commented on GitHub (Jul 20, 2022): ```sh #!/bin/bash -x BUCK=bucket01 ENDPOINT="http://127.0.0.1:8080" echo "File" > data.txt aws s3api create-bucket --bucket $BUCK --endpoint-url $ENDPOINT aws s3api put-bucket-versioning --bucket $BUCK --versioning-configuration Status=Enabled --endpoint-url $ENDPOINT aws s3api put-object --bucket $BUCK --key versioned-object --body data.txt --endpoint-url $ENDPOINT echo "NewFile" > data.txt aws s3api put-object --bucket $BUCK --key versioned-object --body data.txt --endpoint-url $ENDPOINT del=`aws s3api delete-object --bucket $BUCK --key versioned-object --endpoint-url $ENDPOINT | jq .VersionId | sed 's/\"//g'` aws s3api delete-object --bucket $BUCK --version-id $del --key versioned-object --endpoint-url $ENDPOINT aws s3api list-object-versions --bucket $BUCK --endpoint-url $ENDPOINT aws s3api delete-object --bucket $BUCK --key versioned-object --endpoint-url $ENDPOINT aws s3api delete-bucket --bucket $BUCK --endpoint-url $ENDPOINT ``` Reproduced in tree-service (v0.21.1-119-ga8d6916)
sami commented 2025-12-28 17:36:52 +00:00
Author
Owner
Copy link

@KirillovDenis commented on GitHub (Jul 20, 2022):

@vdomnich-yadro Could you provide content of file:///tests/TemporaryDir//delete.json?

@KirillovDenis commented on GitHub (Jul 20, 2022): @vdomnich-yadro Could you provide content of `file:///tests/TemporaryDir//delete.json`?
sami commented 2025-12-28 17:36:52 +00:00
Author
Owner
Copy link

@vdomnich-yadro commented on GitHub (Jul 20, 2022):

@KirillovDenis Sure, here you go:

{ "Objects": [ { "Key": "47bccf87-38a2-4a3f-9990-b98234a875be" } ] }

We are passing just object key, without version.

@vdomnich-yadro commented on GitHub (Jul 20, 2022): @KirillovDenis Sure, here you go: ``` { "Objects": [ { "Key": "47bccf87-38a2-4a3f-9990-b98234a875be" } ] } ``` We are passing just object key, without version.
sami commented 2025-12-28 17:36:52 +00:00
Author
Owner
Copy link

@KirillovDenis commented on GitHub (Jul 21, 2022):

In this case there are two object versions and one delete marker in the bucket. It seems bucket shouldn't be deleted then because of AWS spec: All objects (including all object versions and delete markers) in the bucket must be deleted before the bucket itself can be deleted

@KirillovDenis commented on GitHub (Jul 21, 2022): In this case there are two object versions and one delete marker in the bucket. It seems bucket shouldn't be deleted then because of AWS [spec](https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteBucket.html): `All objects (including all object versions and delete markers) in the bucket must be deleted before the bucket itself can be deleted`
sami commented 2025-12-28 17:36:52 +00:00
Author
Owner
Copy link

@vdomnich-yadro commented on GitHub (Jul 21, 2022):

@KirillovDenis makes sense, let me see if specifying all versions and delete markers will do the trick.

@vdomnich-yadro commented on GitHub (Jul 21, 2022): @KirillovDenis makes sense, let me see if specifying all versions and delete markers will do the trick.
sami commented 2025-12-28 17:36:53 +00:00
Author
Owner
Copy link

@alexvanin commented on GitHub (Jul 21, 2022):

Anyway, we've seen odd behavior when object in dir like some-dir/versioned-object allows to delete bucket without removal of all its versions. Guess this should be fixed.

@alexvanin commented on GitHub (Jul 21, 2022): Anyway, we've seen odd behavior when object in dir like `some-dir/versioned-object` allows to delete bucket without removal of all its versions. Guess this should be fixed.
sami commented 2025-12-28 17:36:53 +00:00
Author
Owner
Copy link

@KirillovDenis commented on GitHub (Jul 21, 2022):

Anyway, we've seen odd behavior when object in dir like some-dir/versioned-object allows to delete bucket without removal of all its versions. Guess this should be fixed.

Done in #603

@KirillovDenis commented on GitHub (Jul 21, 2022): > Anyway, we've seen odd behavior when object in dir like `some-dir/versioned-object` allows to delete bucket without removal of all its versions. Guess this should be fixed. Done in #603
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
master
1255-session-token-v2
new-get
logger
experimental/unsigned-responses
experimental/get-perf
debugprint
ezayats/bump-s3-tests
955-after-a-parallel-deletion-of-versioned-objects-list-objects-returns-info-about-them
v0.42.0
v0.41.5
v0.41.4
v0.41.3
v0.41.2
v0.41.1
v0.41.0
v0.40.0
v0.39.0
v0.38.0
v0.37.0
v0.36.1
v0.36.0
v0.35.0
v0.34.1
v0.34.0
v0.33.0
v0.32.0
v0.31.1
v0.31.0
v0.30.1
v0.30.0
v0.29.0
v0.28.2
v0.28.1
v0.28.0
v0.27.1
v0.27.0
v0.26.1
v0.26.0
v0.25.1
v0.25.0
v0.24.0
v0.23.0
v0.22.0
v0.21.1
v0.21.0
v0.20.0
v0.19.0
v0.18.0
v0.17.0
v0.17.0-rc.1
v0.16.0
v0.15.1-pre
v0.15.0
v0.14.2
v0.14.1
v0.14.0
v0.13.0
v0.12.0-rc.3
v0.12.0-rc.2
v0.12.0-rc.1
Labels
Clear labels   
I2

   
I2

   
I3

   
I4

   
S2

   
S3

   
S4

   
S4

   
U0

   
U1

   
U2

   
U2

   
U3

   
U4

   
U4

   
auth-mate

   
blocked

   
bug

   
config

   
dependencies

   
discussion

   
documentation

   
enhancement

   
epic

   
feature

   
go

   
good first issue

   
help wanted

   
performance

   
question

   
security

   
test

   
tree-service

   
tree-service
No labels
I2
I2
I3
I4
S2
S3
S4
S4
U0
U1
U2
U2
U3
U4
U4
auth-mate
blocked
bug
config
dependencies
discussion
documentation
enhancement
epic
feature
go
good first issue
help wanted
performance
question
security
test
tree-service
tree-service
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
nspcc-dev/neofs-s3-gw#286
No description provided.