nspcc-dev/neofs-s3-gw
1
0
Fork 0
mirror of https://github.com/nspcc-dev/neofs-s3-gw.git synced 2026-03-01 04:29:15 +00:00
Code Issues 24 Projects Packages Wiki Activity

Uploaded parts of multipart-upload keep stored even if multipart upload was aborted. #342

New issue
Closed
opened 2025-12-28 17:37:04 +00:00 by sami · 6 comments
sami commented 2025-12-28 17:37:04 +00:00
Owner
Copy link

Originally created by @lesnya on GitHub (Oct 18, 2022).

Originally assigned to: @masterSplinter01 on GitHub.

Some uploaded parts of s3 multipart-upload keep stored in neofs container even if multipart upload was aborted.

If you upload some relatively small objects (5MB or 100MB) as part of multipart upload and than do multipart-upload abort, they desappear from s3 and objects truly be deleted from neofs container underneath.
But if you upload some big objects (tested and validated for 1GB and 5GB) as part of multipart upload and than do multipart-upload abort - those objects will not be deleted from thier respective neofs container and continue to take up disk space.

Steps to reproduce:

  1. Create bucket
    aws --no-verify-ssl s3api create-bucket --bucket multipart-test --endpoint-url http://node1.neofs:8084

  2. Create multipart upload
    aws --no-verify-ssl s3api create-multipart-upload --bucket multipart-test --key multipart-upload-1 --endpoint http://node1.neofs:8084

  3. Create small (5MB) and big (1GB) files and upload them as parts of multipart upload.

dd if=/dev/urandom of=/home/admin/object.sample.small bs=1M count=5
dd if=/dev/urandom of=/home/admin/object.sample.big bs=1G count=1
aws --no-verify-ssl s3api upload-part --bucket multipart-test --key multipart-upload-1 --upload-id 8c331022-d081-4bca-9f5e-0927483ded02 --part-number 1 --body /home/admin/object.sample.big --endpoint http://node1.neofs:8084
aws --no-verify-ssl s3api upload-part --bucket multipart-test --key multipart-upload-1 --upload-id 8c331022-d081-4bca-9f5e-0927483ded02 --part-number 2 --body /home/admin/object.sample.big --endpoint http://node1.neofs:8084
aws --no-verify-ssl s3api upload-part --bucket multipart-test --key multipart-upload-1 --upload-id 8c331022-d081-4bca-9f5e-0927483ded02 --part-number 3 --body /home/admin/object.sample.small --endpoint http://node1.neofs:8084
  1. Find which container ID represents this s3 bucket in neofs and list objects inside:
    neofs-cli --rpc-endpoint node1.neofs:8080 -w scenarios/files/wallet.json container list-objects --cid 21dKQ9k6tF7azfJ8g2XM3WQXEBHK2GTiyYa1ivBTZMtV

EXPECTED RESULT: Number of objects should match an uploaded parts, i.e. 3 objects in our case:
2dfBnvPEfyyZfb1wXySpPgVDTEK51Cr28hUnCc3XZ6DX
Bq2yrAvDdL3HEbGJb6mFTam2RzgjC4ux2hQ3xr3xCP2N
4pv88qwi2NdHkPgbRkYSFSaGHnzSapXp3BXWTDSzuwQN

  1. Abort multipart upload
    aws --no-verify-ssl s3api abort-multipart-upload --bucket multipart-test --key multipart-upload-1 --upload-id 685318f5-585f-4c3c-9c0a-dff930bba9a7 --endpoint http://node1.neofs:8084

EXPECTED RESULT: s3api list-parts and list-multipart-upload outputs should be empty now.

  1. List neofs container:
    neofs-cli --rpc-endpoint node1.neofs:8080 -w scenarios/files/wallet.json container list-objects --cid 21dKQ9k6tF7azfJ8g2XM3WQXEBHK2GTiyYa1ivBTZMtV

Enter password >
2dfBnvPEfyyZfb1wXySpPgVDTEK51Cr28hUnCc3XZ6DX
Bq2yrAvDdL3HEbGJb6mFTam2RzgjC4ux2hQ3xr3xCP2N

One object has gone (5Mb upload), while other 2 objects are still there.

UPD: ticking epochs doesn't help.

Originally created by @lesnya on GitHub (Oct 18, 2022). Originally assigned to: @masterSplinter01 on GitHub. Some uploaded parts of s3 multipart-upload keep stored in neofs container even if multipart upload was aborted. If you upload some relatively small objects (5MB or 100MB) as part of multipart upload and than do multipart-upload abort, they desappear from s3 and objects truly be deleted from neofs container underneath. But if you upload some big objects (tested and validated for 1GB and 5GB) as part of multipart upload and than do multipart-upload abort - those objects will not be deleted from thier respective neofs container and continue to take up disk space. Steps to reproduce: 1. Create bucket `aws --no-verify-ssl s3api create-bucket --bucket multipart-test --endpoint-url http://node1.neofs:8084` 2. Create multipart upload `aws --no-verify-ssl s3api create-multipart-upload --bucket multipart-test --key multipart-upload-1 --endpoint http://node1.neofs:8084` 3. Create small (5MB) and big (1GB) files and upload them as parts of multipart upload. ``` dd if=/dev/urandom of=/home/admin/object.sample.small bs=1M count=5 dd if=/dev/urandom of=/home/admin/object.sample.big bs=1G count=1 aws --no-verify-ssl s3api upload-part --bucket multipart-test --key multipart-upload-1 --upload-id 8c331022-d081-4bca-9f5e-0927483ded02 --part-number 1 --body /home/admin/object.sample.big --endpoint http://node1.neofs:8084 aws --no-verify-ssl s3api upload-part --bucket multipart-test --key multipart-upload-1 --upload-id 8c331022-d081-4bca-9f5e-0927483ded02 --part-number 2 --body /home/admin/object.sample.big --endpoint http://node1.neofs:8084 aws --no-verify-ssl s3api upload-part --bucket multipart-test --key multipart-upload-1 --upload-id 8c331022-d081-4bca-9f5e-0927483ded02 --part-number 3 --body /home/admin/object.sample.small --endpoint http://node1.neofs:8084 ``` 4. Find which container ID represents this s3 bucket in neofs and list objects inside: `neofs-cli --rpc-endpoint node1.neofs:8080 -w scenarios/files/wallet.json container list-objects --cid 21dKQ9k6tF7azfJ8g2XM3WQXEBHK2GTiyYa1ivBTZMtV` EXPECTED RESULT: Number of objects should match an uploaded parts, i.e. 3 objects in our case: 2dfBnvPEfyyZfb1wXySpPgVDTEK51Cr28hUnCc3XZ6DX Bq2yrAvDdL3HEbGJb6mFTam2RzgjC4ux2hQ3xr3xCP2N 4pv88qwi2NdHkPgbRkYSFSaGHnzSapXp3BXWTDSzuwQN 5. Abort multipart upload `aws --no-verify-ssl s3api abort-multipart-upload --bucket multipart-test --key multipart-upload-1 --upload-id 685318f5-585f-4c3c-9c0a-dff930bba9a7 --endpoint http://node1.neofs:8084` EXPECTED RESULT: s3api list-parts and list-multipart-upload outputs should be empty now. 6. List neofs container: `neofs-cli --rpc-endpoint node1.neofs:8080 -w scenarios/files/wallet.json container list-objects --cid 21dKQ9k6tF7azfJ8g2XM3WQXEBHK2GTiyYa1ivBTZMtV` Enter password > 2dfBnvPEfyyZfb1wXySpPgVDTEK51Cr28hUnCc3XZ6DX Bq2yrAvDdL3HEbGJb6mFTam2RzgjC4ux2hQ3xr3xCP2N One object has gone (5Mb upload), while other 2 objects are still there. UPD: ticking epochs doesn't help.
sami 2025-12-28 17:37:04 +00:00
sami commented 2025-12-28 17:37:04 +00:00
Author
Owner
Copy link

@masterSplinter01 commented on GitHub (Oct 19, 2022):

Couldn't reproduce the bug.
@lesnya Could you please check it one more time? If the bug appear, it will be nice to attach s3 gate and storage nodes logs.

@masterSplinter01 commented on GitHub (Oct 19, 2022): Couldn't reproduce the bug. @lesnya Could you please check it one more time? If the bug appear, it will be nice to attach s3 gate and storage nodes logs.
sami commented 2025-12-28 17:37:04 +00:00
Author
Owner
Copy link

@anatoly-bogatyrev commented on GitHub (Oct 20, 2022):

Issue did not reproduced on the latest versions (master) of all components. Can be closed.

@anatoly-bogatyrev commented on GitHub (Oct 20, 2022): Issue did not reproduced on the latest versions (master) of all components. Can be closed.
sami commented 2025-12-28 17:37:04 +00:00
Author
Owner
Copy link

@anatoly-bogatyrev commented on GitHub (Oct 25, 2022):

Part size: 5.1G /home/admin/object.sample or 1G
versions (latest master):

  • node - v0.33.0-62-g713fdab1
  • s3-gw - v0.24.0-23-g821df3d6
  • neogo - 0.99.4

Create bucket:

 > aws --no-verify-ssl s3api create-bucket --bucket d7b6a5b4-e389-49ab-9e4a-7aaac3c971eb --endpoint http://node1.neofs:8084 --create-bucket-configuration 'LocationConstraint=load-1-4'

Bucket d7b6a5b4-e389-49ab-9e4a-7aaac3c971eb has been created.

Get CID:
4az85vrmCMkvVL1t5chk5Epkg5ZT5VM9gS73ybw2MVom

Displaying a list of objects:

 > neofs-cli --config cli.yaml object search --cid 4az85vrmCMkvVL1t5chk5Epkg5ZT5VM9gS73ybw2MVom --root
 > Found 0 objects.

Multipart-upload start:

 > aws --no-verify-ssl s3api create-multipart-upload --bucket d7b6a5b4-e389-49ab-9e4a-7aaac3c971eb --key multipart-upload-0 --endpoint http://node1.neofs:8084
 > {
    "Bucket": "d7b6a5b4-e389-49ab-9e4a-7aaac3c971eb",
    "Key": "multipart-upload-0",
    "UploadId": "384f78b4-d34d-4b3c-a45a-bb8f63f9f248"
}

Uploading two parts for each of the created multipart-upload sessions:

 > aws --no-verify-ssl s3api upload-part --bucket d7b6a5b4-e389-49ab-9e4a-7aaac3c971eb --key multipart-upload-0 --upload-id 384f78b4-d34d-4b3c-a45a-bb8f63f9f248 --part-number 1 --body /home/admin/object.sample --endpoint http://node1.neofs:8084
{
    "ETag": "d953219d16781c0c5f3dfc3a07f571e5bfcf9fd3833c5c86154052b93ac370ad"
}

 > aws --no-verify-ssl s3api upload-part --bucket d7b6a5b4-e389-49ab-9e4a-7aaac3c971eb --key multipart-upload-0 --upload-id 384f78b4-d34d-4b3c-a45a-bb8f63f9f248 --part-number 2 --body /home/admin/object.sample --endpoint http://node1.neofs:8084
{
    "ETag": "d953219d16781c0c5f3dfc3a07f571e5bfcf9fd3833c5c86154052b93ac370ad"
}

Objects list:

neofs-cli --config cli.yaml object search --cid 4az85vrmCMkvVL1t5chk5Epkg5ZT5VM9gS73ybw2MVom --root
Found 2 objects.
GRzBQ7xpe12BZ9brsfkuDNYs9eXNXe8bTDkWN5F74ZXK
CCHkJqk41PWniewnpUWe85G5Va4GYK8p4FfiviYUGTxR

Stop multipart upload session:

 > aws --no-verify-ssl s3api abort-multipart-upload --bucket d7b6a5b4-e389-49ab-9e4a-7aaac3c971eb  --key multipart-upload-0 --upload-id 384f78b4-d34d-4b3c-a45a-bb8f63f9f248 --endpoint http://node1.neofs:8084
 >

Objects list:

 > neofs-cli --config cli.yaml object search --cid 4az85vrmCMkvVL1t5chk5Epkg5ZT5VM9gS73ybw2MVom --root
 > Found 1 objects.
GRzBQ7xpe12BZ9brsfkuDNYs9eXNXe8bTDkWN5F74ZXK

After several minutes.

Objects list:

 > neofs-cli --config cli.yaml object search --cid 4az85vrmCMkvVL1t5chk5Epkg5ZT5VM9gS73ybw2MVom --root
 > Found 1 objects.
GRzBQ7xpe12BZ9brsfkuDNYs9eXNXe8bTDkWN5F74ZXK

Object GRzBQ7xpe12BZ9brsfkuDNYs9eXNXe8bTDkWN5F74ZXK has not been deleted.

@anatoly-bogatyrev commented on GitHub (Oct 25, 2022): Part size: 5.1G /home/admin/object.sample or 1G versions (latest master): - node - v0.33.0-62-g713fdab1 - s3-gw - v0.24.0-23-g821df3d6 - neogo - 0.99.4 Create bucket: ``` > aws --no-verify-ssl s3api create-bucket --bucket d7b6a5b4-e389-49ab-9e4a-7aaac3c971eb --endpoint http://node1.neofs:8084 --create-bucket-configuration 'LocationConstraint=load-1-4' ``` Bucket d7b6a5b4-e389-49ab-9e4a-7aaac3c971eb has been created. Get CID: 4az85vrmCMkvVL1t5chk5Epkg5ZT5VM9gS73ybw2MVom Displaying a list of objects: ``` > neofs-cli --config cli.yaml object search --cid 4az85vrmCMkvVL1t5chk5Epkg5ZT5VM9gS73ybw2MVom --root > Found 0 objects. ``` Multipart-upload start: ``` > aws --no-verify-ssl s3api create-multipart-upload --bucket d7b6a5b4-e389-49ab-9e4a-7aaac3c971eb --key multipart-upload-0 --endpoint http://node1.neofs:8084 > { "Bucket": "d7b6a5b4-e389-49ab-9e4a-7aaac3c971eb", "Key": "multipart-upload-0", "UploadId": "384f78b4-d34d-4b3c-a45a-bb8f63f9f248" } ``` Uploading two parts for each of the created multipart-upload sessions: ``` > aws --no-verify-ssl s3api upload-part --bucket d7b6a5b4-e389-49ab-9e4a-7aaac3c971eb --key multipart-upload-0 --upload-id 384f78b4-d34d-4b3c-a45a-bb8f63f9f248 --part-number 1 --body /home/admin/object.sample --endpoint http://node1.neofs:8084 { "ETag": "d953219d16781c0c5f3dfc3a07f571e5bfcf9fd3833c5c86154052b93ac370ad" } > aws --no-verify-ssl s3api upload-part --bucket d7b6a5b4-e389-49ab-9e4a-7aaac3c971eb --key multipart-upload-0 --upload-id 384f78b4-d34d-4b3c-a45a-bb8f63f9f248 --part-number 2 --body /home/admin/object.sample --endpoint http://node1.neofs:8084 { "ETag": "d953219d16781c0c5f3dfc3a07f571e5bfcf9fd3833c5c86154052b93ac370ad" } ``` Objects list: > neofs-cli --config cli.yaml object search --cid 4az85vrmCMkvVL1t5chk5Epkg5ZT5VM9gS73ybw2MVom --root > Found 2 objects. GRzBQ7xpe12BZ9brsfkuDNYs9eXNXe8bTDkWN5F74ZXK CCHkJqk41PWniewnpUWe85G5Va4GYK8p4FfiviYUGTxR Stop multipart upload session: ``` > aws --no-verify-ssl s3api abort-multipart-upload --bucket d7b6a5b4-e389-49ab-9e4a-7aaac3c971eb --key multipart-upload-0 --upload-id 384f78b4-d34d-4b3c-a45a-bb8f63f9f248 --endpoint http://node1.neofs:8084 > ``` Objects list: ``` > neofs-cli --config cli.yaml object search --cid 4az85vrmCMkvVL1t5chk5Epkg5ZT5VM9gS73ybw2MVom --root > Found 1 objects. GRzBQ7xpe12BZ9brsfkuDNYs9eXNXe8bTDkWN5F74ZXK ``` After several minutes. Objects list: ``` > neofs-cli --config cli.yaml object search --cid 4az85vrmCMkvVL1t5chk5Epkg5ZT5VM9gS73ybw2MVom --root > Found 1 objects. GRzBQ7xpe12BZ9brsfkuDNYs9eXNXe8bTDkWN5F74ZXK ``` Object GRzBQ7xpe12BZ9brsfkuDNYs9eXNXe8bTDkWN5F74ZXK has not been deleted.
sami commented 2025-12-28 17:37:04 +00:00
Author
Owner
Copy link

@anatoly-bogatyrev commented on GitHub (Oct 25, 2022):

The issue is not reproduced with small objects (part size 100Mb -).

@anatoly-bogatyrev commented on GitHub (Oct 25, 2022): The issue is not reproduced with small objects (part size 100Mb -).
sami commented 2025-12-28 17:37:05 +00:00
Author
Owner
Copy link

@alexvanin commented on GitHub (Nov 16, 2022):

Should be fixed now @lesnya

@alexvanin commented on GitHub (Nov 16, 2022): Should be fixed now @lesnya
sami commented 2025-12-28 17:37:05 +00:00
Author
Owner
Copy link

@alexvanin commented on GitHub (Nov 18, 2022):

Done in https://github.com/nspcc-dev/neofs-sdk-go/issues/360

@alexvanin commented on GitHub (Nov 18, 2022): Done in https://github.com/nspcc-dev/neofs-sdk-go/issues/360
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
master
1255-session-token-v2
new-get
logger
experimental/unsigned-responses
experimental/get-perf
debugprint
ezayats/bump-s3-tests
955-after-a-parallel-deletion-of-versioned-objects-list-objects-returns-info-about-them
v0.42.0
v0.41.5
v0.41.4
v0.41.3
v0.41.2
v0.41.1
v0.41.0
v0.40.0
v0.39.0
v0.38.0
v0.37.0
v0.36.1
v0.36.0
v0.35.0
v0.34.1
v0.34.0
v0.33.0
v0.32.0
v0.31.1
v0.31.0
v0.30.1
v0.30.0
v0.29.0
v0.28.2
v0.28.1
v0.28.0
v0.27.1
v0.27.0
v0.26.1
v0.26.0
v0.25.1
v0.25.0
v0.24.0
v0.23.0
v0.22.0
v0.21.1
v0.21.0
v0.20.0
v0.19.0
v0.18.0
v0.17.0
v0.17.0-rc.1
v0.16.0
v0.15.1-pre
v0.15.0
v0.14.2
v0.14.1
v0.14.0
v0.13.0
v0.12.0-rc.3
v0.12.0-rc.2
v0.12.0-rc.1
Labels
Clear labels   
I2

   
I2

   
I3

   
I4

   
S2

   
S3

   
S4

   
S4

   
U0

   
U1

   
U2

   
U2

   
U3

   
U4

   
U4

   
auth-mate

   
blocked

   
bug

   
config

   
dependencies

   
discussion

   
documentation

   
enhancement

   
epic

   
feature

   
go

   
good first issue

   
help wanted

   
performance

   
question

   
security

   
test

   
tree-service

   
tree-service
No labels
I2
I2
I3
I4
S2
S3
S4
S4
U0
U1
U2
U2
U3
U4
U4
auth-mate
blocked
bug
config
dependencies
discussion
documentation
enhancement
epic
feature
go
good first issue
help wanted
performance
question
security
test
tree-service
tree-service
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
nspcc-dev/neofs-s3-gw#342
No description provided.