nspcc-dev/neofs-node
1
0
Fork 0
mirror of https://github.com/nspcc-dev/neofs-node.git synced 2026-03-01 04:29:10 +00:00
Code Issues 189 Projects Packages Wiki Activity

It looks like removed object can be searched after removal #1410

New issue
Closed
opened 2025-12-28 17:22:54 +00:00 by sami · 4 comments
sami commented 2025-12-28 17:22:54 +00:00
Owner
Copy link

Originally created by @smallhive on GitHub (Apr 25, 2025).

Originally assigned to: @cthulhu-rider on GitHub.

There is s3 gate test, which sometimes fails.
The fail rate is about 1:350, which is very difficult to detect and debug.

Eventually got the next S3 Gate log

info	api/router.go:177	call method	{"host": "localhost:34959", "request_id": "6c0d1246-eb0c-4323-8ddf-816a2d7c7076", "method": "CreateBucket", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "object": ""}
debug	layer/layer.go:411	bucket not found	{"error": "nns get: invocation failed: at instruction 4370 (THROW): unhandled exception: \"token not found\""}
info	handler/put.go:814	bucket is created	{"reqId": "6c0d1246-eb0c-4323-8ddf-816a2d7c7076", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "container_id": "4Z4RimPQS5DFpgnhdhH1p8T6syV9Sse7fjJK1XvAN192"}
info	api/router.go:193	call method result	{"request_id": "6c0d1246-eb0c-4323-8ddf-816a2d7c7076", "status": 200, "description": "OK"}
info	api/router.go:177	call method	{"host": "localhost:34959", "request_id": "0323ec33-dd1f-4582-bafb-39e1c04c4359", "method": "PutBucketVersioning", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "object": ""}
info	api/router.go:193	call method result	{"request_id": "0323ec33-dd1f-4582-bafb-39e1c04c4359", "status": 200, "description": "OK"}
info	api/router.go:177	call method	{"host": "localhost:34959", "request_id": "7f696f48-8bc4-45de-b9d7-e6f232813b36", "method": "GetBucketVersioning", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "object": ""}
info	api/router.go:193	call method result	{"request_id": "7f696f48-8bc4-45de-b9d7-e6f232813b36", "status": 200, "description": "OK"}
info	api/router.go:177	call method	{"host": "localhost:34959", "request_id": "90ccc8f2-da2b-4c91-8186-bb81cc3543f9", "method": "PutObject", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "object": "key_0"}
debug	layer/object.go:288	put object	{"reqId": "90ccc8f2-da2b-4c91-8186-bb81cc3543f9", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "cid": "4Z4RimPQS5DFpgnhdhH1p8T6syV9Sse7fjJK1XvAN192", "object": "key_0", "oid": "DRDei9LyqRW8yx3vwhZsFSJgcbp4MneLMH8kZDJn3jC1", "size": 5}
info	api/router.go:193	call method result	{"request_id": "90ccc8f2-da2b-4c91-8186-bb81cc3543f9", "status": 200, "description": "OK"}
info	api/router.go:177	call method	{"host": "localhost:34959", "request_id": "7cff0881-b4ff-45fc-9357-80ed0508446b", "method": "PutObject", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "object": "key_1"}
debug	layer/object.go:288	put object	{"reqId": "7cff0881-b4ff-45fc-9357-80ed0508446b", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "cid": "4Z4RimPQS5DFpgnhdhH1p8T6syV9Sse7fjJK1XvAN192", "object": "key_1", "oid": "6ek8Nam37vUeUEXXFgbQTG2GQiBfmB3jtGBJjAT78PV5", "size": 5}
info	api/router.go:193	call method result	{"request_id": "7cff0881-b4ff-45fc-9357-80ed0508446b", "status": 200, "description": "OK"}
info	api/router.go:177	call method	{"host": "localhost:34959", "request_id": "c3566d7c-0d0e-45a9-a7e3-a6cdb397449b", "method": "PutObject", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "object": "key_2"}
debug	layer/object.go:288	put object	{"reqId": "c3566d7c-0d0e-45a9-a7e3-a6cdb397449b", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "cid": "4Z4RimPQS5DFpgnhdhH1p8T6syV9Sse7fjJK1XvAN192", "object": "key_2", "oid": "5ssiVKuAqZTUrFvSJ9A2fnaeXCTvYrMJoYnxKgVZCQwX", "size": 5}
info	api/router.go:193	call method result	{"request_id": "c3566d7c-0d0e-45a9-a7e3-a6cdb397449b", "status": 200, "description": "OK"}
info	api/router.go:177	call method	{"host": "localhost:34959", "request_id": "94ab8c0b-0d38-412b-b4a4-8fc24f47681b", "method": "PutObject", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "object": "key_3"}
debug	layer/object.go:288	put object	{"reqId": "94ab8c0b-0d38-412b-b4a4-8fc24f47681b", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "cid": "4Z4RimPQS5DFpgnhdhH1p8T6syV9Sse7fjJK1XvAN192", "object": "key_3", "oid": "b2LpEDyeSsqCDfnAvTEmakfNTiX7WYfK6WbRfTCg3cj", "size": 5}
info	api/router.go:193	call method result	{"request_id": "94ab8c0b-0d38-412b-b4a4-8fc24f47681b", "status": 200, "description": "OK"}
info	api/router.go:177	call method	{"host": "localhost:34959", "request_id": "f05902ee-b064-4c1d-bf03-c83a13dabb1b", "method": "PutObject", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "object": "key_4"}
debug	layer/object.go:288	put object	{"reqId": "f05902ee-b064-4c1d-bf03-c83a13dabb1b", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "cid": "4Z4RimPQS5DFpgnhdhH1p8T6syV9Sse7fjJK1XvAN192", "object": "key_4", "oid": "BizHV9ZBjQcTCuiBCDjpzmssyph35Xnik3whbthVSPus", "size": 5}
info	api/router.go:193	call method result	{"request_id": "f05902ee-b064-4c1d-bf03-c83a13dabb1b", "status": 200, "description": "OK"}
info	api/router.go:177	call method	{"host": "localhost:34959", "request_id": "d9da6a86-62ed-446c-a853-33eb30a0e300", "method": "ListBucketVersions", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "object": ""}
info	api/router.go:193	call method result	{"request_id": "d9da6a86-62ed-446c-a853-33eb30a0e300", "status": 200, "description": "OK"}
info	api/router.go:177	call method	{"host": "localhost:34959", "request_id": "91a48cfb-3a0d-46fb-bdc1-799cb7f9dafa", "method": "DeleteMultipleObjects", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "object": ""}
info	api/router.go:177	call method	{"host": "localhost:34959", "request_id": "2fade456-d1e9-491a-b9de-777690037368", "method": "DeleteMultipleObjects", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "object": ""}
info	api/router.go:177	call method	{"host": "localhost:34959", "request_id": "5bfa6926-2a92-44d6-bbeb-9e4cc29498ef", "method": "DeleteMultipleObjects", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "object": ""}
info	api/router.go:177	call method	{"host": "localhost:34959", "request_id": "20936c66-f5c1-4683-9217-d08c0e6f53c9", "method": "DeleteMultipleObjects", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "object": ""}
debug	layer/object.go:844	delete object	{"reqId": "2fade456-d1e9-491a-b9de-777690037368", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "cid": "4Z4RimPQS5DFpgnhdhH1p8T6syV9Sse7fjJK1XvAN192", "oid": "DRDei9LyqRW8yx3vwhZsFSJgcbp4MneLMH8kZDJn3jC1", "error": "mark object removal via connection pool: status: code = 2052 message = object already removed"}
debug	layer/layer.go:1019	delete object success	{"object": "key_0:DRDei9LyqRW8yx3vwhZsFSJgcbp4MneLMH8kZDJn3jC1", "reqId": "2fade456-d1e9-491a-b9de-777690037368"}
info	api/router.go:177	call method	{"host": "localhost:34959", "request_id": "6f93b598-8e0c-466f-8beb-c81aa7b5a43f", "method": "DeleteMultipleObjects", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "object": ""}
debug	layer/object.go:844	delete object	{"reqId": "20936c66-f5c1-4683-9217-d08c0e6f53c9", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "cid": "4Z4RimPQS5DFpgnhdhH1p8T6syV9Sse7fjJK1XvAN192", "oid": "DRDei9LyqRW8yx3vwhZsFSJgcbp4MneLMH8kZDJn3jC1", "error": "mark object removal via connection pool: status: code = 2052 message = object already removed"}
debug	layer/layer.go:1019	delete object success	{"object": "key_0:DRDei9LyqRW8yx3vwhZsFSJgcbp4MneLMH8kZDJn3jC1", "reqId": "20936c66-f5c1-4683-9217-d08c0e6f53c9"}
debug	layer/object.go:844	delete object	{"reqId": "6f93b598-8e0c-466f-8beb-c81aa7b5a43f", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "cid": "4Z4RimPQS5DFpgnhdhH1p8T6syV9Sse7fjJK1XvAN192", "oid": "DRDei9LyqRW8yx3vwhZsFSJgcbp4MneLMH8kZDJn3jC1", "error": "mark object removal via connection pool: status: code = 2052 message = object already removed"}
debug	layer/layer.go:1019	delete object success	{"object": "key_0:DRDei9LyqRW8yx3vwhZsFSJgcbp4MneLMH8kZDJn3jC1", "reqId": "6f93b598-8e0c-466f-8beb-c81aa7b5a43f"}
debug	layer/object.go:844	delete object	{"reqId": "5bfa6926-2a92-44d6-bbeb-9e4cc29498ef", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "cid": "4Z4RimPQS5DFpgnhdhH1p8T6syV9Sse7fjJK1XvAN192", "oid": "DRDei9LyqRW8yx3vwhZsFSJgcbp4MneLMH8kZDJn3jC1", "error": "mark object removal via connection pool: status: code = 2052 message = object already removed"}
debug	layer/layer.go:1019	delete object success	{"object": "key_0:DRDei9LyqRW8yx3vwhZsFSJgcbp4MneLMH8kZDJn3jC1", "reqId": "5bfa6926-2a92-44d6-bbeb-9e4cc29498ef"}
debug	layer/object.go:844	delete object	{"reqId": "6f93b598-8e0c-466f-8beb-c81aa7b5a43f", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "cid": "4Z4RimPQS5DFpgnhdhH1p8T6syV9Sse7fjJK1XvAN192", "oid": "6ek8Nam37vUeUEXXFgbQTG2GQiBfmB3jtGBJjAT78PV5", "error": "mark object removal via connection pool: status: code = 2052 message = object already removed"}
debug	layer/layer.go:1019	delete object success	{"object": "key_1:6ek8Nam37vUeUEXXFgbQTG2GQiBfmB3jtGBJjAT78PV5", "reqId": "6f93b598-8e0c-466f-8beb-c81aa7b5a43f"}
debug	layer/object.go:844	delete object	{"reqId": "5bfa6926-2a92-44d6-bbeb-9e4cc29498ef", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "cid": "4Z4RimPQS5DFpgnhdhH1p8T6syV9Sse7fjJK1XvAN192", "oid": "6ek8Nam37vUeUEXXFgbQTG2GQiBfmB3jtGBJjAT78PV5", "error": "mark object removal via connection pool: status: code = 2052 message = object already removed"}
debug	layer/layer.go:1019	delete object success	{"object": "key_1:6ek8Nam37vUeUEXXFgbQTG2GQiBfmB3jtGBJjAT78PV5", "reqId": "5bfa6926-2a92-44d6-bbeb-9e4cc29498ef"}
debug	layer/object.go:844	delete object	{"reqId": "91a48cfb-3a0d-46fb-bdc1-799cb7f9dafa", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "cid": "4Z4RimPQS5DFpgnhdhH1p8T6syV9Sse7fjJK1XvAN192", "oid": "DRDei9LyqRW8yx3vwhZsFSJgcbp4MneLMH8kZDJn3jC1"}
debug	layer/layer.go:1019	delete object success	{"object": "key_0:DRDei9LyqRW8yx3vwhZsFSJgcbp4MneLMH8kZDJn3jC1", "reqId": "91a48cfb-3a0d-46fb-bdc1-799cb7f9dafa"}
debug	layer/object.go:844	delete object	{"reqId": "91a48cfb-3a0d-46fb-bdc1-799cb7f9dafa", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "cid": "4Z4RimPQS5DFpgnhdhH1p8T6syV9Sse7fjJK1XvAN192", "oid": "6ek8Nam37vUeUEXXFgbQTG2GQiBfmB3jtGBJjAT78PV5", "error": "mark object removal via connection pool: status: code = 2052 message = object already removed"}
debug	layer/layer.go:1019	delete object success	{"object": "key_1:6ek8Nam37vUeUEXXFgbQTG2GQiBfmB3jtGBJjAT78PV5", "reqId": "91a48cfb-3a0d-46fb-bdc1-799cb7f9dafa"}
debug	layer/object.go:844	delete object	{"reqId": "2fade456-d1e9-491a-b9de-777690037368", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "cid": "4Z4RimPQS5DFpgnhdhH1p8T6syV9Sse7fjJK1XvAN192", "oid": "6ek8Nam37vUeUEXXFgbQTG2GQiBfmB3jtGBJjAT78PV5", "error": "mark object removal via connection pool: status: code = 1024 message = incomplete object PUT by placement: number of replicas cannot be met for list #0: 1 required, 0 nodes remaining (last node error: could not close object stream: replicate object to remote node (key=03af9a767790daf62366499ca902bdac886ae13015aed3ba513930158e2906eb81): /dns4/localhost/tcp/33917: status: code = 1024 message = failed to verify and store object locally: validate payload content: tombstone verification: verifying 6ek8Nam37vUeUEXXFgbQTG2GQiBfmB3jtGBJjAT78PV5 member: heading object: logical error: status: code = 2052 message = object already removed)"}
debug	layer/layer.go:1019	delete object success	{"object": "key_1:6ek8Nam37vUeUEXXFgbQTG2GQiBfmB3jtGBJjAT78PV5", "reqId": "2fade456-d1e9-491a-b9de-777690037368"}
debug	layer/object.go:844	delete object	{"reqId": "91a48cfb-3a0d-46fb-bdc1-799cb7f9dafa", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "cid": "4Z4RimPQS5DFpgnhdhH1p8T6syV9Sse7fjJK1XvAN192", "oid": "5ssiVKuAqZTUrFvSJ9A2fnaeXCTvYrMJoYnxKgVZCQwX", "error": "mark object removal via connection pool: status: code = 2052 message = object already removed"}
debug	layer/layer.go:1019	delete object success	{"object": "key_2:5ssiVKuAqZTUrFvSJ9A2fnaeXCTvYrMJoYnxKgVZCQwX", "reqId": "91a48cfb-3a0d-46fb-bdc1-799cb7f9dafa"}
debug	layer/object.go:844	delete object	{"reqId": "2fade456-d1e9-491a-b9de-777690037368", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "cid": "4Z4RimPQS5DFpgnhdhH1p8T6syV9Sse7fjJK1XvAN192", "oid": "5ssiVKuAqZTUrFvSJ9A2fnaeXCTvYrMJoYnxKgVZCQwX", "error": "mark object removal via connection pool: status: code = 1024 message = incomplete object PUT by placement: number of replicas cannot be met for list #0: 3 required, 1 nodes remaining (last node error: could not close object stream: replicate object to remote node (key=03d2b4580016444af63f9e4dc8e7eb233666141386b501abc647b82bf54f7851ed): /dns4/localhost/tcp/35957: status: code = 1024 message = failed to verify and store object locally: validate payload content: tombstone verification: verifying 5ssiVKuAqZTUrFvSJ9A2fnaeXCTvYrMJoYnxKgVZCQwX member: heading object: logical error: status: code = 2052 message = object already removed)"}
debug	layer/layer.go:1019	delete object success	{"object": "key_2:5ssiVKuAqZTUrFvSJ9A2fnaeXCTvYrMJoYnxKgVZCQwX", "reqId": "2fade456-d1e9-491a-b9de-777690037368"}
debug	layer/object.go:844	delete object	{"reqId": "2fade456-d1e9-491a-b9de-777690037368", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "cid": "4Z4RimPQS5DFpgnhdhH1p8T6syV9Sse7fjJK1XvAN192", "oid": "b2LpEDyeSsqCDfnAvTEmakfNTiX7WYfK6WbRfTCg3cj", "error": "mark object removal via connection pool: status: code = 2052 message = object already removed"}
debug	layer/layer.go:1019	delete object success	{"object": "key_3:b2LpEDyeSsqCDfnAvTEmakfNTiX7WYfK6WbRfTCg3cj", "reqId": "2fade456-d1e9-491a-b9de-777690037368"}
debug	layer/object.go:844	delete object	{"reqId": "20936c66-f5c1-4683-9217-d08c0e6f53c9", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "cid": "4Z4RimPQS5DFpgnhdhH1p8T6syV9Sse7fjJK1XvAN192", "oid": "6ek8Nam37vUeUEXXFgbQTG2GQiBfmB3jtGBJjAT78PV5", "error": "mark object removal via connection pool: status: code = 1024 message = incomplete object PUT by placement: number of replicas cannot be met for list #0: 1 required, 0 nodes remaining (last node error: could not close object stream: replicate object to remote node (key=02e4643c9eea7de165fcf7945a4491224c43dbb702356ba30e0ed84c4bf2111b54): /dns4/localhost/tcp/33759: status: code = 1024 message = failed to verify and store object locally: validate payload content: tombstone verification: verifying 6ek8Nam37vUeUEXXFgbQTG2GQiBfmB3jtGBJjAT78PV5 member: heading object: read object header from NeoFS: /dns4/localhost/tcp/38303: status: code = 2052 message = object already removed)"}
debug	layer/layer.go:1019	delete object success	{"object": "key_1:6ek8Nam37vUeUEXXFgbQTG2GQiBfmB3jtGBJjAT78PV5", "reqId": "20936c66-f5c1-4683-9217-d08c0e6f53c9"}
debug	layer/object.go:844	delete object	{"reqId": "20936c66-f5c1-4683-9217-d08c0e6f53c9", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "cid": "4Z4RimPQS5DFpgnhdhH1p8T6syV9Sse7fjJK1XvAN192", "oid": "5ssiVKuAqZTUrFvSJ9A2fnaeXCTvYrMJoYnxKgVZCQwX", "error": "mark object removal via connection pool: status: code = 2052 message = object already removed"}
debug	layer/layer.go:1019	delete object success	{"object": "key_2:5ssiVKuAqZTUrFvSJ9A2fnaeXCTvYrMJoYnxKgVZCQwX", "reqId": "20936c66-f5c1-4683-9217-d08c0e6f53c9"}
debug	layer/object.go:844	delete object	{"reqId": "20936c66-f5c1-4683-9217-d08c0e6f53c9", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "cid": "4Z4RimPQS5DFpgnhdhH1p8T6syV9Sse7fjJK1XvAN192", "oid": "b2LpEDyeSsqCDfnAvTEmakfNTiX7WYfK6WbRfTCg3cj", "error": "mark object removal via connection pool: status: code = 2052 message = object already removed"}
debug	layer/layer.go:1019	delete object success	{"object": "key_3:b2LpEDyeSsqCDfnAvTEmakfNTiX7WYfK6WbRfTCg3cj", "reqId": "20936c66-f5c1-4683-9217-d08c0e6f53c9"}
debug	layer/object.go:844	delete object	{"reqId": "91a48cfb-3a0d-46fb-bdc1-799cb7f9dafa", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "cid": "4Z4RimPQS5DFpgnhdhH1p8T6syV9Sse7fjJK1XvAN192", "oid": "b2LpEDyeSsqCDfnAvTEmakfNTiX7WYfK6WbRfTCg3cj", "error": "mark object removal via connection pool: status: code = 1024 message = incomplete object PUT by placement: number of replicas cannot be met for list #0: 1 required, 0 nodes remaining (last node error: could not close object stream: replicate object to remote node (key=03af9a767790daf62366499ca902bdac886ae13015aed3ba513930158e2906eb81): /dns4/localhost/tcp/33917: status: code = 1024 message = failed to verify and store object locally: validate payload content: tombstone verification: verifying b2LpEDyeSsqCDfnAvTEmakfNTiX7WYfK6WbRfTCg3cj member: heading object: read object header from NeoFS: /dns4/localhost/tcp/33759: status: code = 2052 message = object already removed)"}
debug	layer/layer.go:1019	delete object success	{"object": "key_3:b2LpEDyeSsqCDfnAvTEmakfNTiX7WYfK6WbRfTCg3cj", "reqId": "91a48cfb-3a0d-46fb-bdc1-799cb7f9dafa"}
debug	layer/object.go:844	delete object	{"reqId": "91a48cfb-3a0d-46fb-bdc1-799cb7f9dafa", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "cid": "4Z4RimPQS5DFpgnhdhH1p8T6syV9Sse7fjJK1XvAN192", "oid": "BizHV9ZBjQcTCuiBCDjpzmssyph35Xnik3whbthVSPus", "error": "mark object removal via connection pool: status: code = 2052 message = object already removed"}
debug	layer/layer.go:1019	delete object success	{"object": "key_4:BizHV9ZBjQcTCuiBCDjpzmssyph35Xnik3whbthVSPus", "reqId": "91a48cfb-3a0d-46fb-bdc1-799cb7f9dafa"}
info	api/router.go:193	call method result	{"request_id": "91a48cfb-3a0d-46fb-bdc1-799cb7f9dafa", "status": 200, "description": "OK"}
debug	layer/object.go:844	delete object	{"reqId": "6f93b598-8e0c-466f-8beb-c81aa7b5a43f", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "cid": "4Z4RimPQS5DFpgnhdhH1p8T6syV9Sse7fjJK1XvAN192", "oid": "5ssiVKuAqZTUrFvSJ9A2fnaeXCTvYrMJoYnxKgVZCQwX"}
debug	layer/layer.go:1019	delete object success	{"object": "key_2:5ssiVKuAqZTUrFvSJ9A2fnaeXCTvYrMJoYnxKgVZCQwX", "reqId": "6f93b598-8e0c-466f-8beb-c81aa7b5a43f"}
debug	layer/object.go:844	delete object	{"reqId": "5bfa6926-2a92-44d6-bbeb-9e4cc29498ef", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "cid": "4Z4RimPQS5DFpgnhdhH1p8T6syV9Sse7fjJK1XvAN192", "oid": "5ssiVKuAqZTUrFvSJ9A2fnaeXCTvYrMJoYnxKgVZCQwX"}
debug	layer/layer.go:1019	delete object success	{"object": "key_2:5ssiVKuAqZTUrFvSJ9A2fnaeXCTvYrMJoYnxKgVZCQwX", "reqId": "5bfa6926-2a92-44d6-bbeb-9e4cc29498ef"}
debug	layer/object.go:844	delete object	{"reqId": "20936c66-f5c1-4683-9217-d08c0e6f53c9", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "cid": "4Z4RimPQS5DFpgnhdhH1p8T6syV9Sse7fjJK1XvAN192", "oid": "BizHV9ZBjQcTCuiBCDjpzmssyph35Xnik3whbthVSPus", "error": "mark object removal via connection pool: status: code = 1024 message = incomplete object PUT by placement: number of replicas cannot be met for list #0: 2 required, 1 nodes remaining (last node error: could not close object stream: replicate object to remote node (key=03af9a767790daf62366499ca902bdac886ae13015aed3ba513930158e2906eb81): /dns4/localhost/tcp/33917: status: code = 1024 message = failed to verify and store object locally: validate payload content: tombstone verification: verifying BizHV9ZBjQcTCuiBCDjpzmssyph35Xnik3whbthVSPus member: heading object: logical error: status: code = 2052 message = object already removed)"}
debug	layer/layer.go:1019	delete object success	{"object": "key_4:BizHV9ZBjQcTCuiBCDjpzmssyph35Xnik3whbthVSPus", "reqId": "20936c66-f5c1-4683-9217-d08c0e6f53c9"}
info	api/router.go:193	call method result	{"request_id": "20936c66-f5c1-4683-9217-d08c0e6f53c9", "status": 200, "description": "OK"}
debug	layer/object.go:844	delete object	{"reqId": "2fade456-d1e9-491a-b9de-777690037368", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "cid": "4Z4RimPQS5DFpgnhdhH1p8T6syV9Sse7fjJK1XvAN192", "oid": "BizHV9ZBjQcTCuiBCDjpzmssyph35Xnik3whbthVSPus"}
debug	layer/layer.go:1019	delete object success	{"object": "key_4:BizHV9ZBjQcTCuiBCDjpzmssyph35Xnik3whbthVSPus", "reqId": "2fade456-d1e9-491a-b9de-777690037368"}
info	api/router.go:193	call method result	{"request_id": "2fade456-d1e9-491a-b9de-777690037368", "status": 200, "description": "OK"}
debug	layer/object.go:844	delete object	{"reqId": "5bfa6926-2a92-44d6-bbeb-9e4cc29498ef", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "cid": "4Z4RimPQS5DFpgnhdhH1p8T6syV9Sse7fjJK1XvAN192", "oid": "b2LpEDyeSsqCDfnAvTEmakfNTiX7WYfK6WbRfTCg3cj", "error": "mark object removal via connection pool: status: code = 2052 message = object already removed"}
debug	layer/layer.go:1019	delete object success	{"object": "key_3:b2LpEDyeSsqCDfnAvTEmakfNTiX7WYfK6WbRfTCg3cj", "reqId": "5bfa6926-2a92-44d6-bbeb-9e4cc29498ef"}
debug	layer/object.go:844	delete object	{"reqId": "5bfa6926-2a92-44d6-bbeb-9e4cc29498ef", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "cid": "4Z4RimPQS5DFpgnhdhH1p8T6syV9Sse7fjJK1XvAN192", "oid": "BizHV9ZBjQcTCuiBCDjpzmssyph35Xnik3whbthVSPus", "error": "mark object removal via connection pool: status: code = 2052 message = object already removed"}
debug	layer/layer.go:1019	delete object success	{"object": "key_4:BizHV9ZBjQcTCuiBCDjpzmssyph35Xnik3whbthVSPus", "reqId": "5bfa6926-2a92-44d6-bbeb-9e4cc29498ef"}
info	api/router.go:193	call method result	{"request_id": "5bfa6926-2a92-44d6-bbeb-9e4cc29498ef", "status": 200, "description": "OK"}
debug	layer/object.go:844	delete object	{"reqId": "6f93b598-8e0c-466f-8beb-c81aa7b5a43f", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "cid": "4Z4RimPQS5DFpgnhdhH1p8T6syV9Sse7fjJK1XvAN192", "oid": "b2LpEDyeSsqCDfnAvTEmakfNTiX7WYfK6WbRfTCg3cj", "error": "mark object removal via connection pool: status: code = 1024 message = incomplete object PUT by placement: number of replicas cannot be met for list #0: 3 required, 1 nodes remaining (last node error: could not close object stream: replicate object to remote node (key=03af9a767790daf62366499ca902bdac886ae13015aed3ba513930158e2906eb81): /dns4/localhost/tcp/33917: status: code = 1024 message = failed to verify and store object locally: validate payload content: tombstone verification: verifying b2LpEDyeSsqCDfnAvTEmakfNTiX7WYfK6WbRfTCg3cj member: heading object: read object header from NeoFS: /dns4/localhost/tcp/33759: status: code = 2052 message = object already removed)"}
debug	layer/layer.go:1019	delete object success	{"object": "key_3:b2LpEDyeSsqCDfnAvTEmakfNTiX7WYfK6WbRfTCg3cj", "reqId": "6f93b598-8e0c-466f-8beb-c81aa7b5a43f"}
debug	layer/object.go:844	delete object	{"reqId": "6f93b598-8e0c-466f-8beb-c81aa7b5a43f", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "cid": "4Z4RimPQS5DFpgnhdhH1p8T6syV9Sse7fjJK1XvAN192", "oid": "BizHV9ZBjQcTCuiBCDjpzmssyph35Xnik3whbthVSPus", "error": "mark object removal via connection pool: status: code = 2052 message = object already removed"}
debug	layer/layer.go:1019	delete object success	{"object": "key_4:BizHV9ZBjQcTCuiBCDjpzmssyph35Xnik3whbthVSPus", "reqId": "6f93b598-8e0c-466f-8beb-c81aa7b5a43f"}
info	api/router.go:193	call method result	{"request_id": "6f93b598-8e0c-466f-8beb-c81aa7b5a43f", "status": 200, "description": "OK"}
info	api/router.go:177	call method	{"host": "localhost:34959", "request_id": "39f9554c-25e2-4d59-b1ee-0952ad035f89", "method": "ListObjectsV1", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "object": ""}
error	handler/object_list.go:38	ListObjectsV1Handler	{"bucket": "key_3", "oid": "b2LpEDyeSsqCDfnAvTEmakfNTiX7WYfK6WbRfTCg3cj"}
info	api/router.go:193	call method result	{"request_id": "39f9554c-25e2-4d59-b1ee-0952ad035f89", "status": 200, "description": "OK"}

Expected Behavior

Object b2LpEDyeSsqCDfnAvTEmakfNTiX7WYfK6WbRfTCg3cj removed

Current Behavior

Object b2LpEDyeSsqCDfnAvTEmakfNTiX7WYfK6WbRfTCg3cj found via search v2

Steps to Reproduce (for bugs)

Run test a lot of times

Regression

No, I don't think so

Originally created by @smallhive on GitHub (Apr 25, 2025). Originally assigned to: @cthulhu-rider on GitHub. <!-- Provide a general summary of the issue in the Title above --> There is [s3 gate test](https://github.com/nspcc-dev/neofs-s3-gw/issues/1129), which sometimes fails. The fail rate is about 1:350, which is very difficult to detect and debug. Eventually got the next S3 Gate log ``` info api/router.go:177 call method {"host": "localhost:34959", "request_id": "6c0d1246-eb0c-4323-8ddf-816a2d7c7076", "method": "CreateBucket", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "object": ""} debug layer/layer.go:411 bucket not found {"error": "nns get: invocation failed: at instruction 4370 (THROW): unhandled exception: \"token not found\""} info handler/put.go:814 bucket is created {"reqId": "6c0d1246-eb0c-4323-8ddf-816a2d7c7076", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "container_id": "4Z4RimPQS5DFpgnhdhH1p8T6syV9Sse7fjJK1XvAN192"} info api/router.go:193 call method result {"request_id": "6c0d1246-eb0c-4323-8ddf-816a2d7c7076", "status": 200, "description": "OK"} info api/router.go:177 call method {"host": "localhost:34959", "request_id": "0323ec33-dd1f-4582-bafb-39e1c04c4359", "method": "PutBucketVersioning", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "object": ""} info api/router.go:193 call method result {"request_id": "0323ec33-dd1f-4582-bafb-39e1c04c4359", "status": 200, "description": "OK"} info api/router.go:177 call method {"host": "localhost:34959", "request_id": "7f696f48-8bc4-45de-b9d7-e6f232813b36", "method": "GetBucketVersioning", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "object": ""} info api/router.go:193 call method result {"request_id": "7f696f48-8bc4-45de-b9d7-e6f232813b36", "status": 200, "description": "OK"} info api/router.go:177 call method {"host": "localhost:34959", "request_id": "90ccc8f2-da2b-4c91-8186-bb81cc3543f9", "method": "PutObject", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "object": "key_0"} debug layer/object.go:288 put object {"reqId": "90ccc8f2-da2b-4c91-8186-bb81cc3543f9", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "cid": "4Z4RimPQS5DFpgnhdhH1p8T6syV9Sse7fjJK1XvAN192", "object": "key_0", "oid": "DRDei9LyqRW8yx3vwhZsFSJgcbp4MneLMH8kZDJn3jC1", "size": 5} info api/router.go:193 call method result {"request_id": "90ccc8f2-da2b-4c91-8186-bb81cc3543f9", "status": 200, "description": "OK"} info api/router.go:177 call method {"host": "localhost:34959", "request_id": "7cff0881-b4ff-45fc-9357-80ed0508446b", "method": "PutObject", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "object": "key_1"} debug layer/object.go:288 put object {"reqId": "7cff0881-b4ff-45fc-9357-80ed0508446b", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "cid": "4Z4RimPQS5DFpgnhdhH1p8T6syV9Sse7fjJK1XvAN192", "object": "key_1", "oid": "6ek8Nam37vUeUEXXFgbQTG2GQiBfmB3jtGBJjAT78PV5", "size": 5} info api/router.go:193 call method result {"request_id": "7cff0881-b4ff-45fc-9357-80ed0508446b", "status": 200, "description": "OK"} info api/router.go:177 call method {"host": "localhost:34959", "request_id": "c3566d7c-0d0e-45a9-a7e3-a6cdb397449b", "method": "PutObject", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "object": "key_2"} debug layer/object.go:288 put object {"reqId": "c3566d7c-0d0e-45a9-a7e3-a6cdb397449b", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "cid": "4Z4RimPQS5DFpgnhdhH1p8T6syV9Sse7fjJK1XvAN192", "object": "key_2", "oid": "5ssiVKuAqZTUrFvSJ9A2fnaeXCTvYrMJoYnxKgVZCQwX", "size": 5} info api/router.go:193 call method result {"request_id": "c3566d7c-0d0e-45a9-a7e3-a6cdb397449b", "status": 200, "description": "OK"} info api/router.go:177 call method {"host": "localhost:34959", "request_id": "94ab8c0b-0d38-412b-b4a4-8fc24f47681b", "method": "PutObject", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "object": "key_3"} debug layer/object.go:288 put object {"reqId": "94ab8c0b-0d38-412b-b4a4-8fc24f47681b", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "cid": "4Z4RimPQS5DFpgnhdhH1p8T6syV9Sse7fjJK1XvAN192", "object": "key_3", "oid": "b2LpEDyeSsqCDfnAvTEmakfNTiX7WYfK6WbRfTCg3cj", "size": 5} info api/router.go:193 call method result {"request_id": "94ab8c0b-0d38-412b-b4a4-8fc24f47681b", "status": 200, "description": "OK"} info api/router.go:177 call method {"host": "localhost:34959", "request_id": "f05902ee-b064-4c1d-bf03-c83a13dabb1b", "method": "PutObject", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "object": "key_4"} debug layer/object.go:288 put object {"reqId": "f05902ee-b064-4c1d-bf03-c83a13dabb1b", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "cid": "4Z4RimPQS5DFpgnhdhH1p8T6syV9Sse7fjJK1XvAN192", "object": "key_4", "oid": "BizHV9ZBjQcTCuiBCDjpzmssyph35Xnik3whbthVSPus", "size": 5} info api/router.go:193 call method result {"request_id": "f05902ee-b064-4c1d-bf03-c83a13dabb1b", "status": 200, "description": "OK"} info api/router.go:177 call method {"host": "localhost:34959", "request_id": "d9da6a86-62ed-446c-a853-33eb30a0e300", "method": "ListBucketVersions", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "object": ""} info api/router.go:193 call method result {"request_id": "d9da6a86-62ed-446c-a853-33eb30a0e300", "status": 200, "description": "OK"} info api/router.go:177 call method {"host": "localhost:34959", "request_id": "91a48cfb-3a0d-46fb-bdc1-799cb7f9dafa", "method": "DeleteMultipleObjects", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "object": ""} info api/router.go:177 call method {"host": "localhost:34959", "request_id": "2fade456-d1e9-491a-b9de-777690037368", "method": "DeleteMultipleObjects", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "object": ""} info api/router.go:177 call method {"host": "localhost:34959", "request_id": "5bfa6926-2a92-44d6-bbeb-9e4cc29498ef", "method": "DeleteMultipleObjects", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "object": ""} info api/router.go:177 call method {"host": "localhost:34959", "request_id": "20936c66-f5c1-4683-9217-d08c0e6f53c9", "method": "DeleteMultipleObjects", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "object": ""} debug layer/object.go:844 delete object {"reqId": "2fade456-d1e9-491a-b9de-777690037368", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "cid": "4Z4RimPQS5DFpgnhdhH1p8T6syV9Sse7fjJK1XvAN192", "oid": "DRDei9LyqRW8yx3vwhZsFSJgcbp4MneLMH8kZDJn3jC1", "error": "mark object removal via connection pool: status: code = 2052 message = object already removed"} debug layer/layer.go:1019 delete object success {"object": "key_0:DRDei9LyqRW8yx3vwhZsFSJgcbp4MneLMH8kZDJn3jC1", "reqId": "2fade456-d1e9-491a-b9de-777690037368"} info api/router.go:177 call method {"host": "localhost:34959", "request_id": "6f93b598-8e0c-466f-8beb-c81aa7b5a43f", "method": "DeleteMultipleObjects", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "object": ""} debug layer/object.go:844 delete object {"reqId": "20936c66-f5c1-4683-9217-d08c0e6f53c9", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "cid": "4Z4RimPQS5DFpgnhdhH1p8T6syV9Sse7fjJK1XvAN192", "oid": "DRDei9LyqRW8yx3vwhZsFSJgcbp4MneLMH8kZDJn3jC1", "error": "mark object removal via connection pool: status: code = 2052 message = object already removed"} debug layer/layer.go:1019 delete object success {"object": "key_0:DRDei9LyqRW8yx3vwhZsFSJgcbp4MneLMH8kZDJn3jC1", "reqId": "20936c66-f5c1-4683-9217-d08c0e6f53c9"} debug layer/object.go:844 delete object {"reqId": "6f93b598-8e0c-466f-8beb-c81aa7b5a43f", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "cid": "4Z4RimPQS5DFpgnhdhH1p8T6syV9Sse7fjJK1XvAN192", "oid": "DRDei9LyqRW8yx3vwhZsFSJgcbp4MneLMH8kZDJn3jC1", "error": "mark object removal via connection pool: status: code = 2052 message = object already removed"} debug layer/layer.go:1019 delete object success {"object": "key_0:DRDei9LyqRW8yx3vwhZsFSJgcbp4MneLMH8kZDJn3jC1", "reqId": "6f93b598-8e0c-466f-8beb-c81aa7b5a43f"} debug layer/object.go:844 delete object {"reqId": "5bfa6926-2a92-44d6-bbeb-9e4cc29498ef", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "cid": "4Z4RimPQS5DFpgnhdhH1p8T6syV9Sse7fjJK1XvAN192", "oid": "DRDei9LyqRW8yx3vwhZsFSJgcbp4MneLMH8kZDJn3jC1", "error": "mark object removal via connection pool: status: code = 2052 message = object already removed"} debug layer/layer.go:1019 delete object success {"object": "key_0:DRDei9LyqRW8yx3vwhZsFSJgcbp4MneLMH8kZDJn3jC1", "reqId": "5bfa6926-2a92-44d6-bbeb-9e4cc29498ef"} debug layer/object.go:844 delete object {"reqId": "6f93b598-8e0c-466f-8beb-c81aa7b5a43f", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "cid": "4Z4RimPQS5DFpgnhdhH1p8T6syV9Sse7fjJK1XvAN192", "oid": "6ek8Nam37vUeUEXXFgbQTG2GQiBfmB3jtGBJjAT78PV5", "error": "mark object removal via connection pool: status: code = 2052 message = object already removed"} debug layer/layer.go:1019 delete object success {"object": "key_1:6ek8Nam37vUeUEXXFgbQTG2GQiBfmB3jtGBJjAT78PV5", "reqId": "6f93b598-8e0c-466f-8beb-c81aa7b5a43f"} debug layer/object.go:844 delete object {"reqId": "5bfa6926-2a92-44d6-bbeb-9e4cc29498ef", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "cid": "4Z4RimPQS5DFpgnhdhH1p8T6syV9Sse7fjJK1XvAN192", "oid": "6ek8Nam37vUeUEXXFgbQTG2GQiBfmB3jtGBJjAT78PV5", "error": "mark object removal via connection pool: status: code = 2052 message = object already removed"} debug layer/layer.go:1019 delete object success {"object": "key_1:6ek8Nam37vUeUEXXFgbQTG2GQiBfmB3jtGBJjAT78PV5", "reqId": "5bfa6926-2a92-44d6-bbeb-9e4cc29498ef"} debug layer/object.go:844 delete object {"reqId": "91a48cfb-3a0d-46fb-bdc1-799cb7f9dafa", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "cid": "4Z4RimPQS5DFpgnhdhH1p8T6syV9Sse7fjJK1XvAN192", "oid": "DRDei9LyqRW8yx3vwhZsFSJgcbp4MneLMH8kZDJn3jC1"} debug layer/layer.go:1019 delete object success {"object": "key_0:DRDei9LyqRW8yx3vwhZsFSJgcbp4MneLMH8kZDJn3jC1", "reqId": "91a48cfb-3a0d-46fb-bdc1-799cb7f9dafa"} debug layer/object.go:844 delete object {"reqId": "91a48cfb-3a0d-46fb-bdc1-799cb7f9dafa", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "cid": "4Z4RimPQS5DFpgnhdhH1p8T6syV9Sse7fjJK1XvAN192", "oid": "6ek8Nam37vUeUEXXFgbQTG2GQiBfmB3jtGBJjAT78PV5", "error": "mark object removal via connection pool: status: code = 2052 message = object already removed"} debug layer/layer.go:1019 delete object success {"object": "key_1:6ek8Nam37vUeUEXXFgbQTG2GQiBfmB3jtGBJjAT78PV5", "reqId": "91a48cfb-3a0d-46fb-bdc1-799cb7f9dafa"} debug layer/object.go:844 delete object {"reqId": "2fade456-d1e9-491a-b9de-777690037368", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "cid": "4Z4RimPQS5DFpgnhdhH1p8T6syV9Sse7fjJK1XvAN192", "oid": "6ek8Nam37vUeUEXXFgbQTG2GQiBfmB3jtGBJjAT78PV5", "error": "mark object removal via connection pool: status: code = 1024 message = incomplete object PUT by placement: number of replicas cannot be met for list #0: 1 required, 0 nodes remaining (last node error: could not close object stream: replicate object to remote node (key=03af9a767790daf62366499ca902bdac886ae13015aed3ba513930158e2906eb81): /dns4/localhost/tcp/33917: status: code = 1024 message = failed to verify and store object locally: validate payload content: tombstone verification: verifying 6ek8Nam37vUeUEXXFgbQTG2GQiBfmB3jtGBJjAT78PV5 member: heading object: logical error: status: code = 2052 message = object already removed)"} debug layer/layer.go:1019 delete object success {"object": "key_1:6ek8Nam37vUeUEXXFgbQTG2GQiBfmB3jtGBJjAT78PV5", "reqId": "2fade456-d1e9-491a-b9de-777690037368"} debug layer/object.go:844 delete object {"reqId": "91a48cfb-3a0d-46fb-bdc1-799cb7f9dafa", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "cid": "4Z4RimPQS5DFpgnhdhH1p8T6syV9Sse7fjJK1XvAN192", "oid": "5ssiVKuAqZTUrFvSJ9A2fnaeXCTvYrMJoYnxKgVZCQwX", "error": "mark object removal via connection pool: status: code = 2052 message = object already removed"} debug layer/layer.go:1019 delete object success {"object": "key_2:5ssiVKuAqZTUrFvSJ9A2fnaeXCTvYrMJoYnxKgVZCQwX", "reqId": "91a48cfb-3a0d-46fb-bdc1-799cb7f9dafa"} debug layer/object.go:844 delete object {"reqId": "2fade456-d1e9-491a-b9de-777690037368", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "cid": "4Z4RimPQS5DFpgnhdhH1p8T6syV9Sse7fjJK1XvAN192", "oid": "5ssiVKuAqZTUrFvSJ9A2fnaeXCTvYrMJoYnxKgVZCQwX", "error": "mark object removal via connection pool: status: code = 1024 message = incomplete object PUT by placement: number of replicas cannot be met for list #0: 3 required, 1 nodes remaining (last node error: could not close object stream: replicate object to remote node (key=03d2b4580016444af63f9e4dc8e7eb233666141386b501abc647b82bf54f7851ed): /dns4/localhost/tcp/35957: status: code = 1024 message = failed to verify and store object locally: validate payload content: tombstone verification: verifying 5ssiVKuAqZTUrFvSJ9A2fnaeXCTvYrMJoYnxKgVZCQwX member: heading object: logical error: status: code = 2052 message = object already removed)"} debug layer/layer.go:1019 delete object success {"object": "key_2:5ssiVKuAqZTUrFvSJ9A2fnaeXCTvYrMJoYnxKgVZCQwX", "reqId": "2fade456-d1e9-491a-b9de-777690037368"} debug layer/object.go:844 delete object {"reqId": "2fade456-d1e9-491a-b9de-777690037368", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "cid": "4Z4RimPQS5DFpgnhdhH1p8T6syV9Sse7fjJK1XvAN192", "oid": "b2LpEDyeSsqCDfnAvTEmakfNTiX7WYfK6WbRfTCg3cj", "error": "mark object removal via connection pool: status: code = 2052 message = object already removed"} debug layer/layer.go:1019 delete object success {"object": "key_3:b2LpEDyeSsqCDfnAvTEmakfNTiX7WYfK6WbRfTCg3cj", "reqId": "2fade456-d1e9-491a-b9de-777690037368"} debug layer/object.go:844 delete object {"reqId": "20936c66-f5c1-4683-9217-d08c0e6f53c9", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "cid": "4Z4RimPQS5DFpgnhdhH1p8T6syV9Sse7fjJK1XvAN192", "oid": "6ek8Nam37vUeUEXXFgbQTG2GQiBfmB3jtGBJjAT78PV5", "error": "mark object removal via connection pool: status: code = 1024 message = incomplete object PUT by placement: number of replicas cannot be met for list #0: 1 required, 0 nodes remaining (last node error: could not close object stream: replicate object to remote node (key=02e4643c9eea7de165fcf7945a4491224c43dbb702356ba30e0ed84c4bf2111b54): /dns4/localhost/tcp/33759: status: code = 1024 message = failed to verify and store object locally: validate payload content: tombstone verification: verifying 6ek8Nam37vUeUEXXFgbQTG2GQiBfmB3jtGBJjAT78PV5 member: heading object: read object header from NeoFS: /dns4/localhost/tcp/38303: status: code = 2052 message = object already removed)"} debug layer/layer.go:1019 delete object success {"object": "key_1:6ek8Nam37vUeUEXXFgbQTG2GQiBfmB3jtGBJjAT78PV5", "reqId": "20936c66-f5c1-4683-9217-d08c0e6f53c9"} debug layer/object.go:844 delete object {"reqId": "20936c66-f5c1-4683-9217-d08c0e6f53c9", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "cid": "4Z4RimPQS5DFpgnhdhH1p8T6syV9Sse7fjJK1XvAN192", "oid": "5ssiVKuAqZTUrFvSJ9A2fnaeXCTvYrMJoYnxKgVZCQwX", "error": "mark object removal via connection pool: status: code = 2052 message = object already removed"} debug layer/layer.go:1019 delete object success {"object": "key_2:5ssiVKuAqZTUrFvSJ9A2fnaeXCTvYrMJoYnxKgVZCQwX", "reqId": "20936c66-f5c1-4683-9217-d08c0e6f53c9"} debug layer/object.go:844 delete object {"reqId": "20936c66-f5c1-4683-9217-d08c0e6f53c9", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "cid": "4Z4RimPQS5DFpgnhdhH1p8T6syV9Sse7fjJK1XvAN192", "oid": "b2LpEDyeSsqCDfnAvTEmakfNTiX7WYfK6WbRfTCg3cj", "error": "mark object removal via connection pool: status: code = 2052 message = object already removed"} debug layer/layer.go:1019 delete object success {"object": "key_3:b2LpEDyeSsqCDfnAvTEmakfNTiX7WYfK6WbRfTCg3cj", "reqId": "20936c66-f5c1-4683-9217-d08c0e6f53c9"} debug layer/object.go:844 delete object {"reqId": "91a48cfb-3a0d-46fb-bdc1-799cb7f9dafa", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "cid": "4Z4RimPQS5DFpgnhdhH1p8T6syV9Sse7fjJK1XvAN192", "oid": "b2LpEDyeSsqCDfnAvTEmakfNTiX7WYfK6WbRfTCg3cj", "error": "mark object removal via connection pool: status: code = 1024 message = incomplete object PUT by placement: number of replicas cannot be met for list #0: 1 required, 0 nodes remaining (last node error: could not close object stream: replicate object to remote node (key=03af9a767790daf62366499ca902bdac886ae13015aed3ba513930158e2906eb81): /dns4/localhost/tcp/33917: status: code = 1024 message = failed to verify and store object locally: validate payload content: tombstone verification: verifying b2LpEDyeSsqCDfnAvTEmakfNTiX7WYfK6WbRfTCg3cj member: heading object: read object header from NeoFS: /dns4/localhost/tcp/33759: status: code = 2052 message = object already removed)"} debug layer/layer.go:1019 delete object success {"object": "key_3:b2LpEDyeSsqCDfnAvTEmakfNTiX7WYfK6WbRfTCg3cj", "reqId": "91a48cfb-3a0d-46fb-bdc1-799cb7f9dafa"} debug layer/object.go:844 delete object {"reqId": "91a48cfb-3a0d-46fb-bdc1-799cb7f9dafa", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "cid": "4Z4RimPQS5DFpgnhdhH1p8T6syV9Sse7fjJK1XvAN192", "oid": "BizHV9ZBjQcTCuiBCDjpzmssyph35Xnik3whbthVSPus", "error": "mark object removal via connection pool: status: code = 2052 message = object already removed"} debug layer/layer.go:1019 delete object success {"object": "key_4:BizHV9ZBjQcTCuiBCDjpzmssyph35Xnik3whbthVSPus", "reqId": "91a48cfb-3a0d-46fb-bdc1-799cb7f9dafa"} info api/router.go:193 call method result {"request_id": "91a48cfb-3a0d-46fb-bdc1-799cb7f9dafa", "status": 200, "description": "OK"} debug layer/object.go:844 delete object {"reqId": "6f93b598-8e0c-466f-8beb-c81aa7b5a43f", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "cid": "4Z4RimPQS5DFpgnhdhH1p8T6syV9Sse7fjJK1XvAN192", "oid": "5ssiVKuAqZTUrFvSJ9A2fnaeXCTvYrMJoYnxKgVZCQwX"} debug layer/layer.go:1019 delete object success {"object": "key_2:5ssiVKuAqZTUrFvSJ9A2fnaeXCTvYrMJoYnxKgVZCQwX", "reqId": "6f93b598-8e0c-466f-8beb-c81aa7b5a43f"} debug layer/object.go:844 delete object {"reqId": "5bfa6926-2a92-44d6-bbeb-9e4cc29498ef", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "cid": "4Z4RimPQS5DFpgnhdhH1p8T6syV9Sse7fjJK1XvAN192", "oid": "5ssiVKuAqZTUrFvSJ9A2fnaeXCTvYrMJoYnxKgVZCQwX"} debug layer/layer.go:1019 delete object success {"object": "key_2:5ssiVKuAqZTUrFvSJ9A2fnaeXCTvYrMJoYnxKgVZCQwX", "reqId": "5bfa6926-2a92-44d6-bbeb-9e4cc29498ef"} debug layer/object.go:844 delete object {"reqId": "20936c66-f5c1-4683-9217-d08c0e6f53c9", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "cid": "4Z4RimPQS5DFpgnhdhH1p8T6syV9Sse7fjJK1XvAN192", "oid": "BizHV9ZBjQcTCuiBCDjpzmssyph35Xnik3whbthVSPus", "error": "mark object removal via connection pool: status: code = 1024 message = incomplete object PUT by placement: number of replicas cannot be met for list #0: 2 required, 1 nodes remaining (last node error: could not close object stream: replicate object to remote node (key=03af9a767790daf62366499ca902bdac886ae13015aed3ba513930158e2906eb81): /dns4/localhost/tcp/33917: status: code = 1024 message = failed to verify and store object locally: validate payload content: tombstone verification: verifying BizHV9ZBjQcTCuiBCDjpzmssyph35Xnik3whbthVSPus member: heading object: logical error: status: code = 2052 message = object already removed)"} debug layer/layer.go:1019 delete object success {"object": "key_4:BizHV9ZBjQcTCuiBCDjpzmssyph35Xnik3whbthVSPus", "reqId": "20936c66-f5c1-4683-9217-d08c0e6f53c9"} info api/router.go:193 call method result {"request_id": "20936c66-f5c1-4683-9217-d08c0e6f53c9", "status": 200, "description": "OK"} debug layer/object.go:844 delete object {"reqId": "2fade456-d1e9-491a-b9de-777690037368", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "cid": "4Z4RimPQS5DFpgnhdhH1p8T6syV9Sse7fjJK1XvAN192", "oid": "BizHV9ZBjQcTCuiBCDjpzmssyph35Xnik3whbthVSPus"} debug layer/layer.go:1019 delete object success {"object": "key_4:BizHV9ZBjQcTCuiBCDjpzmssyph35Xnik3whbthVSPus", "reqId": "2fade456-d1e9-491a-b9de-777690037368"} info api/router.go:193 call method result {"request_id": "2fade456-d1e9-491a-b9de-777690037368", "status": 200, "description": "OK"} debug layer/object.go:844 delete object {"reqId": "5bfa6926-2a92-44d6-bbeb-9e4cc29498ef", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "cid": "4Z4RimPQS5DFpgnhdhH1p8T6syV9Sse7fjJK1XvAN192", "oid": "b2LpEDyeSsqCDfnAvTEmakfNTiX7WYfK6WbRfTCg3cj", "error": "mark object removal via connection pool: status: code = 2052 message = object already removed"} debug layer/layer.go:1019 delete object success {"object": "key_3:b2LpEDyeSsqCDfnAvTEmakfNTiX7WYfK6WbRfTCg3cj", "reqId": "5bfa6926-2a92-44d6-bbeb-9e4cc29498ef"} debug layer/object.go:844 delete object {"reqId": "5bfa6926-2a92-44d6-bbeb-9e4cc29498ef", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "cid": "4Z4RimPQS5DFpgnhdhH1p8T6syV9Sse7fjJK1XvAN192", "oid": "BizHV9ZBjQcTCuiBCDjpzmssyph35Xnik3whbthVSPus", "error": "mark object removal via connection pool: status: code = 2052 message = object already removed"} debug layer/layer.go:1019 delete object success {"object": "key_4:BizHV9ZBjQcTCuiBCDjpzmssyph35Xnik3whbthVSPus", "reqId": "5bfa6926-2a92-44d6-bbeb-9e4cc29498ef"} info api/router.go:193 call method result {"request_id": "5bfa6926-2a92-44d6-bbeb-9e4cc29498ef", "status": 200, "description": "OK"} debug layer/object.go:844 delete object {"reqId": "6f93b598-8e0c-466f-8beb-c81aa7b5a43f", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "cid": "4Z4RimPQS5DFpgnhdhH1p8T6syV9Sse7fjJK1XvAN192", "oid": "b2LpEDyeSsqCDfnAvTEmakfNTiX7WYfK6WbRfTCg3cj", "error": "mark object removal via connection pool: status: code = 1024 message = incomplete object PUT by placement: number of replicas cannot be met for list #0: 3 required, 1 nodes remaining (last node error: could not close object stream: replicate object to remote node (key=03af9a767790daf62366499ca902bdac886ae13015aed3ba513930158e2906eb81): /dns4/localhost/tcp/33917: status: code = 1024 message = failed to verify and store object locally: validate payload content: tombstone verification: verifying b2LpEDyeSsqCDfnAvTEmakfNTiX7WYfK6WbRfTCg3cj member: heading object: read object header from NeoFS: /dns4/localhost/tcp/33759: status: code = 2052 message = object already removed)"} debug layer/layer.go:1019 delete object success {"object": "key_3:b2LpEDyeSsqCDfnAvTEmakfNTiX7WYfK6WbRfTCg3cj", "reqId": "6f93b598-8e0c-466f-8beb-c81aa7b5a43f"} debug layer/object.go:844 delete object {"reqId": "6f93b598-8e0c-466f-8beb-c81aa7b5a43f", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "cid": "4Z4RimPQS5DFpgnhdhH1p8T6syV9Sse7fjJK1XvAN192", "oid": "BizHV9ZBjQcTCuiBCDjpzmssyph35Xnik3whbthVSPus", "error": "mark object removal via connection pool: status: code = 2052 message = object already removed"} debug layer/layer.go:1019 delete object success {"object": "key_4:BizHV9ZBjQcTCuiBCDjpzmssyph35Xnik3whbthVSPus", "reqId": "6f93b598-8e0c-466f-8beb-c81aa7b5a43f"} info api/router.go:193 call method result {"request_id": "6f93b598-8e0c-466f-8beb-c81aa7b5a43f", "status": 200, "description": "OK"} info api/router.go:177 call method {"host": "localhost:34959", "request_id": "39f9554c-25e2-4d59-b1ee-0952ad035f89", "method": "ListObjectsV1", "bucket": "yournamehere-k47mm2av4fkpml4z-11", "object": ""} error handler/object_list.go:38 ListObjectsV1Handler {"bucket": "key_3", "oid": "b2LpEDyeSsqCDfnAvTEmakfNTiX7WYfK6WbRfTCg3cj"} info api/router.go:193 call method result {"request_id": "39f9554c-25e2-4d59-b1ee-0952ad035f89", "status": 200, "description": "OK"} ``` ## Expected Behavior Object `b2LpEDyeSsqCDfnAvTEmakfNTiX7WYfK6WbRfTCg3cj` removed ## Current Behavior Object `b2LpEDyeSsqCDfnAvTEmakfNTiX7WYfK6WbRfTCg3cj` found via search v2 ## Steps to Reproduce (for bugs) Run [test](https://github.com/nspcc-dev/neofs-s3-gw/issues/1129) a lot of times ## Regression No, I don't think so
sami 2025-12-28 17:22:54 +00:00
sami commented 2025-12-28 17:22:54 +00:00
Author
Owner
Copy link

@roman-khimov commented on GitHub (Apr 28, 2025):

It looks like the node doesn't do the request properly or not interpreting 2052 correctly. It's tombstone verificator that is doing this request and freaking out on 2052, not even the gate itself. Same things happens to report uploader from time to time:

Stderr: rpc error: remove object via client: status: code = 1024 message = incomplete object PUT by placement: number of replicas cannot be met for list #0: 1 required, 0 nodes remaining (last node error: could not close object stream: replicate object to remote node (key=03933279423bf3504b61f76912790a566d99929eb047f30c8699f8e49a34ab07f7): /dns4/minerva.nspcc.ru/tcp/8080: status: code = 1024 message = failed to verify and store object locally: validate payload content: tombstone verification: verifying DKL3QfEcgqW3TySTx7ox6wye6PdGSJsQEXpfHeNtyRmp member: heading object: read object header from NeoFS: /dns4/st1.storage.fs.neo.org/tcp/8080: status: code = 2052 message = object already removed)

https://github.com/nspcc-dev/neofs-rest-gw/actions/runs/14710063058/job/41279915286

@roman-khimov commented on GitHub (Apr 28, 2025): It looks like the node doesn't do the request properly or not interpreting 2052 correctly. It's tombstone verificator that is doing this request and freaking out on 2052, not even the gate itself. Same things happens to report uploader from time to time: > Stderr: rpc error: remove object via client: status: code = 1024 message = incomplete object PUT by placement: number of replicas cannot be met for list #0: 1 required, 0 nodes remaining (last node error: could not close object stream: replicate object to remote node (key=03933279423bf3504b61f76912790a566d99929eb047f30c8699f8e49a34ab07f7): /dns4/minerva.nspcc.ru/tcp/8080: status: code = 1024 message = failed to verify and store object locally: validate payload content: tombstone verification: verifying DKL3QfEcgqW3TySTx7ox6wye6PdGSJsQEXpfHeNtyRmp member: heading object: read object header from NeoFS: /dns4/st1.storage.fs.neo.org/tcp/8080: status: code = 2052 message = object already removed) https://github.com/nspcc-dev/neofs-rest-gw/actions/runs/14710063058/job/41279915286
sami commented 2025-12-28 17:22:54 +00:00
Author
Owner
Copy link

@cthulhu-rider commented on GitHub (Apr 29, 2025):

@smallhive may u pls try fix #3332?

@cthulhu-rider commented on GitHub (Apr 29, 2025): @smallhive may u pls try fix #3332?
sami commented 2025-12-28 17:22:54 +00:00
Author
Owner
Copy link

@smallhive commented on GitHub (Apr 29, 2025):

@smallhive may u pls try fix #3332?

Sure

@smallhive commented on GitHub (Apr 29, 2025): > [@smallhive](https://github.com/smallhive) may u pls try fix [#3332](https://github.com/nspcc-dev/neofs-node/pull/3332)? Sure
sami commented 2025-12-28 17:22:54 +00:00
Author
Owner
Copy link

@smallhive commented on GitHub (Apr 29, 2025):

I ran the test 1500 times - everything is green. Previously, it was about 5 failures for such attempts.
It looks like the issue is gone. @cthulhu-rider

@smallhive commented on GitHub (Apr 29, 2025): I ran the test 1500 times - everything is green. Previously, it was about 5 failures for such attempts. It looks like the issue is gone. @cthulhu-rider
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
master
put-ants-pool
lens-meta-resync
policer-boost-mode
fix-shard-evacuation-for-ec
fstree-head-buffer
initial-placement
test-metadata-contract
head-proxy-buffer
feat/meta-data-native-contract
local-head-buffer
head-local-bin
test/notary-lock
feat/meta-side-chain
notary-set-attribute
search-1000
ec-put-parallel
fix/incorrect-TS-placement
new-object-structure
get-forwarding
test-bolt
logs
decrease-prototick
bugfix/listen-maxobjsize-changes
2684-api-server-optimize-sig-verify
v0.51.1
v0.51.0
v0.50.2
v0.50.1
v0.50.0
v0.49.1
v0.49.0
v0.48.3
v0.48.2
v0.48.1
v0.48.0
v0.47.1
v0.47.0
v0.46.1
v0.46.0
v0.45.2
v0.45.1
v0.45.0
v0.44.2
v0.44.1
v0.44.0
v0.43.0
v0.42.1
v0.42.0
v0.41.1
v0.41.0
v0.40.1
v0.40.0
v0.39.2
v0.39.1
v0.39.0
v0.38.1
v0.38.0
v0.37.0
v0.36.1
v0.36.0
v0.35.0
v0.34.0
v0.33.0
v0.32.0
v0.31.0
v0.30.2
v0.30.1
v0.30.0
v0.29.0
v0.28.3
v0.28.2
v0.28.1
v0.28.0
v0.28.0-rc.3
v0.27.7
v0.27.6
v0.28.0-rc.2
v0.28.0-rc.1
v0.27.5
v0.27.4
v0.27.3
v0.27.2
v0.27.1
v0.27.0
v0.27.0-rc.1
v0.26.1
v0.26.0
v0.25.1
v0.25.0
v0.24.1
v0.24.0
v0.23.1
v0.23.0
v0.22.3
v0.22.2
v0.22.1
v0.22.0
v0.21.1
v0.21.0
v0.20.0
v0.19.0
v0.18.0
v0.17.0
v0.16.0
v0.15.0
v0.14.3
v0.14.2
v0.14.1
v0.14.0
v0.14.0-rc.1
v0.13.2
v0.13.1
v0.13.0
v0.13.0-rc.1
v0.12.1
v0.12.0
v0.12.0-rc3
v0.12.0-rc2
v0.12.0-rc1
v0.11.0
v0.10.0
Labels
Clear labels   
I1

   
I2

   
I3

   
I4

   
S0

   
S1

   
S2

   
S3

   
S4

   
U0

   
U1

   
U2

   
U3

   
U4

   
blocked

   
bug

   
config

   
dependencies

   
discussion

   
documentation

   
enhancement

   
enhancement

   
epic

   
feature

   
go

   
good first issue

   
help wanted

   
neofs-adm

   
neofs-cli

   
neofs-cli

   
neofs-cli

   
neofs-ir

   
neofs-lens

   
neofs-storage

   
neofs-storage

   
performance

   
question

   
security

   
task

   
test

   
windows
No labels
I1
I2
I3
I4
S0
S1
S2
S3
S4
U0
U1
U2
U3
U4
blocked
bug
config
dependencies
discussion
documentation
enhancement
enhancement
epic
feature
go
good first issue
help wanted
neofs-adm
neofs-cli
neofs-cli
neofs-cli
neofs-ir
neofs-lens
neofs-storage
neofs-storage
performance
question
security
task
test
windows
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
nspcc-dev/neofs-node#1410
No description provided.