Node can't remove object marked for deletion #1381

New issue

Closed

opened 2025-12-28 17:22:47 +00:00 by sami · 3 comments

sami commented

2025-12-28 17:22:47 +00:00

Owner

Originally created by @roman-khimov on GitHub (Mar 19, 2025).

Originally assigned to: @carpawell on GitHub.

Expected Behavior

Drop whatever data we have, the object is marked for deletion.

Current Behavior

If there is anything bad in the metabase, it can't remove object data:

neofs-node[279122]: warn        metabase/delete.go:87        failed to delete object        {"shard_id": "N5C9rj9NzZZ8Ma3y9eENZi", "addr": "VCVr6RKg1hfJR7v36xvjbaXTncj1waPi94u9EFyEttW/8TAXvA2tsfoZDLHCfUDG5mo9cL6aUnX8Kq1m7YuBzayC", "error": "proto: cannot parse invalid wire-format data"}
metis1 neofs-node[279122]: warn        shard/gc.go:192        could not delete the objects        {"shard_id": "N5C9rj9NzZZ8Ma3y9eENZi", "error": "deleted 999 out of 1000 objects, first error: VCVr6RKg1hfJR7v36xvjbaXTncj1waPi94u9EFyEttW/8TAXvA2tsfoZDLHCfUDG5mo9cL6aUnX8Kq1m7YuBzayC object delete fail: proto: cannot parse invalid wire-format data"}

Possible Solution

Drop things we can't parse.

Context

Mainnet. Related to old replication bugs that left these chunks of data in the metabase.

Your Environment

Version used: 0.45.1

Originally created by @roman-khimov on GitHub (Mar 19, 2025). Originally assigned to: @carpawell on GitHub. ## Expected Behavior Drop whatever data we have, the object is marked for deletion. ## Current Behavior If there is anything bad in the metabase, it can't remove object data: ``` neofs-node[279122]: warn metabase/delete.go:87 failed to delete object {"shard_id": "N5C9rj9NzZZ8Ma3y9eENZi", "addr": "VCVr6RKg1hfJR7v36xvjbaXTncj1waPi94u9EFyEttW/8TAXvA2tsfoZDLHCfUDG5mo9cL6aUnX8Kq1m7YuBzayC", "error": "proto: cannot parse invalid wire-format data"} metis1 neofs-node[279122]: warn shard/gc.go:192 could not delete the objects {"shard_id": "N5C9rj9NzZZ8Ma3y9eENZi", "error": "deleted 999 out of 1000 objects, first error: VCVr6RKg1hfJR7v36xvjbaXTncj1waPi94u9EFyEttW/8TAXvA2tsfoZDLHCfUDG5mo9cL6aUnX8Kq1m7YuBzayC object delete fail: proto: cannot parse invalid wire-format data"} ``` ## Possible Solution Drop things we can't parse. ## Context Mainnet. Related to old replication bugs that left these chunks of data in the metabase. ## Your Environment * Version used: 0.45.1

sami

2025-12-28 17:22:47 +00:00

closed this issue
added the
bug

neofs-storage

S4

I4

U1
labels

sami commented

2025-12-28 17:22:48 +00:00

Author

Owner

@roman-khimov commented on GitHub (Mar 19, 2025):

Unmarshaling is needed to get object parent (and size, but it's the same wrt this), but we already have this data in SearchV2 index. So if we're to reuse it we can avoid unmarshaling and make things a bit faster at the same time.

@roman-khimov commented on GitHub (Mar 19, 2025): Unmarshaling is needed to get object parent (and size, but it's the same wrt this), but we already have this data in SearchV2 index. So if we're to reuse it we can avoid unmarshaling and make things a bit faster at the same time.

sami commented

2025-12-28 17:22:48 +00:00

Author

Owner

@carpawell commented on GitHub (Mar 19, 2025):

@roman-khimov, @cthulhu-rider, just a clarification for this issue: this is an important bug for now for the exact corruption reason but its solution (i meant the exact issue's wording) requires an assumption that searchV2 index (and it does mean its migration in fact) is alive and ok even for corrupted data, do we assume it? Dropping anything we cannot unmarshal is easy of course, but dropping "whatever data we have" is a little bit trickier.

@carpawell commented on GitHub (Mar 19, 2025): @roman-khimov, @cthulhu-rider, just a clarification for this issue: this is an important bug for now for the exact corruption reason but its solution (i meant the exact issue's wording) requires an assumption that searchV2 index (and it does mean its migration in fact) is alive and ok even for corrupted data, do we assume it? Dropping anything we cannot unmarshal is easy of course, but dropping "whatever data we have" is a little bit trickier.

sami commented

2025-12-28 17:22:48 +00:00

Author

Owner

@roman-khimov commented on GitHub (Mar 20, 2025):

No unmarshaling --- no problem. These objects obviously were not migrated (no data to migrate), but that's not a problem either --- no parent is no parent.

@roman-khimov commented on GitHub (Mar 20, 2025): No unmarshaling --- no problem. These objects obviously were not migrated (no data to migrate), but that's not a problem either --- no parent is no parent.