shard: Blobstor `file exists` errors move shard into degraded mode #1068

New issue

Open

opened 2025-12-28 17:21:40 +00:00 by sami · 2 comments

sami commented

2025-12-28 17:21:40 +00:00

Owner

Originally created by @cthulhu-rider on GitHub (Jul 14, 2023).

looks like a bug but needs to be investigated. When local storage is loaded by objects with existing files in blobstor, shard shards switches to degraded mode fast and stops storage service

warn        engine/engine.go:160        could not put object to shard        {"shard_id": "ERHCRoaBixN42afWgnJrzA", "error count": 1, "error": "could not put object to BLOB storage: file exists"}
warn        engine/engine.go:160        could not put object to shard        {"shard_id": "ERHCRoaBixN42afWgnJrzA", "error count": 2, "error": "could not put object to BLOB storage: file exists"}
warn        engine/engine.go:160        could not put object to shard        {"shard_id": "ERHCRoaBixN42afWgnJrzA", "error count": 3, "error": "could not put object to BLOB storage: file exists"}
warn        engine/engine.go:160        could not put object to shard        {"shard_id": "ERHCRoaBixN42afWgnJrzA", "error count": 4, "error": "could not put object to BLOB storage: file exists"}
...
info        shard/mode.go:28        setting shard mode        {"shard_id": "ERHCRoaBixN42afWgnJrzA", "old_mode": "READ_WRITE", "new_mode": "DEGRADED_READ_ONLY"}
info        shard/mode.go:69        shard mode set successfully        {"shard_id": "ERHCRoaBixN42afWgnJrzA", "mode": "DEGRADED_READ_ONLY"}
info        engine/engine.go:105        shard is moved in degraded mode due to error threshold        {"shard_id": "ERHCRoaBixN42afWgnJrzA", "error count": 30}
...
warn        engine/put.go:137        could not put object to shard        {"shard_id": "ERHCRoaBixN42afWgnJrzA", "error": "shard is in read-only mode"}
warn        engine/engine.go:142        could not select objects from shard        {"shard_id": "ERHCRoaBixN42afWgnJrzA", "error": "shard is in degraded mode"}
warn        engine/engine.go:142        can't perform `TreeGetOpLog`        {"shard_id": "ERHCRoaBixN42afWgnJrzA", "error": "pilorama is in a degraded mode"}

it seems pretty strange to me that attempts to rewrite files are considered as blunders and lead to the shard's degradation. Lets analyze can shard feel OK to object exists write exceptions and not degrade.

Originally created by @cthulhu-rider on GitHub (Jul 14, 2023). looks like a bug but needs to be investigated. When local storage is loaded by objects with existing files in blobstor, shard shards switches to degraded mode fast and stops storage service ``` warn engine/engine.go:160 could not put object to shard {"shard_id": "ERHCRoaBixN42afWgnJrzA", "error count": 1, "error": "could not put object to BLOB storage: file exists"} warn engine/engine.go:160 could not put object to shard {"shard_id": "ERHCRoaBixN42afWgnJrzA", "error count": 2, "error": "could not put object to BLOB storage: file exists"} warn engine/engine.go:160 could not put object to shard {"shard_id": "ERHCRoaBixN42afWgnJrzA", "error count": 3, "error": "could not put object to BLOB storage: file exists"} warn engine/engine.go:160 could not put object to shard {"shard_id": "ERHCRoaBixN42afWgnJrzA", "error count": 4, "error": "could not put object to BLOB storage: file exists"} ... info shard/mode.go:28 setting shard mode {"shard_id": "ERHCRoaBixN42afWgnJrzA", "old_mode": "READ_WRITE", "new_mode": "DEGRADED_READ_ONLY"} info shard/mode.go:69 shard mode set successfully {"shard_id": "ERHCRoaBixN42afWgnJrzA", "mode": "DEGRADED_READ_ONLY"} info engine/engine.go:105 shard is moved in degraded mode due to error threshold {"shard_id": "ERHCRoaBixN42afWgnJrzA", "error count": 30} ... warn engine/put.go:137 could not put object to shard {"shard_id": "ERHCRoaBixN42afWgnJrzA", "error": "shard is in read-only mode"} warn engine/engine.go:142 could not select objects from shard {"shard_id": "ERHCRoaBixN42afWgnJrzA", "error": "shard is in degraded mode"} warn engine/engine.go:142 can't perform `TreeGetOpLog` {"shard_id": "ERHCRoaBixN42afWgnJrzA", "error": "pilorama is in a degraded mode"} ``` it seems pretty strange to me that attempts to rewrite files are considered as blunders and lead to the shard's degradation. Lets analyze can shard feel OK to `object exists` write exceptions and not degrade.