Write cache fails to flush objects on every iteration #1431

New issue

Closed

opened 2025-12-28 17:22:59 +00:00 by sami · 0 comments

sami commented

2025-12-28 17:22:59 +00:00

Owner

Originally created by @roman-khimov on GitHub (Jun 19, 2025).

Originally assigned to: @End-rey on GitHub.

Expected Behavior

No errors.

Current Behavior

warn	writecache/flush.go:96	flush scheduler paused due to error	{"shard_id": "ABSWcATTDtT2M93iCYYYB4", "substorage": "write-cache", "delay": 10}
error	writecache/flush.go:60	can't flush batch of objects	{"shard_id": "ABSWcATTDtT2M93iCYYYB4", "substorage": "write-cache", "error": "logical error: status: code = 2049 message = object not found"}

Possible Solution

It looks like the in-memory list of objects and FSTree got out of sync. No deletions were made, so we need to figure out how this could happen.

Steps to Reproduce (for bugs)

Unknown. We have a lot of write cache activity in mainnet for example with nothing like this happening.

Context

Tests.

Regression

Unknown.

Your Environment

Version used: 0.47.1

Originally created by @roman-khimov on GitHub (Jun 19, 2025). Originally assigned to: @End-rey on GitHub. ## Expected Behavior No errors. ## Current Behavior ``` warn writecache/flush.go:96 flush scheduler paused due to error {"shard_id": "ABSWcATTDtT2M93iCYYYB4", "substorage": "write-cache", "delay": 10} error writecache/flush.go:60 can't flush batch of objects {"shard_id": "ABSWcATTDtT2M93iCYYYB4", "substorage": "write-cache", "error": "logical error: status: code = 2049 message = object not found"} ``` ## Possible Solution It looks like the in-memory list of objects and FSTree got out of sync. No deletions were made, so we need to figure out how this could happen. ## Steps to Reproduce (for bugs) Unknown. We have a lot of write cache activity in mainnet for example with nothing like this happening. ## Context Tests. ## Regression Unknown. ## Your Environment * Version used: 0.47.1