Speed up shard evacuation process #1455

New issue

Open

opened 2025-12-28 17:23:03 +00:00 by sami · 0 comments

sami commented

2025-12-28 17:23:03 +00:00

Owner

Originally created by @roman-khimov on GitHub (Jul 7, 2025).

I'm always frustrated when shard evacuation takes more time than it could. We're pulling one object after another and writing them one after another, it's not very productive.

Describe the solution you'd like

Pull objects and push them into write queues (when pushing into multiple shards), accumulate small objects into batches and write batches of objects. Source shard iteration can be done by size if metabase is available.

Originally created by @roman-khimov on GitHub (Jul 7, 2025). ## Is your feature request related to a problem? Please describe. I'm always frustrated when shard evacuation takes more time than it could. We're pulling one object after another and writing them one after another, it's not very productive. ## Describe the solution you'd like Pull objects and push them into write queues (when pushing into multiple shards), accumulate small objects into batches and write batches of objects. Source shard iteration can be done by size if metabase is available.