Speed up metabase resync process #1454

New issue

Open

opened 2025-12-28 17:23:03 +00:00 by sami · 0 comments

sami commented

2025-12-28 17:23:03 +00:00

Owner

Originally created by @roman-khimov on GitHub (Jul 7, 2025).

I'm always frustrated when metabase resync takes too much time. Currently it iterates over all objects and does metabase.Put() for all objects as well as has some special handling for locks and tombstones. Lock and TS handling will be simplified with https://github.com/nspcc-dev/neofs-api/issues/312 and https://github.com/nspcc-dev/neofs-api/issues/311, but doing Put() for every object is problematic on its own because it's a BoltDB Batch() for every object and in absence of concurrent modifications this means max_batch_delay for every object. If it's 10ms per object this means ~3h for 1M objects not even counting reads and real writing time.

Describe the solution you'd like

Rework transaction handling for the resync case, open a transaction, process something like 1K objects, flush the transaction, rinse, repeat.

Describe alternatives you've considered

Keep wasting time for nothing.

Originally created by @roman-khimov on GitHub (Jul 7, 2025). ## Is your feature request related to a problem? Please describe. I'm always frustrated when metabase resync takes too much time. Currently it iterates over all objects and does `metabase.Put()` for all objects as well as has some special handling for locks and tombstones. Lock and TS handling will be simplified with https://github.com/nspcc-dev/neofs-api/issues/312 and https://github.com/nspcc-dev/neofs-api/issues/311, but doing `Put()` for every object is problematic on its own because it's a BoltDB `Batch()` for every object and in absence of concurrent modifications this means `max_batch_delay` for every object. If it's 10ms per object this means ~3h for 1M objects not even counting reads and real writing time. ## Describe the solution you'd like Rework transaction handling for the resync case, open a transaction, process something like 1K objects, flush the transaction, rinse, repeat. ## Describe alternatives you've considered Keep wasting time for nothing.