Allow configurable LevelDB options #1492

New issue

Open

opened 2025-12-28 17:16:38 +00:00 by sami · 2 comments

sami commented

2025-12-28 17:16:38 +00:00

Owner

Originally created by @lock9 on GitHub (Mar 5, 2025).

The current configuration for LevelDB creates too many files. It's possible to decrease the number of files by changing the CompactionTableSize on LevelDB. The large amount of files has a negative impact on our IT infrastructure.

Describe the solution you'd like

Level DB has some parameters that can be tweaked, such as WriteBufferSize, BlockSize, BlockCacheCapacity, CompactionTableSize, CompactionL0Trigger, and OpenFilesCacheCapacity. These values should be configurable through the config file.

Describe alternatives you've considered

I've tried running 'db dump', but it takes a long time to complete.

Additional context

I've implemented this feature here, without any kind of test.
linkd-academy/neo-go@8ca4a311c0

Result with 5GB CompactionTableSize (may not be ideal, 1 GB could be better):

Changing some parameters made it run faster on my macOS. My computer was barely usable during synchronization.

Don't forget to add labels!

enhancement

Originally created by @lock9 on GitHub (Mar 5, 2025). ## Is your feature request related to a problem? Please describe. The current configuration for LevelDB creates too many files. It's possible to decrease the number of files by changing the CompactionTableSize on LevelDB. The large amount of files has a negative impact on our IT infrastructure. ## Describe the solution you'd like Level DB has some parameters that can be tweaked, such as WriteBufferSize, BlockSize, BlockCacheCapacity, CompactionTableSize, CompactionL0Trigger, and OpenFilesCacheCapacity. These values should be configurable through the config file. ## Describe alternatives you've considered I've tried running 'db dump', but it takes a long time to complete. ## Additional context I've implemented this feature here, without any kind of test. https://github.com/linkd-academy/neo-go/commit/8ca4a311c01f7627967bf6fe818083b02271bb96 Result with 5GB CompactionTableSize (may not be ideal, 1 GB could be better): ![Image](https://github.com/user-attachments/assets/22a82a69-87f7-41d0-8df1-c99428c9edff) Changing some parameters made it run faster on my macOS. My computer was barely usable during synchronization. ## Don't forget to add labels! - `enhancement`

sami added the

good first issue

enhancement

config

labels

2025-12-28 17:16:38 +00:00

sami commented

2025-12-28 17:16:38 +00:00

Author

Owner

@AnnaShaleva commented on GitHub (Mar 5, 2025):

I've implemented this feature here, without any kind of test.
linkd-academy/neo-go@8ca4a311c0

Patches are always welcomed! Open a PR, and we'll review&merge eventually.

@AnnaShaleva commented on GitHub (Mar 5, 2025): > I've implemented this feature here, without any kind of test. https://github.com/linkd-academy/neo-go/commit/8ca4a311c01f7627967bf6fe818083b02271bb96 Patches are always welcomed! Open a PR, and we'll review&merge eventually.

sami commented

2025-12-28 17:16:38 +00:00

Author

Owner

@fyfyrchik commented on GitHub (Mar 6, 2025):

Btw, boltdb also could also have some love.
There are InitialMmapSize and AllocSize which regulate how big the database is initially and how fast it grows.
It may be important, because to grow a database we need to remap it, and we cannot remap if there is an existing session iterator, so all transactions hang. I have reproduced it multiple times in privnet scenario with heavy iterator usage.
With these 2 settings the probability of such event can be made much smaller.

neofs-node also used to configure MaxBatchSize and MaxBatchDelay but I have found them not that useful for neo-go.

@fyfyrchik commented on GitHub (Mar 6, 2025): Btw, boltdb also could also have some love. There are `InitialMmapSize` and `AllocSize` which regulate how big the database is initially and how fast it grows. It may be important, because to grow a database we need to remap it, and we cannot remap if there is an existing session iterator, so all transactions hang. I have reproduced it multiple times in privnet scenario with heavy iterator usage. With these 2 settings the probability of such event can be made much smaller. neofs-node also used to configure `MaxBatchSize` and `MaxBatchDelay` but I have found them not that useful for neo-go.