Dynamic write cache threading #1435

New issue

Open

opened 2025-12-28 17:22:59 +00:00 by sami · 0 comments

sami commented

2025-12-28 17:22:59 +00:00

Owner

Originally created by @roman-khimov on GitHub (Jun 23, 2025).

I'm always frustrated when we can have better user experience but we don't yet. Our write cache is designed to be greedy in terms of used disk I/O and flushes things as fast as it can. But in a mixed RW/RO scenario this makes RO requests suffer a bit more than needed technically for short write load spikes. If this write burst completely fits into write cache size node can serve a bit more read requests while slowly flushing WC to the backing store.

Describe the solution you'd like

We can throttle WC flushing by limiting the number of threads involved. Some threads can be parked (waiting for signal) and only activated once WC hits some percentage of used space (like a single thread until 10% and then add threads linearly so that we'd have all of them doing their thing at 50% used space).

Notice that this will degrade pure writing load performance since WC will be depleted earlier in this case, but it's acceptable since scenarios with short write bursts and ~constant RO load are more probable in real life. This can be mitigated by increasing the size of WC as well.

Originally created by @roman-khimov on GitHub (Jun 23, 2025). ## Is your feature request related to a problem? Please describe. I'm always frustrated when we can have better user experience but we don't yet. Our write cache is designed to be greedy in terms of used disk I/O and flushes things as fast as it can. But in a mixed RW/RO scenario this makes RO requests suffer a bit more than needed technically for short write load spikes. If this write burst completely fits into write cache size node can serve a bit more read requests while slowly flushing WC to the backing store. ## Describe the solution you'd like We can throttle WC flushing by limiting the number of threads involved. Some threads can be parked (waiting for signal) and only activated once WC hits some percentage of used space (like a single thread until 10% and then add threads linearly so that we'd have all of them doing their thing at 50% used space). Notice that this will degrade pure writing load performance since WC will be depleted earlier in this case, but it's acceptable since scenarios with short write bursts and ~constant RO load are more probable in real life. This can be mitigated by increasing the size of WC as well.