Stop consensus if N last blocks failed to persist #1238

New issue

Open

opened 2025-12-28 17:15:43 +00:00 by sami · 3 comments

sami commented

2025-12-28 17:15:43 +00:00

Owner

Originally created by @fyfyrchik on GitHub (Dec 15, 2023).

Problem

neo-go database lives on a disk. If there is no more space, the database can no longer grow and we will accumulate blocks in memory with these messages in logs:

2023-12-11T19:34:26.820Z        WARN        failed to persist blockchain        {"error": "write /var/lib/neo-go/mainnet.bolt: no space left on device"}

Nothing wrong in public networks, but in privnet scenario, where all nodes are likely to be similar in configuration, this situation can happen at the same time. This is pretty bad, because then we fail with OOM, restart neo-go and lose some of the last blocks (the exact number depends on the amount of RAM and timing), not to mention possible problems with consensus after restart.
Another problem is that clients connected to this particular node could not be prepared to time travelling.

Proposed solution

Add MaxFailedToPersistBlockCount: N config setting, which allows to stop consensus service if last N blocks failed to be persisted. The administrator can then extend partition or clean space via other means.
I am ready to do a PR if we accept this solution.
This is proved to be useful and we have encountered this situation twice. The second time this solution worked as expected and we were able to clean space on the partition and continue consensusing without restarting neo-go or losing any blocks.

2023-12-11T19:34:26.820Z        WARN        failed to persist blockchain        {"error": "write /var/lib/neo-go/mainnet.bolt: no space left on device"}
2023-12-11T19:34:26.820Z        INFO        stopping consensus service
....
2023-12-15T09:20:03.126Z        WARN        failed to persist blockchain        {"error": "write /var/lib/neo-go/mainnet.bolt: no space left on device"}
2023-12-15T09:20:10.957Z        INFO        persisted to disk        {"blocks": 6, "keys": 190, "headerHeight": 243508, "blockHeight": 243508, "took": "6.830437873s"}
2023-12-15T09:20:11.394Z        INFO        starting consensus service

Originally created by @fyfyrchik on GitHub (Dec 15, 2023). # Problem neo-go database lives on a disk. If there is no more space, the database can no longer grow and we will accumulate blocks in memory with these messages in logs: ``` 2023-12-11T19:34:26.820Z WARN failed to persist blockchain {"error": "write /var/lib/neo-go/mainnet.bolt: no space left on device"} ``` Nothing wrong in public networks, but in privnet scenario, where all nodes are likely to be similar in configuration, this situation can happen at the same time. This is pretty bad, because then we fail with OOM, restart neo-go and lose some of the last blocks (the exact number depends on the amount of RAM and timing), not to mention possible problems with consensus after restart. Another problem is that clients connected to this particular node could not be prepared to time travelling. # Proposed solution Add `MaxFailedToPersistBlockCount: N` config setting, which allows to stop consensus service if last N blocks failed to be persisted. The administrator can then extend partition or clean space via other means. I am ready to do a PR if we accept this solution. This is proved to be useful and we have encountered this situation twice. The second time this solution worked as expected and we were able to clean space on the partition and continue consensusing without restarting neo-go or losing any blocks. ``` 2023-12-11T19:34:26.820Z WARN failed to persist blockchain {"error": "write /var/lib/neo-go/mainnet.bolt: no space left on device"} 2023-12-11T19:34:26.820Z INFO stopping consensus service .... 2023-12-15T09:20:03.126Z WARN failed to persist blockchain {"error": "write /var/lib/neo-go/mainnet.bolt: no space left on device"} 2023-12-15T09:20:10.957Z INFO persisted to disk {"blocks": 6, "keys": 190, "headerHeight": 243508, "blockHeight": 243508, "took": "6.830437873s"} 2023-12-15T09:20:11.394Z INFO starting consensus service ```

sami added the

enhancement

labels

2025-12-28 17:15:43 +00:00

sami commented

2025-12-28 17:15:43 +00:00

Author

Owner

@roman-khimov commented on GitHub (Dec 15, 2023):

In general, I don't like new magic options, alternatives are:

write failures can be treated as critical for CNs. But then they will just die and lose the block for sure which is not a nice scenario either.
consensus can be stopped in the event of write failure instead, so the node will stay alive for as long as it needs to, but it won't create new blocks. Maybe this one is better.
we can write internal consensus state to the disk separately and not allow to proceed to the next dBFT round until it's OK. This is reminiscent of what C# node does already, but obviously affects performance.

At the same time, some reaction to write failures can be useful for ordinary nodes too.

Any other suggestions?

@roman-khimov commented on GitHub (Dec 15, 2023): In general, I don't like new magic options, alternatives are: * write failures can be treated as critical for CNs. But then they will just die and lose the block for sure which is not a nice scenario either. * consensus can be stopped in the event of write failure instead, so the node will stay alive for as long as it needs to, but it won't create new blocks. Maybe this one is better. * we can write internal consensus state to the disk separately and not allow to proceed to the next dBFT round until it's OK. This is reminiscent of what C# node does already, but obviously affects performance. At the same time, some reaction to write failures can be useful for ordinary nodes too. Any other suggestions?

sami commented

2025-12-28 17:15:43 +00:00

Author

Owner

@fyfyrchik commented on GitHub (Dec 15, 2023):

consensus can be stopped in the event of write failure instead, so the node will stay alive for as long as it needs to, but it won't create new blocks. Maybe this one is better.

So this is similar to MaxFailedToPersistBlockCount: 1 in the proposed solution? Looks also good to me. The option was initially there to behave smoothly in case of transient failures, even though I don't have any particular example, besides "no space" which is not transient. No options does indeed look better.

@fyfyrchik commented on GitHub (Dec 15, 2023): >consensus can be stopped in the event of write failure instead, so the node will stay alive for as long as it needs to, but it won't create new blocks. Maybe this one is better. So this is similar to `MaxFailedToPersistBlockCount: 1` in the proposed solution? Looks also good to me. The option was initially there to behave smoothly in case of transient failures, even though I don't have any particular example, besides "no space" which is not transient. No options does indeed look better.

sami commented

2025-12-28 17:15:43 +00:00

Author

Owner

@roman-khimov commented on GitHub (Dec 15, 2023):

So this is similar to MaxFailedToPersistBlockCount: 1 in the proposed solution?

Looks like.

in case of transient failures

Yeah, usually it's either "out of space" or "your disk/FS is broken, have fun".

@roman-khimov commented on GitHub (Dec 15, 2023): > So this is similar to MaxFailedToPersistBlockCount: 1 in the proposed solution? Looks like. > in case of transient failures Yeah, usually it's either "out of space" or "your disk/FS is broken, have fun".