Failed to apply MPT changes during `storeBlock` #1530

New issue

Open

opened 2025-12-28 17:16:45 +00:00 by sami · 0 comments

sami commented

2025-12-28 17:16:45 +00:00

Owner

Originally created by @AnnaShaleva on GitHub (Jun 2, 2025).

Current Behavior

Originally ported from https://github.com/nspcc-dev/neo-go/issues/3082#issuecomment-2890471825, turns out that the cause of this issue is different from https://github.com/nspcc-dev/neo-go/issues/3082#issue-1838866157.

To sum up, the problem of error while trying to apply MPT changes: key not found happens on attempt to persist new block. It happens only on BoltDB (until now there was no such cases on LevelDB, but we don't use LevelDB extensively on prod), potentially after ungraceful node shutdown (but not proven). The DB ends up in a corrupted state, see https://github.com/nspcc-dev/neo-go/issues/3082#issuecomment-2922888710 for detailed description. Ref. https://github.com/nspcc-dev/neo-go/issues/3082#issuecomment-2925049385.

Latest known occurrence is documented by @532910 at moros1 on neogo-fschain-cn --version -> 0.109.1 with BoltDB:

May 29 11:05:19 moros1 neogo-fschain-cn[1263406]: WARN        queue: failed to add item into the blockchain        {"mode": 0, "error": "error while trying to apply MPT changes: key not found", "height": 12962799, "nextIndex": 12962800}
May 29 11:05:45 moros1 neogo-fschain-cn[1263406]: WARN        queue: failed to add item into the blockchain        {"mode": 0, "error": "error while trying to apply MPT changes: key not found", "height": 12962799, "nextIndex": 12962800}
May 29 11:05:47 moros1 neogo-fschain-cn[1263406]: WARN        queue: failed to add item into the blockchain        {"mode": 0, "error": "error while trying to apply MPT changes: key not found", "height": 12962799, "nextIndex": 12962800}
May 29 11:05:49 moros1 neogo-fschain-cn[1263406]: WARN        queue: failed to add item into the blockchain        {"mode": 0, "error": "error while trying to apply MPT changes: key not found", "height": 12962799, "nextIndex": 12962800}
May 29 11:05:55 moros1 neogo-fschain-cn[1263406]: WARN        queue: failed to add item into the blockchain        {"mode": 0, "error": "error while trying to apply MPT changes: key not found", "height": 12962799, "nextIndex": 12962800}
May 29 11:05:57 moros1 neogo-fschain-cn[1263406]: WARN        queue: failed to add item into the blockchain        {"mode": 0, "error": "error while trying to apply MPT changes: key not found", "height": 12962799, "nextIndex": 12962800}

Expected Behavior

No error.

Originally created by @AnnaShaleva on GitHub (Jun 2, 2025). ## Current Behavior Originally ported from https://github.com/nspcc-dev/neo-go/issues/3082#issuecomment-2890471825, turns out that the cause of this issue is different from https://github.com/nspcc-dev/neo-go/issues/3082#issue-1838866157. To sum up, the problem of `error while trying to apply MPT changes: key not found` happens on attempt to persist new block. It happens only on BoltDB (until now there was no such cases on LevelDB, but we don't use LevelDB extensively on prod), potentially after ungraceful node shutdown (but not proven). The DB ends up in a corrupted state, see https://github.com/nspcc-dev/neo-go/issues/3082#issuecomment-2922888710 for detailed description. Ref. https://github.com/nspcc-dev/neo-go/issues/3082#issuecomment-2925049385. Latest known occurrence is documented by @532910 at moros1 on `neogo-fschain-cn --version -> 0.109.1` with BoltDB: ``` May 29 11:05:19 moros1 neogo-fschain-cn[1263406]: WARN queue: failed to add item into the blockchain {"mode": 0, "error": "error while trying to apply MPT changes: key not found", "height": 12962799, "nextIndex": 12962800} May 29 11:05:45 moros1 neogo-fschain-cn[1263406]: WARN queue: failed to add item into the blockchain {"mode": 0, "error": "error while trying to apply MPT changes: key not found", "height": 12962799, "nextIndex": 12962800} May 29 11:05:47 moros1 neogo-fschain-cn[1263406]: WARN queue: failed to add item into the blockchain {"mode": 0, "error": "error while trying to apply MPT changes: key not found", "height": 12962799, "nextIndex": 12962800} May 29 11:05:49 moros1 neogo-fschain-cn[1263406]: WARN queue: failed to add item into the blockchain {"mode": 0, "error": "error while trying to apply MPT changes: key not found", "height": 12962799, "nextIndex": 12962800} May 29 11:05:55 moros1 neogo-fschain-cn[1263406]: WARN queue: failed to add item into the blockchain {"mode": 0, "error": "error while trying to apply MPT changes: key not found", "height": 12962799, "nextIndex": 12962800} May 29 11:05:57 moros1 neogo-fschain-cn[1263406]: WARN queue: failed to add item into the blockchain {"mode": 0, "error": "error while trying to apply MPT changes: key not found", "height": 12962799, "nextIndex": 12962800} ``` ## Expected Behavior No error.