7 IR node deployment test fails from time to time

sami commented

2025-12-28 17:22:47 +00:00

Owner

Originally created by @roman-khimov on GitHub (Mar 18, 2025).

Expected Behavior

Green.

Current Behavior

Red: https://rest.fs.neo.org/HXSaMJXk2g8C14ht8HSi7BBaiYZ1HeWh2xnWPGQCg4H6/2781-1742235841/index.html#suites/a4a271aa801d19e5d94000d236bb642b/4533b6b117edb6e2/
Some other failures happened also: https://rest.fs.neo.org/HXSaMJXk2g8C14ht8HSi7BBaiYZ1HeWh2xnWPGQCg4H6/2781-1742235841/index.html#suites/a4a271aa801d19e5d94000d236bb642b/4533b6b117edb6e2/history

Possible Solution

Unknown, but it doesn't seem to be a test issue, the test just waits and IR is stuck at INITIALIZING_NETWORK.

Your Environment

Version used: 0.45.1+

Originally created by @roman-khimov on GitHub (Mar 18, 2025). ## Expected Behavior Green. ## Current Behavior Red: https://rest.fs.neo.org/HXSaMJXk2g8C14ht8HSi7BBaiYZ1HeWh2xnWPGQCg4H6/2781-1742235841/index.html#suites/a4a271aa801d19e5d94000d236bb642b/4533b6b117edb6e2/ Some other failures happened also: https://rest.fs.neo.org/HXSaMJXk2g8C14ht8HSi7BBaiYZ1HeWh2xnWPGQCg4H6/2781-1742235841/index.html#suites/a4a271aa801d19e5d94000d236bb642b/4533b6b117edb6e2/history ## Possible Solution Unknown, but it doesn't seem to be a test issue, the test just waits and IR is stuck at INITIALIZING_NETWORK. ## Your Environment * Version used: 0.45.1+

sami

2025-12-28 17:22:47 +00:00

closed this issue
added the
bug

neofs-ir

S4

U3

I4
labels

sami commented

2025-12-28 17:22:48 +00:00

Author

Owner

@roman-khimov commented on GitHub (Mar 18, 2025):

There are no IR logs there, btw. @evgeniiz321?

@roman-khimov commented on GitHub (Mar 18, 2025): There are no IR logs there, btw. @evgeniiz321?

sami commented

2025-12-28 17:22:48 +00:00

Author

Owner

@roman-khimov commented on GitHub (Mar 27, 2025):

https://rest.fs.neo.org/HXSaMJXk2g8C14ht8HSi7BBaiYZ1HeWh2xnWPGQCg4H6/965-1743022436/index.html#suites/a4a271aa801d19e5d94000d236bb642b/d607dce14eba2615/

@roman-khimov commented on GitHub (Mar 27, 2025): https://rest.fs.neo.org/HXSaMJXk2g8C14ht8HSi7BBaiYZ1HeWh2xnWPGQCg4H6/965-1743022436/index.html#suites/a4a271aa801d19e5d94000d236bb642b/d607dce14eba2615/

sami commented

2025-12-28 17:22:48 +00:00

Author

Owner

@roman-khimov commented on GitHub (Apr 4, 2025):

https://rest.fs.neo.org/HXSaMJXk2g8C14ht8HSi7BBaiYZ1HeWh2xnWPGQCg4H6/1003-1743774939/index.html#suites/a4a271aa801d19e5d94000d236bb642b/8255fb6a13096e96/

@roman-khimov commented on GitHub (Apr 4, 2025): https://rest.fs.neo.org/HXSaMJXk2g8C14ht8HSi7BBaiYZ1HeWh2xnWPGQCg4H6/1003-1743774939/index.html#suites/a4a271aa801d19e5d94000d236bb642b/8255fb6a13096e96/

sami commented

2025-12-28 17:22:48 +00:00

Author

Owner

@roman-khimov commented on GitHub (Apr 4, 2025):

info deploy/deploy.go:627 distributing NEO to the Alphabet contracts...

info deploy/funds.go:428 have available NEO on the committee multi-sig account, going to transfer to the Alphabet contracts {"balance": "100000000"}
info deploy/funds.go:434 going to transfer NEO from the committee multi-sig account to the Alphabet contract {"contract": "b54cf49305f33e98cdfff32408e022a1764ff485", "amount": 14285715}
info deploy/funds.go:434 going to transfer NEO from the committee multi-sig account to the Alphabet contract {"contract": "910784cef90bf37cb042354bb8587dcd4a0ef2aa", "amount": 14285715}
info deploy/funds.go:434 going to transfer NEO from the committee multi-sig account to the Alphabet contract {"contract": "f1e0e2aad190c17320e1135549c698f58ef88d42", "amount": 14285714}
info deploy/funds.go:434 going to transfer NEO from the committee multi-sig account to the Alphabet contract {"contract": "04739e8354f952373d573365c859ac8b21bfdab7", "amount": 14285714}
info deploy/funds.go:434 going to transfer NEO from the committee multi-sig account to the Alphabet contract {"contract": "fdcd06749c38c24202bbcb18f9f16e764b4d468c", "amount": 14285714}
info deploy/funds.go:434 going to transfer NEO from the committee multi-sig account to the Alphabet contract {"contract": "fbfdc3f1fff59a1f1b995b532e5da0ddf61a0185", "amount": 14285714}
info deploy/funds.go:434 going to transfer NEO from the committee multi-sig account to the Alphabet contract {"contract": "f8bce3d3e5e0f9bdd5a56f55a7931d753bd33b28", "amount": 14285714}

debug core/blockchain.go:1619 done processing headers {"headerIndex": 95, "blockHeight": 94, "took": 0.000551542}
warn core/blockchain.go:1760 contract invocation failed {"tx": "481347c7046774a0354bf276ca8253313fc9fd47222e2f88dcb01d924cfffcc1", "block": 95, "error": "at instruction 90 (ASSERT): ASSERT failed"}
warn core/blockchain.go:1760 contract invocation failed {"tx": "2635940ac872a412d9c45a25bd69e80c104412c9fbe41be707f981e866870269", "block": 95, "error": "at instruction 90 (ASSERT): ASSERT failed"}
warn core/blockchain.go:1760 contract invocation failed {"tx": "51c310e4bcf9fc5214918da9828bb214ad3b2074348bb1fada7d4cabd880438e", "block": 95, "error": "at instruction 90 (ASSERT): ASSERT failed"}
warn core/blockchain.go:1760 contract invocation failed {"tx": "1a3ac3e29028bd360528d45bfb157c7813842a2cbee980b93475d1a81958f490", "block": 95, "error": "at instruction 90 (ASSERT): ASSERT failed"}
warn core/blockchain.go:1760 contract invocation failed {"tx": "765f6b4e81c603636c0fa3ac4329ce0c72387e81f10428d3b9301ef81e24b001", "block": 95, "error": "at instruction 90 (ASSERT): ASSERT failed"}
warn core/blockchain.go:1760 contract invocation failed {"tx": "5f28e70b7612769cc5f12fb3ed1a94e052dbf5660f9e7ab53c5703757256d5be", "block": 95, "error": "at instruction 6632 (THROW): unhandled exception: "not witnessed by admin""}
warn core/blockchain.go:1760 contract invocation failed {"tx": "345bc6892ad2cdec0458a56d81298a568b62f13e1e21ffaf39e247b0f9d8d820", "block": 95, "error": "at instruction 6632 (THROW): unhandled exception: "not witnessed by admin""}

@roman-khimov commented on GitHub (Apr 4, 2025): > info deploy/deploy.go:627 distributing NEO to the Alphabet contracts... > info deploy/funds.go:428 have available NEO on the committee multi-sig account, going to transfer to the Alphabet contracts {"balance": "100000000"} info deploy/funds.go:434 going to transfer NEO from the committee multi-sig account to the Alphabet contract {"contract": "b54cf49305f33e98cdfff32408e022a1764ff485", "amount": 14285715} info deploy/funds.go:434 going to transfer NEO from the committee multi-sig account to the Alphabet contract {"contract": "910784cef90bf37cb042354bb8587dcd4a0ef2aa", "amount": 14285715} info deploy/funds.go:434 going to transfer NEO from the committee multi-sig account to the Alphabet contract {"contract": "f1e0e2aad190c17320e1135549c698f58ef88d42", "amount": 14285714} info deploy/funds.go:434 going to transfer NEO from the committee multi-sig account to the Alphabet contract {"contract": "04739e8354f952373d573365c859ac8b21bfdab7", "amount": 14285714} info deploy/funds.go:434 going to transfer NEO from the committee multi-sig account to the Alphabet contract {"contract": "fdcd06749c38c24202bbcb18f9f16e764b4d468c", "amount": 14285714} info deploy/funds.go:434 going to transfer NEO from the committee multi-sig account to the Alphabet contract {"contract": "fbfdc3f1fff59a1f1b995b532e5da0ddf61a0185", "amount": 14285714} info deploy/funds.go:434 going to transfer NEO from the committee multi-sig account to the Alphabet contract {"contract": "f8bce3d3e5e0f9bdd5a56f55a7931d753bd33b28", "amount": 14285714} > debug core/blockchain.go:1619 done processing headers {"headerIndex": 95, "blockHeight": 94, "took": 0.000551542} warn core/blockchain.go:1760 contract invocation failed {"tx": "481347c7046774a0354bf276ca8253313fc9fd47222e2f88dcb01d924cfffcc1", "block": 95, "error": "at instruction 90 (ASSERT): ASSERT failed"} warn core/blockchain.go:1760 contract invocation failed {"tx": "2635940ac872a412d9c45a25bd69e80c104412c9fbe41be707f981e866870269", "block": 95, "error": "at instruction 90 (ASSERT): ASSERT failed"} warn core/blockchain.go:1760 contract invocation failed {"tx": "51c310e4bcf9fc5214918da9828bb214ad3b2074348bb1fada7d4cabd880438e", "block": 95, "error": "at instruction 90 (ASSERT): ASSERT failed"} warn core/blockchain.go:1760 contract invocation failed {"tx": "1a3ac3e29028bd360528d45bfb157c7813842a2cbee980b93475d1a81958f490", "block": 95, "error": "at instruction 90 (ASSERT): ASSERT failed"} warn core/blockchain.go:1760 contract invocation failed {"tx": "765f6b4e81c603636c0fa3ac4329ce0c72387e81f10428d3b9301ef81e24b001", "block": 95, "error": "at instruction 90 (ASSERT): ASSERT failed"} warn core/blockchain.go:1760 contract invocation failed {"tx": "5f28e70b7612769cc5f12fb3ed1a94e052dbf5660f9e7ab53c5703757256d5be", "block": 95, "error": "at instruction 6632 (THROW): unhandled exception: \"not witnessed by admin\""} warn core/blockchain.go:1760 contract invocation failed {"tx": "345bc6892ad2cdec0458a56d81298a568b62f13e1e21ffaf39e247b0f9d8d820", "block": 95, "error": "at instruction 6632 (THROW): unhandled exception: \"not witnessed by admin\""}

sami commented

2025-12-28 17:22:48 +00:00

Author

Owner

@roman-khimov commented on GitHub (Apr 30, 2025):

https://rest.fs.neo.org/HXSaMJXk2g8C14ht8HSi7BBaiYZ1HeWh2xnWPGQCg4H6/1090-1745995281/index.html#suites/a4a271aa801d19e5d94000d236bb642b/294d20e1839c4a34/

@roman-khimov commented on GitHub (Apr 30, 2025): https://rest.fs.neo.org/HXSaMJXk2g8C14ht8HSi7BBaiYZ1HeWh2xnWPGQCg4H6/1090-1745995281/index.html#suites/a4a271aa801d19e5d94000d236bb642b/294d20e1839c4a34/

sami commented

2025-12-28 17:22:48 +00:00

Author

Owner

@roman-khimov commented on GitHub (Jul 25, 2025):

Looks like it happens more frequently with dynamic block time:
https://rest.fs.neo.org/HXSaMJXk2g8C14ht8HSi7BBaiYZ1HeWh2xnWPGQCg4H6/1268-1753393473/index.html#suites/a4a271aa801d19e5d94000d236bb642b/e7c1bd6e388ad8a1/

Some nodes pass through deployment fine, but some end up with

2025-07-24T20:48:56.565Z error deploy/funds.go:418 no NEO on the committee multi-sig account, nothing to transfer, skip
2025-07-24T20:48:56.565Z info deploy/deploy.go:621 NEO distribution to the Alphabet contracts successfully completed
2025-07-24T20:48:56.565Z debug rpcsrv/server.go:617 processing local rpc request {"method": "unsubscribe", "params": "[0]"}
2025-07-24T20:48:56.566Z info deploy/notary.go:937 Notary balance tracker stopped by context {"error": "context canceled"}
2025-07-24T20:48:56.566Z info deploy/notary.go:1056 stop listening to committee notary requests (context is done) {"error": "context canceled"}
2025-07-24T20:48:56.566Z debug network/server.go:1503 got msg {"addr": "127.0.0.1:58542", "type": "CMDInv"}
2025-07-24T20:48:56.566Z debug network/server.go:1503 got msg {"addr": "127.0.0.1:58542", "type": "CMDInv"}
2025-07-24T20:48:56.566Z debug network/server.go:1503 got msg {"addr": "127.0.0.1:58542", "type": "CMDP2PNotaryRequest"}

And then do nothing (except for consensus).

It looks like the process gets stuck in server.fsChainClient.InitFSChainScope() call which is likely caused by inappropriate subscription management.

@roman-khimov commented on GitHub (Jul 25, 2025): Looks like it happens more frequently with dynamic block time: https://rest.fs.neo.org/HXSaMJXk2g8C14ht8HSi7BBaiYZ1HeWh2xnWPGQCg4H6/1268-1753393473/index.html#suites/a4a271aa801d19e5d94000d236bb642b/e7c1bd6e388ad8a1/ Some nodes pass through deployment fine, but some end up with > 2025-07-24T20:48:56.565Z error deploy/funds.go:418 no NEO on the committee multi-sig account, nothing to transfer, skip 2025-07-24T20:48:56.565Z info deploy/deploy.go:621 NEO distribution to the Alphabet contracts successfully completed 2025-07-24T20:48:56.565Z debug rpcsrv/server.go:617 processing local rpc request {"method": "unsubscribe", "params": "[0]"} 2025-07-24T20:48:56.566Z info deploy/notary.go:937 Notary balance tracker stopped by context {"error": "context canceled"} 2025-07-24T20:48:56.566Z info deploy/notary.go:1056 stop listening to committee notary requests (context is done) {"error": "context canceled"} 2025-07-24T20:48:56.566Z debug network/server.go:1503 got msg {"addr": "127.0.0.1:58542", "type": "CMDInv"} 2025-07-24T20:48:56.566Z debug network/server.go:1503 got msg {"addr": "127.0.0.1:58542", "type": "CMDInv"} 2025-07-24T20:48:56.566Z debug network/server.go:1503 got msg {"addr": "127.0.0.1:58542", "type": "CMDP2PNotaryRequest"} And then do nothing (except for consensus). It looks like the process gets stuck in `server.fsChainClient.InitFSChainScope()` call which is likely caused by inappropriate subscription management.

sami commented

2025-12-28 17:22:48 +00:00

Author

Owner

@roman-khimov commented on GitHub (Jul 25, 2025):

Looks like unsubscription needs to be done at deploy-level code, only it can synchronize all the internal channel reads to properly drain their channels.

@roman-khimov commented on GitHub (Jul 25, 2025): Looks like unsubscription needs to be done at deploy-level code, only it can synchronize all the internal channel reads to properly drain their channels.

sami commented

2025-12-28 17:22:48 +00:00

Author

Owner

@roman-khimov commented on GitHub (Jul 29, 2025):

A different one: https://rest.fs.neo.org/HXSaMJXk2g8C14ht8HSi7BBaiYZ1HeWh2xnWPGQCg4H6/1277-1753795110/index.html#suites/a4a271aa801d19e5d94000d236bb642b/8bef2706dffed413/

@roman-khimov commented on GitHub (Jul 29, 2025): A different one: https://rest.fs.neo.org/HXSaMJXk2g8C14ht8HSi7BBaiYZ1HeWh2xnWPGQCg4H6/1277-1753795110/index.html#suites/a4a271aa801d19e5d94000d236bb642b/8bef2706dffed413/

Rows
Columns

7 IR node deployment test fails from time to time #1382

Expected Behavior

Current Behavior

Possible Solution

Your Environment