"object already removed" is not handled properly for object removal attempts #16

New issue

Closed

opened 2025-12-28 18:15:45 +00:00 by sami · 0 comments

sami commented

2025-12-28 18:15:45 +00:00

Owner

Originally created by @roman-khimov on GitHub (Mar 13, 2025).

Originally assigned to: @evgeniiz321 on GitHub.

Current Behavior

Uploader fails to delete some objects and fails completely:

    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command 'NEOFS_CLI_PASSWORD=*** neofs-cli --rpc-endpoint st1.storage.fs.neo.org:8080 --wallet /home/runner/work/_actions/nspcc-dev/gh-push-to-neofs/v0.3.0/wallet.json object delete --cid HXSaMJXk2g8C14ht8HSi7BBaiYZ1HeWh2xnWPGQCg4H6 --oid '9VxgMVwJPnoD2pej9RBabHEtXHCoDiNDkhD23mCf7aWq'' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/runner/work/_actions/nspcc-dev/gh-push-to-neofs/v0.3.0/push-to-neofs.py", line 302, in <module>
    push_files_to_neofs(
  File "/home/runner/work/_actions/nspcc-dev/gh-push-to-neofs/v0.3.0/push-to-neofs.py", line 294, in push_files_to_neofs
    delete_objects(endpoint, wallet, password, cid, flat_existing_objects)
  File "/home/runner/work/_actions/nspcc-dev/gh-push-to-neofs/v0.3.0/push-to-neofs.py", line 162, in delete_objects
    neofs_cli_execute(cmd)
  File "/home/runner/work/_actions/nspcc-dev/gh-push-to-neofs/v0.3.0/helpers/neofs.py", line 39, in neofs_cli_execute
    raise Exception(
Exception: Command failed: NEOFS_CLI_PASSWORD=*** neofs-cli --rpc-endpoint st1.storage.fs.neo.org:8080 --wallet /home/runner/work/_actions/nspcc-dev/gh-push-to-neofs/v0.3.0/wallet.json object delete --cid HXSaMJXk2g8C14ht8HSi7BBaiYZ1HeWh2xnWPGQCg4H6 --oid '9VxgMVwJPnoD2pej9RBabHEtXHCoDiNDkhD23mCf7aWq'

Expected Behavior

But the object is already deleted, it's not a failure.

Possible Solution

In general this is not expected to happen, but in practice it can because of some concurrency issues, so this error should be ignored (IIRC CLI returns some specific error code for it).

Steps to Reproduce

https://github.com/nspcc-dev/neofs-node/actions/runs/13832276437/job/38706488529?pr=3226

Context

https://github.com/nspcc-dev/neofs-testcases/issues/1001

Regression

No.

Your Environment

Version of the product used: 0.3.0

Originally created by @roman-khimov on GitHub (Mar 13, 2025). Originally assigned to: @evgeniiz321 on GitHub. ## Current Behavior Uploader fails to delete some objects and fails completely: ``` raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command 'NEOFS_CLI_PASSWORD=*** neofs-cli --rpc-endpoint st1.storage.fs.neo.org:8080 --wallet /home/runner/work/_actions/nspcc-dev/gh-push-to-neofs/v0.3.0/wallet.json object delete --cid HXSaMJXk2g8C14ht8HSi7BBaiYZ1HeWh2xnWPGQCg4H6 --oid '9VxgMVwJPnoD2pej9RBabHEtXHCoDiNDkhD23mCf7aWq'' returned non-zero exit status 1. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/runner/work/_actions/nspcc-dev/gh-push-to-neofs/v0.3.0/push-to-neofs.py", line 302, in <module> push_files_to_neofs( File "/home/runner/work/_actions/nspcc-dev/gh-push-to-neofs/v0.3.0/push-to-neofs.py", line 294, in push_files_to_neofs delete_objects(endpoint, wallet, password, cid, flat_existing_objects) File "/home/runner/work/_actions/nspcc-dev/gh-push-to-neofs/v0.3.0/push-to-neofs.py", line 162, in delete_objects neofs_cli_execute(cmd) File "/home/runner/work/_actions/nspcc-dev/gh-push-to-neofs/v0.3.0/helpers/neofs.py", line 39, in neofs_cli_execute raise Exception( Exception: Command failed: NEOFS_CLI_PASSWORD=*** neofs-cli --rpc-endpoint st1.storage.fs.neo.org:8080 --wallet /home/runner/work/_actions/nspcc-dev/gh-push-to-neofs/v0.3.0/wallet.json object delete --cid HXSaMJXk2g8C14ht8HSi7BBaiYZ1HeWh2xnWPGQCg4H6 --oid '9VxgMVwJPnoD2pej9RBabHEtXHCoDiNDkhD23mCf7aWq' ``` ## Expected Behavior But the object is already deleted, it's not a failure. ## Possible Solution In general this is not expected to happen, but in practice it can because of some concurrency issues, so this error should be ignored (IIRC CLI returns some specific error code for it). ## Steps to Reproduce https://github.com/nspcc-dev/neofs-node/actions/runs/13832276437/job/38706488529?pr=3226 ## Context https://github.com/nspcc-dev/neofs-testcases/issues/1001 ## Regression No. ## Your Environment * Version of the product used: 0.3.0