BUG: Changing .neptune location is broken with NeptuneLogger

[Wouter Zwerink]

Describe the bug

I have code that changes the location of .neptune according to the docs:
A clear and concise description of what the bug is.

After updating from an older neptune-client to the latest neptune (1.3.1), this broke completely.
Now, it crashes when trying to log something.

Reproduction

Create run using the setup shown above.
Create lightning NeptuneLogger with the run as argument.
Try to log something.
Will add minimal code example later.

Expected behavior

.neptune folder is moved, logging still works

Traceback

    logger = pl.loggers.NeptuneLogger(
  File "/home/micromamba/lib/python3.10/site-packages/pytorch_lightning/loggers/neptune.py", line 258, in __init__
    self._retrieve_run_data()
  File "/home/micromamba/lib/python3.10/site-packages/pytorch_lightning/loggers/neptune.py", line 273, in _retrieve_run_data
    root_obj.wait()
  File "/home/micromamba/lib/python3.10/site-packages/neptune/metadata_containers/metadata_container.py", line 433, in wait
    self._op_processor.wait()
  File "/home/micromamba/lib/python3.10/site-packages/neptune/internal/operation_processors/async_operation_processor.py", line 122, in wait
    raise NeptuneSynchronizationAlreadyStoppedException()
neptune.exceptions.NeptuneSynchronizationAlreadyStoppedException:

Environment

The output of pip list:

<output goes here>

The operating system you're using: Amazon linux
The output of python --version: 3.10.6

Additional context

Add any other context about the problem here.

[Prince Canuma]

Hi @wouterzwerink,

Thanks for reaching out!

I believe I managed to replicate it.

Does it also throw this error for you?

Unexpected error occurred in Neptune background thread: Killing Neptune asynchronous thread. All data is safe on disk and can be later synced manually using `neptune sync` command.
Exception in thread NeptuneAsyncOpProcessor:
Traceback (most recent call last):
  File "/Users/prince_canuma/opt/anaconda3/envs/neptune_test_env/lib/python3.9/threading.py", line 980, in _bootstrap_inner
    self.run()
  File "/Users/prince_canuma/opt/anaconda3/envs/neptune_test_env/lib/python3.9/site-packages/neptune/internal/operation_processors/async_operation_processor.py", line 230, in run
    super().run()
  File "/Users/prince_canuma/opt/anaconda3/envs/neptune_test_env/lib/python3.9/site-packages/neptune/internal/threading/daemon.py", line 53, in run
    self.work()
  File "/Users/prince_canuma/opt/anaconda3/envs/neptune_test_env/lib/python3.9/site-packages/neptune/internal/operation_processors/async_operation_processor.py", line 246, in work
    self.process_batch([element.obj for element in batch], batch[-1].ver)
  File "/Users/prince_canuma/opt/anaconda3/envs/neptune_test_env/lib/python3.9/site-packages/neptune/internal/threading/daemon.py", line 76, in wrapper
    result = func(self_, *args, **kwargs)
  File "/Users/prince_canuma/opt/anaconda3/envs/neptune_test_env/lib/python3.9/site-packages/neptune/internal/operation_processors/async_operation_processor.py", line 259, in process_batch
    processed_count, errors = self._processor._backend.execute_operations(
  File "/Users/prince_canuma/opt/anaconda3/envs/neptune_test_env/lib/python3.9/site-packages/neptune/internal/backends/hosted_neptune_backend.py", line 496, in execute_operations
    op.clean(operation_storage=operation_storage)
  File "/Users/prince_canuma/opt/anaconda3/envs/neptune_test_env/lib/python3.9/site-packages/neptune/internal/operation.py", line 214, in clean
    os.remove(self.get_absolute_path(operation_storage))
FileNotFoundError: [Errno 2] No such file or directory: '.neptune/async/run__99868482-f0d8-4f7f-bc04-fbdd3bcd3c88/exec-1688030278.345917-2023-06-29_11.17.58.345917/upload_path/source_code_diff-1688030278.474701-2023-06-29_11.17.58.474701.patch'

[Wouter Zwerink]

Yes! Interestingly, it seems to not happen when my git directory is clean, only when there are uncommitted changes.

[Prince Canuma]

Thanks for the update. We identified the cause of the issue as related to a feature we introduced recently that allows you to track uncommitted changes.

[Siddhant Sadangi]

Hello @wouterzwerink 👋

We just released neptune v1.4.0 that fixes this 🎉

Please let me know if this works for you 🙏

[Wouter Zwerink]

@SiddhantSadangi sounds great! Is the dataloader fork bug also fixed in 1.4? I don't see it in the changelog, but that bug is very impactful

[Siddhant Sadangi]

Not yet, unfortunately 😞
We are still discussing the best course of action there.