Wandb on ludwigai/ludwig-ray-gpu:latest + ray throws AttributeError: module 'pydantic.fields' has no attribute 'ModelField'

[Rahul V Ramesh]

Describe the bug
Wandb on ludwigai/ludwig-ray-gpu:latest + ray throws AttributeError: module 'pydantic.fields' has no attribute 'ModelField'

To Reproduce
Steps to reproduce the behavior:
1, Setup Ray Cluster With Docker Image (latest tag)
2, Submit Job

ray job submit \
    --address=http://<ip>:8265 \
    --runtime-env-json='{"conda": {"channels":["conda-forge"],"dependencies": ["wandb"]},"setup_timeout_seconds": 30,"working_dir": ".", "env_vars": {}}' \
    -- ludwig train --config llama-config.yml \
    --dataset "ludwig://alpaca" --wandb

Note : by default wandb triggers no module found issue.

Please provide code, yaml config file and a sample of data in order to entirely reproduce the issue.
Issues that are not reproducible will be ignored.

model_type: llm
base_model: TinyLlama/TinyLlama-1.1B-Chat-v1.0


adapter:
  type: lora

prompt:
  template: |
    ### Instruction:
    {instruction}

    ### Input:
    {input}

    ### Response:

input_features:
  - name: prompt
    type: text
    preprocessing:
      max_sequence_length: 256

output_features:
  - name: output
    type: text
    preprocessing:
      max_sequence_length: 256

trainer:
  type: finetune
  learning_rate: 0.0001
  batch_size: 1
  gradient_accumulation_steps: 16
  epochs: 1
  learning_rate_scheduler:
    warmup_fraction: 0.01

backend:
  type: ray
  cache_dir: s3://fine-tuning-testing/cache
  trainer:
    use_gpu: true
    num_workers: 2
    strategy:
      type: deepspeed
      zero_optimization:
        stage: 3
        offload_optimizer:
          device: cpu
          pin_memory: true
      bf16:
        enabled: true

Expected behavior
should install wandb and start pushing metrics

Error

tatus message: Unexpected error occurred: The actor died because of an error raised in its creation task, ray::_ray_internal_job_actor_raysubmit_LFQdYns3Zsi5495P:JobSupervisor.__init__() (pid=53445, ip=172.31.77.187, repr=<ray.dashboard.modules.job.job_manager.JobSupervisor object at 0x7faa547b5f70>)
  File "/tmp/ray/session_2024-03-23_07-37-24_381042_194/runtime_resources/conda/66324431b46322e8e8f74b368def6d448f095eb8/lib/python3.8/concurrent/futures/_base.py", line 437, in result
    return self.__get_result()
  File "/tmp/ray/session_2024-03-23_07-37-24_381042_194/runtime_resources/conda/66324431b46322e8e8f74b368def6d448f095eb8/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/tmp/ray/session_2024-03-23_07-37-24_381042_194/runtime_resources/conda/66324431b46322e8e8f74b368def6d448f095eb8/lib/python3.8/site-packages/ray/util/serialization_addons.py", line 58, in apply
    register_pydantic_serializer(serialization_context)
  File "/tmp/ray/session_2024-03-23_07-37-24_381042_194/runtime_resources/conda/66324431b46322e8e8f74b368def6d448f095eb8/lib/python3.8/site-packages/ray/util/serialization_addons.py", line 21, in register_pydantic_serializer
    pydantic.fields.ModelField,
AttributeError: module 'pydantic.fields' has no attribute 'ModelField'

Environment (please complete the following information):

• OS: Ubuntu
• Version 20
• Python 3.8.13

Additional context

• tried by downgrading pydantic, which triggers more module errors.