High cpu RAM usage when loading the IP-Adapter in Colab

[amida47]

Attempted to utilize the IP-Adapter within a Colab environment running the ComfyUI interface. I'm employing the SDXL version, and everything appears to function smoothly. I'm able to successfully load the base SDXL and clip vision. However, upon attempting to load the IP-Adapter node, an immediate surge in CPU RAM usage is observed, resulting in the Colab instance shutting down automatically. Refer to the provided screenshot for a visual representation of the resource consumption pattern. Notably, the resource spikes coincide with the IP-Adapter loading attempt.

the GPU is a T4
Edit:
Forgot to add the colab I use, here it is https://colab.research.google.com/github/camenduru/sdxl-colab/blob/main/sdxl_v1.0_controlnet_comfyui_colab.ipynb
And I add this code to install the ComfyUI manager

%cd custom_nodes
![ ! -d ComfyUI-Manager ] && echo -= Initial setup ComfyUI-Manager =- && git clone https://github.com/ltdrdata/ComfyUI-Manager
%cd ComfyUI-Manager
!git pull

The IP-adapter is installed by the manager