README 📖

Welcome to the final project repository for the "3D Computer Vision" course at NYCU! This project utilizes techniques from GaussianEditor to experiment with 3D scene editing.

A. Installation 🛠️

This project is tested on Ubuntu 22.04 with CUDA 11.8. Follow these steps to set up your environment:

# Set up the environment with conda
conda env create -f environment.yaml
conda activate 3dcv_final

# Download weights for Wonder3D and DPT
cd GaussianEditor
bash download_wonder3d.sh
mkdir -p .cache/dpt
gdown 1Jrh-bRnJEjyMCS7f-WsaFlccfPjJPPHI -O ./cache/dpt

B. Preparing Dataset 🎥

B-1. Download Videos

Access our dataset via Google Drive:

Video	Description	Time (s)
IMG_1991.MOV	Desk scene	~80
IMG_1992.MOV	Human scene	~30

Or download directly using:

gdown 11rxX2uWAcH4LKjtvq2zpUIfofpWybNYe
gdown 1v1g1wYLPhPfpK3pXnwT7FQRD1UAE-zbX

B-2. Videos to Frames

Extract frames from videos at the specified frame rates using ffmpeg:

# For the first video at 15 FPS
mkdir -p desk/input
ffmpeg -i IMG_1991.MOV -vf fps=15 desk/input/img%5d.png

# For the second video at 30 FPS
mkdir -p human/input
ffmpeg -i IMG_1992.MOV -vf fps=30 human/input/img%5d.png

C. Scene Reconstruction 🏗️

C-1. Obtaining Point Clouds

Ensure you have colmap and ImageMagick installed. Execute the following:

# Resize images in the 'desk' directory to conserve resources
python convert.py -s ../desk --resize

# Convert images in the 'human' directory
python Converter -s ../human

The following structure will be created in the desk and human folders:

📦 desk or human
┣ 📂 images
┃ ┗ 📜 *.png
┣ 📂 sparse/0
┃ ┣ 📜 cameras.bin
┃ ┣ 📜 images.bin
┃ ┗ 📜 points3D.bin
┗ 📂 (other auxiliary files)

Note that our convert.py is different from that in gaussian_splatting. We have modified the ImageMagick command to fit the latest version.

C-2. Constructing Gaussian Splatting

Run the Gaussian splatting training script:

cd gaussian_splatting
OAR_JOB_ID=desk python train.py -s desk -r 2  # Downsample to save GPU resources
OAR_JOB_ID=human python train.py -s human

Find the training results in output/desk and output/human.

D. Gaussian Editor 🖌️

Edit 3D scenes after generating point clouds:

cd GaussianEditor

# Desk scene
python webui.py --gs ../gaussian-splatting/output/desk/point_cloud/iteration_30000/point_cloud.ply --colmap_dir ../desk

# Human scene
python webui.py --gs ../gaussian-splatting/output/human/point_cloud/iteration_30000/point_cloud.ply --colmap_dir ../human

For detailed guidance on using the webUI, refer to this guide. Note that our webui.py is different from the official one, we support single camera tracing and negative depth scale.

D-1. Object Deleting

Use specific prompts to delete objects:

For the remote control, use the prompt remote control.
For the human, use man with orange t-shirt.

D-2. Object Adding

Add objects by generating and refining inpainting results:

Apple: Use a red apple on the table followed by make it a realistic apple.
Teddy Bear: Use a teddy bear on the ground followed by make it a realistic teddy bear.

D-3. Object Editing

Edit objects using Gaussian tracing:

Remote Control: Enable SAM and add SAM points to the object. Start tracing with the prompt Make it a Miro painting.
Human Clothes: Similarly, enable SAM for the clothes and use the prompt Turn his shirt out of leather.

Justin900429 / NYCU_3DCV