fusheng-ji / Coordinate_based_MLPs

Experiment of coordinate based MLPs Network with Pytorch-Lightning

Coordinate_based_MLPs

Intro

Experiments of coordinate-based MLPs based on Pytorch-lightning

graph LR
	1[x]-->3[P.E.]-->4[Linear layers + activation<br>256]-->5[Linear layers + activation<br>256]-->6[Linear layers + activation<br>256]-->7[Linear layers + Sgmoid<br>256]
	2[y]-->3[P.E.]
	7-->8[R]
	7-->9[G]
	7-->10[B]

Loading

Positional Encoding	Equation
Fourier feature mapping	$\gamma(v) = [..., \text{cos}(2\pi \sigma^{\frac{j}{m}}v), \text{sin}(2\pi \sigma^{\frac{j}{m}}v), ...]^T \ \text{for j in } [0, m-1]$
Fourier feature mapping (Gaussian distribution)	$\gamma(v) = [..., \text{cos}(2\pi Bv), \text{sin}(2\pi Bv), ...]^T \text{where each entry in B} \in \mathbb{R}^{m\times d} \text{is sampled from }\mathcal{N}(0, \sigma^2)$
Activation function	Equation
ReLU	$\text{max}(0,x)$
Siren	$\text{sin}(w_0\times W_x + b)$
Gaussian	$e^{\frac{-0.5x^2}{a^2}}$
Quadratic	$\frac{1}{1+(ax)^2}$
Multi Quadratic	$\frac{1}{\sqrt{1+(ax)^2}}$
Laplacian	$e^{\frac{-\lvert x \rvert}{a}}$
Super-Gaussian	$\left [ e^{\frac{-0.5x^2}{a^2}}\right ]^b$
ExpSin	$e^{-\text{sin}(ax)}$

How to run?

Data preparation

• build a directory "data/"
• make sure your own images put in "data/"
• Data used in My Experiment: Pluto image: NASA

Run single experiment

# raw MLPs with RuLU activation function without positional encoding
python train.py --arch=relu --use_pe=False --exp_name=raw_mlps_800*800_1024

Run all experiments at once

# run exp with defualt setting: Image_wh=800*800 batch_size=1024
bash exp.sh
# run exp with setting: Image_wh=800*800 batch_size=800*800
bash exp_640000.sh

Experiment logs

ReLU P.E.(Fourier Mapping)

Without Positional Encoding

	Image size 800*800, batch size 1024	Image size 800*800, batch size 800*800
MAX_PSNR	33.262	21.601

With Positional Encoding

	Image size 800*800, batch size 1024	Image size 800*800, batch size 800*800
MAX_PSNR	22.405	16.957

ReLU Fourier Mapping( Gaussian distribution)

2π

	Image size 800*800, batch size 1024	Image size 800*800, batch size 800*800
MAX_PSNR	31.523	24.383

20π

	Image size 800*800, batch size 1024	Image size 800*800, batch size 800*800
MAX_PSNR	30.004	25.222

200π

	Image size 800*800, batch size 1024	Image size 800*800, batch size 800*800
MAX_PSNR	8.585	7.146

Siren activation

As Siren dependent on quality of initialization, in this experiment I didn't initialize it specially, so the outputs are bad.

omega_0 = 30

	Image size 800*800, batch size 1024	Image size 800*800, batch size 800*800
MAX_PSNR	8.767	22.326

Gaussian activation

a=0.1 with P.E.

	Image size 800*800, batch size 1024	Image size 800*800, batch size 800*800
MAX_PSNR	8.36	8.61

a=0.1 without P.E.

	Image size 800*800, batch size 1024	Image size 800*800, batch size 800*800
MAX_PSNR	30.677	23.899

Super-Gaussian activation

a=0.1 b=2 with P.E.

	Image size 800*800, batch size 1024	Image size 800*800, batch size 800*800
MAX_PSNR	8.347	9.494

a=0.1 b=2 without P.E.

	Image size 800*800, batch size 1024	Image size 800*800, batch size 800*800
MAX_PSNR	30.644	24.183

ExpSin activation

a=0.1 with P.E.

	Image size 800*800, batch size 1024	Image size 800*800, batch size 800*800
MAX_PSNR	9.589	9.259

a=01 without P.E.

	Image size 800*800, batch size 1024	Image size 800*800, batch size 800*800
MAX_PSNR	24.268	9.204

Laplacian activation

a=0.1 with P.E.

	Image size 800*800, batch size 1024	Image size 800*800, batch size 800*800
MAX_PSNR	8.273	9.268

a=0.1 without P.E.

	Image size 800*800, batch size 1024	Image size 800*800, batch size 800*800
MAX_PSNR	31.283	24.006

Quadratic activation

a=10 with P.E.

	Image size 800*800, batch size 1024	Image size 800*800, batch size 800*800
MAX_PSNR	8.417	8.572

a=10 without P.E.

	Image size 800*800, batch size 1024	Image size 800*800, batch size 800*800
MAX_PSNR	31.222	23.861

Multi-Quadratic activation

a=20 with P.E.

	Image size 800*800, batch size 1024	Image size 800*800, batch size 800*800
MAX_PSNR	8.448	8.846

a=20 without P.E.

	Image size 800*800, batch size 1024	Image size 800*800, batch size 800*800
MAX_PSNR	30.355	23.944

Related papers

• Implicit Neural Representations with Periodic Activation Functions
• Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains
• Beyond Periodicity: Towards a Unifying Framework for Activations in Coordinate-MLPs
• Gaussian Activated Neural Radiance Fields for High Fidelity Reconstruction & Pose Estimation
• Spline Positional Encoding for Learning 3D Implicit Signed Distance Fields

Acknowledgment

Thanks for kwea123's wonderful live stream and his repo