pandas9 / gan-latent-space-tsne-js

Robust way to explore GAN model latent space on web using three.js & t-SNE

GAN Latent Space Web Explorer w/ t-SNE Visualization

Robust way to explore GAN model latent space on web using t-SNE statistical method
GAN explorer example

This is example on how you can create latent space web explorer for any GAN model using js and python
Example is based on three.js library, t-SNE statistical method and vanilla js

t-SNE allows us to position images near other similar-looking images in the 2D coordinate space and three.js library to build a WebGL-powered visualization that can display tens of thousands of images in an interactive 3D environment

Inside examples folder there is celeba_gan_e30.h5 that has been trained on celeba dataset in case someone wants to try and build custom GAN explorer but doesn't have machines to train model

Run Locally

1. clone github project and set githubPages to false inside main.js
   
2. npm install
   command inside project root folder
   
3. node app.js
   serve app using node
   

Build Gan Explorer For Your Model

1. Clone the project and run calculate_latent_size.py and set variables inside file that will fit your model this is important for atlases and how your latent space is visualized with three.js
   python calculate_latent_size.py
   

2. Using your GAN model generate latent points and save each as image i.e
   

# example.py

def generate_images(examples, n):
	for i in range(n):
		pyplot.imsave(f'./faces/generated_face_{i}.png', examples[i, :, :])

model = load_model('celeba_gan_e30.h5')
n = 35301
latent_points = generate_latent_points(100, n)
X  = model.predict(latent_points)
X = (X + 1) / 2.0
generate_images(X, n)

3. Use classify_images.py to create vectorized representation of each image
   python classify_images.py 'faces/*'
   

4. Run create_tsne_projection.py to get t-SNE coordinates
   python create_tsne_projection.py
   inside file make sure to change latent_size and other variables same goes for other files below
   

5. Split latent space images into equal parts for atlas
   python split_latent_space.py
   

6. Generate atlases and serve the app
   python create_atlases.py
   
   by now you should have image_tsne_projections.json and images_..jpg those are your atlases next is to create new folder inside /src/assets/ and copy generated files inside, create new config config.js inside /src/ folder that will fit your model and add inside main.js - index.html option to have your config selectable inside web page
   
   serve app using node node app.js and visit localhost:3000 to explore your GAN latent space
   

Requirements

python3
node.js
imagemagick

To do

• Port to React.js
  
• Convert classify_images.py to tf2 and have script only create vectorized representations of each image
  
• Add 3D visualization
  
• Allow objects to be interactive for interpolation / zoom / vector arithmetic ... etc
  
• Dynamic atlas size
  
• Indication of load progress
  
• Guide through the visualization
  

Acknowledgements

machinelearningmastery.com/how-to-interpolate-and-perform-vector-arithmetic-with-faces-using-a-generative-adversarial-network/
medium.com/mlearning-ai/latent-space-representation-a-hands-on-tutorial-on-autoencoders-in-tensorflow
douglasduhaime.com/posts/visualizing-tsne-maps-with-three-js
github.com/YaleDHLab/pix-plot
artsexperiments.withgoogle.com/tsnemap
towardsdatascience.com/understanding-latent-space-in-machine-learning-de5a7c687d8d