nitthilan / CPISADGAN

Reproducing Controllable Person Image Synthesis with Attribute-Decomposed GAN result

Reproduction of Controllable Person Image Synthesis with Attribute-Decomposed GAN

@inproceedings{men2020controllable,
  title={Controllable Person Image Synthesis with Attribute-Decomposed GAN},
  author={Men, Yifang and Mao, Yiming and Jiang, Yuning and Ma, Wei-Ying and Lian, Zhouhui},
  booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
  year={2020}
}

Repo

Reusing code for generation of input from the following repositories:

• Controllable Person Image Synthesis with Attribute-Decomposed GAN paper, code - not updated
• Part segmentation is done using Pyramid Scene Parsing Network - paper, pytorch, dataset
• Pose estimation using Pose-Transfer - paper, pytorch

• StyleGAN2 - paper,tensorflow,pytorch

• StyleGAN - paper

• Image Harmonization - DoveNet: Deep Image Harmonization via Domain Verification - paper, pytorch, Image Colorization - paper list

• Dataset: DeepFashion

• Intrinsic Image Popularity Assessment - paper

• SPADE - Semantic Image Synthesis with Spatially-Adaptive Normalization - pytorch,project page, paper

• VideoPose Estimation - code, awesome papers

• Adversarial Latent Autoencoders - paper, pytorch,

• Depth Estimation - Consistent Depth project

• Animating using reinforcement learning - project

• Principles of animation - youtube

• Lighting and illumination

  • DeepLight youtube, paper, project
  • HDRI explained youtube details
  • Animation - Cinematic lighting

• Understanding textures maps: albedo - youtube

• Correcting compositing - paper, project, youtube

Other references:

• A Style-Based Generator Architecture for Generative Adversarial Networks - paper

• First Order Motion Model for Image Animation - code, paper

• Basic GAN pytorch implementation - code

• VFX experiments blog

• Rigging - RigNet paper, youtube, project, code-not available, dattaset - Predicting Animation Skeletons for 3D Articulated Models via Volumetric Nets paper, supplement, code

• what is skin weights in animation - NeuroSkinning - paper

  • autorigging deeplearning

• how is skeleton mapped to required motion

• 3d walk cycle - how is skeleton mapped to required motion

• High resolution 3d human digitization - code, project

Companies: mixamo, https://www.gleechi.com/

catastrophic forgetting deep learning

• making learning rate small for already learnt network
• Having old and new networks. The old weights do not change or get update less
• Measuring Catastrophic Forgetting in Neural Networks - file:///Users/kannappanjayakodinitthilan/Downloads/16410-77483-1-PB.pdf
• Continual Learning and Catastrophic Forgetting https://www.cs.uic.edu/~liub/lifelong-learning/continual-learning.pdf
• Continual lifelong learning with neural networks : A review - https://reader.elsevier.com/reader/sd/pii/S0893608019300231?token=0A2F8D43B5B7E04B9C4B16B24769AA0EAC2D14F7C954CF29B6DC4E4A0F6D0851031F84CB2D8B0AC6F5FB68E1C58CE9DD
• CRITICAL LEARNING PERIODS IN DEEP NETWORKS - https://openreview.net/pdf?id=BkeStsCcKQ
• Image Difficulty Curriculum for Generative Adversarial Networks (CuGAN) - https://openaccess.thecvf.com/content_WACV_2020/papers/Soviany_Image_Difficulty_Curriculum_for_Generative_Adversarial_Networks_CuGAN_WACV_2020_paper.pdf

People crowd simulation:

• generation of shadows
• steering people
• interaction between people like not overlapping with each other
• ability to walk jump fall etc..
• ability to choose gender, age, cloth color etc
• Reference1

Animation:

• https://www.nukeygara.com/

List of ideas:

• Can we have a discriminator which learn progressively i.e. initially uses basic input samples to restrict the combinatorial search space. Then gradually feed finer and finer variations to adapt easily
• Can we use GAN based object generattion like small coffettis or lot of people kind of objects and encode them separately such tthat we can store the number and object and recreate them on the other side for decoders?
• Crown simulatitons for car orientation detection deep learning, car videos dataset, trees dataset
• pytorch3d to render outputs with known input and then learning a generator to approximate the output
• SPADE based image painting - bg generation and then add humans

List of ideas 3D sculpting:

• 3d mesh models pixar from the movies extracting the object from differnt views - https://www.renderhub.com/lykomodels/left-shark, search google 3d mesh models