Giters

Cloud-CV / GSoC-Ideas

CloudCV GSoC Ideas

Home Page:https://gsoc.cloudcv.org

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Evaluating submission code in Docker containers

RishabhJain2018 opened this issue · comments

commented

Project Title: Evaluating submission code in Docker containers on EvalAI

Description:

The rise of reinforcement learning based problems or any problem which requires that an agent must interact with an environment introduces additional challenges for benchmarking. In contrast to the supervised learning setting where performance is measured by evaluating on a static test set, it is less straightforward to measure generalization performance of these agents in the context of the interactions with the environment. Evaluating these agents involves running the associated code on a collection of unseen environments that constitutes a hidden test set for such a scenario. The goal of this project is to set up a robust pipeline for uploading prediction code in the form of Docker containers (as opposed to test prediction file) that will be evaluated on remote machines and the results will be displayed on the leaderboard.

Deliverable:

  • Setup the pipeline for uploading the image on ECR registry
  • Add feature to display the image uploaded by a user on UI
  • Setup the pipeline to spawn a GPU based AWS machine automatically and start the evaluation of the uploaded docker image
  • Pipeline to send the evaluation results back to EvalAI and kill the spawned instance
  • Display the stderr, stdout, results corresponding to the submission on EvalAI
  • Setup a dummy environment based challenge on EvalAI
  • Optimize the submission evaluation by evaluating it on the cluster of machines
  • Write robust test cases for the code
  • Write Documentation for setting it up on server

The aforementioned features should be implemented by the end of the GSoC period. Moreover, we expect students to also have implemented improvements that they have proposed.

Mentors: Deepesh Pathak @fristonio , Rishabh Jain @RishabhJain2018 , Deshraj @deshraj

Skills Required: Docker, AWS-CLI, Django, DRF [Knowledge of Reinforcement Learning is not necessary.]

Skill Level: Difficult/Ambitious

Get started: Try to fix some issues in EvalAI (note that there are some issues labeled with GSOC-2019)

Tutorials:

a) Docker
b) AWS-CLI
c) Buildpack
d) jupyter-repo2docker

Important Links:

commented

@deshraj @RishabhJain2018 @fristonio
This sounds exciting! I also would love to work on this project. 😄

commented

Thanks for your interest @vkartik97! Looking forward to your GSOC Proposal :)

commented

I am also interested, but have knowledge of python and some basics of docker. I am a beginner but interested in this.

commented

Thanks for your interest in the project @adi0509
The project is mostly around docker so I would recommend you to familiarize yourself with AWS and docker ecosystem. Here are some links for the same.

Let me know if you have some specific doubts.

commented

Thanks for your interest in the project @adi0509
The project is mostly around docker so I would recommend you to familiarize yourself with AWS and docker ecosystem. Here are some links for the same.

Let me know if you have some specific doubts.

@fristonio Thanks for your suggestion. I will surely follow it.

commented

@deshraj @RishabhJain2018 @fristonio Thank you all for your kind help! I am interested in this topic and will look into it. Cheers!

commented

Hi @Helinia, Thanks for your interest. Looking forward to your proposal.

commented

Hi @RishabhJain2018 ,i am interested in this topic want to contribute, i do not fulfill all the prerequisites though :(

commented

Hi, @RishabhJain2018 I have good knowledge of Azure Enterprise Cloud and Python and Docker. I Believe I can contribute to this one.

commented

@fristonio @RishabhJain2018 Hi, this seems pretty interesting! Just a quick question. What level of comfort with Django would this demand? And on that note, for someone fairly proficient in Python, what would be an efficient way to pick Django up?

commented

This looks cool and I would love to work on this project.

commented

Hi Everyone, i liked the project and i'm very much excited to contribute in it.I have intermediate knowledge of python and beginner of docker. @RishabhJain2018 Please provide me with the link (if required) so i can catch up to the pace.Thank you

commented

Hi @yashbhutoria @cyphysan @krritik @Aaryaman09 Thanks for your interest and looking forward to your GSoC proposals.

commented

Just a quick question. What level of comfort with Django would this demand? And on that note, for someone fairly proficient in Python, what would be an efficient way to pick Django up?

@cyphysan We don't demand a student to be an expert but should have a basic understanding of the concepts and workflow in Django. In order to get started with Django, please refer to the documentation and its tutorial app.

commented

please provide me with the link (if required) so I can catch up to the pace.Thank you

@Aaryaman09 What link are you talking about? Please join the gitter channel in order to stay updated with the project.

commented

Hey @RishabhJain2018 i liked the project. I am intermediate in docker,aws and python. i would love to work on this project.

commented

Thanks for your interest @mani1soni looking forward to your proposal.

commented

Hey, @RishabhJain2018 I really liked the project. I did work on docker and python but intermediate in aws.
I would Love to contribute.

commented

Hey,@RishabhJain2018 I really liked the project. I have done mt last project on Django, python and this time I am working on aws but I am intermediate in docker. I would love to contribute.

commented

Hi, @RishabhJain2018 I am interested in this project. I have worked on CI/CD pipeline with Docker during my internship in RedHat and I am familiar with python. (But starter in Django). I have worked on some research in deep learning and I am happy to contribute.

commented

Hey @NagpalVarun, @9916103020, @theodore3131 Thanks for your interest in the project. I'm looking forward to your GSoC proposals. Please feel free to share with us so that we can review before the final submission to google.

commented

@RishabhJain2018 Why does the participant provide the docker container for evaluating the code? I mean, the submission of model file could have a predict method which could be used instead and the evaluation code docker container can be provided by the challenge host
(Also, the context of user in the description is related to challenge host or participant ?)

commented

@yashdusing please read section 6.2 of this paper https://arxiv.org/pdf/1902.03570.pdf. This will clear your doubt. :)

commented

So, the participant submits the same docker image with prediction code which he used for training (with little changes but the underlying API for simulation of environments stays the same. The only change is in the environment files)?
If so, I wrongly assumed that the challenge host specifies the docker image. But, the latter approach could actually abstract the details of the implementation from the participant and make the challenge host implement the docker script. This could be easier for participants because most of the participants need not know of docker (it's better to assume that the challenge host knows of docker than the participant). This way, the participant focuses on the model snapshot and the challenge host focuses on creating the docker image (Also, we have to trust the participant to not do something malicious in the former approach while in the latter approach all we are getting from the participant is the model file)
@deshraj @RishabhJain2018 can you please tell me what I’ve written is right or wrong

commented

Hi, Im intrested in this project, I know some basics of Aws(worked on it during my first internship at TCS) , I am familiar with python and kinda starting with Django

commented

I would love to join in this project. Could someone show me where can i start digging in because this is my first time to GSOC.

commented

@zduymz Firstly become familiar with the whole idea of the project using this link:https://arxiv.org/pdf/1902.03570.pdf
and then see the template of the GSoC 2019 link: https://github.com/Cloud-CV/GSoC-Ideas/wiki/GSOC-2019-Proposal-Template

commented

Hello Sir,
Divyansh Here @deshraj @RishabhJain2018 @Ram81 @fristonio Could someone please review my proposal? Only a few days are left so it would be really helpful if someone could give feedbacks.
Thanks in advance.

commented

Sir, would you please reply?

commented

I am starting to work on this. If anyone is already working on this, ping me.

commented

@RishabhJain2018
is the issue still open for GSOC'20 or we'll have to wait for some other issues?

commented

Hi @spectre1811 , Thanks a lot for your interest in the organization. We will be opening new issues for this year's GSoC.