CI/CD Github integration (mirror project)

[Seungpil Park]

오리지널 프로젝트와 포크 프로젝트 합병 방법.

https://stackoverflow.com/questions/3903817/pull-new-updates-from-original-github-repository-into-forked-github-repository

Nginx Reponse Html modify

https://github.com/openresty/lua-nginx-module

[Seungpil Park]

case1. clone origin and push to mirror.

git clone --bare https://github.com/SeungpilPark/mirror-test
cd mirror-test.git
git remote set-url --push origin http://gitlab.pas-mini.io/root/mirror-test
git config remote.origin.fetch 'refs/heads/*:refs/heads/*'


git --bare fetch -p origin
git push --mirror -f

• test. if remote(gitlab) updated? -> Ok with git push --mirror -f
• test. mirror push event trigger a new gitlab pipeline? -> OK with 'refs/heads/*:refs/heads/*'

case2. upstream origin and push to mirror.

git clone --bare http://gitlab.pas-mini.io/root/mirror-test
git remote add mirror https://github.com/SeungpilPark/mirror-test
git remote add origin http://gitlab.pas-mini.io/root/mirror-test


git pull
git revert 825d3fb7c6dda0528505c90f49ed17c966121f9d


git fetch mirror
git checkout master
git merge -X theirs mirror/master


git push origin master

• test. if remote(gitlab) updated? -> Not work!!
  • Problem. merge make a new gitlab commit, and trigger pipeline run as that commit sha.
  • Is there a scenario which needs a link between github commit and gitlab pipeline?

[Seungpil Park]

TODO list.

Menu:
template project:
  list apps
  select repo type

import project:
  insert git url
  select repo type

manage exist github project:
  list github repository



createApp:
  step1.
  how to manage your application?
    - 1. Create new in gitlab
    - 2. Create new in github
    - 3. Manage existing github project

  step2.
  if Create new in gitlab
    (icons)
    Blank project
    create from template
    import git url

  if Create new in github
    (icons)
    Blank project
    create from template
    import git url
     
  if Manage existing github project
    list github owned project.

  step3.
  if Manage new in gitlab
  if Manage new in github
    your project path
    gitlab:
      - /:user/name
      - /:org/name
    github:
      - /:user/name
      - /:org/name
  step4.
    app Name, resource, domain, etc... finish creation.

---

• import project 는 create project 시 git url 로 생성.

• Manage existing github project 는 git url 로 생성.

• Blank project 는 create project 시 empty template.

• create from template 는 create project 시 template fork.

• 생성 후 template 일 경우는 ci 실행. 아니면 대기.

• 어느 경우던 시작은 gitlab 에 프로젝트 생성됨.

---

Entity addition.

OauthUser: 
  metaData: github access token.
AppEntity:
  repositoryType: gitlab/github
  githubRepoId: int
AppCreate:
  repositoryType: gitlab/github
  importGitUrl: string
  namespace: string
  repositoryName: string
  gitlabProjectId: int
  githubRepoId: int

---

case: Manage existing github project

cloud server

1. check access level about repository. (pull need.) => get repository, and check permissions.admin is true
2. create gitlab project. (empty template, do not excutePipelineTrigger.)
3. send mirror service github project build event

---

case: Create new in github

cloud server

1. create gitlab project. (do not excutePipelineTrigger.)
2. send mirror service github project build event

---

case: Create new in gitlab

cloud server

1. create gitlab project.

---

mirror service

github project build event

• params: appEntity, userEntity,org name,repo name

if new github project,

1. create github project.
2. create github webhook.
3. push gitlab source codes to github. (trigger mirror ci)
4. trigger project ci

if manage existing github project,

1. create github webhook.
2. send mirror service sync repository event

push event

• It comes from github wehook.

1. send mirror service sync repository event

sync repository event

• params: branch, appEntity
• pre check. if app mirror is working, re register kafka.
• lock app mirror on start.
• unlock app mirror on finish.

1. check if cache exist.

2. if not, clone bare first.

3. if exist, download cache.

4. set env, fetch and push

5. upload to s3.

6. delete dir.

[Seungpil Park]

Error case

1. github access token life check.
2. github repository already exist check.

• in createNewGithubProject, cancel creation and request to user delete repo or change repo name manually.

3. github webhook already exist check.

• in manageExistGithubProject case, create hook if not exist.

Next test

1. manage exist github repository check. => ok
2. Save mirroring status on execute with pipeline id / remove mirroring status from pipeline hook
3. gitlab impersonationToken for push force

[Seungpil Park]

Mirroring project per each App

• 미러링 프로젝트를 각 프로젝트 별로 하나씩 생성하도록. (easy search). ok
• 프로젝트 삭제시 미러링 프로젝트도 함께 삭제. ok
• 생성 실패시 미러링 프로젝트 존재시 함께 삭제. ok
• 미러 프로젝트 CI 파이프라인 redis 저장, app log 저장, ui 노티. ok
• 미러 프로젝트 CI 파이프라인 rest api. ok