-
install dependency
pip install gevent -
登陆 Github token 设置，创建个人 token
-
修改 main.py 参数：
- lang: 目标语言
- num_of_repo: 爬取库的数量
- download_files_when_generate_datasamples: 生成数据样本的同时下载对应代码文件
- only_download_changed_files: 只下载对应文件而非整个 repo
-
将个人 token 复制到 get_changes.py 第 12 行：
GITHUB_TOKEN = ‘’ -
运行程序
python main.py
- get_changes(lang, num_of_repo): 从目标语言下 star 数最多的库开始爬取，获得每个库的 commit sha，再获得每个 commit 下每条修改。修改会被记录在 ./changes/{lang} 下的jsonl文件中，文件名为：{user_name}_{proj_name}_{sha}_{parent_sha}.jsonl，每条数据结构如下：
{
"func_name": the name of function that this change belongs to,
"del_line_idx": the index of lines of code that deleted in the old version,
"add_line_idx": the index of lines of code that are newly added in the new version,
"del_line": the lines of code that deleted,
"add_line": the lines of code that added,
"file_path": the file path of this change belongs to
}- get_datasample(lang): 读取修改文件，根据sha下载对应历史版本代码库，并形成数据样本，结构如下：
{
'old_file_path': f'./repos/{user_name}_{proj_name}_{parent_sha}/{file_path_within_proj}',
'new_file_path': f'./repos/{user_name}_{proj_name}_{sha}/{file_path_within_proj}',
'changes': [
{
"func_name": the name of function that this change belongs to,
"del_line_idx": the index of lines of code that deleted in the old version,
"add_line_idx": the index of lines of code that are newly added in the new version,
"del_line": the lines of code that deleted,
"add_line": the lines of code that added
}
...
]
'commit_msg': commit message,
'pull_msg': pull message,
'html_url': html url from commit information
}- 数据保存在 ./dataset 中
- 当超出 GitHub 允许的一小时请求次数时，r.status_code == 403，此时程序 sleep 一小时