增量预训练后,效果变得特别差
listwebit opened this issue · comments
Reminder
- I have searched the Github Discussion and issues and have not found anything similar to this.
Environment
- OS:
- Python:
- PyTorch:
- CUDA:
老师好,我用你们推荐的,https://github.com/hiyouga/LLaMA-Factory 做了增量预训练,但是效果特别差,能帮忙分析一下原因吗?
我用的Yi-34B 基础语言模型做的增量预训练,用了5G的医疗领域数据,5G wudao中的百科数据,用了3台机器做了pt预训练,然后用opencompass 分别对数据集ceval和cmmlu做了PPL测试,结果原始的Yi-34B模型效果很好,预训练之后的效果非常差劲,能帮忙分析一下原因吗?如果说是因为灾难遗忘导致的,但是掉点也不至于这么大吧,而且其中和医疗相关的测评,ceval-basic_medicine ,ceval-clinical_medicine也都掉点很多
这是LLaMA-Factory的启动参加配置:
deepspeed --hostfile=./hostfile --master_port=9901 src/train_bash.py
--deepspeed ./ds_config.json
--stage pt
--do_train
--model_name_or_path ../Yi-34B
--dataset input_test
--finetuning_type full
--lora_target q_proj,v_proj
--output_dir Yi-34B_output
--overwrite_cache
--per_device_train_batch_size 4
--gradient_accumulation_steps 4
--lr_scheduler_type cosine
--logging_steps 5
--save_steps 300
--learning_rate 5e-5
--num_train_epochs 1.0
--plot_loss
--bf16
这是YI-34B 原始的结果
dataset version metric mode yi-34b-hf
---------------------------------------------- --------- -------- ------ -----------
cmmlu-agronomy 18e759 accuracy ppl 79.88
cmmlu-anatomy e2b12f accuracy ppl 91.89
cmmlu-ancient_chinese 7faca5 accuracy ppl 52.44
cmmlu-arts cc4476 accuracy ppl 96.25
cmmlu-astronomy b81d60 accuracy ppl 61.21
cmmlu-business_ethics 5a294c accuracy ppl 73.68
cmmlu-chinese_civil_service_exam d2f770 accuracy ppl 85.62
cmmlu-chinese_driving_rule 0e8a93 accuracy ppl 99.24
cmmlu-chinese_food_culture 8e2055 accuracy ppl 84.56
cmmlu-chinese_foreign_policy 484713 accuracy ppl 85.98
cmmlu-chinese_history b336e0 accuracy ppl 94.74
cmmlu-chinese_literature a9a252 accuracy ppl 78.43
cmmlu-chinese_teacher_qualification eacc26 accuracy ppl 93.85
cmmlu-clinical_knowledge 9993e7 accuracy ppl 85.23
cmmlu-college_actuarial_science 1f3eb3 accuracy ppl 40.57
cmmlu-college_education 7dbe6b accuracy ppl 97.2
cmmlu-college_engineering_hydrology 44f211 accuracy ppl 84.91
cmmlu-college_law 524735 accuracy ppl 86.11
cmmlu-college_mathematics b94953 accuracy ppl 35.24
cmmlu-college_medical_statistics 96aa86 accuracy ppl 76.42
cmmlu-college_medicine 5d2d59 accuracy ppl 91.21
cmmlu-computer_science ca1243 accuracy ppl 89.22
cmmlu-computer_security c21394 accuracy ppl 94.74
cmmlu-conceptual_physics 6dffab accuracy ppl 90.48
cmmlu-construction_project_management 4dd438 accuracy ppl 76.26
cmmlu-economics 2a5014 accuracy ppl 81.76
cmmlu-education 8fb812 accuracy ppl 82.82
cmmlu-electrical_engineering c692a4 accuracy ppl 84.3
cmmlu-elementary_chinese ecace0 accuracy ppl 83.33
cmmlu-elementary_commonsense f0a3cc accuracy ppl 86.87
cmmlu-elementary_information_and_technology edaf57 accuracy ppl 93.28
cmmlu-elementary_mathematics 099f4c accuracy ppl 56.09
cmmlu-ethnology 201b78 accuracy ppl 90.37
cmmlu-food_science ff494e accuracy ppl 79.72
cmmlu-genetics aa522b accuracy ppl 71.02
cmmlu-global_facts 0e84cf accuracy ppl 92.62
cmmlu-high_school_biology 4114ed accuracy ppl 88.76
cmmlu-high_school_chemistry 540bcf accuracy ppl 78.79
cmmlu-high_school_geography 70e849 accuracy ppl 88.98
cmmlu-high_school_mathematics 04285b accuracy ppl 52.44
cmmlu-high_school_physics d09967 accuracy ppl 72.73
cmmlu-high_school_politics d047b7 accuracy ppl 86.71
cmmlu-human_sexuality cc75c8 accuracy ppl 79.37
cmmlu-international_law f420cb accuracy ppl 74.59
cmmlu-journalism 5ef649 accuracy ppl 80.23
cmmlu-jurisprudence 6a38fc accuracy ppl 88.81
cmmlu-legal_and_moral_basis 6c3e30 accuracy ppl 97.66
cmmlu-logical 56a1ca accuracy ppl 74.8
cmmlu-machine_learning f26e28 accuracy ppl 80.33
cmmlu-management ed634c accuracy ppl 94.29
cmmlu-marketing acb874 accuracy ppl 90.56
cmmlu-marxist_theory 1fd3ba accuracy ppl 96.83
cmmlu-modern_chinese 805e5f accuracy ppl 66.38
cmmlu-nutrition 49ad23 accuracy ppl 87.59
cmmlu-philosophy 71f30b accuracy ppl 88.57
cmmlu-professional_accounting f1c72e accuracy ppl 90.86
cmmlu-professional_law 2651ae accuracy ppl 75.36
cmmlu-professional_medicine 0a6df4 accuracy ppl 87.77
cmmlu-professional_psychology d689f4 accuracy ppl 90.09
cmmlu-public_relations 1091db accuracy ppl 78.16
cmmlu-security_study 69d6ff accuracy ppl 92.59
cmmlu-sociology d58f6c accuracy ppl 81.86
cmmlu-sports_science e669b1 accuracy ppl 84.24
cmmlu-traditional_chinese_medicine 60cd30 accuracy ppl 86.49
cmmlu-virology 4ace32 accuracy ppl 89.94
cmmlu-world_history b012c4 accuracy ppl 93.17
cmmlu-world_religions c59782 accuracy ppl 88.12
ceval-computer_network 9b9417 accuracy ppl 73.68
ceval-operating_system b2b8cf accuracy ppl 89.47
ceval-computer_architecture 1bd275 accuracy ppl 85.71
ceval-college_programming 2d0833 accuracy ppl 75.68
ceval-college_physics fb7e04 accuracy ppl 68.42
ceval-college_chemistry 916b7d accuracy ppl 79.17
ceval-advanced_mathematics 5cad2a accuracy ppl 42.11
ceval-probability_and_statistics a6b30e accuracy ppl 33.33
ceval-discrete_mathematics 68be68 accuracy ppl 25
ceval-electrical_engineer 056c2e accuracy ppl 59.46
ceval-metrology_engineer 4a757a accuracy ppl 83.33
ceval-high_school_mathematics a8ed21 accuracy ppl 44.44
ceval-high_school_physics e1fc86 accuracy ppl 84.21
ceval-high_school_chemistry 9021c6 accuracy ppl 84.21
ceval-high_school_biology c7f5a1 accuracy ppl 89.47
ceval-middle_school_mathematics 213989 accuracy ppl 52.63
ceval-middle_school_biology ce0420 accuracy ppl 90.48
ceval-middle_school_physics 78f3af accuracy ppl 94.74
ceval-middle_school_chemistry d071d2 accuracy ppl 100
ceval-veterinary_medicine cd3a07 accuracy ppl 82.61
ceval-college_economics a35346 accuracy ppl 65.45
ceval-business_administration 69dd6a accuracy ppl 87.88
ceval-marxism 283ce0 accuracy ppl 94.74
ceval-mao_zedong_thought f38cd1 accuracy ppl 100
ceval-education_science fbd65c accuracy ppl 89.66
ceval-teacher_qualification c77f1f accuracy ppl 93.18
ceval-high_school_politics bbac37 accuracy ppl 94.74
ceval-high_school_geography 730a30 accuracy ppl 89.47
ceval-middle_school_politics 15b2d7 accuracy ppl 90.48
ceval-middle_school_geography b00167 accuracy ppl 91.67
ceval-modern_chinese_history 5a04cd accuracy ppl 91.3
ceval-ideological_and_moral_cultivation 0829ff accuracy ppl 100
ceval-logic c9c394 accuracy ppl 95.45
ceval-law cbd3c5 accuracy ppl 79.17
ceval-chinese_language_and_literature 716ab3 accuracy ppl 65.22
ceval-art_studies 476114 accuracy ppl 78.79
ceval-professional_tour_guide 70f30f accuracy ppl 96.55
ceval-legal_professional f19cf5 accuracy ppl 73.91
ceval-high_school_chinese 931614 accuracy ppl 78.95
ceval-high_school_history 4d6364 accuracy ppl 100
ceval-middle_school_history 7f6356 accuracy ppl 100
ceval-civil_servant a5dcb8 accuracy ppl 76.6
ceval-sports_science 192553 accuracy ppl 89.47
ceval-plant_protection f7ff86 accuracy ppl 90.91
ceval-basic_medicine a95a09 accuracy ppl 94.74
ceval-clinical_medicine 664b54 accuracy ppl 90.91
ceval-urban_and_rural_planner fdae6f accuracy ppl 82.61
ceval-accountant d810a1 accuracy ppl 85.71
ceval-fire_engineer bb924d accuracy ppl 80.65
ceval-environmental_impact_assessment_engineer d59200 accuracy ppl 80.65
ceval-tax_accountant 9e16f2 accuracy ppl 87.76
ceval-physician 0e90d5 accuracy ppl 89.8
这是增量预训练的结果:
dataset version metric mode yi-34b-hf
---------------------------------------------- --------- -------- ------ -----------
cmmlu-agronomy 18e759 accuracy ppl 68.64
cmmlu-anatomy e2b12f accuracy ppl 63.51
cmmlu-ancient_chinese 7faca5 accuracy ppl 37.8
cmmlu-arts cc4476 accuracy ppl 84.38
cmmlu-astronomy b81d60 accuracy ppl 49.09
cmmlu-business_ethics 5a294c accuracy ppl 69.86
cmmlu-chinese_civil_service_exam d2f770 accuracy ppl 59.38
cmmlu-chinese_driving_rule 0e8a93 accuracy ppl 87.02
cmmlu-chinese_food_culture 8e2055 accuracy ppl 66.18
cmmlu-chinese_foreign_policy 484713 accuracy ppl 71.96
cmmlu-chinese_history b336e0 accuracy ppl 77.09
cmmlu-chinese_literature a9a252 accuracy ppl 58.33
cmmlu-chinese_teacher_qualification eacc26 accuracy ppl 83.8
cmmlu-clinical_knowledge 9993e7 accuracy ppl 62.45
cmmlu-college_actuarial_science 1f3eb3 accuracy ppl 26.42
cmmlu-college_education 7dbe6b accuracy ppl 78.5
cmmlu-college_engineering_hydrology 44f211 accuracy ppl 61.32
cmmlu-college_law 524735 accuracy ppl 57.41
cmmlu-college_mathematics b94953 accuracy ppl 30.48
cmmlu-college_medical_statistics 96aa86 accuracy ppl 61.32
cmmlu-college_medicine 5d2d59 accuracy ppl 69.23
cmmlu-computer_science ca1243 accuracy ppl 72.06
cmmlu-computer_security c21394 accuracy ppl 84.8
cmmlu-conceptual_physics 6dffab accuracy ppl 70.07
cmmlu-construction_project_management 4dd438 accuracy ppl 55.4
cmmlu-economics 2a5014 accuracy ppl 70.44
cmmlu-education 8fb812 accuracy ppl 76.07
cmmlu-electrical_engineering c692a4 accuracy ppl 70.35
cmmlu-elementary_chinese ecace0 accuracy ppl 60.71
cmmlu-elementary_commonsense f0a3cc accuracy ppl 69.19
cmmlu-elementary_information_and_technology edaf57 accuracy ppl 89.08
cmmlu-elementary_mathematics 099f4c accuracy ppl 40.43
cmmlu-ethnology 201b78 accuracy ppl 68.89
cmmlu-food_science ff494e accuracy ppl 65.03
cmmlu-genetics aa522b accuracy ppl 57.95
cmmlu-global_facts 0e84cf accuracy ppl 73.83
cmmlu-high_school_biology 4114ed accuracy ppl 60.36
cmmlu-high_school_chemistry 540bcf accuracy ppl 52.27
cmmlu-high_school_geography 70e849 accuracy ppl 79.66
cmmlu-high_school_mathematics 04285b accuracy ppl 38.41
cmmlu-high_school_physics d09967 accuracy ppl 45.45
cmmlu-high_school_politics d047b7 accuracy ppl 66.43
cmmlu-human_sexuality cc75c8 accuracy ppl 68.25
cmmlu-international_law f420cb accuracy ppl 53.51
cmmlu-journalism 5ef649 accuracy ppl 69.19
cmmlu-jurisprudence 6a38fc accuracy ppl 72.26
cmmlu-legal_and_moral_basis 6c3e30 accuracy ppl 95.33
cmmlu-logical 56a1ca accuracy ppl 69.92
cmmlu-machine_learning f26e28 accuracy ppl 64.75
cmmlu-management ed634c accuracy ppl 83.81
cmmlu-marketing acb874 accuracy ppl 83.89
cmmlu-marxist_theory 1fd3ba accuracy ppl 85.19
cmmlu-modern_chinese 805e5f accuracy ppl 50
cmmlu-nutrition 49ad23 accuracy ppl 69.66
cmmlu-philosophy 71f30b accuracy ppl 70.48
cmmlu-professional_accounting f1c72e accuracy ppl 76
cmmlu-professional_law 2651ae accuracy ppl 52.13
cmmlu-professional_medicine 0a6df4 accuracy ppl 66.49
cmmlu-professional_psychology d689f4 accuracy ppl 80.6
cmmlu-public_relations 1091db accuracy ppl 67.82
cmmlu-security_study 69d6ff accuracy ppl 77.04
cmmlu-sociology d58f6c accuracy ppl 69.03
cmmlu-sports_science e669b1 accuracy ppl 71.52
cmmlu-traditional_chinese_medicine 60cd30 accuracy ppl 63.24
cmmlu-virology 4ace32 accuracy ppl 76.33
cmmlu-world_history b012c4 accuracy ppl 76.4
cmmlu-world_religions c59782 accuracy ppl 76.88
ceval-computer_network 9b9417 accuracy ppl 57.89
ceval-operating_system b2b8cf accuracy ppl 73.68
ceval-computer_architecture 1bd275 accuracy ppl 71.43
ceval-college_programming 2d0833 accuracy ppl 78.38
ceval-college_physics fb7e04 accuracy ppl 31.58
ceval-college_chemistry 916b7d accuracy ppl 66.67
ceval-advanced_mathematics 5cad2a accuracy ppl 52.63
ceval-probability_and_statistics a6b30e accuracy ppl 16.67
ceval-discrete_mathematics 68be68 accuracy ppl 31.25
ceval-electrical_engineer 056c2e accuracy ppl 48.65
ceval-metrology_engineer 4a757a accuracy ppl 79.17
ceval-high_school_mathematics a8ed21 accuracy ppl 27.78
ceval-high_school_physics e1fc86 accuracy ppl 68.42
ceval-high_school_chemistry 9021c6 accuracy ppl 57.89
ceval-high_school_biology c7f5a1 accuracy ppl 63.16
ceval-middle_school_mathematics 213989 accuracy ppl 52.63
ceval-middle_school_biology ce0420 accuracy ppl 85.71
ceval-middle_school_physics 78f3af accuracy ppl 78.95
ceval-middle_school_chemistry d071d2 accuracy ppl 85
ceval-veterinary_medicine cd3a07 accuracy ppl 73.91
ceval-college_economics a35346 accuracy ppl 61.82
ceval-business_administration 69dd6a accuracy ppl 66.67
ceval-marxism 283ce0 accuracy ppl 68.42
ceval-mao_zedong_thought f38cd1 accuracy ppl 91.67
ceval-education_science fbd65c accuracy ppl 72.41
ceval-teacher_qualification c77f1f accuracy ppl 88.64
ceval-high_school_politics bbac37 accuracy ppl 84.21
ceval-high_school_geography 730a30 accuracy ppl 84.21
ceval-middle_school_politics 15b2d7 accuracy ppl 85.71
ceval-middle_school_geography b00167 accuracy ppl 66.67
ceval-modern_chinese_history 5a04cd accuracy ppl 82.61
ceval-ideological_and_moral_cultivation 0829ff accuracy ppl 100
ceval-logic c9c394 accuracy ppl 63.64
ceval-law cbd3c5 accuracy ppl 54.17
ceval-chinese_language_and_literature 716ab3 accuracy ppl 52.17
ceval-art_studies 476114 accuracy ppl 78.79
ceval-professional_tour_guide 70f30f accuracy ppl 82.76
ceval-legal_professional f19cf5 accuracy ppl 60.87
ceval-high_school_chinese 931614 accuracy ppl 47.37
ceval-high_school_history 4d6364 accuracy ppl 75
ceval-middle_school_history 7f6356 accuracy ppl 77.27
ceval-civil_servant a5dcb8 accuracy ppl 46.81
ceval-sports_science 192553 accuracy ppl 73.68
ceval-plant_protection f7ff86 accuracy ppl 77.27
ceval-basic_medicine a95a09 accuracy ppl 78.95
ceval-clinical_medicine 664b54 accuracy ppl 54.55
ceval-urban_and_rural_planner fdae6f accuracy ppl 69.57
ceval-accountant d810a1 accuracy ppl 73.47
ceval-fire_engineer bb924d accuracy ppl 51.61
ceval-environmental_impact_assessment_engineer d59200 accuracy ppl 64.52
ceval-tax_accountant 9e16f2 accuracy ppl 59.18
ceval-physician 0e90d5 accuracy ppl 75.51
### Current Behavior
是不是学习率最好和你们预训练结束时候的一致,然后继续预训练才能保持比较好的效果呢?能否说一下你们预训练最后的学习率是多少呢?谢谢各位大佬
### Expected Behavior
我用的三台A800 *8的机器,最后的学习率和loss日志如下:
{"current_steps": 5590, "total_steps": 5614, "loss": 1.4313, "eval_loss": null, "predict_loss": null, "reward": null, "learning_rate": 2.2546591544991837e-09, "epoch": 1.0, "percentage": 99.57, "elapsed_time": "5 days, 17:51:46", "remaining_time": "0:35:30"}
{"current_steps": 5595, "total_steps": 5614, "loss": 1.4016, "eval_loss": null, "predict_loss": null, "reward": null, "learning_rate": 1.4130842386717025e-09, "epoch": 1.0, "percentage": 99.66, "elapsed_time": "5 days, 17:59:09", "remaining_time": "0:28:06"}
{"current_steps": 5600, "total_steps": 5614, "loss": 1.406, "eval_loss": null, "predict_loss": null, "reward": null, "learning_rate": 7.672180148132757e-10, "epoch": 1.0, "percentage": 99.75, "elapsed_time": "5 days, 18:06:27", "remaining_time": "0:20:42"}
{"current_steps": 5605, "total_steps": 5614, "loss": 1.4244, "eval_loss": null, "predict_loss": null, "reward": null, "learning_rate": 3.170655392792377e-10, "epoch": 1.0, "percentage": 99.84, "elapsed_time": "5 days, 18:13:48", "remaining_time": "0:13:19"}
{"current_steps": 5610, "total_steps": 5614, "loss": 1.4334, "eval_loss": null, "predict_loss": null, "reward": null, "learning_rate": 6.263033621722869e-11, "epoch": 1.0, "percentage": 99.93, "elapsed_time": "5 days, 18:21:07", "remaining_time": "0:05:55"}
{"current_steps": 5614, "total_steps": 5614, "loss": null, "eval_loss": null, "predict_loss": null, "reward": null, "learning_rate": null, "epoch": 1.0, "percentage": 100.0, "elapsed_time": "5 days, 18:26:58", "remaining_time": "0:00:00"}
### Steps to Reproduce
环境什么的都是按照官方配置的
### Anything Else?
谢谢老师
增量数据样例如下,医疗数据和通用数据做了shuff:
{"text": "肾上腺腺瘤的早期症状 通常都是轻到中度肥胖,典型的表现是满月脸、水牛背、颈部短粗等,四肢正常或偏瘦。\n2. 蛋白质代谢异常皮肤比较薄,腰、腹、股、腋窝等处有宽大紫纹,肌肉萎缩无力,>严重骨质疏松。\n3. 糖尿病多饮、多尿、多食,体重减轻。\n4. 性功能障碍男性出现性欲减退、阴茎勃起功能障碍、早泄等;女性出现闭经、月经紊乱或减少、多毛等。\n5. 精神异常轻度患者出现失眠、记忆力
减退、注意力不能集中的表现。中度患者出现忧郁、躁狂等。严重患者出现抑郁症甚至精神分裂症的症状。\n6. 高血压。\n3. 原发性醛固酮增多症有以下表现大部分原发性醛固酮增多症的患者,早期症状不典型>,仅出现高血压。随着病情发展,出现低钾血症。晚期病人的症状发生频率依次如下高血压>低钾血症>肢体麻木和肌肉无力>夜尿增多。\n1. 高血压血压会逐渐升高,可出现头痛、乏力、视物模糊等。\n2. 神经系
统功能障碍早期会出现肌肉麻木和隐隐作痛,之后会出现全身乏力、肌肉酸痛、下肢麻痹等。\n3. 肾脏表现多尿、夜尿增多。\n4. 心脏表现,部分患者会出现阵发性室上性心动过速,严重时会出现心室颤动,甚>至出现心功能衰竭"}
{"text": "编写一份人力资源管理手册,包含员工招聘、培训、福利和考核等方面的制度规定。 人力资源管理手册\n第一章 员工招聘\n1.1 招聘流程\n1.1.1 发布招聘广告\n1.1.2 筛选简历\n1.1.3 安排面试\n1.1.4 面试评分\n1.1.5 决定录用与否\n1.2 招聘标准\n1.2.1 岗位职责与描述\n1.2.2 经验要求\n1.2.3 学历要求\n1.2.4 技能要求\n1.2.5 素质要求\n第二章 培训\n2.1 培训流程\n2.1.1 识别培训需求\n2.1.2 制定培训计划\n2.1.3 筛选培训方式\n2.1.4 培训执行\n2.1.5 培训评估\n2.2 培训类型\n2.2.1 入职培训\n2.2.2 技能培训\n2.2.3 职业发展培训\n2.2.4 领导力培训\n2.2.5 员工福利培训\n第三章 福利\n3.1 薪酬制度\n3.1.1 填写薪酬调查问卷\n3.1.2 提出薪酬方案\n3.1.3 实施薪酬方案\n3.1.4 监测薪酬效果\n3.2 假期制度\n3.2.1 年假\n3.2.2 病假\n3.2.3 婚假\n3.2.4 产假\n3.2.5 陪产假\n3.2.6 哺乳假\n3.2.7 丧假\n3.2.8 探亲假\n3.2.9 调休\n3.3 保险福利\n3.3.1 社会保险\n3.3.2 商业保险\n3.3.3 公积金\n第四章 考核\n4.1 考核流程\n4.1.1 制定考核指标\n4.1.2 员工自我评估\n4.1.3 直属经理评估员工\n4.1.4 员工评估直属经理\n4.1.5 综合评估结果\n4.1.6 告知评估结果\n4.2 考核指标\n4.2.1 工作绩效\n4.2.2 工作态度\n4.2.3 专业技能\n4.2.4 团队意识\n4.2.5 学习能力\n4.2.6 创新能力\n4.3 奖惩措施\n4.3.1 优秀员工奖励\n4.3.2 差评员工警告\n4.3.3 不合格员工处理\n4.3.4 绩效考核带薪调整\n4.3.5 申诉机制\n以上为本公司人力资源管理手册规定,旨在维护企业和员工的权益。违反本手册规定者,将根据
相应的企业规定进行处罚。"}
{"text": "有一年多了,脸上痘坑里都是黑色的,不是很明显,会不会自动消除? 1. 使用含有果酸、维生素C等有效成分的护肤品,可以帮助淡化黑色痘坑。\n2. 定期进行去角质,可以促进新的皮肤细胞生长,>帮助淡化黑色痘坑。\n3. 使用面膜,如炭黑面膜,可以有效清洁毛孔,减少黑头和痘痘,从而改善黑色痘坑的外观。\n4. 如果黑色痘坑比较明显,可以考虑采取医学美容手段,如激光治疗、微针、微晶瓷等,以>达到更好的效果。总之,如果您想改善黑色痘坑的外观,需要耐心和坚持。同时,也要注意避免刺激皮肤,保持皮肤清洁和滋润。"}
{"text": "今天才开始涨乳,孩子已经快三个月了,吸奶器也吸不出来,孩子也吸不出来,该怎么办 1. 经常喂奶将孩子放在乳房上经常喂奶,这有助于减轻乳腺堵塞和疼痛。\n2. 使用热敷使用热毛巾或热水袋在
乳房上进行热敷,这有助于促进血液循环,减轻乳腺堵塞。\n3. 使用吸奶器尝试使用吸奶器进行吸奶,可以轻松地将过多的乳汁吸出来,减轻乳腺堵塞和疼痛。\n4. 按摩乳房使用按摩的方式,轻轻地按摩乳房,>促进乳汁的流动和排出,缓解乳腺堵塞和疼痛。如果以上方法都没有效果,建议咨询医生或产科护士的建议。他们会根据你的具体情况给出更具体的建议和治疗方案。"}
{"text": "西安哪个治淋巴瘤医院最好,能治好淋巴瘤吗,西安哪个治淋巴瘤医院最好,能治好淋巴瘤吗? 淋巴瘤能治好吗与病人的身体机能密切相关。身体机能好,免疫力强,才能抵抗癌肿的发展,耐受各种药
物治疗。因此,提高免疫机能,增强对肿瘤的抵抗力对恶性淋巴瘤患者极为重要。生物细胞治疗可以有效控制癌细胞转移扩散,能够增强机体免疫功能以达到抑制癌细胞生长,同时又不产生副作用,在治癌抗癌的>同时,增强机体免疫力,以达到完全战胜癌症的目的。"}
{"text": "炎性改变的临床表现有些什么? 低弱回声"}
{"text": "您好,我有一个问题想请教,我和我的爱人有三次胎停育,之前在河南省人民医院生殖中心做了很多检查,最后做了免疫治疗,治疗后的阴性会转阳性吗?现在备孕已经大于半年了,还是没有怀孕,请>
问需要怎么处理? 您好,根据您提供的情况,我能理解您的忧虑和焦虑。对于复发性流产的治疗,免疫治疗是其中一种选择,但是其适用范围很有限,因为组织相容性抗原(HLA)抗体与复发性流产之间的关系并>
不是十分明确。免疫治疗可能在一定程度上减轻因HLA抗体导致的反复失败,但是其疗效存在争议,而且很多医生也不推荐这种治疗方式。在您的情况下,如果免疫治疗已经进行过,并且出现了阴性,那么通常情况
下不会再次转为阳性。但是建议需要进行进一步的检查来确定是否需要继续进行免疫治疗。此外,您与您的爱人备孕超过半年了,建议考虑进行一些额外的检查,例如女方可以进行内分泌检查、排卵监测及输卵管>
造影等来确定是否存在相应的问题;男方则可以进行精液检查。若以上检查均没有发现明显的异常,则可以考虑寻求辅助生殖技术,例如人工授精或者体外受精等。总之,建议您先要对您的身体进行全面的检查,>
这样才能明确原因并选择最合适的治疗方式。希望我的回答对您有所帮助。"}{"text": "癫痫可以治疗痊愈吗 ,我们家的小孩由于以前比较顽皮,在一次和别人玩耍时不小心把头给打破了,后来这个是痊愈了,但是这次创伤给他带来了癫痫病的后遗症,前一段时间还发作了,很吓人,幸亏
我们及时发现,然后送医院了,但是我还怕它会再发作。 你好,癫痫病史可以通过治疗完全痊愈的,癫痫病的治疗周期比较长,而且痊愈的时间是和病人的病情严重程度有关系,需要及时关注病情,积极配合医生的治疗。,癫痫病患者在及时治疗之外,患者在生活中还需要注意要保持良好的心情,好的心情对疾病的恢复很有帮助,希望上述的答案可以帮助到你,谢谢!"} {"text": "阐明个人品牌建立的必要性和方法,并给出至少3个行之有效的建议。 个人品牌建立的必要性: \n1. 在职场竞争中脱颖而出:个人品牌可以帮助我们在职场竞争中脱颖而出,让我们更快地被他人认可>
和记住。\n2. 在社交媒体上建立专业形象:在社交媒体上建立一个专业、令人信任的形象,能够增加我们的个人可信度,并且对于求职等方面也有好处。\n3. 凸显自身价值:通过建立个人品牌,我们可以更好地>展示自身价值、经验和技能,从而获得更多的职位机会和收入。\n个人品牌建立的方法:\n1. 创建一个独特的品牌名:选择一个个人品牌名,最好能够简洁易记。\n2. 建立强大的线上和线下存在感:在社交媒体>上和现实生活中展示自己的品牌。比如定期发布有价值的文章、参加有关自己所在领域的活动等等。\n3. 打造个人形象:包括个人形象设计、网站设计、简历设计等等,要保持一致性和专业性。\n行之有效的建议
:\n1. 定期更新自己的社交媒体,分享有关自己所在领域的见解、经验和思考。\n2. 参加行业活动、交流会议,拓展自己的人际关系。\n3. 虚心学习、接收反馈,不断提升自身能力和品牌价值。"}
{"text": "双贝特的副作用(不良反应) 偶有恶心、腹胀、腹泻、嗜睡、无力、脱发、白细胞减少、皮疹、瘙痒、肌强直、肌痉挛、肌酸磷酸激酶及谷草转氨酶升高等。"}
{"text": "刘小平(武汉理工大学化学工程学院教师)。刘小平,武汉理工大学化学工程学院教师。刘小平基本资料 出生年月 1958年2月 学 位 医学学士 刘小平教育经历 1982年1月,毕业于湖北中医学院中药专
业 刘小平工作简历 1982.1-2002.4,湖北中医学院附属医院 2002.5 —— ,武汉理工大学化学工程学院制药工程系 刘小平研究领域 药物新剂型与新技术研究;中药活性成分与制剂研究;药物分析与药品质量标准 百度百科内容由网友共同编辑,如您发现自己的词条内容不准确或不完善,欢迎使用本人词条编辑服务(免费)参与修正。立即前往"}
请问一下这个lr scheduler和batchsize是怎么设置的
上面写的有,deepspeed --hostfile=./hostfile --master_port=9901 src/train_bash.py
--deepspeed ./ds_config.json
--stage pt
--do_train
--model_name_or_path ../Yi-34B
--dataset input_test
--finetuning_type full
--lora_target q_proj,v_proj
--output_dir Yi-34B_output
--overwrite_cache
--per_device_train_batch_size 4
--gradient_accumulation_steps 4
--lr_scheduler_type cosine
--logging_steps 5
--save_steps 300
--learning_rate 5e-5
--num_train_epochs 1.0
--plot_loss
--bf16
上面写的有,deepspeed --hostfile=./hostfile --master_port=9901 src/train_bash.py
--deepspeed ./ds_config.json
--stage pt
--do_train
--model_name_or_path ../Yi-34B
--dataset input_test
--finetuning_type full
--lora_target q_proj,v_proj
--output_dir Yi-34B_output
--overwrite_cache
--per_device_train_batch_size 4
--gradient_accumulation_steps 4
--lr_scheduler_type cosine
--logging_steps 5
--save_steps 300
--learning_rate 5e-5
--num_train_epochs 1.0
--plot_loss
--bf16
你增量预训练的数据集医疗数据和通用数据的配比是怎么样的,而且你评测的两个数据集都是考试相关的,两个是同质的。你能不能多测几个不同测试集。最关键的是,在你实际应用场景效果是否有提升,你有测试过吗?
你好,你的问题已经在微信群里回答啦。
增量预训练是一件挺难的事情。预训练阶段的语料质量我们已经做了极其严格的把控,你的医疗语料很有可能在通用语料中出现过。增量训练的一些考量我们也在Yi-9B的公众号文章,以及TechReport(https://arxiv.org/abs/2403.04652)中提到过。祝好运~