for Redhat Linux (centos)
Install Python3.7 and pip3
sudo yum update
sudo yum -y install zlib-devel bzip2-devel openssl-devel ncurses-devel sqlite-devel readline-devel tk-devel gcc make libffi-devel
sudo yum -y install epel-release
sudo yum -y install python-pip
sudo pip install wget
wget https://www.python.org/ftp/python/3.7.5/Python-3.7.5.tgz
tar -zxvf Python-3.7.5.tgz
cd Python-3.7.5
./configure prefix=/usr/local/python3
sudo make install
sudo ln -s /usr/local/python3/bin/python3.7 /usr/bin/python3
sudo ln -s /usr/local/python3/bin/pip3.7 /usr/bin/pip3
python3 -V
sudo yum -y install python-devel python3-devel python-Levenshtein
for Redhat Linux (centos)
sudo rpm -Uvh https://yum.postgresql.org/10/redhat/rhel-7-x86_64/pgdg-centos10-10-2.noarch.rpm
sudo yum -y install postgresql10-server postgresql10
sudo yum -y install postgresql-contrib postgresql-libs
sudo /usr/pgsql-10/bin/postgresql-10-setup initdb
systemctl start postgresql-10.service
systemctl enable postgresql-10.service
create a new database and setting db users
sudo su - postgres -c "psql"
\conninfo
\password postgres
CREATE DATABASE [name of database];
\q
finally make sure pg_hba.conf is trust all localhost
postgres=# show_hba_file;
for Redhat Linux (centos)
sudo yum -y install epel-release yum-utils
sudo yum -y install http://rpms.remirepo.net/enterprise/remi-release-7.rpm
sudo yum-config-manager --enable remi
sudo yum -y install redis
sudo systemctl start redis
sudo systemctl enable redis
sudo systemctl status redis
for Redhat Linux (centos)
sudo yum -y install nginx
sudo systemctl start nginx
for both linux sysyem, allow 80 and 443 port in firewall
sudo firewall-cmd --permanent --zone=public --add-service=http
sudo firewall-cmd --permanent --zone=public --add-service=https
sudo firewall-cmd --reload
for Redhat Linux (centos)
sudo yum -y install python-virtualenv
make sure pip version > 19.0.x
sudo pip -V
for both Linux system
sudo pip install uwsgi
make project folder and clone the project. for example the project name is "ai-opt":
- 1.1. Clone the project
mkdir /ai-opt/chatfilter
cd /ai-opt/chatfilter
git clone ...
cd /ai-opt/chatfilter/main
- 1.2. Setting nginx config
copy and chang your network setting in nginx.conf file
cp nginx.conf.example nginx.conf
make symbolic link to niginx configs
sudo ln -s /ai-opt/chatfilter/main/nginx.conf /etc/nginx/sites-enabled/
or
sudo ln -s /ai-opt/chatfilter/main/nginx.conf /etc/nginx/conf.d/
- 1.3. Setting Project config
copy setting.ini and chagne config you need
cp setting.ini.example setting.ini
nano setting.ini
change "DATABASE" setting
[DATABASE]
DATABASE_NAME = DB_NAME
DATABASE_USER = DB_USER_NAME
DATABASE_PASSWORD = DB_PASSWORD
- 0.4. Create logs directory in project and make sure the logs folder changeable for supervisor(python)
mkdir /ai-opt/logs
chmod -R 777 /ai-opt/logs
Create virtual environment named venv:
cd /ai-opt
python3 -m venv venv
chmod -R 777 venv
source venv/bin/activate
python -V
pip -V
should be seen the python version at least with 3.7.5 and pip is 19+
before doing this you've make sure you already got "venv" environment install what python's need in "venv"
pip install tensorflow==2.3.1
pip install tensorflow_datasets==1.3.0
pip install -r requirement.txt
pip install psycopg2-binary
pip install websocket
pip install websocket-client
pip install zhconv
pip install xlwt
pip install django-import-export
pip install grpcio
pip install grpcio-tools
pip install djangogrpcframework
pip install opencc-python-reimplemented
pip install psycopg2-binary==2.8.6
pip install tensorflow-text
build up the database instruction
python manage.py migrate
python manage.py loaddata service/seed/initial.json
python manage.py loaddata ai/json/knowledge.json
create django admin superuser with following the guiding steps to finish
python manage.py createsuperuser
..fill all the form
collect and copy the static file in project to improve performance
python manage.py collectstatic
before you train you may need to check your vocabulary dictionary
python manage.py knowledge -i ai/assets/chinese/dict_taiwan.xlsx -lan TW -f 2
python manage.py knowledge -i ai/assets/english/dict.xls -lan EN -f 3
python manage.py knowledge -di ai/assets/chinese_digits/dict.xls
python manage.py knowledge -i ai/assets/chinese/dict_china_common.xls -lan CN -f 24
python manage.py knowledge -i ai/assets/chinese/dict_china_single.xls -lan CN -f 2
python manage.py freq -i ai/assets/textbook/json/2022-01-12.json
start training
python manage.py train -i ai/assets/textbook/json/grammar/2022-01-14.json -f 0.9985 -grm -t 3
python manage.py train -i ai/assets/textbook/json/pinyin/2020-08-31.json -f 0.9987 -t 12
python manage.py train -i ai/assets/textbook/json/english/2020-09-08.json -eng -t 1
open tcp port for chatting socket if need
sudo firewall-cmd --permanent --zone=public --add-port=8025/tcp
sudo firewall-cmd --permanent --zone=public --add-port=9001/tcp
sudo firewall-cmd --reload
setting supervisor http://supervisord.org/configuration.html (if you are in sourced on venv you have to deactivate it)
deactivate
for Redhat Linux (centos)
sudo yum -y install supervisor
sudo systemctl start supervisord
sudo systemctl enable supervisord
sudo systemctl status supervisord
copy and edit config
cp supervisor.conf.example supervisor.conf
nano supervisor.conf
-
change all directory
/opt/
to your project's folder
directory=/ai-opt/chatfilter/main
-
change all
/opt/venv/bin/python
to your virtual environment python
command=/ai-opt/venv/bin/python tcpsocket/main.py -p 8025
-
change all
/opt/logs/
to your logs folder
stdout_logfile=/ai-opt/logs/tcpsocket.log
symbolic link to supervisor config
sudo ln -s /ai-opt/chatfilter/main/supervisor.conf /etc/supervisord.d/ai-chatfilter-service.ini
reload supervisor
sudo supervisorctl reload
sudo supervisorctl reread
sudo supervisorctl update
reload nginx
sudo systemctl reload nginx
sudo systemctl restart nginx
Check the SELinux and add policy to nginx or just disable it
sestatus
setsebool -P httpd_can_network_connect 1
SELinux might block the socket connection between nginx and supervisord
Test the django web site is working, type domain:port on browser for example:
http://127.0.0.1:8000/
you should see the screen with 404 not found page but has content like below
Using the URLconf defined in service.urls, Django tried these URL patterns, in this order:
chat/
admin/
auth/
auth/
The empty path didn't match any of these.
that means the website is working fine and next we change url to
http://127.0.0.1:8000/chat/
, try typeing something to test websockets
use tcpsocket client to test chatting binary packages.
cd /ai-opt/chatfilter/main
python tcpsocket/client.py -h 127.0.0.1 -p 8025
you will see
Please choose package type:
1.hearting
2.login
3.login response
4.chatting
5.chat response
Enter number:
everything is fine!!
python manage.py predict -t [speaksome..] -s
python manage.py predict -i [ai/assets/..] -s
python manage.py testsocket -i [ai/assets/..] -p 8025 -h 127.0.0.1
some commod tips dump and restore blockword data do not use until you know all about database
python manage.py dumpdata service > service/seed/initial.json
python manage.py loaddata service/seed/initial.json
python manage.py upsert -i ../excel file -model textbook
python manage.py parsexcel -i ai/assets/textbook/pinyin
python manage.py parsexcel -i ai/assets/textbook/grammar
python manage.py freq -i ai/assets/textbook/json/pinyin
python manage.py backupdatabase
python -m grpc_tools.protoc --proto_path=./ --python_out=./ --grpc_python_out=./ ./account.proto
/usr/pgsql-10/bin/pg_dump ai_chat -U postgres -c --if-exists -f ./bak.sql -t [表名]
docker exec -it e3d37 pg_restore -U postgres -d aichatfilter -1 ./bak_bad_0512.sql
docker pull django
docker pull postgres
docker pull redis