AutoRclone: rclone copy/move/sync (automatically) with service accounts (still in the beta stage)

create service accounts using script
add massive service accounts into rclone config file
add massive service accounts into groups for your organization
automatically switch accounts when rclone copy/move/sync
Windows system is supported

Step 1. Copy code to your VPS or local machine

Before everything, install python3. Because we use python as our programing language.

For Linux system: Install screen, git and latest rclone. If in Debian/Ubuntu, directly use this command

sudo apt-get install screen git && curl https://rclone.org/install.sh | sudo bash

After all dependency above are successfully installed, run this command

sudo git clone https://github.com/xyou365/AutoRclone && cd AutoRclone && sudo pip3 install -r requirements.txt

For Windows system: Directly download this project then install latest rclone. Then run this command (type in cmd command windows or PowerShell windows) in our project folder

pip3 install -r requirements.txt

Step 2. Generate service accounts

Let us create as many service accounts as possible.

Enable the Drive API in Python Quickstart and save the file credentials.json into project directory.

If you do not have any project in your account, just run python3 gen_sa_accounts.py --quick-setup 5 to

create 6 projects (project 0 to project 5)
enable the required services
create 600 (6 projects, each with 100) Service Accounts
and download their credentials into a folder named accounts

If you have already N projects and want to create service accounts only in newly created projects, run python3 gen_sa_accounts.py --quick-setup 2 --new-only to

create additional 2 projects (project N+1 to project N+2)
enable the required services
create 200 (2 projects, each with 100) Service Accounts
and download their credentials into a folder named accounts

If you want to create some service accounts using existing projects (do not create more projects), run python3 gen_sa_accounts.py --quick-setup -1.Note that this will overwrite the existing service accounts.

After it is finished, there will be many json files in one folder named accounts.

What is service account

How to use service account in rclone.

Step 3. Add service accounts to Google Groups (Optional)

We use Google Groups to manager our service accounts considering the
Official limits to the members of Team Drive (Limit for individuals and groups directly added as members: 600).

For GSuite Admin

Turn on the Directory API following official steps (save the generated json file to folder credentials).
Create group for your organization in the Admin console. After create a group, you will have an address for examplesa@yourdomain.com.
Run python3 add_to_google_group.py -g sa@yourdomain.com

For meaning of above flags, please run python3 add_to_google_group.py -h

For normal user

Create Google Group then add the service accounts as members by hand.

Step 4. Add service accounts or Google Groups into Team Drive

If you do not use Team Drive, just skip.

If you have already created Google Groups (Step 2) to manager your service accounts, add the group address sa@yourdomain.com or sa@googlegroups.com to your source Team Drive (tdsrc) and destination Team Drive (tddst).

Otherwise, add service accounts directly into Team Drive.

Enable the Drive API in Python Quickstart and save the credentials.json into project root path if you have not done it in Step 2.

Add service accounts into your source Team Drive: python3 add_to_team_drive.py -d SharedTeamDriveSrcID

Add service accounts into your destination Team Drive: python3 add_to_team_drive.py -d SharedTeamDriveDstID

Step 5. Start your task

Let us copy hundreds of TB resource using service accounts.

For server side copy

publicly shared folder to Team Drive
Team Drive to Team Drive
publicly shared folder to publicly shared folder (with write privilege)
Team Drive to publicly shared folder

python3 rclone_sa_magic.py -s SourceID -d DestinationID -dp DestinationPathName -b 1 -e 600

For meaning of above flags, please run python3 rclone_sa_magic.py -h
Add --disable_list_r if rclone cannot read all contents of public shared folder.
Please make sure the Rclone can read your source and destination directory. Check it using rclone size:

rclone --config rclone.conf size --disable ListR src001:
rclone --config rclone.conf size --disable ListR dst001:

For local to Google Drive (need test)

local to Team Drive
local to private folder
private folder to any (think service accounts cannot do anything about private folder)

python3 rclone_sa_magic.py -sp YourLocalPath -d DestinationID -dp DestinationPathName -b 1 -e 600

Run command tail -f log_rclone.txt to see what happens in details (linux only).

Also let's talk about this project in Telegram Group AutoRclone

Blog（中文） | Google Drive Group | Google Drive Channel

nsdown / AutoRclone