secc - Distributed compiler with modern web technology.

a project of 'Second Compiler'.

Key Features

RESTful API in Daemon/Scheduler
Preprocessed and Pump Mode
Memory Cache(works like remote Ccache)
Debug fission(-gsplit-dwarf option) supports.
Monitoring by Browser(Navigate to http://SCHEDULER:PORT)
Cross-compile(Various version/machine of compiler in network) supports
Gcc / Clang support

Quick Start

use DOCKER to start a scheduler and daemons

How to use

install by git clone (then npm install)

git clone https://github.com/ivere27/secc.git
cd secc
npm install

make sure you've already installed node.js, npm and (optionally) redis

Ubuntu $ sudo apt-get install nodejs npm nodejs-legacy redis-server
Mac $ brew install nodejs redis

Client - your PC

set PATH and NUMBER_OF_PROCESSORS(-jX)

edit "client" part in 'settings.json' file. set scheduler's address and port.

$ export PATH=/path/to/secc/bin:$PATH
$ export NUMBER_OF_PROCESSORS="8"
$ # then,
$ clang -c test.c


or just use CC, CXX and NUMBER_OF_PROCESSORS env
```sh
$ SECC_ADDRESS=172.17.42.1 SECC_CACHE=1 SECC_CROSS=0 \
CC=/path/to/secc/bin/clang CXX=/path/to/secc/bin/clang++ ./configure
$ SECC_ADDRESS=172.17.42.1 SECC_CACHE=1 SECC_CROSS=0 \
CC=/path/to/secc/bin/clang CXX=/path/to/secc/bin/clang++ make -j8

upload your compiler archive by ./tool/secc-upload-archive.js

$ # for clang
$ node secc-upload-archive.js --clang /path/to/clang /path/to/clang++ archivetool.js http://SCHEDULER:PORT
$ # for gcc
$ node secc-upload-archive.js --gcc /path/to/gcc /path/to/g++ archivetool.js http://SCHEDULER:PORT

in linux case(you can use the specific compiler version),

 node secc-upload-archive.js --gcc /usr/bin/gcc-5 /usr/bin/g++-5 ./secc-create-archive-linux.js http://172.17.42.1:10509

 node secc-upload-archive.js --clang /usr/bin/clang /usr/bin/clang++ ./secc-create-archive-linux.js http://172.17.42.1:10509

Daemon - n PC

edit "daemon" part in 'settings.json' file. set scheduler's address and port.

if you want to use cache, go #Caches section.

run 'node secc-daemon.js' as root for chroot-jail.

$ sudo DEBUG=secc* node secc-daemon.js

Scheduler - 1 PC

$ DEBUG=secc* node secc-scheduler.js

Modes

MODE 1 - Preprocessed Mode(default) send one preprocessed source to daemon. (cause CPU load)

MODE 2 - Pump Mode send headers and source to daemon. (cause Memory/Bandwidth load)

MODE 3 - Git Mode(not yet supported.)

Debug

DEBUG=* to watch every verbose logs.
DEBUG=secc* to watch only SECC's log.
SECC_LOG=/path/to/log.txt - redirect logs to a file

$ DEBUG=* SECC_MODE=1 SECC_CACHE=1 /path/to/secc/bin/gcc -c test.c

Caches

install REDIS in a daemon computer. then, enable "cache" in "daemon" part of 'settings.json' file.

Environment Variable

ENV	secc-client	secc-daemon	secc-scheduler
SECC_ADDRESS	scheduler address	scheduler address	-
SECC_PORT	scheduler port	scheduler port	listening port
DEBUG	log level	log level	log level
SECC_LOG	log file path	log file path	log file path
SECC_CMDLINE	log command line	-	-
SECC_MODE	processed or pump	-	-
SECC_CACHE	cache prefer	cache enable	-
SECC_CROSS	cross-compiling prefer	-	-
SECC_CC	c compiler path	-	-
SECC_CXX	c++ compiler path	-	-
REDIS_ADDRESS	-	redis address	-
REDIS_PORT	-	redis port	-
SECC_EXPOSE_ADDRESS	-	address for client	-
SECC_EXPOSE_PORT	-	port for client	-

How It Works

Scheduler - RESTful/WebSocket/Monitoring WebServer
Daemon - RESTful WebServer + WebSocket Client + Cache Server(redis)
Client - http client

(once) Client uploads a Compiler Archive to Scheduler
Scheduler and Daemons are connected by WebSocket
Client asks to Scheduler which daemon is available by REST API
(Optional) Client tries to get caches if possible
Client sends a source or/and dependencies to a Daemon by REST API
(once) Daemon downloads the archive from Scheduler if not exists
Daemon compiles the sources by Client's Compiler Archive
Daemon responds to Client with a object(+@)
(Optional) Objects are stored in Daemon's MemoryDB(redis)

License

MIT

jason-hwang / secc