A service to function as a small scale simulation of how Kochava distributes data to third parties in real time.
The project consists of a php application to ingest http requests and a go application to deliver http responses with a kafka job queue between them.
The interaction flow between the components is the following:
Web request → Ingestion Agent (php) → Delivery Queue (kafka) → Delivery Agent (go) → Web response
The detailed app operation is:
- Ingestion agent accepts incoming http request
- Ingestion agent pushes a postback object to the kafka delivery queue for each "data" object contained in accepted request.
- Delivery agent continuously pulls postback objects from kafka delivery queue.
- Delivery agent delivers each postback object to the http endpoint after having filled the query parameters with the data values.
- Delivery agent logs delivery time, response code, response time, and response body.
The configuration commands are for Windows 10. The commands for a Linux machine will be similar. The configuration of constants will be on the header of the relevant files.
Run composer update on the ingestion-agent folder to retrieve all the dependencies.
To setup the PHP server two options were tested:
- Download and unzip PHP. It may be useful to add it to the PATH environtment variable. You can then startup the server with:
php -S localhost:80 -t "$PWD"
- Download and install XAMPP. Run the XAMPP Control Panel and start the Apache module. Using XAMPP will support logging and other useful features out of the box. By default the PHP code will be on "C:\xampp\htdocs". If you wish to change the default directory on the XAMPP app go to
Config->httpd.confand change the DocumentRoot and Directory:
DocumentRoot "C:/xampp/htdocs"
<Directory "C:/xampp/htdocs">
The PHP versions used are 8.1.6 with XAMPP v3.3.0 and 8.1.10 with the regular PHP.
Download and unzip Apache Kafka. Go to the unzipped Kafka folder and run the following commands:
bin/windows/zookeeper-server-start.bat config/zookeeper.properties
bin/windows/kafka-server-start.bat config/server.properties
bin/windows/kafka-topics.bat --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic postback
bin/windows/kafka-topics.bat --list --bootstrap-server localhost:9092
The Apache Kafka version used is kafka_2.13-3.2.1
Download and install Golang
Run go get -u github.com/segmentio/kafka-go on the delivery-agent folder to install the needed dependency
To startup the delivery agent run go run deliver.go. By default the service will log to logfile.log
The used Go version is 1.19
After setting up everything you can send a POST request to the php ingest agent:
curl "localhost:80/ingest.php" -H 'Content-Type: application/json' -X POST -d '@request_weather.json'
You can also use Postman or any other software to send the request to the php ingest agent.