The Python Peer is used to request Eulerian Data Warehouse Services.
Explore the docs »
View Demo
·
Report Bug
·
Request Feature
To get a local copy up and running follow these simple steps.
Peer rely on python package requests : https://docs.python-requests.org/en/master/
- pip
python -m pip install requests
Peer rely on python package ijson : https://pypi.org/project/ijson/
- pip
python -m pip install ijson
- Clone the repo
git clone https://github.com/THORCOMP/Eulerian-Data-Warehouse-Python-Peer.git
Helper script Peer.sh is provided to simplify the python Peer.py usage.
This script rely on the setup of a configuration file named Peer.conf located in the same directory than the script, it is in charge of providing mandatories parameters to ./Peer.py script.
./Peer.sh file1 file2 ... filexIf used directly, python script help is as following :
./Peer.py <options> file1 file2 ... filexWith options :
Mandatories :
--grid=<grid> : Set Eulerian customer Grid name.
--ip=<ip> : Set Eulerian customer IPv4 address.
--token=<token> : Set Eulerian customer AES token.
Optionals :
--accept=<accept> : Set accepted reply format. ( default : 'application/json' ).
--wdir=<directory> : Set Working directory used to download replies. ( default : '/tmp' ).
--download-only : Download the replies into working directory.
Experts :
--peer=<class> : Set transport class. ( default : Eulerian.Edw.Peers.Rest ).
--hook=<class> : Set output hook class. ( default : Eulerian.Edw.Hooks.CSV ).
--hook-options=<path> : Set output hook options. ( default : Hook.conf ).
--host=<host> : Set remote host. ( default : None ).
--ports=<ports> : Set remote ports. ( default : 80,443 ).
--platform=<platform> : Set the name of authority platform. ( default : france ).
--unsecure : Use unsecure transport layer.class Eulerian.Edw.Peers.Rest
/**
* @brief Eulerian Data Warehouse Rest Peer constructor.
*/
Rest()
/**
* @brief Defines Eulerian Grid Name.
* This attribute is mandatory.
* Its default value is None.
*
* @param grid - Grid Name.
*/
set_grid( grid )
/**
* @brief Defines Eulerian Grid Name.
* This attribute is mandatory.
* Its default value is None.
*
* @param token - Token.
*/
set_token( token ) /**
* @brief Defines Eulerian Platform Name.
* 'france' or 'canada' are the only accepted values.
* Default value is 'france'.
*
* @param platform - Platform name.
*/
set_platform( platform )
/**
* @brief Defines Accepted reply format.
* 'application/json', 'text/csv' and 'application/parquet' are the only accepted values.
* Default value is 'application/json'.
*
* @param accept - Accept format.
*/
set_accept( accept )
/**
* @brief Defines local working directory.
* Any writable directory is valid.
* Default value is '/tmp'.
*
* @param dir - Working directory.
*/
set_wdir( dir )set_host( host ) : Defines Remote host. set_ports( ports ) : Defines Remotes ports( secure/unsecure ). set_kind( kind ) : Defines Authority token type ( Access/Session ).
request( command ) : Do command
Create a Job download the resulting file
MyPeer.py
from Eulerian.Edw.Peers.Rest import Rest as Rest
/* Setup mandatory parameters */
platform = 'france' // ( Can be france or canada )
grid = <GridName> // ( Eulerian Customer Grid )
token = <Token> // ( Eulerian Customer Token )
request = <request> // ( Eulerian Data Warehouse Request )
/* Create new Peer Instance */
peer = Rest()
/* Setup mandatory parameters */
peer.set_platform( platform )
peer.set_grid( grid )
peer.set_token( token )
peer.set_download( True )
/* Send request to the server, wait end of the job, download
resulting file, return path to the file */
path = peer.request( request )Create a Job download the resulting file, parse streamed file and call Hook functions
MyHook.py
from Eulerian.Edw.Hook import Hook as Hook
class MyHook( Hook ) :
def on_add( self, uuid, rows ) :
for row in rows :
string = ''
for col in row :
if type( col ) is bytes :
col.col.decode()
string += col
string += '#'
print( string )
MyPeer.py
from Eulerian.Edw.Peers.Rest import Rest as Rest
import MyHook
/* Setup mandatory parameters */
platform = 'france' // ( Can be france or canada )
grid = <GridName> // ( Eulerian Customer Grid )
token = <Token> // ( Eulerian Customer Token )
request = <request> // ( Eulerian Data Warehouse Request )
/* Create new Peer Instance */
peer = Rest()
/* Setup mandatory parameters */
peer.set_platform( platform )
peer.set_grid( grid )
peer.set_token( token )
peer.set_hooks( MyHook() )
peer.set_accept( "application/json" )
/* Send request to the server, wait end of the job, download
resulting file, parse the file call hooks */
peer.request( request )Create a Job download the resulting file, load its content into Pandas Data Frame
MyPeer.py
from Eulerian.Edw.Peers.Rest import Rest as Rest
import pandas as Pandas
/* Setup mandatory parameters */
platform = 'france' // ( Can be france or canada )
grid = <GridName> // ( Eulerian Customer Grid )
token = <Token> // ( Eulerian Customer Token )
request = <request> // ( Eulerian Data Warehouse Request )
/* Create new Peer Instance */
peer = Rest()
/* Setup mandatory parameters */
peer.set_platform( platform )
peer.set_grid( grid )
peer.set_token( token )
peer.set_download( True )
/* Setup format used to load data into Pandas */
peer.set_accept( "application/parquet" )
/* or
peer.set_accept( "text/csv" ) */
/* Send request to the server, wait end of the job, download
resulting file, return path to the file */
path = peer.request( request )
/* Load resulting file ( parquet format ) */
frame = Pandas.read_parquet( path )
/* or
frame = Pandas.read_csv( path ) */
See the open issues for a list of proposed features (and known issues)
Distributed under the MIT License. See LICENSE for more information.
THORILLON Xavier - x.thorillon@eulerian.com
Project Link: https://github.com/EulerianTechnologies/Eulerian-Data-Warehouse-Python-Peer