bgpat / glue2bq

Convert the table schema from AWS Glue to Google BigQuery

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

glue2bq

Convert the table schema from AWS Glue to Google BigQuery

example

$ aws glue get-table --database-name=default --name=cloudtrail_logs | ruby main.rb
[{"name":"eventversion","type":"STRING","mode":"NULLABLE"},{"name":"useridentity","type":"RECORD","mode":"NULLABLE","fields":[{"name":"type","type":"STRING","mode":"NULLABLE"},{"name":"principalid","type":"STRING","mode":"NULLABLE"},{"name":"arn","type":"STRING","mode":"NULLABLE"},{"name":"accountid","type":"STRING","mode":"NULLABLE"},{"name":"invokedby","type":"STRING","mode":"NULLABLE"},{"name":"accesskeyid","type":"STRING","mode":"NULLABLE"},{"name":"username","type":"STRING","mode":"NULLABLE"},{"name":"sessioncontext","type":"RECORD","mode":"NULLABLE","fields":[{"name":"attributes","type":"RECORD","mode":"NULLABLE","fields":[{"name":"mfaauthenticated","type":"STRING","mode":"NULLABLE"},{"name":"creationdate","type":"STRING","mode":"NULLABLE"}]},{"name":"sessionissuer","type":"RECORD","mode":"NULLABLE","fields":[{"name":"type","type":"STRING","mode":"NULLABLE"},{"name":"principalid","type":"STRING","mode":"NULLABLE"},{"name":"arn","type":"STRING","mode":"NULLABLE"},{"name":"accountid","type":"STRING","mode":"NULLABLE"},{"name":"username","type":"STRING","mode":"NULLABLE"}]}]}]},{"name":"eventtime","type":"STRING","mode":"NULLABLE"},{"name":"eventsource","type":"STRING","mode":"NULLABLE"},{"name":"eventname","type":"STRING","mode":"NULLABLE"},{"name":"awsregion","type":"STRING","mode":"NULLABLE"},{"name":"sourceipaddress","type":"STRING","mode":"NULLABLE"},{"name":"useragent","type":"STRING","mode":"NULLABLE"},{"name":"errorcode","type":"STRING","mode":"NULLABLE"},{"name":"errormessage","type":"STRING","mode":"NULLABLE"},{"name":"requestparameters","type":"STRING","mode":"NULLABLE"},{"name":"responseelements","type":"STRING","mode":"NULLABLE"},{"name":"additionaleventdata","type":"STRING","mode":"NULLABLE"},{"name":"requestid","type":"STRING","mode":"NULLABLE"},{"name":"eventid","type":"STRING","mode":"NULLABLE"},{"name":"resources","mode":"NULLABLE","type":"RECORD","fields":[{"name":"arn","type":"STRING","mode":"NULLABLE"},{"name":"accountid","type":"STRING","mode":"NULLABLE"},{"name":"type","type":"STRING","mode":"NULLABLE"}]},{"name":"eventtype","type":"STRING","mode":"NULLABLE"},{"name":"apiversion","type":"STRING","mode":"NULLABLE"},{"name":"readonly","type":"STRING","mode":"NULLABLE"},{"name":"recipientaccountid","type":"STRING","mode":"NULLABLE"},{"name":"serviceeventdetails","type":"STRING","mode":"NULLABLE"},{"name":"sharedeventid","type":"STRING","mode":"NULLABLE"},{"name":"vpcendpointid","type":"STRING","mode":"NULLABLE"}]

About

Convert the table schema from AWS Glue to Google BigQuery

License:MIT License


Languages

Language:Ruby 100.0%