ingestion-pipeline

There are 2 repositories under ingestion-pipeline topic.

• bruin-data / ingestr

  ingestr is a CLI tool to copy data between any databases with a single command seamlessly.

  bigquerycopy-databasedata-ingestiondata-integrationdata-pipelineduckdbingestion-pipelinemssqlpostgresqlsnowflake
  Language:Python 2422

• opensemanticsearch / open-semantic-etl

  Python based Open Source ETL tools for file crawling, document processing (text extraction, OCR), content analysis (Entity Extraction & Named Entity Recognition) & data enrichment (annotation) pipelines & ingestor to Solr or Elastic search index & linked data graph database

  annotationdocumentselasticsearchenrichmentetlextractextract-informationextract-textextractoringestingestion-pipelineingests-documentsnamed-entity-recognitionnlpocrpdfpythonrdfsolrsolr-dataimporter
  Language:Python 254

• KnudsenMorten / AzLogDcrIngestPS

  AzLogDcrIngestPS - Unleashing the power of Log Ingestion API with Azure LogAnalytics custom table v2, Azure Data Collection Rules and Azure Data Ingestion Pipeline

  azuredatadata-collection-rulesingestion-pipelinelogmanipulationazure-pipelinedcrlog-ingestionpowershell
  Language:PowerShell 30

• Morphl-AI / MorphL-Model-User-Search-Intent

  Google Cloud Storage connector, pre-processor and model for predicting user search intent based on keywords

  machine-learningnlp-machine-learningpysparkpreprocessoringestion-pipelinepredictmorphl-platform
  Language:Python 21

• Morphl-AI / MorphL-Model-Publishers-Churning-Users

  Google Analytics connector, pre-processor and model for predicting churning users for digital publishers.

  google-analyticspredictionpreprocessorpysparkmachine-learningingestion-pipelinemorphl-platform
  Language:Python 10
• Emancipitaion-of-Apache-Spark

  akshaybahadur21 / Emancipitaion-of-Apache-Spark

  My experiments with Apache Spark for Humans ⭐

  sparkarchitectureapache-sparkingestion-pipeline
  Language:Java 6

• tmcgrath / cassandra-ingest

  DataStax or Cassandra Ingest from Relational Databases with StreamSets

  cassandraingestingestion-pipelinerdbmscdcdatastaxstreamsets
  Language:PLSQL 4

• azuregig / work_with_OrdnanceSurvey_data

  Sample Azure Data Factory pipeline for ingesting Data Packages directly from the Download API of the Ordnance Survey Data Hub into Azure Storage.

  geospatial-processinggeospatial-dataazureazure-data-factoryingestion-pipelineingestionordnance-surveyordnancesurveymastermapos-data-hub-api
  3

• siddharth271101 / Stock-Exchange-Analysis

  Created a data pipeline using sqoop to ingest data from sql server into the hive table and used hive for feature engineering and analysis.

  big-datahiveingestion-pipelinemysqlsqoop
  Language:Shell 2

• mbsuraj / postgresql_ingestion_script

  Ingest any format data into postgreSQL database

  ingestioningestion-pipelinepostgresqlpostgresql-databasecsv-importpython3quickeasy-to-useeasyready-to-useready-to-runclean-code
  Language:Python 1

• rachita27 / AUTOMATING

  Automating Ingestion Excel Files On To Azure Data Studio (SQL-Server)

  azurefunctionspythonjupyter-notebookingestion-pipelineingestionsqlsqlserverpandaspyodbcpython3excel
  Language:Jupyter Notebook 1

• rohitshubham / Cloud-pipeline

  A real-life end-to-end cloud sub-system scenario

  docker-composecloud-pipelinesparkspark-streamingkafkakafka-brokersmongodbingestion-pipelinestreaming-analytics
  Language:Python 1

• amyth-singh / multinational-retail-data-centralisation

  The multinational retail data contralisation project is a data warehousing project that focuses on ingesting data from disparate sources to create a centralised warehouse

  data-warehouseingestion-pipeline
  Language:Python 0

• fnldesign / crypthobot-ingestion

  A cryptho currency automated bot

  ingestion-pipelinecryptocurrencyautomated-machine-learningbot
  0

• Ludovik99 / Analysis-of-Gas-Stations-with-Apache-Spark

  Simulating a consultancy project for Repsol, the repository contains both the code notebook and the analysis.

  apache-sparkhadoop-hdfsingestion-pipelinenifistorage
  Language:Jupyter Notebook 0

• waqarg2001 / Covid-19-DE-Project

  ETL process applied on covid-19 dataset of European countries using Azure services such as databricks, keyvault, sql database, data factory etc. Finally power bi dashbaord was also made.

  azureazure-pipelinescovid-19datadata-engineeringdata-factorydata-lakedatabricksetletl-pipelineingestion-pipelinekey-vaultpower-bipysparkpythonsql
  Language:Python 0

• alanzhaonys / workmail-intercepter-excel-to-csv

  Transform incoming AWS WorkMail email with Excel attachment to CSV and save to S3 bucket

  aws-lambdadataingestion-pipelineworkmail
  Language:Python

• Charanaicore / multinational-retail-data-centralisation

  The multinational retail data contralisation project is a data warehousing project that focuses on ingesting data from disparate sources to create a centralised warehouse

  data-warehouseingestion-pipeline
  Language:Python

• Sqooba / mssql-to-avro-with-spark

  Apache Spark example reading from MSSQL and converting in AVRO format.

  sparkmssqldataingestion-pipelineavro
  Language:Java

• vivek-bombatkar / Graph-Datastructure-for-Movielens-dataset

  graphhbasepyspark-notebookspark-streamingmachine-learningingestion-pipelinepandas
  Language:Jupyter Notebook