AWS Data Pipeline Restore

💸A python module for building portfolio assessment pipeline

This is a basic example of using a pipeline in data science.

Learn to design models, build data warehouses and data lakes, automate data pipelines and work with massive datasets.

This is the data pipeline for the url-shortner application. Deprecated in favor of https://github.com/Dukes-Wine-Co/request-parsing-api

ETL pipeline with PySpark on Dataproc for data lake on Google Cloud Storage

UC Davis Distributed Computing with Spark SQL (with Databricks) and Databricks Apache Spark SQL for Data Analysts

An easy to use, reliable and well designed python module that domain experts and data scientists can use to fetch, visualise, and transform publicly available satellite and LIDAR data.

Get started with Prefect by scheduling your Prefect flows with GitHub Actions

This project aims to do exploratory data analysis of the listings on Airbnb NYC 2019 Dataset (48895, 16). Airbnb has a global reach, and data analysis plays a crucial role in its operations.

Checking the scalability of a data pipeline involving MySQL, Spark and Machine Learning Models using Latency.

This repository contains code for comparing the performance of three different ELT (Extract, Load, Transform) methods on CSV files of different sizes. The three methods are implemented in Python using different approaches and libraries, and their execution times are compared and plotted for analysis.

House prices dataset exploration and prediction. Workflow includes useful examples of Tensorflow pipelines including k-Nearest Neighbors imputer, Decision Tree Regression and XGBoost Regression

The project works with aerospace component data, high number of part removals end up being unscheduled and that results in less flight time for the aircraft, this project deals with the historical data and tries to predict the parts failure on component level using survival analysis to avoid the mechanical induced disruptions.

A custom Airbyte connector to fetch football data from the Football-Data.org API. It allows users to retrieve match results, league tables, and player statistics for specific leagues, making it a versatile tool for football data analysis.

Batch/stream ETL pipeline of NOAA GLM dataset, using Python frameworks: Dagster, PySpark and Parquet storage.

Airflow DAG tutorial with docker compose local setup

Optimizing offers/discounts send to Starbucks Clients using Machine Learning models on historical data.

LOGVERZ PORTAL ACCESS. Logverz portal access is the "login" component of the Logverz application bundle (LogverzReleases Repository). Logverz is a serverless adaptive data pipeline, the fastest route from AWS S3 to instant reports.

Pipeline de dados automatizado para extração e armazenamento de previsões meteorológicas para o setor de turismo.