https://javipc.github.io/bigdata/

Tests with Spark local, kafka streaming and socket streaming

CS 143 Project 2 Code: Implement Disk Based Hashing for SQL DataBase

Official repo for DK908 - IOT Big Data Processing

Spark examples by Scala

A method for detection of DoS/DDoS attacks based on an evaluation of the incoming/outgoing packet volume ratio and its variance to the long-time ratio.

Spark kafka pipeline for Anomaly detection

Projects related to Big Data technologies

Implementation of Data Mining algorithms

Amazon Customer Reviews (a.k.a. Product Reviews) is one of Amazon’s iconic products. In a period of over two decades since the first review in 1995, millions of Amazon customers have contributed over a hundred million reviews to express opinions and describe their experiences regarding products on the Amazon.com website. Over 130+ million custom…

Spark Streaming Big Data Hadoop

spark streaming process for data coming from kafka

Linear Regression in PySpark

As a Data Engineer for a fictional E-commerce startup, this project addresses the task of analyzing the web server logs to find the number of product pages visited and the number of items in the cart.

Analysis for a streaming daily retail data using Spark structured streaming and querying this data to get insights

This project is basically for collecting enormous data and analyzing it. It includes live streaming of data from FOREX trading API and Electric Vehicle stocks API. The data is fetched and processed using Kafka Streaming and Spark streaming.Throughout this project stocks of Forex data and Electric Vehicle parts making companies data were analyzed…

divide city to some points as nodes of a traffic graph and process location data from kafka and calculate edge between nodes with spark aggregations

Transcription and NER in streaming environment