Whether your organization needs to monitor cybersecurity threats, fraudulent financial transactions, product defects, or equipment health, artificial intelligence can help catch data abnormalities before they impact your business. AI models can be trained and deployed to automatically analyze datasets, define “normal behavior,” and identify breaches in patterns quickly and effectively. These models can then be used to predict future anomalies. With massive amounts of data available across industries and subtle distinctions between normal and abnormal patterns, it’s critical that organizations use AI to quickly detect anomalies that pose a threat.
In this workshop, we learned how to identify anomalies and failures in time-series data, estimate the remaining useful life of the corresponding parts, and map anomalies to failure conditions. More specifically, how to prepare time-series data for AI model training, develop an XGBoost ensemble tree model, build a deep learning model using a long short-term memory (LSTM) network, and create an autoencoder that detects anomalies for predictive maintenance. At the end of the workshop, we are able to use AI to estimate the condition of equipment and predict when maintenance should be performed.
The overall goals of this course were the following:
- Prepare data and build, train, and evaluate models using XGBoost, autoencoders, and GANs;
- Detect anomalies in datasets with both labeled and unlabeled data;
- Classify anomalies into multiple categories regardless of whether the original data was labeled.
More detailed information and links for the course can be found on the course website.
The description of the files in this repository can be found below:
- 1 - Anomaly Detection in Network Data Using GPU-Accelerated XGBoost:
- Lab1-Anomaly Detection in Network Data using GPU-Accelerated XGBoost - Notebook;
- Lab1-preprocess_data - Script;
- Lab1-Presentation - Slides;
Learn how to detect anomalies using supervised learning:
- Prepare data for GPU acceleration using the provided dataset.
- Train a binary and multi-class classifier using the popular machine learning algorithm XGBoost.
- Assess and improve your model’s performance before deployment.
- 2 - Anomaly Detection in Network Data Using GPU-Accelerated Autoencoder:
- Lab2-Network Anomaly Detection using Autoencoders - Notebook;
- Lab2-preprocess_data - Script;
- Lab2-Presentation - Slides;
Learn how to detect anomalies using modern unsupervised learning:
- Build and train a deep learning-based autoencoder to work with unlabeled data.
- Apply techniques to separate anomalies into multiple classes.
- Explore other applications of GPU-accelerated autoencoders.
- 3 - Anomaly Detection in Network Data Using GANs:
- Lab3-Anomaly Detection in Network Data using GANS - Notebook;
- Lab3-preprocess_data - Script;
- Lab3-Presentation - Slides;
Learn how to detect anomalies using GANs:
- Train an unsupervised learning model to create new data.
- Use that new data to turn the problem into a supervised learning problem.
- Compare the performance of this new approach to more established approaches.
- 4 - Workshop Assessment - Building and training an Xgboost model, an autoencoder neural network and detecting anomalies using different thresholding methods:
- Workshop Assessment - Notebook;
- Assessment-preprocess_data - Script;
- Assessment-Presentation - Slides;
Learn how to detect anomalies using GANs:
- Train an unsupervised learning model to create new data.
- Use that new data to turn the problem into a supervised learning problem.
- Compare the performance of this new approach to more established approaches.
The certificate for the workshop can be found below:
"Applications of AI for Anomaly Detection" - NVIDIA Deep Learning Institute (Issued On: March 2023 - date mismatch because of account change)