AWS-Keyspaces-Cassandra-IOT-Drilling-App

This repo contains source code for a real-time well data drilling application.

The application can be used to deliver real-time well drilling operation data into Amazon Keyspaces (for Apache Cassandra) for subsequent use in AIML applications, including:

(1) Anonamly Detections.
(2) Classifications.

The drilling operation could be any of the followings:

1. Oil and gas drilling operation (For shale, conventional, and heavy oil/bitumen reservoirs).
2. Water well drilling operation.
3. Geothermal well drilling operation for renewable energy.

The data can be streamed by the application directly or through an intermediate file (.CSV) that is offloaded to S3 bucket at set intermittent interval.

The source code files include:

1. keyspaces_drlling_client.py - Main Python client application code for:
   a) Connecting to the DBaaS.
   b) Querying (DML, DQL and DDL queries) DBaaS.
   

2. keyspaces_connection_options.json - json file for specifying all connection options, which include:
   a) Amazon Keyspaces service-specific credentials.
   b) Connection port.
   c) The DBaaS endpoint.
   d) SSL certificate path.
   

3. keyspaces_drilling_cfn.yaml - AWS Cloud Formation IaC stack for:
   a) Creating the NoSQL DBaaS (Amazon Keystore).
   b) Defining (ddl) the Data Model (keyspaces, tables and parametized unique tags).

4. ddl_timeseries_keyspace.cql and ddl_drilling_table.cql - Cassandra Query Language - CQL's DDL equivalent of of the keyspaces_drilling_cfn.yaml file. These can be used to define the DBaaS' keyspace and table via:
   a) AWS Management Console's CQL Editor or
   b) CQLSH (CSL Shell) using Amazon Keyspaces service-specific credentials.
   

5. dml_insert.cql - sample dml for inserting data into the DBaaS.

6. dml_update.cql - sample dml for updating data on the DBaaS.

7. dml_delete.cql - sample dml for deleting data on the DBaaS.

8. dql_select.cql - sample dql for selecting data from the DBaaS.

9. ddl_alter.cql - sample ddl for altering table capacity units on the DBaaS.

The repo is based on the following languages, frameworks, packages and database servers:

1. Python v3.8 - (https://www.python.org/downloads/release/python-380/)
2. Java 8 - Java-8-openjdk (https://openjdk.java.net/)
3. AWS SDK for Python - Boto3 (https://boto3.amazonaws.com/v1/documentation/api/latest/index.html)
4. Python Driver for Apache Cassandra (https://github.com/datastax/python-driver)
5. CQL Language Reference for Amazon Keyspaces (for Apache Cassandra) - (https://docs.aws.amazon.com/keyspaces/latest/devguide/cql.html)
6. Amazon Keyspaces (for Apache Cassandra) - (https://docs.aws.amazon.com/keyspaces/latest/devguide/what-is-keyspaces.html)
7. Apache Cassandra (includes CQLSH) - (https://cassandra.apache.org/)

License

Copyright © 2015 - present. MongoExpUser

Licensed under the MIT license.