This repository contains examples of use cases that utilize Decodable streaming solution as well as demos for related open-source projects such as Apache Flink, Debezium, and Postgres.
| Example | Description |
|---|---|
| Flink Learn | Apache Flink tutorials and webinar |
| AsyncAPI | Publishing Data Products with AsyncAPI |
| Opinionated Data Pipelines | Building data pipelines with schema on write streams. |
| Postman | Building data pipelines with Postman. |
| Change Streams | Using change streams to build materialized views in Postgres |
| XML Processing | Parse XML and transform to JSON |
| OSQuery Routing | Route OSQuery logs with SQL |
| Masking | Ways to mask data |
| Apache Pinot | Transforming osquery logs to Apache Pinot and Superset |
| Apache Druid | This example sends covid 19 data to Decodable using it's REST API. The data is then cleansed using Decodable SQL and send the data to a Kafka sink. |
| Rockset | We will be utilizing a cloud MQTT broker and AWS Kinesis to capture and stream data. Decodable will be responsible for preparing and aggregating the data prior to reaching the real-time analytical database (Rockset) |
| Tinybird | We write data to Tinybird and build a simple real time web application. |
| Apache Kafka | Installing Apache Kafka on EC2 and writing to S3 with Decodable |
| Apache Kafka mTLS | We install Apache Kafka on EC2 and configure it with mTLS and configure Decodable to read from it |
| Snowflake + Snowpipe | We setup a snowpipe at the end of a Decodable S3 sink. |
| Confluent | Clickstream from Confluent Cloud joined with CDC user data from Postgres |
| Snowflake + Snowpipe + Merge | Leveraging Snowpipe, we send CDC data from Postgres to be processed in Snowflake using an Append Only Stream in Snowflake to merge CDC data in a Snowflake table. Essentially mirroring the table in Postgres in a Snowflake table. |
| Reading S3 Events in a Lambda Function | We configure an S3 bucket with a Lambda notification to send data to Kinesis to be processed in Decodable |
| MSSQL CDC | We enable a MSSQL in Docker with CDC. We then stand up a Debezium server to read from MSSQL and write the change events into AWS Kinesis |
| Oracle CDC | We configure a Oracle AWS RDS with LogMiner. We then stand up a Debezium server to read change events into AWS Kinesis |
| DynamoDb CDC | We configure a DynamoDB to send change data to Kinesis. Then we read those changes into Decodable for transformation or replication. |
| Logical Decoding Message Examples | We show how to retrieve logical decoding messages from the Postgres WAL |
| GitHub Webhooks | We show how to process GitHub Webhook events using the Decodable REST source connector |
| PyFlink | We run a basic PyFlink job on Kubernetes |
| Kafka / Flink / Iceberg | Integrating Apache Kafka with Apache Iceberg through Apache Flink. As presented at Kafka Summit London 2024 |
| Flink SQL Troubleshooting | A set of Docker Compose environments for demonstrating various Flink SQL troubleshooting scenarios (see related blog) |
| Array Aggregation | Using the array_agg() UDF for denormalizing data in a pipeline from MySQL to OpenSearch |
| Kafka with ngrok | Docker Compose for running Apache Kafka locally, accessible from the internet using ngrok |
This code base is available under the Apache License, version 2.