confluentinc / demo-scene

Scripts and samples to support Confluent Platform talks. May be rough around the edges. For automated tutorials and QA'd code, see https://github.com/confluentinc/examples/

Requirements

You need to allocate Docker 8GB when running these. Avoid allocating all your machine's cores to Docker as this may cause the machine to become unresponsive when running large stacks. On a four-core Mac Book two cores for Docker should be fine.

Contents

• Applications

  • Building a Telegram bot with Go, Apache Kafka, and ksqlDB (🎥 talk)
  • Streaming Pac-Man

• Data pipelines (Database + KSQL + Elasticsearch)

  • Pipeline to the cloud - on-premises RDBMS to Cloud datawarehouse e.g. Snowflake (✍️ blog)
  • Rail data streaming pipeline (🗣️talk)
  • Apache Kafka and KSQL in Action: Let’s Build a Streaming Data Pipeline! (🗣️talk)
  • MySQL / Debezium CDC / KSQL / Elasticsearch
  • Oracle CDC / KSQL / Elasticsearch
  • Postgres / Debezium CDC / KSQL / Elasticsearch
  • CDC demo with MySQL (🗣️talk)
  • CDC demo with Oracle
  • Building data pipelines with Confluent Cloud and GCP (BigQuery, GCS, etc)
  • MS SQL with Debezium and ksqlDB (✍️ blog)

• ksqlDB (previously known as KSQL)

  • Introduction to ksqlDB 01 (🗣️talk)
  • Introduction to KSQL 02
  • Using Twitter data with ksqlDB
  • KSQL UDF Advanced Example
  • KSQL Troubleshooting
  • ATM Fraud detection with Kafka and KSQL (🗣️talk)
  • Kafka Streams/KSQL Movie Demo
  • KSQL MQTT demo
  • KSQL Dump Utility
  • KSQL workshop (more recent version is here)
  • Multi-node ksqlDB and Kafka Connect clusters

• Kafka Connect

  • From Zero to Hero with Kafka Connect (🗣️talk)
  • Kafka Connect Converters and Serialization (✍️ blog)
  • Building a Kafka Connect cluster
  • Kafka Connect error handling (✍️ blog)
  • Multi-node ksqlDB and Kafka Connect clusters
  • Specific connectors
    • 👉 S3 Sink (🎥 tutorial)
    • 👉 Database (tutorial 🎥 1 / 🎥 2)
    • 👉 Elasticsearch (🎥 Tutorial)
    • Kafka Connect JDBC Source demo environment (✍️ blog)
    • InfluxDB & Kafka Connect (✍️ blog)
    • RabbitMQ into Kafka (✍️ blog)
    • MQTT Connect Connector Demo
    • Example Kafka Connect syslog configuration and Docker Compose (see blog series 1/2/3 and standalone articles here and here)
    • Azure SQL Data Warehouse Connector Sink Demo
    • IBM MQ Connect Connector Demo
    • Solace Sink/Source Demo
    • RSS feed into Kafka

• Confluent Cloud

  • Getting Started with Confluent Cloud using Java
  • Getting Started with Confluent Cloud using Go
  • Streaming Pac-Man
  • "The Cube" Demo
  • Using Replicator with Confluent Cloud

• Confluent Platform

  • Self-Balancing Clusters Demo
  • Tiered Storage Demo
  • Cluster Linking Demo

• Misc

  • Hacky export/import between Kafka clusters using kafkacat
  • Docker Compose for just the community licensed components of Confluent Platform
  • Topic Tailer, stream topics to the browser using websockets
  • KPay payment processing example
  • Industry themes (e.g. banking Next Best Offer)
  • Distributed tracing
  • Analysing Sonos data in Kafka (✍️ blog)
  • Analysing Wi-Fi pcap data with Kafka
  • Twitter streams and Operator

Feedback & Questions

Raise an issue on this github project, or head to http://cnfl.io/slack and join our Community Slack group.