Hello Hello! I Detected a Change on This Database.

The story of this tryout is as follows:

In this tryout I used debezium/example-mysql:1.4 Docker image for the database server and it comes with an example database called inventory along with some example tables with sample data.

Let’s have a look at our producer (Debezium MySQL connector) configuration (register-mysql.json):

Our Debezium connector will listen to changes on the inventory database and publish the changes to dbserver1.inventory prefixed topic (if changes are detected on customers table, the related topic will be dbserver1.inventory.customers) of the Kafka cluster.

And now it’s time to check our consumer (Apache Pinot) configuration (pinot/def-table.json):

This is a table definition for Apache Pinot and it also specifies from which Kafka topic it will automatically pull the messages and in this tryout, it is dbserver1.inventory.customers.

Finally, we are using the Apache Pinot UI Query Console to execute our SQL queries to observe the changes. (Most probably, in a production environment, the API of Apache Pinot will be preferred. ✅ )

Note: If you get Docker Container exited with code 137 errors during running and experimenting, chances are that your Docker Desktop -> Resources -> Memory amount is not enough. I have set my resource settings as follows: CPUs 6, Memory 8 GB, Swap 1 GB.

Open up a terminal and browse to the cloned folder and execute the following command to see the magic happen:

docker-compose up

Or to have everything run at the background silently, add -d

docker-compose up -d

Open another terminal and execute a script to start Debezium MySQL connector and create table in Apache Pinot::

./run.sh

Browse Apache Pinot UI to query some data

Open an SQL client and use the following information to connect:

Note: Some SQL clients (like Sequel Pro) are incompatible with the Docker image.

Insert new records to / update existing records in / delete records from customers table of the inventory database.

Execute the following SQL to see the captured changes:

Related posts:

While volatility is nice when you’re looking for an investment and can give you an indication of the risk-reward ratio, it’s exactly the opposite of what you would look for when it comes to building…

7 anak yang ada di rumah kini sedang berada di lantai 1 rumah mereka. Ada yang di dapur dan ada yang bersantai di ruang tamu Kini Gerrel bangun dari duduknya dan entah apa tujuannya, ia melempar…

CXFS is one of the world’s first trade shows dedicated to customer experience in the financial services industry. CXFS is held annually in Boston; show attendance grew by 62% this year and I met…