rmoff / flink-jdbc

Apache Flink writing and writing from Postgres with JDBC Catalog

Launch stack

docker compose build
docker compose up

Create a table and populate it in Postgres

Connect to postgres:

docker compose exec -it postgres psql --user postgres

Create and populate table

CREATE TABLE t_foo (c1 varchar, c2 int);
INSERT INTO t_foo VALUES ('a',42);

Do SQL Stuff in Flink

Run SQL Client

docker compose exec -it flink ./bin/sql-client.sh

Define catalog:

CREATE CATALOG c_jdbc WITH (
   'type' = 'jdbc',
   'base-url' = 'jdbc:postgresql://postgres:5432',
   'default-database' = 'postgres',
   'username' = 'postgres',
   'password' = 'postgres'
   );

Set catalog as active:

USE CATALOG `c_jdbc`;

List Postgres tables:

Flink SQL> SHOW TABLES;
+--------------+
|   table name |
+--------------+
| public.t_foo |
+--------------+
1 row in set

Read some data:

Flink SQL> SET 'execution.runtime-mode' = 'batch';
>
[INFO] Execute statement succeed.

Flink SQL> SET 'sql-client.execution.result-mode' = 'tableau';
[INFO] Execute statement succeed.

Flink SQL> SELECT * FROM `public.t_foo`;
+----+----+
| c1 | c2 |
+----+----+
|  a | 42 |
+----+----+
1 row in set

Write some data:

Flink SQL> INSERT INTO t_foo VALUES ('foo',0);
[INFO] Submitting SQL update statement to the cluster...
[INFO] SQL update statement has been successfully submitted to the cluster:
Job ID: e150849e90fcaa91f3f00459ec0e8010

Check the data back in psql:

postgres=# SELECT * FROM t_foo;
 c1  | c2
-----+----
 a   | 42
 foo |  0
(2 rows)

Poke around

Flink Dashboard: http://localhost:8081/

Logs from Flink

docker compose exec -t flink bash -c "tail -f log/*"