Postgres data pipeline - prepare the source data
The source data comes from a PostgreSQL server that has to meet the minimal configuration mentioned here.
PostgreSQL in Docker
Docker is a simple way to start playing with a local database and data pipelines. To capture changes from PostgreSQL, the Write Ahead Log level has to be set at least to logical
. By default, the PostgreSQL docker image has its Log Level set at level lower than this; thus you must specify it explicitly when the image is run:
docker run -tid -p 5432:5432 -e POSTGRES_PASSWORD=docker -e PGDATA=/tmp postgres:12.6-alpine -c wal_level=logical
Database Table
Following is the SQL to create the table that will be used as a data source. Any record inserted in the table will be processed by the data pipeline in the Genesis platform:
CREATE TYPE valid_sides AS ENUM ('buy', 'sell');
CREATE TYPE valid_states AS ENUM ('new', 'mod', 'canc');
CREATE TABLE trades (
trd_id VARCHAR ( 12 ) PRIMARY KEY,
inst VARCHAR ( 5 ) NOT NULL,
price DOUBLE PRECISION NOT NULL,
quantity INTEGER NOT NULL,
side VALID_SIDES NOT NULL,
traded_at TIMESTAMP NOT NULL,
trader VARCHAR (30) NOT NULL,
trade_state valid_states NOT NULL,
unsolicited BOOL,
orig_trd_id VARCHAR (12),
trade_notes VARCHAR (500)
);