bakhti / gomysql-playground

Data validator tool in Go

this document is created using emacs/org-mode - all the code blocks can be executed directly from emacs (with some changes in headers)

Prepare the database

docker-compose up -d mysql-1 mysql-2

Create the database and two tables

CREATE DATABASE IF NOT EXISTS validator;
CREATE TABLE validator.table1 (id bigint(20) AUTO_INCREMENT, data varchar(16), primary key(id));
CREATE TABLE validator.table2 (id bigint(20) AUTO_INCREMENT, data TEXT, primary key(id));

Seed some random data

DROP procedure if exists doSeeding;
delimiter //
CREATE PROCEDURE doSeeding()
BEGIN
  DECLARE i INT DEFAULT 1;
  WHILE i <= 350 DO
    INSERT INTO table1 (id, data) VALUES (i, LEFT(MD5(uuid()), 16));
    INSERT INTO table2 (id, data) VALUES (i, LEFT(MD5(uuid()), 16));
    SET i = i + 1;
  END WHILE;
END //
delimiter ;
call doSeeding;

Ideas for a start

• [ ] Connect to source, check if table exist, get its schema
• [ ] Get the greatest primary key in a table

SELECT MAX(id) FROM validator.table1

• [ ] Do the same for destination and compare result with the one from source
• [ ] Get the checksum of a random row from source and compare it to destination

SELECT MD5(CONCAT(id, IFNULL(data, ''))) FROM validator.table1 WHERE id = 1

Development

• local env
  • [ ] docker-compose
  • [ ] mysql-client
  • [ ] go
• ci/cd
  • [ ] travis

Notes

• Ghostferry is a great source of ideas
• https://github.com/siddontang/go-mysql
• https://github.com/go-sql-driver/mysql
• https://www.percona.com/doc/percona-toolkit/3.0/pt-table-checksum.html