Make easy for new contributors discover and test against databases and remote filesystems

[Juarez Rudsatz]

Problem description

• Today we use tox and tox.ini for local testing
• But testing against databases and Remote Filesystem is not easy discoverable.
• So local testing is skipped most of the times

TODO

• Create docker-compose / docker setup for running the needed databases and remote filesystems
• Make it easy for the contributor set up the environment and run the tests
• Hook it in IDE like VSCode, and others
• Replace the hacks in the CI tests with a proper solution
• Enable using docker testing of hard to set up formats like:
  • HDF5
  • Bcolz/ctabls
  • whoosh

CI Test hacks

Databases

• MySql:
  • create
  • fromdb/todb
• PostgreSQL:
  • Create
  • fromdb/todb

Remote Filesystems

• This functionality needs fsspec
• For SMB:
  • Container setup command:
    • docker run -it --name samba -p 139:139 -p 445:445 -d "dperson/samba" -p -u "petl;test" -s "public;/public-dir;yes;no;yes;all"
  • Environment variable used:
    • PETL_TEST_SMB=smb://WORKGROUP;petl:test@localhost/public/
• For SFTP:
  • Container setup command:
    • docker run -it --name sftp -p 2244:22 -d atmoz/sftp petl:test:::public
  • Environment variable used:
    • PETL_TEST_SFTP=sftp://petl:test@localhost:2244/public/
• For testing additional Built-in fsspec implementations:
  • Works with a similar environment variable: PETL_TEST_XXXXX
  • In conjunction with a URI understood by fspec
• For testing additional Cloud fsspec implementations.
  • Idem.