marekschor / mql-sqlite3

SQLite3 binding for the MQL language (both 32bit MT4 and 64bit MT5)

mql-sqlite3

SQLite3 binding for the MQL language (both 32bit MT4 and 64bit MT5)

Introduction

This is a complete binding of the SQLite3 library (version 3180000) for the MQL4/5 language used by MetaTrader4/5.

SQLite is an fast embeded SQL engine written in C and widely used by a lot of projects. We always have the need to persistent states of Expert Advisors and SQLite seems to be the best solution.

This binding tries to remain compatible between MQL4/5. Users of both versions can use this binding, with a single set of headers. MQL4 and MQL5 are basically the same in that they are merged in recent versions. The difference is in the runtime environment (MetaTrader5 is 64bit by default, while MetaTrader4 is 32bit). The trading system is also different, but it is no concern of this binding.

Files

This binding contains three sets of files:

1. The binding itself is in the Include/SQLite3 directory.

2. There is a simple testing script called TestSQLite3.mq4 in Scripts/Test directory. The script files are mq4 by default, but you can change the extension to mq5 to use them in MetaTrader5. Currently there is only one script. I am going to add more in the future.

3. Precompiled DLLs of both 64bit (Library/MT5) and 32bit (Library/MT4) are provided. Copy the corresponding DLLs to the Library folder of your MetaTrader terminal. If you are using MetaTrader5 32bit, use the 32bit version from Library/MT4. The DLLs have no special dependencies and should work in any Windows version after NT. Note that these DLLs are copied from official binary release, without any modification. You can download/compile your own if you don't trust these binaries.

API comparison

Below is a detailed comparison table of MQL and C/C++ APIs. If you are reading the official documentation, and you want to find the corresponding MQL API for the C API, use this table. This table also has comments about why a particular API is not included in this binding. This table might change if new APIs get added.

SQLite3 C API	mql-sqlite3	Comment
sqlite3_aggregate_context	N/A	for extension
sqlite3_aggregate_count	N/A	deprecated
sqlite3_auto_extension	N/A	for extension
sqlite3_backup_finish	Backup::~Backup
sqlite3_backup_init	Backup::Backup
sqlite3_backup_pagecount	Backup::getPageCount
sqlite3_backup_remaining	Backup::getRemaining
sqlite3_backup_step	Backup::step
sqlite3_bind_blob	Statement::bind
sqlite3_bind_blob64	N/A	too large for ordinary use
sqlite3_bind_double	Statement::bind
sqlite3_bind_int	Statement::bind
sqlite3_bind_int64	Statement::bind
sqlite3_bind_null	Statement::bind
sqlite3_bind_parameter_count	Statement::getParameterCount
sqlite3_bind_parameter_index	Statement::getParameterIndex
sqlite3_bind_parameter_name	Statement::getParameterName
sqlite3_bind_text	Statement::bind
sqlite3_bind_text16	N/A	always UTF-8
sqlite3_bind_text64	N/A	too large for ordinary use
sqlite3_bind_value	N/A	for extension
sqlite3_bind_zeroblob	Statement::zero
sqlite3_bind_zeroblob64	Statement::zero
sqlite3_blob_bytes	Blob::size
sqlite3_blob_close	Blob::~Blob
sqlite3_blob_open	Blob::Blob
sqlite3_blob_read	Blob::read
sqlite3_blob_reopen	Blob::moveTo
sqlite3_blob_write	Blob::write
sqlite3_busy_handler	N/A	Need callback
sqlite3_busy_timeout	SQLite3::setBusyTimeout
sqlite3_cancel_auto_extension	N/A	for extension
sqlite3_changes	SQLite3::getChanges
sqlite3_clear_bindings	Statement::clearBindings
sqlite3_close	N/A	use v2
sqlite3_close_v2	SQLite3::~SQLite3
sqlite3_collation_needed	N/A	for extension
sqlite3_collation_needed16	N/A	for extension
sqlite3_column_blob	Statement::getColumn
sqlite3_column_bytes	Statement::getColumnBytes
sqlite3_column_bytes16	N/A	always UTF-8
sqlite3_column_count	Statement::getColumnCount
sqlite3_column_database_name	Statement::getColumnDatabaseName
sqlite3_column_database_name16	N/A	always UTF-8
sqlite3_column_decltype	Statement::getColumnDeclareType
sqlite3_column_decltype16	N/A	always UTF-8
sqlite3_column_double	Statement::getColumn
sqlite3_column_int	Statement::getColumn
sqlite3_column_int64	Statement::getColumn
sqlite3_column_name	Statement::getColumnName
sqlite3_column_name16	N/A	always UTF-8
sqlite3_column_origin_name	Statement::getColumnOriginName
sqlite3_column_origin_name16	N/A	always UTF-8
sqlite3_column_table_name	Statement::getColumnTableName
sqlite3_column_table_name16	N/A	always UTF-8
sqlite3_column_text	Statement::getColumn
sqlite3_column_text16	N/A	always UTF-8
sqlite3_column_type	Statement::getColumnType
sqlite3_column_value	N/A	for extension
sqlite3_commit_hook	N/A	need callback
sqlite3_compileoption_get	N/A	diagnostic/optional
sqlite3_compileoption_used	N/A	diagnostic/optional
sqlite3_complete	Statement::isComplete
sqlite3_complete16	N/A	always UTF-8
sqlite3_config	SQLite3::set* static methods	partial
sqlite3_context_db_handle	N/A	for extension
sqlite3_create_collation	N/A	for extension
sqlite3_create_collation16	N/A	for extension
sqlite3_create_collation_v2	N/A	for extension
sqlite3_create_function	N/A	for extension
sqlite3_create_function16	N/A	for extension
sqlite3_create_function_v2	N/A	for extension
sqlite3_create_module	N/A	for extension
sqlite3_create_module_v2	N/A	for extension
sqlite3_data_count	Statement::getDataCount
sqlite3_db_cacheflush	SQLite3::flush
sqlite3_db_config	SQLite3::setDbLookAside SQLite3::setMainDbName SQLite3::isTriggerEnabled SQLite3::setTriggerEnabled SQLite3::isForeignKeyEnabled SQLite3::setForeignKeyEnabled SQLite3::isFTS3TokenizerEnabled SQLite3::setFTS3TokenizerEnabled SQLite3::isLoadExtensionEnabled SQLite3::setLoadExtensionEnabled SQLite3::isCheckpointsOnCloseEnabled SQLite3::setCheckpointsOnCloseEnabled
sqlite3_db_filename	SQLite3::getDbFilename
sqlite3_db_handle	Statement::getConnectionHandle
sqlite3_db_mutex	N/A	internal use only
sqlite3_db_readonly	SQLite3::isReadonly	SQLite3::hasDb check if contains db
sqlite3_db_release_memory	SQLite3::releaseMemory
sqlite3_db_status	SQLite3::getStatus
sqlite3_declare_vtab	N/A	for extension
sqlite3_enable_load_extension	SQLite3::setLoadExtension
sqlite3_enable_shared_cache	SQLite3::setSharedCache
sqlite3_errcode	SQLite3::getErrorCode
sqlite3_errmsg	SQLite3::getErrorMsg
sqlite3_errmsg16	N/A	always UTF-8
sqlite3_errstr	SQLite3::errorCode2Msg
sqlite3_exec	N/A	convenience wrapper only
sqlite3_expanded_sql	SQLite3::getExpandedSql
sqlite3_expired	N/A	deprecated
sqlite3_extended_errcode	SQLite3::getExtendedErrorCode
sqlite3_extended_result_codes	SQLite3::setResultCodes
sqlite3_file_control	N/A	too low level
sqlite3_finalize	SQLite3::~SQLite3
sqlite3_free	N/A	internal use only
sqlite3_free_table	N/A	convenience wrapper only
sqlite3_get_autocommit	SQLite3::isAutoCommit
sqlite3_get_auxdata	N/A	for extension
sqlite3_get_table	N/A	convenience wrapper only
sqlite3_global_recover	N/A	deprecated
sqlite3_initialize	SQLite3::initialize	static
sqlite3_interrupt	SQLite3::interrupt
sqlite3_last_insert_rowid	SQLite3::getLastInsertRowId
sqlite3_libversion	SQLite3::getVersion	static
sqlite3_libversion_number	SQLite3::getVersionNumber	static
sqlite3_limit	SQLite3::setLimit
sqlite3_load_extension	SQLite3::loadExtension
sqlite3_log	N/A	for extension
sqlite3_malloc	N/A	internal use only
sqlite3_malloc64	N/A	internal use only
sqlite3_memory_alarm	N/A	deprecated
sqlite3_memory_highwater	SQLite3::getMemoryHighwater, SQLite3::resetMemoryHighwater
sqlite3_memory_used	SQLite3::getMemoryUsed
sqlite3_mprintf	N/A	internal use only
sqlite3_msize	N/A	internal use only
sqlite3_mutex_alloc	N/A	internal use only
sqlite3_mutex_enter	N/A	internal use only
sqlite3_mutex_free	N/A	internal use only
sqlite3_mutex_held	N/A	internal use only
sqlite3_mutex_leave	N/A	internal use only
sqlite3_mutex_notheld	N/A	internal use only
sqlite3_mutex_try	N/A	internal use only
sqlite3_next_stmt	N/A	no use
sqlite3_open	N/A	use v2
sqlite3_open16	N/A	always UTF-8
sqlite3_open_v2	SQLite3::SQLite3
sqlite3_os_end	N/A	internal use only
sqlite3_os_init	N/A	internal use only
sqlite3_overload_function	N/A	for extension
sqlite3_prepare	N/A	use v2
sqlite3_prepare16	N/A	always UTF-8
sqlite3_prepare16_v2	N/A	always UTF-8
sqlite3_prepare_v2	Statement::Statement
sqlite3_preupdate_count	N/A	for extension
sqlite3_preupdate_depth	N/A	for extension
sqlite3_preupdate_hook	N/A	for extension
sqlite3_preupdate_new	N/A	for extension
sqlite3_preupdate_old	N/A	for extension
sqlite3_profile	N/A	deprecated
sqlite3_progress_handler	N/A	need callback
sqlite3_randomness	N/A	use MQL Random
sqlite3_realloc	N/A	internal use only
sqlite3_realloc64	N/A	internal use only
sqlite3_release_memory	SQLite3::releaseMemoryInBytes	static
sqlite3_reset	Statement::reset
sqlite3_reset_auto_extension	N/A	for extension
sqlite3_result_blob	N/A	for extension
sqlite3_result_blob64	N/A	for extension
sqlite3_result_double	N/A	for extension
sqlite3_result_error	N/A	for extension
sqlite3_result_error16	N/A	for extension
sqlite3_result_error_code	N/A	for extension
sqlite3_result_error_nomem	N/A	for extension
sqlite3_result_error_toobig	N/A	for extension
sqlite3_result_int	N/A	for extension
sqlite3_result_int64	N/A	for extension
sqlite3_result_null	N/A	for extension
sqlite3_result_subtype	N/A	for extension
sqlite3_result_text	N/A	for extension
sqlite3_result_text16	N/A	for extension
sqlite3_result_text16be	N/A	for extension
sqlite3_result_text16le	N/A	for extension
sqlite3_result_text64	N/A	for extension
sqlite3_result_value	N/A	for extension
sqlite3_result_zeroblob	N/A	for extension
sqlite3_result_zeroblob64	N/A	for extension
sqlite3_rollback_hook	N/A	need callback
sqlite3_set_authorizer	N/A	need callback
sqlite3_set_auxdata	N/A	for extension
sqlite3_set_last_insert_rowid	SQLite3::setLastInsertRowId
sqlite3_shutdown	SQLite3::shutdown
sqlite3_sleep	N/A	use MQL Sleep
sqlite3_snapshot_cmp(exp)	N/A	experimental
sqlite3_snapshot_free(exp)	N/A	experimental
sqlite3_snapshot_get(exp)	N/A	experimental
sqlite3_snapshot_open(exp)	N/A	experimental
sqlite3_snapshot_recover(exp)	N/A	experimental
sqlite3_snprintf	N/A	internal use only
sqlite3_soft_heap_limit	N/A	deprecated
sqlite3_soft_heap_limit64	SQLite3::setSoftHeapLimit	static
sqlite3_sourceid	SQLite3::getSourceId	static
sqlite3_sql	Statement::getSql
sqlite3_status	N/A	use sqlite3_status64
sqlite3_status64	SQLite3::status
sqlite3_step	Statement::step
sqlite3_stmt_busy	Statement::isBusy
sqlite3_stmt_readonly	Statement::isReadonly
sqlite3_stmt_scanstatus	N/A	advanced/optional
sqlite3_stmt_scanstatus_reset	N/A	advanced/optional
sqlite3_stmt_status	Statement::getCounter, Statement::resetCounter
sqlite3_strglob	N/A	internal use only
sqlite3_stricmp	N/A	internal use only
sqlite3_strlike	N/A	internal use only
sqlite3_strnicmp	N/A	internal use only
sqlite3_system_errno	N/A	internal use only
sqlite3_table_column_metadata	SQLite3::getDbColumnMetadata
sqlite3_test_control	N/A	internal use only
sqlite3_thread_cleanup	N/A	deprecated
sqlite3_threadsafe	SQLite3::isThreadSafe
sqlite3_total_changes	SQLite3::getTotalChanges
sqlite3_trace	N/A	deprecated
sqlite3_trace_v2	N/A	need callback
sqlite3_transfer_bindings	N/A	deprecated
sqlite3_unlock_notify	N/A	need callback
sqlite3_update_hook	N/A	need callback
sqlite3_uri_boolean	N/A	for extension
sqlite3_uri_int64	N/A	for extension
sqlite3_uri_parameter	N/A	for extension
sqlite3_user_data	N/A	for extension
sqlite3_value_blob	N/A	for extension
sqlite3_value_bytes	N/A	for extension
sqlite3_value_bytes16	N/A	for extension
sqlite3_value_double	N/A	for extension
sqlite3_value_dup	N/A	for extension
sqlite3_value_free	N/A	for extension
sqlite3_value_int	N/A	for extension
sqlite3_value_int64	N/A	for extension
sqlite3_value_numeric_type	N/A	for extension
sqlite3_value_subtype	N/A	for extension
sqlite3_value_text	N/A	for extension
sqlite3_value_text16	N/A	for extension
sqlite3_value_text16be	N/A	for extension
sqlite3_value_text16le	N/A	for extension
sqlite3_value_type	N/A	for extension
sqlite3_version	N/A	not a function
sqlite3_vfs_find	Vfs::find, Vfs::getDefault
sqlite3_vfs_register	Vfs::register, Vfs::registerDefault
sqlite3_vfs_unregister	Vfs::unregister
sqlite3_vmprintf	N/A	internal use only
sqlite3_vsnprintf	N/A	internal use only
sqlite3_vtab_config	N/A	for extension
sqlite3_vtab_on_conflict	N/A	for extension
sqlite3_wal_autocheckpoint	SQLite3::setAutoCheckpoint
sqlite3_wal_checkpoint	N/A	use v2
sqlite3_wal_checkpoint_v2	SQLite3::checkpoint
sqlite3_wal_hook	N/A	need callback

About string encoding and API choices

MQL strings are Win32 UNICODE strings (basically 2-byte UTF-16). While for several APIs, SQLite3 provides both UTF-8 and UTF-16 versions, only UTF-8 strings are accepted in other APIs. For unification and maximum interoperabibility, we only use the UTF-8 version of the APIs. In this binding all strings are converted to utf-8 strings before sending to the dll layer. And strings in the database are also UTF-8 encoded.

About callbacks, internal APIs, and convenience wrappers

A lot of SQLite3 APIs are for internal use only, or for extension development, or need a C function as callback. These are not included in this binding. This will not affect the general applicability of this binding for most application level usage.

APIs like sqlite3_exec, sqlite3_get_table, etc. are convenience wrappers, which are not included. Maybe I will implement them in MQL.

Usage

You'd better read the official documentation or some books about SQLite before using this library. Documentation about this binding will be added when I have the time to finish them. You can find a simple test script in Scripts/Test. Here is a sample from TestSQLite3.mq4:

//+------------------------------------------------------------------+
//|                                                  TestSQLite3.mq4 |
//|                                          Copyright 2017, Li Ding |
//|                                            [e-mail] |
//+------------------------------------------------------------------+
#property copyright "Copyright 2017, Li Ding"
#property link      "[e-mail]"
#property version   "1.00"
#property strict

#include <SQLite3/Statement.mqh>
//+------------------------------------------------------------------+
//| Script program start function                                    |
//+------------------------------------------------------------------+
void OnStart()
  {
//--- optional but recommended
   SQLite3::initialize();

//--- ensure the dll and the lib is of the same version
   Print(SQLite3::getVersionNumber(), " = ", SQLITE_VERSION_NUMBER);
   Print(SQLite3::getVersion(), " = ", SQLITE_VERSION);
   Print(SQLite3::getSourceId(), " = ", SQLITE_SOURCE_ID);

//--- create an empty db
#ifdef __MQL5__
   string filesPath=TerminalInfoString(TERMINAL_DATA_PATH)+"\\MQL5\\Files";
#else
   string filesPath=TerminalInfoString(TERMINAL_DATA_PATH)+"\\MQL4\\Files";
#endif
   string dbPath=filesPath+"\\test.db";
   Print(dbPath);

   SQLite3 db(dbPath,SQLITE_OPEN_READWRITE|SQLITE_OPEN_CREATE);
   if(!db.isValid()) return;

   Print("DB created.");
   string sql="create table buy_orders"
              "(a int, b text);";
   if(Statement::isComplete(sql))
      Print(">>> SQL is complete");
   else
      Print(">>> SQL not complete");

   Statement s(db,sql);

   if(!s.isValid())
     {
      Print(db.getErrorMsg());
      return;
     }

   int r=s.step();
   if(r == SQLITE_OK)
      Print(">>> Step finished.");
   else if(r==SQLITE_DONE)
      Print(">>> Successfully created table.");
   else
      Print(">>> Error executing statement: ",db.getErrorMsg());

//--- optional but recommended
   SQLite3::shutdown();
  }

Here is a sample for select:

//+------------------------------------------------------------------+
//|                                            TestSQLite3Select.mq4 |
//|                                          Copyright 2018, Li Ding |
//|                                            [e-mail] |
//+------------------------------------------------------------------+
#property copyright "Copyright 2018, Li Ding"
#property link      "[e-mail]"
#property version   "1.00"
#property strict

#include <SQLite3/Statement.mqh>
//+------------------------------------------------------------------+
//| Script program start function                                    |
//+------------------------------------------------------------------+
void OnStart()
  {
//--- optional but recommended
   SQLite3::initialize();

//--- open database
#ifdef __MQL5__
   string filesPath=TerminalInfoString(TERMINAL_DATA_PATH)+"\\MQL5\\Files";
#else
   string filesPath=TerminalInfoString(TERMINAL_DATA_PATH)+"\\MQL4\\Files";
#endif
   string dbPath=filesPath+"\\test.db";
   Print(dbPath);

   SQLite3 db(dbPath,SQLITE_OPEN_READWRITE);
   if(!db.isValid()) return;

   string sql="select a, b from buy_orders;";

   Statement s(db,sql);

   if(!s.isValid())
     {
      Print(db.getErrorMsg());
      return;
     }

   int r=s.step();
   do
     {
      if(r==SQLITE_ROW)
        {
         Print(">>> New row!");
         int c=s.getColumnCount();
         for(int i=0; i<c; i++)
           {
            if(i==0)
              {
               int value;
               s.getColumn(i,value);
               Print(s.getColumnName(i),": ",value);
              }
            else if(i==1)
              {
               string value;
               s.getColumn(i,value);
               Print(s.getColumnName(i),": ",value);
              }
           }
        }
      else
         break;

      r=s.step();
     }
   while(r!=SQLITE_DONE);

//--- optional but recommended
   SQLite3::shutdown();
  }
//+------------------------------------------------------------------+

TODO

1. Write more tests.
2. Documentation.

Changes

• 2017-06-21: Initial version. Alpha quality, needs more tests.