migrate - a Tarantool rock for migration from Tarantool 1.5 to 1.10+
Tarantool 1.10.0+ with header files:
* tarantool && tarantool-dev packages on Ubuntu/Debian
* tarantool && tarantool-devel package on Fedora/Centos
* tarantool package on Mac OS X if you are using Homebrew
If building with LuaRocks:
* libsmall && libsmall-dev/devel
* msgpuck-dev/devel
Clone the repository with submodules and build it using CMake:
git clone --recursive https://github.com/tarantool/migrate.git
cd migrate && cmake . -DCMAKE_BUILD_TYPE=RelWithDebInfo
make
make install
You can also use LuaRocks:
luarocks install https://raw.githubusercontent.com/tarantool/migrate/master/migrate-scm-1.rockspec
See TarantoolRocks for LuaRocks configuration details.
Run tests using ctest
:
$ ctest
Test project <path>/xlog
Start 1: snap_test
1/3 Test #1: snap_test ........................ Passed 0.05 sec
Start 2: xlog_test
2/3 Test #2: xlog_test ........................ Passed 0.09 sec
Start 3: xdir_test
3/3 Test #3: xdir_test ........................ Passed 0.02 sec
100% tests passed, 0 tests failed out of 3
Total Test time (real) = 0.15 sec
box.cfg{
wal_mode = 'none',
logger = 'tarantool.log'
}
local s = box.schema.create_space('test_256', { if_not_exists = true })
local i = s:create_index('primary', {
type = 'TREE',
parts = {1, 'STR'},
if_not_exists = true
})
s:truncate()
local migrate = require('migrate')
local toolkit = migrate.reader({
dir = {
xlog = 'xlog' -- path to dir with xlogs
snap = 'snap' -- path to dir with snapshots
},
spaces = {
-- space id to look in snapshots/xlogs
[0] = {
new_id = s.name,
index = {
new_id = i.name,
parts = {1}
},
fields = {'str', 'num', 'num'},
default = 'str'
}
}
})
-- Load all available tuples/rows from snapshots/xlogs
toolkit:resume()
-- Do something to let the service know it works with the new Tarantool
-- ...
-- Load the rest of rows from xlogs
toolkit:resume()
-- box.snapshot() -- save snapshot
-- Now your Tarantool is ready to go!
local migrate = require('migrate')
Create a migration object. This function returns the new migration object.
Configuration is a table object that consists of:
dir = {xlog = 'xlog_dir', snap = 'snap_dir'}
ordir = 'xlog_snap_dir'
. Required field.spaces
- table with space definition, described later. Required field.throw
- boolean value that indicates whether to ignore error conversion or not. Errors happen if you cannot insert/delete/update or if you said that some field must be a number but its size is not 4 nor 8. Error messages are logged.true
by default.commit
- usebox.begin()
/box.commit()
before and after batch processing.true
by default.batch_count
- count of tuples/rows to read from disk, parse, and process at a time.500
by default.return_type
- type of objects to convert tuples to. Can be'table'
or'tuple'
(default). Use'table'
if you want to modify tuples and then push them into Tarantool (this way is slower).
spaces
is a table that associates old space number and table with definitions:
new_id
- string or number with spacename
/number
.index
- index definition table that includes:new_id
- new primary index name/number (string
ornumber
),parts
- table of key positions in primary index.
fields
- table of key types ({'str'/'num', ..}
).default
- default field type ('str'
/'num'
) if it is not defined in fields.insert
- custom insert functionfunction(tuple, flags)
. If not defined, a function that inserts atuple
into a space withnew_id
.delete
- custom delete functionfunction(key, flags)
. If not defined, a function that deletes a tuple with akey
from a space withnew_id
.update
- custom update functionfunction(key, ops, flags
. If not defined, a function that executesupdate
with operationsops
in a space withnew_id
on a tuple with PKkey
.
Resume loading of xlogs/snapshots. It uses a mechanism similar to that of Tarantool 1.5 to find all snapshots and xlogs necessary to load:
- When you run this method for the first time - it loads the last snapshot and xlogs.
- When you run this method next time and subsequently - it loads only the xlogs that contain rows with LSN greater than the last processed.