What's the correct way to create/migrate hbase tables using this gem?

Question

What's the correct way to create/migrate hbase tables using this gem?

pranny opened this issue 8 years ago · comments

So, I have been struggling to create a new hbase table using the gem. My model schema and code goes like

require 'massive_record'

class Enterprise < MassiveRecord::ORM::Table
  default_scope select(:info)

  column_family :info do
    field :id
    field :name
  end
end

MassiveRecord::ORM::Base.connection_configuration = {
  host: '127.0.0.1',
  port: 9090
}

Enterprise.table.save
#  => true. At this point the table exists in hbase but no column families defined

Enterprise.table.destroy
# => true

Enterprise.table.column_families
 => []

Enterprise.column_families
 => #<MassiveRecord::ORM::Schema::ColumnFamilies: {#<MassiveRecord::ORM::Schema::ColumnFamily:0x007fd51c3f5ba8 @fields=#<MassiveRecord::ORM::Schema::Fields: {#<MassiveRecord::ORM::Schema::Field:0x007fd51c3f55b8 @fields=#<MassiveRecord::ORM::Schema::Fields: {...}>, @name="id", @column=nil, @type=:string, @default=nil, @allow_nil=true, @coder=#<MassiveRecord::ORM::Coders::JSON:0x007fd51c75ea60>, @validation_context=nil, @errors={}>, #<MassiveRecord::ORM::Schema::Field:0x007fd51c3f4910 @fields=#<MassiveRecord::ORM::Schema::Fields: {...}>, @name="name", @column=nil, @type=:string, @default=nil, @allow_nil=true, @coder=#<MassiveRecord::ORM::Coders::JSON:0x007fd51c75ea60>, @validation_context=nil, @errors={}>}>, @name="info", @column_families=#<MassiveRecord::ORM::Schema::ColumnFamilies: {...}>, @autoload_fields=nil, @validation_context=nil, @errors={}>}>

MassiveRecord::ORM::Persistence::Operations::TableOperationHelpers.hbase_create_table!(Enterprise)
# Abort trap: 6 Really Weird. On OS X El Capitan

Enterprise.table.column_families = Enterprise.column_families

Enterprise.table.save
NoMethodError: undefined method `descriptor' for #<MassiveRecord::ORM::Schema::ColumnFamily:0x007f936290b1e8>
    from /Users/pranav/.rvm/gems/ruby-2.1.5/gems/massive_record-0.2.2/lib/massive_record/adapters/thrift/table.rb:48:in `block in save'
    from /Users/pranav/.rvm/rubies/ruby-2.1.5/lib/ruby/2.1.0/set.rb:263:in `each_key'
    from /Users/pranav/.rvm/rubies/ruby-2.1.5/lib/ruby/2.1.0/set.rb:263:in `each'
    from /Users/pranav/.rvm/gems/ruby-2.1.5/gems/massive_record-0.2.2/lib/massive_record/adapters/thrift/table.rb:48:in `collect'
    from /Users/pranav/.rvm/gems/ruby-2.1.5/gems/massive_record-0.2.2/lib/massive_record/adapters/thrift/table.rb:48:in `save'

Enterprise.table.init_column_families
# <MassiveRecord::Adapters::Thrift::Table:0x007fc96ab8bc90 @connection=#<MassiveRecord::Adapters::Thrift::Connection:0x007fc96ab7db90 @timeout=4000, @host="127.0.0.1", @port=9090, @instrumenter=#<ActiveSupport::Notifications::Instrumenter:0x007fc96b447618 @id="c1f9d1c99ab34dad716c", @notifier=#<ActiveSupport::Notifications::Fanout:0x007fc96ab3d770 @subscribers=[#<ActiveSupport::Notifications::Fanout::Subscriber:0x007fc96ab3d6f8 @pattern="load.massive_record", @delegate=#<MassiveRecord::ORM::LogSubscriber:0x007fc96ab9b6e0>>, #<ActiveSupport::Notifications::Fanout::Subscriber:0x007fc96ab3d5b8 @pattern="identity_map.massive_record", @delegate=#<MassiveRecord::ORM::LogSubscriber:0x007fc96ab9b6e0>>, #<ActiveSupport::Notifications::Fanout::Subscriber:0x007fc96ab3d450 @pattern="query.massive_record", @delegate=#<MassiveRecord::ORM::LogSubscriber:0x007fc96ab9b6e0>>, #<ActiveSupport::Notifications::Fanout::Subscriber:0x007fc96ab3d310 @pattern="logger.massive_record", @delegate=#<MassiveRecord::ORM::LogSubscriber:0x007fc96ab9b6e0>>], @listeners_for={"adapter_connecting.massive_record"=>[]}>>, @transport=#<Thrift::BufferedTransport:0x007fc96b4470f0 @transport=#<Thrift::Socket:0x007fc96b447168 @host="127.0.0.1", @port=9090, @timeout=4000, @desc="127.0.0.1:9090", @handle=#<Socket:fd 9>>, @wbuf="", @rbuf="\x80\x01\x00\x02\x00\x00\x00\vdeleteTable\x00\x00\x00\x00\x00", @index=24>, @client=#<Apache::Hadoop::Hbase::Thrift::Hbase::Client:0x007fc96b446f60 @iprot=#<Thrift::BinaryProtocol:0x007fc96b446f88 @trans=#<Thrift::BufferedTransport:0x007fc96b4470f0 @transport=#<Thrift::Socket:0x007fc96b447168 @host="127.0.0.1", @port=9090, @timeout=4000, @desc="127.0.0.1:9090", @handle=#<Socket:fd 9>>, @wbuf="", @rbuf="\x80\x01\x00\x02\x00\x00\x00\vdeleteTable\x00\x00\x00\x00\x00", @index=24>, @strict_read=true, @strict_write=true>, @oprot=#<Thrift::BinaryProtocol:0x007fc96b446f88 @trans=#<Thrift::BufferedTransport:0x007fc96b4470f0 @transport=#<Thrift::Socket:0x007fc96b447168 @host="127.0.0.1", @port=9090, @timeout=4000, @desc="127.0.0.1:9090", @handle=#<Socket:fd 9>>, @wbuf="", @rbuf="\x80\x01\x00\x02\x00\x00\x00\vdeleteTable\x00\x00\x00\x00\x00", @index=24>, @strict_read=true, @strict_write=true>, @seqid=0>>, @name="enterprises", @column_families=[], @table_exists=false>
Enterprise.table.save
# => true
Enterprise.find('1')
MassiveRecord::ORM::ColumnFamiliesMissingError: hbase are missing some column families for class 'Enterprise', table 'enterprises': info. Please migrate the database.
    from /Users/pranav/.rvm/gems/ruby-2.1.5/gems/massive_record-0.2.2/lib/massive_record/orm/finders.rb:187:in `rescue in query_hbase'
    from /Users/pranav/.rvm/gems/ruby-2.1.5/gems/massive_record-0.2.2/lib/massive_record/orm/finders.rb:175:in `query_hbase'
    from /Users/pranav/.rvm/gems/ruby-2.1.5/gems/massive_record-0.2.2/lib/massive_record/orm/finders.rb:138:in `find_one'
    from /Users/pranav/.rvm/gems/ruby-2.1.5/gems/massive_record-0.2.2/lib/massive_record/orm/identity_map.rb:157:in `find_one'
    from /Users/pranav/.rvm/gems/ruby-2.1.5/gems/massive_record-0.2.2/lib/massive_record/orm/query_instrumentation.rb:16:in `block in find_one'
    from /Users/pranav/.rvm/gems/ruby-2.1.5/gems/activesupport-3.0.20/lib/active_support/notifications.rb:52:in `block in instrument'
    from /Users/pranav/.rvm/gems/ruby-2.1.5/gems/activesupport-3.0.20/lib/active_support/notifications/instrumenter.rb:21:in `instrument'
    from /Users/pranav/.rvm/gems/ruby-2.1.5/gems/activesupport-3.0.20/lib/active_support/notifications.rb:52:in `instrument'
    from /Users/pranav/.rvm/gems/ruby-2.1.5/gems/massive_record-0.2.2/lib/massive_record/orm/query_instrumentation.rb:11:in `find_one'
    from /Users/pranav/.rvm/gems/ruby-2.1.5/gems/massive_record-0.2.2/lib/massive_record/orm/finders.rb:130:in `find_by_ids'
    from /Users/pranav/.rvm/gems/ruby-2.1.5/gems/massive_record-0.2.2/lib/massive_record/orm/finders.rb:108:in `do_find'
    from /Users/pranav/.rvm/gems/ruby-2.1.5/gems/massive_record-0.2.2/lib/massive_record/orm/finders/rescue_missing_table_on_find.rb:10:in `block in do_find'
    from /Users/pranav/.rvm/gems/ruby-2.1.5/gems/massive_record-0.2.2/lib/massive_record/orm/finders/rescue_missing_table_on_find.rb:32:in `create_table_and_retry_if_table_missing'
    from /Users/pranav/.rvm/gems/ruby-2.1.5/gems/massive_record-0.2.2/lib/massive_record/orm/finders/rescue_missing_table_on_find.rb:10:in `do_find'
    from /Users/pranav/.rvm/gems/ruby-2.1.5/gems/massive_record-0.2.2/lib/massive_record/orm/single_table_inheritance.rb:26:in `do_find'
    from /Users/pranav/.rvm/gems/ruby-2.1.5/gems/massive_record-0.2.2/lib/massive_record/orm/finders/scope.rb:99:in `find'
    from /Users/pranav/.rvm/gems/ruby-2.1.5/gems/massive_record-0.2.2/lib/massive_record/orm/finders.rb:8:in `find'

I tried a bunch of other similar methods, but I am unable to create a table with appropriate column family. What am I missing here? I am using the cloudera version of hbase.

Pranav Prakash · Answer 1 · Sun Sep 11 2016 01:05:35 GMT+0800 (China Standard Time)

Even using the raw adapter throws exception

require 'massive_record'
conn = MassiveRecord::Wrapper::Connection.new(:host => '127.0.0.1', :port => 9090)
# => #<MassiveRecord::Adapters::Thrift::Connection:0x007fedbc5f0d50 @timeout=4000, @host="127.0.0.1", @port=9090, @instru..

conn.open
# => true

 table = MassiveRecord::Wrapper::Table.new(conn, :people)
# => #<MassiveRecord::Adapters::Thrift::Table:0x007fedbc4f2b38 @connec....

column = MassiveRecord::Wrapper::ColumnFamily.new(:info)
# => #<MassiveRecord::Adapters::Thrift::ColumnFamily:0x007fedbc4e19c8 @name=:info, @max_versions=10, @columns=[]>

table.column_families.push(column)
# => [#<MassiveRecord::Adapters::Thrift::ColumnFamily:0x007fedbc4e19c8 @name=:info, @max_versions=10, @columns=[]>]

table.save
# Abort trap: 6 !!

If I skip the table.column_families.push(column) step, then the table is created but without column_family

Pranav Prakash · Answer 2 · Sun Sep 11 2016 01:08:58 GMT+0800 (China Standard Time)

I am on ruby 2.1.5. Could that be a pain? I tried downgrading and using 1.9.3 (since rvm couldn't find anything for 1.9.2) and most of my app is designed for ruby > 2.1. So downgrading doesn't look like an option here.

Vincent Peres · Answer 3 · Mon Sep 12 2016 19:16:36 GMT+0800 (China Standard Time)

Sorry we haven't tested with ruby > 1.9.3. It might not work there. Using the ORM, tables are created (with column families) whenever you perform an action toward the database. It will look up for the table name in the list of existing tables, and create it if missing.

Pranav Prakash · Answer 4 · Mon Sep 12 2016 20:20:46 GMT+0800 (China Standard Time)

In the meanwhile, I have this gist which provides a work around this problem.