Giters

druid-io / druid-io.github.io

Druid Project Website

Home Page:http://druid.io

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Druid -s3 Migration

BigDataArtist opened this issue · comments

commented

I am trying to migrate the data from druid to s3 bucket. Below are my configurations

Error

020-04-21T12:57:26,235 INFO [main] org.apache.druid.cli.CliPeon - * java.vm.specification.name: Java Virtual Machine Specification
2020-04-21T12:57:26,235 INFO [main] org.apache.druid.cli.CliPeon - * java.vm.specification.vendor: Oracle Corporation
2020-04-21T12:57:26,235 INFO [main] org.apache.druid.cli.CliPeon - * java.vm.specification.version: 1.8
2020-04-21T12:57:26,235 INFO [main] org.apache.druid.cli.CliPeon - * java.vm.vendor: Oracle Corporation
2020-04-21T12:57:26,236 INFO [main] org.apache.druid.cli.CliPeon - * java.vm.version: 25.241-b07
2020-04-21T12:57:26,236 INFO [main] org.apache.druid.cli.CliPeon - * line.separator:

2020-04-21T12:57:26,236 INFO [main] org.apache.druid.cli.CliPeon - * log4j.shutdownCallbackRegistry: org.apache.druid.common.config.Log4jShutdown
2020-04-21T12:57:26,236 INFO [main] org.apache.druid.cli.CliPeon - * log4j.shutdownHookEnabled: true
2020-04-21T12:57:26,236 INFO [main] org.apache.druid.cli.CliPeon - * os.arch: x86_64
2020-04-21T12:57:26,236 INFO [main] org.apache.druid.cli.CliPeon - * os.name: Mac OS X
2020-04-21T12:57:26,236 INFO [main] org.apache.druid.cli.CliPeon - * os.version: 10.15.1
2020-04-21T12:57:26,237 INFO [main] org.apache.druid.cli.CliPeon - * path.separator: :
2020-04-21T12:57:26,237 INFO [main] org.apache.druid.cli.CliPeon - * socksNonProxyHosts: local|.local|169.254/16|.169.254/16
2020-04-21T12:57:26,237 INFO [main] org.apache.druid.cli.CliPeon - * sun.arch.data.model: 64
2020-04-21T12:57:26,237 INFO [main] org.apache.druid.cli.CliPeon - * sun.boot.class.path: /Library/Java/JavaVirtualMachines/jdk1.8.0_241.jdk/Contents/Home/jre/lib/resources.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_241.jdk/Contents/Home/jre/lib/rt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_241.jdk/Contents/Home/jre/lib/sunrsasign.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_241.jdk/Contents/Home/jre/lib/jsse.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_241.jdk/Contents/Home/jre/lib/jce.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_241.jdk/Contents/Home/jre/lib/charsets.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_241.jdk/Contents/Home/jre/lib/jfr.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_241.jdk/Contents/Home/jre/classes
2020-04-21T12:57:26,238 INFO [main] org.apache.druid.cli.CliPeon - * sun.boot.library.path: /Library/Java/JavaVirtualMachines/jdk1.8.0_241.jdk/Contents/Home/jre/lib
2020-04-21T12:57:26,238 INFO [main] org.apache.druid.cli.CliPeon - * sun.cpu.endian: little
2020-04-21T12:57:26,238 INFO [main] org.apache.druid.cli.CliPeon - * sun.cpu.isalist:
2020-04-21T12:57:26,238 INFO [main] org.apache.druid.cli.CliPeon - * sun.io.unicode.encoding: UnicodeBig
2020-04-21T12:57:26,238 INFO [main] org.apache.druid.cli.CliPeon - * sun.java.command: org.apache.druid.cli.Main internal peon var/druid/task/index_parallel_test_ijbajejj_2020-04-21T12:57:10.447Z/task.json var/druid/task/index_parallel_test_ijbajejj_2020-04-21T12:57:10.447Z/32a4549d-e110-49a5-9e56-69e994674447/status.json var/druid/task/index_parallel_test_ijbajejj_2020-04-21T12:57:10.447Z/32a4549d-e110-49a5-9e56-69e994674447/report.json
2020-04-21T12:57:26,238 INFO [main] org.apache.druid.cli.CliPeon - * sun.java.launcher: SUN_STANDARD
2020-04-21T12:57:26,238 INFO [main] org.apache.druid.cli.CliPeon - * sun.jnu.encoding: UTF-8
2020-04-21T12:57:26,238 INFO [main] org.apache.druid.cli.CliPeon - * sun.management.compiler: HotSpot 64-Bit Tiered Compilers
2020-04-21T12:57:26,238 INFO [main] org.apache.druid.cli.CliPeon - * sun.os.patch.level: unknown
2020-04-21T12:57:26,239 INFO [main] org.apache.druid.cli.CliPeon - * user.country: US
2020-04-21T12:57:26,239 INFO [main] org.apache.druid.cli.CliPeon - * user.dir: /Users/adityaverma/Downloads/apache-druid-0.17.1
2020-04-21T12:57:26,239 INFO [main] org.apache.druid.cli.CliPeon - * user.home: /Users/adityaverma
2020-04-21T12:57:26,239 INFO [main] org.apache.druid.cli.CliPeon - * user.language: en
2020-04-21T12:57:26,239 INFO [main] org.apache.druid.cli.CliPeon - * user.name: adityaverma
2020-04-21T12:57:26,239 INFO [main] org.apache.druid.cli.CliPeon - * user.timezone: UTC
2020-04-21T12:57:29,816 ERROR [main] org.apache.druid.cli.CliPeon - Error when starting up. Failing.
com.google.inject.ProvisionException: Unable to provision, see the following errors:

  1. Error injecting constructor, java.lang.IllegalArgumentException: Can not create a Path from an empty string
    at org.apache.druid.storage.hdfs.HdfsDataSegmentKiller.(HdfsDataSegmentKiller.java:48)
    while locating org.apache.druid.storage.hdfs.HdfsDataSegmentKiller
    at org.apache.druid.storage.hdfs.HdfsStorageDruidModule.configure(HdfsStorageDruidModule.java:83) (via modules: com.google.inject.util.Modules$OverrideModule -> org.apache.druid.storage.hdfs.HdfsStorageDruidModule)
    while locating org.apache.druid.segment.loading.DataSegmentKiller annotated with @com.google.inject.multibindings.Element(setName=,uniqueId=155, type=MAPBINDER, keyType=java.lang.String)
    at org.apache.druid.guice.Binders.dataSegmentKillerBinder(Binders.java:40) (via modules: com.google.inject.util.Modules$OverrideModule -> org.apache.druid.storage.s3.S3StorageDruidModule -> com.google.inject.multibindings.MapBinder$RealMapBinder)
    while locating java.util.Map<java.lang.String, org.apache.druid.segment.loading.DataSegmentKiller>
    for the 1st parameter of org.apache.druid.segment.loading.OmniDataSegmentKiller.(OmniDataSegmentKiller.java:38)
    while locating org.apache.druid.segment.loading.OmniDataSegmentKiller
    at org.apache.druid.cli.CliPeon.bindPeonDataSegmentHandlers(CliPeon.java:362) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.cli.CliPeon$1)
    while locating org.apache.druid.segment.loading.DataSegmentKiller
    for the 6th parameter of org.apache.druid.indexing.common.TaskToolboxFactory.(TaskToolboxFactory.java:120)
    at org.apache.druid.cli.CliPeon.bindTaskConfigAndClients(CliPeon.java:400) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.cli.CliPeon$1)
    while locating org.apache.druid.indexing.common.TaskToolboxFactory
    for the 1st parameter of org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner.(SingleTaskBackgroundRunner.java:95)
    at org.apache.druid.cli.CliPeon$1.configure(CliPeon.java:210) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.cli.CliPeon$1)
    while locating org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner
    while locating org.apache.druid.indexing.overlord.TaskRunner
    for the 4th parameter of org.apache.druid.indexing.worker.executor.ExecutorLifecycle.(ExecutorLifecycle.java:80)
    at org.apache.druid.cli.CliPeon$1.configure(CliPeon.java:194) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.cli.CliPeon$1)
    while locating org.apache.druid.indexing.worker.executor.ExecutorLifecycle

1 error
at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1028) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1050) ~[guice-4.1.0.jar:?]
at org.apache.druid.guice.LifecycleModule$2.start(LifecycleModule.java:141) ~[druid-core-0.17.1.jar:0.17.1]
at org.apache.druid.cli.GuiceRunnable.initLifecycle(GuiceRunnable.java:115) [druid-services-0.17.1.jar:0.17.1]
at org.apache.druid.cli.CliPeon.run(CliPeon.java:280) [druid-services-0.17.1.jar:0.17.1]
at org.apache.druid.cli.Main.main(Main.java:113) [druid-services-0.17.1.jar:0.17.1]
Caused by: java.lang.IllegalArgumentException: Can not create a Path from an empty string
at org.apache.hadoop.fs.Path.checkPathArg(Path.java:163) ~[?:?]
at org.apache.hadoop.fs.Path.(Path.java:175) ~[?:?]
at org.apache.druid.storage.hdfs.HdfsDataSegmentKiller.(HdfsDataSegmentKiller.java:50) ~[?:?]
at org.apache.druid.storage.hdfs.HdfsDataSegmentKiller$$FastClassByGuice$$b7e9ca1b.newInstance() ~[?:?]
at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:89) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:111) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:90) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:268) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:56) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:194) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1019) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1015) ~[guice-4.1.0.jar:?]
at com.google.inject.spi.ProviderLookup$1.get(ProviderLookup.java:104) ~[guice-4.1.0.jar:?]
at com.google.inject.spi.ProviderLookup$1.get(ProviderLookup.java:104) ~[guice-4.1.0.jar:?]
at com.google.inject.multibindings.MapBinder$RealMapBinder$ValueProvider.get(MapBinder.java:821) ~[guice-multibindings-4.1.0.jar:?]
at com.google.inject.multibindings.MapBinder$RealMapBinder$RealMapProvider.get(MapBinder.java:605) ~[guice-multibindings-4.1.0.jar:?]
at com.google.inject.multibindings.MapBinder$RealMapBinder$RealMapProvider.get(MapBinder.java:586) ~[guice-multibindings-4.1.0.jar:?]
at com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:81) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.InternalFactoryToInitializableAdapter.provision(InternalFactoryToInitializableAdapter.java:53) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:61) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.InternalFactoryToInitializableAdapter.get(InternalFactoryToInitializableAdapter.java:45) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:110) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:90) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:268) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:56) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:194) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:110) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:90) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:268) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:194) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:110) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:90) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:268) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) ~[guice-4.1.0.jar:?]
at org.apache.druid.guice.LifecycleScope$1.get(LifecycleScope.java:68) ~[druid-core-0.17.1.jar:0.17.1]
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:56) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:110) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:90) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:268) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) ~[guice-4.1.0.jar:?]
at org.apache.druid.guice.LifecycleScope$1.get(LifecycleScope.java:68) ~[druid-core-0.17.1.jar:0.17.1]
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1019) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1085) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1015) ~[guice-4.1.0.jar:?]

DEEP Storage properties

common.runtime.properties

//I commented out for the localdisk

For local disk (only viable in a cluster if this is a network mount):

#druid.storage.type=local
#druid.storage.storageDirectory=var/druid/segments

For S3:

druid.storage.type=s3
druid.storage.bucket=druids3migration
druid.storage.baseKey=druid/segments
druid.s3.accessKeyi=AZKB47
druid.s3.secretKey=d1b*ABExuCxvxv3Use+Uzs

-Daws.region=ap-south-1 --> middleManager/jvm.properties

// middleManager/runtime.properties
druid.indexer.runner.javaOpts=-server -Xms1g -Xmx1g -XX:MaxDirectMemorySize=1g -Duser.timezone=UTC -Dfile.encoding=UTF-8 -XX:+ExitOnOutOfMemoryError -Daws.region=ap-south-1 -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager