Just wanting to Know whether we can use plugin to Backup/Restore Elastic Search 5.2 or newer version
codervarun opened this issue · comments
Please comment if anybody has idea. I can see that in commit somebody has written ES 5.2.
If plugin is made compatible for 5.x versions, then please write down the steps to download the plugin and install it.
Hi, we are in the process of refactoring the codebase. We have compiled it successfully against ES 5.2 but haven't yet released it officially because of said refactor. Watch this space :)
@synhershko Cool but It would be great if you can share the time when you are planning to release it as this would help us taking decision whether we can use swift to backup/restore ES indexes ?
Don't like to be a bearer of a bad news, but current Elastics search is more like 6.2 :-)
Just saying, no pressure...
We are well aware :) currently busy with shipping other products but this will happen very soon :)
Create tickets, happy to help! Now I do not know what your plan is, therefore hard to pitch in...
Hi @synhershko !
Is there any news from you side regarding Elastic >= 5.x support ?
Could you please push your modifications to a specific branch ? so that anyone can start reviewing and helping.
Regards,
Hi @synhershko : Any update for new release which supports 5.x and 6.x ?
Hi all,
A quick status update.
Since 5.x versions are too old at this point, we don't have plans to support them. Only 6.x.
I have just pushed some work + tests to fix some issues the plugin had, and make it compatible with 6.x. Currently it compiles for 6.0.0 but making it work with any 6.x version is rather trivial at this point.
We are still working on testing the plugin and making sure it works flawlessly - it previously had no tests and during the process of adding tests we indeed found various issues with it and fixed what we found.
If any of you can give it a try we will be more than happy to support your tests and fix whatever you might find. Compiling it at this point should be trivial but we can provide binaries for any version > 6.x in case you need it.
Thank you for your understanding.
@synhershko : I can test it in my openstack environment. Could you please tell me how to install test version of this plugin ?
git clone -> change the ES version in build.gradle -> gradle build -> install the plugin file from file path (in /build folder)
Build is failing
changed as below:
elasticsearch = '6.0.0' -->> elasticsearch = '5.4.3'
./gradlew build.gradle
Picked up _JAVA_OPTIONS: -Dhttps.proxyHost=xxxxxxx -Dhttps.proxyPort=8080
> Configure project :
=======================================
Elasticsearch Build Hamster says Hello!
=======================================
Gradle Version : 4.10.1
OS Info : Linux 4.4.0-134-generic (amd64)
JDK Version : Oracle Corporation 1.8.0_131 [Java HotSpot(TM) 64-Bit Server VM 25.131-b11]
JAVA_HOME : /usr/lib/jvm/java-8-oracle
Random Testing Seed : F3804EC030B83C32
FAILURE: Build failed with an exception.
* What went wrong:
Task 'build.gradle' not found in root project 'elasticsearch-repository-swift'.
* Try:
Run gradlew tasks to get a list of available tasks. Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 5.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/4.10.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 0s
Like I said, we are not going to support ES 5.x, only ES 6.x. Tho the error you pasted seems to be related to running gradle build in the wrong folder.
ok. I did not change version of elasticsearch this time. I followed below steps. however, i am getting same error. From which directory should i run gradle build ?
sakhtar@sakhtar:~/SRE/elasticsearch_backup$ git clone https://github.com/BigDataBoutique/elasticsearch-repository-swift.git
Cloning into 'elasticsearch-repository-swift'...
remote: Enumerating objects: 26, done.
remote: Counting objects: 100% (26/26), done.
remote: Compressing objects: 100% (17/17), done.
remote: Total 761 (delta 3), reused 17 (delta 1), pack-reused 735
Receiving objects: 100% (761/761), 279.75 KiB | 415.00 KiB/s, done.
Resolving deltas: 100% (312/312), done.
Checking connectivity... done.
sakhtar@sakhtar:~/SRE/elasticsearch_backup$ cd elasticsearch-repository-swift/
sakhtar@sakhtar:~/SRE/elasticsearch_backup/elasticsearch-repository-swift$ ls
build.gradle gradle gradlew gradlew.bat LICENSE NOTICE README.md settings.gradle src
sakhtar@sakhtar:~/SRE/elasticsearch_backup/elasticsearch-repository-swift$ ./gradlew build.gradle
Picked up _JAVA_OPTIONS: -Dhttps.proxyHost=x.x.x.x -Dhttps.proxyPort=8080
> Configure project :
=======================================
Elasticsearch Build Hamster says Hello!
=======================================
Gradle Version : 4.10.1
OS Info : Linux 4.4.0-134-generic (amd64)
JDK Version : Oracle Corporation 1.8.0_131 [Java HotSpot(TM) 64-Bit Server VM 25.131-b11]
JAVA_HOME : /usr/lib/jvm/java-8-oracle
Random Testing Seed : 885E5C8B865712B5
FAILURE: Build failed with an exception.
* What went wrong:
Task 'build.gradle' not found in root project 'elasticsearch-repository-swift'.
* Try:
Run gradlew tasks to get a list of available tasks. Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 5.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/4.10.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 0s
``
now i get below errors
sakhtar@sakhtar:~/SRE/elasticsearch_backup/elasticsearch-repository-swift$ ./gradlew build
> Configure project :
=======================================
Elasticsearch Build Hamster says Hello!
=======================================
Gradle Version : 4.10.1
OS Info : Linux 4.4.0-134-generic (amd64)
JDK Version : Oracle Corporation 1.8.0_131 [Java HotSpot(TM) 64-Bit Server VM 25.131-b11]
JAVA_HOME : /usr/lib/jvm/java-8-oracle
Random Testing Seed : 8ED6A944C4F9FB53
> Task :integTestCluster#installElasticsearchRepositorySwiftPlugin FAILED
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':integTestCluster#installElasticsearchRepositorySwiftPlugin'.
> A problem occurred starting process 'command '/home/sakhtar/SRE/elasticsearch_backup/elasticsearch-repository-swift/build/cluster/integTestCluster node0/elasticsearch-6.0.0/bin/elasticsearch-plugin''
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 5.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/4.10.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 6s
23 actionable tasks: 23 executed
sakhtar@sakhtar:~/SRE/elasticsearch_backup/elasticsearch-repository-swift$
Can you try again? it failed for me on first run and succeeds in subsequent runs
In subsequent runs it is successful. Thanks for your support
Where can i find artifacts and how can i install this plugin, i would like to keep these artifacts in artifactory.
> Configure project :
=======================================
Elasticsearch Build Hamster says Hello!
=======================================
Gradle Version : 4.10.1
OS Info : Linux 4.4.0-134-generic (amd64)
JDK Version : Oracle Corporation 1.8.0_131 [Java HotSpot(TM) 64-Bit Server VM 25.131-b11]
JAVA_HOME : /usr/lib/jvm/java-8-oracle
Random Testing Seed : 6FD239D6E76C650F
> Task :test
==> Test Info: seed=1DFDA67A3C7EE2FC; jvms=3; suites=3
==> Test Summary: 3 suites, 6 tests
> Task :integTestRunner
==> Test Info: seed=A18247DEC8C8546A; jvm=1; suite=1
==> Test Summary: 1 suite, 1 test
Deprecated Gradle features were used in this build, making it incompatible with Gradle 5.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/4.10.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD SUCCESSFUL in 25s
34 actionable tasks: 19 executed, 15 up-to-date
build/distributions/repository-swift-3.0.0-es6.0.0.zip
Installation will be like below ?
elasticsearch-plugin install/remove <artifactoryurl>/repository-swift-3.0.0-es6.0.0.zip
yes, please follow the docs in https://www.elastic.co/guide/en/elasticsearch/plugins/6.0/plugin-management-custom-url.html
@synhershko
Thanks alot for your support.
After installation of this plugin in all 3 nodes, I restarted elasticsearch service. Then when I try to crate repository with curl command provided. my cluster crash.
do I need to also configure path.repo in elasticsearch.yml file ?
Hello,
I've tried to build it againt elasticsearch 6.2.4 without success (using openJDK 9.0). The test task always fails. Am I missing something ?
Suite:org.wikimedia.elasticsearch.swift.SwiftBlobStoreTests
1> [2019-01-22T16:46:25,544][INFO ][o.w.e.s.SwiftBlobStoreTests] [testContainerCreationAndDeletion]: before test
1> [2019-01-22T16:46:25,918][INFO ][o.w.e.s.SwiftBlobStoreTests] [testContainerCreationAndDeletion]: after test
2> REPRODUCE WITH: gradle :test -Dtests.seed=FCD7975541B64D0C -Dtests.class=org.wikimedia.elasticsearch.swift.SwiftBlobStoreTests -Dtests.method="testContainerCreationAndDeletion" -Dtests.security.manager=true -Dtests.locale=mzn -Dtests.timezone=Asia/Novosibirsk
ERROR 0.49s J1 | SwiftBlobStoreTests.testContainerCreationAndDeletion <<< FAILURES!
Throwable #1: java.lang.NoClassDefFoundError: javax/activation/MimetypesFileTypeMap
at __randomizedtesting.SeedInfo.seed([FCD7975541B64D0C:EF9D89AAC4AE4329]:0)
at org.javaswift.joss.swift.SwiftStoredObject.uploadObject(SwiftStoredObject.java:86)
at org.javaswift.joss.swift.Swift.uploadObject(Swift.java:330)
at >
org.javaswift.joss.command.mock.object.UploadObjectCommandMock.callSwift(UploadObjectCommandMock.java:28)
at org.javaswift.joss.command.mock.core.CommandMock.call(CommandMock.java:62)
at org.javaswift.joss.client.core.AbstractStoredObject.directlyUploadObject(AbstractStoredObject.java:204)
at org.javaswift.joss.client.core.AbstractStoredObject.uploadObject(AbstractStoredObject.java:147)
at org.javaswift.joss.client.core.AbstractStoredObject.uploadObject(AbstractStoredObject.java:208)
at org.wikimedia.elasticsearch.swift.repositories.blobstore.SwiftBlobContainer.lambda$writeBlob$4(SwiftBlobContainer.java:172)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at org.wikimedia.elasticsearch.swift.SwiftPerms.exec(SwiftPerms.java:33)
at org.wikimedia.elasticsearch.swift.repositories.blobstore.SwiftBlobContainer.writeBlob(SwiftBlobContainer.java:171)
at org.elasticsearch.repositories.ESBlobStoreTestCase.writeBlob(ESBlobStoreTestCase.java:83)
at org.elasticsearch.repositories.ESBlobStoreTestCase.testContainerCreationAndDeletion(ESBlobStoreTestCase.java:44)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at java.base/java.lang.Thread.run(Thread.java:844)
Caused by: java.lang.ClassNotFoundException: javax.activation.MimetypesFileTypeMap
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:582)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:185)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:496)
... 49 more
2> NOTE: leaving temporary files on disk at: /home/quentin/git/elasticsearch-repository-swift/build/testrun/test/J1/temp/org.wikimedia.elasticsearch.swift.SwiftBlobStoreTests_FCD7975541B64D0C-001
2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=1146, maxMBSortInHeap=6.041239543663229, sim=RandomSimilarity(queryNorm=false): {}, locale=mzn, timezone=Asia/Novosibirsk
2> NOTE: Linux 4.20.2-107.current amd64/Oracle Corporation 9.0.4 (64-bit)/cpus=8,threads=1,free=496285304,total=536870912
2> NOTE: All tests run in this JVM: [SwiftBlobStoreTests]
Completed [1/3] on J1 in 2.20s, 1 test, 1 error <<< FAILURES!
Thanks for your work.