realxujiang / aliyun-emapreduce-sdk

Hadoop/Spark on Aliyun, supporting interactions with Aliyun's base services.

E-MapReduce SDK

Requirements

• Spark 1.3+

Introduction

• This SDK supports interaction with Aliyun's base service, e.g. OSS, ODPS, LogService and ONS, in Spark runtime environment.

Build and Install

	    git clone https://github.com/aliyun/aliyun-spark-sdk.git
	    cd  aliyun-spark-sdk
	    mvn clean package -DskipTests

Use SDK in Eclipse project directly

• copy sdk jar to your project
• right click Eclipse project -> Properties -> Java Build Path -> Add JARs
• choose and import the sdk
• you can use the sdk in your Eclipse project

Maven

        <dependency>
            <groupId>com.aliyun.emr</groupId>
            <artifactId>emr-maxcompute_2.11</artifactId>
            <version>2.0.0-SNAPSHOT</version>
        </dependency>

        <dependency>
            <groupId>com.aliyun.emr</groupId>
            <artifactId>emr-logservice_2.11</artifactId>
            <version>2.0.0-SNAPSHOT</version>
        </dependency>

        <dependency>
            <groupId>com.aliyun.emr</groupId>
            <artifactId>emr-tablestore</artifactId>
            <version>2.0.0-SNAPSHOT</version>
        </dependency>

        <dependency>
            <groupId>com.aliyun.emr</groupId>
            <artifactId>emr-ons_2.11</artifactId>
            <version>2.0.0-SNAPSHOT</version>
        </dependency>

        <dependency>
            <groupId>com.aliyun.emr</groupId>
            <artifactId>emr-datahub_2.11</artifactId>
            <version>2.0.0-SNAPSHOT</version>
        </dependency>
        
        <dependency>
            <groupId>com.aliyun.emr</groupId>
            <artifactId>emr-jdbc_2.11</artifactId>
            <version>2.0.0-SNAPSHOT</version>
        </dependency>
        
        <dependency>
            <groupId>com.aliyun.emr</groupId>
            <artifactId>emr-druid_2.11</artifactId>
            <version>2.0.0-SNAPSHOT</version>
        </dependency>

        <dependency>
            <groupId>com.aliyun.emr</groupId>
            <artifactId>emr-hbase_2.11</artifactId>
            <version>2.0.0-SNAPSHOT</version>
        </dependency>

        <dependency>
            <groupId>com.aliyun.emr</groupId>
            <artifactId>emr-kudu_2.11</artifactId>
            <version>2.0.0-SNAPSHOT</version>
        </dependency>
        
        <dependency>
            <groupId>com.aliyun.emr</groupId>
            <artifactId>emr-redis_2.11</artifactId>
            <version>2.0.0-SNAPSHOT</version>
        </dependency>

        <dependency>
            <groupId>com.aliyun.emr</groupId>
            <artifactId>emr-oss</artifactId>
            <version>2.0.0-SNAPSHOT</version>
        </dependency>
        
        <dependency>
            <groupId>com.aliyun.emr</groupId>
            <artifactId>emr-common_2.11</artifactId>
            <version>2.0.0-SNAPSHOT</version>
        </dependency>

Run tests

• How to run tests

OSS support

• Hadoop/Spark on OSS

MaxCompute support

• Spark on MaxCompute

ONS support

• Spark on ONS

LogService support

• Spark on LogService

TableStore support

• HadoopMR on TableStore
• Spark on TableStore
• Hive/SparkSQL on TableStore

License

Licensed under the Apache License 2.0