Developers want to run kylin test cases or applications at their development machine.
By following this tutorial, you will be able to build kylin test cubes by running a specific test case, and you can further run other test cases against the cubes having been built.
Environment on the Hadoop CLI
Off-Hadoop-CLI installation requires you having a hadoop CLI machine (or a hadoop sandbox) as well as your local develop machine. To make things easier we strongly recommend you starting with running Kylin on a hadoop sandbox, like http://hortonworks.com/products/hortonworks-sandbox/. In the following tutorial we’ll go with Hortonworks Sandbox 2.2.4. It is recommended that you provide enough memory to your sandbox, 8G or more is preferred.
In Hortonworks sandbox, ambari helps to launch hadoop:
ambari-agent start ambari-server start
With both command successfully run you can go to ambari home page at http://yoursandboxip:8080 (user:admin,password:admin) to check everything’s status. By default ambari disables HBase, you’ll need manually start the
For other hadoop distribution, basically start the hadoop cluster, make sure HDFS, YARN, Hive, HBase are running.
Environment on the dev machine
The latest maven can be found at http://maven.apache.org/download.cgi, we create a symbolic so that
mvn can be run anywhere.
cd ~ wget http://xenia.sote.hu/ftp/mirrors/www.apache.org/maven/maven-3/3.2.5/binaries/apache-maven-3.2.5-bin.tar.gz tar -xzvf apache-maven-3.2.5-bin.tar.gz ln -s /root/apache-maven-3.2.5/bin/mvn /usr/bin/mvn
Manually install spark-1.6.3-bin-hadoop2.6 in a local folder like /usr/local/spark
wget -O /tmp/spark-1.6.3-bin-hadoop2.6.tgz http://d3kbcqa49mib13.cloudfront.net/spark-1.6.3-bin-hadoop2.6.tgz cd /usr/local tar -zxvf /tmp/spark-1.6.3-bin-hadoop2.6.tgz ln -s spark-1.6.3-bin-hadoop2.6 spark
Upload the spark-assembly jar to HDFS as /kylin/spark/spark-assembly-1.6.3-hadoop2.6.0.jar (avoid repeatedly uploading the jar to HDFS):
hadoop fs -mkdir /kylin/spark/ hadoop fs -put /usr/local/spark/lib/spark-assembly-1.6.3-hadoop2.6.0.jar /kylin/spark/
Create local temp folder for hbase client (if it doesn’t exist):
mkdir -p /hadoop/hbase/local/jars chmod 777 /hadoop/hbase/local/jars
First clone the Kylin project to your local:
git clone https://github.com/apache/kylin.git
Install Kylin artifacts to the maven repo
mvn clean install -DskipTests
Modify local configuration
Local configuration must be modified to point to your hadoop sandbox (or CLI) machine.
- In examples/test_case_data/sandbox/kylin.properties
sandboxand replace with your hadoop hosts (if you’re using HDP sandbox, this can be skipped)
kylin.job.run.as.remote.cmdand change it to “true” (in code repository the default is false, which assume running it on hadoop CLI)
kylin.job.remote.cli.password, fill in the user name and password used to login hadoop cluster for hadoop command execution; If you’re using HDP sandbox, the default username is
rootand password is
- In examples/test_case_data/sandbox
- For each configuration xml file, find all occurrences of
sandbox.hortonworks.com, replace with your hadoop hosts; (if you’re using HDP sandbox, this can be skipped)
- For each configuration xml file, find all occurrences of
An alternative to the host replacement is updating your
hosts file to resolve
sandbox.hortonworks.com to the IP of your sandbox machine.
Run unit tests
Run unit tests to validate basic function of each classes.
mvn test -fae -Dhdp.version=<hdp-version> -P sandbox
Run integration tests
Before actually running integration tests, need to run some end-to-end cube building jobs for test data population, in the meantime validating cubing process. Then comes with the integration tests.
It might take a while (maybe one hour), please keep patient.
mvn verify -fae -Dhdp.version=<hdp-version> -P sandbox
To learn more about test, please refer to How to test.
Launch Kylin Web Server locally
Copy server/src/main/webapp/WEB-INF to webapp/app/WEB-INF
cp -r server/src/main/webapp/WEB-INF webapp/app/WEB-INF
Download JS for Kylin web GUI.
npm is part of
Node.js, please search about how to install it on your OS.
cd webapp npm install -g bower bower --allow-root install
In IDE, launch
org.apache.kylin.rest.DebugTomcat with working directory set to the /server folder. (By default Kylin server will listen on 7070 port; If you want to use another port, please specify it as a parameter when run `DebugTomcat)
Check Kylin Web at
http://localhost:7070/kylin (user:ADMIN, password:KYLIN)
For IntelliJ IDEA users, need modify “server/kylin-server.iml” file, replace all “PROVIDED” to “COMPILE”, otherwise an “java.lang.NoClassDefFoundError: org/apache/catalina/LifecycleListener” error may be thrown.
Setup IDE code formatter
In case you’re writting code for Kylin, you should make sure that your code in expected formats.
For Eclipse users, just format the code before committing the code.
For intellij IDEA users, you have to do a few more steps:
Install “Eclipse Code Formatter” and use “org.eclipse.jdt.core.prefs” and “org.eclipse.jdt.ui.prefs” in core-common/.settings to configure “Eclipse Java Formatter config file” and “Import order”
Go to Preference => Code Style => Java, set “Scheme” to Default, and set both “Class count to use import with ‘*’” and “Names count to use static import with ‘*’” to 99.
Disable intellij IDEA’s “Optimize imports on the fly”
Format the code before committing the code.
Setup IDE license header template
Each source file should include the following Apache License header
Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
The checkstyle plugin will check the header rule when packaging also. The license file locates under
dev-support/checkstyle-apache-header.txt. To make it easy for developers, please add the header as
Copyright Profile and set it as default for Kylin project.