Skip to main content
Version: 5.0.0

How to debug Kylin in IDEA without Hadoop

Background

Why debug Kylin in IDEA without Hadoop

This article aims to introduce a simple and useful way to develop and debug Kylin for developers.

Deployment architecture

Following is the architecture of the current deployment.

debug_in_laptop

This guide assumes you have prepared the following things:

  • A laptop with macOS installed to do development work (Windows is not verified at the moment)
  • Kylin's source code is cloned into some directory on your laptop
For Windows Dev Machine

For Windows dev machine, setup the Kylin dev env in Windows Subsystem for Linux is the best option. Follow this guide on how to install WSL with GUI, and install both the Kylin code and your favorite IDE (but not the docker) in WSL for best performance.

Prepare IDEA and build source code

Step 1: Check Software Requirement

Please visit Software Requirement, and make sure your laptop has met the requirement.

Step 2: Build source code

  • Build backend source code before your start debugging.

    cd <path-to-kylin-source>
    mvn clean install -DskipTests
  • Build front-end source code.

    (Please use node.js v18.20.4, for how to use a specific version of node.js, please check how to switch to a specific node js )

    cd kystudio
    npm install -g yarn
    yarn install

Step 3: Install IntelliJ IDEA and build the source

  1. Install the IDEA Community edition (the Ultimate edition is ok too).

  2. Import the source code into IDEA. Click the Open, and choose the directory of Kylin source code.

  3. Install the scala plugin and restart

  4. Configure SDK(JDK and Scala), make sure you use JDK 1.8.X and Scala 2.12.X.

  5. Reload maven projects, and the scala directory will be marked as source root(in blue color).

  6. Build the projects.(make sure you have executed mvn clean package -DskipTests, otherwise some source code is not generated by maven javacc plugin)

Step 4: Prepare IDEA configuration

  1. Download spark and create running IDEA configuration for debug purpose and initialize the frontend env.
./dev-support/contributor/local/local.sh init

Following is the shell output.

(base) ➜  kylin git:(kylin5) ✗ ./dev-support/contributor/local/local.sh init
# KYLIN_HOME=/path/to/kylin
# SPARK_HOME=/path/to/kylin/build/spark
* Spark already exists, will be deleted and re-downloaded
* Downloading spark...
spark_pkg_file_name : spark-newten-3.3.0-4.6.26.0.tgz
* Setting IDEA run configurations...
# IDEA run configuration version is community...
* Build Frontend...
* Init Done!

Prepare the Requirements

Deploy Zookeeper(Current only need Zookeeper)

  1. Install latest docker desktop in your laptop

  2. Check available resource of your docker desktop in laptop, make sure you leave 1 CPUs and 1 GB memory at least .

Following is the setting page of Docker Desktop of MacBook.

resources

  1. Start Zookeeper service via docker compose on laptop.
./dev-support/contributor/local/local.sh up

Following is the shell output.

(base) ➜  kylin git:(kylin5) ✗ ./dev-support/contributor/local/local.sh up
# KYLIN_HOME=/path/to/kylin
# SPARK_HOME=/path/to/kylin/build/spark
* Starting container...
[+] Running 9/9
⠿ zookeeper Pulled 469.1s
⠿ b2ddfd337773 Pull complete 403.5s
⠿ 35794a35c4aa Pull complete 403.8s
⠿ 261269474501 Pull complete 466.3s
⠿ 48a873c16c38 Pull complete 466.4s
⠿ 91bcf425c56c Pull complete 466.4s
⠿ 71c2395ae368 Pull complete 466.5s
⠿ 5515e61ba09a Pull complete 466.7s
⠿ 8ce907db3e1a Pull complete 466.8s
[+] Running 2/2
⠿ Network local_local Created 0.0s
⠿ Container zookeeper Started 0.5s
* Up Done!

Check status of Require services

  • Check health of Zookeeper, you can use following command to check status
./dev-support/contributor/local/local.sh ps

Following output content shows all hadoop component are in health state.

# KYLIN_HOME=/path/to/kylin
# SPARK_HOME=/path/to/kylin/build/spark
NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS
zookeeper zookeeper:3.6 "/docker-entrypoint.…" zookeeper 2 minutes ago Up 2 minutes 2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp, 8080/tcp

Debug Kylin in IDEA

Start backend in IDEA

  • Select "BootstrapServer[Local]" on top of IDEA and click Run .

click BootstrapServer[local]

  • Check if SparkUI of Sparder is started.

Spark UI

Start frontend in IDEA

  • Set up dev proxy
cd kystudio
npm run devproxy

setup front end

Validate Kylin's core functions

  • Visit Kylin WEB UI in your laptop

setup front end

  • Create a new project, load table and create model

  • Validate Cube Build and Query function

build job

query a agg index

Command manual

  1. Use ./dev-support/local/local.sh stop to stop all containers
  2. Use ./dev-support/local/local.sh start to start all containers
  3. Use ./dev-support/local/local.sh ps to check status of all containers
  4. Use ./dev-support/local/local.sh down to stop all containers and delete them