Multi-level Partitioning

Reminders:

1. Please read Access and Authentication REST API and understand how authentication works.
2. On Curl command line, don't forget to quote the URL if it contains the special char &.

Set partition column

For multi-level partition model define partition column, please refer to Define Partition Column directly.

Add Sub-Partition value

You can add it by specifying sub_partition_values when Load Segment directly or execute following API.

• POST http://host:port/kylin/api/models/{model_name}/segments/multi_partition/sub_partition_values

• URL Parameters

  • model_name - required string,model name.

• HTTP Body: JSON Object

  • project - required string,project name.

  • sub_partition_values - required array,sub-partition values.

• HTTP Header

  • Accept: application/vnd.apache.kylin-v4-public+json
  • Accept-Language: en
  • Content-Type: application/json;charset=utf-8

• Curl Request Example

  curl -X POST \
      http://host:port/kylin/api/models/multi_level_partition/segments/multi_partition/sub_partition_values \
      -H 'Accept: application/vnd.apache.kylin-v4-public+json' \
      -H 'Authorization: Basic QURNSU46S1lMSU4=' \
      -H 'Content-Type: application/json;charset=utf-8' \
      -d '{
        "project":"multi_level_partition",
        "sub_partition_values":[["5"]]
    }'

• Response Example

  {
       "code": "000",
       "data": "",
       "msg": ""
  }

Load Segment

For building multi-level partition model, please refer directly to Load Segment API.

Build Sub-Partition

• POST http://host:port/kylin/api/models/{model_name}/segments/multi_partition

• URL Parameters

  • model_name - required string,model name.

• HTTP Body: JSON Object

  • project - required string,project name.

  • segment_id - required string,Segment id.

  • sub_partition_values - required array,sub-partition values.

  • parallel_build_by_segment - optional boolean, whether to build concurrently,the default value is false.

  • build_all_sub_partitions - optional boolean whether to build all sub-partition values,the default value is false.

  • yarn_queue - optional string, specify the YARN queue used by the job, it can be set after these two parameters were set: kylin.engine-yarn.queue.in.task.enabled (whether to allow set specified YARN queue for build task, default value is false), kylin.engine-yarn.queue.in.task.available (available YARN queues, separate them with English commas)

  • tag - optional object, job tag, if the field is set, when calling the Get Job List API, the field will be the same back when returning the job. It can be used for system integration, mark the job and deal with it accordingly. By default, the maximum size of value is 1024 KB , which can be set by the configure kylin.job.tag-max-size=1024.

• HTTP Header

  • Accept: application/vnd.apache.kylin-v4-public+json
  • Accept-Language: en
  • Content-Type: application/json;charset=utf-8

• Curl Request Example

  curl -X POST \
    http://host:port/kylin/api/models/multi_partition_model/segments/multi_partition \
    -H 'Accept: application/vnd.apache.kylin-v4-public+json' \
    -H 'Authorization: Basic QURNSU46S1lMSU4=' \
    -H 'Content-Type: application/json;charset=utf-8' \
    -d '{
      "project": "multi_partition_project",
      "segment_id": "983904fa-d573-4944-acb8-558c59598a48",
      "sub_partition_values": [
          [
              "5"
          ]
      ],
      "parallel_build_by_segment": false,
      "build_all_sub_partitions": false
  }'

• Response Example

  {
      "code": "000",
      "data": {
          "jobs": [
              {
                  "job_name": "SUB_PARTITION_BUILD",
                  "job_id": "0282e2c9-63e3-46b0-85f6-6e74ca5bf984"
              }
          ]
      },
      "msg": ""
  }

Delete Sub-Partition value corresponding index data

• DELETE http://host:port/kylin/api/models/segments/multi_partition

• URL Parameters

  • project - required string,project name.

  • model - required string,model name.

  • segment_id - required string, Segment Id.

  • sub_partition_values - required string, sub-partition values.

• HTTP Header

  • Accept: application/vnd.apache.kylin-v4-public+json
  • Accept-Language: en
  • Content-Type: application/json;charset=utf-8

• Curl Request Example

  curl -X DELETE \
    'http://host:port/kylin/api/models/segments/multi_partition?project=multi_partition_project&model=multi_partition_model&segment_id=ff839b0b-2c23-4420-b332-0df70e36c343&sub_partition_values=0,1' \
    -H 'accept: application/vnd.apache.kylin-v4-public+json' \
    -H 'authorization: Basic QURNSU46S1lMSU4=' \
    -H 'Content-Type: application/json;charset=utf-8' \

• Response Example

  {
      "code": "000",
      "data": "",
      "msg": ""
  }

Refresh the index data corresponding to the Sub-Partition value

• PUT http://host:port/kylin/api/models/{model_name}/segments/multi_partition

• URL Parameters

  • model_name - required string,model name.

• HTTP Body: JSON Object

  • project - required string,project name.

  • segment_id - required string,Segment id.

  • sub_partition_values - required array,sub-partition values.

  • yarn_queue - optional string, specify the YARN queue used by the job, it can be set after these two parameters were set: kylin.engine-yarn.queue.in.task.enabled (whether to allow set specified YARN queue for build task, default value is false), kylin.engine-yarn.queue.in.task.available (available YARN queues, separate them with English commas)

  • tag - optional object, job tag, if the field is set, when calling the Get Job List API, the field will be the same back when returning the job. It can be used for system integration, mark the job and deal with it accordingly. By default, the maximum size of value is 1024 KB , which can be set by the configure kylin.job.tag-max-size=1024.

• HTTP Header

  • Accept: application/vnd.apache.kylin-v4-public+json
  • Accept-Language: en
  • Content-Type: application/json;charset=utf-8

• Curl Request Example

  curl -X PUT \
    http://host:port/kylin/api/models/SSB_LINEORDER/segments/multi_partition \
    -H 'accept: application/vnd.apache.kylin-v4-public+json' \
    -H 'authorization: Basic QURNSU46S1lMSU4=' \
    -H 'Content-Type: application/json;charset=utf-8' \
    -d '{"project":"default","segment_id":"60615c5e-2ae1-4ee9-b88d-f00f1f101a8a","sub_partition_values":[["1"], ["2"]]}'

• Response Example

  {
      "code": "000",
      "data": {
          "jobs": [
              {
                  "job_name": "SUB_PARTITION_REFRESH",
                  "job_id": "4fa1a0a1-6a97-4be9-bdf6-8957c5f80114"
              }
          ]
      },
      "msg": ""
  }

Get Segment List

Please refer to Get Segment List for multi-level partition model to get Segment list.

Get Segment details

• GET http://host:port/kylin/api/models/{model_name}/segments/multi_partition

• URL Parameters

  • project - required string, project name.

  • segment_id - required string, Segment Id.

  • page_offset - optional int, pagination page, the default value is 0.

  • page_size - optional int, page size, the default value is 10.

  • model_name - required string, model name.

• HTTP Header

  • Accept: application/vnd.apache.kylin-v4-public+json
  • Accept-Language: en
  • Content-Type: application/json;charset=utf-8

• Curl Request Example

  curl -X GET \
   'http://host:port/kylin/api/models/multi_partition_model/segments/multi_partition?project=multi_partition_project&segment_id=ffdfc037-7c63-4499-986d-9b680276161e' \
  -H 'Accept: application/vnd.apache.kylin-v4-public+json' \
  -H 'Authorization: Basic QURNSU46a3lsaW5AMjAxOQ==' \
  -H 'Content-Type: application/json;charset=utf-8' \

• Response Example

  {
      "code": "000",
      "data": {
          "value": [
              {
                  "id": 2,
                  "values": [
                      "3"
                  ],
                  "status": "ONLINE",
                  "last_modified_time": 1609213843048,
                  "source_count": 0,
                  "bytes_size": 3747
              },
              {
                  "id": 3,
                  "values": [
                      "4"
                  ],
                  "status": "ONLINE",
                  "last_modified_time": 1609213843048,
                  "source_count": 0,
                  "bytes_size": 3747
              }
          ],
          "offset": 0,
          "limit": 10,
          "total_size": 2
      },
      "msg": ""
  }

Query mapping settings

• PUT http://host:port/kylin/api/models/{model_name}/multi_partition/mapping

• URL Parameters

  • model_name - required string,model name.

• HTTP Body: JSON Object

  • project - required string,project name.

  • alias_columns - required,array<string>, The corresponding column name that the sub-partition column needs to be mapped to

  • multi_partition_columns - required,array<string>, sub-partition column names

  • value_mapping - required,array<object>, partition value mapping

    • origin - required,array<string>, sub-partition column value

    • target - required, array<string>, sub-partition column value mapping value

• HTTP Header

  • Accept: application/vnd.apache.kylin-v4-public+json
  • Accept-Language: en
  • Content-Type: application/json;charset=utf-8

• Curl Request Example

  curl -X PUT \
    http://host:port/kylin/api/models/test_multi/multi_partition/mapping \
    -H 'Accept: application/vnd.apache.kylin-v4-public+json' \
    -H 'Authorization: Basic QURNSU46a3lsaW5AMjAxOQ==' \
    -H 'Content-Type: application/json;charset=utf-8' \
    -d '{
  	"project": "multi_partition_project",
      "alias_columns": ["KYLIN_SALES.LEAF_CATEG_ID"],
      "multi_partition_columns": ["KYLIN_SALES.LSTG_SITE_ID"],
      "value_mapping": [
        {
          "origin": ["Beijing"], 
          "target":["North"]
        },
        {
          "origin":["Shanghai"],
          "target":["South"]
        }
      ]
  }'

• Response Example

  {
      "code": "000",
      "data": "",
      "msg": ""
  }