Model Management
Kylin provides REST APIs on checking model information, index build and model management.
- On how authentication works, see Access and Authentication REST API.
- On Curl command line, don't forget to quote the URL if it contains the special char
&.- On Curl command line, escape
%25if it contains the special char%.
Create Model
-
POST http://host:port/kylin/api/models -
Request Permission: MANAGEMENT permission and above
-
HTTP Header
Accept: application/vnd.apache.kylin-v4-public+jsonAccept-Language: cnContent-Type: application/json;charset=utf-8
-
HTTP Body: JSON Object
project-requiredstring, project namefact_table-requiredstring, fact table nameuuid-optionalstring, model ID, default auto generatedalias-requiredstring, model alias namemanagement_type-requiredstring, for model creation please use: MODEL_BASEDsimplified_measures-requiredJSON Object[], measures-
name-requiredstring, measure name -
expression-requiredstring, function, including: SUM,MIN,MAX,TOP_N,COUNT,COUNT_DISTINCT,CORR,PERCENTILE_APPROX,COLLECT_SET -
parameter_value-requiredJSON Object[], measure parameterstype-requiredstring, parameter type: constant,columnvalue-requiredint|string, parameter value (Set value 1 when 'expression' is 'COUNT' and 'type' is 'constant', while others should be: TABLE.COLUMN)
-
return_type-requiredstring, function return type with arguments including: topn(10)、topn(100)、percentile(100) etc., using empty string("") when no return type neededSupported return_type values:
TOP_N:
* Top 10: topn(10)
* Top 100: topn(100)
* Top 1000: topn(1000)
COUNT_DISTINCT:
* Error Rate < 9.75%: hllc(10)
* Error Rate < 4.88%: hllc(12)
* Error Rate < 2.44%: hllc(14)
* Error Rate < 1.72%: hllc(15)
* Error Rate < 1.22%: hllc(16)
* Precisely: bitmap
PERCENTILE_APPROX:
* percentile(100)
* percentile(1000)
* percentile(10000) -
comment-optionalstring, comments
-
simplified_dimensions-requiredJSON Object[], dimensions infocolumn-requiredstring, format: TABLE.COLUMNname-requiredstring, column alias, may set the same as column namestatus-requiredstring, fixed value: DIMENSION
computed_columns-optionalJSON Object[], computed columns infocolumnName-requiredstring, the new column nameexpression-requiredstring, expressiondatatype-requiredstring, data type: VARCHAR,INT,BIGINT ...standard sql typestableIdentity-requiredstring, format: SCHEMA.TABLEtableAlias-requiredstring, table alias
join_tables-requiredJSON Object[], join infotable-requiredstring, table namejoin-requiredJSON Object, join info-
type-requiredstring, join type: INNER,LEFT -
foreign_key-requiredstring[], foreign key -
primary_key-requiredstring[], primary key -
simplified_non_equi_join_conditions-optionalJSON Object, non-equivalent join conditions(note1: The support of this settings should have 'Support History Table' enabled in advance. Seeing Slowly Changing Dimension
(note2: Join relationship >= and < must be used in pairs, and same column must be joint in both conditions)
foreign_key-string, foreign keyprimary_key-string, primary keyop-string, join relationship: LESS_THAN,GREATER_THAN_OR_EQUAL
-
kind-optionalstring, table kind: FACT,LOOKUP, default: LOOKUPalias-optionalstring, aliasflattenable-requiredstring, precomputing associations(precomputing: flatten, no-precomputing: normalized), flatten is suggestedjoin_relation_type-optionalstring, join type: MANY_TO_ONE,MANY_TO_MANY, default: MANY_TO_ONE
partition_desc-optionalJSON Object, partition columns infopartition_date_column-requiredstring, partition date column, format: TABLE.COLUMNpartition_date_format-requiredstring, partition date column format, including: yyyy-MM-dd, yyyyMMdd... Supported date format please check "Model Design Basics"partition_type-optionalstring, partition type, including: APPEND, default: APPEND
owner-optionalstring, the owner of model, default current userdescription-optionalstring, model descriptioncapacity-optionalstring, model capacity, including: SMALL,MEDIUM,LARGE, default: MEDIUMfilter_condition-optionalstring, data filter condition. Data filter condition is an additional data filter during data loading.with_base_index-optionalboolean, adding base indexes or not, including base aggregate index and base table index. Base indexes include all dimensions and measures of the model and automatically update as the model changes by default. default: false; It is recommended to usebase_index_typeafter version 4.6.6,with_base_indexdoes not take effect after configuringbase_index_typebase_index_type-optionalArray[String], select the base index type to be added, optional valuesBASE_AGG_INDEX,BASE_TABLE_INDEX, effective version: 4.6.6[BASE_AGG_INDEX, BASE_TABLE_INDEX]Both the base aggregate index and the base table index are added[BASE_AGG_INDEX]only add the base aggregate index[BASE_TABLE_INDEX]only add the base table index[]does not add base indexes
-
Curl Request Example
curl -X POST \
'http://host:port/kylin/api/models' \
-H 'Accept: application/vnd.apache.kylin-v4-public+json' \
-H 'Accept-Language: cn' \
-H 'Authorization: Basic QURNSU46S1lMSU4=' \
-H 'Content-Type: application/json;charset=utf-8' \
### Stringify the following JSON Object when use
-d '{
"project": "pj01",
"fact_table": "SSB.P_LINEORDER",
"alias": "test_model_01",
"management_type": "MODEL_BASED",
"simplified_measures": [
{
"name": "COUNT_ALL",
"expression": "COUNT",
"parameter_value": [
{
"type": "constant",
"value": 1
}
],
"return_type": ""
}
],
"simplified_dimensions": [
{
"column": "P_LINEORDER.LO_ORDERKEY",
"name": "LO_ORDERKEY",
"status": "DIMENSION"
},
{
"column": "P_LINEORDER.LO_CUSTKEY",
"name": "LO_CUSTKEY",
"status": "DIMENSION"
},
{
"column": "P_LINEORDER.LO_ORDERDATE",
"name": "LO_ORDERDATE",
"status": "DIMENSION"
},
{
"column": "P_LINEORDER.LO_ORDERPRIOTITY",
"name": "LO_ORDERPRIOTITY",
"status": "DIMENSION"
},
{
"column": "P_LINEORDER.V_REVENUE",
"name": "V_REVENUE",
"status": "DIMENSION"
},
{
"column": "DATES.D_YEAR",
"name": "D_YEAR",
"status": "DIMENSION"
},
{
"column": "CUSTOMER.C_NAME",
"name": "C_NAME",
"status": "DIMENSION"
},
{
"column": "CUSTOMER.C_PHONE",
"name": "C_PHONE",
"status": "DIMENSION"
}
],
"join_tables": [
{
"table": "SSB.DATES",
"join": {
"type": "LEFT",
"foreign_key": [
"P_LINEORDER.LO_ORDERDATE"
],
"primary_key": [
"DATES.D_DATE"
]
},
"alias": "DATES",
"flattenable": "flatten",
"join_relation_type": "MANY_TO_ONE"
},
{
"table": "SSB.CUSTOMER",
"join": {
"type": "LEFT",
"foreign_key": [
"P_LINEORDER.LO_CUSTKEY"
],
"primary_key": [
"CUSTOMER.C_CUSTKEY"
]
},
"alias": "CUSTOMER",
"flattenable": "flatten",
"join_relation_type": "MANY_TO_ONE"
}
],
"with_base_index": true,
"base_index_type": ["BASE_AGG_INDEX", "BASE_TABLE_INDEX"]
}' -
Response Details
code-string, response code, succeed:000, failed:999msg-string, response messagedata-JSON Object, response database_table_index-JSON Object, base index infodimension_count-int, dimension countmeasure_count-int, measure countlayout_id-long, layout idoperate_type-string, operation type, including: UPDATE,CREATE
base_agg_index-JSON Object, base aggregation index info, same structure asbase_table_indexwarn_code-string, warning code message
-
Response Example
{
"code": "000",
"msg": "",
"data": {
"base_table_index": {
"dimension_count": 8,
"measure_count": 0,
"layout_id": 20000000001,
"operate_type": "CREATE"
},
"base_agg_index": {
"dimension_count": 8,
"measure_count": 1,
"layout_id": 1,
"operate_type": "CREATE"
},
"warn_code": null
}
} -
Failed Response Example
{
"code": "999",
"data": null,
"msg": "KE-010001002(Empty Project Name):Can’t find project information. Please select a project.",
"stacktrace": "KE-010001002(Empty Project Name) \norg.apache.kylin.common.exception.KylinException: KE-010001002(Empty Project Name):Can’t find project information. ...",
"exception": "KE-010001002(Empty Project Name):Can’t find project information. Please select a project.",
"url": "http://host:port/kylin/api/models"
} -
Error Code (Specific error message please check real api return.)
KE-010001001: Project Not ExistKE-010001002: Empty Project NameKE-010006002: Invalid Partition ColumnKE-010000003: Invalid ParameterKE-010000002: Invalid RangeKE-010000004: Invalid NameKE-010006005: Timestamp Column Not ExistKE-010002010: Failed to Add ModelsKE-010011001: Duplicate Computed Column NameKE-010007001: Table Not Exist
Get Model List
Call this API to get a list of models under specified project that satisfies certain conditions.
-
GET http://host:port/kylin/api/models -
URL Parameters
project-required,string, project name.page_offset-optional,int, offset of returned result,0by default.page_size-optional,int, quantity of returned result per page,10by default.status-optional,string, model status.model_name-optional,string, model name.exact-optional,boolean, whether exactly match the model name,trueby default.
-
HTTP Header
Accept: application/vnd.apache.kylin-v4-public+jsonAccept-Language: cnContent-Type: application/json;charset=utf-8
-
Curl Request Example
curl -X GET \
'http://host:port/kylin/api/models?project=doc_expert' \
-H 'Accept: application/vnd.apache.kylin-v4-public+json' \
-H 'Accept-Language: cn' \
-H 'Authorization: Basic QURNSU46S1lMSU4=' \
-H 'Content-Type: application/json;charset=utf-8' -
Response Details
Code:String, response code. Returned value:000(request processing success),999(request processing failed)data:JSON, returned datavalue: details of the returned data, which consists of:name:String, model namelookups:JSON, detailed list of all dimension tablessize_kb:Long, total size of all segments of the modelsinput_records_count:Long, number of flattened tables within all segmentsinput_records_size:Long, size of flattened tables within all segmentsproject:String, project nameuuid:String, model IDlast_modified:Long, last modified time of the modelcreate_time:Long, model creation timemvcc:Long, version number with metadata modifiedalias:String, model aliasowner:String, model ownerconfig_last_modifier:String, last user who modified the configurationfact_table:String, fact table (one model contains only one fact table)fact_table_alias:String, fact table aliasjoin_tables:JSON, joined tablespartition_desc:JSON, partition columnall_measures:JSON, all measures within the modelmulti_partition_desc:JSON, multiple partitionscomputed_columns:JSON, computed columnscanvas:JSON, position of model canvasstatus:String, model status; Returned value:online,offline,broken,warninglast_build_end:String, building time of the last segmentstorage:Long, total storage size of the model; Unit: bytesource:Long, sourceByte sum of all segments in the modelexpansion_rate:String, model expansion rate; Unit: %usage:Long, queried times of the modelmodel_broken:Boolean, if the model is broken or notroot_fact_table_deleted:Boolean, if the fact table is deletedsegments:JSON, segment listsrecommendations_count:Integer, number of recommendationssimplified_measures:JSON, measure list of the modelssimplified_dimensions:JSON, dimension list of the modelssimplified_tables:JSON, all tables in the model
offset: page numberlimit: models listed in each pagetotal_size: total number of models
-
Response Example
{
"code":"000",
"data":{
"value":[
{
"name":"Model_03",
"lookups":Array[1],
"is_streaming":false,
"size_kb":0,
"input_records_count":0,
"input_records_size":0,
"project":null,
"uuid":"0464241b-fd7d-49a9-a3c9-b4f7e32cf489",
"last_modified":1574750372949,
"create_time":1574761225505,
"version":"4.0.0.0",
"mvcc":4,
"alias":"Model_03",
"owner":"ADMIN",
"config_last_modifier":null,
"config_last_modified":0,
"description":"",
"fact_table":"SSB.LINEORDER",
"fact_table_alias":null,
"management_type":"MODEL_BASED",
"join_tables":Array[1],
"filter_condition":"",
"partition_desc":Object{...},
"capacity":"MEDIUM",
"segment_config":Object{...},
"data_check_desc":null,
"semantic_version":0,
"all_named_columns":Array[26],
"all_measures":Array[2],
"column_correlations":Array[0],
"multilevel_partition_cols":Array[0],
"computed_columns":Array[0],
"canvas":Object{...},
"status":"ONLINE",
"last_build_end":"902016000000",
"storage":24694,
"source":5585164,
"expansion_rate":"0.44",
"usage":0,
"model_broken":false,
"root_fact_table_deleted":false,
"segments":null,
"recommendations_count":0,
"simplified_measures":Array[2],
"simplified_dimensions":Array[9],
"simplified_tables":Array[2],
"multi_partition_desc": {
"columns": ["KYLIN_SALES.LSTG_SITE_ID"],
"partitions": [
{
"id": 0,
"values": [
"1"
]
},
{
"id": 1,
"values": [
"2"
]
}
],
"max_partition_id": 1
}
},
Object{...},
Object{...}
],
"offset":0,
"limit":10,
"total_size":3
},
"msg":""
}