COLUMN clause for XGBoost

[brightcoder01]

The root discussion issue for Data analysis and transform is here.

Transformation Requirements
XGBoost model only accept the numerical values as input.
For numerical input, we don't need transformation such as normalization/standardization, issue.
For categorical input, we need convert it into numerical before feeding it to the model. The typical transformation is convert to one hot vector, convert to integer id by looking up vocabulary and hash. We can use ONEHOT, VOCABULARIZE, HASH transform function in SQLFlow COLUMN clause.

SELECT cont_1, cont_2, cont_3, cat_1, cat_2, cat_3
FROM source_table
TO TRAIN xgboost
COLUMN VOCABULARIZE(cat_1), HASH(cat_2), ONEHOT(cat_3)
INTO my_model

Let's take VOCABULARIZE function for example. The workflow contains two steps:

1. Do data analysis on the column of string type, and get the vocabulary.
2. Do the transformation of mapping string to id at the training stage.

Discussion Points

1. How do we execute the string to id mapping transformation?

• Do this transformation during the process of generating DMatrix from the source table.
  For example, load the source data as numpy array -> do transformation on the numpy array -> build DMatrix from this preprocessed numpy array -> feed DMatrix to XGBoost for training. Please check the official document.
• Do this transformation in each iteration of XGBoost training.
  Does XGBoost have the flexibility? - To be investigated
• Build a sklearn pipeline, the pipeline contains transformation and XGBoost training. sample video.

2. How to save the transform logic together with the model?

• Export both the model and transform logic together to PMML.
  We can try to use the sklearn pipeline to compose the preprocess logic and XGBoost model. And then use sklearn2pmml to export the transformation and model to PMML

3. Distributed data preprocess and XGBoost training?

• [Need Investigation] Can we use sklearn pipeline for XGBoost distributed training?
• × We can investigate Spark Pipeline and XGBoost on Spark. (Heavy dependency)

[wangkuiyi]

I am not familiar with scikit-learn and not sure how sklearn could help build a pipeline that calls XGBoost. To participate in the discussion and contribute some ideas, I need to learn that. Could @brightcoder01 provide more details about the pipeline building?

Is https://machinelearningmastery.com/develop-first-xgboost-model-python-scikit-learn/ a good explanation about calling XGBoost from scikit-learn?

[weiguoz]

sklearn2pmml requires Java >=1.8.
Instead, I found Nyoka, a Python library to export ML/DL models into PMML. Here are examples: [1, 2, 3]

[brightcoder01]

Export sklearn pipeline (preprocess + XGBoost model) to PMML

• Sklearn2PMML
  From the sample, we need build a sklearn2pmml.PMMLPipeline instead of sklearn.pipeline.Pipeline. The latter one is the native class for sklearn.

• nyoka
  From the sample, we can build a sklearn.pipeline.Pipeline and export the sklearn native pipeline directly to a PMML file.

Summary: Use nyoka is more sklearn friendly.

[terrytangyuan]

3. Distributed data preprocess and XGBoost training?

Check out these tutorials:

• Distributed XGBoost on Kubernetes
• XGBoost on Spark
• Distributed XGBoost with Dask

[sneaxiy]

I investigate XGBoost + Dask + sklearn2pmml.

The reason why I choose sklearn2pmml instead of Nyoka is that:

• Sklearn2pmml supports more data preprocessing than Nyoka. Please compare https://github.com/jpmml/jpmml-sklearn and https://github.com/nyoka-pmml/nyoka . For example, OrdinalEncoder is not supported in Nyoka, which is a quite common encoder to convert string to integer.
• Nyoka has no unified APIs for different pipelines. Users have to call skl_to_pmml to save pure sklearn pipeline, and call skl_to_xgboost to save sklearn pipeline with XGBoost. Not quite sure whether skl_to_xgboost works when the pipeline contains both native sklearn transformers and XGBoost classifer/regressor.

I have tried the following cases:

• XGBoost(using sklearn pipeline) + sklearn2pmml. It works well but XGBoost version must be <= 1.0.2. See Sklearn Pipeline with XGBoost model cannot be saved jpmml/sklearn2pmml#221 .
• XGBoost(using sklearn pipeline + Dask transformers) + Dask + sklearn2pmml. Training process works well, but saving the pipeline to pmml would fail. This is because Dask transformers are intrusive wrapper of original sklearn transformers, and they cannot be recognized by sklearn2pmml.

[vruusmann]

Found this issue via a mention in the SkLearn2PMML project. Here's what I've learned while investigating XGBoost models.

XGBoost model only accept the numerical values as input.

XGBoost models can accept both continuous and categorical features. The splits on the former are encoded using comparison operators (eg. age < 33), whereas the latter are encoded using equality check operators (eg. color == green, or fruit != apple).

It is simply that most wrapper ML frameworks don't support non-continuous features. For example, it is Scikit-Learn that requires you to convert string columns to numeric columns (eg. via one-hot-encoding), because Scikit-Learn is otherwise unable to validate and pass this data forward to XGBoost.

XGBoost even has the concept of a "feature map", which is a description of model columns (data type, operational type, plus valid value space for categorical features).

SkLearn2PMML vs Nyoka

I would advise you to export an XGBoost pipeline using both conversion packages, and then compare the resulting PMML XML files. Their results look and perform very differently.

SkLearn2PMML and XGBoost compatibility

XGBoost changed its binary file format between 1.0.0 and 1.1.0 versions (there's a new "binf" header). SkLearn2PMML versions 0.56.2 and newer know about it now.

TLDR: If you're interested in the internals of XGBoost models, then please export them into PMML representation using the SkLearn2PMML package.