Xgboost-With-Imbalance-And-Focal-Loss

This small project includes the codes of Weighted Imbalance Loss and Focal Loss [1] for Xgboost [2](<\url> https://github.com/dmlc/xgboost) in 2-class classification problems. The principal reason for us to use Weighted Imbalance Loss and Focal Loss is to address the problem of label-imbalanced data, which could significantly degrade the performance of Xgboost. The original Xgboost program provides a convinient method to customize the loss function, but one will be needing to compute the first+second order derivatives and implement them. The major contribution of the project to the drivation of the gradients and the implementations of them.
For both of the loss functions, since the task is 2-class classification, the activation would be sigmoid:
$y_{i} = \frac{1}{1+\text{exp}(-z_{i})}$
And below the two types of loss will be discussed respectively.

1. Weighted Imbalance (Cross-entropoy) Loss

And combining with $\hat{y}$, which are the true labels, the weighted imbalance loss for 2-class data could be denoted as:
$l_{w} = -\sum_{i=1}^{m}(\alpha\hat{y}_{i}\text{log}(y_{i})+(1-\hat{y}_{i})\text{log}(1-y_{i})))$
Where $\alpha$ is the 'imbalance factor'. And $\alpha$ value greater than 1 means to put extra loss on 'classifying 1 as 0'.
The gradient would be:
$\frac{\partial L_{w}}{\partial z_{i}} = -\alpha^{\hat{y}_{i}}(\hat{y}_{i}-y_{i})$
And the second order gradient would be:
$\frac{\partial L_{w}^{2}}{\partial^{2} z_{i}} = \alpha^{\hat{y}_{i}}(1-y_{i})(y_{i})$

2. Focal Loss

The focal loss is proposed in [1] and the expression of it would be:
$L_{w} = -\sum_{i=1}^{m}\hat{y}_{i}(1-y_{i})^{\gamma}\text{log}(y_{i}) + (1-\hat{y}_{i})y_{i}^{\gamma}\text{log}(1-y_{i})$
The first order gradient would be:
$\frac{\partial L_{w}}{\partial z_{i}} = -(\hat{y}_{i}+(-1)^{\hat{y}_{i}}y_{i})^{\gamma}[(\hat{y}_{i}-y_{i})+\gamma(1-\hat{y}_{i}-y_{i})+log(1-\hat{y}_{i}-(-1)^{\hat{y}_{i}}y_{i})]$
And the second order gradient would be a little bit complex. To simplify the expression, we firstly denotes the terms in the 1-st order gradient as the following notations:
$\begin{cases} g_{1} = (\hat{y}_{i}+(-1)^{\hat{y}_{i}}y_{i})^{\gamma} \\ g_{2} = (\hat{y}_{i}-y_{i}) \\ g_{3} = \gamma(1-\hat{y}_{i}-y_{i}) \\ g_{l} = \text{log}(1-\hat{y}_{i}-(-1)^{\hat{y}_{i}}y_{i}) \end{cases}$
Then the 2-nd order derivative will be:
$\tiny \ \frac{\partial L_{w}^{2}}{\partial^{2} z_{i}} = -[\gamma(\hat{y}_{i}+(-1)^{\hat{y}_{i}}y_{i})^{\gamma-1}y_{i}(1-y_{i})(g_{2}+g_{3}+g_{l}))+ g_{1}(-y_{i}(1-y_{i})-\gamma y_{i}(1-y_{i})g_{l}-\frac{1}{1-\hat{y}_{i}(-1)^{\hat{y}_{i}}y_{i}}y_{i}(1-y_{i}))]$

Scared by the above equations? It's ok

That's the reason I upload this to github: I have done the job and coded them into the script, you can simply use it and ignore the equations! Remember to call Xgboost_classsifier_sklearn class and specify the parameter special_objective when implementing the class to an object. Also, you can change the prarameter $\alpha$ or $\gamma$ inside the script.

Enjoy Using!

@author: Chen Wang, College of Science and Art, Rutgers University (previously affiliated with University College London, Sichuan University and Northwestern Polytechnical University)
@version: 0.1

References

[1] Lin, Tsung-Yi, Priyal Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. "Focal loss for dense object detection." IEEE transactions on pattern analysis and machine intelligence (2018).
[2] Chen, Tianqi, and Carlos Guestrin. "Xgboost: A scalable tree boosting system." In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pp. 785-794. ACM, 2016.

mathlf2015 / Xgboost-With-Imbalance-And-Focal-Loss