Cross Entropy Loss

The cross entropy between two probability distributions over the same underlying set of events measures the average number of bits needed to identify an event drawn from the set.

Cross entropy can be used to calculate loss.

Regularization

Regularization is the process of introducing additional information to prevent overfitting and reduce loss, including:

L1 - Lasso Regression; variable selection and regularization
L2 - Ridge Regression; useful to mitigate multicollinearity

An Example

An example is:

$\\ p=\left ( 1.0,0.0,0.0 \right ) \\ \\ q=\left ( 0.7,0.2,0.1 \right ) \\ \\ p_{2} = p_{3}=0\\ \\ H\left ( p,q \right ) = -\left ( 1.0\cdot ln\left ( 0.7 \right ) \right )=-\left ( 1.0\cdot -.357 \right )=.357$

References

Google Scholar Search
Web Search
Wikipedia Cross Entropy Article
YouTube Search