Prediction is up-or-down but the score is not binary. Actually the score is "the return you would have made if you take that position (assume up is long, down is short)". If you predict a asset "up" and the asset actually rise by 2% then you get +log(1.02) while you loose if the asset fell by 2%. Additionally, the return is divided by the period (timeframe you choose) to annualize it. Also, a constant is added to the denominator to adjust for luck with small sample.