Alcohol wipes. The big ones, not the tiny 'prep' pads. Alcohol will take the oil right off. I always have a couple big pads in my pack when I'm out running around.
Here's 1 example. There are many brands. Just get some that are big enough to really wet your skin down.
ALCOHOL WIPESA bottle of rubbing alcohol works fine, too, but you need to use it before the oil has time to start burning.