I'm thinking on the next loading I'll size in two steps, first run the expander ball through the neck, then remove it and FL size without it.
If you've got a pair of dial calipers around, you could take the neck expander out of the die, chuck it up in a hand drill and spin it while pinching a piece of fine sandpaper around it until you reduce its diameter .002" or so. To smooth it out afterwards use a piece of Scotchbrite on it while it's spun in the drill.
Wear a glove. It's going to get hot during the sanding process.