You probably know this, but just in case.....
Before you try to drift that sight, lift the leaf and carefully take out the elevator. Then put a piece of thin plastic, or Tyvek, or even oiled paper under the leaf to protect the barrel's finish. Also, use a proper brass drift and hammer to avoid whoopsies. Drift left to right, looking from the rear.
