Are you sure a #4 Rolling Block has enough moxie to contain a .25-20? They were only made in rimfire chamberings, and not of the highest quality alloy steels at that. I guess if you kept handloads down to rimfire pressure levels you might get away with it, but what if/when it falls into the hands of a yahoo who tries to ratchet things up? Remember, it's only the pins that support the breech block and hammer that absorb the backthrust of the cartridge's discharge, and those pins are mighty small in a #4. If you tried that trick with even a #44 Stevens, which at least has a modicum of receiver support for the breech block (if fitted up properly), every critic with a conscience would advise against it.
Best to barrel it as a .22LR, stick to standard velocity ammo, and call it good.
