The simple question of how fast leads to a less than simple answer of it depends. It depends on your bullet alloy and hardness, bullet size, chamber throat dimensions, bullet lube and the peak pressure your load produces. Make sure you scrub all jacket fouling out of your bore before shooting cast bullets as jacket fouling can really exacerbate leading. Here's how I look at various velocities with cast bullets in rifles and the difficulty you'll have achieving accuracy at those speeds
Up to 1800 fps, childs play, nothing special needed other than a properly sized bullet and a clean bore
Up to 2200 fps, college level, some tinkering may be needed to get good results but nothing exceptional
Up to 2400 fps, graduate level, expect some serious tinkering including casting your own bullets and playing with alloys, hardness and different lubes.
Over 2400 fps, doctoral level, you'll be putting in some time to get success at this level and most will give up before achieving success, not impossible but you have to figure out how much time you have to invest and the benefits.
