In lots of ways, the performance of aircraft in the late 1950s wasn't all that inferior to those of today. Most of the advancement has been in avionics and weapons systems as opposed to outright raw performance.
I would imagine that given that in the space of about twenty years, the state of the art for a fighter aircraft went from 350 mph at 25,000 feet with three or four .30 caliber machine guns to Mach 2, 60,000 ft. ceiling, and missiles firing BFR, that designers of that era would be flabbergasted that current fighters are not closer to something from Buck Rogers than the things they designed back then.
It all depends on what you call "performance". If it's fast and squirrly, then yeah, you're right. But advancements have been in speed with low level stability, lower radar cross section, MUCH longer range, all weather, overall lifespan of the airframe, ease in maintenance, and ability to fly all day long, rotating out pilots. Those are some of the "performance" upgrades we have made.
But it's true that a F-15 isn't going to go a whole lot faster than an F-104. Service ceiling, rate of climb, and top speed are probably all pretty darned close. (I'm sure the F-15 is ahead in every category, but I doubt there are any huge differences).
