It was inevitable that somewhere along the way, the power that had been assumed by the federal government would be challenged. That's where the "Civil War" came from. The slavery issue is just the boogeyman that the federal government used to push the issue.

Good read on the matter.

http://www.sobran.com/articles/tyranny.shtml

excerpt: But reading it all is eye opening.

To sum up this little constitutional history. The history of the Constitution is the story of its inversion. The original understanding of the Constitution has been reversed. The Constitution creates a presumption against any power not plainly delegated to the federal government and a corresponding presumption in favor of the rights and powers of the states and the people. But we now have a sloppy presumption in favor of federal power. Most people assume the federal government can do anything it isn’t plainly forbidden to do.