Memorial Day weekend will be jammed.
You might do well to camp somewhere near Pond's Lodge or Mack's Inn, and commute. Or maybe even St. Anthony.
A real out of the way part of the park is Cave Falls. Check the map, and you'll see why it doesn't get much traffic. There is camping available there.