.264 is measured in the depth of the groove. .256 is measured from land to land. In the original Newton rifles, they used a Metford form of rifling where the 'grooves' were rounded somewhat. Here's an article by Charles Newton from 1917 and published in the American Rifleman describing the rifling they used.
http://ps-2.kev009.com/ohlandl/Cast_Bullet/Vault/Newton-Pope_Rifling_Oval_Bore.html
