The Ledyard, New London rivalry