We believe we've identified the right Viscount Milton.
As the son of William Fitzwilliam, 4th Earl Fitzwilliam, Charles would have held the title Viscount Milton until the death of his father when he became known as Earl Fitzwilliam (we think this is analogous to the title 'Prince of Wales' being overridden by the title King). His father died in 1833 so in 1830, when the plaque was erected, Charles was the current Viscount Milton.
His father had inherited great wealth from his uncle, Charles The Marquess of Rockingham, and Charles was an only child so we have to assume he also was very rich, and we can guess who he was named for.
