The manor of Bayfield in the fifteenth, sixteenth and seventeenth centuries belonged to the Clere and Yelverton families. The estate was acquired from the Yelvertons circa 1620 by John Jermy of the Middle Temple and of Gunton. The family was strongly on the side of Parliament in the Civil War. To this John must be attributed the original E-shaped house with mullioned windows and central porch, and wings to which shaped gables and quoins were added in the late 17th century. The last of the Jermys to own the hall was William, High Sheriff in 1748, who dying in 1752 left a widow Frances (nee Preston) who subsequently married John Michell, a wine merchant of Boston, Lincolnshire. He died here and was buried at Letheringsett in 1776, but sixteen years later was removed to Forcett in Yorkshire. It was about this time that Henry Jodrell, barrister and Recorder of Yarmouth purchased the property and built the present hall, with part of E-shaped house embedded in it.
Ref: Cozens-Hardy, B. 1961. Some Norfolk Halls. Norfolk Archaeology. 32. 163-208