Alexander Fyfe erected Hillside in 1862. The land was originally taken up by him in 1855 and was intended to be a two storied home. After the ground floor was completed, Captain Fyfe chartered a ship to bring decorative temple stones from Burma for the second storey. The ship was lost at sea and the captain was unable to complete his home. He died on 8 May 1903 at the age of 76. He arrived in Australia with his wife in 1848 on the ship Stag. All of their ten children survived. He was a very prominent public figure in the early days of Victoria and Queensland, having been elected to the first Legislative Council for Geelong in the middle fifties at the early age of 24 years. He was the first person to open up a foreign trade between Geelong and the East Indies, having as many as five sailing vessels trading between these places. Migrating to Queensland in 1861 he took over a large squatting property. In 1868 he was elected to represent Rockhampton, and held the seat until 1874, when he returned to Victoria.
James Henty then purchased the house, adding the land to his holdings nearby and used it as a country holiday home for a few years.
James Stewart tendered for a four year lease of 682 acres for the years 1875-1879.
The next owner was Andrew White. He was born in 1829 at Stretton-on-Dunsmore in Warwickshire. He married Emily Walker in 1852. Andrew and Emily arrived in Australia in 1853 aboard the James L Bogert. He ran livery stables in Geelong which were well established by 1856. Emily died in 1861. His second wife Jane died in 1871. He had large land holdings in Connewarre. He purchased Hillside in 1878 and moved from Stretton Park with his family soon after. He became a councillor in the Barrabool Shire in 1878 and served until 1886. In July 1885 he was appointed a justice of the peace. He was also a South Barwon Shire councillor and served for ten years between 1878 and 1899 with several terms as president (1878-1879, 1885-1887, 1889, 1894, 1895-1896 and 1898-1899). He died intestate on 28 May 1900 in his 72nd year. His estate was valued at £46,305/17/5 and was divided between his widow, Catherine, and his children at the time of his death:
William Andrew White of Aphrasia Street, Newtown
Charles White late of "Darragh", Myamyn Street, Malvern
Emily Ann McGregor of Morangarell, NSW
Alfred Gilbert White of Hillside Mount Duneed
Henrietta Jane Whitelaw of Fitzroy Street, Geelong
Adeline Charlotte Collins of Prospect Road, Newtown
Laura Amelia Richardson of Pevensey Crescent, Geelong
His third son Alfred Gilbert (1868-19 January 1946) purchased Hillside from his father's estate in 1900 and lived there until his death. He was a former pupil of Mount Duneed School from 1878 until 1880. He was the municipal clerk at South Barwon Shire from 1894-1900, and served on the South Barwon Council for 46 years from 1900 until his death on 19 January 1946 being president for the following terms — 1903-1904, 1906-1907, 1921-1922 and 1931-1933. He married Elizabeth Orchard Matthews. His children were Mildred Bessie (born 1895), Constance (born 1897), Ruth Audrey (born 1899) Douglas Alfred (born 1900), Noel Stretton (born 1902) and John Wilfred (born 1907).
His grandson John Wilfred White (1907—5 August 1978), was elected to the Barrabool Shire Council in 1933, serving until 1946 with two terms as president (1936 and 1945). He moved to Hillside in 1947.
Grapes were grown at Hillside from about 1860 and a century later the old grape vines were still hanging on the rafters of the homestead. On one side of the property was a bluestone quarry and on the other side a scoria quarry. Scoria from this quarry was used for mixing with scoria imported from Italy for the 1956 Olympic Games. The Barrabool Shire and contractors paid royalty on all scoria removed, mainly being used for roads. The pistol club used the quarry after it closed.