In 1861 his farmhouse coach house and stables, now known as Neuchatel, were built on the corner of Barrabool and Merrawarp Roads. He married Hannah (née Arnold) in 1860. Hannah was born in Kent England in 1834 and died in October 1863. She was buried in the Church of England section of the Highton Cemetery. They had two children:
Justine Laura — born and died aged 4 months in 1861
Gustave Adolphe — born on 22 October 1862, married Eleanor Anne Florence Charlton in October 1891 in Durham England, died on 24 July 1918 at Glasgow Scotland
In 1864 he married Catherine née Baglin. She was born about 1844. The had the following children:
Frederick William Baglin — born on 23 May 1864 in Geelong, died on 14 April 1884 at Caulfield Grammar School aged 19 years of heart disease, buried at the St Kilda Cemetery
Abram Louis — born on 5 March 1866 at Ceres, died on 21 January 1867 aged 11 months at Ceres
Fanny Elizabeth — born in 1867, married Samuel Whitehead in 1897, died on 17 June 1954 at the Alfred Hospital in Prahran
Justin Auguste — born on 23 March 1869 at Ceres, married Rose Hannah Neilson in 1892, died on 23 April 1951 at Geelong West aged 82 years
Mary Helen — born on 1 October 1870 at Ceres, married William James Haylock on 16 December 1902 in Western Australia, died on 11 April 1930 at Wooroloo in Western Australia
Laura Alice — born on 21 August 1872 at Ceres, married Joseph Mattinson, died on 17 April 1961 at Williamstown
Albert Augustin — born on 23 November 1874 at Ceres, married Marion Saddlier in 1899, died on 10 March 1917 at Ascot Vale
Cecile Lucie — born on 1 August 1875 at Ceres, died on 1 June 1936 at Ballarat
Charles Benjamin — born in 1878, died in 1959 at Werribee aged 82 years
Josephine — born in 1878 at Ceres, died on 28 April 1896 aged 17 years, buried at St Kilda Cemetery
David Ulysses — born in 1881 at Ceres, died in 1962 aged 81 at Geelong
Hannah Jane — born in 1883 at Ceres, died on 10 November 1968 at Hawthorn aged 86 years, buried in the lawn section of the Highton Cemetery
Frederick William — born in 1884 at Ceres, married Ida Phipps in 1929, died on 30 March 1976 aged 91 years at Geelong
Esther — born on 28 August 1885 at Ceres, died in October 1885 at Ceres
Frederick Ulysses' uncle, Frederick, was one of the first Swiss vignerons to settle in the Geelong district. He planted a vineyard in Partnership with David Pettavel at Pollocksford. This was taken over by Frederick in partnership with Louis Aeschlimann and Henry Louis Pernet, a firm which was dissolved on 22 Novmeber 1862, and also ran Suguar Loaf Vineyard. Frederick later moved to the "Suisse" vineyard on the corner of Barrabool and Merrawarp Roads. A house with the name "Newchatel" was built on the property.
Frederick was a Barrabool Shire councillor serving from 1867 until 1873 including a term as president in 1872-73. He died on 23 June 1914 aged 81 years and was buried in the Church of England section of the Highton Cemetery with Catherine who died on 1 August 1904 aged 62 years, Hannah who died on 13 October 1863 aged 29 years at the Swiss Vineyard and their daughter Justine who died at the age of 4 months.