George and Ann soon began their large family:
Mary Jane — born in 1841 in a small shack on the corner of Elizabeth and Bourke Streets In Melbourne, married John Richard Musgrove in 1848, died in 1920 at Ballarat
George John — born in 1844 at Connewarre, died in 1865 at Connewarre aged 21 years
James Henry — born in 1847 at Geelong, married Sarah Monk in 1867, died in 1931 at Cobden
Samuel Edwin — born in 1848 at Geelong, married Mary Jane Matthews in 1871
Thomas Alfred — born in 1851 at Saltwater Creek, married Alice Louisa Edsall in 1880, died on 27 June 1911 at Colac, buried in the Methodist section of the Colac Cemetery
Esther Ann — born in 1851 at Saltwater Creek, married Alfred Kidwell Shaw in 1884, died in 1926 at Geelong
William Tilly — born in 1855 at Saltwater Creek, married Sarah McKinnon in 1885, died in 1917 at Frankston
Caroline Matilda — born in 1856 at Connewarre, married John George Graham in 1888, died on 28 December 1912 at Connewarre East, buried in the Methodist section of the Mount Duneed Cemetery
Louisa Harding — born on 15 September 1857 at Connewarre, married William Henry Middleton in 1880, died on 25 June 1941 in Geelong West
Richard Dabbige — born on 1 February 1859 at Connewarre, died in 1860 at Connewarre
Charles Priddle — born in 1860 at Connewarre, married Adelaide Mary Edsall in 1886, died in 1900 at Coorgulmerang
Leslie Edward — born in 1862 in Connewarre, married Lily Butterworth, died on 5 February 1952 at Colac, buried in the general section of the Colac Cemetery
Elizabeth Frances Amelia — born in 1864 at Connewarre, married George Angus in 1891, died in 1925 at Colac, buried in the Church of England section of the Colac Cemetery
Victoria Adeline — born in 1865 at Connewarre, died in 1869 at Connewarre aged 3 years, buried in an unknown location of the Mount Duneed Cemetery
In 1842 the Vagg family headed for Geelong on the paddle steamer Aphrasia, landing at Point Henry. George secured a small piece of land at Barrabool Hills. During George's time there his work was mainly cartage, using his bullock team and wagon, to deliver goods for the early settlers. When the gold rush started he was often away for months at a time earning £100 per trip to the goldfields. His prosperity induced his brother Harry and his family to emigrate, settling in the Western District. Later both George and Ann's sisters were to arrive with their families.
George's family settled east of the mouth of Bream Creak (now called Thompson Creek). He became a member of the Connewarre Road Board for six years. By 1864 he was a joint auditor of the board for which he received £1/1/-. By 1895 he was a joint auditor for the Connewarre Farmers' Common.
By the time of his death he was living at Point Henry. He died on 18 August 1901 aged 81 years at Connewarre and was buried in the Methodist section of the Geelong Eastern Cemetery. Ann died in 1902 in Geelong aged 81 years and was buried with him.