Governor John Hunter ordered a fort to be built and Fort Phillip was commissioned in 1804 by the third New South Wales Governor (in office 1800–1806), Philip Gidley King, partly as a response to external threats such as a possible French attack in 1803 and partly due to the internal unrest from rebellious convicts reflected in Australia's only major convict rebellion at Castle Hill in March 1804. This was dubbed the Battle of Vinegar Hill as most of the convict rebels were Irish. Windmill Hill (later Observatory Hill) was chosen as a fort location as it was the highest point above the colony, affording commanding views of the Harbour approaches from east and west, the river and road to Parramatta, surrounding country and of the entire town below. The fort was never fully completed and never fired a single shot in anger.
In 1840 a new signal station, designed by the colonial architect Mortimer Lewis, was built to replace the flagstaff. The fort was subsequently demolished leaving the eastern rampart as the only remaining evidence of the fort's existence.
Sydney Observatory was built on the site in 1858 to provide accurate time readings for the colony and to observe stars in the southern sky.
Fort Philip (Sydney) Facts for Kids. Kiddle Encyclopedia.