The Dolphins Swim Club had its origins in West Pennant Hills when Pat Norton (a former Olympic Swimmer who swam backstroke at the Berlin Games), opened a Swim school at the Sun Valley Pool in 1955. In 1968 as the swim school grew club social races were introduced. In the late 1960s Jan Coulter came to work for Pat Norton and became the club coach.
In 1973 the club was officially registered with NSW Swimming as West Pennant Hills Amateur Swimming Club ASC.
Max Ruddock was the federal member for the Hills at the time and became patron of the club. His son Phillip Ruddock also lived locally and his children all learnt to swim at the Sun Valley Pool. The Max and Emmie Ruddock trophies are still presented each year and usually Phillip Ruddock finds time to personally present them at the annual presentation.
In 1974 the Castle Hill RSL relocated from Old Northern Rd to its present Castle St location. The new club had an indoor 25m pool in the lower floor referred to as the moat by the locals. In 1978 the RSL engaged Jan Coulter to operate the swim school. The West Pennant Hills Swimming Club was no longer able to operate from Sun Valley and relocated to the Blind School pool at North Rocks for a short period.
By 1980 the Coulter Swim School had grown and Jan Coulter introduced Friday Night Developmental Races at the RSL for her competitive swimmers to swim time trials and races. She applied for the RSL to close the pool to public swimmers from 6-8pm each Friday Night. The swimmers still competed in NSW Swimming events as members of the West Pennant Hills Swimming club. The numbers were high and the West Pennant Hills Swim Club won the Castlereagh District Championships for 5 years in a row from 1978 through until 1982.
In 1982 it was decided to change the name of the club from West Pennant Hills Amateur Swimming Club to the Castle Hill RSL Dolphins ASC. The club would swim weekly in the Friday timeslot. Swimmers also participated in NSW State RSL Swimming Championships representing Castle Hill as part of Zone 4, the local district. At the time the club was one of the biggest in the Castlereagh District and hence the Dolphins Swimming Club as we know it today was born.