In the fifties, very few Chinese were interested in the game of golf and only a handful of the Hong Kong Chinese were members of the Royal Hong Kong Golf Club. Shortly after the Civil War in China, quite a number of Chinese from Shanghai migrated to Hong Kong and started to enroll in the Royal Hong Kong Golf Club.

As many of them were beginners and ignorant of the golf rules and etiquette, it had caused some embarrassment in the Club.

Mr. Fisher Yu, a veteran from Shanghai, with a view to avoid the Chinese members from being ridiculed, was determined to set up a golfing fraternity with the sole objective of teaching these new members the basic rules of golf and golfing etiquette. Thus the "Duffers" was founded in 1954.

The events were held monthly with competition-cum-lecture and dinner at Deep Water Bay Golf Club. Mr. K.U. Zyung and Mr. Willie Woo being the most experienced and better golfers at the time were asked to run such events.

All participants were required to pay a small registration fee for each function, which was used to purchase token prizes like putter or golf bag or balls for the various winners.

On 12th September 1972, Duffers (Hong Kong) limited is officially incorporated in Hong Kong as a limited liability company under the Hong Kong Company Ordinance. Due to the popularity in golf, Duffers membership grew rapidly over the years. Currently, there are close to 600 life members in the fraternity of which , over 180 are active members.

The Duffers has built strong relationship throughout the years with various clubs and associations in different parts of Asia. The Duffers also arranged and participated in golf tournaments and competitions for amateurs in the Asian Pacific Region.

The "North & South" Golf Fraternity, the "Tai Pan" Group and the "Senior & Junior" Golfing Societies all were the "Off Shoots" of the "Duffers".