Why 'five of the clock'? Why not four or six? The 'five o'clock shadow' coinage was based on the 19th century upper-crust English habit of taking tea at five o'clock. Not that the notably upper-crust 7th Earl of Shaftesbury had much time for it. He is reported in Edwin Hodder's biography, 1886, as saying:
Five o'clock tea, that pernicious, unprincipled and stomach-ruining habit.