If you consider the cumulative effect of small talk, you will be surprised as to how many years gone in this not so important talk. Day in and day out we are dragged into some conversation which may not add value. These are rumors are so interesting that we are drawn into them.
If you carefully log your activities for a week and analyze the same, your conscious will put you back in your normal course. You will discover a chunk of time that could have been used for constructive purposes.
I am not suggesting that you should become entirely focused only on your goal-related activities. Life is full of opportunities and challenges. One of the challenges is to refocus your attention on the priorities on hand. The easiest thing is to blame others for taking away your valuable time. The difficult thing is to analyze as to what you could have done to prevent it.