Also note early on that in America saying something is bugging you or someone is a bugger refers to someone being annoying. Like a bug or stinging insect. Not something anatomically disturbing. This will prevent more awkward moments.
Honestly, that was my first experience of this word, as heard used by family (who clearly didn't mean it to mean anything bad enough that they didn't feel it was fine to use in front of a small child, and had I used it myself at an old enough age I don't know that I would've been reprimanded). As in, "that bugger cut me off when the light turned green" or "it was a bugger mowing the whole lawn today when it was 90 degrees out." (In fact, I rather get the impression it may be a substitution for a worse word in both cases.)
I've never in my life heard anyone refer to something/someone "bugging" them in any way except as a synonym for "bothering." As in, "my car is still making that noise and it's bugging me" or "today my boss was bugging me to get that report done." (However, "bugger
ing" is a different story, but in my area I'm not sure how many people would even know what it meant if they heard it.)
Perhaps sometimes maybe a case of something that is used so often that many people have lost the sense of what it means and now consider it just a word rather than a word with a definition, and hence it is not as "bad" as it may once have been considered?