It was my understanding that the idle can become a bit rough because you don't have quite so much rotational mass to smooth it out. The idle, after all, is just a bunch of small controlled explosions, so it will have a natural lumpiness to it.
Picture trying to keep a merry-go-round spinning when empty. It might visibly slow between each shove you gave it, but it would be easier to move.
Now picture it loaded up with a bunch of fat harley dudes. It won't slow down much between pushes, but it would take a lot more to get it going.