I rarely ever drill #80 holes any more ... just for the reasons you note and a few of my own. I can't remember the last time I put a #80 in a pin vise!
My "minimum drill size" is #78. When drilling any hole that will have something installed in them I go larger by 1 or 2 numbers. It makes the parts -fit- a lot easier. Even if they are 'loose'.
Then I will use either a very small amount of Tenax or Krystal Klear as the adhesive. Which melts and bonds to the shape in the case of Tenax or fills in the gap in the case of KK. When using Tenax I pre-assemble and apply the glue after. When using KK I put the KK in the hole(s) and put the part in after.
The finished model will be
"what you wanted/envisioned it to look like all along" ...
