Here's a sample of some of the tools I like to use.
View attachment 206233
The left is a set of mini files from big box store. The benefit of these is the low form factor. You can make a hole to barely the size you need without extra. I like this husky brand one because it comes with a handle that's wider than the cheap Amazon ones. However these are very slow to remove material.
The one on the right you might recognize as from the set of Amazon mini rasps that are sold under various China brands. They remove material fast, but if you want a narrow tang hole, like 3mm, you probably won't be able to get it due to the thickness of the tool.
The final one in the middle was my splurge. This is a handle broach from John perry. They remove material fast, can be made narrow, and are deep enough for longer tangs. Length from front of handle to tip of broach is just over 140mm. Compared to the one linked above which is only about 90mm long. (for reference, I've had knives I rehandled that had tangs up to about 115mm). John makes these in 3/32, 1/8, 3/16, and 1/4. Very high quality steel and a lovely walnut handle to go with it. Downside is that this will set you back about $100 before shipping.