Don't hand punch , it distorts the aluminum into a well area, and bends as you apply pressure to punch the holes, plus the punch can be a bitch to get in close or round angled parts...
Just buy a load of CHEAP HS drills 10 for a $ or something, and as long as you don't apply too much pressure, it won't burr when the drill punches thru, and if it does , take it off with a stiff wire brush or file.
Square off all the holes from a guaranteed straight edge, then 'centre' mark the drill spot for all your holes, to prevent the drill bit from 'wandering' when you start the hole.
(if you want to be super accurate use a .5 or 1mm modelers hand held drill to drill the pilot.), and just zip thru the job in quick succession.
I have built mining rigs from ali.
Notice the 'corner rods', this allows me to adjust the width of the rigs, by dropping in a new doweling, or fold the whole thing down if it needs to be transported or stored.
also see the "double riveted" top part & bottom, this is to guarantee the squareness and to prevent folding
I'm currently working on another 30 board rig.