The measurement of 3.53mm is the size of the screw head. The size of the thread will be smaller - either 3mm or 2.5mm. To drill it out, you need to centre-punch right in the middle before attempting to drill otherwise the drill will walk off centre. Next, you need to start with a smaller drill of about 1.5 to 2mm just to get it started. You have to be really steady because the drill will break easily, and if it breaks off in the hole, you're out of the game. If you can make a hole a couple of mm deep, then you can change to the bigger drill. I'd start with a 2.5mm one. Again, it's really easy to break the drill, especially when it comes out the other side of the screw and jumps a bit. Don't push too hard.
Regarding the allen screw on the other end: It's only a question of finding the right sized key. I suspect that your screwdriver damaged the end a bit so that the right sized key doesn't go in so easily. Use the screw that you already removed to test for a nice snug fit on a key. When you're happy that you got the right sized key, then see if it goes in the other screw. If it doesn't, you can tap it in with a hammer. If it does, but spins, then you've rounded the hexagon inside. You might still be able to get it out with one of those star-shaped bits. You use the one that nearly goes in and give it a tap with the hammer. It should then be nice and tight and should be able to turn the screw out.