I have owned and enjoyed a belt drive bike for more than four years. It's great. As for downsides I've experienced none but am aware of the extra work when it comes to, as you note, removing the wheel for puncture repair. You remind me to purchase a special sort of inner tube that has a break...