Are you taking about the edge where the brake shoe pushes against the drum or where the magnet rubs against the drum? If it's rough where the magnet rides, the magnet is usually shot and the brake assembly needs to be replaced (which should also give you new shoes - it's an easy DYI job). If it is where the brake shoes ride, it may need to be turned to smooth it out. Depending on how bad it is the drum may need to be replaced.
If you end up having to replace the drum, I would also replace the brake assembly and if you don't have self adjusting assemblies, that is also the time for that upgrade. It's a simple bolt on option and the whole thing can be a DIY (just watch some youtube videos if you have never done it before).
