I understand that there is a legitimate purpose to restricting the rate at which comments can be added to a post.
Still users should be encouraged to remove comments when they are no longer needed or relevant, and the site should make that as easy and simple as possible.
To that end, I suggest that removing one's own comment not be subject to the timer restriction. It is annoying when trying to be good and clean up comments that are no longer relevant to be told that you have to wait 5 seconds to clean up the second one, and it adds no value to the site.