While there isn't an official songwriting rulebook, there are quite a few tried-and-true songwriting conventions that over time have essentially make up a few of its basic rules. In almost all instances, these "rules" are there because they tend to make songs more commercial or catchy or both.
That being said, breaking these rules can be a very effective tool, assuming it's done with the full understanding that you are doing so. In other words, breaking the rules by accident doesn't count and can often result in a song that isn't as effective as it could be. This article walks through the when, how, and why of breaking some of the rules of popular songwriting.
Breaking a songwriting rule should be done when the end result actually draws the listener's attention to that moment in the song for the better. I think the best way to illustrate this would be by example. One songwriting rule that's fairly commonly observed is that your lyric should be conversational. Part of what I take conversational to mean is that you put the accent on the proper syllable of each word.
The best time to break this rule would be when you're trying to draw the listener's ear to a particular word. By emphasizing a syllable in that word that usually doesn't get emphasis, your listener will most likely notice what you've done.
Sometimes it's not something they consciously notice, but it can make the song more fun to sing thus, more memorable. However, if your song is full of words where you're accenting an unusual syllable, it will not only lose its effectiveness but also make the song uncomfortable to listen to.
Breaking songwriting rules can be done in a variety of ways. As I mentioned above, you can take a specific rule and just break it once to draw attention to a particular moment in a song. However, you can also take a more global approach.
For example, if the general rule in country music singles is to keep the songs short and the topics relatively light, then writing a song that is over four minutes about a teenage girl losing her virginity would, in most cases, be considered pretty seriously breaking the rules. Yet that's exactly what the songwriters of "Strawberry Wine" did, and it was still a huge hit and career song for the artist (Deana Carter).
My experience is that the best reason to break a songwriting rule is that the song will be less effective by not breaking it. There are simply times when taking the safe route will water down a song that would otherwise have that magic "thing" we all look for when we write. If that's the case, then by all means, break the rule and make the song the best it can possibly be. I'd only suggest that it should be an active decision and that you be aware that you are, in fact, breaking a rule.
Personally, I'm not convinced that breaking any of the many unwritten songwriting rules is something you should make into a regular part of your songwriting practice. As I mentioned earlier, these rules exist because they tend to make your songs more commercial and effective. However, if you find yourself in a position where breaking the rules is your best bet, then I'd recommend embracing it fully.