I think unless the punter is holding the ball while he's hit it's roughing/running into the kicker. period. (or if it's tipped of course)
The real question is, at what point can you "block" the punter? I mean, you're allowed to block the punter if the return guy starts to run a kick back. Do you have to wait until the punt is fielded before you're allowed to block the punter?