Interesting idea, but I don't think most trolls care enough whether or not they are financially harming or helping the websites they are trolling on - it's more about the reaction they get from other posters. If anything, it might encourage them as they would enjoy seeing other people spend their own money because of the trolling.
I believe the adage holds true: DON'T FEED THE TROLLS. Internet trolls are unpredictable and any sort of reaction, especially one involving money, is likely to encourage them.
I know what you mean, but that is why they will make money to the website they are trolling and not to themselves. You are not in this case feeding the troll as the troll becomes a mule for the website. I am sure it will be fun for the troll to be trolling MORE when learning about the bitTroll deal at the very start. By definition the troll is an annoyance and won't go away. You don't feed the troll. You don't have to answer back to him/her. When you reply to the troll, you enter the keyword that would trigger a troll donation directly to the webmaster.
By the way this is not a concept but a proven technology. On reddit when someone makes a comment you like you can reply with keywords that will send a x amount of bitcoins directly to that person's wallet by replying. My twist would be that the keyword linked to trolling would send a donation to the webmaster's wallet and not the person commenting like right now.