I actually went through the same scenario with a court martial. The character really messed up big time and on numerous occassions. He knew he was in trouble. So, we talked about it before we actually gamed. I gave him one whole time in between gaming sessions to think things over and we made sure it was clear why he was going through this. Luckily, he understood.
The trial itself was pretty easy. I brought in the other PCs as witnesses and I ran the judges and JAG officers. Things went well and in the end, he only ended up with a major reprimand. Of course, later on down the road he quit Starfleet because he was going to undergo another court martial.
Someone earlier posted about bringing in guest players to act as the judges or JAGs. I have to strongly disagree from experience. I once brought in guest players for roles like this and also guest played roles like this. Both results were disastrous.
The problem is the guest players, do not have any true knowledge of the campaign or what they should expect out of the players. The players may have problems with the fate of their characters being in the hands of a strangers. And if you give the guest players a "script", then its not true role-playing as no matter what the characters do, the end result would be the same.
I hope this helps!