is there a way to ban someone specific on my server?

Yes, in the expert section add the steam ip separated by commas and save. In the next restart you can not enter.