If you want to remove all contacts of a group, try removing the group instead.
To remove contacts, select the contact row (you may select more than one contacts by Ctrl + Click) and click Remove button.