If you want to remove a color, select it in the color list of the Color inspector and click the [–] button in the left bottom corner. Chances are that you will then have to choose a replacement color for the removed color:
