You can deselect objects from the current selection by holding the <Shift> key while you select objects to remove from the selection.