Right-click (anywhere in a view - there is no need to be over an element in the reference). Select Deactivate <model_name>.

or

In the References dialog's list box, select the list entry for the reference. In the list entry's Activate Status column, click the black dot.