TLDR: nice hiking and scenic views with a cool history and museum about immigration history on the island

The ferry to the island was quick from SF, less than 30 minutes. The views were very scenic even on a cloudy day. We enjoyed the immigration museum (free) and the detention center ($5) on our home around the island. From the port to the immigration museum is about a 35 minute walk (some uphill involved including long flights of stairs) but the roundabout way without stairs takes another 15-20 minutes longer.