Built in 1861, this restored mansion (2616 Pleasant St., 250/380-6506, daily 10 a.m.–4 p.m. May–mid-Sept., adult $5, senior $4, child $3) sits amid beautiful gardens along the gorge on Point Ellice, less than two kilometers (1.2 miles) from the Inner Harbour [1]. The best reason to visit is to enjoy a traditional English afternoon tea served 11 a.m.–3 p.m. ($19.95 includes admission).
To get there from the Inner Harbour, jump aboard a Victoria Harbour Ferry (10 minutes and $4 each way), or by road take Government or Douglas Street north from downtown, turn left on Bay Street, and turn left again on Pleasant Street.
Links:
[1] http://www.moon.com/destinations/vancouver-victoria/vancouver-island/victoria/inner-harbour-sights