In 1840, Inverness Mayor James Saffran built the distillery on the ruins of the Muirtown brewery, beside the Caledonian Canal, allowing convenient access to southern markets by sea. However, nine years after opening, the distillery suffered a fire and severe damage. In 1855, Saffran's property was confiscated. The distillery and its buildings stood vacant for 20 years, during which time it was used only as a mill. After 1884, grain merchant William Grigor acquired it and built a larger distillery on the original site. Manager John Birnie, in partnership with Charles Mackinlay & Company, purchased land across the way in 1892 and built Glen Mhor Distillery (see related entry). In 1920, the two companies acquired Glen Albyn. In 1972, Glen Albyn was purchased by D.C.L. and transferred to S.M.D. Both distilleries closed in 1983, and the Glen Albyn site is now a supermarket.

During World War I, Glen Elgin was reportedly requisitioned by the Admiralty as a naval base for manufacturing mines, though other sources claim it served as a station for building breakwaters. In 1954, Saladin boxes replaced the distillery's traditional floor malting. Despite being just a stone's throw away, Glen Elgin's whisky is entirely different from Glenmorangie and commands higher praise.