Temples in the area are said to date back to imperial orders issued by Emperor Kōnin in 774, but Kōzan-ji (formally Toganōsan Kōsan-ji) was officially founded by scholar and monk Myōe (1173 – 1232) who served at nearby Jingo-ji before he was granted the land to construct a temple by Emperor Go-Toba in 1206.
There may already have been a temple on the site with Myōe doing a restoration job, but there's a diagram housed at Jingo-ji drawn in 1230, showing the thirteenth century layout of the temple, which consisted of a large gate, a main hall, a three-storied pagoda, halls dedicated to Amitabha and Lohan, a bell tower, a scripture hall (originally the residence of a member of the Imperial family), and a Shinto shrine.
Structures on the site have been destroyed numerous times by fire and war and the oldest buildings standing today is the scripture hall, now known as Sekisui-in and Myōe's former residence, two of the few remaining examples of Kamakura Era architecture, with a roof of thatch and shingles.
The grounds also hold the oldest tea field in Japan, planted by Myoe with seeds brought from China by the Zen priest Eisai. Tea helped monks stay awake during late-night meditation.
Kōzan-ji is home to numerous national treasures and important cultural properties though most of them of them are currently on loan to museums in Kyoto and Tokyo. The Chōjū-jinbutsu-giga (The Scroll of Frolicking Animals and Humans), a set of four picture scrolls of ink paintings from the twelfth and thirteenth centuries, sometimes described as Japan's first comic, that takes the mickey out of human foibles by showing frogs, monkeys, birds, and rabbits engaged in worldly pursuits. The original is currently housed in Tokyo National Museum but apparently there are precise replicas on display on site. I say apparently because I was more interested in the ambience among the towering cedar trees and moss-covered ground than checking out areas that were likely to attract a crowd.
Heading back down the hill I wasn't exactly looking forward to our other temple visit since the morning was getting on and there were obvious expectations of substantial crowds among the roadside merchandisers. Most of the stall holders had just about finished setting up as we retraced our tracks along the riverside side road, and when we made it back to the original starting point it looked like we were in for another experience in crowded sightseeing.