I eventually made ikiwiki go much faster than what I was experiencing before. It had turned out that the aggregate plugin left some trash in my tree of files, I some how ended up accumulating about 10k number of small files.

After I had removed the loose files I ran a rebuild/refresh, and now its down to ~3seconds for a refresh of my site. Not bad I guess.