http://www.intellij.net/tracker/idea/viewSCR?publicId=24203
The simplest way to reproduce this is to add a folder to the project path and inside put a symlink to it (eg foo -> .) Granted, not the best idea, but it happens. With this or any type of more complicated circular reference, IDEA will (it seems) try to traverse the whole tree and get caught on an infinite loop consuming memory, until it runs out.
This also seems to happen with compilation, even if all the folders that have recursive directories in the project path have been excluded. In this case classes will be compiled, but then IDEA will use up all the memory anyway.
It would be nice if it realized when there was a circular reference and stopped. Maybe this could be done by keeping track of all the canonical names for the (sub)folders visited, and not visiting a folder more than once.