The project is set up so that every module has a test.AllTests test suite that, in a hierarchical fashion, includes all the tests in that module. Once such a test suite has been invoked for a module, however, IDEA keeps reusing that run configuration for the similar test suites from other modules (I guess it indexes then based on FQN of the test class, or something). By reusing I mean the set of test that are being run is always the one corresponding to the existing run config, not the tests grouped by the new test suite (the one being invoked through right click/run). This is, obviously, not the desired behaviour, as one has to delete the first run config in order to be able to invoke another test suite.