Right now,. it's not possible to use a non-standard doclet from the tools->javadoc interface.
It seems like the reason is that the options used by the itnerface (and which cannot be changed) include options that are only available to the standard doclet. the javadoc tool recognizes two sets of options- those that can be used by all doclets, and those that are special to the standard doclet.
Behind the scenes, IntelliJ invokes javadoc with those special-to-standard-doclet options. The effect is, you can't specify a non-standard doclet and have it work; you'll get (at least)
javadoc: error - invalid flag: -splitindex
Of course, it's not impossible for a doclet developer to support all the command line flags, but that' s a lot of work for no reason and not what's expected from doclet writers.