Kune

Htmlunit process is controlled from this servlet filter class

cc.kune.core.server.searcheable.SearchEngineServletFilter

Init method sets thread configuration:

  public void init(final FilterConfig filterConfig) throws ServletException {
    this.filterConfig = filterConfig;
    cache = new Cache();
    executor = Executors.newFixedThreadPool(THREADS);
  }

We have to define following issues to address a solution:

• WHEN have the server to avoid launching htmlunit? current server load %, current mem usage %...
• WHAT behavior will the server have in that case? response HTTP 404?, response HTTP 500? I think crawlers are very sensitive to this!

Other thing I was thinking (and trying without success), was to maintain open the htmlunit WebClient, and not to do a client.closeAllWindows(); only when servlet destroy. I was trying to cache and make the page request faster, but... maybe we have to try again.

By the way, as we are using cloudflare CDN, I've added this rule:
kune.cc/*escaped_fragment*
Cache level: Aggressive caching

After our last conversation, and your work with #70 I just added some mbean methods, and also I have refactorized a little bit this servlet. In short (when this is installed in kune.cc) we can debug this.