Using built-in JavaCompiler with a custom classloader

Java 6 has a great feature built into the JVM (when it comes with JDK apparently): a programmatical access to javac functionality. There's a great article on developer works that helped me a lot when I was implementing a feature in my application that was using this functionality. In fact I just took the code from that article, applied it to my project and everything worked like magic when I was running locally from my IDE =)

So it was fine until I deployed the app to a real-life server. There the code stopped working completely. Which was not really a big surprise to me but quite frustrating. Looking into details I was able to find out the reason. The application utilising this code was loaded on the server by a custom classloader. I.e. the jvm was started with a single jar in classpath that was capable of loading different versions of the application in separate classloaders (don't ask me for reasons pls). So the compiler was failing unable to resolve dependencies.

Ok, let's do a simple example.
Imagine that this loading jar is called loader.jar and the application itself is contained in a jar app.jar. Among others there is an interface ru.atamur.Compiled there. And finally I'm trying to compile a source of:

package ru.atamur.compiled;

import ru.atamur.*;

public class Generated implements Compiled {
  @Overrides public int sum(int a, int b) { return a+b; }
}

Apparently I need the interface to access my compiled methods, so I have this dependency on classes inside the app.jar.

So why was it running locally?
if you check out the code from the article it contains this line:

JavaFileManager fileManager 
  = compiler.getStandardFileManager(diagnostics, null, null);

Basically JavaFileManager is something that allows javac to ask for classes it can't find in the source code - the dependencies.
compiler.getStandarfFileManager matching its name returns you a standard manager. Probably the one that's used by command line javac. The runtime version uses the jvm classpath to resolve all the FQNs encountered. When I was running locally - my whole app (the app.jar) was on my classpath, so this standard file manager was doing just fine, but when running on the server the only jar on the classpath was loader.jar that didn't contain ru.atamur.Compiled causing a compile error.

This might seem a bit artificial but the same problem will occur in any user code that's run inside a container (app server, web server, ioc container etc).

The most irritating detail was: all the classes the compiler needs are here in current thread's classloader, but there's no way to tell it about this fact!

Browsing google I wasn't able to find a proper solution so here's what I finally came up with: use my very own implementation of JavaFileManager that resolves the user class dependencies using a given classloader and resolves system dependencies (like java.lang) using standard file manager.

The solution might be not enough for every possible application but was just fine for my app.

First we modify that line from CharSequenceCompiler seen above:

    StandardJavaFileManager standardJavaFileManager 
      = compiler.getStandardFileManager(diagnostics, null, null);
    final JavaFileManager fileManager
      = new CustomClassloaderJavaFileManager(loader, standardJavaFileManager);

So this introduces a new class called CustomClassloaderJavaFileManager which takes as a parameter a classloader that knows where application classes reside and the standard file manager that can resolve dependencies to standard java classes.

Here is the definition of CustomClassloaderJavaFileManager:

package ru.atamur.compilation;

import java.io.*;
import java.util.*;
import javax.tools.*;

/**
 * @author atamur
 * @since 15-Oct-2009
 */
public class CustomClassloaderJavaFileManager implements JavaFileManager {
 private final ClassLoader classLoader;
 private final StandardJavaFileManager standardFileManager;
 private final PackageInternalsFinder finder;

 public CustomClassloaderJavaFileManager(ClassLoader classLoader, StandardJavaFileManager standardFileManager) {
  this.classLoader = classLoader;
  this.standardFileManager = standardFileManager;
  finder = new PackageInternalsFinder(classLoader);
 }

 @Override
 public ClassLoader getClassLoader(Location location) {
  return classLoader;
 }

 @Override
 public String inferBinaryName(Location location, JavaFileObject file) {
  if (file instanceof CustomJavaFileObject) {
   return ((CustomJavaFileObject) file).binaryName();
  } else { // if it's not CustomJavaFileObject, then it's coming from standard file manager - let it handle the file
   return standardFileManager.inferBinaryName(location, file);
  }
 }

 @Override
 public boolean isSameFile(FileObject a, FileObject b) {
  throw new UnsupportedOperationException();
 }

 @Override
 public boolean handleOption(String current, Iterator<String> remaining) {
  throw new UnsupportedOperationException();
 }

 @Override
 public boolean hasLocation(Location location) {
  return location == StandardLocation.CLASS_PATH || location == StandardLocation.PLATFORM_CLASS_PATH; // we don't care about source and other location types - not needed for compilation
 }

 @Override
 public JavaFileObject getJavaFileForInput(Location location, String className, JavaFileObject.Kind kind) throws IOException {
  throw new UnsupportedOperationException();
 }

 @Override
 public JavaFileObject getJavaFileForOutput(Location location, String className, JavaFileObject.Kind kind, FileObject sibling) throws IOException {
  throw new UnsupportedOperationException();
 }

 @Override
 public FileObject getFileForInput(Location location, String packageName, String relativeName) throws IOException {
  throw new UnsupportedOperationException();
 }

 @Override
 public FileObject getFileForOutput(Location location, String packageName, String relativeName, FileObject sibling) throws IOException {
  throw new UnsupportedOperationException();
 }

 @Override
 public void flush() throws IOException {
  // do nothing
 }

 @Override
 public void close() throws IOException {
  // do nothing
 }

 @Override
 public Iterable<JavaFileObject> list(Location location, String packageName, Set<JavaFileObject.Kind> kinds, boolean recurse) throws IOException {
  if (location == StandardLocation.PLATFORM_CLASS_PATH) { // let standard manager hanfle
   return standardFileManager.list(location, packageName, kinds, recurse);
  } else if (location == StandardLocation.CLASS_PATH && kinds.contains(JavaFileObject.Kind.CLASS)) {
   if (packageName.startsWith("java")) { // a hack to let standard manager handle locations like "java.lang" or "java.util". Prob would make sense to join results of standard manager with those of my finder here
    return standardFileManager.list(location, packageName, kinds, recurse);
   } else { // app specific classes are here
    return finder.find(packageName);
   }
  }
  return Collections.emptyList();

 }

 @Override
 public int isSupportedOption(String option) {
  return -1;
 }

}

So this class implements only a subset of functionality defined by JavaFileManager interface but that appears to be enough to handle the compilation (at least in my case). It also has some not-that-pretty code to handle both custom classloader classes and standard java classes.

There are 2 new classes used here PackageInternalsFinder and CustomJavaFileObject. FIrst one is a utility class that translates JavaFileManager requests to classloader requests and the second one is a custom implementation of JavaFileObject interface. JDK already contains an implementation of JavaFileObject: SimpleJavaFileObject based on URI that seemed suitable for my needs but it heavily depends on the URI having non null path which is not true for JAR URIs, so I had to develop my own version:

package ru.atamur.compilation;

import java.net.URI;
import java.io.*;
import javax.tools.JavaFileObject;
import javax.lang.model.element.NestingKind;
import javax.lang.model.element.Modifier;

/**
* @author atamur
* @since 15-Oct-2009
*/
class CustomJavaFileObject implements JavaFileObject {
 private final String binaryName;
 private final URI uri;
 private final String name;

 public CustomJavaFileObject(String binaryName, URI uri) {
  this.uri = uri;
  this.binaryName = binaryName;
  name = uri.getPath() == null ? uri.getSchemeSpecificPart() : uri.getPath(); // for FS based URI the path is not null, for JAR URI the scheme specific part is not null
 }

 @Override
 public URI toUri() {
  return uri;
 }

 @Override
 public InputStream openInputStream() throws IOException {
  return uri.toURL().openStream(); // easy way to handle any URI!
 }

 @Override
 public OutputStream openOutputStream() throws IOException {
  throw new UnsupportedOperationException();
 }

 @Override
 public String getName() {
  return name;
 }

 @Override
 public Reader openReader(boolean ignoreEncodingErrors) throws IOException {
  throw new UnsupportedOperationException();
 }

 @Override
 public CharSequence getCharContent(boolean ignoreEncodingErrors) throws IOException {
  throw new UnsupportedOperationException();
 }

 @Override
 public Writer openWriter() throws IOException {
  throw new UnsupportedOperationException();
 }

 @Override
 public long getLastModified() {
  return 0;
 }

 @Override
 public boolean delete() {
  throw new UnsupportedOperationException();
 }

 @Override
 public Kind getKind() {
  return Kind.CLASS;
 }

 @Override // copied from SImpleJavaFileManager
 public boolean isNameCompatible(String simpleName, Kind kind) {
  String baseName = simpleName + kind.extension;
  return kind.equals(getKind())
    && (baseName.equals(getName())
    || getName().endsWith("/" + baseName));
 }

 @Override
 public NestingKind getNestingKind() {
  throw new UnsupportedOperationException();
 }

 @Override
 public Modifier getAccessLevel() {
  throw new UnsupportedOperationException();
 }

 public String binaryName() {
  return binaryName;
 }


 @Override
 public String toString() {
  return "CustomJavaFileObject{" +
    "uri=" + uri +
    '}';
 }
}

again I didn't implement methods that are not absolutely required for doing compilation - and that's fine in my case.

Last bit to actually work out URIs for classes needed by the compiler:

package ru.atamur.compilation;

import java.util.List;
import java.util.ArrayList;
import java.util.Enumeration;
import java.util.Collection;
import java.util.jar.JarEntry;
import java.io.IOException;
import java.io.File;
import java.net.URL;
import java.net.JarURLConnection;
import java.net.URI;
import javax.tools.JavaFileObject;

/**
* @author atamur
* @since 15-Oct-2009
*/
class PackageInternalsFinder {
 private ClassLoader classLoader;
 private static final String CLASS_FILE_EXTENSION = ".class";

 public PackageInternalsFinder(ClassLoader classLoader) {
  this.classLoader = classLoader;
 }

 public List<JavaFileObject> find(String packageName) throws IOException {
  String javaPackageName = packageName.replaceAll("\\.", "/");

  List<JavaFileObject> result = new ArrayList<JavaFileObject>();

  Enumeration<URL> urlEnumeration = classLoader.getResources(javaPackageName);
  while (urlEnumeration.hasMoreElements()) { // one URL for each jar on the classpath that has the given package
   URL packageFolderURL = urlEnumeration.nextElement();
   result.addAll(listUnder(packageName, packageFolderURL));
  }

  return result;
 }

 private Collection<JavaFileObject> listUnder(String packageName, URL packageFolderURL) {
  File directory = new File(packageFolderURL.getFile());
  if (directory.isDirectory()) { // browse local .class files - useful for local execution
   return processDir(packageName, directory);
  } else { // browse a jar file
   return processJar(packageFolderURL);
  } // maybe there can be something else for more involved class loaders
 }

 private List<JavaFileObject> processJar(URL packageFolderURL) {
  List<JavaFileObject> result = new ArrayList<JavaFileObject>();
  try {
   String jarUri = packageFolderURL.toExternalForm().split("!")[0];

   JarURLConnection jarConn = (JarURLConnection) packageFolderURL.openConnection();
   String rootEntryName = jarConn.getEntryName();
   int rootEnd = rootEntryName.length()+1;

   Enumeration<JarEntry> entryEnum = jarConn.getJarFile().entries();
   while (entryEnum.hasMoreElements()) {
    JarEntry jarEntry = entryEnum.nextElement();
    String name = jarEntry.getName();
    if (name.startsWith(rootEntryName) && name.indexOf('/', rootEnd) == -1 && name.endsWith(CLASS_FILE_EXTENSION)) {
     URI uri = URI.create(jarUri + "!/" + name);
     String binaryName = name.replaceAll("/", ".");
     binaryName = binaryName.replaceAll(CLASS_FILE_EXTENSION + "$", "");

     result.add(new CustomJavaFileObject(binaryName, uri));
    }
   }
  } catch (Exception e) {
   throw new RuntimeException("Wasn't able to open " + packageFolderURL + " as a jar file", e);
  }
  return result;
 }

 private List<JavaFileObject> processDir(String packageName, File directory) {
  List<JavaFileObject> result = new ArrayList<JavaFileObject>();

  File[] childFiles = directory.listFiles();
  for (File childFile : childFiles) {
   if (childFile.isFile()) {
    // We only want the .class files.
    if (childFile.getName().endsWith(CLASS_FILE_EXTENSION)) {
     String binaryName = packageName + "." + childFile.getName();
     binaryName = binaryName.replaceAll(CLASS_FILE_EXTENSION + "$", "");

     result.add(new CustomJavaFileObject(binaryName, childFile.toURI()));
    }
   }
  }

  return result;
 }
}

This code is tested under both windows and linux OS and both from command line and IDE.

Hope someone will find this in time of need =)