Selenium IE Webdriver in Java

Recently at the company I work for, we've been using the Robot Framework and Selenium for automated testing.  We've had great success so far, Firefox and Chrome have been working great.  However, IE was a bit more problematic for us.  The default IE web driver for IE has been very flaky for us.  We've had issue such as xpath expressions failing, random javascript errors and even crashes.  Even when testing on Sauce Labs, we've experienced these issues.  Given that these same tests worked for Chrome and Firefox without issue, naturally we blamed the IE driver.

After we gave up on the standard IE driver, we started researching other solutions for testing IE.  Watir was the first interesting project we looked at.  It looked good, but it was a Ruby project and we don't have much Ruby expertise.  I found a java equivalent called Watij and that showed more promise for us.  Unfortunately, the project is several years old and didn't support 64-bit JVM's.  In addition, Watij is based on a set of proprietary COM API's from a company called teamdev.  This made it difficult to track down issues when the COM calls would fail.  So I started evaluating calling IE COM directly from Java.

I finally settled on COM4j for the bulk of the IE interaction with addtional help from JNA to call some of the win32 API's.  This combination has worked well for us and we were able to get a Java IE WebDriver going.  We've created a googlecode project to host this effort:
http://code.google.com/p/java-ie-webdriver/

So far we've had great success with this driver and one of our projects has 50+ robot tests that are all passing in both IE 8 and IE 9 using this driver.  At this point, most of the driver is implemented, including all the element finder methods and full file upload support. So feel free to give it a try and let us know how it works for you.

CXF and MS CRM 2011

Background
Early last year, Microsoft released a new version of their CRM product, Dynamics CRM 2011. My company has a Java product that integrates with MS CRM and we were successfully able to call the webservice that is included with CRM 2011 using SAML tokens. Everything was working great until we reconfigured CRM for on-premise authentication. In the previous version of MS CRM, the on-premise authentication used transport layer NTLM/Kerberos authentication which Java supports. With CRM 2011, however, Microsoft is using WCF 4.0 which uses message level NTLM/Kerberos based on SPNEGO WS-Trust. (Specification) We searched extensively, but could not find an implementation of this protocol for any open source Java Web Service framework. So I along with a coworker set out to implement this for CXF. After several weeks, we were successful and were able to invoke the CRM web service from Java. We submitted this functionality back to the CXF project. (JIRA) Recently, Colm O hEigeartaigh was kind enough to review the patch and integrate it into the trunk of the CXF codebase. (With some major refactoring to make it fit better) This works out of the box with the latest CXF code, but the configuration of this is far from trivial, so this blog post explains how to configure this new functionality.

CXF and MS CRM 2011
The first step in getting this all to work is to setup Java to authenticate using Kerberos with the Active Directory primary domain controller. This requires you to create a login.conf as follows:

spnego-client {
 com.sun.security.auth.module.Krb5LoginModule required;
};

and a krb5.conf as follows:

[libdefaults]
 default_realm = <NTdomain>
 default_tkt_enctypes = aes128-cts rc4-hmac des3-cbc-sha1 des-cbc-md5 des-cbc-crc
 default_tgs_enctypes = aes128-cts rc4-hmac des3-cbc-sha1 des-cbc-md5 des-cbc-crc
 permitted_enctypes   = aes128-cts rc4-hmac des3-cbc-sha1 des-cbc-md5 des-cbc-crc

[realms]
 <NTdomain>  = {
  kdc = <PrimaryDomainController>
  default_domain = <NTdomain>
}

[domain_realm]
 .<NTdomain> = <NTdomain>

Then, in my Java code, I manually configure the following the system properies so that JAAS will use these files:

System.setProperty("java.security.auth.login.config", "C:\\projects\\login.conf");
   System.setProperty("java.security.krb5.conf", "C:\\projects\\krb5.conf");

CRM 2011 includes a custom policy element in the WSDL that needs to be asserted in CXF. If this is not done, CXF will error out because it doesn't have a policy provider to assert the custom policy. This custom policy can be asserted using the following classes, the first is the PolicyProvider itself:

public class XRMAuthPolicyProvider extends AbstractPolicyInterceptorProvider
{
 public XRMAuthPolicyProvider()
 {
   super(Arrays
     .asList(new QName[]{new QName("http://schemas.microsoft.com/xrm/2011/Contracts/Services",
       "AuthenticationPolicy", "ms-xrm")}));
   getInInterceptors().add(new XRMAuthPolicyInterceptor());
 }
}

and the second is the associated CXF interceptor:

public class XRMAuthPolicyInterceptor extends AbstractSoapInterceptor
{
 public XRMAuthPolicyInterceptor()
 {
   super(Phase.PRE_PROTOCOL);
   addAfter(PolicyBasedWSS4JInInterceptor.class.getName());
   addAfter(PolicyBasedWSS4JOutInterceptor.class.getName());
 }

 public void handleMessage(SoapMessage message) throws Fault
 {
   AssertionInfoMap aim = message.get(AssertionInfoMap.class);
   if (null == aim)
   {
     return;
   }
   QName qname = new QName("http://schemas.microsoft.com/xrm/2011/Contracts/Services",
     "AuthenticationPolicy", "ms-xrm");
   Collection ais = aim.get(qname);
   if (null == ais || ais.size() == 0)
   {
     return;
   }
   for (AssertionInfo ai : ais)
   {
     ai.setAsserted(true);
   }
 }
}

The policy provider can be added to web service client as follows:

   Client client = ClientProxy.getClient(port);
   Bus bus = ((EndpointImpl) client.getEndpoint()).getBus();
   PolicyInterceptorProviderRegistry pipr = bus
     .getExtension(PolicyInterceptorProviderRegistry.class);
   pipr.register(new XRMAuthPolicyProvider());

Once the policy provider is added, CXF needs to be configured to use Kerberos authentication. For my setup, I was using an Active Directory username and password, so I had to create an implementation of SpnegoClientAction that uses GSSName.NT_USER_NAME rather than the default GSSName.NT_HOSTBASED_SERVICE:

public class XRMSpnegoClientAction implements SpnegoClientAction
{
  private org.apache.commons.logging.Log log = org.apache.commons.logging.LogFactory
    .getLog(DefaultSpnegoClientAction.class);

  protected String serviceName;
  protected GSSContext secContext;
  protected boolean mutualAuth;

  /**
   * Whether to enable mutual authentication or not.
   */
  public void setMutualAuth(boolean mutualAuthentication)
  {
    mutualAuth = mutualAuthentication;
  }

  /**
   * The Service Name
   */
  public void setServiceName(String serviceName)
  {
    this.serviceName = serviceName;
  }

  /**
   * Obtain a service ticket
   */
  public byte[] run()
  {
    try
    {
      GSSManager gssManager = GSSManager.getInstance();
      Oid oid = new Oid("1.3.6.1.5.5.2");

      GSSName gssService = gssManager.createName(serviceName, GSSName.NT_USER_NAME);
      secContext = gssManager.createContext(gssService, oid, null, GSSContext.DEFAULT_LIFETIME);

      secContext.requestMutualAuth(mutualAuth);
      secContext.requestCredDeleg(Boolean.FALSE);

      byte[] token = new byte[0];
      return secContext.initSecContext(token, 0, token.length);
    }
    catch (GSSException e)
    {
      if (log.isDebugEnabled())
      {
        log.debug("Error in obtaining a Kerberos token", e);
      }
    }

    return null;
  }

  /**
   * Get the GSSContext that was created after a service ticket was obtained
   */
  public GSSContext getContext()
  {
    return secContext;
  }
}

Then I added the following code to hook Kerberos into my web service client:

   // Active Directory domain username and password
   final String username = "<username>";
   final String password = "<password>";
   // Kerberos service provider name, e.g. RestrictedKrbHost/<computername>
   String spn = "RestrictedKrbHost/<crm_server>";
   // Kerberos JAAS client as configured in login.conf
   String jaasClient = "spnego-client";
   // Active Directory username and password
   CallbackHandler callbackHandler = new NamePasswordCallbackHandler(username, password);

   client.getRequestContext().put("ws-security.kerberos.jaas.context", jaasClient);
   client.getRequestContext().put("ws-security.kerberos.spn", spn);
   client.getRequestContext().put("ws-security.callback-handler", callbackHandler);
   client.getRequestContext().put("ws-security.spnego.client.action", new XRMSpnegoClientAction());

Once the custom policy provider is added and the Kerberos authentication is setup, the client configuration is complete.

For reference, the complete client code example is below:

   URL wsdlURL = OrganizationService.WSDL_LOCATION;

   System.setProperty("java.security.auth.login.config", "C:\\projects\\login.conf");
   System.setProperty("java.security.krb5.conf", "C:\\projects\\krb5.conf");

   // Active Directory domain username and password
   final String username = "<username>";
   final String password = "<password>";
   // Kerberos service provider name, e.g. RestrictedKrbHost/<computername>
   String spn = "RestrictedKrbHost/<crm_server>";
   // Kerberos JAAS client as configured in login.conf
   String jaasClient = "spnego-client";
   // Active Directory username and password
   CallbackHandler callbackHandler = new NamePasswordCallbackHandler(username, password);

   OrganizationService ss = new OrganizationService(wsdlURL, SERVICE_NAME);
   IOrganizationService port = ss.getCustomBindingIOrganizationService();

   Client client = ClientProxy.getClient(port);
   Bus bus = ((EndpointImpl) client.getEndpoint()).getBus();
   PolicyInterceptorProviderRegistry pipr = bus
     .getExtension(PolicyInterceptorProviderRegistry.class);
   pipr.register(new XRMAuthPolicyProvider());

   client.getRequestContext().put("ws-security.kerberos.jaas.context", jaasClient);
   client.getRequestContext().put("ws-security.kerberos.spn", spn);
   client.getRequestContext().put("ws-security.callback-handler", callbackHandler);
   client.getRequestContext().put("ws-security.spnego.client.action", new XRMSpnegoClientAction());

   // call webservice
   ColumnSet colSet = new ColumnSet();
   colSet.setAllColumns(true);
   colSet.setColumns(new ArrayOfstring());
   QueryByAttribute qba = new QueryByAttribute();
   qba.setEntityName("account");
   qba.setColumnSet(colSet);
   ArrayOfstring aos2 = new ArrayOfstring();
   aos2.getString().add("accountid");
   qba.setAttributes(aos2);
   ArrayOfanyType aoat = new ArrayOfanyType();
   aoat.getAnyType().add("2D08FEC1-9734-E111-9454-000C295D5DEA");
   qba.setValues(aoat);
   EntityCollection entityCol = port.retrieveMultiple(qba);
   List props = entityCol.getEntities().getEntity().get(0)
     .getAttributes().getKeyValuePairOfstringanyType();
   for (KeyValuePairOfstringanyType prop : props)
   {
     if ("name".equals(prop.getKey()))
     {
       System.out.println(prop.getKey() + " = " + prop.getValue());
     }
   }

Since our team spent several weeks working on this example, I hope this walkthrough is helpful for other developers out there. This will not only work for CRM 2011, but any WCF webservice that uses message level NTLM/Kerberos authentication.

**Update (2013-08-26)** With newer versions of CXF (tested against CXF 2.6.9) it is necessary to include a bindings file when generating the client code using wsdl2java:

<jaxb:bindings version="2.1"
xmlns:jaxb="http://java.sun.com/xml/ns/jaxb"
xmlns:xjc="http://java.sun.com/xml/ns/jaxb/xjc"
xmlns:xs="http://www.w3.org/2001/XMLSchema">
  <jaxb:globalBindings generateElementProperty="false" mapSimpleTypeDef="true"/>
</jaxb:bindings>