java.io.IOException: HTTPS hostname wrong

While I was testing creating a secure connection to a new REST service am integrating with & despite I made sure I imported their certificate, I was getting the exception below:

java.io.IOException: HTTPS hostname wrong:  should be <connect2.bglobale.com>
at com.ibm.net.ssl.www2.protocol.https.c.b(c.java:117)
at com.ibm.net.ssl.www2.protocol.https.c.afterConnect(c.java:138)
at com.ibm.net.ssl.www2.protocol.https.d.connect(d.java:64)
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1024)
at com.ibm.net.ssl.www2.protocol.https.b.getOutputStream(b.java:51)
at temporary.test.TestConnection.main(TestConnection.java:36)


After some investigation, I found that a self signed certificate might raise this exception even if it is created with the right domain (connect2.bglobale.com in my case) & the only way around it, is to force Java to fully trust the host-name which be achieved using the "Hostname Verifier" as shown in the example below:

package temporary.test;

import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.io.OutputStreamWriter;
import java.net.URL;
import javax.net.ssl.HostnameVerifier;
import javax.net.ssl.HttpsURLConnection;
import javax.net.ssl.SSLSession;


public class TestConnection {

public static void main (String args[]){
StringBuffer jsonString=new StringBuffer();
String line;

HostnameVerifier hv = new HostnameVerifier(){
   public boolean verify(String urlHostName, SSLSession session){
       System.out.println("Warning: URL Host: " + urlHostName + " vs. " + session.getPeerHost());
       return true;
   }
};

try{
       URL url = new URL("https://connect2.bglobale.com/Browsing/AppSettings?merchantGUID=1fccd7e5"); 
       HttpsURLConnection connection = (HttpsURLConnection) url.openConnection();
       connection.setHostnameVerifier(hv);
       connection.setDoOutput(true);
       connection.setDoInput(true);
       connection.setRequestMethod("POST");
       
       String data = "";
       connection.setRequestProperty("Content-Length",""+data.length());
   
       OutputStreamWriter writer = new OutputStreamWriter(connection.getOutputStream());
       writer.write("" + data);
       writer.close();
       
       System.out.println("Response code: "+connection.getResponseCode());
       System.out.println("Response message: "+connection.getResponseMessage());
       
       BufferedReader br = new BufferedReader(new InputStreamReader(connection.getInputStream()));
       while ((line = br.readLine()) != null) {
               jsonString.append(line);
       }
       System.out.println("JSON String = "+jsonString);
       br.close();
       connection.disconnect();
}catch(Exception e){
e.printStackTrace();
}
}
}

The code highlighted in red is the one required to fix the problem.

Chrome vs Firefox with jQuery AJAX async parameter set to false

We had a button and upon click on it, we display a progress image, send three AJAX requests using jQuery and async set to false and we once all requests are complete, we hide the progress image.

We noticed a strange behaviour when comparing Chrome vs Firefox. On Firefox, the browser responsiveness is normal, it displays the progress image and things execute as designed, while on Chrome, it stops responding for around a second, suddenly progress image shows up and disappears again right away as if all actions took place suddenly.

Changing async parameter to true, made Chrome to behave normally again and in a way similar to Firefox. We didn't reach a clear explanation for why this happens on Chrome but seems it causes some sort of threading issue.

Setting JVM arguments for ANT Java task

Setting the JVM properties for a Java ANT task can be bit confusing and to make it clear, am showing two samples below to compare the wrong way for doing vs the right way.

Wrong way

<java classname="com.tmpl.commerce.catalog.dataload.datareader.CatalogEntryCatalogGroupRelationPreProcessor" fork="true" failonerror="true" output="${log.dir}/dataload-preprocess-${dataload.startTime}.log">
<arg value="${dataload.preprocess}"/>
<arg value="${dataload.catalogId}"/>
<arg value="${db.user}"/>
<arg value="${db.pwd.encrypted}"/>
<arg value="${db.name}"/>
<arg value="${db.server}"/>
<arg value="${db.port}"/>
<arg value="1000" />
<arg value="false"/>
<jvmarg value="-Dwc.home=/usr/WebSphere/CommerceServer70 -Djava.util.logging.config.file=logging.properties" />
<classpath>
<path refid="classpath" />
</classpath>
</java>

Right way

<java classname="com.tmpl.commerce.catalog.dataload.datareader.CatalogEntryCatalogGroupRelationPreProcessor" fork="true" failonerror="true" output="${log.dir}/dataload-preprocess-${dataload.startTime}.log">
<arg value="${dataload.preprocess}"/>
<arg value="${dataload.catalogId}"/>
<arg value="${db.user}"/>
<arg value="${db.pwd.encrypted}"/>
<arg value="${db.name}"/>
<arg value="${db.server}"/>
<arg value="${db.port}"/>
<arg value="1000" />
<arg value="false"/>
<jvmarg value="-Dwc.home=/usr/WebSphere/CommerceServer70 " />
  <jvmarg value="-Djava.util.logging.config.file=logging.properties" />
<classpath>
<path refid="classpath" />
</classpath>
</java>

Paypal sandbox SSL exception & SSL 3.0 Protocol Vulnerability

Problem

Connection to Paypal sandbox was broken on/after 13th of October and the SSL exception below was thrown from Paypal core Java client.

Caused by: javax.net.ssl.SSLException: Unsupported record version Unknown-0.0
                at com.ibm.jsse2.b.b(b.java:102)
                at com.ibm.jsse2.b.a(b.java:212)
                at com.ibm.jsse2.SSLSocketImpl.a(SSLSocketImpl.java:814)
                at com.ibm.jsse2.SSLSocketImpl.h(SSLSocketImpl.java:704)
                at com.ibm.jsse2.SSLSocketImpl.a(SSLSocketImpl.java:12)
                at com.ibm.jsse2.SSLSocketImpl.startHandshake(SSLSocketImpl.java:498)
                at com.ibm.net.ssl.www2.protocol.https.c.afterConnect(c.java:59)
                at com.ibm.net.ssl.www2.protocol.https.d.connect(d.java:31)
                at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1195)
                at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:379)
                at com.ibm.net.ssl.www2.protocol.https.b.getResponseCode(b.java:91)
                at com.paypal.core.HttpConnection.execute(HttpConnection.java:93)
                at com.paypal.core.APIService.makeRequestUsing(APIService.java:176)
                at com.paypal.core.BaseService.call(BaseService.java:258)
                at urn.ebay.api.PayPalAPI.PayPalAPIInterfaceServiceService.setExpressCheckout(PayPalAPIInterfaceServiceService.java:2196)
                at urn.ebay.api.PayPalAPI.PayPalAPIInterfaceServiceService.setExpressCheckout(PayPalAPIInterfaceServiceService.java:2148)

Fix

Paypal sandbox was updated on/around 13th of October to disable SSLv3 and enable TLS instead as a response to POODLE Security Vulnerability. This change on their side mandates replacing Paypal client Jars to latest version especially paypal-core which needs to be on version 1.6.4 at least as it has the necessary code change to initialize a TLS connection.

To prepare a fix, you need to build new jars from Paypal GitHub links provided below:

Paypal SDK Core Java
Paypal Merchant SDK Java

For quick testing, you can use the two jars below which I build myself:

Download PayPal Merchant SDK
Download Paypal SDK Core Java

Remarks

• The issues above were encountered while using Paypal Mercahant SDK and don't know if their newer library (REST SDK) had a similar issue or not

Warning !

• The jars provided above are for quick testing only and for use on a test non-live environment
• Never deploy a payment related jar from untrusted source (like myself). Once you are sure the jars fixes your problem, please contact Paypal to prepare a new jar for you or do it yourself and build their code available on GitHub.

When JSP coding turns Dyncache into a nightmare

The two code samples below looks very similar and they both work fine when tested in a local development environment without caching enabled. But once WebSphere Dynacache is enabled, the first code sample will start causing weird effects on front end. To be more specific, on first access to the page, the content that is generated by CategoryRecommendation.jsp will render correctly without problems, but subsequent access (e.g. page refresh) which is supposed to fetch content from cache, the content will suddenly disappear.

Whenever you see this issue, first thing to look for is your JSP flush which needs to be positioned exactly before and after c:import without anything in-between.

That is one of the reasons I personally prefer to use jsp:import as you can pass flush as an attribute to the tag and you don't have to worry about doing such a mistake.

Code sample 1

<%out.flush();%>
<c:import var="fscontent" url="/ArgSharedStore/Widgets/Tmpl/ESpot/CategoryRecommendation/CategoryRecommendation.jsp">
<c:param name="emsName" value="${spotName}" />
<c:param name="cacheWithParent" value="false" />
<c:param name="catalogId" value="${catalogId}" />
<c:param name="align" value="${align}" />
</c:import>
<c:out value="${fscontent}" escapeXml="false" />
<%out.flush();%>

Code sample 2

<%out.flush();%>
<c:import var="fscontent" url="/ArgSharedStore/Widgets/Tmpl/ESpot/CategoryRecommendation/CategoryRecommendation.jsp">
<c:param name="emsName" value="${spotName}" />
<c:param name="cacheWithParent" value="false" />
<c:param name="catalogId" value="${catalogId}" />
<c:param name="align" value="${align}" />
</c:import>
<%out.flush();%>

<c:out value="${fscontent}" escapeXml="false" />

Tip for handling exceptions in WebSphere Commerce

Overview

When integrating with 3rd party jar files (e.g. Paypal as shown in examples below), it is important when catching exceptions thrown by those jars, to include the original exception (known as cause) when throwing an application specific exception.

In sample code given below, the first method will throw PayPalException and just the exception message while the second code sample will include the original exception itself too.

What difference will that make ? if you face a problem on production, a stack trace similar to the one given below will show up, but the second code sample will show more details which are marked in red in stack trace.

As you can see from the stack trace, you will get more details on what happened within your 3rd party jar files and if you have to contact their support team, you are able to give them a proper exception stack trace that helps them (and subsequently yourself too) to identify the problem and advise you with proper fix.

Sample code

private SetExpressCheckoutResponseType callPayPalSetExpressCheckOut(SetExpressCheckoutReq setExpressCheckoutReq,

PayPalAPIInterfaceServiceService service) throws PayPalException {

final String METHOD_NAME = "callPayPalSetExpressCheckOut()";

LOGGER.entering(CLASSNAME, METHOD_NAME);

String responseToken = null;

SetExpressCheckoutResponseType setExpressCheckoutResponse;

try {

//Execute the API operation and obtain the response.

setExpressCheckoutResponse = service.setExpressCheckout(setExpressCheckoutReq);

} catch (Exception ex) {

throw new PayPalException(CLASSNAME, METHOD_NAME, ex.getMessage());

}

LOGGER.exiting(CLASSNAME, METHOD_NAME);

return setExpressCheckoutResponse;

}

private SetExpressCheckoutResponseType callPayPalSetExpressCheckOut(SetExpressCheckoutReq setExpressCheckoutReq,

PayPalAPIInterfaceServiceService service) throws PayPalException {

final String METHOD_NAME = "callPayPalSetExpressCheckOut()";

LOGGER.entering(CLASSNAME, METHOD_NAME);

String responseToken = null;

SetExpressCheckoutResponseType setExpressCheckoutResponse;

try {

//Execute the API operation and obtain the response.

setExpressCheckoutResponse = service.setExpressCheckout(setExpressCheckoutReq);

} catch (Exception ex) {

throw new PayPalException(CLASSNAME, METHOD_NAME, ex.getMessage(),ex);

}

LOGGER.exiting(CLASSNAME, METHOD_NAME);

return setExpressCheckoutResponse;

}

Stacktrace

 com.tmpl.commerce.payment.exceptions.PayPalException: Unsupported record version Unknown-0.0

at com.tmpl.commerce.payment.paypal.request.PayPalECPaymentRequestHandler.callPayPalSetExpressCheckOut(PayPalECPaymentRequestHandler.java:457)

at com.tmpl.commerce.payment.paypal.request.PayPalECPaymentRequestHandler.setExpressCheckOut(PayPalECPaymentRequestHandler.java:107)

at com.tmpl.commerce.payment.paypal.processor.PayPalECService.setExpressCheckOut(PayPalECService.java:84)

at com.tmpl.commerce.payment.paypal.commands.PayPalECRequestCmdImpl.performExecute(PayPalECRequestCmdImpl.java:174)

at com.ibm.commerce.command.ECCommandTarget.executeCommand(ECCommandTarget.java:157)

at com.ibm.ws.cache.command.CommandCache.executeCommand(CommandCache.java:332)

at com.ibm.websphere.command.CacheableCommandImpl.execute(CacheableCommandImpl.java:166)

at com.ibm.commerce.command.AbstractECTargetableCommand.execute(AbstractECTargetableCommand.java:236)

at com.ibm.commerce.component.BaseComponentImpl.executeCommand(BaseComponentImpl.java:202)

at com.ibm.commerce.component.WebAdapterComponentImpl.executeCommand(WebAdapterComponentImpl.java:46)

at com.ibm.commerce.component.objimpl.WebAdapterServiceBeanBase.executeCommand(WebAdapterServiceBeanBase.java:58)

at com.ibm.commerce.component.objects.EJSLocalStatelessWebAdapterService_ce749a4a.executeCommand(EJSLocalStatelessWebAdapterService_ce749a4a.java:31)

at com.ibm.commerce.component.objects.WebAdapterServiceAccessBean.executeCommand(WebAdapterServiceAccessBean.java:160)

at com.ibm.commerce.webcontroller.WebControllerHelper.executeCommand(WebControllerHelper.java:2778)

at com.ibm.commerce.struts.AjaxAction.invokeService(AjaxAction.java:501)

at com.ibm.commerce.struts.AjaxAction.executeAction(AjaxAction.java:312)

at com.ibm.commerce.struts.AjaxAction.execute(AjaxAction.java:125)

at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:431)

at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:236)

at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1196)

at com.ibm.commerce.struts.ECActionServlet.processRequest(ECActionServlet.java:231)

at com.ibm.commerce.struts.ECActionServlet.doPost(ECActionServlet.java:186)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:738)

at com.ibm.commerce.struts.ECActionServlet.service(ECActionServlet.java:739)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:831)

at com.ibm.ws.cache.servlet.ServletWrapper.serviceProxied(ServletWrapper.java:307)

at com.ibm.ws.cache.servlet.CacheHook.handleFragment(CacheHook.java:576)

at com.ibm.ws.cache.servlet.CacheHook.handleServlet(CacheHook.java:250)

at com.ibm.ws.cache.servlet.ServletWrapper.service(ServletWrapper.java:259)

at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1667)

at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1602)

at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:149)

at com.ibm.commerce.campaigns.filter.CampaignsFilter.doFilter(CampaignsFilter.java:148)

at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:190)

at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:125)

at com.ibm.commerce.likeminds.filter.LikeMindsFilter.doFilter(LikeMindsFilter.java:183)

at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:190)

at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:125)

at com.ibm.commerce.dynacache.filter.CacheFilter$1.run(CacheFilter.java:390)

at com.ibm.commerce.dynacache.filter.CacheFilter.doFilter(CacheFilter.java:553)

at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:190)

at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:125)

at com.ibm.commerce.webcontroller.RuntimeServletFilter.doFilterAction(RuntimeServletFilter.java:736)

at com.ibm.commerce.webcontroller.RuntimeServletFilter.access$0(RuntimeServletFilter.java:523)

at com.ibm.commerce.webcontroller.RuntimeServletFilter$1.run(RuntimeServletFilter.java:433)

at com.ibm.commerce.webcontroller.RuntimeServletFilter.doFilter(RuntimeServletFilter.java:458)

at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:190)

at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:125)

at com.ibm.ws.webcontainer.filter.WebAppFilterChain._doFilter(WebAppFilterChain.java:80)

at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:908)

at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:939)

at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:507)

at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:181)

at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:91)

at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:878)

at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1592)

at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:191)

at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:453)

at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:515)

at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:306)

at com.ibm.ws.http.channel.inbound.impl.HttpICLReadCallback.complete(HttpICLReadCallback.java:84)

at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:175)

at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)

at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)

at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)

at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)

at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)

at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)

at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1656)

Caused by: javax.net.ssl.SSLException: Unsupported record version Unknown-0.0

at com.ibm.jsse2.b.b(b.java:102)

at com.ibm.jsse2.b.a(b.java:212)

at com.ibm.jsse2.SSLSocketImpl.a(SSLSocketImpl.java:814)

at com.ibm.jsse2.SSLSocketImpl.h(SSLSocketImpl.java:704)

at com.ibm.jsse2.SSLSocketImpl.a(SSLSocketImpl.java:12)

at com.ibm.jsse2.SSLSocketImpl.startHandshake(SSLSocketImpl.java:498)

at com.ibm.net.ssl.www2.protocol.https.c.afterConnect(c.java:59)

at com.ibm.net.ssl.www2.protocol.https.d.connect(d.java:31)

at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1195)

at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:379)

at com.ibm.net.ssl.www2.protocol.https.b.getResponseCode(b.java:91)

at com.paypal.core.HttpConnection.execute(HttpConnection.java:93)

at com.paypal.core.APIService.makeRequestUsing(APIService.java:176)

at com.paypal.core.BaseService.call(BaseService.java:258)

at urn.ebay.api.PayPalAPI.PayPalAPIInterfaceServiceService.setExpressCheckout(PayPalAPIInterfaceServiceService.java:2196)

at urn.ebay.api.PayPalAPI.PayPalAPIInterfaceServiceService.setExpressCheckout(PayPalAPIInterfaceServiceService.java:2148)

at com.tmpl.commerce.payment.paypal.request.PayPalECPaymentRequestHandler.callPayPalSetExpressCheckOut(PayPalECPaymentRequestHandler.java:455)

... 68 more

WebSphere Commerce static crawler configuration tips

A simple way for for executing crawler.sh is to use the syntax below:

crawler.sh -cfg /usr/WebSphere/AppServer70/profiles/search/solr/home/droidConfig.xml -instance <instancename> 

Which will do the following:

• Crawl your static content starting from home URL as defined by location in droidConfig.xml
• Invoke a delta index by pushing documents directly to index if autoindex is enabled (again as defined in droidConfig.xml

This works perfectly but it has a major drawback. If your removing existing static content html files, they will not be cleaned up from index and you will end up with search results which are not valid anymore and customer will get the famous 404 if tried to click on any one of them.

In such case, it might make more sense to use the steps below:

crawler.sh -cfg /usr/WebSphere/AppServer70/profiles/search/solr/home/droidConfig.xml -instance <instancename> -dbuser <dbuser> -dbuserpwd <password> -dbhost <db2_hostname> -dbname <databasename> -dbport 50000 -dbtype db2

/di-buildindex.sh -instance <wcs_instance> -masterCatalogId <catalogId> -indexSubType WebContent -dbuser <dbuser> -dbuserpwd <password> -fullbuild true -statusInterval <interval> -localename <lang> -force true -webcontentDelete true

di-buildindex.sh -instance <wcs_instance> -masterCatalogId <instancename> -indexSubType WebContent -dbuser <dbuser> -dbuserpwd <password> -fullbuild true -statusInterval <interval> -localename <lang>-force true

What will happen in such case ?

• The first command crawler.sh if configured with databse configuration paramters, it will attempt to connect to the database and update SRCHCONFEXT table where indexsubtype equals WebContent record and set the location of the newly crawled files (in column config)
• Running di-buildindex.sh with  -webcontentDelete set to true will force cleaning up index content
• The running di-buildindex.sh again without this option (it is false by default) will build a clean index.

Remarks:

• The process is expected to happen on staging and index is propagated to production so cleaning up index shouldn't affect your production data
• You can mix between full index cleanup and delta index updates by setting autoIndex to true. In such case, you will have more freedom in setting a different schedule for WebContent index update.

Some insights into WebSphere Dynacache

Let's assume the following scenario:

• You have two JSPs, a parent JSP called parent.jsp and a child which it dynamically includes a child JSP called child.jsp.
• Child JSP requires one parameter to function properly and you need this parameter to be passed with user request to generate some dynamic content that doesn't depend on cache. Let's assume this parameter is called action1.
• You want to cache the parent but you don't wish to cache the child/
• The cachespec.xml is defined something as shown below.

<cache-entry><class>servlet</class><name>/StorefrontAssetStore/parent.jsp</name><property name="do-not-consume">true</property><property name="save-attributes">false</property><property name="consume-subfragments">true</property><cache-id><component id="someparam" type="parameter"><required>true</required></component></cache-id></cache-entry>
<cache-entry><class>servlet</class><name>/StorefrontAssetStore/child.jsp</name><property name="do-not-consume">true</property><property name="do-not-cache">true</property>
<property name="save-attributes">false</property><property name="consume-subfragments">true</property><cache-id><component id="action1" type="parameter"><required>true</required></component></cache-id></cache-entry>

Now, when you are developing the JSPs, you will most probably dynamically include the child from the parent as follows:

<jsp:include page="child.jsp" flush="true"><jsp:param name="action1" value="${param.action1}"/>
</jsp:include>

So question is, will this work as intended ? parent.jsp cached once and child.jsp is dynamically invoked with every request with a fresh value for action1 ?

Unfortunately, this will not happen and despite the child JSP is not cached, it will behave as if it is cached.

Why ?

Things will be more clear, if you opened WebSphere Cache monitor and checked how parent JSP is cached in first place, you will notice something as the below in its cache content:

[include: /StorefrontAssetStore/child.jsp?action1=somevalue]

Which means, whenever parent JSP is accessed from cache, it will dynamically call child JSP but within all calls, it is always passing one cached value which is <somevalue>

So how to get over this ?

Simply, your parent JSP needs to look something as below:

<jsp:include page="child.jsp" flush="true"/>

and within child JSP make sure you always access action1 using param.action1. This will ensure that child JSP is completely free from cache content and able to pickup parameters properly each time from the request stream and do something different each time.

WebSphere Commerce access policy not working for a command

Problem

You developed a new custom WebSphere Commerce command, you defined access policy for it and you are sure it is correct but for some reason, your command executes without confirming to your access policy.

What might be the root cause ?

Fix

Access policy are defined for interfaces and not for the actual command. So if your command is not picking up the access policy, then you didn't define the relation between your command and its interface properly. Example:

Let's assume, your new custom command is called ProcessRefundCmdImpl and its interface is ProcessRefundCmd.  Let's assume as well, you defined NAME & defaultCommandClassName properly in your interface.

If your ProcessRefundCmdImpl is defined as shown below, two things will happen:

• The command will execute as you expect and implementation class ProcessRefundCmdImpl will be picked up from CMDREG or defaultCommandClassName (as you expect)
• Once you apply access policy, command will not work anymore.

public class ProcessRefundCmdImpl extends ControllerCommandImpl

Now, once you change your code to be as shown below, your access policy will start working as expected.

public class ProcessRefundCmdImpl extends ControllerCommandImpl implements ProcessRefundCmd

It is one of those small mistakes that can take hours or even days to find out. It would have been much better, if the command never executes if it doesn't implement the proper interface in first place, but it is there for developers to have fun with.

Why would you need to define access policy WebSphere Commerce data beans ?

We -WebSphere Commerce developers- don't pay much attention to security data beans, we put all attention to WebSphere Commerce commands and views because they are supposed to be entry point to any store and they will enforce security, while data beans are only accessible from a JSP which is the last step in the cycle and you can't get it to it unless you go through a command or view first.

That was my belief till a network administrator send me the curl command below which he used to simulate some request parameters, send it over to one of our JSPs (CustomerSearchOutput.jsp) and without providing any credentials, he managed to extract customer sensitive data from the database.

curl -k --data "storeId=10151&qlist=1n2n3n4n5n6n7n8n9n10n11n12n11n12n&useraction=searchclicked&sortclicked=defaultsortclicked&searchOption1=findbylastname&searchOption2=&searchTerm1=Duncan&searchTerm2=&csrLogonId=Rumi&sorttype=desc" https://wcs_testserver/webapp/wcs/csr/servlet/CustomerSearchOutput.jsp ?

As you might imagine, we worked till late the next couple of day to get it sorted out. For better security, you need to make sure no one can access your databeans and use them as a gateway to extract data from your database.

For details on how to get it done, please check my earlier blog Implementing access control in WebSphere commerce data beans

Implementing access control for WebSphere commerce data beans

Introduction

Tried recently to secure some data beans, followed WebSphere Commerce infocenter and didn't manage to make it work ? banged your head against a brick wall for hours and still didn't work ? Welcome in club, you are not alone.

Assumptions

Let's assume we have the following details to start with:

• Data bean to secure is com.mycompany.beans.SampleDataBean
• The bean is declared as follows

public class SampleDataBean extends SmartDataBeanImpl

• The bean is intended for administrators only and should be available for members of group CSRSystemAdmins

Steps

• Change your bean to implement both Protectable & Delegator as shown below

public class SampleDataBean extends BaseSearchBean implements Protectable, Delegator

• Implement getDelegate as shown below. This method is the most important for security to work and most properly, things didn't work for you because you didn't implement getDelegate and left it to just return null, which for Commerce (DataBeanManager to be more specific) means security will be ignored (even if you defined access policies). You need to return a reference to any object that implements Protectable and since my sample implements it, I will return a reference to it (which is this). 

@Override
public Protectable getDelegate() throws Exception {
String METHOD_NAME = "getDelegate()";
LOGGER.entering(CLASS_NAME, METHOD_NAME);
return this;
}

• You can leave fulfills without a particular implementation as shown below. In the sample scenario explained here, I enabled tracing and am sure fulfills was never called during bean initialization which means it will not impact how am expecting bean to work. On other hand, if you are doing more changes and you will define a relation between policy and resource, then you need to have a proper implementation.

@Override
public boolean fulfills(Long member, String relationship) throws Exception, java.rmi.RemoteException {
String METHOD_NAME = "fulfills(Long member, String relationship)";
LOGGER.entering(CLASS_NAME, METHOD_NAME);
return false;
}

• Implement getOwner as shown below. Again it doesn't make a big difference, I just decided that the owner is whoever executing it. You can change it to whatever you see more appropriate.

@Override public Long getOwner() throws Exception, java.rmi.RemoteException { String METHOD_NAME = "getOwner()"; LOGGER.entering(CLASS_NAME, METHOD_NAME); return getCommandContext().getUserId(); }

• Now for access policies, you need to amend your access policies to have something as shown below. I marked in red, the pieces you need to focus on while adding the new access policy. Remember the order of the sections below is important.

<Action Name="DisplayDatabean" CommandName="Display"/>
<ActionGroup Name="DisplayDatabeanActionGroup" OwnerID="RootOrganization"><ActionGroupAction Name="DisplayDatabean"/></ActionGroup>   <ResourceCategory Name="SampleDataBean" ResourceBeanClass="com.mycompany.beans.SampleDataBean"><ResourceAction Name="DisplayDatabean"/></ResourceCategory> <ResourceGroup Name="DataBeansForAdmins" OwnerID="RootOrganization"><ResourceGroupResource Name="SampleDataBean"/></ResourceGroup> <Policy Name="DisplayDataBeansForAdmins"OwnerID="RootOrganization"UserGroup="CSRSystemAdmins"ActionGroupName="DisplayDatabeanActionGroup"ResourceGroupName="DataBeansForAdmins"PolicyType="groupableStandard"></Policy>

• Load the new policy using acpload
• Make sure you don't have any errors in acpload.log
• Open Organization administration console and review the entries are properly loaded
• Enable access control tracing as explained in an older blog Debugging WebSphere Commerce
• Restart server
• Start testing and access your JSPs as usual. You need to access the JSPs with a guest user, a logged-in user who is not a member of group CSRSystemAdmins
• If security check is successful and user ganted access, you should something similar to the trace output below

com.ibm.commerce.accesscontrol.policymanager.PolicyManagerImpl isExecutionAllowed isAllowed? User=344003; Action=Display; Resource=com.mycompany.beans.SampleDataBean; Owner=344003; Resource Ancestor Orgs=-2001; Resource Applicable Orgs=-2001

com.ibm.commerce.accesscontrol.policymanager.PolicyManagerImpl isExecutionAllowed Found PolicyName: DisplayDataBeansForAdmin; PolicyType: 2; PolicyOwner: -2001

com.ibm.commerce.accesscontrol.policymanager.PolicyManagerImpl getPolicyApplicableOrgs Entry

com.ibm.commerce.accesscontrol.policymanager.PolicyManagerImpl getPolicyApplicableOrgs Policy Applicable Orgs=-2001

com.ibm.commerce.accesscontrol.policymanager.PolicyManagerImpl getPolicyApplicableOrgs Exit

com.ibm.commerce.accesscontrol.policymanager.PolicyManagerImpl evaluatePolicy Evaluating PolicyName: DisplayDataBeansForAdmin

com.ibm.commerce.accesscontrol.policymanager.PolicyManagerImpl evaluatePolicy CommandLevelCheck: false; StoreId: 10151

com.ibm.commerce.accesscontrol.policymanager.PolicyManagerImpl evaluatePolicy No Relationship or RelationshipGroup to check

com.ibm.commerce.accesscontrol.policymanager.PolicyManagerImpl isExecutionAllowed PASSED? =true

com.ibm.commerce.accesscontrol.policymanager.PolicyManagerImpl isExecutionAllowed Exit

com.ibm.commerce.accesscontrol.policymanager.PolicyManagerImpl isAllowed PASSED? =true

com.ibm.commerce.accesscontrol.policymanager.PolicyManagerImpl isAllowed Exit

• If on other hand, user denied access, you will see something as shown below

com.ibm.commerce.accesscontrol.policymanager.PolicyManagerImpl isExecutionAllowed isAllowed? User=333003; Action=Display; Resource=com.mycompany.beans.SampleDataBean; Owner=333003; Resource Ancestor Orgs=-2000,-2001; Resource Applicable Orgs=-2000

com.ibm.commerce.accesscontrol.policymanager.PolicyManagerImpl isExecutionAllowed Found PolicyName: DisplayDataBeansForAdmin; PolicyType: 2; PolicyOwner: -2001

com.ibm.commerce.accesscontrol.policymanager.PolicyManagerImpl getPolicyApplicableOrgs Entry

com.ibm.commerce.accesscontrol.policymanager.PolicyManagerImpl getPolicyApplicableOrgs Policy Applicable Orgs=-2000

com.ibm.commerce.accesscontrol.policymanager.PolicyManagerImpl getPolicyApplicableOrgs Exit

com.ibm.commerce.accesscontrol.policymanager.PolicyManagerImpl evaluatePolicy Evaluating PolicyName: DisplayDataBeansForAdmin

com.ibm.commerce.accesscontrol.policymanager.PolicyManagerImpl evaluatePolicy CommandLevelCheck: false; StoreId: 10151

com.ibm.commerce.accesscontrol.policymanager.PolicyManagerImpl evaluatePolicy Normal UserGroup does not match

com.ibm.commerce.accesscontrol.policymanager.PolicyManagerImpl isExecutionAllowed PASSED? =false

com.ibm.commerce.accesscontrol.policymanager.PolicyManagerImpl isExecutionAllowed Exit

com.ibm.commerce.accesscontrol.policymanager.PolicyManagerImpl isAllowed PASSED? =false

com.ibm.commerce.accesscontrol.policymanager.PolicyManagerImpl isAllowed Exit

Final Recommendation

• It is better you always test your security by enabling trace to make sure that a user is denied or granted access for the right expected reasons and according to your expectations for how the policy should work. Sometimes you can get the right results but for wrong reasons, for example, you have another policy (that you created for sake of testing long time ago) in your local environment that happens to do what you expect. In such cases, once your code is deployed to another environment, it will stop working and it will take you a long time to figure out why.

References

Implementing access control in data beans

Protecting Data Beans

Define WebSphere Commerce SEO pattern to match a filename

If you are trying to define a token for filenames which accepts lower or upper case characters as well as numbers & hyphen, here is the right way to define it:

<seourl:tokenValue value="[[a-zA-Z0-9\- ]*]"/>

JSP code fragment for WebSphere Commerce to check if user is logged in or not

The sample code fragment below will set a variable loggedIn to true or false after based on current user register type.

<%
CommandContext commandContext= (CommandContext)request.getAttribute(ECConstants.EC_COMMANDCONTEXT );

Long userId=commandContext.getUserId();

UserDataBean userBean=new UserDataBean();
userBean.setDataBeanKeyMemberId(userId.toString());
userBean.populate();

String userType=userBean.getRegisterType();

if(userType.equalsIgnoreCase(ECConstants.EC_GENERIC_USER_TYPE)){
request.setAttribute("loggedIn",false);
}else{
request.setAttribute("loggedIn",true);
}
%>

<c:if test="${loggedIn}">
    <!-- Do something  -->
</c:if>

Few days later, I discovered it is much easier this way :)

<%
String userType=(String)request.getAttribute("DC_userType");
if(userType.equalsIgnoreCase(ECConstants.EC_GENERIC_USER_TYPE)){
request.setAttribute("loggedIn",false);
}else{
request.setAttribute("loggedIn",true);
}

%>
<c:if test="${loggedIn}">
    <!-- Do something  -->
</c:if>

Access policy looks fine but some entries are not loaded properly

Problem

You have created a custom access policy file e.g. CustomACPolicies.xml, you load it using acpload, you don't get any error but still when you check the entries in database or using Organization admin console, you realize some of the entries are missing and were not loaded properly.

Fix

Make sure to review the order of the elements in the file, because the order of different section makes a whole big difference, for example a ResourceCategory or ResourceGroup can't come after a policy. Here is the order you need to follow within the XML file:

• Action
• ActionGroup
• ResourceCategory
• ResourceGroup
• Policy
• PolicyGroup

Example

The following extract from an access policy when executes, it creates the policy as well as resource group as expected, but the resource group is empty and it doesn't contain the resource com.beans,ResultDataBean. 

The reason is, the ResourceCategory should come before the ResourceGroup.

<ResourceGroup Name="PenguinDataBeansForCustomerServiceSupervisors" OwnerID="RootOrganization">
<ResourceGroupResource Name="com.beans.ResultDataBean"/>
</ResourceGroup> 

<ResourceCategory Name="com.beans.ResultDataBean" ResourceBeanClass="com.beans.ResultDataBean">
<ResourceAction Name="DisplayDatabean"/>
</ResourceCategory> 

<Policy Name="DisplayPenguinDataBeanForCustomerServiceRepresentatives"
OwnerID="RootOrganization"
UserGroup="CustomerServiceRepresentatives"
ActionGroupName="DisplayDatabeanActionGroup"
ResourceGroupName="PenguinDataBeansForCustomerServiceRepresentatives"
RelationName="owner"
PolicyType="groupableStandard">
</Policy>

Access WebSphere CommandContext from scriptlet

The code snippet below is used to access CommandContext from a JSP scriptlet

<%
CommandContext commandContext= (CommandContext)request.getAttribute(ECConstants.EC_COMMANDCONTEXT );
 %> 

Tip for WebSphere Commerce SEO

While customizing SEO and doing some testing on your changes, make sure you don't copy existing files and make a backup of them by copying them in the same folder as shown below.

After restarting server, WebSphere Commerce will load all of those files including the one called "Copy of SEOURLPatterns-ext.xml" and this will definitely conflict on the new changes you are making and can take long hours of debug time till you realize the mistake.

SEO Tip

Configure access to static documents on HTTP server

While customizing WebSphere Commerce stores, if you need to provide links to static content deployed on your web server, lets' say for example you need to have a link for size guidelines as shown below:

http://<hostname>/infodocs/size_guidelines.html

To access this file from Commerce, you will need to use a link which might look as follows:

http://<hostname>/info/size_guidelines

You will need to make the following changes to httpd.conf:

• For VirtualHost 80 and 443. add the following entry

Alias   /infodocs     "/opt/webserver/assets/infodocs"

• For rewrite rules, add the following line:

RewriteCond %{REQUEST_URI} !^/info.*$ 

Where infodocs is a folder you need to create your HTTP server (or any other you see appropriate).


General considerations:

• The URL used to access the html file from HTTP server directly can't be identical to the one used by Commerce and that is why I used two different links /infodocs and /info
• For Commerce SEO to work properly you can't use a file extension and that is why I removed .html from the URL 
• The Commerce needs to be customized to properly display the content of a static html file. For example, let's assume your SEO will use view CustomStatiContent to handle /info/ and let's assume this view will be rendered using CustomStaticContent.jsp, in such case, this jsp should have some logic that looks as below:

<c:catch var="e">
<c:import var="pageContents" url="${filePath}" charEncoding="UTF-8"/>
</c:catch>
<c:if test="${!empty pageContents && empty e }">
<c:out value="${pageContents}" escapeXml="false"/>
</c:if>

Where filePath variable will hold the following URL (given our example above) http://<hostname>/infodocs/size_guidelines.html

Configure Websphere Commerce crawler to crawl static html files

High Level description

• The implementation below is based on OOB configuration but it changes the default crawling page to another static content page (staticcontentindex.html) which customer can update manually and upload to server to define a list of static files for crawler to manage.
• The configuration is not required on production. Crawling and indexing will happen on staging and index is propagated to production (as usual)
• It is assumed SEO is configured on WebSphere Commerce so static content html are accessed using /info/ e.g. http://<hostname>/info/contactus
• It is assumed that web server is configured so all static html files are located under /opt/webserver/assets/infodocs. For example, if you try to access file http://hostname/info/contactus, it is expected to have the following file available on http server/opt/webserver/assets/infodocs/contactus.html
• The customization done to Commerce to display static html files are not included as part of the article.

For case where Solr is local to Commerce box (e.g. UAT-Staging)

• Update droidConfig.xml (environment specific) to change following entries (file attached as reference)
• hostname
• storePathDirecttory. I don’t like the default storePathDirectory as it will copy crawled data in subfolders under it. So feel free to define another location that makes more sense if you wish to.
• Add a new var called solrHostname (in most environment the value will be identical to hostname)
• Add a new var called solrPort
• Change location to the following value http://${hostname}/info/staticcontentindex
• Change relative path to empty string. This will make sure all crawler links will have a full URL and will avoid awkward issues that might popup with relative ones. Just make sure the hostname used is one that customers can access externally. More on this might come in future posts. 
• Set autoIndex enable=”true” and set URL as shown below.

http://${solrHostname}:${solrPort}/solr/MC_${catalogId}_CatalogEntry_Unstructured_${localename}/webdataimport?command=full-import&amp;storeId=${storeId}&amp;basePath=

• Use attached filters.txt instead of default one which allows crawling of static html files and ignore all others. Because your static files might include your megamenu, it simply means the crawler will attempt to crawl the entire store. You need to make sure the rules defined in filters.txt will prohibit this from happening. In attached sample file, I included two stop rules which are -.*(search).* & -.*(category).* because my SEO uses both of them and I need to make sure they are filtered out. As a consequence you need to avoid to use the same SEO pattern as folder names for your static content.
• Copy staticcontentindex.html to /opt/webserver/assets/infodocs. It is just provided as a sample for testing purposes, so don’t replace it in case it already exist on server.

For case where Solr server is remote to Commerce server

• You need to follow technote provided by IBM http://www-01.ibm.com/support/docview.wss?uid=swg21612155
• Apply steps above on the remote Solr server

Database

• In table SRCHCONFEXT, for INDEXSUBTYPE WebContent, make sure CONFIG column has something similar to what is below where storePathDirectory is as defined in droidConfig.xml

BasePath=<storePathDirectory>\StaticContent\en_US\,SearchServerPort=<solrPort>,SearchServerName=<solrHostname>,StoreId=<storeId>

Crontab

• We need to add crawler.sh to crontab and define an appropriate schedule for it

Testing

• In case Solr/Commerce are on same box, change directory to commerce server bin directory & execute command shown below. In case Solr is on a separate box, you need to run crawler.sh from the Solr box instead of Commerce.

./crawler.sh -cfg /usr/WebSphere/AppServer70/profiles/search/solr/home/droidConfig.xml -instance <instancename>

• Verify that crawler.sh completed successfully without errors
• Verify indexing status using the following link

http://<solrHostname>:<solrPort>/solr/MC_10001_CatalogEntry_Unstructured_en_US/webdataimport?command=status

• Run index update for WebContent as shown below and make sure it completes successfully without errors

/usr/WebSphere/CommerceServer70/bin/di-buildindex.sh -instance <instanceName> -masterCatalogId <masterCatalogId> -indexSubType WebContent -dbuser <user> -dbuserpwd <password> -fullbuild true -statusInterval 10000 -localename en_US

More considerations

Please check my newest post regarding crawler configuration tips you need to take into consideration.

References

droidConfid.xml
filters.txt
staticontentindex.html

WebSphere commerce wcf:url returns an empty string

Problem

In a simple JSP, the code fragment below was always returning var GenericErrorViewPage as an empty string.

<wcf:url var="GenericErrorViewPage" value="GenericApplicationError">

     <wcf:param name="page" value="FileNotFound"/>

 </wcf:url>

Fix

Make sure the JSP includes EnvironmentSetup.jspf and not just JSTLEnvironmentSetup.jsp, if environment is not properly initialized, wcf:url doesn't work properly

Add hostname as a cache-id in Dynacache

Sample below shows how to do it. It can be useful if the same store is accessed using different domain names, one for external customers and another for internal support when they place orders on behalf of a customer. In such case, you might need to create separate cache entries to avoid problems that might raise up because of hard coded links in any of the JSPs or any other logic that might break if you store one cache entry for both.

<component id="host" type ="header">
     <required>false</required>
</component>