WebSphere Commerce static crawler configuration tips

A simple way for for executing crawler.sh is to use the syntax below:

crawler.sh -cfg /usr/WebSphere/AppServer70/profiles/search/solr/home/droidConfig.xml -instance <instancename> 

Which will do the following:

• Crawl your static content starting from home URL as defined by location in droidConfig.xml
• Invoke a delta index by pushing documents directly to index if autoindex is enabled (again as defined in droidConfig.xml

This works perfectly but it has a major drawback. If your removing existing static content html files, they will not be cleaned up from index and you will end up with search results which are not valid anymore and customer will get the famous 404 if tried to click on any one of them.

In such case, it might make more sense to use the steps below:

crawler.sh -cfg /usr/WebSphere/AppServer70/profiles/search/solr/home/droidConfig.xml -instance <instancename> -dbuser <dbuser> -dbuserpwd <password> -dbhost <db2_hostname> -dbname <databasename> -dbport 50000 -dbtype db2

/di-buildindex.sh -instance <wcs_instance> -masterCatalogId <catalogId> -indexSubType WebContent -dbuser <dbuser> -dbuserpwd <password> -fullbuild true -statusInterval <interval> -localename <lang> -force true -webcontentDelete true

di-buildindex.sh -instance <wcs_instance> -masterCatalogId <instancename> -indexSubType WebContent -dbuser <dbuser> -dbuserpwd <password> -fullbuild true -statusInterval <interval> -localename <lang>-force true

What will happen in such case ?

• The first command crawler.sh if configured with databse configuration paramters, it will attempt to connect to the database and update SRCHCONFEXT table where indexsubtype equals WebContent record and set the location of the newly crawled files (in column config)
• Running di-buildindex.sh with  -webcontentDelete set to true will force cleaning up index content
• The running di-buildindex.sh again without this option (it is false by default) will build a clean index.

Remarks:

• The process is expected to happen on staging and index is propagated to production so cleaning up index shouldn't affect your production data
• You can mix between full index cleanup and delta index updates by setting autoIndex to true. In such case, you will have more freedom in setting a different schedule for WebContent index update.

Configure Websphere Commerce crawler to crawl static html files

High Level description

• The implementation below is based on OOB configuration but it changes the default crawling page to another static content page (staticcontentindex.html) which customer can update manually and upload to server to define a list of static files for crawler to manage.
• The configuration is not required on production. Crawling and indexing will happen on staging and index is propagated to production (as usual)
• It is assumed SEO is configured on WebSphere Commerce so static content html are accessed using /info/ e.g. http://<hostname>/info/contactus
• It is assumed that web server is configured so all static html files are located under /opt/webserver/assets/infodocs. For example, if you try to access file http://hostname/info/contactus, it is expected to have the following file available on http server/opt/webserver/assets/infodocs/contactus.html
• The customization done to Commerce to display static html files are not included as part of the article.

For case where Solr is local to Commerce box (e.g. UAT-Staging)

• Update droidConfig.xml (environment specific) to change following entries (file attached as reference)
• hostname
• storePathDirecttory. I don’t like the default storePathDirectory as it will copy crawled data in subfolders under it. So feel free to define another location that makes more sense if you wish to.
• Add a new var called solrHostname (in most environment the value will be identical to hostname)
• Add a new var called solrPort
• Change location to the following value http://${hostname}/info/staticcontentindex
• Change relative path to empty string. This will make sure all crawler links will have a full URL and will avoid awkward issues that might popup with relative ones. Just make sure the hostname used is one that customers can access externally. More on this might come in future posts. 
• Set autoIndex enable=”true” and set URL as shown below.

http://${solrHostname}:${solrPort}/solr/MC_${catalogId}_CatalogEntry_Unstructured_${localename}/webdataimport?command=full-import&amp;storeId=${storeId}&amp;basePath=

• Use attached filters.txt instead of default one which allows crawling of static html files and ignore all others. Because your static files might include your megamenu, it simply means the crawler will attempt to crawl the entire store. You need to make sure the rules defined in filters.txt will prohibit this from happening. In attached sample file, I included two stop rules which are -.*(search).* & -.*(category).* because my SEO uses both of them and I need to make sure they are filtered out. As a consequence you need to avoid to use the same SEO pattern as folder names for your static content.
• Copy staticcontentindex.html to /opt/webserver/assets/infodocs. It is just provided as a sample for testing purposes, so don’t replace it in case it already exist on server.

For case where Solr server is remote to Commerce server

• You need to follow technote provided by IBM http://www-01.ibm.com/support/docview.wss?uid=swg21612155
• Apply steps above on the remote Solr server

Database

• In table SRCHCONFEXT, for INDEXSUBTYPE WebContent, make sure CONFIG column has something similar to what is below where storePathDirectory is as defined in droidConfig.xml

BasePath=<storePathDirectory>\StaticContent\en_US\,SearchServerPort=<solrPort>,SearchServerName=<solrHostname>,StoreId=<storeId>

Crontab

• We need to add crawler.sh to crontab and define an appropriate schedule for it

Testing

• In case Solr/Commerce are on same box, change directory to commerce server bin directory & execute command shown below. In case Solr is on a separate box, you need to run crawler.sh from the Solr box instead of Commerce.

./crawler.sh -cfg /usr/WebSphere/AppServer70/profiles/search/solr/home/droidConfig.xml -instance <instancename>

• Verify that crawler.sh completed successfully without errors
• Verify indexing status using the following link

http://<solrHostname>:<solrPort>/solr/MC_10001_CatalogEntry_Unstructured_en_US/webdataimport?command=status

• Run index update for WebContent as shown below and make sure it completes successfully without errors

/usr/WebSphere/CommerceServer70/bin/di-buildindex.sh -instance <instanceName> -masterCatalogId <masterCatalogId> -indexSubType WebContent -dbuser <user> -dbuserpwd <password> -fullbuild true -statusInterval 10000 -localename en_US

More considerations

Please check my newest post regarding crawler configuration tips you need to take into consideration.

References

droidConfid.xml
filters.txt
staticontentindex.html