Setting up an application based on Discover Electronics

Setting up LifeStyle Endeca Application and LifeStyle Reference Application

We can create an Endeca application based on discover-data deployment deployment template by following the steps provided in Getting Started Guide (http://docs.oracle.com/cd/E38680_01/Common.311/pdf/GettingStarted.pdf - Deploying a Reference Application) and then view the sample pages using discover-electronics reference application (http:localhost:8006/discover)

As beginners, we tend to play around the reference application by changing the pipeline, creating a new cartridge templates and then seeing the changes in the reference application. To do this we should have a reference application that is specific to our application so that we can make all the changes as we need without effecting the base reference application.

I have followed the below steps to create a new Endeca application and corresponding new reference application to make the changes.

Check that the Endeca HTTP Service and Endeca Tools Service are up and running

1. Create a new Endeca application named lifestyle based on discover-data deployment template

a) Navigate to cd C:\Endeca\ToolsAndFrameworks\3.1.1\deployment_template\bin and execute

deploy.bat --app ..\..\reference\discover-data\deploy.xml

Provide the below parameter values

• Application Name: lifestyle
• Deployment Directory : C:\Endeca\apps
• EAC Port : 8888
• Workbench port : 8006
• Live Dgraph port : 18000
• Authoring Dgraph port : 18002
• Log Server Port : 18010  

Ensure that the application is successfully deployed


b) Navigate to the control directory of the newly created application and execute the initialize_services.bat to provision the application to EAC
C:\Endeca\apps\lifestyle\control>initialize_services.bat

c) Execute the load_baseline_test_data to load the test data
C:\Endeca\apps\lifestyle\control>load_baseline_test_data.bat

d) Run the baselineupdate to perform full indexing
C:\Endeca\apps\lifestyle\control>baseline_update.bat

e) Execute promote content to promote the content to work bench
C:\Endeca\apps\lifestyle\control>promote_content

2. Navigate to C:\Endeca\ToolsAndFrameworks\3.1.1\reference directory, copy of the discover-electronics reference application folder, paste it in the same directory and name it as lifestyle

3. Modify the following properties in assembler.properties file of lifestyle reference application

C:\Endeca\ToolsAndFrameworks\3.1.1\reference\lifestyle\WEB-INF\assembler.properties

• workbench.app.name=lifestyle
• workbench.host=localhost
• workbench.port=8006
• workbench.publishing.serverPort=8007
• mdex.host=localhost
• mdex.port=18000
• logserver.host=localhost
• logserver.port=18010

4. Navigate to ENDECA_TOOLS_CONF sub directory C:\Endeca\ToolsAndFrameworks\3.1.1\server\workspace\conf\Standalone\localhost, copy the discover.xml file, paste it in the same directory and rename it to lifestyle.xml and make the below changes

• path="/lifestyle"
• docbase="${catalina.base}/../../reference/lifestyle"

5. Restart the EndecaToolsService

6. Verify that lifestyle application is present in the Endeca Workbench with the sample cartridge templates and sample pages.

http://localhost:8006/admin > login and select the lifestyle Endeca Application

7. Verify that the lifestyle  reference application is available with sample page views configured in the experience manager with camera test data.

http://localhost:8006/lifestyle to view the lifestyle reference web application

We have successfully created lifestyle Endeca application using discover-data deployment template and successfully have corresponding lifestyle web application running. We can now create a new pages in experience manager, make corresponding view changes in the reference application and see the changes.

Stepping into Oracle Endeca Commerce 3.1.1

Since Oracle has released its latest version of Endeca Commerce 3.1.1 very recently, I decided to migrate my current Endeca Installations to that. Please find below the steps followed to achieve the same.

Release Document of Endeca Commerce 3.1.1 states that only the following packages have been updated

• Tools and Frameworks 3.1.1
• Mdex Engine 6.4.0
• Content Acquistion System 3.1.1

Since my current installations are from Oracle Endeca Commerce 3.1.0 and Platform Services has not been updated in this release, I decided to upgrade only Tools and Frameworks and Mdex Engine. Since I am not going to use ATG 10.1.2 to integrate with Endeca, I am not installing CAS and will not be using discover-data-catalog-integration deployment template for creating the Endeca Application.

Since I need to upgrade my Mdex Engine from 6.3.0 to 6.4.0 and Tools and Frameworks from 3.1.0 to 3.1.1, I followed the below steps.

1. Identify the current Endeca applications that have been installed and remove them.

a) Navigate to PlatformServices Installation Directory/bin (Eg: C:\Endeca\PlatformServices\6.1.3\bin) and execute the command eaccmd.bat list-apps to get the list of Endeca Applications that are provisioned to EAC

b) Since there is only application Discover , next step is to remove the application. Navigate to the control directory of the Discover application and execute the runcommand.bat with remove--app arguments

c) Remove the Instance Configurations completely, using emgr_update.bat with remove_all_settings argument

emgr_update.bat --host localhost:8006 --app_name My_app --action remove_all_settings --prefix My_prefix

d) If you are using CAS Record Store Instances to load the data and have created the Endeca application using ProductCatalogIntegration deployment template, the you need to remove the record store instances for your application

C:\Endeca\CAS\3.0.2\bin>component-manager-cmd.bat delete-component -n {record_store_name}

To get the list of recod stores you can use the below command

C:\Endeca\CAS\3.0.2\bin>component-manager-cmd.bat list-components

e) Close the command prompt (if open), navigate to the Endeca applications directory (Eg: C:\Endeca\apps) and delete the Discover folder completely.

Follow the above steps to remove all the Endeca applications completely that are listed using list-apps command described above.

2. Navigate to the Tools&Frameworks/server/bin folder (Eg: C:\Endeca\ToolsAndFrameworks\3.1.0\server\bin) folder and execute the uninstall_service.bat file

Navigate to the Endeca Installation Directory(Eg: C:\Endeca) and delete the ToolsAndFrameworks folder completely.

3. Stop the HTTP Service from the services console (You will not see the Endeca Tools Service as it is uninstalled in the above step) and uninstall the MDEX Engine from the Control Panel/Add or Remove programs.

Navigate to the Endeca Installation Directory (Eg: C:\Endeca) and delete the MDEX folder completely

4. Execute the mdex_6.4.0.692722_x86_64pc-win32.exe file and complete Mdex Installation successfully.

5. Extract the Endeca-Tools-Frameworks-with-Experience-Manager-2.1.2-V35745-01  zip file to Endeca Installation Directory and ensure that editor_sdk folder is present in C:\Endeca\ToolsAndFrameworks\3.1.1 directory (Tools and Frameworks installation directory) to confirm it  supports experience manager and not guided search.

6. Execute the install_services.bat from the ToolAndFrameworks/server/bin folder

7. Since we have upgraded the MDEX without re-installing the platform services, we need to inform it about the change in MDEX version. So we need to update the mdexRoot path in eac.properties present in the location (C:\Endeca\PlatformServices\workspace\conf\eac.properties). Failing to do this step would produce Errors when a baseline update is performed on any Endeca application that is deployed later. i.e

Error Communicating with EAC Agent while starting component. Occurred while executing line 25 of valid Bean shell script

8. The MDEX 6.4 zip file contains the Presentation API jars that are compatible with MDEX 6.4. Copy those jars to the Endeca_JSP Ref web application to avoid ENEException later.

For Eg: Copy endeca_logging.jar, endeca_navigation.jar from  C:\Endeca\presAPI_6.4.0.692722_x86_64pc-win32\PresentationAPI\6.4.0\java\lib to C:\Endeca\PlatformServices\6.1.3\tools\server\webapps\endeca_jspref\WEB-INF\lib.

Failing to do so might produce ENEExecption in the EndecaJSPRef application because of the version mismatch 

Oracle Endeca - JSP Reference Application

status>>ENE error

ENEException

com.endeca.navigation.ENEException: Navigation Engine not able to process request 'http://localhost:15000/graph?node=0&offset=0&nbins=10&iversion=620'.

9. Start the Endeca HTTP Service and Tools Service from services console.

By following the above steps I have successfully migrated from Oracle Endeca Commerce 3.1.0 to Oracle Endeca Commerce 3.1.1. Next job would be to create a sample Endeca Application and test if my installations are successful !!!

New look to Oracle ATG Commerce with Oracle Endeca Commerce

With Oracle acquiring both ATG and Endeca, it has now grouped both the products under one umbrella called ORACLE COMMERCE comprising ORACLE ATG COMMERCE and ORACLE ENDECA COMMERCE.  Oracle has already announced that they would stop their support for ATG search in next couple of years and are encouraging all the existing applications developed in ATG Search to migrate to Endeca.

As part of the products integration, Oracle has released two major versions of ATG for integration of Oracle ATG Commerce to Oracle Endeca Commerce. With the latest version of Oracle ATG Commerce 10.1.2 one can easily integrate with the latest version Oracle Endeca Commerce 3.1.1 with the help of many OOTB components provided in ATG as well as the Deployment Template provided in Endeca.

Endeca Commerce, by itself is a well proven product for Search and with the power of ATG Commerce for the core commerce functionality we can build an Enterprise e-commerce application that can give the best experience to the customers which is highly stable and scalable.

Now that the gates are open for the integration of ATG Commerce with Endeca Commerce, there are some challenges as well when we take into consideration of the existing clients who would like to integrate their ATG applications with Endeca Commerce.

1) Do I need to migrate my existing ATG application which is currently running in (7.0 or 9.0) to 10.1.2 to integrate with Endeca Commerce 3,1,1. Aren't there any steps to integrate my existing application with Endeca Commerce?

2) If my existing ATG application containing a lot of custom BCC changes as per my requirement is running well today, what are the challenges in front to migrate to ATG 10.1.2 so as to integrate with Endeca Commerce? As ATG 10.0 and above uses flex in Merchandising UI, is it advisable to migrate to 10 so as to integrate with Endeca ?

3) Though one can index the data from ATG product catalog to Endeca using the OOTB components, we still have to make use of Developer Studio for things such as creating a search interface and so on. Will we be able to do all the required stuff from ATG itself going forward?

As change is inevitable, without worrying about the challenges ahead one should start making use of Endeca Commerce capabilities with the best possible way for their current ATG solution.

Note: As I am completely new to Endeca, I am planning to pen down all the configuration steps, issues faced, migration steps and so in my upcoming blogs.

ACC Changes versus dyn/admin changes

Normally we always have a confusion on what is happening behind the scene when a change is made through dyn/admin to that of ACC

Changes made through dyn/admin are not written to the properties file so they will be lost once you restart the server. One pretty common scenario when we tend to change the component property values through dyn/admin is while diagnosing some problem as we may want to enable debug logs and any other types of output messages which can give us more information. 

When changing component property values from ACC, the changes are actually written to the corresponding component properties files and those properties file will go to the config layer with the highest precedence. And in case you want your changes to be written to a particular config layer then you can select that config layer also within ACC from the menu Tools -> Set Update Layer. By default the config layer at <ATG>/home/localconfig is at the highest precedence so if you don't explicitly specify any other config layer, ACC will write your changes at the component path within <ATG>/home/localconfig.

<ATG>/home/localconfig has a CONFIG.properties having the property defaultForUpdates=true which makes it the default for any updates. You can change this default by changing defaultForUpdates to false in <ATG>/home/localconfig/CONFIG.properties and create a CONFIG.properties with defaultForUpdates=true within your modules' config layer so that ACC by default can write your changes at the component path in your module's config layer.

Transactional Pattern to be followed in form handlers

Below is an article that I found very useful for understanding the transactional pattern that need to be followed in form handlers. Thanks to Oracle Knowledge Base :)


Code that modifies ATG Commerce Object must follow a specific transactional pattern to avoid exceptions and dead locks. The transactional pattern is implemented by the form handlers that work with order objects.Same transactional pattern can be followed in other form handlers as well

ATG Commerce form handlers that modify Order objects typically extends atg.commerce.order.purchase.PurchaseProcessFormHandler class. This subclass of atg.droplet.GenericFormHander is an abstract class that implements transaction management through its beforeSet, afterSet, and handler methods.

beforeSet - Called once before any form handler property is set or the handle method is called. It implements the following transactional steps:


If the formhandler's useLocksAroundTransactions property is true (default), obtain a transaction lock before the transaction is created.
This prevents a user from modifying an order in multiple concurrent threads. The lock name used defaults to the current profile ID. For more information, see the API documentation for atg.commerce.util.TransactionLockFactory. (Note that use of locking has a small performance impact.)
Check for an existing transaction and, if no transaction exists, create one.

 public boolean beforeSet(DynamoHttpServletRequest pRequest,
DynamoHttpServletResponse pResponse) throws DropletFormException
{
  try {
    acquireTransactionLock(pRequest);
  }
  catch (DeadlockException de) {

    // We are going to log the exception here and then ignore it because
    // the worst that should happen is that the user will get a concurrent
    // update exception if two threads try to modify the same order, and we
    // can recover from that.

    if (isLoggingError())
      logError(de);
  }

  Transaction t = ensureTransaction();
  if (t != null)
    setTransactionCreated(pRequest, pResponse);

  if (isLoggingDebug()) {
    if (t != null)
      logDebug("beforeSet created transaction " + t);
    else
      logDebug("beforeSet did not create a transaction.");
  }

  return super.beforeSet(pRequest,pResponse);
}

The handler methods implement the following transactional steps:

   a) Synchronize on the Order object.

    b) Execute logic for modifying the Order object.  For example, the CartModifierFormHandler subclass   has a handleAddItemToOrder method that executes the logic of adding an item to an order.

  c)  Call the OrderManager object’s updateOrder method to save the order data to the repository.

  d) End the synchronization.

The following example shows the handleAddItemToOrder method of the CartModifierFormHandler:

public boolean handleAddItemToOrder (DynamoHttpServletRequest pRequest,

DynamoHttpServletResponse pResponse)

  throws ServletException, IOException

{

  RepeatingRequestMonitor rrm = getRepeatingRequestMonitor();

  String myHandleMethod = "CartModifierOrderFormHandler.handleAddItemToOrder";

  if ((rrm == null) || (rrm.isUniqueRequestEntry(myHandleMethod)))

  {

    Transaction tr = null;

    try {

      tr = ensureTransaction();

      if (getUserLocale() == null) setUserLocale(getUserLocale(pRequest, pResponse));

      //If any form errors found, redirect to error URL:

      if (! checkFormRedirect(null, getAddItemToOrderErrorURL(), pRequest, pResponse))

        return false;

      synchronized(getOrder()) {

        preAddItemToOrder(pRequest, pResponse);

        //If any form errors found, redirect to error URL:

        if (! checkFormRedirect(null, getAddItemToOrderErrorURL(), pRequest, pResponse))

          return false;

        addItemToOrder(pRequest, pResponse);

        //If any form errors found, redirect to error URL:

        if (! checkFormRedirect(null, getAddItemToOrderErrorURL(), pRequest, pResponse))

          return false;

        postAddItemToOrder(pRequest, pResponse);

        updateOrder(getOrder(), MSG_ERROR_UPDATE_ORDER, pRequest, pResponse);

      } // synchronized

      //If NO form errors are found, redirect to the success URL.

      //If form errors are found, redirect to the error URL.

      return checkFormRedirect (getAddItemToOrderSuccessURL(), getAddItemToOrderErrorURL(), pRequest, pResponse);

    }

    finally {

      if (tr != null) commitTransaction(tr);

      if (rrm != null)

        rrm.removeRequestEntry(myHandleMethod);

    }

  }

  else {

    return false;

  }

}

afterSet Method

This method is called once after all form handler processing is completed. It implements the following transactional steps:

    Commit or roll back any transaction that was created in the beforeSet method.  If the transaction was already in place before the beforeSet method was called, the afterSet method does not end the transaction automatically; this is the application’s responsibility.  If a transaction lock was acquired in the beforeSet method, release the lock.

    The following example shows the afterSet method of the PurchaseProcessFormHandler:

public boolean afterSet(DynamoHttpServletRequest pRequest,

DynamoHttpServletResponse pResponse) throws DropletFormException

{

  try

  {

    Transaction t = getCurrentTransaction();

    if (isLoggingDebug()) {

      if (t != null)

        logDebug("afterSet sees currentTransaction as " + t);

      else

        logDebug("afterSet sees no current transaction.");

    }

    // Try to keep the response to this page from being cached.

    ServletUtil.setNoCacheHeaders(pResponse);

    if (t != null && isTransactionCreated(pRequest, pResponse)) {

      if (isLoggingDebug())

        logDebug("afterSet committing transaction " + t);

      commitTransaction(t);

      unsetTransactionCreated(pRequest, pResponse);

    }

  }

Java client for invoking a RESTful service

Below is a sample REST Java client written using Apache Commons HttpClient. Though the same can be achieved using java.net classes HttpClient is preferred as it provides higher and more functional  level of abstraction.



import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.IOException;
import java.io.InputStreamReader;
import java.io.StringWriter;

import org.apache.commons.httpclient.HttpClient;
import org.apache.commons.httpclient.HttpException;
import org.apache.commons.httpclient.methods.GetMethod;

import atg.core.util.StringUtils;
import atg.nucleus.GenericService;

/**
 * @author kshabarinath
 * This is a sample REST client which is used to access a service exposed using URL pattern
 *
 */


public static void main(String[] args) {
String requestURL = "http://www.thomas-bayer.com/sqlrest/CUSTOMER/999";
HttpClient client = new HttpClient();
GetMethod method = new GetMethod(requestURL);
String line;
InputStream rstream;
try {
int statusCode = client.executeMethod(method);
String response = getResponseBody(method);
System.out.println("RESPONSE::"+response);
} catch(HttpException e){
e.printStackTrace();

}catch (IOException e) {

// TODO Auto-generated catch block
e.printStackTrace();
}finally{
method.releaseConnection();
}
}



private static String getResponseBody(GetMethod method){
if(method!=null && method.hasBeenUsed()){
BufferedReader in = null;
StringWriter stringOut = new StringWriter();
BufferedWriter dumpOut = new BufferedWriter(stringOut,8192);
try{
in=new BufferedReader(new InputStreamReader(method.getResponseBodyAsStream()));
String line = "";
while((line=in.readLine())!=null){
dumpOut.write(line);
dumpOut.newLine();
}
}catch(IOException e){
e.printStackTrace();
}finally{
try{
dumpOut.flush();
dumpOut.close();
if(in!=null){
in.close();
}
}catch(IOException e){
e.printStackTrace();
}
}
return stringOut.toString();
}
return null;
}

OUTPUT of the Program


RESPONSE::<?xml version="1.0"?><CUSTOMER xmlns:xlink="http://www.w3.org/1999/xlink">
    <ID>999</ID>
    <FIRSTNAME>Sylvia</FIRSTNAME>
    <LASTNAME>Ott</LASTNAME>
    <STREET>361 College Av.</STREET>
    <CITY>Viena</CITY>
</CUSTOMER>

Error Codes - Some of the frequently issues

1. 403 - Cannot Serve URLs with character in them


The HTMLServlet and the FileFinderServlet servlets in the request handling servlet pipeline check for the characters '..' and 'null'. If these appear in the URL the "Cannot serve URLs" error message is shown.


These characters are hard coded in the source and cannot be parametrized.


Source Code Snippet: - FileFinderPipelineServlet


 if(pPathInfo.indexOf("..") != -1)
                {
                    sendError(pResponse, 403, "Cannot serve URLs with ..'s in them");
                    return false;
                }
                if(pPathInfo.indexOf('\0') != -1)
                {
                    sendError(pResponse, 403, "Cannot serve URLs with nulls in them");
                    return false;
                }


2. 404 Error Pages Rendered From web.xml Definition Not Multisite Aware 


A 404 error-page was added to the application's web.xml  file
<error-page> 
    <error-code>404</error-code> 
    <location>/error.jsp</location>
</error-page>


However, the request does not appear to be processed by the pipleline, as none of the site formatting or branding for Multisite is being displayed with the 404 error.jsp


The 404 exceptions are handled via the Application Server (ie WebLogic, JBoss, WebShpere). If the web.xml file does not include instructions for how the Applications Servers, Dispatcher should handle ERRORs, the Application Server will not forward the Request to the PageFilter servlet -> DynamoHandler -> SiteContextPipelineServlet , and on down the pipeline. Hence, none of the Formatting/Branding handled by Multisite and the SiteContextPipelineServlet are triggered.


We need to abide to the pattern followed in CRS web.xml when we define the error pages for error codes:


1. Define the filters
<filter>
    <filter-name>PageFilter</filter-name>
    <filter-class>atg.filter.dspjsp.PageFilter</filter-class>
 </filter>
 <filter>
    <filter-name>ErrorFilter</filter-name>
    <filter-class>atg.servlet.ErrorFilter</filter-class>
  </filter>


2. Define the filter mappings
 <filter-mapping>
    <filter-name>ErrorFilter</filter-name>
    <url-pattern>/*</url-pattern>
    <dispatcher>ERROR</dispatcher>
  </filter-mapping>
  <filter-mapping>
    <filter-name>PageFilter</filter-name>
    <url-pattern>/*</url-pattern>
    <dispatcher>REQUEST</dispatcher>
    <dispatcher>ERROR</dispatcher>
    <dispatcher>FORWARD</dispatcher>
  </filter-mapping>


3. Include all the error page elements
 <error-page>
    <error-code>404</error-code>
    <location>/global/pageNotFound.jsp</location>
  </error-page>
  <error-page>
    <error-code>500</error-code>
    <location>/global/serverError.jsp</location>
  </error-page>