Tuesday, 25 November 2014

Export the data from CAS Record Stores from one environment to other (ATG - ENDECA)


When the baseline indexing is triggered from ATG, the product catalog data is exported from ATG to CAS Record Store Instances. The EndecaScriptService component then triggers the baseline update script of the Endeca Application to invoke the forge which loads the CAS record stores data and indexes the same based on the pipeline definition.

Sometimes when we want to debug the index of UAT/Production Environment in local we need to build the index in local with the same data that is in production. Since we use Endeca CAS record store instances which stores the data in binary format it is not straight forward to build the index in local with the same data.

We can follow the below steps to export the Endeca CAS Record Store Instances from one environment and other and index the same.


Navigate to CAS Installation directory and list the record stores

/u01/home/app/atg/soft/endeca/CAS/11.0.0/bin >> sh component-manager-cmd.sh list-components

Note the Record Store Instances for the Endeca Application that you are interested in



/u01/home/app/atg/soft/endeca/CAS/11.0.0/bin >> 
sh recordstore-cmd.sh list-active-transactions -a Lifestyle_en_dimvals
sh recordstore-cmd.sh list-active-transactions -a Lifestyle_en_prules
sh recordstore-cmd.sh list-active-transactions -a Lifestyle_en_data
sh recordstore-cmd.sh list-active-transactions -a Lifestyle_en_schema


/u01/home/app/atg/soft/endeca/CAS/11.0.0/bin >> 

sh recordstore-cmd.sh get-configuration -a Lifestyle_en_dimvals -f cfg_Lifestyle_en_dimvals.xml
sh recordstore-cmd.sh get-configuration -a Lifestyle_en_prules -f cfg_Lifestyle_en_prules.xml
sh recordstore-cmd.sh get-configuration -a Lifestyle_en_data -f cfg_Lifestyle_en_data.xml
sh recordstore-cmd.sh get-configuration -a Lifestyle_en_schema -f cfg_Lifestyle_en_schema.xml


/u01/home/app/atg/soft/endeca/CAS/11.0.0/bin >> 
sh recordstore-cmd.sh read-baseline -a Lifestyle_en_dimvals -f baseline_Lifestyle_en_dimvals.xml
sh recordstore-cmd.sh read-baseline -a Lifestyle_en_prules -f baseline_Lifestyle_en_prules.xml
sh recordstore-cmd.sh read-baseline -a Lifestyle_en_schema -f baseline_Lifestyle_en_schema.xml
sh recordstore-cmd.sh read-baseline -a Lifestyle_en_data -f baseline_Lifestyle_en_data.xml

Copy the configuration and baseline data files to another environments CAS Installations directory and run the below commands



recordstore-cmd.bat set-configuration -a Lifestyle_en_dimvals -f cfg_Lifestyle_en_dimvals.xml
recordstore-cmd.bat set-configuration -a Lifestyle_en_prules -f cfg_Lifestyle_en_prules.xml
recordstore-cmd.bat set-configuration -a Lifestyle_en_schema -f cfg_Lifestyle_en_schema.xml
recordstore-cmd.bat set-configuration -a Lifestyle_en_data -f cfg_Lifestyle_en_data.xml



recordstore-cmd.bat write -a Lifestyle_en_dimvals -f baseline_Lifestyle_en_dimvals.xml
recordstore-cmd.bat write -a Lifestyle_en_prules -f baseline_Lifestyle_en_prules.xml
recordstore-cmd.bat write -a Lifestyle_en_schema -f baseline_Lifestyle_en_schema.xml
recordstore-cmd.bat write -a Lifestyle_en_data -f baseline_Lifestyle_en_data.xml


The newly created index will be similar to the one in other environment from which the data is exported.

Hope this helps !!!

Thursday, 7 November 2013

Using the Assembler to connect to Lifestyle Endeca Application

In my earlier posts, I have detailed on how to create a new Endeca Appication 'Lifestyle' based on discover-data deployment template and new reference web application based on discover-electronics reference application.

I shall now detail how to connect to Endeca Application using an Assembler


  • Windows>Preferences (Expand server and select Runtime Environments) and click ADD
  • Select Apache Tomcat and provide the tomcat installation directory path (C:\Endeca\ToolsAndFrameworks\3.1.2\server\apache-tomcat-6.0.32)
  • Once done, create a new server using the 'Servers' tab and provide the server name as 'lifestyleWebServer'
  • Start the 'lifestyleWebServer' and confirm it starts up successfully using the console tab


  • Create a new dynamic web project 'Lifestyle'
  • Add the JARs in the directory (C:\Endeca\ToolsAndFrameworks\3.1.2\reference\discover-electronics-authoring\WEB-INF\lib) to the Web App using Java Build Path. Also click on the 'Order and Export' tab and select the JARs that we have just added
  • Right-click on Web App, select properties -> Deployment Assembly. Click Add and select the Java Built Path Entries and Click Next. Select all the entries and finish
  • Importing classes from discover-authoring to Lifestyle web app
    • Lifestyle>Web Content>WEB-INF and choose import from file system. Navigate to directory (C:\Endeca\ToolsAndFrameworks\3.1.2\reference\discover-electronics-authoring\WEB-INF\classes)
    • Select the 'classes' folder, and check the two options (Override existing resources and create top-level folfers)
  • Configure Assembler
    • Copy the files (assembler.properties, assembler-context.xml, endeca-url-config.xml,perf-logging-config.xml,urlrewrite.xml and web.xml) from the directory (C:\Endeca\ToolsAndFrameworks\3.1.2\reference\discover-electronics-authoring\WEB-INF) to a local directory (C:\Endeca|Lab\AssemblerFiles) and do the below modifications
    • Replace the web-app xmlns entry in web.xml as (<web-app xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://java.sun.com/xml/ns/javaee" xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd" id="WebApp_ID" version="2.5">)
    • Ensure that you are pointing to the correct application and mdex host by verifying the configured values in assembler.properties
    • Search for 'endeca-seo-url-config.xml' entry in assembler-context.xml and replace it with 'endeca-url-config.xml'
    • Search for 'urlFormatter' entry in assembler-context.xml and replace the text ref="seoUrlFormatter" to ref="basicUrlFormatter"
    • Lifestyle> Web Content>WEB-INF and choose import from file system. Navigate to directory (C:\Endeca|Lab\AssemblerFiles)
    • Select all the files, but ensure that 'Create top-level folder is not selected'
  • Right click on WEB-INF directory and create a new folder 'services'
  • Create a new JSP file 'lifestyleAssemble.jsp'
  • Remove all the HTML content and replace the same with below
<%@page import="com.endeca.infront.assembler.Assembler"%>
<%@page import="com.endeca.infront.assembler.AssemblerFactory"%>
<%@page import="com.endeca.infront.assembler.ContentItem"%>
<%@page import="com.endeca.infront.cartridge.ContentInclude"%>
<%@page import="com.endeca.infront.serialization.JsonSerializer"%>
<%@page import="com.endeca.infront.serialization.XmlSerializer"%>
<%@page import="com.endeca.infront.navigation.UserState"%>
<%@page import="com.endeca.mobile.services.detection.DeviceManager"%>
<%@page import="org.springframework.web.context.WebApplicationContext"%>
<%@page import="org.springframework.web.context.support.WebApplicationContextUtils"%>
<%@page import="java.util.Properties"%>
<%@page import="java.util.LinkedHashMap"%>
<%@page import="java.util.Map"%>
<%@page import="java.util.ArrayList"%>
<%@page import="java.util.List"%>

<%@page language="java" contentType="text/html; charset=UTF-8"%>
<%@ taglib prefix="c" uri="http://java.sun.com/jsp/jstl/core"%>

WebApplicationContext webappCtx = WebApplicationContextUtils
AssemblerFactory assemblerFactory = (AssemblerFactory) webappCtx
Assembler assembler = assemblerFactory.createAssembler();
ContentItem contentItem = new ContentInclude("/pages/browse");
ContentItem responseContentItem = assembler.assemble(contentItem);

String format = request.getParameter("format");
if ("json".equals(format)) {
response.setHeader("content-type", "application/json");
new JsonSerializer(response.getWriter())
} else if ("xml".equals(format)) {
response.setHeader("content-type", "application/xml");
new XmlSerializer(response.getWriter())
  • Search for the servlet entry 'assemble' in web.xml and change the jsp-file entry to 'lifestyleAssemble.jsp'

  • Click on the servers tab, right click on 'lifestyleWebserver' and choose Add and Remove. Add the 'Lifestyle' web app 
  • Restart the tomcat server instance
  • Navigate to the URL (http://localhost:8080/Lifestyle/browse?format=json') from browser and you should see the JSON reponse for the browse page as configured in Endeca Experience Manager rendered using the Assembler

Sunday, 22 September 2013

Environment Variables - Oracle ATG Commerce & Oracle Endeca Commerce

Environment Variables for Oracle ATG Commerce & Oracle Endeca Commerce Set Up

We always miss few very important environmental variables which is main reason to land into many installation issues

Once the installation set up is done successfully following the steps described in the documentation, ensure that below user environment variables are set

  • DYNAMO_ROOT            C:\Kande\ATG\ATG10.2
  • DYNAMO_HOME                    %DYNAMO_ROOT%\home
  • JAVA_HOME                    C:\Kande\Java\jdk1.6.0_45
  • JBOSS_HOME                    C:\Kande\jboss-eap-5.1\jboss-as
  • ENDECA_CONF                            C:\Kande\Endeca\PlatformServices\workspace
  • ENDECA_REFERENCE_DIR           C:\Kande\Endeca\PlaformServices\reference
  • ENDECA_ROOT                            C:\Kande\Endeca\PlatformServices\6.1.3
  • ENDECA_TOOLS_CONF                C:\Kande\Endeca\ToolsAndFrameworks\3.1.2\server\workspace
  • ENDECA_TOOLS_ROOT                C:\Kande\Endeca\ToolsAndFrameworks\3.1.2
  • Path                                            %JAVA_HOME%\bin;%Path%;%DYNAMO_HOME%\bin;%JBOSS_HOME%\bin
  • PERL5LIB                                     %ENDECA_ROOT%\perl;%ENDECA_ROOT%\perl\5.8.3\lib
  • PERLLIB                                       %ENDECA_ROOT%\perl;%ENDECA_ROOT%\perl\5.8.3\lib

Thursday, 7 February 2013

Setting up preview application based on Discover Electronics Authoring Application

Setting up Lifestyle authoring application based on Discover Electronics Authoring Application

In my previous post, I have described the steps that need to be followed to create a new reference application(Lifestyle) based on the Discover-Electronics reference application. Having made any changes to the pages in lifestyle app in experience we would want to see how the page looks before promoting the content. That's where the preview application comes into pic. Clicking the preview button in the experience manager against each page designed in the experience manager opens up a new window depicting how the page looks with the changed configurations.

By default, when we create an Endeca application using discover-data, the preview settings are defines as below:

  • Preview URL - http://localhost:8006/discover-authoring
  • Link Service URL - http://localhost:8006/discover-authoring/link.json
Since we are using lifestyle application for application for any page changes, we shall focus on using lifesyle-authoring reference application for preview. Below are the steps to be followed to create a new authoring application based on discover-authoring reference application.

1. Navigate to C:\Endeca\ToolsAndFrameworks\3.1.1\referencedirectory, copy the 
discover-electronics-authoring reference application folder, paste it in the same directory and name it as lifestyle-authoring

2. Modify the following properties in assembler.properties file of lifestyle-authoring reference application

  • workbench.app.name=lifestyle
  • workbench.host=localhost
  • workbench.port=8006
  • workbench.publishing.serverPort=8007
  • mdex.host=localhost
  • mdex.port=18002
  • logserver.host=localhost
  • logserver.port=18010 

3. Navigate to ENDECA_TOOLS_CONF sub directory C:\Endeca\ToolsAndFrameworks\3.1.1\server\workspace\conf\Standalone\localhost, copy the
 discover-authoring.xml file, paste it in the same directory and rename it to lifestyle-authoring.xml and make the below changes

4. Under the preview settings for the lifestyle application in the experience manager, make the below changes
  • Preview URL : http://localhost:8006/lifestyle-authoring
  • Link Service URL : http://localhost:8006/lifestyle-authoring/link.json
5. Restart the EndecaToolsService

6. You should now be able to access http://localhost:8006/lifestyle-authoring reference application which is also the preview app that gets loaded when the preview button is used in the experience manager.


Thursday, 31 January 2013

Setting up an application based on Discover Electronics

Setting up LifeStyle Endeca Application and LifeStyle Reference Application

We can create an Endeca application based on discover-data deployment deployment template by following the steps provided in Getting Started Guide (http://docs.oracle.com/cd/E38680_01/Common.311/pdf/GettingStarted.pdf - Deploying a Reference Application) and then view the sample pages using discover-electronics reference application (http:localhost:8006/discover)

As beginners, we tend to play around the reference application by changing the pipeline, creating a new cartridge templates and then seeing the changes in the reference application. To do this we should have a reference application that is specific to our application so that we can make all the changes as we need without effecting the base reference application.

I have followed the below steps to create a new Endeca application and corresponding new reference application to make the changes.

Check that the Endeca HTTP Service and Endeca Tools Service are up and running

1. Create a new Endeca application named lifestyle based on discover-data deployment template

a) Navigate to cd C:\Endeca\ToolsAndFrameworks\3.1.1\deployment_template\bin and execute

deploy.bat --app ..\..\reference\discover-data\deploy.xml

Provide the below parameter values

  • Application Name: lifestyle
  • Deployment Directory : C:\Endeca\apps
  • EAC Port : 8888
  • Workbench port : 8006
  • Live Dgraph port : 18000
  • Authoring Dgraph port : 18002
  • Log Server Port : 18010  

Ensure that the application is successfully deployed

b) Navigate to the control directory of the newly created application and execute the initialize_services.bat to provision the application to EAC

c) Execute the load_baseline_test_data to load the test data

d) Run the baselineupdate to perform full indexing

e) Execute promote content to promote the content to work bench

2. Navigate to C:\Endeca\ToolsAndFrameworks\3.1.1\reference directory, copy of the discover-electronics reference application folder, paste it in the same directory and name it as lifestyle

3. Modify the following properties in assembler.properties file of lifestyle reference application


  • workbench.app.name=lifestyle
  • workbench.host=localhost
  • workbench.port=8006
  • workbench.publishing.serverPort=8007
  • mdex.host=localhost
  • mdex.port=18000
  • logserver.host=localhost
  • logserver.port=18010

4. Navigate to ENDECA_TOOLS_CONF sub directory C:\Endeca\ToolsAndFrameworks\3.1.1\server\workspace\conf\Standalone\localhost, copy the discover.xml file, paste it in the same directory and rename it to lifestyle.xml and make the below changes
  • path="/lifestyle"
  • docbase="${catalina.base}/../../reference/lifestyle"

5. Restart the EndecaToolsService

6. Verify that lifestyle application is present in the Endeca Workbench with the sample cartridge templates and sample pages.

http://localhost:8006/admin > login and select the lifestyle Endeca Application

7. Verify that the lifestyle  reference application is available with sample page views configured in the experience manager with camera test data.

http://localhost:8006/lifestyle to view the lifestyle reference web application

We have successfully created lifestyle Endeca application using discover-data deployment template and successfully have corresponding lifestyle web application running. We can now create a new pages in experience manager, make corresponding view changes in the reference application and see the changes.

Wednesday, 30 January 2013

Stepping into Oracle Endeca Commerce 3.1.1

Since Oracle has released its latest version of Endeca Commerce 3.1.1 very recently, I decided to migrate my current Endeca Installations to that. Please find below the steps followed to achieve the same.

Release Document of Endeca Commerce 3.1.1 states that only the following packages have been updated

  • Tools and Frameworks 3.1.1
  • Mdex Engine 6.4.0
  • Content Acquistion System 3.1.1
Since my current installations are from Oracle Endeca Commerce 3.1.0 and Platform Services has not been updated in this release, I decided to upgrade only Tools and Frameworks and Mdex Engine. Since I am not going to use ATG 10.1.2 to integrate with Endeca, I am not installing CAS and will not be using discover-data-catalog-integration deployment template for creating the Endeca Application.

Since I need to upgrade my Mdex Engine from 6.3.0 to 6.4.0 and Tools and Frameworks from 3.1.0 to 3.1.1, I followed the below steps.

1. Identify the current Endeca applications that have been installed and remove them.

a) Navigate to PlatformServices Installation Directory/bin (Eg: C:\Endeca\PlatformServices\6.1.3\bin) and execute the command eaccmd.bat list-apps to get the list of Endeca Applications that are provisioned to EAC

b) Since there is only application Discover , next step is to remove the application. Navigate to the control directory of the Discover application and execute the runcommand.bat with remove--app arguments

c) Remove the Instance Configurations completely, using emgr_update.bat with remove_all_settings argument
emgr_update.bat --host localhost:8006 --app_name My_app --action remove_all_settings --prefix My_prefix

d) If you are using CAS Record Store Instances to load the data and have created the Endeca application using ProductCatalogIntegration deployment template, the you need to remove the record store instances for your application

C:\Endeca\CAS\3.0.2\bin>component-manager-cmd.bat delete-component -n {record_store_name}

To get the list of recod stores you can use the below command

C:\Endeca\CAS\3.0.2\bin>component-manager-cmd.bat list-components

e) Close the command prompt (if open), navigate to the Endeca applications directory (Eg: C:\Endeca\apps) and delete the Discover folder completely.

Follow the above steps to remove all the Endeca applications completely that are listed using list-apps command described above.

2. Navigate to the Tools&Frameworks/server/bin folder (Eg: C:\Endeca\ToolsAndFrameworks\3.1.0\server\bin) folder and execute the uninstall_service.bat file

Navigate to the Endeca Installation Directory(Eg: C:\Endeca) and delete the ToolsAndFrameworks folder completely.

3. Stop the HTTP Service from the services console (You will not see the Endeca Tools Service as it is uninstalled in the above step) and uninstall the MDEX Engine from the Control Panel/Add or Remove programs.

Navigate to the Endeca Installation Directory (Eg: C:\Endeca) and delete the MDEX folder completely

4. Execute the mdex_6.4.0.692722_x86_64pc-win32.exe file and complete Mdex Installation successfully.

5. Extract the Endeca-Tools-Frameworks-with-Experience-Manager-2.1.2-V35745-01  zip file to Endeca Installation Directory and ensure that editor_sdk folder is present in C:\Endeca\ToolsAndFrameworks\3.1.1 directory (Tools and Frameworks installation directory) to confirm it  supports experience manager and not guided search.

6. Execute the install_services.bat from the ToolAndFrameworks/server/bin folder

7. Since we have upgraded the MDEX without re-installing the platform services, we need to inform it about the change in MDEX version. So we need to update the mdexRoot path in eac.properties present in the location (C:\Endeca\PlatformServices\workspace\conf\eac.properties). Failing to do this step would produce Errors when a baseline update is performed on any Endeca application that is deployed later. i.e
Error Communicating with EAC Agent while starting component. Occurred while executing line 25 of valid Bean shell script

8. The MDEX 6.4 zip file contains the Presentation API jars that are compatible with MDEX 6.4. Copy those jars to the Endeca_JSP Ref web application to avoid ENEException later.

For Eg: Copy endeca_logging.jar, endeca_navigation.jar from  C:\Endeca\presAPI_6.4.0.692722_x86_64pc-win32\PresentationAPI\6.4.0\java\lib to C:\Endeca\PlatformServices\6.1.3\tools\server\webapps\endeca_jspref\WEB-INF\lib.

Failing to do so might produce ENEExecption in the EndecaJSPRef application because of the version mismatch 

Oracle Endeca - JSP Reference Application
status>>ENE error
com.endeca.navigation.ENEException: Navigation Engine not able to process request 'http://localhost:15000/graph?node=0&offset=0&nbins=10&iversion=620'.

9. Start the Endeca HTTP Service and Tools Service from services console.

By following the above steps I have successfully migrated from Oracle Endeca Commerce 3.1.0 to Oracle Endeca Commerce 3.1.1. Next job would be to create a sample Endeca Application and test if my installations are successful !!!

Sunday, 13 January 2013

New look to Oracle ATG Commerce with Oracle Endeca Commerce

With Oracle acquiring both ATG and Endeca, it has now grouped both the products under one umbrella called ORACLE COMMERCE comprising ORACLE ATG COMMERCE and ORACLE ENDECA COMMERCE.  Oracle has already announced that they would stop their support for ATG search in next couple of years and are encouraging all the existing applications developed in ATG Search to migrate to Endeca.

As part of the products integration, Oracle has released two major versions of ATG for integration of Oracle ATG Commerce to Oracle Endeca Commerce. With the latest version of Oracle ATG Commerce 10.1.2 one can easily integrate with the latest version Oracle Endeca Commerce 3.1.1 with the help of many OOTB components provided in ATG as well as the Deployment Template provided in Endeca.

Endeca Commerce, by itself is a well proven product for Search and with the power of ATG Commerce for the core commerce functionality we can build an Enterprise e-commerce application that can give the best experience to the customers which is highly stable and scalable.

Now that the gates are open for the integration of ATG Commerce with Endeca Commerce, there are some challenges as well when we take into consideration of the existing clients who would like to integrate their ATG applications with Endeca Commerce.

1) Do I need to migrate my existing ATG application which is currently running in (7.0 or 9.0) to 10.1.2 to integrate with Endeca Commerce 3,1,1. Aren't there any steps to integrate my existing application with Endeca Commerce?

2) If my existing ATG application containing a lot of custom BCC changes as per my requirement is running well today, what are the challenges in front to migrate to ATG 10.1.2 so as to integrate with Endeca Commerce? As ATG 10.0 and above uses flex in Merchandising UI, is it advisable to migrate to 10 so as to integrate with Endeca ?

3) Though one can index the data from ATG product catalog to Endeca using the OOTB components, we still have to make use of Developer Studio for things such as creating a search interface and so on. Will we be able to do all the required stuff from ATG itself going forward?

As change is inevitable, without worrying about the challenges ahead one should start making use of Endeca Commerce capabilities with the best possible way for their current ATG solution.

Note: As I am completely new to Endeca, I am planning to pen down all the configuration steps, issues faced, migration steps and so in my upcoming blogs.