Lotus Connections Profiles Population Wizard, What is Really Going On Under the Covers

I have been doing quite a bit of data integration lately to bring together data from four different sources in to Lotus Connections Profiles.  Along the way I have learned quite a bit about Tivoli Directory Integrator (TDI).  TDI while a little complicated at first is a pretty flexible tool capable of carrying out all the data integration both initially, and maintaining the data across systems. If you are installing Lotus Connections your first interaction with TDI will be via the Database Population Wizard, a guided interface which executes TDI scripts in the background.  I thought it would be a good idea to put together some information on what is actually going on in the background, as a way of explaining TDI a little, and gain a better understanding of the process. I am using the following software TDI 6.1.1 (Part Number C9666ML) TDI 6.1.1 Fix Pack 3 TDI 6.1.1 IF0008 Connections Database Configuration and Profiles Population Wizard (Part Number C1L11ML) My database is DB2 though Oracle and MS SQL are also supported ( I have worked extensively with Oracle and DB2) When you launch the population wizard the first screen is the Database Configuration As you fill in the fields, the values are actually being stored in a file named profiles_tdi.properties (this file is located under the Wizards directory in “TDIPopulationTDISOL” this is the TDI Solutions Directory that will be used when executing TDI for Profiles.  We will come back to this file a little later but here are the relevant sections dbrepos_jdbc_url=jdbc:db2://db2.the80s.com:50000/PEOPLEDB dbrepos_password={encr}k2bdc4849865965597ULZ7kTRrzm7q/4pAxPtLgfQXT1TVdC8l7cYVotd+Tryy8287TpX5HM8wMAnfo= dbrepos_username=lcuser dbrepos_jdbc_driver=com.ibm.db2.jcc.DB2Driver The next screen is where you configure your LDAP Connection and the relevant lines being populated in the profiles_tdi.properties source_ldap_url=ldap://sitcom.the80s.com:389 source_ldap_use_ssl=false Next up is the credentials to connect to LDAP and the relevant lines being populated in the profiles_tdi.properties source_ldap_user_login=Sitcom Administrator source_ldap_user_password={encr}Xs9eJLne8LBtcYq5Yo+91Hrd1jnS6RG4sd9699655VHZYmBeEbSOpvkds7+9IXvlDjgwu7cGcfAJvzCu/X Next is defining the search base, and the LDAP search filter in this example my O is ‘the80s’, and I am filtering for Person Documents and the relevant lines being populated in the profiles_tdi.properties source_ldap_search_base=o=the80s source_ldap_search_filter=(&(uid=*)(objectclass=inetOrgPerson)) In the next screen you map the attributes from your LDAP to the fields in Profiles, these values will be saved in map_dbrepos_from_source.properties which is also under the Wizards directory in “TDIPopulationTDISOL” These values should be set as they apply to your environment.  While not completely relevant to this topic the key field is the PROF_GUID field which needs to be set specifically for your LDAP.  For Domino LDAP is is ‘dominoUNID’  This table will display the specifics for the various LDAP configurations supported in Connections.   Any field you don’t want populated should be left blank, and will contain the value ‘null’ in the propertied file. PROF_GUID=dominoUNID PROF_FLOOR=null Finally when you populate the database you have the option to run other jobs at the same time, these values are stored in the profiles_tdi.properties country_table_csv_file=isocc.csv – this is to populate country code information in to the COUNTRY Table (isocc.csv is provided in the solutions directory if you are using standard ISO country codes) The remaining options are more company specific so you have to find the source for the data in your organization department_table_csv_file=deptinfo.csv populates the DEPARTMENT table with your organizations department scheme organization_table_csv_file=orginfo.csv populates the ORAGNIZATION table with your organizations organizational units emp_type_table_csv_file=emptype.csv populates the EMP_TYPE table with employee type information workloc_table_csv_file=workloc.csv populates the WORKLOC  table with your organizations work location In addition for each job there is a line defining the separator, they are all defaulted to ‘;’ workloc_table_csv_separator=; To change them you need to edit the file, there is no option in the UI The final option is to run the Mark Managers task, run this to mark managers so that Reporting Structure will display properly in Connections (only run this task if you are providing the org structure in the data load) This is all the configuration the wizard requires and if you simply let it run, it will populate the database with the parameters you selected.  What you will see next is how you can leverage the setup information provided in the wizard to run these jobs individually,  schedule them to run on an interval to maintain the data, or leverage the configuration already completed to run your own custom jobs. TDI should be launched from the Solutions Directory  in this case WizardsTDIPopulationTDISOL.  Edit tdienv.bat for your environment, run it, and then execute ibmditk.bat in from where you installed TDI (Program FilesIBMTDIV6.1.1 is the default) Once launched open the profiles_tdi.xml, and expand the Assemblylines sections.   Each assembly line represents one set of data to be managed, you will notice the various jobs that correspond to the steps we looked at earlier for example “populate_country” is the job to populate the COUNTRY table. Opening up the populate_country assembly line we can see the various components of the assembly line. “http://www.flickr.com/photos/curiousmitch/3119673012/” title=”tdi3 by CuriousMitch, on Flickr”> The first thing you should note is that there are multiple config tabs, this is a recurring them throughout TDI and takes a little getting used to.  The top Config Tab is the configuration for the entire assembly line.  The one below it is the config for a specific flow in this case “update_db”.  An assembly line can contain one or multiple Feeds and Flows. Though they don’t appear that way each of the items in the red box is a clickable item.  For example JDBC URL which as you can see contains the value we specified earlier in the properties file.   We could simply type it in here, but look how many assembly lines and how many configurations we have that might require the same value.   If we click on “JDBC URL” we see that the Expression is {properties.profiles:dbrepos_jdbc_url} If we navigate away from the Assembly line and over to the Properties section we can see the various “Properties” files attached to this configuration On the Connector Configuration tab we see that this is where the assembly line is connected to the “profiles_tdi.properties” file we set up earlier On the Editor tab we can see the values assigned By defining the properties here, and configuring the assembly lines to use references instead of hard coding them, the TDI solution becomes much more manageable.  For example if you have a production and a test environment you can  maintain your assembly lines to be identical and simply use different property files this will save you lots of time when you make changes to your solutions. As you develop your own assembly lines you can define your own property files and properties with in them for ease of use. Lastly you can execute an assembly line without launching TDI to do that you need to run ibmdisrv.bat which is in the directory where TDI is installed (again run it from the solution directory) followed by the xml file which contains the assembly  line with a -c parameter followed by the assembly line to run with the -r parameter.  To run the populate_country assebly line we looked at above the command would be “ibmdisrv -c profiles_tdi.xml -r populate_country Additional Resources Lotus Connections Infocenter: Populating the Profiles database Tivoli Directory Integrator documentation IBM Tivoli Directory Integrator Users Group You Tube has a few videos with examples of TDI, the search results are here. .