1551 Wednesday, June 1, 2016 |
Suresh Maurya
Dynamics CRM Consultant at IBM

Understanding Entity Change Tracking in Microsoft Dynamics CRM

In CRM integration projects, your business requirements often dictate that you should only retrieve those transaction records that have been recently created or modified in your source system since last integration. When your CRM system is the source, you would typically use CRM FetchXML query to pull data from the CRM system by comparing the record's modifiedon field, which is not a solution that is reliable enough and may not perform well when the source entity has a significant number of records.

To address this particular challenge, CRM Online 2015 Update 1 release introduced the Change Tracking feature, which offers a reliable and efficient way to track transactional data changes for CRM entities.

Turn on Change Tracking

In order to track data changes of a particular entity, you have to first turn on "Change Tracking" option for the entity. You would do so by going to the entity's customization page, and tick "Change Tracking" option under "Data Services" section.

Retrieve Changes

After Change Tracking has been enabled for the entity, you can use CRM SDK RetrieveEntityChangesRequest message to pull changes from CRM server. The following is a snippet that demonstrates how to pull changes from the CRM server.

 using (var service = new OrganizationService(crmConnection))
{
var request = new RetrieveEntityChangesRequest();

// Set which entity to get changes for.
request.EntityName = "account";
request.Columns = new ColumnSet("accountnumber", "name", "creditlimit");

// Set paging preferences.
request.PageInfo = new PagingInfo() { Count = 5000, PageNumber = 1 };

// Set change token returned from last pull.
request.DataVersion = changeToken; // set to null or remove this line to do an initial pull

// Get the changes.
var response = (RetrieveEntityChangesResponse)service.Execute(request);

// TODO: Process all the changed records (see the code snippet below)

// Save the token somewhere for future use
changeToken = response.EntityChanges.DataToken;
Console.WriteLine(changeToken);
}

Walk through Changes

After retrieving the changes, you will get two types of results. The first type of result is the newly added or updated records, and the second one is for the records that have been deleted since the last pull.

// Replace the TODO line in the above snippet with the following code
foreach (var change in response.EntityChanges.Changes)
{
if (change.Type == ChangeType.NewOrUpdated)
{
var changedItem = (NewOrUpdatedItem)change;
Entity newOrChangedEntity = changedItem.NewOrUpdatedEntity;
// TODO: Process new or updated entity record
}
else if (change.Type == ChangeType.RemoveOrDeleted)
{
var deleteditem = (RemovedOrDeletedItem)change;
EntityReference deletedEntityReference = deleteditem.RemovedItem;
// TODO: Process deleted entity records
}
}

Note that the returned NewOrChangedEntity is an instance of Entity class, while the RemovedItem (the deleted record) is an instance of EntityReference class.

Some Closing Notes

There are a few things that you should be aware when using this feature:

  • The "Change Tracking" option is an entity-level setting. It has to be enabled for the entity before you can use RetrieveEntityChangesRequest to pull data. If the option is not enabled, and you try to use this feature, you will get an error message telling you "Entity: account isn't enabled for change tracking" where account can be any entity name that you are working with.
  • Due to the fact that the "Change Tracking" option works at entity level, you would need to keep track of the change token for each entity individually. There is no organization level tracking token.
  • There is one special situation that you should watch out for. Suppose there is a newly added record after last pull and it was deleted before the new pull, you will get the record in the Delete result set, which you may not have knowledge about it.
  • In the case that you have selected to return a lookup field, the field's Name property is not returned.
  • When change collection has more records than the page size that you have specified, you would have to page through the change collection.
  • Based on my preliminary testing, it appears that CRM keeps multiple token versions, which is really nice. However, I imagine keeping too many versions on the server side would not make sense as each version would consume a certain amount of database space. There must be a limit in terms of how many versions are kept on the CRM server side. It could be either a time-based limit (say 90 days probably, which could still be a lot of change versions if I keep pulling from the server every 30 seconds) or a number-based limit. This is something that has yet to be confirmed.
  • When you retrieve changes using this feature, you will get the records in their current status, the old values before the change are not returned.
  • The user account needs to have organization level read (or so-called "Root Read") privileges for the concerned entity in order to uses RetrieveEntityChangesRequest to retrieve changes.
  • More details about this feature can be found at the CRM SDK documentation page: Use change tracking to synchronize data with external systems.

Images in Microsoft Dynamics CRM

MS CRM Notes


Images in Microsoft Dynamics CRM

Images in Dynamics CRM are snapshot of record in SQL server, before and after the core system operation.

Pre-Image

Pre-Image, returns the image of the record as is stored in the SQL database before the CRM Platform action has been performed.

Post-Image

Post-Image, returns the image of the record as is stored in the SQL database after the CRM Platform action has been performed.

Message Stage Pre-Image Post-Image
Create PRE No No
Create POST No Yes
Update PRE Yes No
Update POST Yes Yes
Delete PRE Yes No
Delete POST Yes No

Example Usage:

  • Comparison of record before and after the operation to build custom audit functionality.
  • In update operation get value of attributes in plugin which are not changed.
  • To improve performance by avoiding retrieve query to Dynamics CRM.

Msxrmtools.com

How to Create a General Role for a Dynamics CRM 2011 User

Dynamics CRM 2011: Sales Force Automation

What's New in CRM 2011 Session 2: Charts and Dashboards

Email Router Demystified in MS CRM Part -1

Troubleshooting Email Router Configuration Wizard issues

The first part of the Email Router to ensure you don’t run into any issues is around the Email Router Configuration Manager.  If you run into errors within the Email Router Configuration Manager, you will face these same errors when the Email Router is trying to process the email requests.  We need to ensure that the configuration is error free.

Load Data

The first step of ensuring the Email Router is configured properly is to click on Load Data.  When clicking on Load Data, it is going out to the Organization service and retrieving a list of CRM users that have their User Profile configured for E-mail Router for the Incoming/Outgoing E-mail Access Configuration.  If you run into an issue when clicking on Load Data, this indicates an issue accessing the CRM services, either the Discovery or Organization service endpoints.  To troubleshoot this issue, ensure you take the URL provided within the Deployment Profile and append it with the following: URL/XrmServices/2011/Discovery.svc?wsd
URL/XRMServices/2011/Organization.svc?wsdl

This should show an XML response like the following:

If you get any error accessing this URL, then you will need to troubleshoot those errors before the Load Data will ever work.  Since there are a vast number of problems that can come up when accessing this URL, we will not cover any of these. However, knowing that Load Data is directly related to the Deployment Profile should help you

Service Account

If you are able to access the Organization and Discovery services within Internet Explorer and not able to within Load Data, it could be an issue with the account running the Email Router service.  When clicking on Load Data, it is running under the context of the Email Router service account.  This account can be found within Services (Start > Administrative Tools > Services) with the name of Microsoft CRM Email Router.  By default, the account running the Email Router service is Local System.

NOTE: The service account needs to be a Microsoft Dynamics CRM user or it needs to be added to the PrivUserGroup for CRM OnPremise customers.

If you experience an error where the Email Router Configuration Manager is not able to Load Data, try changing the service account running the Microsoft CRM Email Router service to a domain user.  I have seen many situations where the Email Router is not able to connect to CRM due to proxy configuration.  Within these environments, the internet proxy configuration is pushed out to users and not applied to servers/workstations. 

If you still experience issues while clicking on Load Data after changing the service account to a domain user, then look at using a web debugger, called Fiddler, to try and give more information about what actual HTTP errors we are getting.  Again, when clicking on Load Data, we are trying to access a couple of HTTP endpoints from CRM.  In order for Fiddler to log the HTTP requests coming from the Load Data request, the Microsoft CRM Email Router service account must be the logged in user.  Fiddler will only capture HTTP requests of the user that opened the application.  We will cover using Fiddler in a later section but let’s view a situation where Fiddler can be used to help identify an issue with Load Data.

 

Fiddler 

In this scenario, I can see that when clicking on Load Data I am getting a very generic response in my OnPremise environment with Claims and IFD enabled.  When using the Auth endpoint for the Deployment Profile, the error I get from Load Data shows:

However, using Fiddler, I can get a better understanding of where the failure is occurring:

The error shows that it’s coming from ADFS.  This could help me with troubleshooting as I now know to look at the ADFS logs and see what exception is being raised there.

Test Access

After we have successfully loaded the users from the Load Data button, the next step to ensure the Email Router will work properly is to click on a user and then click on Test Access.  So when we click on Test Access, what exactly is the Email Router doing?  There is a lot of speculation of what this is exactly doing so let’s demystify and document what it actually does.

Incoming

  1. The first request being made is out to Exchange using EWS to do a GetFolder request for the default folder providing the user’s Mailbox email address and authenticating as the account within the Incoming Profile.  The response that comes back should be that of the Inbox.
  2. Now, as the account within the Deployment Profile, it logs into CRM and does a series of Organization service requests.  The first one is a WhoAmIRequest.  The WhoAmI response contains the user’s UserId, BusinessUnitId, and OrganizationId
  3. The next CRM request being made is CheckRouterCompatibility.  This is to verify that we are not using an older or newer version of the Email Router that is not supported by that version of Microsoft Dynamics CRM.
  4. Lastly, we do a CheckIncomingEmailRequest and checking if the result comes back with a ReasonCode of ShouldDeliver

 

Outgoing

  1. The first thing that happens is a WhoAmIRequest being made to the CRM server as the account specified in the Deployment Profile.  The WhoAmI response contains the user’s UserId, BusinessUnitId, and OrganizationId
  2. The next request being made is CheckRouterCompatibility.  This is to verify that we are not using an older or newer version of the Email Router that is not supported by that version of Microsoft Dynamics CRM.
  3. Lastly, it does one of the following:
    1. SMTP: The Email Router is doing a HELO request.  The router is looking for a ReplyCode of 220 within the response that comes back from the server.  We can emulate a similar request by using a Telnet as discussed HERE:

       
    2. EWS: The router is doing a GetFolder EWS request for the Default folder.  The response that comes back should be that of the Inbox.

As long as no error is presented during these requests, then you should see Succeeded responses within the Email Router Configuration Wizard.

 

The next part of this post is to cover Troubleshooting Outgoing email issues.

What's New in CRM 2011, Session 2 Intro: Sales Force Automation

Intro to the Sales Force Automation session. Includes foundation topics (teams, queues, sales literature...); working with Charts and Dashboards; working with Goal Metrics and Goals.

Microsoft Dynamics CRM 2015 Features and Functionalities Part 2 of 2 - Support, Upgrade, & Fields

Microsoft Dynamics CRM for the Financial Advisor

Microsoft Dynamics CRM 2011: Configuring Filtered lookups

By configuring filtered lookups in Dynamics CRM 2011, you can ensure that only relevent records appear in lookup dialog windows.
 
 

 

Microsoft Dynamics CRM: Entering a Case

Color Grid for Dynamics 365

Color Form for Dynamics 365