Recently we observed that post PVA and Omnichannel implementation, there was sudden exponential increase in the storage consumption and we were almost close to go over capacity risk. This was the triggering point to check if there is any tool or solution available to archive the data in some economical storage like Azure data lake instead of paying too much for data verse storage.
When we checked with Microsoft about this issue, Microsoft suggested the latest feature of Long term data retention which is in preview at the moment. We all know about Data Export Service which was Microsoft’s first data archival solution did not fly well as once we archive the data we were not able to view it from dynamics. We had to put lot of efforts like creating virtual entities and connecting those to these tables residing in Azure sql server, creating forms to view the data ,still it was not that that effective and did not sell through. We were not able to satisfy the customers with this solution. Having this background, I was bit skeptical about this new data retention solution as well and hence started exploring more about this.
Let us first understand that this feature is in preview state at the moment, hence it may change in future and we should not use this in production. Exploring this solution and submitting our feedback to Microsoft would help them to understand the customer expectations and accordingly improve the solution.
Basically this long term data retention feature enables the administrators to select the data which fulfills below needs and archive that data to Dataverse data lake. Yes you read it right😊, its Dataverse hosted data lake and not a open Azure Datalake we can can operate in our Azure subscription
- No live transaction is needed for this data
- Data is to be retained only for reference or for audit compliance purpose
Once we identify the data which is to be archived, we can set the data retention rules and the schedule the archival. This would then automatically start moving this data from data verse database to data lake. Most important provision of this feature is once this data is archived, we will be able to access this data from the dynamics usual interface. In existing filters, it would give option to select the active data or retained data. We won’t be able to do any transaction on this data. There are other restrictions as well on this data which will see in later part of this post .
As we have understood the solution at high level, let us actually see how to use this step by step
Selecting the data to be retained
Before we start creating the data retention policies, we need to identify the tables. Let us go to tables and select the table
Note - Currently, audit, elastic tables, and images aren't supported for long term retention
From the extended menu, select properties
Select Advanced Options
As we scroll down, we can see a property called “Enable Long term retention”. we have select that
Note : The checkbox for “Enable Long Term Retention” gets enables after few second when we open the advanced option. So if do not see it enabled to select, wait for few seconds . One more point to be noted is we can only select the root level tables for retentions for example Contact, Account, Conversations etc which are parent entities and automatically all child entities will be enabled for data retention.
After enabling the tables for data retention, we also need to create the view to filter the records which are to be retained. We need to select these views while creating the retention policy.
Let us see how to create retention policies in my next post
Vrushali's Techno Blog 