Automerge Failure Process Update

Occasionally, failures will occur when merging due to issues in the platform with both first and third party plugins. The Automerge processes has been re-evaluated to offer better insight should failures occur.

Added Dataverse Search Rule

Added a new dedupe+ rule for online only

Improved Rule Efficiency

A number of improvements were made, particularly around Exact Match and Or rules.

Better Type Checking on Multi Record Merging

Safer type matching was added, particularly when dealing with the currency type.

Duplicare Copilot

Build your Dedupe+ Rules with AI using the new Duplicare Copilot. Create a new Dedupe+ Rule and use natural language to describe what records you want to match, such as “active contacts with the same name and either phone number or email address” and let the AI convert it to a full Dedupe+ Rule ready for you to review.

Work Around “TransactionCurrencyId needs to be supplied to format the money field” error

A recent Microsoft change can trigger this error when updating the value of a money field on the master record during a merge. While hopefully Microsoft will resolve the underlying problem soon, this change works around the error within Duplicare as well.

Added Intersect to Dedupe+ Jobs

Previously intersect could only be used in real time but this feature has now been extended to batch jobs.

Improved Duplicates Detected +

Further fixes around Duplicates Detected + that were affecting using taking very specific steps during the data entry. These steps have been fixed to have the expected behaviour.

Handling Errored Metadata

There are some situations in fields provided by Microsoft out of the box that some fields can miss metadata that should be required. A fix was deployed to handle the few situations where it may not be configured quite as it should be.

API Request Improvements

A number of improvements in decreasing the API requests used by duplicare when performing cross entity matching were added.

Merge Window Composite Updates

Composite fields now break down to their unique field names and render those names accordingly i.e. firstname and lastname instead of fullname.

Improved Duplicates Detected +

When choosing “Ignore and Save” on Duplicates Detected +, a second duplication check was taking place causing the out of the box duplicates detected window to also display. This has now been fixed.

Customizable Behaviour When Merging Custom Tables

When a merge was taking place for a table that isn’t normally mergeable out of the box (account, contact, lead, case) the behaviour of related records can now be amended on a per relationship basis – this is for situations where you don’t want re-parenting to take place.

Improved Address Standardization for businesses

There was a bug preventing businesses being treated as businesses despite having the company name field specified in the address field. This has now been rectified.

Introduced async merging in QuickMerge Dedupe+ Jobs

Dedupe+ Jobs can now optionally be enabled to run async in QuickMerge Dedupe+ Jobs. Each merge that we find will be run as a system job asynchronously so the dedupe+ job itself will finish much quicker whilst the merges are still completing in the system job queue. 

Introduced optional “title” to fuzzy name match dedupe rule

The fuzzy name match rule can now have an optional “title” field added to it, allowing for even better matching as well as removing some false positives.

Introduced optional “Company Name” to standardized address & fuzzy address rule.

An optional company name field has been introduced when created an address field. This name won’t change the standardized result if you have it enabled but does allow the “company name” field to be taken into account during a fuzzy address match to improve matching logic, particularly with a B2B focus.

Introduced an optional “Opt in” field to standardized address.

In address fields if you have standardize enabled then by default we will standardize every address, both historically and in the future. We have an introduced a new option for “opt in” which is a boolean field you can specify. If this boolean is set to true, we will standardize that address otherwise we will leave it as it was. There are occasions, or routes data takes that means sometimes standardizing of addresses requires to be skipped and this new fields allows this to be configured as such.

Fixed issue when combining static value and OR rules

An issue was found when multiple static value rules are combined with an OR in a rule. This has now been rectified in both real time and batch.

Efficiency changes when editing Dedupe+ Rules

The system was performing some unnecessary maintenance when editing dedupe+ rules, particularly those without global search enabled. Whilst this wasn’t an issue, the bug was fixed to save time on the UX.

Fixed tracking of merging when merging tables that don’t have OOTB merge

The “Duplicate Management Dashboard” and respective “Duplicate Report” tables were not tracking when a merge was taking place for a table that isn’t normally mergeable out of the box (account, contact, lead, case).

Server Side Efficiency Changes

A number of changes were made at Data8 server side, where batch procedures are ran so jobs are picked up and ran quicker.

Merging Orders

When merging sales order records, the subordinate order records are now correctly marked as canceled. An “Order Close” activity will be created to indicate that it is lost because it has been merged into another order.

Merging records with missing currency

When merging records that have currency-type values, a warning will now be shown if the selected currency is blank

Duplicates Detected+ and special ribbon buttons

The Duplicates Detected+ window is no longer shown when a record is updated by one of the special ribbon buttons such as Activate, Deactivate, Qualify, Disqualify etc. It will still be shown as normal when the standard Save button is used instead.

Closed Quote/Order/Invoice records

When merging custom entities that have related quote/order/invoice type records, any such records that are in a closed state will automatically be moved to the master record without requiring system administrator privileges.

Merge error diagnostics

If an error occurs during a merge, additional information is now shown alongside the main error message. If the error is triggered by a plugin, the name of that plugin will be shown. The option to download a log file with the full error message and other details is also available.

Merging Quotes

When merging quote records, the subordinate quote records are now correctly marked as lost. A “Quote Close” activity will be created to indicate that it is lost because it has been merged into another quote.

Use Master Merge Rule

The “Use Master” merge rule now has a new option to ignore any blank values. When this option is enabled, additional rules can then be added to perform other merging logic when the master record doesn’t contain a value, e.g. take the most popular value from the other records being merged.

Expression Validation

Additional validation is now applied to expressions being used for the master selection rule and sort/filter expressions in merge rules. This avoids issues that could lead to unexpected values being used during a merge. For example, if you use a sort expression in a Filter & Sort merge rule to prefer taking values from records based on a specific industry, the expected syntax would be something like:

CASE WHEN industrycode = 100001 THEN 1 ELSE 0 END

Common errors we see are when the numeric value is enclosed in quotes, or a text value is entered instead:

CASE WHEN industrycode = ‘100001’ THEN 1 ELSE 0 END
or
CASE WHEN industrycode = ‘Agriculture’ THEN 1 ELSE 0 END

In both these cases, the equality expression would have always evaluated to false and therefore the sort order would not have had the expected result.

Both these two incorrect examples will now trigger an error in the merge rule editor and will need to be corrected to the first correct example before the rule can be saved.

Automerge Updates

When an automerge fails due to the Dataverse configuration, rather than stopping the entire dedupe+ job we will now continue. Any failed updates will be pushed in and the error be shown on screen to allow users to fix and retry. Dedupe+ Jobs will also now finish with a status of “Completed with Errors”.

Assign Duplicate Groups

Duplicate groups can now be reassigned. They can either be reassigned using the standard “Assign” from a subgrid or they can be reassign from the multi record page.

Fixed an issue around lead -> contact merging

A behind the scenes change in the platform introduced a breaking change for some lead to contact merging. This has now been rectified.

NOT rule added

A new “NOT” rule has been added to dedupe+ rules.

Batch Scheduling Update

The batch schedule process has been rewritten to dramatically reduce API usage and become much more efficient.

New Version Alert

The new version alert is now instant and if you update, the alert will disappear straight away. Previously it took around 10 minutes to disappear.

Validate against metadata – Apply to memo field

The new validate against metadata options wasn’t applying to memo fields, this is now rectified

Validate against metadata

A new feature added to merge+ rules allows you to validate the potential merged data against the metadata in Dataverse. Examples such as:

• Invalid option sets will now be removed from calculations
• Sum rules will now limit to max values
• Concat rules will now truncate to max length.

This will default to one for new merge+ rules but will need to be enabled for existing merge+ to ensure backwards compatibility.

API Usage reduction bug fix

A very specific set of criteria existed that could cause the merge window to stop rendering, introduced in 6.2.0. This has now been fixed.

Misc Bug Fixes

A number of smaller bugs were fixed around the global search and PCF controls included.

API Usage dramatically reduced

A refactor of the core parts of the product has reduced the API calls required for general usage of the tool.

Most Popular Across Fields Merge Implementation

After the fix in 6.1.54 for the UI, an issue was uncovered in potential inconsistency in the result calculation depending on record sort order. This has now been rectified and a consistent sort order is now applied.

Domain Match Rule

In some scenarios, the domain match rule did not follow the specified exclusions when finding duplicates – this is now fixed.

Multi Record Merge through the API

An issue was found when calling the legacy multi record merge through the API i.e. via Power Automate. This method is different than the multi record merge action ran during the normal merging UX.

Merge Window Display Name

The merge window was showing the display name specified in Dataverse and was not using any form overridden values if specified – this is now rectified and display name is showing correctly.

Most Popular Across Fields Merge Rule

Fixes a bug when editing this merge rule type that could cause it to not save the value for the “Take the <n>th most popular value” setting.

Controls Update

A number of bug fixes were added across the controls throughout the solution.

Browser Storage changes

Occasionally Duplicare uses storage in a browser – this has been improved to minimize any browser issues.

On-Premise Updates

A number of updates have been made to the on-premise version to bring the performance in line with the online version of the solution.

Controls Update

A number of configuration controls were updated to move away from AutoComplete to dropdown boxes for a better user experience

New Assignment Strategy for Dedupe+ Jobs

Dedupe+ Jobs can now have a team specified as the assignment strategy, meaning you can evenly split merging records evenly across team members

Custom Duplicate ID storage

A specific field can now be specified for “Store Duplicate ID” in Dedupe+ Jobs making for easier analysis of job results. 

Merge+ Rule Editor Fix

Fixed a bug causing headers to not be displayed for some form layouts in the Merge+ Rule editor.

Duplicate Detected PCF Reworked

Designed to be more efficient with API calls and also now works across table/entity

Added Name Matching Option

A number of name matches were added to the fuzzy company name option for better international support

Miscellaneous Bug Fixes

A number of bug fixes are now included

New Scorer

Added new Scoring rule type.

Fixed Merging Permission Issue

When a user has merging permissions via team inheritance, it was incorrectly showing no permission.

Global Search & Search All Records Behaviour

The behaviour across global search and search all records was improved to perform as expected. Additional notes were also added through to ensure proper usage.

Fixed Merge interface bugs

The merging interface will now handle vector icon images properly and will also wrap multi-line text fields.

Merge+ Rule Tweak

The “Merged Status Reason” will no longer be visible when configuring an out of the box merging enabled table.

Added new Dedupe+ Rule editor

The full dedupe+ rule editor got a face lift to look a little more modern.

Added new Merge+ Rule editor

The full merge+ rule editor got a face lift to look a little more modern.

Fixed UCI Issue with Dedupe+ Rules

Occasional transactional issues would occur when creating, updating or deleting dedupe+ rules due to a change in how the unified interface interacts with the data on screen. This has now been amended.

Added “IsGhostRecord” variable for Merge+ Rules

This allows for better rules when using lead to contact matching.

Added columns to Global Search results

The new global search experience has been updated to enable configuration of additional fields to be shown on the results of global search.

NOTE: The legacy global search (web resource named: data8_GlobalSearch.html) is deprecated for online users. All online users should now use the web resource named data8_advancedlookup.index.html

Extended multi field rule

Additional options were exposed in multi field rule to allow a better match set up. You can now 

Allow Blank Match and Allow Non Conflicting Matches in multi field rule, much like you always could for Exact Match Rule.

Fixed a bug with permissions on Duplicate Detection Action

A bug meant specific system set ups could throw an error when searching for duplicates.

Fixed a bug on Duplicates Detected +

A bug meant the merge button would always show, even if users didn’t have merge permissions. The user wouldn’t have been able to persist the merge so it was a poor UX.

Miscellaneous Fixes

Fixed display of merge screen when using forms that contain a “File” type column

Improved naming of automatically scheduled Dedupe+ Jobs

Duplicates Detected+ Release

A brand new duplicates detected+ feature was added into the product. 

Improved International Fuzzy Company Name

A number of improvements were added to the fuzzy company name feature for better international matches.

Geo-Restriction on Address Standardization

The restriction for running address standardization in Azure regions has now been removed.

Handle status bug upon merge

Occasionally a record could be have been left as inactive incorrectly. This has now been fixed.

Better handle UTC issues on merge

Occasionally an issue occurred with dates when users spanning multiple time zones were merged.

Miscellaneous bug fixes and performance improvements

A wide range of minor bug fixes and performance improvements were added.

Lookup+

Lookup+ was added to the product. 

Duplicare SDK Client

A C# NuGet package was introduced.

Application Insights

Adding some basic application insights anonymous usage tracking. Also added the ability to enable advanced logging in the event of requiring assistance from Data8.

Minor UI Tweaks

A number of error messages have been improved to be more accurate and the user interface in the config area has been updated slightly.

Merge Permissions Fixed

The logic to calculate merging permissions has been updated.

Improved Dedupe+ Job Statistics

The statistics generated for a Dedupe+ Job now correctly take into account any records that have been ignored due to previously-created exclusion rules.

Improved Merge+ Rule editing

When selecting an existing “Use most common value across a set of fields” rule, the previously selected fields that the rule will use are now highlighted.

Dedupe+ Rule Creation Fix

Resolves a “Business Process Error: Object reference not set to an instance of an object” error when creating a new Dedupe+ Rule.

Data Imports

Enhances duplicate detection during data imports. When data is imported that includes a lookup field that is important to duplicate detection, this may have led to the duplicates not being correctly identified in earlier versions. This version fixes this and will leave the duplicate record in a “Partially Imported” state within the import job. To trigger the solution to re-synchronise existing rules with the new functionality, you need to make a change to the rule definition (e.g. add a new “exact match” rule, save it, then remove and save it again)

Advanced Find

Displays the correct merge button on the Advanced Find window ribbon (requires deactivating and reactivating the Merge+ Rule and publishing customisations)

Merging Fixes

Fixes issues with required field handling in the merge screen and fixes issues merging multi-select picklist records

Canadian Address Improvements

The calculations for using fuzzy matching on addresses can vary by country and a number of improvements have been made for canadian addresses based on official guidance.

Server side efficiency changes for batch jobs

Some efficiency changes where made on the server side, allowed for faster batch dedupe+ jobs

Updated latest version checker

The configuration page has a latest version checker and a bug was identified during the move to a 3 number based version number as of 4.0.0. This has now been fixed.

Fixed bug with empty multi select picklists

Minor bug fix which prevented records with a blank multi select picklist being merged.

Azure data processing

Option to choose where your data is processed from a selection of Azure regions

Added new PCF component

New PCF custom control added for quicker feedback on duplicates, speeding up the duplicate detection process even quicker.

Extended duplicate detection to optional search across the whole system

Optionally use the system organisation to search for duplicates across the whole system, regardless of permissions. The duplicates returned will not share data it shouldn’t and will just give the count of duplicates.

Fuzzy Name Matching Improvements

Improvements were made to the fuzzy name, including an updated blacklist of names.

Bug Fixes & Performance Updates

A number of bugs fix and a number performance updates were added throughout, particularly in the web resources used to manage the duplicare specific rules.

Versioning

Versioning numbers where brought back to be inline, regardless of premise or online. Also moved to a 3 digit version number rather than a 4.