Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

"Real Time" data change detection in SQL Server

We have a requirement for notifying external systems of changes in data in various tables in a SQL Server database. The choice of which data to monitor is somewhat under the control of the user (gets to choose from a list of what we support). The recipients of the notifications may be on a locally connected network (i.e., in the same data center) or they may be remote.

We currently handle this by application code within our data access layer that detects changes and queues notifications on a Service Broker queue which is monitored by a Windows service that performs the actual notification. Not quite real time but close enough.

This has proven to have some maintenance problems so we are looking at using one of the change detection mechanisms that are built into SQL Server. Unfortunately none of the ones I have looked at (I think I looked at them all) seem to fit very well:

Change Data Capture and Change Tracking: Major problem is that they require polling the captured information to determine changes that are to be passed on to recipients. I suspect that will introduce too much overhead.

Notification Services: Essentially uses SQL Server as a web server, which is a horrible waste of licenses. It also requires access through at least two firewalls in the network, which is unacceptable from a security perspective.

Query Notification: Seems the most likely candidate but does not seem to lend itself particularly well to dynamically choosing the data elements to watch. The need to re-register the query after each notification is sent means that we would keep SQL Server busy with managing the registrations

Event Notification: Designed to notify on database or instance level events, not really applicable to data change detection.

About the best idea I have come up with is to use CDC and put insert triggers on the change data tables. The triggers would queue something to a Service Broker queue that would be handled by some other code to perform the notifications. This is essentially what we do now except using a SQL Server feature to do the change detection. I'm not even sure that you can add triggers to those tables but I thought I'd get feedback before spending a lot of time with a POC.

That seems like an awful roundabout way to get the job done. Is there something I have missed that will make the job easier or have I misinterpreted one of these features?

Thanks and I apologize for the length of this question.

like image 887
Some Guy in Seattle Avatar asked Jul 13 '15 17:07

Some Guy in Seattle


People also ask

How do you track data changes in SQL?

Right click on the table you want to track changes. Click Properties, click Change Tracking, then in the right pane set Change Tracking to TRUE.

What is DMF in SQL Server?

Dynamic management views and functions return server state information that can be used to monitor the health of a server instance, diagnose problems, and tune performance. Important. Dynamic management views and functions return internal, implementation-specific state data.

How SQL is used in real time?

SQL simplifies real-time analytics by encapsulating the underlying complexity of the data management operations. SQL queries can be optimized automatically over distributed systems for significantly (100X) better performance and open source frameworks such as Storm.


1 Answers

Why don't you use update and insert triggers? A trigger can execute clr code, which is explained enter link description here

like image 149
Luc Avatar answered Nov 09 '22 09:11

Luc