AD Change Detection Component and Preloading Delta db.

[bill valvik]

I need to detect changes in AD from last change. since the Delta DB does not have the pre changes version until it loads it once, how can I preload the Delta DB with everything in AD so it will properly then detect changes from last detection?

Some background: I have setup an iterator pointing to the LDAP and set the Delta DB in that Delta tab. Set it to collect all the attributes and let it run to the end.

Went back to my chand detection and it is still reporting all attributes as being changed for every entry that it detects.

Thank you,

Bill Valvik

[Eddie Hartman]

Post by bill valvik
I need to detect changes in AD from last change. since the Delta DB does not have the pre changes version until it loads it once, how can I preload the Delta DB with everything in AD so it will properly then detect changes from last detection?
Some background: I have setup an iterator pointing to the LDAP and set the Delta DB in that Delta tab. Set it to collect all the attributes and let it run to the end.
Went back to my chand detection and it is still reporting all attributes as being changed for every entry that it detects.
Thank you,
Bill Valvik

The easiest way is to set up an Iteration using Delta and read through the entire directory, Bill. You were thinking easier than that?

-Eddie :)

[yn2000]

Hi Bill,
I wonder I read the question correctly.
You want to set up a Delta on AD Change Detection???
I believe Delta is for normal LDAP iterator read; while probably you need a 'persistent store' when you deal with AD Change Detection.
If you have the authority to choose how you read, please try solve your issue using Delta and a normal LDAP iterator read and avoid using AD Change Detection. I hate using AD Change Detection, because, depending on your AD environment and your solution, AD may send more than 80% of their transactions on something that have nothing to do with your solution. What worse, sometime AD is sending create and update transactions for the same user so close to another. In addition, most of AD replicate their DCs once an hour. It means that running the AL every hour is still a viable solution... well, most of the time.
Rgds. YN.

[Eddie Hartman]

Post by yn2000
Hi Bill,
I wonder I read the question correctly.
You want to set up a Delta on AD Change Detection???
I believe Delta is for normal LDAP iterator read; while probably you need a 'persistent store' when you deal with AD Change Detection.
If you have the authority to choose how you read, please try solve your issue using Delta and a normal LDAP iterator read and avoid using AD Change Detection. I hate using AD Change Detection, because, depending on your AD environment and your solution, AD may send more than 80% of their transactions on something that have nothing to do with your solution. What worse, sometime AD is sending create and update transactions for the same user so close to another. In addition, most of AD replicate their DCs once an hour. It means that running the AL every hour is still a viable solution... well, most of the time.
Rgds. YN.

Thanks for those insights, YN! My experience with the AD CDC is minimal, but I have struggled with the rapid-fire create + modify change combo that results in me only picking up the modify change. Also, I've had difficulties getting access to CN=Deleted Objects on some DCs, again since my exposure to AD is sporadically little.

If anyone else has lessons-learned to share, please enlighten us!

Always learning (and sometimes remembering),
-Eddie

[bill valvik]

YN, thank you for the reply. To update my scenario, I do have a persistent store, and have iterated the entire AD to capture the changes. I agree with the oddity of AD in that when a change is made all the attributes are reflected as (replace) thus it is near impossible to know what exactly has changed. Fortunately I am looking for container changes on user accounts and I then compare the distinguishedname attribute with what I have in ISIM to determine if it was just moved of modified within the container. It satisfies my immediate needs however it would be nice to get a true response from the Delta store on what is really being changed. So far I cannot confirm which attribute is detected under .getOperation() method for each attribute. Everything comes back as modified.

If anyone else has any details on AD CDC I would like to hear about it.

Thank you all,

[yn2000]

While waiting for others to chime in...
In my opinion, your situation/requirement is exactly the reason why LDAP (Iterator+Delta) option provides a better solution than AD CDC option.
In LDAP (Iterator+Delta) option: In the first run, the Delta Store will contain all of the data 'that matters' from a specific AD container. In the second run, you will know exactly 'what was changed' up to the attribute level, because TDI will compare 'all' of the new data from that AD container with the ones stored in the Delta Store.
In AD CDC option, you should not turn on Delta, because AD CDC is already sending 'what was changed'. For example: uSNChanged#1: is changing givenName value, then uSNChanged#2: is changing sn value. When processing uSNChanged#2, the TDI Delta will interpret as add a new sn operation, because it was not existed in uSNChanged#1. (Well, not exactly like that but that is the simplified version of it)
That is why I asked, why you want to turn on TDI Delta to detect 'what was changed' on the data that is already sending 'what was changed'?
I believe, the problem in your situation is the fact that AD is sending (read: recording) transaction data that does not really matter to you, so that your data source from AD CDC seems to send a false (read: not changed) information.
So, I don't think TDI Delta is the appropriate use of feature to filter out 'bad' transactions from AD. In my opinion, custom filtering/scripting would do much better for your situation. But, then why choose a custom scripting if you have a better option?
Rgds. YN.

[bill valvik]

Post by yn2000
While waiting for others to chime in...
In my opinion, your situation/requirement is exactly the reason why LDAP (Iterator+Delta) option provides a better solution than AD CDC option.
In LDAP (Iterator+Delta) option: In the first run, the Delta Store will contain all of the data 'that matters' from a specific AD container. In the second run, you will know exactly 'what was changed' up to the attribute level, because TDI will compare 'all' of the new data from that AD container with the ones stored in the Delta Store.
In AD CDC option, you should not turn on Delta, because AD CDC is already sending 'what was changed'. For example: uSNChanged#1: is changing givenName value, then uSNChanged#2: is changing sn value. When processing uSNChanged#2, the TDI Delta will interpret as add a new sn operation, because it was not existed in uSNChanged#1. (Well, not exactly like that but that is the simplified version of it)
That is why I asked, why you want to turn on TDI Delta to detect 'what was changed' on the data that is already sending 'what was changed'?
I believe, the problem in your situation is the fact that AD is sending (read: recording) transaction data that does not really matter to you, so that your data source from AD CDC seems to send a false (read: not changed) information.
So, I don't think TDI Delta is the appropriate use of feature to filter out 'bad' transactions from AD. In my opinion, custom filtering/scripting would do much better for your situation. But, then why choose a custom scripting if you have a better option?
Rgds. YN.

I am beginning to realize that the delta is not giving me any benefit on this situation. I would be nice for the AD to cooperate and only transfer true deltas when a change is made.
Thank you for the input, I do need the CDC for the imediate response requirements, however the delta is being turned off.

[yn2000]

Here is my analysis on AD a decade ago, so I am not sure it is true anymore.

As you know that AD contains 'everything', so to speak. Then, there are many AD DC (Domain Controller) in the environment, most likely. AD DC replicate each other every hour (configurable). Unfortunately, we can only tap into one of the DC and this DC received update from many other DCs. And on top of that each DC maintain each own uSNChanged value.

uSNChanged value (the one that is usually used in the AD CDC persistent store) would changed if anything changed in the AD. Therefore, technically, AD is really sending a true delta, like you wish, but 80% (very conservative number) has nothing to do with the one (user data) that you need.

Now, talking about "...I do need the CDC for the immediate response requirements..."

That is a bogus requirement. Yeah, yeah, the customer say so, but in our world customer is not always right. I did mentioned that "AD DC replicate each other every hour (configurable)", right? So, technically, AD DC itself is violating the requirement to get immediate response from other DC.
How could they asked you to provide an immediate response if the source that you read could be 1 hour late?

So, the argument that I usually use is by asking 'how long does it take for AD DC to replicate?' then I configure TDI to schedule iterator at the same time period.

And last... treat yourself a beer if the customer agree to it, because this decision will safe you (and your customer) from future maintenance nightmare.

:-) Rgds. YN.

[bill valvik]

Post by yn2000
Here is my analysis on AD a decade ago, so I am not sure it is true anymore.
As you know that AD contains 'everything', so to speak. Then, there are many AD DC (Domain Controller) in the environment, most likely. AD DC replicate each other every hour (configurable). Unfortunately, we can only tap into one of the DC and this DC received update from many other DCs. And on top of that each DC maintain each own uSNChanged value.
uSNChanged value (the one that is usually used in the AD CDC persistent store) would changed if anything changed in the AD. Therefore, technically, AD is really sending a true delta, like you wish, but 80% (very conservative number) has nothing to do with the one (user data) that you need.
Now, talking about "...I do need the CDC for the immediate response requirements..."
That is a bogus requirement. Yeah, yeah, the customer say so, but in our world customer is not always right. I did mentioned that "AD DC replicate each other every hour (configurable)", right? So, technically, AD DC itself is violating the requirement to get immediate response from other DC.
How could they asked you to provide an immediate response if the source that you read could be 1 hour late?
So, the argument that I usually use is by asking 'how long does it take for AD DC to replicate?' then I configure TDI to schedule iterator at the same time period.
And last... treat yourself a beer if the customer agree to it, because this decision will safe you (and your customer) from future maintenance nightmare.
:-) Rgds. YN.

So, to explain my requirements for immediate updates, I need to know when I user account contains the value ou=Admin Lockout in it distinguished name so I can trigger an complete suspend of the person and all other accounts managed in ISIM.

This actually works immediately no matter on the AD DC which is updating the distinguished name I had one of our AD admins move a test user on another DC back and forth and The DC I was following picked up the changes immediately.

So, for now all is working just fine.

Thank you,

[yn2000]

OK... not to persuade you to change your solution, this is just extended discussion, in case that I am (or others) in your shoe.
How many entries in the ou=Admin Lockout? Hundreds? Thousands? LDAP (Iterator+Delta) can swoop those entries in minutes, right?
So, I would still choose to use LDAP (Iterator+Delta) to run every 10 minutes.
Imagine this... one day a manager asked, why this person is suspended, when he/she is not in ou=Admin Lockout? With AD CDC approach, you will have a challenge to provide forensic prove that something might screwed up in the uSNChanged sequencing (or maybe just system down/error issue)... aka 'not your fault'; plus you will have additional challenge to recover from the issue, because the source (the event/transaction) is not there anymore or not easy to backtrack to the uSNChanged that has issue. With LDAP (Iterator+Delta) approach, the source will always be there and you can easily delete the Delta and re-run the same AL to fix any data sync issue... just saying...
Rgds. YN.