All records eventually deleted by syncrepl when running in REFRESH_ONLY

[matthewh]

I noticed an oddity with syncrepl_client when using REFRESH_ONLY mode. This issue does not present when I use REFRESH_AND_PERSIST so I'll admit I may not understand how the LDAP Synchronize spec is supposed to behave with REFRESH_ONLY.

After running my script several times over a span of 10 minutes, an entry gets flagged as deleted by syncrepl_client even though it still exists on the server. That entry will not be added again by the snycrepl_client until I modify it on the server. The cookie value on the execution that triggers the delete matches the cookie value from the previous execution so I'm perplexed as to why
snycrepl_client think it needs to delete something? Is this the expected behavior of REFRESH_ONLY?

I'm testing with FreeIPA, Directory 389, running inside docker (https://hub.docker.com/r/freeipa/freeipa-server/).

LDAP_SCOPE = SUBTREE
LDAP_FILTER_STR = "(objectclass=organizationalperson)"
LDAP_DN = "cn=users,cn=accounts,dc=example,dc=com"

Thanks

[A. Karl Kornel]

Hmmmm, that's going to be a tough one.

I guess the first question is, are you using LDAPS or STARTTLS, and what type of bind is this? If you are not using LDAPS or STARTTLS, and you are not using GSSAPI bind, then your LDAP traffic should be going in the clear, which means you can capture it.

The question that has to be answered is, how is the deletion being communicated to the client? There is either an explicit deletion from the server (which would make a direct call to syncrepl_client.syncrepl_delete), or the deletion is being relayed to us through a call to syncrepl_client.syncrepl_present). The latter case is more likely, but the "present" call has so many meanings, it's possible for a record to be deleted simply because it was wasn't mentioned by the server.

You can gather more info by putting some print lines at the start of syncrepl_client.syncrepl_present (syncrepl_client/init.py line 1130) recording the uuids list (which may be empty), and the value of refreshDeletes. There should be at least one present message from the LDAP server at the end of the refresh, and there may be additional present messages interspersed through the rest of the refresh.

So, please have a look at the present messages coming in, and see what you get!

P.S. What ldap client are you using? pyldap or python-ldap?

[matthewh]

I'm using pyldap 2.4.37 and unencrypted traffic to port 389 while debugging. Thanks for pointing me to those function calls - I'll some debugging and watch the traffic.

pip freeze

Jinja2==2.9.6
MarkupSafe==1.0
pyasn1==0.2.3
pyldap==2.4.37
PyYAML==3.11
simple-settings==0.12.0
six==1.11.0
structlog==17.2.0

[A. Karl Kornel]

Hi @matthewh, good evening!

Since this issue has sat for a while, do you have any problems with it being closed?

Please let me know!

[matthewh]

Give me 24 more hours and if I haven't found the time to work on it we can close it.

[matthewh]

Directory 389 does the following in REFRESH:

1. On the first run, add all records.
2. On the second run, delete all records.
   There are no uuids and refreshDeletes is False.

   syncrepl_present None False
   

[Michael Brown]

I have also observed this problem. The root cause is that in refreshOnly mode, 389 directory server will unconditionally send the syncDoneControl with refreshDeletes omitted (i.e. false); this indicates to the client that any entries not explicitly mentioned in the search results (or in a syncIdSet) should be deleted.

When the client has provided an initial sync cookie, 389 directory server will send any modified entries plus syncIdSets for any deleted entries. The server should therefore send a syncDoneControl with refreshDeletes set to true, indicating that it has provided explicit lists of deleted entries.

A conformant client will therefore respond exactly as you have observed: on the second run in refreshOnly mode, all entries will be deleted.