MS AD to OpenLDAP sync tooling

This tool is used to synchronize groups and users between MS AD and OpenLDAP. In general it is run via a cron job, but can also be used on an ad-hoc basis.

Release process

Make a new branch and merge your changes to it
- Make sure your changes in your feature branch are checked in
- make test Make sure all tests pass (you might need -i if you have outstanding test failures)
- In gitlab, create a merge request from your feature branch to main
  - Make sure you squash commits to reduce the amount of clutter in the comments on main
- Create your new release branch from main
  - git branch to make sure you are on "main"
  - git checkout main to swap to main if you are not there - git pull to ensure you pull your earlier merge from the feature branch
  - git checkout -b v1.0 to create and switch to the new feature branch
  - git push --set-upstream origin v1.0 to push your new release branch to the git server
Deploy your release to the production server
- ssh server.name to log in to the LDAP master
- sudo -u service_user -i to become the service account user
- Disable the existing sync scripts
  - crontab -e and comment out (with i and Esc) with a # at the start of each line, both cron jobs (one for user_sync, and one for group_sync)
  - :wq to save and quit the cron editor
- Update the git repo to your new release version
  - cd ad-to-ldap-sync/ to change to the git repo
  - git pull to get the new branch names
  - git checkout v1.0 to change to the new release branch
  - You are now using the new feature branch
- Update config file if necessary to include and new config options
  - diff -i --side-by-side --suppress-common-lines config/config-template.yaml config/config.yaml | grep -Ev 'bind_user|bind_pass|ca_certs_file'
- Make sure that python has all the requirements installed
  - pip install -r requirements.txt
- Update the commented out cron lines to take account of any new command line options
- Run a manual test of user_sync and group_sync in noop mode to verify they run as expected. Look for unexpected errors
  - /home/service_user/.pyenv/versions/3.10.1/bin/python /home/service_user/ad-to-ldap-sync/ad_to_ldap_sync.py group_sync --config_file /home/service_user/ad-to-ldap-sync/config/config.yaml --console error for group_sync
  - /home/service_user/.pyenv/versions/3.10.1/bin/python /home/service_user/ad-to-ldap-sync/ad_to_ldap_sync.py user_sync --config_file /home/service_user/ad-to-ldap-sync/config/config.yaml --console error for user_sync
  - Check the monitoring log files both have "True" in them
    - for i in *monitoring.log; do echo $(cat $i); done
  - Look for errors or warnings in either of the script log files
    - less group_sync_ad-to-ldap-sync.log for group sync
    - less user_sync_ad-to-ldap-sync.log for user sync
- Reinstate the cron job so that the scripts will run automatically
  - crontab -e and remove the two commented out (with i and Esc) lines by removing # from the start (one for user_sync, and one for group_sync)
  - :wq to save and quit the cron editor
- Release is completed. Bask in the glorious admiration of your colleagues.

Configuration details

Auto-generated API documentation

API documentation

Metrics

Development

Use NumPy/SciPy Docstrings
Use Sphinx

Breakpoints

At any point in the code you can add breakpoint() to add a breakpoint. In addition to that, set the environment variable $PYTHONBREAKPOINT = 'web_pdb.set_trace'. This will then launch Pdb on port 5555 (web-pdb required). Use your browser of preference to open it and step through.

Sphinx setup

Create the basic configuration:

mkdir api_docs && pushd api_docs
sphinx-quickstart \
--no-sep \
--project "MS AD <-> OpenLDAP sync" \
--author "Peter & Martinus" \
--ext-autodoc \
--ext-coverage \
--extensions sphinx.ext.napoleon \
--makefile \
--no-batchfile \
--language en \
--release "v1"

sphinx-apidoc -o . ../src
popd

Install the autodocsumm module:

pip install autodocsumm

Set the path correctly:

sed -i '1 i\
import os\
import sys\
\
sys.path.insert(0, os.path.abspath(".."))' api_docs/conf.py

Set the autodoc options

sed -i '/Options for HTML output/i\
autodoc_default_options = {\
    "autosummary": True,\
    "members": True,\
    "undoc-members": True,\
    "private-members": True,\
}' api_docs/conf.py

Set the theme as the default is ....

sed -i 's/alabaster/classic/' api_docs/conf.py

Set the side bar and body width:

echo 'html_theme_options = {"sidebarwidth": 400, "body_max_width": "none"}' >> api_docs/conf.py

Ignore the modules.rst file

sed -i '1 i\:orphan:\' api_docs/modules.rst

Docstring coverage

pip install docstr-coverage

Profiling

If you want to see where things are slow:

python -m cProfile -o output.pstats ad_to_ldap_sync.py --conf config/config.yaml --exc config/exceptions.yaml --cons debug --count config/country_control.yaml

And to then graph it use gprof2dot:

gprof2dot -f pstats output.pstats | dot -Tpng -o output.png

Creating a compiled executeable

In order to easily share the program over a wide set of systems and not worry about either containers or Python versions and module, you can compile the entire program into a single binary.

For further information see PyInstaller

NOTE: As of this writing, UPX is not used on Linux.

Ensure you have shared libraries

If you use pyenv:

env PYTHON_CONFIGURE_OPTS="--enable-shared" pyenv install 3.10.1

Create the binary

pyinstaller --clean -F ad-to-ldap-sync.py

The binary will be located with the same name in the dist/ directory.

TODO

New feature(s)

Email user with link to Password reset page (in check_change) (Future)
Cope with large LDAP queries, and pagination (Future)
If the AD object in a group is itself a group, recursively descend and build full member list. (soonish)
Set recursion limit based on len(called_groups) and config file in _flatten_nested_group
Investigate below 'md4' as it is a broken algorythm: https://cwe.mitre.org/data/definitions/327.html
Need integration testing for adding multiple users at once.
Duplicate duplicate duplicate code between user and group sync
Clean up unit tests so they don't leave log files behind.
Ensure 100% code coverage for unit tests.
Create integration tests for user_sync.

Testing

We would benefit from AD "test users" instead of named ones. This would make testing cleaner and safer.
Configure OpenLDAP Docker container for integration tests.
Run through all methods and compare with unit-tests, verify we do all the things

graphcore / ad-to-ldap-sync