OpenLDAP has been replacing proprietary directory server offerings in the private sector, public sector and the financial sector at an increasing pace. This is largely due to its performance and scalability, dynamic configuration capabilities and flexible extensibility via bundled modules. OpenLDAP would not have earned its place in these sectors without enterprise grade replication options. In this talk, an overview of the latest production ready OpenLDAP 2.4 replication features will be discussed and the numerous best practice strategies will be presented covering the most common deployment configurations found in the wild.
Text of OpenLDAP Replication Strategies
1.OpenLDAP Replication StrategiesGavin Henry Managing Director, Suretec Systems Ltd.- firstname.lastname@example.org Documentation Developer, OpenLDAP Project - email@example.com
2. AgendaOverview The History of Replication Replication Technology Deployment Alternatives Replication Best Practices Questions and Answers http://creativecommons.org/licenses/by/3.0/ 02/04/09 2 3. Overview OpenLDAP is an open source code project Founded 1998 Three core team members A dozen or so contributors (engineering team) Feature releases every 12-18 months Maintenance releases roughly monthly 4. OverviewReplicated directories are a fundamental requirement for delivering a resilient enterprise deployment.Slurpd is now completely removed from 2.4 There's new terminology Provider/Consumer MirrorMode and Multi-Master now available Replication needs to support complex environments Wonderful things can be done with the Dynamic Configuration Backend http://creativecommons.org/licenses/by/3.0/ 02/04/09 4 5. The History of ReplicationSlurpd was the first form of replication Slurpd was a standalone daemon plagued with problems (in brief):slurpd never rerouted requests It was not reliable It was extremely sensitive to the ordering of records inthe replogmore.... http://creativecommons.org/licenses/by/3.0/ 02/04/095 6. The History of ReplicationIt could easily go out of sync, at which point manualintervention was requiredIt wasn't very tolerant of unavailable servers. It only worked in push mode It required stopping and restarting the master to add newslavesIt only supported single master replication Slurpd is no longer part of OpenLDAP 2.4 http://creativecommons.org/licenses/by/3.0/ 02/04/09 6 7. The History of ReplicationSyncrepl has none of those weaknesses Syncrepl was born on April Fools Day 2003 and is documented in the Admin Guide and RFC4533 - LDAP Content SynchronizationOperationIt is extremely flexible and JGOWI Push Based, Pull based, Proxies..... http://creativecommons.org/licenses/by/3.0/ 02/04/09 7 8. The History of ReplicationOpenLDAP 2.4 adds: MirrorMode (Active-Active Hot-standby) N-Way Multimaster Replication More sophisticated Syncrepl configurations Replicating slapd Configuration (syncrepl and cn=config)http://creativecommons.org/licenses/by/3.0/ 02/04/098 9. Deployment AlternativesSyncrepl Delta-syncrepl N-Way Multi-Master MirrorMode Syncrepl Proxy Mode http://creativecommons.org/licenses/by/3.0/ 02/04/099 10. LDAP Content Synchronization Protocol (LDAP Sync)RFC 4533 - LDAP Content Synchronization Operation - June 2006refreshOnly and refreshAndPersist Eventually-convergent content synchronization Client can be notified that a complete reload is needed (used in Delta-syncrepl)Copy of DIT Fragment http://creativecommons.org/licenses/by/3.0/ 02/04/09 10 11. LDAP Content Synchronization Protocol (LDAP Sync)Inconsistencies resolved on subsequent syncs Not good for bandwidth challenged apps/deploymentsNot for use with apps that need transactional data consistencyExample of refreshOnly and refreshAndPresist to followhttp://creativecommons.org/licenses/by/3.0/ 02/04/09 11 12. http://creativecommons.org/licenses/by/3.0/ 02/04/09 12 13. http://creativecommons.org/licenses/by/3.0/ 02/04/09 13 14. LDAP Sync Replication (Syncrepl)LDAP Sync Replication engine, syncrepl for short Consumer-side replication engine Resides at the consumer and executes as one of the slapd(8) threads.Uses the LDAP Content Synchronization protocol (or LDAP Sync for short) - RFC4533LDAP Sync provides a stateful replication which supports both pull-based and push-based synchronization and doesnot mandate the use of a history store. http://creativecommons.org/licenses/by/3.0/ 02/04/0914 15. LDAP Sync Replication (Syncrepl)Pull-based replication - periodically polls the provider for updates.Push-based replication - consumer listens for updates that are sent in realtimeSyncrepl tracks status of the replication content by maintaining and exchanging synchronization cookiesConsumer replica can be constructed from a consumer-side or a provider-side backup at any synchronization status.Syncrepl can automatically resynchronize the consumer replica up-to-date with the current provider content.http://creativecommons.org/licenses/by/3.0/ 02/04/0915 16. LDAP Sync Replication (Syncrepl)Session log can be used in the provider which stores the entryUUIDs of a finite number of entries deleted from adatabase in order to use the delete phaseLDAP Sync provider maintains a contextCSN in the suffix entry (change sequence number = CSN)It is the largest entryCSN in the provider context (depending on outstanding transactions)contextCSN maintained primarily in memory and written at shutdown, but can be checkpointed.http://creativecommons.org/licenses/by/3.0/ 02/04/09 16 17. LDAP Sync Replication (Syncrepl)Whole database scanned if contextCSN not found and new one generatedConsumer also stores its replica state, which is the provider's contextCSN received as a synchronizationcookieNew Consumer doesn't change provider config No provider restarts needed Consumer replication can stop without the need for provider-side changes and restart. http://creativecommons.org/licenses/by/3.0/ 02/04/0917 18. Delta-syncreplDisadvantages of LDAP Sync LDAP Sync replication is an object-based replicationBoth the changed and unchanged attribute values are processedExcess traffic generated for small changes changelog-based variant of syncrepl http://creativecommons.org/licenses/by/3.0/ 02/04/0918 19. Delta-syncreplMaintains a changelog on the provider Consumer checks the changelog for the changes it needsIf a replica is too far out of sync, switches to conventional syncreplSwitches back to the delta-syncrepl mode when fully sync'd http://creativecommons.org/licenses/by/3.0/ 02/04/0919 20. http://creativecommons.org/licenses/by/3.0/ 02/04/09 20 21. N-Way Multi-MasterUses Syncrepl to replicate data to multiple provider (quot;Masterquot;) Directory servers.Avoids a single point of failure Supports complex topologies Providers can be located in several physical sites Good for Automatic failover/High Availability Requires synchronised time source - ntp http://creativecommons.org/licenses/by/3.0/ 02/04/0921 22. N-Way Multi-MasterIt has NOTHING to do with load balancing Providers must propagate writes to all the other servers Network traffic and write load spreads across all of the servers the same as for single-master.Server utilization and performance are at best identical for Multi-Master and Single-Master replicationSingle-Master is superior because indexing can be tuned differently to optimize for the different usage patternsbetween the provider and the consumers.http://creativecommons.org/licenses/by/3.0/ 02/04/09 22 23. N-Way Multi-MasterBreaks the data consistency guarantees of the directory modelCrashed, or network link has failed??? More: http://www.watersprings.org/pub/id/draft-zeilenga-ldup-harmful-02.txt http://www.openldap.org/faq/data/cache/1240.html http://creativecommons.org/licenses/by/3.0/ 02/04/09 23 24. http://creativecommons.org/licenses/by/3.0/ 02/04/09 24 25. MirrorModeIs a Active-Active Hot-Standby solution External front end needed Not a Multi-Master solution Syncrepl also allows the provider nodes to re- synchronize after any downtimeDelta-Syncrepl is not yet supported http://creativecommons.org/licenses/by/3.0/ 02/04/09 25 26. MirrorModeTwo providers are set up to replicate from each otherAn external frontend is employed to direct all writes to only one of the two servers.The second provider will only be used for writes if the first provider crashesautomatically catch up to any changes on the running provider and resync.http://creativecommons.org/licenses/by/3.0/ 02/04/09 26 27. MirrorModehttp://creativecommons.org/licenses/by/3.0/ 02/04/09 27 28. Syncrepl Proxy ModerefreshAndPersist must still be initiated from theconsumerFirewalls may need provider initiated push-modereplicationslapd-ldap proxy is set up near or with the provider thatpoints to the consumer syncrepl engine runs on the proxy and points to providerhttp://creativecommons.org/licenses/by/3.0/ 02/04/0928 29. Syncrepl Proxy Modehttp://creativecommons.org/licenses/by/3.0/ 02/04/09 29 30. Replication Best PracticesUse a replication user You can restart replication by adding/removing syncrepl statement if needed when changingreplication credentials/certificatesMirrorMode always good enough for most enterprises dependent on global requirementsSlapo-chain to chain writes back to provider http://creativecommons.org/licenses/by/3.0/ 02/04/09 30 31. Replication Best Practices[ghenry@suretec~]$ldapsearchxLLLHldap[?]://master:389sbaseb'dc=suretecsystems,dc=com'contextCSN[?]dn:dc=suretecsystems,dc=comcontextCSN:20081025222436.822813Z&000000;000#000000 [ghenry@suretec~]$ldapsearchxLLLHldap://slave:389sbaseb'dc=suretecsystems,dc=com'contextCSNdn:dc=suretecsystems,dc=comcontextCSN:20081025222436.822813Z&000000;000#000000 Backups online(ldapsearch)/offline(slapcat)/phyiscal Checkpointing Monitoring: Nagios ZenOSS etc. http://creativecommons.org/licenses/by/3.0/ 02/04/0931 32. Conclusion OpenLDAP is the fastest directory software in the world. Now it also provides the best replicationfeatures in the world.There is always a replication model to suit your needs, if not send us a patch!Don't hide in a corner, join our community, ask questions and help others! - http://www.openldap.org http://creativecommons.org/licenses/by/3.0/ 02/04/09 32 33. Questions?http://creativecommons.org/licenses/by/3.0/ 02/04/09 33