possible LDAP over SSl bug in OS 10, 10.4, 10.5, 10.6

From: Schley Andrew Kutz (_at_)
Date: 11/16/04 

Date: Tue, 16 Nov 2004 07:11:01 -0600

This was submitted to Apple Bug Reporter on Oct. 23rd 2004. 

---
23-Oct-2004 02:57 PM Schley Kutz:
Summary:
--------
As I have been reminded, for all intensive purposes I represent the 
greater community of Mac users here at the University of Texas at 
Austin who want to get Entourage to be as feature-full as it is 
advertised to be.  UT@Austin is the largest public university in the 
nation, and at one time it was the largest concentration of Macs in the 
nation as well.  Hence, this is something that *really* needs to work.
Microsoft released Entourage 2004 into the ready hands of users 
clamoring for a product that was the equivalent of Outlook 2003 and 
would give Mac users a portal to access Exchange.  Entourage 2004 had 
some problems however.  Particularly it did not let users securely 
access the GAL (Directory Access in Entourage) or securely acknowledge 
themselves as delegates.  These were both LDAP over SSL issues.  These 
problems were thought to be on Entourage's part.
Then Service Pack 1 for Office 2004 came out.  I applied it with great 
hopes that it would fix the LDAP over SSL issue that plagued so many 
users.  It did not.  I then composed a *very* long email to our 
Microsoft Technical Account Manager (TAM).  The email detailed all the 
steps I had taken with Ethereal and ssltap to determine that Entourage 
could decidedly still not properly talk to LDAP over SSL.  Towards the 
end of composing the email I decided to reconfirm that native OS X apps 
would access a LDAP server over SSL.  Much to my surprise (since this 
had been tested before) Address Book could not talk to 
austin.utexas.edu via SSL.
So the problem is deeper than Entourage 2004.  The problem that I found 
seems to be an issue with the way that OS X's Directory Services 
framework interacts with OpenLDAP.
Steps to Reproduce:
-------------------
Thus began my journey on a different path.  I compiled ssldump for OS X 
and began examining different configurations of LDAP queries.  I 
realized that I had to test these configurations on more than just our 
Active Directory so Eric Irrgang and Robert Kennedy of the ITS Unix 
group graciously assisted me with this.  Eric set me up with an account 
on the Unix group's internal LDAP server, and Robert showed me some 
basic openldap command line commands such as ldapsearch.  In fact, I 
used several programs to test LDAP over SSL functionality on OS X.  
They are:
a) Address Book.app
b) JXplorer
c) ldapsearch
d) Entourage 2004 - Directory Service
e) Entourage 2004 - Delegates
d and e may seem surprising but it is the case while Directory Service 
(the GAL) and Delegates in Entourage 2004 both issue the same type of 
LDAP requests (I watch unencrypted traffic in Ethereal to prove this), 
they produce different errors in ssldump when they do not work as 
advertised.
There were three distinctly different results when trying to access 
LDAP over SSL:
1) LDAP over SSL works successfully and much giddiness ensued
2) LDAP over SSL works and ssldump reports (in several fashions) an 
Unknown Certificate Authority (ca|CA) error.  This permutation will in 
some cases report an error to the user but in fact all the traffic is 
still encrypted and as far as Entourage's Directory Service is 
concerned the data is still retrieved.
3) LDAP over SSL fails and ssldump reports that an Unknown CA was the cause.
4) LDAP over SSL fails completely.  This is what sparked the OS X 
investigation.  What is odd about this error was the ssldump output.  
It showed that although the commands were being issued over port 636 
(ldaps port), the client was trying to initiate a NON SSL connection to 
the server.  At this point the server basically says "Um, yes, I am 
terribly sorry sir but it would be entirely improper of me if I were to 
allow you to continue this obvious assault on my person.  Please make 
haste in your efforts to retreat from my doorstep lest I be forced to 
commit a barbaric act of v101ence."
I exported two Certification Authority certificates to Base64 format.  
They were the Secure Server Certification Authority certificate (SSCA) 
(a standard Verisign CA), which is used by our Active Directory, and 
the ITS Unix CA certificate, used by the Unix group's LDAP server.  I 
imported these into Keychain Access's X509Anchors keychain (I removed 
the original SSCA CA from the keychain prior to adding mine) which 
should be used by Address Book.app and Entourage 2004.  Next, I 
imported these into Jxplorer's Key Store.  Finally, I used the 
TLS_CACERT option in my .ldaprc file to specify the CA to use when 
executing the OpenLDAP command ldapsearch.  I then removed ALL of the 
certificates and ran the tests again in order to get a full suite of 
debugging information.
Here is a simple chart of the different configurations I tested.  I 
have attached text files of the logs for these configurations.  
Although I tested this on 5 different systems (see Isolation for OS 
versions), I only generated log files for my primary test system.
For interpreting the chart please refer to the alpha and numeric scheme 
I used for lableling the applications I tested and the results I found.
yca = with Certificate Authority certificate installed for the ldap 
server I was accessing
nca = without Certificate Authority certificate installed for the ldap 
server I was accessing
          a     |     b    |     c    |     d    |     e    |
     |----------|----------|----------|----------|----------|
yca  |    4     |     1    |     1    |     2    |     4    |
     |----------|----------|----------|----------|----------|
nca  |    4     |     3    |     3    |     2    |     4    |
     |----------|----------|----------|----------|----------|
yca/a - Address Book.app fails completely nca/a - Address Book.app 
fails completely
yca/b - JXplorer works
nca/b - Jxplorer fails because of unknown CA
yca/c - ldapsearch works
nca/c - ldapsearch fails because of unknown CA
yca/d - Entourage Directory Service throws an error but continues to 
return good data that IS encrypted nca/d - Entourage Directory Service 
throws an error but continues to return good data that IS encrypted
yca/e - Entourage Delegates fails completely nca/e - Entourage 
Delegates fails completely
So what do these results tell me?  Well, what is MOST interesting is 
that 2 different parts of the application that started this all, 
Entourage, must use ENTIRELY DIFFERENT methods of doing a directory 
lookup.  I know this because they throw completely different errors.
What is telling about the Delegates error is that it is exactly the 
same as the Address Book.app's error.  This got me thinking.  How can I 
duplicate all of Entourage's errors (Address Book.app's error is the 
same as Entourage's Delegates error so if I can dupe the Delegates 
error I will have duped the Address Book.app error) errors with 
OpenLDAP's command line tool, ldapsearch?
First, the not so shocking discovery.  The error that the Entourage 
Directory Service gives is that it cannot find the CA.  Well that is 
easy enough to reproduce.  I just removed the TLS_CACERT definition 
that pointed to the SSCA cert that our AD uses from my .ldaprc file.  I 
then defined TLS_CERT as 'request' and did an ldapsearch against our 
AD.  This produced the exact same error that Entourage gives when using 
its Directory Service.  Essentially, when Entourage issues a directory 
request from its Directory Service it gets to OpenLDAP as a call that 
does not use a CA but requests one.  This causes the client to say 
"Hey, I know you have a CA to give me, but I don't have one to give 
you.  Eh, don't worry about it.  I'll take a warning sure, but we can 
still talk in that crazy encrypted lingo that those cats like to call 
SSL."  The logs show a warning but all traffic still occurs and is 
still encrypted.
When I removed the TLS_CACERT definition and executed the same command:
ldapsearch -H "ldaps://server.name.utexas.edu:636" -W -D "CN=not 
real,OU=yeah right,OU=not on your 
life,DC=server,DC=name,DC=utexas,DC=edu"
I received this error:
TLS certificate verification: Error, unable to get local issuer certificate
This just proves that it is at a different location than OpenLDAP that 
the request of the CA from KeyChain happens.  I could not find a 
reference to X509 or Keychain (grep -ri) in any of the DirectoryAccess 
sources either.  I could not find a header for OpenSSL on a clean  10.3 
install either.  Hmmm.  Duh!  I finally found ssl.h in the DevSDK and 
openssl header includes in the DirectoryAccess headers.  OpenSSL 
sources do not exist, that I found, in a clean install of OS X, but 
rather are compiled into /usr/bin/openssl and the OS X Directory 
Service.
This still begs the question, why is the "Cannot find root certificate" 
error occurring?  1 or 2 of 2 things is happening.  Either something is 
improperly using /usr/bin/openssl to generate a temporary root ca from 
KeyChain or something is improperly hooking into the OpenSSL libraries.
Second, the discovery that left me saying, "Huh?"  Error #4 was just 
odd.  All the other ssldumps showed the client's first contact with the 
server as "C->S SSLv2 Compatible Hello (636)."  Basically the client 
was talking to the server over 636 in SSL lingo, as it should.  But the 
Address Book.app and Entourage's Delegates generated this log, "C->S 
(636)."  It was as if the client was trying to issue LDAP commands with 
a normal ldap uri "ldap://" over 636.  Well this won't fly cat.  Nope 
man, you've got to use "ldaps://" if you want to dig on 636.  At least 
with OpenLDAP you do.  Anyway, so I imitated this with OpenLDAP.  I 
executed this command to both LDAP servers:
ldapsearch -H "ldap://server.name.utexas.edu:636" -W -D "CN=not 
real,OU=yeah right,OU=not on your 
life,DC=server,DC=name,DC=utexas,DC=edu"
Voilla!  Same error as Address Book.app and Entourage's Delegates.  I 
know it sounds crazy, but it looks like those two programs are issuing 
LDAP commands with a non secure form of the ldap uri over port 636.  
This is just silly.  Of course it won't work.
How, you might ask, could this have happened?  Well, really, it is not 
that hard to imagine.  LDAP is a protocol that is used to access what 
has been, until recently, philosophically something that is "public" 
information.  Directories, large ones especially, are by their nature 
meant to be open and read by the public.  If a directory is small and 
needs to be secure there are better solutions than a LDAP enabled 
directory.
Anyway.
So I was able to deduce quite a bit, but I was still kind of ticked 
off.  I may have possibly sort of perhaps figured out *what* was going 
on, but *where* is the on going?  That is the real zinger of the month 
... Well, as I mentioned, Entourage seems to hook into OS X's 
networking API at different places, depending on where it is calling it 
from.  Also, I realized that if OpenLDAP is not grabbing goodies from 
X509Anchors it must be Directory Access doing the dirty deed.
This is where my good friend and hated enemy Nicholas "St. Who?" 
Lauland came in.  I absolutely despise him (tongue in cheek there) for 
informing me that "You can get the source code for Darwin Directory 
Service off of ADC.  Well crap.  I already knew that I could get the 
source for OpenLDAP (duh!  ->OPEN<-LDAP), but this new revelation just 
meant that I was going to be compelled to see this through much further 
along than I was mentally capable of.
Do you know what happens after you press "Search" on the Address 
Book.app and expect it to search a configured LDAP server?
Neither do I.  But I do know where things go from when 
CLDAPv3Plugin::ProcessRequest invokes CLDAPv3Plugin::HandleRequest.  
And that is really the point where the story can start anyway since 
CLDAPv3Plugin inherits the CServerPlugin class.  Any LDAP-centric 
configuration is going to take place once a CLDAPv3Plugin object is 
instantiated.
I have attached the OpenLDAP headers and sources that contain the 
functions I am going to reference.  I also attached the 
DirectoryService sources and headers.  The Adc membership it takes to 
request them is free, so whomever could actually go online an get them. 
 Here is a full list of the headers and sources that contain the 
functions that I am going to reference.  In fact, they are listed in 
the order in which they are invoked, so I will only list them here:
CServerPlugin::ProcessRequest - 
DirectoryService-257.1/Server/CServerPlugin.h .cpp
CServerPlugin::HandleRequest - 
DirectoryService-257.1/Server/CServerPlugin.h .cpp
CLDAPv3Plugin::OpenDirNode - 
DirectoryService-257.1/Plugins/LDAPv3/CLDAPv3Plugin.h .cpp
CLDAPNode::SafeOpen - DirectoryService-257.1/Plugins/LDAPv3/CLDAPNode.h .cpp
CLDAPNode::AuthOpen - DirectoryService-257.1/Plugins/LDAPv3/CLDAPNode.h .cpp
CLDAPNode::BindProc - DirectoryService-257.1/Plugins/LDAPv3/CLDAPNode.h .cpp
ldap_bind - OpenLDAP-37.2.1/include/ldap.h /libraries/libldap/bind.c
ldap_simple_bind - OpenLDAP-37.2.1/include/ldap.h /libraries/libldap/sbind.c
ldap_sasl_bind - OpenLDAP-37.2.1/include/ldap.h /libraries/libldap/sasl.c
ldap_send_initial_request - OpenLDAP-37.2.1/include/ldap.h 
/libraries/libldap/request.c
ldap_send_server_request - OpenLDAP-37.2.1/include/ldap.h 
/libraries/libldap/request.c
The error occurs in the last function.  The code that throws the error 
is as follows:
		if ( srv == NULL ) {
			if ( !use_ldsb ) {
				ber_sockbuf_free( lc->lconn_sb );
			}
		    LDAP_FREE( (char *)lc );
		    ld->ld_errno = LDAP_SERVER_DOWN;
		    return( NULL );
		}
The text of the error reads, "Can't contact LDAP Server."  The error is 
defined in OpenLDAP-37.2.1/include/ldap.h and looks like this:
	#define LDAP_SERVER_DOWN		0x51
For laughs I decided to set Address Book.app and Entourage Delegates to 
use SSL over port 389.  Guess what?  Instead of acting in error like 
they do when you specify port 636/3239 (Global Catalog SSL, GCS) and 
trying to talk to the server without first initiating a compatible 
SSLv2 Hello, they issue a compatible SSLv2 Hello!  However, I used 
ethereal and verified that the traffic is in fact traveling in the 
clear when you do this.  BUT, it does cause Address Book.app and 
Entourage Delegates to issue the correct Hello.
It seems to me that the combination of specifying SSL and talking over 
port 636 causes some type of confusion in the Directory Access API.  I 
do not know where since I do not have the sources or headers for 
Address Book.app and/or Entourage.
I confirmed the errors by duplicating the theorized process with 
ldapsearch.  This was the command I used:
ldapsearch -H "ldaps://server.name.utexas.edu:389" -W -D "CN=not 
real,OU=yeah right,OU=not on your 
life,DC=server,DC=name,DC=utexas,DC=edu"
I also tested all of the above by adding a LDAPv3 entry into OS X's 
Directory Access preferences for austin.utexas.edu.  I checked it by 
searching against OS X's Directory Services in Address Book.app and by 
doing a "lookupd -d <return> userWithName: akutz."  Both failed.
Expected Results:
-----------------
Well, that table I drew earlier should look like this:
a) Address Book.app
b) JXplorer
c) ldapsearch
d) Entourage 2004 - Directory Service
e) Entourage 2004 - Delegates
1) LDAP over SSL works successfully and much giddiness ensued
2) LDAP over SSL works and ssldump reports (in several fashions) an 
Unknown Certificate Authority (ca|CA) error.  This permutation will in 
some cases report an error to the user but in fact all the traffic is 
still encrypted and as far as Entourage's Directory Service is 
concerned the data is still retrieved.
3) LDAP over SSL fails and ssldump reports that an Unknown CA was the cause.
4) LDAP over SSL fails completely.  This is what sparked the OS X 
investigation.  What is odd about this error was the ssldump output.  
It showed that although the commands were being issued over port 636 
(ldaps port), the client was trying to initiate a NON SSL connection to 
the server.  At this point the server basically says "Um, yes, I am 
terribly sorry sir but it would be entirely improper of me if I were to 
allow you to continue this obvious assault on my person.  Please make 
haste in your efforts to retreat from my doorstep lest I be forced to 
commit a barbaric act of v101ence."
yca = with Certificate Authority certificate installed for the ldap 
server I was accessing
nca = without Certificate Authority certificate installed for the ldap 
server I was accessing
          a     |     b    |     c    |     d    |     e    |
     |----------|----------|----------|----------|----------|
yca  |    1     |     1    |     1    |     1    |     1    |
     |----------|----------|----------|----------|----------|
nca  |   2/3    |    2/3   |    2/3   |    2/3   |    2/3   |
     |----------|----------|----------|----------|----------|
yca/a - Address Book.app works
nca/a - Address Book.app throws an error but continues to return good 
data that IS encrypted OR fails because of unknown CA
yca/b - JXplorer works
nca/b - Jxplorer throws an error but continues to return good data that 
IS encrypted OR fails because of unknown CA
yca/c - ldapsearch works
nca/c - ldapsearch throws an error but continues to return good data 
that IS encrypted OR fails because of unknown CA
yca/d - Entourage Directory Service works nca/d - Entourage Directory 
Service throws an error but continues to return good data that IS 
encrypted OR fails because of unknown CA
yca/e - Entourage Delegates works
nca/e - Entourage Delegates throws an error but continues to return 
good data that IS encrypted OR fails because of unknown CA
Why 2 OR 3 without the certificate authority?  Because I OS X 's 
Directory Service should, and from what I can tell from my testing it 
apparently is not, respect any .ldaprc file in my homedir or any 
ldap.conf file in /etc/openldap/ldap.conf.  The ldap.conf can be in 1 
or 2 more places, but the lid to my powerbook is closed, and I enjoy 
watching the button light throb too much to open the lid back up to 
bring up ldap.conf 's man page.  Meaning Directory Service should 
respect whether or not I tell OpenLDAP to ignore TLS_REQCERT and 
TLS_CHECKPEER.  The ldapsearch command does (of course) and anything 
that implements OpenLDAP should too.
Actual Results:
---------------
a) Address Book.app
b) JXplorer
c) ldapsearch
d) Entourage 2004 - Directory Service
e) Entourage 2004 - Delegates
1) LDAP over SSL works successfully and much giddiness ensued
2) LDAP over SSL works and ssldump reports (in several fashions) an 
Unknown Certificate Authority (ca|CA) error.  This permutation will in 
some cases report an error to the user but in fact all the traffic is 
still encrypted and as far as Entourage's Directory Service is 
concerned the data is still retrieved.
3) LDAP over SSL fails and ssldump reports that an Unknown CA was the cause.
4) LDAP over SSL fails completely.  This is what sparked the OS X 
investigation.  What is odd about this error was the ssldump output.  
It showed that although the commands were being issued over port 636 
(ldaps port), the client was trying to initiate a NON SSL connection to 
the server.  At this point the server basically says "Um, yes, I am 
terribly sorry sir but it would be entirely improper of me if I were to 
allow you to continue this obvious assault on my person.  Please make 
haste in your efforts to retreat from my doorstep lest I be forced to 
commit a barbaric act of v101ence."
yca = with Certificate Authority certificate installed for the ldap 
server I was accessing
nca = without Certificate Authority certificate installed for the ldap 
server I was accessing
          a     |     b    |     c    |     d    |     e    |
     |----------|----------|----------|----------|----------|
yca  |    4     |     1    |     1    |     2    |     4    |
     |----------|----------|----------|----------|----------|
nca  |    4     |     3    |     3    |     2    |     4    |
     |----------|----------|----------|----------|----------|
yca/a - Address Book.app fails completely nca/a - Address Book.app 
fails completely
yca/b - JXplorer works
nca/b - Jxplorer fails because of unknown CA
yca/c - ldapsearch works
nca/c - ldapsearch fails because of unknown CA
yca/d - Entourage Directory Service throws an error but continues to 
return good data that IS encrypted nca/d - Entourage Directory Service 
throws an error but continues to return good data that IS encrypted
yca/e - Entourage Delegates fails completely nca/e - Entourage 
Delegates fails completely
Workaround:
-----------
I have not found one.  Nick suggested that I recompile OpenLDAP and 
throw in some extra debugging statements.  I even thought of checking 
for the presence of port 636 and forcing the uri to start with "ldaps." 
 Quite frankly, that's what the folks at Apple get paid to do : )  For 
now I know the limitations of OS X and subsequently Entourage 2004 / 
SP1.
All we Entourage users here at the University can do is wait.
Isolation:
----------
The same errors have occurred on 5 different Macs.  Here are they 
operating systems:
10.3.5
10.3.5
10.3.5
10.3.4
10.3
'one file to bind them.zip' and 'System Profile.spx'were successfully uploaded
-- 
-a
Loading