RE: SP1 Problem SOAPException doesn't return quote and Umlaute correcty

From: Dan Rogers (danro_at_microsoft.com)
Date: 12/16/04 

Date: Thu, 16 Dec 2004 21:26:11 GMT

Hi Dany,

I do understand your desire to see the behavior change. You've explained
your perspective well. I think you'll find though, that the behavior you
are seeing _is_ the bug fix. The point of view that makes that the current
state stems from the belief that it is NOT unreasonable to expect a web
client to UrlDecode data that was passed over the wire before displaying it
to the end user.

At the core of it is : Should SOAP fault description strings be intended
to be end use ready?

 I understand your argument for the answer being "YES".

I'm glad you've found a work around,

Dan

--------------------
Date: Thu, 16 Dec 2004 11:28:08 +0100
References: <uCfPi7f2EHA.3132@TK2MSFTNGP14.phx.gbl>
        <#H865Er4EHA.3908@TK2MSFTNGP12.phx.gbl>
        <UuBAG$s4EHA.4068@cpmsftngxa10.phx.gbl>
X-Newsreader: Groupwise 6.5
From: Dany <Dany@newsgroups.com>
Subject: RE: SP1 Problem SOAPException doesn't return quote and Umlaute
        correcty
Message-ID: <OaVG9n14EHA.2540@TK2MSFTNGP09.phx.gbl>
Newsgroups: microsoft.public.dotnet.framework.webservices
NNTP-Posting-Host: gonzo.opus.ch 213.200.237.2
Lines: 1
Path:
cpmsftngxa10.phx.gbl!TK2MSFTFEED02.phx.gbl!TK2MSFTNGP08.phx.gbl!TK2MSFTNGP09
phx.gbl
Xref: cpmsftngxa10.phx.gbl
microsoft.public.dotnet.framework.webservices:8127
X-Tomcat-NG: microsoft.public.dotnet.framework.webservices

Hi Dan

The problem is, that the web service provides for different clients, and I
can't change them all. Also it is very difficult telling the implementers
of the clients to make a hack (decoding twice instead of once) to correct a
bug in the webservice on their side dependent of the server having .net
framework SP1 installed or not. At the moment we've resorted to
implementing a SOAP Extension to correct the error on the server side. But
this is just a workaround for the moment and performance-wise not ideal.
Also we have to to check if .net Framework SP1 is installed on installation
of a server and only then install the SOAP Extension. As you can imagine
this is not the way I expect bugs to be handled.

What is the best approach (I'm an msdn universal subscriber) to have a
chance of MS fixing the bug at least for a future service pack or version?

Regards
Dany

>>> Dan Rogers<danro@microsoft.com> 15.12.2004 18:58:39 >>>
Hi Dany,

Given your example, I expect the behavior you cite. The issue is that the
unicode string is being URLEncoded as part of converting it to XML. To
display it, try URLDecoding it.

Dan
--------------------
Date: Wed, 15 Dec 2004 15:20:23 +0100
References: <uCfPi7f2EHA.3132@TK2MSFTNGP14.phx.gbl>
        <#f4Owvd4EHA.3644@tk2msftngp13.phx.gbl>
        <iwvrpDk4EHA.2788@cpmsftngxa10.phx.gbl>
X-Newsreader: Groupwise 6.5
From: Dany <Dany@newsgroups.com>
Subject: RE: SP1 Problem SOAPException doesn't return quote and Umlaute
        correcty
Message-ID: <#H865Er4EHA.3908@TK2MSFTNGP12.phx.gbl>
Newsgroups: microsoft.public.dotnet.framework.webservices
NNTP-Posting-Host: gonzo.opus.ch 213.200.237.2
Lines: 1
Path:
cpmsftngxa10.phx.gbl!TK2MSFTNGXA01.phx.gbl!TK2MSFTNGP08.phx.gbl!TK2MSFTNGP12
phx.gbl
Xref: cpmsftngxa10.phx.gbl
microsoft.public.dotnet.framework.webservices:8106
X-Tomcat-NG: microsoft.public.dotnet.framework.webservices

Hi Dan

I can only supply the XML for the Fault/detail element, not for the
Fault/faultstring element. The Fault/faultstring is by definition the
human-readable explanation of why the Fault occurred. That is the string we
need. But this is the string, that is converted incorrectly from .net
Framework SP1. So I don't see where I can provide an XML Node for the
faultstring (not the fault detail). Or have I not understood your
explanation correctly?

I'm still absolutely sure there is a bug here in SP1 which needs to be
fixed in the .net framework. And believe me I've studied this problem quite
intensly. Just try the example I posted, maybe then you'll see what I mean.

Dany

>>> Dan Rogers<danro@microsoft.com> 15.12.2004 01:56:27 >>>
Hi Dany,

You can manuallyl supply the XML for the fault node. The issue is
converting a unicode string to XML. When you have unicode in strings, and
then let the default system convert it to XML, the first thing it's going
to do is URLEncode the string, and then include THAT in the fault node.
Try creating the XML yourself (Xml.Document.CreateNode .. or something like
that). And populating the fault yourself. I think you'll find that
SoapException has some overrides in the constructor that let you do this.

It's been a while... but I remember doing this some time back myself.

Dan
--------------------
Date: Tue, 14 Dec 2004 13:53:37 +0100
References: <uCfPi7f2EHA.3132@TK2MSFTNGP14.phx.gbl>
        <et$A51P3EHA.1524@TK2MSFTNGP09.phx.gbl>
        <034VKAV4EHA.768@cpmsftngxa10.phx.gbl>
X-Newsreader: Groupwise 6.5
From: Dany <Dany@newsgroups.com>
Subject: RE: SP1 Problem SOAPException doesn't return quote and Umlaute
        correcty
Message-ID: <#f4Owvd4EHA.3644@tk2msftngp13.phx.gbl>
Newsgroups: microsoft.public.dotnet.framework.webservices
NNTP-Posting-Host: gonzo.opus.ch 213.200.237.2
Lines: 1
Path:
cpmsftngxa10.phx.gbl!TK2MSFTFEED02.phx.gbl!TK2MSFTNGP08.phx.gbl!tk2msftngp13
phx.gbl
Xref: cpmsftngxa10.phx.gbl
microsoft.public.dotnet.framework.webservices:8068
X-Tomcat-NG: microsoft.public.dotnet.framework.webservices

Hi Dan

We are not creating any fault node data. All we do is:

throw new SoapException( errMsg,new XmlQualifiedName("Server.Problem"));

where errMsg is a normal string, which can contain characters like """,
"<", ">", "ä","ü", etc.

The way I understand, we have no influence as to how the SOAP response is
created here. So I don't understand what we could do differently. Please
elaborate if you have an idea.

The way we see it the SOAP subsystem serializes the SOAPException to a SOAP
response and falsly converts the characters in the fault string to XML
twice. The fact that a pure .net consumer can't even handle converting the
fault string back to a correct string seems to prove, that it can't be
correct.

We have done a workaround (=hack) now, assuming that MS won't be able to
fix the bug quick enough. We've created a SOAP Extension that takes the
SOAP response and undoes the wrongly converted characters. But this is by
no meens something I want to have to keep in a productive system.

I really would appreciate if you could pass this bug on to someone in MS
who could take care of it.
Reproducing is very easy:
1. In VS create a new web service project.
2. Add a HelloWorld webservice method as so:
                [WebMethod]
                public string HelloWorld()
                {
                                                string msgStr = "This is a
quote: ""
                        throw new SoapException( "Now I've got a problem \"here\". ><",new
XmlQualifiedName("Server.Problem"));
                }
3. Create a new WindowsApplication
4. Add a web reference to the web service created above (call it
TestWebService)
5. Add a button and add the following code for it's click handler
                private void button1_Click(object sender, System.EventArgs e)
                {
                        try
                        {
                                TestWebService.Service1 appserv = new TestWebService.Service1();
                                string result = appserv.HelloWorld();
                                MessageBox.Show( result );
                        }
                        catch (Exception exc)
                        {
                                                                
MessageBox.Show( exc.Message );
                        }
                }
6. Now run the client and after pressing the button1 you will see a message
box with the incorrectly converted error message. If you use a proxy to
analyse the SOAP request and response you can see that the fault string in
the SOAPFault is: This is a quote &amp;quot; or this is a lesser than
symbol &amp;lt; or these are german Umlaute &amp;#246;, &amp;#228;,
&amp;#252;
This is of course the wrong XML representation and should actually be: This
is a quote &quot; or this is a lesser than symbol &lt; or these are german
Umlaute &#246;, &#228;, &#252;

Regards
Dany

>>> Dan Rogers<danro@microsoft.com> 13.12.2004 21:12:11 >>>
Hi Dany,

The SOAP subsystem doesn't actually do much when it comes to getting the
fault string on the wire. What it will do is take an XML node and insert
it into the proper place in a fault. It sounds like there may be the case
where the code you are using to set up the XML is double converting. This
could happen if you try and treat a string with embedded angle brackets in
it like XML. Try creating the fault node data by using XmlElement's if
this is the case.

I hope this helps

Dan
--------------------
Date: Wed, 8 Dec 2004 09:11:03 +0100
References: <uCfPi7f2EHA.3132@TK2MSFTNGP14.phx.gbl>
        <eDppBzF3EHA.2316@TK2MSFTNGP15.phx.gbl>
        <#1p7txJ3EHA.4068@cpmsftngxa10.phx.gbl>
X-Newsreader: Groupwise 6.5
From: Dany <Dany@newsgroups.com>
Subject: RE: SP1 Problem SOAPException doesn't return quote and Umlaute
        correcty
Message-ID: <et$A51P3EHA.1524@TK2MSFTNGP09.phx.gbl>
Newsgroups: microsoft.public.dotnet.framework.webservices
NNTP-Posting-Host: gonzo.opus.ch 213.200.237.2
Lines: 1
Path:
cpmsftngxa10.phx.gbl!TK2MSFTNGXA01.phx.gbl!TK2MSFTNGP08.phx.gbl!TK2MSFTNGP09
phx.gbl
Xref: cpmsftngxa10.phx.gbl
microsoft.public.dotnet.framework.webservices:7978
X-Tomcat-NG: microsoft.public.dotnet.framework.webservices

Hi Dan

The more I look into it the more I get the impression there is really
something very wrong here in the .net Framework SP1. I've been talking to
an XML specialist and he noticed that it is not correct XML generated when
there is an Umlaut (ä, ö, ü) f.e. the error message for "Bitte erfassen sie
ein Abreisedatum für das Dokument." is represented in the SOAP fault as

<faultstring>Bitte erfassen sie ein Abreisedatum f&amp;#252;r das
Dokument.</faultstring>

Only when using a proxy to log the messages on the wire did I notice this.
Now when the XML Fault is read through an XML parser the &amp; is correctly
converted to a "&" and the result is "Bitte erfassen sie ein Abreisedatum
f&#252;r das Dokument." which is of course wrong after a conversion.

So it looks like the SOAP fault string is converted twice by the .net
Framework before landing on the wire. The first time everything would be Ok
and the ü character is converted to a valid character reference "&#252;".
But then this is replaced by character references once again which results
in "&amp;#252;" which is of course not a valid XML representation of the
"ü" character.

In my view this is a serious bug in the handling of SOAPExceptions by the
net Framework. I'd be really thankful if you could check with the
serialization team. Reproducing the error should be quite simple. Hope they
have a solution then.

Thanks & Regards
Dany

>>> Dan Rogers<danro@microsoft.com> 07.12.2004 21:36:25 >>>
Hi Dany,

I suspect that there is an encoding issue in the way you are loading the
XML. Is there an inconsistency between the XML encoding mark up and the
actual encoding on the wire? UrlEncoded XML is supported by the DOM, but
if someone somewhere along the way is loading what looks like the XML into
a unicode string without doing an encoding conversion first, issues can
happen.

As for how/why this happened - I'm not that guy. For the most part, 1.1
fixed bugs. It is possible that it introduced some as well. I'll see what
I can dig up from the serialization team - can't promise they have a fix,
but I'll see.

Dan
--------------------
Date: Tue, 7 Dec 2004 14:00:36 +0100
References: <uCfPi7f2EHA.3132@TK2MSFTNGP14.phx.gbl>
        <hzdWUQ92EHA.3200@cpmsftngxa10.phx.gbl>
X-Newsreader: Groupwise 6.5
From: Dany <Dany@newsgroups.com>
Subject: RE: SP1 Problem SOAPException doesn't return quote and Umlaute
        correcty
Message-ID: <eDppBzF3EHA.2316@TK2MSFTNGP15.phx.gbl>
Newsgroups: microsoft.public.dotnet.framework.webservices
NNTP-Posting-Host: gonzo.opus.ch 213.200.237.2
Lines: 1
Path:
cpmsftngxa10.phx.gbl!TK2MSFTFEED01.phx.gbl!TK2MSFTNGP08.phx.gbl!TK2MSFTNGP15
phx.gbl
Xref: cpmsftngxa10.phx.gbl
microsoft.public.dotnet.framework.webservices:7944
X-Tomcat-NG: microsoft.public.dotnet.framework.webservices

Found out .net web service clients are just as helpless with handling these
character references.

I've built a simple web service consumer in VC#. On a button click the
following code is executed:
                        try
                        {
                                localhost.UmbrellaNTAppServer appserv = new
localhost.UmbrellaNTAppServer();
                                string result = appserv.HelloWorld();
                                MessageBox.Show( result );
                        }
                        catch (Exception exc)
                        {
                                                               
MessageBox.Show( "Error has occured: " + exc.Message );
                        }

If an exception is raised in the web service method saying i.e. 'The method
"HelloWorld" is under construction.' as the exception message. The user
will be shown 'Error has occured: The method &quot;HelloWorld&quot; is
under construction. In a real world example the user would be sure to
complain about this display.

Before SP1 everything was fine, but now handling of exception messages
coming from web service calls must be handled specifically.
In normal SOAP responses (not SOAP faults) strings containing special
characters are correctly sent. So if 'The method "HelloWorld" is under
construction.' is returned the client will also receive it unchanged and
without any character references.

So I wonder why this inconsistent behaviour was built into the SP1 by MS.

Regards Dany

>>> Dan Rogers<danro@microsoft.com> 06.12.2004 21:42:15 >>>
Hi,

What you are describing is the use of encoded strings. These are, to my
knowlege, valid in XML and the responsibility of the parser to convert
properly.

What is the source of the problem, given that these are valid
representations of the data? Is the client not able to process these
properly as it should?

Regards

Dan Rogers
Microsoft Corporation

--------------------
Date: Sat, 4 Dec 2004 13:43:30 +0100
X-Newsreader: Groupwise 6.5
From: Dany <Dany@newsgroups.com>
Subject: SP1 Problem SOAPException doesn't return quote and Umlaute correcty
Message-ID: <uCfPi7f2EHA.3132@TK2MSFTNGP14.phx.gbl>
Newsgroups: microsoft.public.dotnet.framework.webservices
NNTP-Posting-Host: adsl-62-167-138-146.adslplus.ch 62.167.138.146
Lines: 1
Path:
cpmsftngxa10.phx.gbl!TK2MSFTFEED02.phx.gbl!TK2MSFTNGP08.phx.gbl!TK2MSFTNGP14
phx.gbl
Xref: cpmsftngxa10.phx.gbl
microsoft.public.dotnet.framework.webservices:7851
X-Tomcat-NG: microsoft.public.dotnet.framework.webservices

Our web service was working fine until we installed .net Framework 1.1
service pack 1. Uninstalling SP1 is not an option because our largest
customer says service packs marked as "critical" by Microsoft must be
installed on their servers.

Now german Umlaute (ä, ü, ö) and quotes are returned incorrectly in SOAP
fault responses.
This can be easily verified:

Implement the following in a web service method (just raises a
SOAPException with a message text containing quote characters ("):

  string errMsg = "Here is a \"quote\".";
  throw new SoapException( errMsg,new XmlQualifiedName("Server.Problem"));

The following SOAP response is created when calling this SOAP-Exception:

HTTP/1.1 500 Internal Server Error.
Server: Microsoft-IIS/5.1
Date: Fri, 03 Dec 2004 13:02:23 GMT
X-Powered-By: ASP.NET
X-AspNet-Version: 1.1.4322
Cache-Control: private
Content-Type: text/xml; charset=utf-8
Content-Length: 431

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <soap:Body>
    <soap:Fault>
      <faultcode>Server.Problem</faultcode>
      <faultstring>Here is a &quot;quote&quot;.</faultstring>
      <detail />
    </soap:Fault>
  </soap:Body>
</soap:Envelope>

All though the encoding should be utf-8 and "& q u o t ;" is not correct.
This worked fine in .net 1.1 without the SP1. The correct <faultstring>
should be:
      <faultstring>Here is a "quote".</faultstring>

This problem also occurs when using german Umlaute: f.e. with an exception
message of "Fahrvergnügen". It is converted incorrectly to
<faultstring>Fahrvergn&amp;#252;gen</faultstring>.

This is a serious problem for us, because in german and french many error
messages will have special characters like ö,ä, é, à in them and these are
not correctly processed.

Thanks for any help!
Dany