Re: SP1 Problem SOAPException doesn't return quote and Umlaute correcty

From: Dany (Dany_at_newsgroups.com)
Date: 12/22/04 

Date: Wed, 22 Dec 2004 09:22:04 +0100

Hi Dave

I'm glad at least someone else is experiencing the same problems. I almost started to believe either we are the only ones using the SOAPException fault string with special characters or almost no one uses SOAP Webservices in .net productivly today.

I'm still waiting for MS to acknowledge this bug. Don't know how to get through to them. It's quite frustrating.

Regards
Dany

>>> Dave<dhoek@bakerhill.com> 17.12.2004 20:56:07 >>>
Dany,

I too am experiencing this problem with .Net 1.1 SP1. There definitely
is some double encoding of special characters going on.

Our app returns XML in the fault string. Our client uses the
webservice behavior. Before SP1, we could have a webservice that
returned <whatever> in the fault string of a soap exception. This
would get properly encoded as &lt;whatever&gt;. You can verify that by
looking at the webservice behavior's errorDetail.raw.xml - you can see
&lt;whatever&gt; in the faultstring element. errorDetail.string would
properly return <whatever>.

Now, with SP1, errorDetail.raw.xml shows &amp;lt;whatever&amp;gt;.
This very clearly indicates that SP1 causes special characters to be
encoded twice. This bug broke some code that parses the fault string
on the client.

Dave

Dany wrote:
> Hi Dan
>
> Sorry, for being so insistant, but I think I might not have made the
problem clear enough.
>
> You are absolutely correct, that the SOAP client needs to URL decode
the fault description. This is what for example a .net webservice
consumer will also do and this works correctly if the web service is
not running on a server with .net Framework SP1.
> But the way it is now in SP1 it would have to be decoded twice, which
of course a web service caller will not do normally and is not
according to W3C SOAP specs.
>
> The SOAP fault string returned in the SOAP fault contains f.e. a
quote (") as:
>
> &amp;quot;
>
> Now the normal decoding which is done will generate:
>
> &quot;
>
> But this is still not enough the client would have to decode once
again to receive a ".
>
> The need to decode a message 2 times is just not correct. Correct
would be the SOAP response containing:
>
> &quot;
>
> This is then decoded on the client side correctly to a " before
displaying to the user.
>
> I've tested this with different implementations of web service
clients and non of them does the conversion twice. I have also checked
the SOAP and XML specs and they all state that the correct
representation of a doublequote is &quot; and not &amp;quot;.
>
> So there is no way to see the new behavior as a bug fix, it is a bug.
The client is not required to decode the message TWICE before being
able to display it to the end user.
>
> Hope the problem is more clear now. Or is there an error in the
example or my reasoning?
>
> Regards
> Dany
>
> >>> Dan Rogers<danro@microsoft.com> 16.12.2004 22:26:11 >>>
> Hi Dany,
>
> I do understand your desire to see the behavior change. You've
explained
> your perspective well. I think you'll find though, that the behavior
you
> are seeing _is_ the bug fix. The point of view that makes that the
current
> state stems from the belief that it is NOT unreasonable to expect a
web
> client to UrlDecode data that was passed over the wire before
displaying it
> to the end user.
>
> At the core of it is : Should SOAP fault description strings be
intended
> to be end use ready?
>
> I understand your argument for the answer being "YES".
>
> I'm glad you've found a work around,
>
> Dan
>
> --------------------
> Date: Thu, 16 Dec 2004 11:28:08 +0100
> References: <uCfPi7f2EHA.3132@TK2MSFTNGP14.phx.gbl>
> <#H865Er4EHA.3908@TK2MSFTNGP12.phx.gbl>
> <UuBAG$s4EHA.4068@cpmsftngxa10.phx.gbl>
> X-Newsreader: Groupwise 6.5
> From: Dany <Dany@newsgroups.com>
> Subject: RE: SP1 Problem SOAPException doesn't return quote and
Umlaute
> correcty
> Message-ID: <OaVG9n14EHA.2540@TK2MSFTNGP09.phx.gbl>
> Newsgroups: microsoft.public.dotnet.framework.webservices
> NNTP-Posting-Host: gonzo.opus.ch 213.200.237.2
> Lines: 1
> Path:
>
cpmsftngxa10.phx.gbl!TK2MSFTFEED02.phx.gbl!TK2MSFTNGP08.phx.gbl!TK2MSFTNGP09
> .phx.gbl
> Xref: cpmsftngxa10.phx.gbl
> microsoft.public.dotnet.framework.webservices:8127
> X-Tomcat-NG: microsoft.public.dotnet.framework.webservices
>
> Hi Dan
>
> The problem is, that the web service provides for different clients,
and I
> can't change them all. Also it is very difficult telling the
implementers
> of the clients to make a hack (decoding twice instead of once) to
correct a
> bug in the webservice on their side dependent of the server having
.net
> framework SP1 installed or not. At the moment we've resorted to
> implementing a SOAP Extension to correct the error on the server
side. But
> this is just a workaround for the moment and performance-wise not
ideal.
> Also we have to to check if .net Framework SP1 is installed on
installation
> of a server and only then install the SOAP Extension. As you can
imagine
> this is not the way I expect bugs to be handled.
>
> What is the best approach (I'm an msdn universal subscriber) to have
a
> chance of MS fixing the bug at least for a future service pack or
version?
>
> Regards
> Dany
>
> >>> Dan Rogers<danro@microsoft.com> 15.12.2004 18:58:39 >>>
> Hi Dany,
>
> Given your example, I expect the behavior you cite. The issue is
that the
> unicode string is being URLEncoded as part of converting it to XML.
To
> display it, try URLDecoding it.
>
> Dan
> --------------------
> Date: Wed, 15 Dec 2004 15:20:23 +0100
> References: <uCfPi7f2EHA.3132@TK2MSFTNGP14.phx.gbl>
> <#f4Owvd4EHA.3644@tk2msftngp13.phx.gbl>
> <iwvrpDk4EHA.2788@cpmsftngxa10.phx.gbl>
> X-Newsreader: Groupwise 6.5
> From: Dany <Dany@newsgroups.com>
> Subject: RE: SP1 Problem SOAPException doesn't return quote and
Umlaute
> correcty
> Message-ID: <#H865Er4EHA.3908@TK2MSFTNGP12.phx.gbl>
> Newsgroups: microsoft.public.dotnet.framework.webservices
> NNTP-Posting-Host: gonzo.opus.ch 213.200.237.2
> Lines: 1
> Path:
>
cpmsftngxa10.phx.gbl!TK2MSFTNGXA01.phx.gbl!TK2MSFTNGP08.phx.gbl!TK2MSFTNGP12
> .phx.gbl
> Xref: cpmsftngxa10.phx.gbl
> microsoft.public.dotnet.framework.webservices:8106
> X-Tomcat-NG: microsoft.public.dotnet.framework.webservices
>
> Hi Dan
>
> I can only supply the XML for the Fault/detail element, not for the
> Fault/faultstring element. The Fault/faultstring is by definition the

> human-readable explanation of why the Fault occurred. That is the
string we
> need. But this is the string, that is converted incorrectly from .net

> Framework SP1. So I don't see where I can provide an XML Node for the

> faultstring (not the fault detail). Or have I not understood your
> explanation correctly?
>
> I'm still absolutely sure there is a bug here in SP1 which needs to
be
> fixed in the .net framework. And believe me I've studied this problem
quite
> intensly. Just try the example I posted, maybe then you'll see what I
mean.
>
> Dany
>
>
>
> >>> Dan Rogers<danro@microsoft.com> 15.12.2004 01:56:27 >>>
> Hi Dany,
>
> You can manuallyl supply the XML for the fault node. The issue is
> converting a unicode string to XML. When you have unicode in
strings, and
> then let the default system convert it to XML, the first thing it's
going
> to do is URLEncode the string, and then include THAT in the fault
node.
> Try creating the XML yourself (Xml.Document.CreateNode .. or
something like
> that). And populating the fault yourself. I think you'll find that
> SoapException has some overrides in the constructor that let you do
this.
>
> It's been a while... but I remember doing this some time back myself.
>
> Dan
> --------------------
> Date: Tue, 14 Dec 2004 13:53:37 +0100
> References: <uCfPi7f2EHA.3132@TK2MSFTNGP14.phx.gbl>
> <et$A51P3EHA.1524@TK2MSFTNGP09.phx.gbl>
> <034VKAV4EHA.768@cpmsftngxa10.phx.gbl>
> X-Newsreader: Groupwise 6.5
> From: Dany <Dany@newsgroups.com>
> Subject: RE: SP1 Problem SOAPException doesn't return quote and
Umlaute
> correcty
> Message-ID: <#f4Owvd4EHA.3644@tk2msftngp13.phx.gbl>
> Newsgroups: microsoft.public.dotnet.framework.webservices
> NNTP-Posting-Host: gonzo.opus.ch 213.200.237.2
> Lines: 1
> Path:
>
cpmsftngxa10.phx.gbl!TK2MSFTFEED02.phx.gbl!TK2MSFTNGP08.phx.gbl!tk2msftngp13
> .phx.gbl
> Xref: cpmsftngxa10.phx.gbl
> microsoft.public.dotnet.framework.webservices:8068
> X-Tomcat-NG: microsoft.public.dotnet.framework.webservices
>
> Hi Dan
>
> We are not creating any fault node data. All we do is:
>
> throw new SoapException( errMsg,new
XmlQualifiedName("Server.Problem"));
>
> where errMsg is a normal string, which can contain characters like
""",
> "<", ">", "ä","ü", etc.
>
> The way I understand, we have no influence as to how the SOAP
response is
> created here. So I don't understand what we could do differently.
Please
> elaborate if you have an idea.
>
> The way we see it the SOAP subsystem serializes the SOAPException to
a SOAP
> response and falsly converts the characters in the fault string to
XML
> twice. The fact that a pure .net consumer can't even handle
converting the
> fault string back to a correct string seems to prove, that it can't
be
> correct.
>
> We have done a workaround (=hack) now, assuming that MS won't be able
to
> fix the bug quick enough. We've created a SOAP Extension that takes
the
> SOAP response and undoes the wrongly converted characters. But this
is by
> no meens something I want to have to keep in a productive system.
>
> I really would appreciate if you could pass this bug on to someone in
MS
> who could take care of it.
> Reproducing is very easy:
> 1. In VS create a new web service project.
> 2. Add a HelloWorld webservice method as so:
> [WebMethod]
> public string HelloWorld()
> {
> string msgStr = "This
is a
> quote: ""
> throw new SoapException( "Now I've got a problem \"here\". ><",new

> XmlQualifiedName("Server.Problem"));
> }
> 3. Create a new WindowsApplication
> 4. Add a web reference to the web service created above (call it
> TestWebService)
> 5. Add a button and add the following code for it's click handler
> private void button1_Click(object sender, System.EventArgs e)
> {
> try
> {
> TestWebService.Service1 appserv = new TestWebService.Service1();
> string result = appserv.HelloWorld();
> MessageBox.Show( result );
> }
> catch (Exception exc)
> {
>
> MessageBox.Show( exc.Message );
> }
> }
> 6. Now run the client and after pressing the button1 you will see a
message
> box with the incorrectly converted error message. If you use a proxy
to
> analyse the SOAP request and response you can see that the fault
string in
> the SOAPFault is: This is a quote &amp;quot; or this is a lesser than

> symbol &amp;lt; or these are german Umlaute &amp;#246;, &amp;#228;,
> &amp;#252;
> This is of course the wrong XML representation and should actually
be: This
> is a quote &quot; or this is a lesser than symbol &lt; or these are
german
> Umlaute &#246;, &#228;, &#252;
>
> Regards
> Dany
>
>
> >>> Dan Rogers<danro@microsoft.com> 13.12.2004 21:12:11 >>>
> Hi Dany,
>
> The SOAP subsystem doesn't actually do much when it comes to getting
the
> fault string on the wire. What it will do is take an XML node and
insert
> it into the proper place in a fault. It sounds like there may be the
case
> where the code you are using to set up the XML is double converting.
This
> could happen if you try and treat a string with embedded angle
brackets in
> it like XML. Try creating the fault node data by using XmlElement's
if
> this is the case.
>
> I hope this helps
>
> Dan
> --------------------
> Date: Wed, 8 Dec 2004 09:11:03 +0100
> References: <uCfPi7f2EHA.3132@TK2MSFTNGP14.phx.gbl>
> <eDppBzF3EHA.2316@TK2MSFTNGP15.phx.gbl>
> <#1p7txJ3EHA.4068@cpmsftngxa10.phx.gbl>
> X-Newsreader: Groupwise 6.5
> From: Dany <Dany@newsgroups.com>
> Subject: RE: SP1 Problem SOAPException doesn't return quote and
Umlaute
> correcty
> Message-ID: <et$A51P3EHA.1524@TK2MSFTNGP09.phx.gbl>
> Newsgroups: microsoft.public.dotnet.framework.webservices
> NNTP-Posting-Host: gonzo.opus.ch 213.200.237.2
> Lines: 1
> Path:
>
cpmsftngxa10.phx.gbl!TK2MSFTNGXA01.phx.gbl!TK2MSFTNGP08.phx.gbl!TK2MSFTNGP09
> .phx.gbl
> Xref: cpmsftngxa10.phx.gbl
> microsoft.public.dotnet.framework.webservices:7978
> X-Tomcat-NG: microsoft.public.dotnet.framework.webservices
>
> Hi Dan
>
> The more I look into it the more I get the impression there is really

> something very wrong here in the .net Framework SP1. I've been
talking to
> an XML specialist and he noticed that it is not correct XML generated
when
> there is an Umlaut (ä, ö, ü) f.e. the error message for "Bitte
erfassen sie
> ein Abreisedatum für das Dokument." is represented in the SOAP fault
as
>
> <faultstring>Bitte erfassen sie ein Abreisedatum f&amp;#252;r das
> Dokument.</faultstring>
>
> Only when using a proxy to log the messages on the wire did I notice
this.
> Now when the XML Fault is read through an XML parser the &amp; is
correctly
> converted to a "&" and the result is "Bitte erfassen sie ein
Abreisedatum
> f&#252;r das Dokument." which is of course wrong after a conversion.
>
> So it looks like the SOAP fault string is converted twice by the .net

> Framework before landing on the wire. The first time everything would
be Ok
> and the ü character is converted to a valid character reference
"&#252;".
> But then this is replaced by character references once again which
results
> in "&amp;#252;" which is of course not a valid XML representation of
the
> "ü" character.
>
> In my view this is a serious bug in the handling of SOAPExceptions by
the
> .net Framework. I'd be really thankful if you could check with the
> serialization team. Reproducing the error should be quite simple.
Hope they
> have a solution then.
>
> Thanks & Regards
> Dany
>
>
> >>> Dan Rogers<danro@microsoft.com> 07.12.2004 21:36:25 >>>
> Hi Dany,
>
> I suspect that there is an encoding issue in the way you are loading
the
> XML. Is there an inconsistency between the XML encoding mark up and
the
> actual encoding on the wire? UrlEncoded XML is supported by the DOM,
but
> if someone somewhere along the way is loading what looks like the XML
into
> a unicode string without doing an encoding conversion first, issues
can
> happen.
>
> As for how/why this happened - I'm not that guy. For the most part,
1.1
> fixed bugs. It is possible that it introduced some as well. I'll
see what
> I can dig up from the serialization team - can't promise they have a
fix,
> but I'll see.
>
> Dan
> --------------------
> Date: Tue, 7 Dec 2004 14:00:36 +0100
> References: <uCfPi7f2EHA.3132@TK2MSFTNGP14.phx.gbl>
> <hzdWUQ92EHA.3200@cpmsftngxa10.phx.gbl>
> X-Newsreader: Groupwise 6.5
> From: Dany <Dany@newsgroups.com>
> Subject: RE: SP1 Problem SOAPException doesn't return quote and
Umlaute
> correcty
> Message-ID: <eDppBzF3EHA.2316@TK2MSFTNGP15.phx.gbl>
> Newsgroups: microsoft.public.dotnet.framework.webservices
> NNTP-Posting-Host: gonzo.opus.ch 213.200.237.2
> Lines: 1
> Path:
>
cpmsftngxa10.phx.gbl!TK2MSFTFEED01.phx.gbl!TK2MSFTNGP08.phx.gbl!TK2MSFTNGP15
> .phx.gbl
> Xref: cpmsftngxa10.phx.gbl
> microsoft.public.dotnet.framework.webservices:7944
> X-Tomcat-NG: microsoft.public.dotnet.framework.webservices
>
> Found out .net web service clients are just as helpless with handling
these
> character references.
>
> I've built a simple web service consumer in VC#. On a button click
the
> following code is executed:
> try
> {
> localhost.UmbrellaNTAppServer appserv = new
> localhost.UmbrellaNTAppServer();
> string result = appserv.HelloWorld();
> MessageBox.Show( result );
> }
> catch (Exception exc)
> {
>
> MessageBox.Show( "Error has occured: " + exc.Message );
> }
>
> If an exception is raised in the web service method saying i.e. 'The
method
> "HelloWorld" is under construction.' as the exception message. The
user
> will be shown 'Error has occured: The method &quot;HelloWorld&quot;
is
> under construction. In a real world example the user would be sure to

> complain about this display.
>
> Before SP1 everything was fine, but now handling of exception
messages
> coming from web service calls must be handled specifically.
> In normal SOAP responses (not SOAP faults) strings containing special

> characters are correctly sent. So if 'The method "HelloWorld" is
under
> construction.' is returned the client will also receive it unchanged
and
> without any character references.
>
> So I wonder why this inconsistent behaviour was built into the SP1 by
MS.
>
> Regards Dany
>
> >>> Dan Rogers<danro@microsoft.com> 06.12.2004 21:42:15 >>>
> Hi,
>
> What you are describing is the use of encoded strings. These are, to
my
> knowlege, valid in XML and the responsibility of the parser to
convert
> properly.
>
> What is the source of the problem, given that these are valid
> representations of the data? Is the client not able to process these

> properly as it should?
>
> Regards
>
> Dan Rogers
> Microsoft Corporation
>
> --------------------
> Date: Sat, 4 Dec 2004 13:43:30 +0100
> X-Newsreader: Groupwise 6.5
> From: Dany <Dany@newsgroups.com>
> Subject: SP1 Problem SOAPException doesn't return quote and Umlaute
correcty
> Message-ID: <uCfPi7f2EHA.3132@TK2MSFTNGP14.phx.gbl>
> Newsgroups: microsoft.public.dotnet.framework.webservices
> NNTP-Posting-Host: adsl-62-167-138-146.adslplus.ch 62.167.138.146
> Lines: 1
> Path:
>
cpmsftngxa10.phx.gbl!TK2MSFTFEED02.phx.gbl!TK2MSFTNGP08.phx.gbl!TK2MSFTNGP14
> .phx.gbl
> Xref: cpmsftngxa10.phx.gbl
> microsoft.public.dotnet.framework.webservices:7851
> X-Tomcat-NG: microsoft.public.dotnet.framework.webservices
>
> Our web service was working fine until we installed .net Framework
1.1
> service pack 1. Uninstalling SP1 is not an option because our largest

> customer says service packs marked as "critical" by Microsoft must be

> installed on their servers.
>
> Now german Umlaute (ä, ü, ö) and quotes are returned incorrectly
in SOAP
> fault responses.
> This can be easily verified:
>
> Implement the following in a web service method (just raises a
> SOAPException with a message text containing quote characters ("):
>
> string errMsg = "Here is a \"quote\".";
> throw new SoapException( errMsg,new
XmlQualifiedName("Server.Problem"));
>
> The following SOAP response is created when calling this
SOAP-Exception:
>
> HTTP/1.1 500 Internal Server Error.
> Server: Microsoft-IIS/5.1
> Date: Fri, 03 Dec 2004 13:02:23 GMT
> X-Powered-By: ASP.NET
> X-AspNet-Version: 1.1.4322
> Cache-Control: private
> Content-Type: text/xml; charset=utf-8
> Content-Length: 431
>
> <?xml version="1.0" encoding="utf-8"?>
> <soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"

> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xmlns:xsd="http://www.w3.org/2001/XMLSchema">
> <soap:Body>
> <soap:Fault>
> <faultcode>Server.Problem</faultcode>
> <faultstring>Here is a &quot;quote&quot;.</faultstring>
> <detail />
> </soap:Fault>
> </soap:Body>
> </soap:Envelope>
>
> All though the encoding should be utf-8 and "& q u o t ;" is not
correct.
> This worked fine in .net 1.1 without the SP1. The correct
<faultstring>
> should be:
> <faultstring>Here is a "quote".</faultstring>
>
> This problem also occurs when using german Umlaute: f.e. with an
exception
> message of "Fahrvergnügen". It is converted incorrectly to
> <faultstring>Fahrvergn&amp;#252;gen</faultstring>.
>
> This is a serious problem for us, because in german and french many
error
> messages will have special characters like ö,ä, é, à in them and
these are
> not correctly processed.
>
> Thanks for any help!
> Dany

Loading