Re: Difficulties in deriving a truly universal GUI Scripting Language

I spent a lot of time worrying about user-interface problems. I used to be able to do
protocol analysis, although it has been decades since I last tried this and I'd have to
come up to speed again.

As I said in my earlier response, it is easy to confuse great technology with a great
solution. To my pain, I learned the hard way that they are not the same. Great
technology doesn't sell well. Great solutions (which we didn't have) will. Great
solutions require great techology. Great technology does not produce great solutions in
and of itself.
joe

On Mon, 21 Aug 2006 06:04:58 -0500, "Peter Olcott" <olcott@xxxxxxx> wrote:


"Joseph M. Newcomer" <newcomer@xxxxxxxxxxxx> wrote in message
news:u6n6e25dli6fibcoheivk151fkjicoaeao@xxxxxxxxxx
See below...
On Wed, 16 Aug 2006 12:34:09 -0500, "Peter Olcott" <olcott@xxxxxxx> wrote:


"Joseph M. Newcomer" <newcomer@xxxxxxxxxxxx> wrote in message
news:a2i6e2lulnaf3qvevprd9gvd4hj8f6etnk@xxxxxxxxxx
See below...
On Tue, 15 Aug 2006 20:13:35 -0500, "Peter Olcott" <olcott@xxxxxxx> wrote:


"Joseph M. Newcomer" <newcomer@xxxxxxxxxxxx> wrote in message
news:9hp4e2lufbla2bpg0fkc0evn4c8r4jj32n@xxxxxxxxxx
A caret means whatever the app *wants* the caret to mean. The APIs that
create
the caret

From your discussion below it looks more like this statement means that most
every aspect of the caret's meaning must be implemented from scratch.
****
True
****
It could
possibly be used in entirely unconventional ways.
****
I thought of another problem dealing with insert/overwrite mode, which does
confuse the
interpretation, but yes, it *is* possible, although unlikely, the caret
could
have other
meanings.
****
I would also further estimate
that almost no one ever violates the conventional meaning.
****
True, but as I pointed out concerning horizontal scrolling there can be
serious and
unexpected behavior of the caret even within those guidelines.
****
I would also estimate
the operating system helps to enforce this conventional meaning to some
extent,
****
Not really
****
not permitting oddly shaped or sized carets,
****
The caret can be a bitmap of arbitrary shape and size. Consider that there
are often
special carets to indicate overwrite mode, and in the case of Word, italic
mode. These
are entirely at the discretion of the application
****
and not allowing more than one
caret at a time.
****
This is the only limitation enforced by the operating system.
****
My estimate is that the caret is simply the MS Windows
equivalent of the MSDOS text cursor.
****
In the abstract, yes. At the detail level, it can vary. For example, in
our
"stupid
glass tty" emulator, we used a caret that was a character block size.
****
Are these estimates correct?
****
As indicated above.

Now the overwrite problem is a potentially serious one. There is no way to
determine if
the window containing the caret is going to accept characters (readonly),
insert them
("normal" mode) or replace the character to the right (or under) the caret
(overwrite).
Even the rich edit control, in one of the most glaring API oversights I've
seen in a long
time, has no way to either report its overwrite state or allow it to be
explicitly set.
This can lead to interesting problems in scripting, particularly if the
state
is set by
some mechanism other than coming up by default in insert state (for example,
the
application comes up in the mode it was last in)..
****

In that case the script author would have to have the script do the same thing
that people would do, look for an onscreen indication of the current mode, or
simply try it an see what happens.
****
Note that this adds whole new dimensions to the complexity of scripting. Not
that it
can't be done, but emulating human behavior--perception and feedback--has been
notoriously
difficult for the field of Artificial Intelligence.

For any particular application version on any specific platform, there would be
a simple fixed way to accomplish this. If it was possible for a person to tell
what was needed, then the script would simply use this same method. There is no
reason why the script would ever need to have AI technology, it would simply
rely upon the script programmer to instruct it what to do in each specific case.
If the only way to tell whether the mode is overwrite or insert is by trying it,
then the script could simply try it and see what happens.

****


merely create a graphical artifact and position it. Interpreting that
artifact and what
it means requires code exist that detects the caret position, derives the
textual location
information from that, and then proceeds to do something in that vicinity.
Back when I
taught the introductory Win32 course, we actually had the students create
a
micro-text-editor (using fixed-pitch font to simplify the
coordinate-to-character
mapping). Their job was to take a mouse click, convert it to a character
cell
coordinate,
and place the caret on the screen at the start of that cell. If the
window
scrolled, they
had to move the caret to follow. If the user typed something, they had to
figure out
where in the text the caret represented information and put the new text
there. Then they
had to add highlighting, and maintain the selection *and* the caret.
These
are all purely
graphical artifacts, and the code that gives them meaning is just code.
So
we
all work by
the conventions. Concepts like overstrike vs. insert mode are implemented
entirely by
code.
joe

On Tue, 15 Aug 2006 18:49:46 -0500, "Peter Olcott" <olcott@xxxxxxx> wrote:


"Joseph M. Newcomer" <newcomer@xxxxxxxxxxxx> wrote in message
news:fuk4e2pi9k40t63dspkft1aa5gcmsja18m@xxxxxxxxxx
But note that the caret is a graphical artifact with no semantic
content.
What the caret
means is whatever the control that is displaying it chooses to have it
mean.
I see
possible risks here.


I thought that the caret's entire purpose for being was to come up with an
alternative term than cursor, so that the mouse could use the cursor. This
purpose includes showing the current location of text data entry, or text
editing. I was unaware of any other purposes.

Now, the next issue is what is meant by "re-examining the screen". Do
you
specify in the
scripting language "I want this caret before the 'i' in 'is'" and let it
do
whatever it
has to do to achieve that (and I can think of a lot of ways to get the
wrong
result here)

In the case that I have in mind, the scripting language would always be
able
to
"see" what's on the screen. It could also determine the precise pixel
coordinate
location of every onscreen element including the mouse cursor, and the
text
caret. It could also control the mouse and the keyboard.

If the text cursor tended to jump around a little bit immediately after
entering
an editbox, then the currently displayed editbox would be obtained
immediately
after the caret enters the editbox. From this information, the script
could
redetermine the relative position of the text cursor to the intended
target
of
manipulation by the script.

or does the scripting language have to contain a loop by which the
scripter
writes out
what needs to be done (same problems of getting the wrong answer, but
more
work).

Automated "black box" testing harnesses were challenging even in the
days
of
text-only
scripting, where we had to look for responses from the compiler and
such.
An
unexpected
response could ruin a whole night's testing run because the script had
no
idea
how to
handle it. Ultimately it all depends on how much the script has to be
"perfect" and how
much it only matters that *something* happened.
joe

Unforeseen circumstances will always be a problem for any script
programmer
to
some extent until AI is much more fully developed. When I refer to a truly
universal scripting language, I am referring to a language capable of
controlling essentially every single program that runs under modern
operating
system versions, including old MSDOS programs.


On Tue, 15 Aug 2006 16:12:08 -0500, "Peter Olcott" <olcott@xxxxxxx>
wrote:


"Joseph M. Newcomer" <newcomer@xxxxxxxxxxxx> wrote in message
news:vgu3e2135n02nmvl2m5776cmcbmqfg2jn6@xxxxxxxxxx
Just another set of glitches I thought of, in rereading that article.
Consider an edit
control with horizontal scrolling. You click into it, between a
couple
characters,
expecting that's where the caret is going to end up. For example

[This is a t]est

where the est is off the end of the control. You want to change 'is'
to
'was', so you
click between the i and the s. You might find something like

Thi[s is a |test]

where the | indicates the caret position. This is because horizontal
edit
controls tend
of spontaneously scroll underneath you. I'd suggest the problem of
recovering
from this
is going to be at least one problem in writing a script.

You would merely have to re-examine the screen display after every input
action
whenever you encounter a control with this behavior. You might also need
to
have
a way to determine the current location of the caret, if the screen
caret
was
capable of adjusting its precise position from where it was placed.
There
are
API calls to determine the precise location of the mouse cursor and
screen
caret, aren't there?


Another is the fact that you might have to search for the phrase "This
is
a
test", but it
doesn't actually appear on the screen. But it might. Or not. For
example,
if the user
has selected "large fonts", you will have a potentially different
screen
display than for
the default fonts. Now, in the case of an edit control, GetWindowText
will
suffice, but
consider a situation where a listbox or combobox has the text scrolled
off
the
edge of the
screen and is owner-draw. A human would see ambiguous strings and
know
to
scroll, but how
would a scripting language? What if the programmer actually failed to
provide
a
horizontal scrollbar in the listbox?

The scripting language might not know to scroll, but the author of the
script
might know to tell the script to look for scroll-bars, and to determine
the
attributes of the screen display fonts. Automated testing could change
things
such as the screen font, and see if the correct results are consistently
produced, reporting any errors in the end of its run.

joe

On Tue, 15 Aug 2006 11:28:21 -0500, "Peter Olcott" <olcott@xxxxxxx>
wrote:

http://groups.google.com/group/microsoft.public.vc.mfc/browse_thread/thread/32f6ea916eb799d8/74db496855f60a0f?#74db496855f60a0f

The above post has some really good insights into this problem. I want
to
delve
a little deeper into this question, and see if any of the prior
insights
have
changed. It looks like one of the most difficult aspects of this
problem
is
providing a way so that the Mouse can always "see" where to click.

Joseph M. Newcomer [MVP]
email: newcomer@xxxxxxxxxxxx
Web: http://www.flounder.com
MVP Tips: http://www.flounder.com/mvp_tips.htm

Joseph M. Newcomer [MVP]
email: newcomer@xxxxxxxxxxxx
Web: http://www.flounder.com
MVP Tips: http://www.flounder.com/mvp_tips.htm

Joseph M. Newcomer [MVP]
email: newcomer@xxxxxxxxxxxx
Web: http://www.flounder.com
MVP Tips: http://www.flounder.com/mvp_tips.htm

Joseph M. Newcomer [MVP]
email: newcomer@xxxxxxxxxxxx
Web: http://www.flounder.com
MVP Tips: http://www.flounder.com/mvp_tips.htm

Joseph M. Newcomer [MVP]
email: newcomer@xxxxxxxxxxxx
Web: http://www.flounder.com
MVP Tips: http://www.flounder.com/mvp_tips.htm

Joseph M. Newcomer [MVP]
email: newcomer@xxxxxxxxxxxx
Web: http://www.flounder.com
MVP Tips: http://www.flounder.com/mvp_tips.htm
.

Loading