Re: javascript compression

From: benben (benhongh_at_yahoo.com.au)
Date: 11/30/04 

Date: Tue, 30 Nov 2004 20:34:51 +0800

Ifall you want is to reduce the size of the script (as your news title
prompts) then I have written you a tool in javascript, which will take off
all the unnecessary white spaces and line breaks (it catches lexical errors
as well :D). The script is appended at the end of this reply, just save it
to a .js file and run it in cscript.exe, as

cscript.exe jscompact.js inputfile.js outputfile.js

However if you want to obfuscate your script by prefixing all variables,
errors that are very hard to find is guaranteed. If you foresee your script
being used in mixed environment you should have prefixed your codes already
:). Think about the following code:

function proc_a()
{
    //...
}

function proc_b()
{
    var i = proc_a(); // error, proc_a() is renamed to abc_proc_a()
}

Were the functions renamed, say abc_proc_a() and abc_proc_b(), the function
call in proc_b will be invalid. Even if your obfuscator is powerful enough
to recode the inner call, there are still protential risks, for example
eval()'s or window.setTimeout()'s:

function proc_c()
{
    var exec_str = "proc_a()";
    eval(exec_str); // error, proc_a() is renamed to abc_proc_a()
    window.setTimeout("proc_b()", 1000); // error again!
}

My advice is, don't leave the risks to the user, keep them, solve them.

benben

> can anybody please advise me on the best way to obfuscate
> my javascript code. There is one little catch. The js
> files are going to be included within a page that I have
> no control over. To eleviate the confusion of variable
> names I would like to prefix all my new obfuscated
> variables with a pre determined tag e.g. _abc. Does
> anybody know of a javascript compression tool that will
> allow this, or of any good compression tools that are
> worth trying out.
>
> Many thanks

// jscompact.js

////////////////////////////////////////////////////////////////////////////
///////////
//option block
// refer to formal documentation

var RingBellOnFatalError = false;
var RingBellOnWarning = false;
var AbortOnFatalError = true;

////////////////////////////////////////////////////////////////////////////
///////////
// general helper functions

/////////////////////////////////////////////////////////////////
// function: assert()
// input: condition (meant to be the result of an assertion statement)
// process: call abort() if condition is not fulfilled
// output: will never return
function assert(condition)
{
 if (!condition) abort();
}

// The following two variables are added for multi-input support.
var defaultHf = null; //default input that abort(), etc handles
var openedFiles = new Array(); //opened file handlers, for post clean up

/////////////////////////////////////////////////////////////////
// function: abort()
// input: none
// process: terminates the program
// output: will never return
function abort()
{
 WScript.StdErr.WriteLine("!Aborting compilation...");

 //close file handles
 if (defaultHf.opened)
 {
  while (openedFiles.length > 0)
  {
   defaultHf = openedFiles.pop();
   if (!defaultHf.opened) continue;

   WScript.StdErr.WriteLine("!" + GetInputPositionMessage() + "Terminating
input...");
   CloseFileInput(defaultHf);
  }
 }

 WScript.Quit(-1);
}

/////////////////////////////////////////////////////////////////
// function: GetInputPositionMessage()
// input: none
// process: ...
// output: returns a string detailing the line and char position of default
input
function GetInputPositionMessage()
{
 if (defaultHf == null) abort();

 return "(" + defaultHf.name + ") line " + defaultHf.line + " character " +
defaultHf.character + " :";
}

// bell - character ASCII 7, for beeping service
var bell = String.fromCharCode(7);

/////////////////////////////////////////////////////////////////
// function: FatalError()
// input: msg (string message containing a description of error)
// process: issue a fatal error in compilation phases
// output: will never return; prints error message
function FatalError(msg)
{
 if (RingBellOnFatalError)
  WScript.StdErr.Write(bell);

 WScript.StdErr.WriteLine("!Fatal Error: " + GetInputPositionMessage() +
msg);

 if (AbortOnFatalError)
  abort();
}

/////////////////////////////////////////////////////////////////
// function: Warning()
// input: msg (string message containing a warning)
// process: issue a warning to the user.
// output: no return; prints warning message
function Warning(msg)
{
 if (RingBellOnWarning)
  WScript.StdErr.Write(bell);

 WScript.StdErr.WriteLine("!Warning: " + GetInputPositionMessage() + msg);
}

/////////////////////////////////////////////////////////////////
// function: MessageOut()
// input: msg (message)
// process: print message
// output: return a reference to itself for chained call.
function MessageOut(msg)
{
 WScript.StdOut.Write(msg);
 return MessageOut;
}

/////////////////////////////////////////////////////////////////
// function: SetDefaultInput()
// input: hf (handle of file input)
// process: set default input to hf
// output: none
function SetDefaultInput(hf)
{
 defaultHf = hf;
}

///////////////////////////////////////////////////////////////////////////
// input routines

/////////////////////////////////////////////////////////////////
// structure: InputStreamHandle
// input: streamObject (native stream object with ReadLine() and
AtEndOfStream implemented
function InputStreamHandle(streamObject)
{
 this.name = ""; //name of the input, usually the path of the file
 this.sobj = streamObject; //the actual stream object
 this.linebuffer = ""; //contains the currently reading line

 this.line = 0; //line number
 this.character = 0; //character position in the line
 this.eof = false; //End Of File flag, trivial. Use IsEOF() for querying
end of stream unless when debugging
 this.opened = true; //is the handle referring to an opened stream

 //put back support
 this.putbackBufferSize = 10; //can at most put back 10 characters
 this.putbackBuffer = new Array(); //putback buffer

 //token support
 this.putbackTokens = new Array(); //multi-input token putback support
}

/////////////////////////////////////////////////////////////////
// function: AtEndOfLine()
// input: isHandle (handle to input)
// process: checks if current reading position is at end of line
// output: boolean flag indicating if the position of reading is at end of
line
function AtEndOfLine(isHandle)
{
 return (isHandle.character == isHandle.linebuffer.length);
}

/////////////////////////////////////////////////////////////////
// function: FetchLine()
// input: isHandle (handle to input)
// process: Read a new line, if any, from the input stream, to the line
buffer
// output: boolean value indicating if a line has been actually read in.
function FetchLine(isHandle)
{
 if (IsEOF(isHandle))
 {
  ValidateISHandle(isHandle);
  return false;
 }

 isHandle.linebuffer = isHandle.sobj.ReadLine() + char_LineFeed;
 isHandle.line++;
 isHandle.character = 0;
 return true;
}

/////////////////////////////////////////////////////////////////
// function: GetNextCharFromLine()
// input: isHandle (handle to input)
// process: read a character from line buffer, increment the reading
position
// output: the character read
function GetNextCharFromLine(isHandle)
{
 var c;
 if (isHandle.putbackBuffer.length > 0)
  c = isHandle.putbackBuffer.pop();
 else
  c = isHandle.linebuffer.charAt(isHandle.character);

 isHandle.character++;

 return c;
}

/////////////////////////////////////////////////////////////////
// function: ValidateISHandle()
// input: isHandle (handle to input)
// process: validates the state of the passed in input handle
// output: none
function ValidateISHandle(isHandle)
{
 isHandle.eof = IsEOF(isHandle);
}

/////////////////////////////////////////////////////////////////
// function: GetChar()
// input: isHandle (handle to input)
// process: Read a character from the input
// correct the line and character position
// validate the handle state
// output: the character read
function GetChar(isHandle)
{

 if (IsEOF(isHandle))
  FatalError("unexpected end of file");

 if (AtEndOfLine(isHandle))
 {
  FetchLine(isHandle);
 }

 var c = GetNextCharFromLine(isHandle);
 ValidateISHandle(isHandle);

 //WScript.StdOut.WriteLine("GetChar: " + c.charCodeAt(0));

 return c;
}

/////////////////////////////////////////////////////////////////
// function: ReplaceCharacter()
// input: str - string in which a character is to be replaced by another
// pos - position of the character to be replaced
// character - new character to replace
// process: replace a character at position pos in string str with a new
character character
// output: string in which the character is replaced by another
function ReplaceCharacter(str, pos, character)
{
 if (str.charAt(pos) == character) return str;

 var part1 = str.substring(0, pos);
 var part2 = str.substring(pos + 1, str.length);

 return part1 + character + part2;
}

/////////////////////////////////////////////////////////////////
// function: Putback()
// input: isHandle - handle to input
// character - character to be put back to input
// process: put a character back to input stream
// output: always true
function Putback(isHandle, character)
{
 isHandle.character--;
 isHandle.putbackBuffer.push(character.charAt(0));
 return true;
}

/////////////////////////////////////////////////////////////////
// function: IsEOF()
// input: isHandle - handle to input
// process: verify if the input is at its end
// output: boolean value indication such
function IsEOF(isHandle)
{
 if (!AtEndOfLine(isHandle)) return false;

 return isHandle.sobj.AtEndOfStream;
}

// a set of characters for convenience
//used by the lexical analyzer
var char_LineFeed = '\n';
var char_FormFeed = String.fromCharCode(12);
var char_CarrageReturn = '\r';
var char_Space = ' ';
var char_Tab = '\t';
var char_Underscore = '_';

// file system object constants
var ForReading = 1;
var ForWriting = 2;
var ForAppending = 8;

var OpenAsDefault = -2;
var OpenAsASCII = 0;
var OpenAsUnicode = -1;
var Fso = new ActiveXObject("Scripting.FileSystemObject");

/////////////////////////////////////////////////////////////////
// function: OpenFileInput()
// input: path - path to file to be opened
// process: open a file and create a handle to it
// output: the handle
function OpenFileInput(path)
{
 var f = Fso.OpenTextFile(path, ForReading, false, OpenAsASCII);
 var hf = new InputStreamHandle(f);
 hf.name = path;
 ValidateISHandle(hf);

 openedFiles.push(hf);

 return hf;
}

/////////////////////////////////////////////////////////////////
// function: CloseFileInput()
// input: hf - handle of input
// process: close the filed handled by hf
// output: none
function CloseFileInput(hf)
{
 if (hf.opened)
 {
  hf.opened = false;
  hf.sobj.Close();
 }
}

////////////////////////////////////////////////////////////////////////////
//////
// tokenizer

/////////////////////////////////////////////////////////////////
// structure: Token()
// input: type - type of token
// assoc - associated information
function Token(type, assoc)
{
 this.type = type;
 this.assoc = assoc;
}

// tokens - an array of predifined tokens
var tokens = new Array();

// tokenIndexBase - base index for operator tokens
var tokenIndexBase = 60;

// PTN - incremental indexer for declaration convenience
var PTN = tokenIndexBase;

/////////////////////////////////////////////////////////////////
// function: IsPunctuator()
// input: str
// process: check if a string is a punctuator
// output: boolean value indicating so
function IsPunctuator(str)
{
 for (var i = 0; i < tokens.length; i++)
 {
  if (str.length <= tokens.length)
  {
   if (str == tokens[i].substring(0, str.length))
    return true;
  }
 }

 return false;
}

/////////////////////////////////////////////////////////////////
// function: ReadPunctuator()
// input: initial - the first character read in
// hf - handle to input
// process: read the longest punctuator possible from current position of
input
// output: returns a string containing the punctuator read
function ReadPunctuator(initial, hf)
{
 while (!IsEOF(hf))
 {
  var c = GetChar(hf);

  if (!IsPunctuator(initial + c))
  {
   Putback(hf, c);
   return initial;
  }

  initial += c;
 }

 return initial;
}

/////////////////////////////////////////////////////////////////
// function: RT()
// input: sample - a punctuator sample
// process: register a punctuator type of token
// output: returns the index to the token
function RT(sample) // register token
{
 PTN ++;
 tokens.push(sample);
 return PTN - 1;
}

//token definition

//basic token types
var tok_identifier = 10;
var tok_number = 11;
var tok_eof = 12;
var tok_string_literal = 13;
var tok_character_literal = 14;

//further punctuator type tokens.
//declared by calling RT(), see RT()

//arithmetic operators
var tok_plus = RT("+");
var tok_minus = RT("-");
var tok_multiply = RT("*");
var tok_divide = RT("/");
var tok_modulus = RT("%");

//implicit assignment
var tok_increment = RT("++");
var tok_decrement = RT("--");

//bitwise operators
var tok_bitwise_and = RT("&");
var tok_bitwise_or = RT("|");
var tok_bitwise_xor = RT("^");
var tok_left_shift = RT("<<");
var tok_right_shift = RT(">>");

//logical operators
var tok_and = RT("&&");
var tok_or = RT("||");
var tok_not = RT("!");

//comparison
var tok_greater_than = RT(">");
var tok_less_than = RT("<");
var tok_greater_equal = RT(">=");
var tok_less_equal = RT("<=");
var tok_equal = RT("==");
var tok_not_equal = RT("!=");

//assignment
var tok_assign = RT("=");
var tok_plus_assign = RT("+=");
var tok_minus_assign = RT("-=");
var tok_multiply_assign = RT("*=");
var tok_divide_assign = RT("/=");
var tok_modulus_assign = RT("%=");
var tok_bitwise_and_assign = RT("&=");
var tok_bitwise_or_assign = RT("|=");
var tok_bitwise_xor_assign = RT("^=");
var tok_left_shift_assign = RT("<<=");
var tok_right_shift_assign = RT(">>=");

//grouping
var tok_left_paren = RT("(");
var tok_right_paren = RT(")");
var tok_left_bracket = RT("[");
var tok_right_bracket = RT("]");
var tok_left_brace = RT("{");
var tok_right_brace = RT("}");

var tok_block_begin = tok_left_brace;
var tok_block_end = tok_right_brace;

//further punctuators that are not operators
var tok_semmi_colon = RT(";");
var tok_semmicolon = tok_semmi_colon;
var tok_colon = RT(":");
var tok_comma = RT(",");
var tok_period = RT(".");
var tok_destructor = RT("~");
var tok_dotdot = RT("..");
var tok_et_al = RT("...");

//also some unsupported operators for error handling and recovery
var tok_hash = RT("#");
var tok_back_slash = RT("\\");
var tok_question_mark = RT("?");
var tok_left_ushift_assign = RT("<<<=");
var tok_right_ushift_assign = RT(">>>=");
var tok_left_ushift = RT("<<<");
var tok_right_ushift = RT(">>>");

//comment
var tok_comment_start = RT("/*");
var tok_comment_end = RT("*/");
var tok_inlinecomment_start = RT("//");

// a set of character to declared, trivial
var charcode_a = 'a'.charCodeAt(0);
var charcode_A = 'A'.charCodeAt(0);
var charcode_z = 'z'.charCodeAt(0);
var charcode_Z = 'Z'.charCodeAt(0);
var charcode_0 = '0'.charCodeAt(0);
var charcode_9 = '9'.charCodeAt(0);

/////////////////////////////////////////////////////////////////
// function: IsNumeric()
// input: c - a character
// process: verify that c is numerical, i.e., one of 0,1,2,3,4,5,6,7,8,9
// output: boolean value indicating so
function IsNumeric(c)
{
 var c_code = c.charCodeAt(0);

 return (c_code >= charcode_0 && c_code <= charcode_9);
}

/////////////////////////////////////////////////////////////////
// function: IsBlankSpace()
// input: c - a character
// process: verify that c is considered a blankspace, i.e., [space] or
[tab]
// output:
function IsBlankSpace(c)
{
 c = c.charAt(0);
 return (c == char_Space || c == char_Tab)
}

/////////////////////////////////////////////////////////////////
// function: IsComment()
// input: c - first character already read in
// hf - handle to input
// process: verify if the input is at the start of a comment block
// output: returns 0 if not, 1 if a traditional comment block, or 2 if an
inline comment block
function IsComment(c, hf)
{
 if (c != '/') return 0;
 if (IsEOF(hf)) return 0;
 var d = GetChar(hf);

 if (d == '*')
 {
  return 1;
 }
 if (d == '/')
 {
  return 2
 }

 Putback(hf, d);
 return 0;
}

//this is a sequence of hexadecimal digits, trivial
var hexDigits = "0123456789abcdefABCDEF";

/////////////////////////////////////////////////////////////////
// function: IsHexDigit()
// input: c
// process: verify if c is a hex digit
// output: boolean value indicating so
function IsHexDigit(c)
{
 for (var i = 0; i < hexDigits.length; i++)
 {
  if (c == hexDigits.charAt(i))
   return true;
 }

 return false;
}

/////////////////////////////////////////////////////////////////
// function: ReadHexadecimalNumber()
// input: hf - handle of input
// process: read in the longest possible hexadecimal number
// output: a string literal of the hex number
function ReadHexadecimalNumber(hf)
{
 var hex = "";

 while (!IsEOF(hf))
 {
  var d = GetChar(hf);
  if (!IsHexDigit(d))
  {
   Putback(hf, d);
   return hex;
  }

  hex += d;
 }

 return hex;
}

/////////////////////////////////////////////////////////////////
// function: ReadNonDecimalNumber()
// input: initial - first character already read in
// hf - handle to input
// process: reading the longest possible non decimal number (integer) from
input
// output: string literal
function ReadNonDecimalNumber(initial, input)
{
 var c;
 var hf = input;

 while (!IsEOF(input))
 {
  c = GetChar(input);
  if (IsNumeric(c))
  {
   initial += c;
  }
  else
  {
   Putback(input, c);
   break;
  }
 }

 return initial;
}

/////////////////////////////////////////////////////////////////
// function: ReadNumber()
// input:
// process:
// output:
function ReadNumber(initial, input)
{
 var c;
 var hf = input;

 if (initial == '0')
 {
  if (IsEOF(hf)) return '0';

  var d = GetChar(hf);
  if (d == 'x') return "0x" + ReadHexadecimalNumber(hf);

  Putback(hf, d);
 }

 initial = ReadNonDecimalNumber(initial, hf);

 if (!IsEOF(hf))
 {
  var d = GetChar(hf);

  if (d == '.')
  {
   if (IsEOF(hf))
   {
    Putback(hf, d);
    return initial;
   }

   var e = GetChar(hf);

   if (!IsNumeric(e))
   {
    Putback(hf, e);
    Putback(hf, d);
    return initial;
   }

   initial += '.' + ReadNonDecimalNumber(e, input);
   return initial;
  }
  else
  {
   Putback(hf, d);
  }
 }

 return initial;
}

function IsAlpha(c)
{
 var c_code = c.charCodeAt(0);

 return (
  (c_code >= charcode_a && c_code <= charcode_z) ||
  (c_code >= charcode_A && c_code <= charcode_Z)
 );
}

function ReadIdentifier(initial, input)
{
 var c;
 while (!IsEOF(input))
 {
  c = GetChar(input);
  if (IsAlpha(c) || IsNumeric(c) || c == char_Underscore)
  {
   initial += c;
  }
  else
  {
   Putback(input, c);
   return initial;
  }
 }

 return initial;
}

function GetEscapeCharacter(c)
{
 switch (c)
 {
  case 'a': return String.fromCharCode(7);
  case 'b': return String.fromCharCode(8);
  case 'f': return String.fromCharCode(12);
  case 'v': return String.fromCharCode(11);
  case 'n': return '\n';
  case 'r': return '\r';
  case 't': return '\t';
 }

 return c;
}

function ReadStringLiteral(hf)
{
 var str = "";

 while (true)
 {
  if (IsEOF(hf)) break;

  var c = GetChar(hf);
  if (c == '\\')
  {
   if (IsEOF(hf))
    FatalError("\" (double quote) expected");

   c = GetEscapeCharacter(GetChar(hf));
  }
  else if (c == '\"')
  {
   break;
  }

  str += c;
 }

 var line = hf.line;
 var nexttok = GetToken(hf);
 if (nexttok.type == tok_string_literal)
 {
  //aggregation of if blocks: order of condition is critical,
  //do NOT use logical and operation.

  if (hf.line != line)
  {
   str += nexttok.assoc; //join the two string literals
   return str;
  }
 }

 PutbackToken(hf, nexttok);
 return str;
}

function ReadCharacterLiteral(hf)
{
 if (IsEOF(hf)) FatalError("character literal expected");

 var c = GetChar(hf);
 if (c == "\\")
 {
  if (IsEOF(hf))
   FatalError("character literal expected");

  c = GetEscapeCharacter(GetChar(hf));
 }

 if (IsEOF(hf)) FatalError("\' (single quote) expected");
 var e = GetChar(hf);
 if (e != '\'')
  FatalError("\' (single quote) expected");

 return c;
}

function ReadComment(hf)
{
 while (true)
 {
  if (IsEOF(hf)) return;

  var c = GetChar(hf);
  if (c == '*')
  {
   Putback(hf, c);
   var t = GetToken(hf);

   if (t.type == tok_comment_end) return;
  }
 }
}

function ReadInlineComment(hf)
{
 while (true)
 {
  if (IsEOF(hf)) return;

  var c = GetChar(hf);
  if (c == char_LineFeed) return;
 }
}

function TranslatePunctuator(p)
{
 for (var i = 0; i < tokens.length; i++)
 {
  if (p == tokens[i])
  {
   i += tokenIndexBase;
   return new Token(i, p);
  }
 }

 return new Token(-1, 0);
}

function GetToken(hf)
{
 if (hf.putbackTokens.length > 0) return hf.putbackTokens.pop();

 var c;

 //loop over redundant characters like space, tab, comments, linefeed, etc
 //and if already at end of file return eof
 //
 //This loop keeps the recurrence of GetToken() low throughout the
compilation
 while (true)
 {
  if (IsEOF(hf)) return new Token(tok_eof, "");
  c = GetChar(hf);

  if (IsBlankSpace(c))
   continue;

  if (c == char_LineFeed || c == char_FormFeed)
   continue;

  switch (IsComment(c, hf))
  {
   case 1:
    ReadComment(hf);
    continue;

   case 2:
    ReadInlineComment(hf);
    continue;
  }

  break;
 }

 //sieve the retrieved character

 //identifiers and numbers
 if (IsAlpha(c) || c == '_')
  return new Token(tok_identifier, ReadIdentifier(c, hf));

 if (IsNumeric(c))
  return new Token(tok_number, ReadNumber(c, hf));

 //comments
 if (c == '\'')
  return new Token(tok_character_literal, ReadCharacterLiteral(hf));

 if (c == '\"')
  return new Token(tok_string_literal, ReadStringLiteral(hf));

 //further process the input if a punctuator is found
 if (IsPunctuator(c))
 {
  var p = ReadPunctuator(c, hf);

  //get a proper token object with the puntuator, and return
  var t = TranslatePunctuator(p);
  return t;
 }

 //if the character goes this far, it must be invalid, issue a fatal error

 FatalError("\'" + c + "\'(ASCII " + c.charCodeAt(0) + ") is not a valid
operator" );
}

function PutbackToken(hf, token)
{
 hf.putbackTokens.push(token);
}

var keywords = new Array();
PTN += 10;
var keyword_indexbase = PTN;

function keyword(sample)
{
 PTN++;
 keywords.push(sample);
 return PTN - 1;
}

function GetKeywordIndex(key)
{
 for (var i = 0; i < keywords.length; i++)
 {
  if (keywords[i] == key)
   return i + keyword_indexbase;
 }

 return -1;
}

var tok_terminate = tok_semmicolon;

//keywords

//global keywords
var tok_import = keyword("import");
var tok_require = keyword("require");
var tok_using = keyword("using");
var tok_namespace = keyword("namespace");

//class
var tok_class = keyword("class");
var tok_struct = keyword("struct");
var tok_interface = keyword("interface");
var tok_enum = keyword("enum");
var tok_typedef = keyword("typedef");

//access control
var tok_public = keyword("public");
var tok_private = keyword("private");
var tok_protected = keyword("protected");
var tok_get = keyword("get");
var tok_set = keyword("set");

//typeid
var tok_typeid = keyword("typeid");

//declaration decorators
var tok_static = keyword("static");
var tok_virtual = keyword("virtual");
var tok_pure = keyword("pure");
var tok_const = keyword("const");

//build-in types
var tok_int = keyword("int");
var tok_char = keyword("char");
var tok_bool = keyword("bool");
var tok_double = keyword("double");
var tok_byte = keyword("byte");
var tok_string = keyword("string");
var tok_void = keyword("void");

//literals
var tok_null = keyword("null");
var tok_true = keyword("true");
var tok_false = keyword("false");

//control flow
var tok_if = keyword("if");
var tok_else = keyword("else");
var tok_switch = keyword("switch");
var tok_case = keyword("case");
var tok_default = keyword("default");
var tok_for = keyword("for");
var tok_while = keyword("while");
var tok_do = keyword("do");
var tok_break = keyword("break");
var tok_continue = keyword("continue");
var tok_throw = keyword("throw");
var tok_catch = keyword("catch");
var tok_try = keyword("try");
var tok_return = keyword("return");

var tok_integer = 30;
var tok_float = 31;
var tok_hex_integer = 32;

function TranslateToken(token)
{
 if (token.type == tok_identifier)
 {
  var k = GetKeywordIndex(token.assoc);
  if (k != -1)
  {
   token.type = k;
  }

  return;
 }

 if (token.type == tok_number)
 {
  var assoc = token.assoc;

  if (assoc.substring(0, 2) == "0x")
  {
   token.type == tok_hex_integer;
   return token;
  }

  var p = assoc.split('.');

  if (p.length > 1)
  {
   token.type = tok_float;
   return token;
  }

  token.type = tok_integer;
  return token;
 }

 switch (token.type)
 {
  case tok_left_ushift:
  case tok_left_ushift_assign:
  case tok_right_ushift:
  case tok_right_ushift_assign:
  case tok_dotdot:
  case tok_hash:
  case tok_back_slash:
   FatalError("\'" + token.assoc + "\' is not a valid operator.")
 }
}

function GetTokenEx(hf)
{
 var t = GetToken(hf);
 TranslateToken(t);
 return t;
}

////////////////////////////////////////////////////////////////////////////
///////////
// syntactical analyzer

PTN += 10;

function FreeTokenID()
{
 PTN++;
 return PTN;
}

function FormatString(str)
{
 var s = "";

 for (var i = 0; i < str.length; i++)
 {
  var c = str.charAt(i);

  switch (c)
  {
   case '\f': s += "\\f"; break;
   case '\n': s += "\\n"; break;
   case '\r': s += "\\r"; break;
   case '\t': s += "\\t"; break;
   case '\\': s += "\\\\"; break;
   case '\'': s += "\\\'"; break;
   case '\"': s += "\\\""; break;

   default:
    s += c;
  }

 }

 return "\"" + s + "\"";
}

function main()
{
 var inputfile = WScript.Arguments(0);
 var outputfile = WScript.Arguments(1);

 if (!Fso.FileExists(inputfile))
 {
  MessageOut(inputfile)(": file not found!\r\n");
  abort();
 }

 if (Fso.FileExists(outputfile))
 {
  MessageOut("deleting file: ")(outputfile)("\r\n");
 }

 var hf = OpenFileInput(inputfile);
 SetDefaultInput(hf);

 var outf = Fso.OpenTextFile(outputfile, ForWriting, true, OpenAsASCII);

 var addspace = false;

 while (true)
 {
  var token = GetTokenEx(hf);

  if (token.type == tok_eof) break;

  if (token.type == tok_string_literal || token.type ==
tok_character_literal)
  {
   token.assoc = FormatString(token.assoc);
  }

  switch (token.type)
  {
   case tok_colon: addspace = false; break;
   case tok_semmicolon: addspace = false; break;
   case tok_comma: addspace = false; break;
   case tok_period: addspace = false; break;
   case tok_left_paren: addspace = false; break;
   case tok_right_paren: addspace = false; break;
   case tok_left_bracket: addspace = false; break;
   case tok_right_bracket: addspace = false; break;
   case tok_left_brace: addspace = false; break;
   case tok_right_brace: addspace = false; break;
   case tok_question_mark: addspace = false; break;
   case tok_string_literal: addspace = false; break;
   case tok_character_literal: addspace = false; break;
   case tok_assign: addspace = false; break;
   case tok_equal: addspace = false; break;

   default:
    if (addspace)
     token.assoc = " " + token.assoc;
    addspace = true;
    break;
  }

  outf.Write(token.assoc);
 }

 outf.Close();
 CloseFileInput(hf);
}

main();

Relevant Pages

  • Re: Bug found in TstringList
    ... enlighten yourself with the knowledge that *standard* for the text files ... is to have *Control-Z* character as the EOF character. ... >> TStringList tries to split up the string that it loaded into multiple ...
    (alt.comp.lang.borland-delphi)
  • Re: Calculate the length of text in bytes
    ... The term "physical" means in this context, ... e.g. cyrillic would be 2 bytes per character. ... String stores each character as a 16 bit value. ... var a=String.fromCharCode; ...
    (comp.lang.javascript)
  • Re: Armenian, Sumerian, Burushaski, and Turkic languages
    ... - sort of a blank character, ... maybe it really is a Norwegian thing. ... Det var en liten gutt som gikk og græt og var så lei, ... med blanke ark og fargestifter tel. ...
    (sci.lang)
  • Re: regexp help
    ... Within that string I need to find whether a whole word exists. ... The word to test for is held in the var 'k'. ... contains one character that is not a word character, ... evangelism: XHTML 1.0 Strict ...
    (comp.lang.javascript)
  • Re: Trying to limit input ot certain numbers
    ... " part is and were in the help files I can learn about it. ... If you want one character from a string, then use the array-access notation to extract just that one character: ...
    (alt.comp.lang.borland-delphi)