Difference between revisions of "Tokenize"

Revision as of 22:15, 27 November 2005

Fills the $0 $1 $2 ... $N identifiers with tokens in <text> separated by character c

/tokenize <c> <text>

If you e.g. want to support different syntaxes, you can use /tokenize to bring it to a "standard format" (look at this script to get an example).

Example

; set a test string
var %mystring = this.is.my.string
; actually tokenize the string. 46 is the ascii number for . which is our seperator.
tokenize 46 this.is.my.string
; lets loop through all $1, $2, $3 ...
var %i = 1 
; $0 returns the total number of "$'s".
while (%i <= $0) {
 ; echo the current "$".
 echo -a %i $+ : $ [ $+ [ %i ] ]
 ; increase looping variable
 inc %i
}

This example will echo:

1: this
2: is
3: my
4: string

See Also

To work with tokens, take a look at the Token Identifier.