Tokenize

Fills the $1 $2 ... $N identifiers with tokens in <text> separated by character c

/tokenize <c> <text>

If you e.g. want to support different syntaxes, you can use /tokenize to bring it to a "standard format" (look at this script to get an example).

Note that the $0 identifier is also filled with the total amount of tokens.

Example

; set a test string
var %mystring = this.is.my.string
; actually tokenize the string. 46 is the ascii number for . which is our seperator.
tokenize 46 %mystring
; lets loop through all $1, $2, $3 ...
var %i = 1 
; $0 returns the total number of "$'s".
while (%i <= $0) {
 ; echo the current "$".
 echo -a %i $+ : $ [ $+ [ %i ] ]
 ; increase looping variable
 inc %i
}

This example will echo:

1: this
2: is
3: my
4: string

See Also

• Token Identifiers for more information about tokens.