It is currently prohibitively difficult to use replacement tokens such as $EffectiveUsername$within strings in dynamic scripts, because it cannot be guaranteed that characters that would terminate the string are not included within the replacement value.
(This is not a new problem, and has been described at length in the past, e.g. here and here – this offers a possible alternative solution.)
For example, given the PowerShell $my_username = '$EffectiveUsername$', you cannot guarantee that the effective username does not contain the quote char (such as joe.o'brien), which would render the resulting PowerShell as $my_username = 'joe.o'brien' (which fails to parse, starting at brien).
Even using here-strings (@'...'@) is fraught (as much due to idioms of PowerShell); more cases are covered, but you still cannot guarantee that the closing '@ does not appear in the replacement value.
I don’t see any reasonable way for an end-user to craft a PowerShell script which can guard against this type of replacement, at least not without some assistance from the tool doing the replacement.
My suggestion to resolve this is for there to be some supported syntax within the replacement token which indicates that RoyalTS should itself quote and/or escape the token as part of the replacement. For example, using /q (i.e. quoted) at the end of the token name: $EffectiveUsername/q$.
The exact syntax element is not important – it just needs to be something reasonable that can be reliably located by RoyalTS and wouldn’t have worked before, so existing scripts are less likely to be affected. Other candidates might be $'EffectiveUsername'$ (which may be a little esoteric), or a functional-style $Quote(EffectiveUsername)$ (assuming brackets cannot currently be part of tokens); or even just anything prefixed with ‘Quoted’, i.e. $QuotedEffectiveUsername$.
The resulting replacement would then be interpreter-specific, i.e. for each of the interpreters you support, you define the best non-interpolated string-quoting function you can, and apply its transformation to the existing replacement value.
e.g. for PowerShell, the replacement strategy is probably to escape any single-quote by doubling it, then prepend and append a single-quote, i.e. if $EffectiveUsername$ returns joe.o'brien, $EffectiveUsername/q$ would return 'joe.o''brien'. The PowerShell script we would then write in the dynamic folder definition would become $my_username = $EffectiveUsername/q$ (note how the PS script now does not provide its own quotes).
Some potential interpreter-specific approaches are:
| Intepreter | Strategy | Example | Possible C# Implementation |
|---|---|---|---|
| PowerShell | duplicate squo; wrap with squo | 'joe.o''brien wrote "Hello" to C:\test.txt' |
string.Concat("'", value.Replace("'", "''"), "'"); |
| JSON/JavaScript | duplicate bksl; replace newlines with bksl+r and/or bksl+n; replace dquo with bksl+dquo; wrap with dquo (or use the JSON Serializer in your codebase’s libraries) | "joe.o'brien wrote \"Hello\" to C:\\test.txt" |
string.Concat("\"", value.Replace("\\", "\\\\").Replace("\r", "\\r").Replace("\n", "\\n").Replace("\"", "\\\""), "\""); |
| Perl/PHP/Ruby† | duplicate bksl; replace squo with bksl+squo; wrap in squo | 'joe.o\'brien wrote "Hello" to C:\test.txt' |
string.Concat("'", value.Replace("\\", "\\\\").Replace("'", "\\'"), "'"); |
| Bash | replace squo with magic sequence squo+dquo+squo+dquo+squo; wrap in squo | 'joe.o'"'"'brien wrote "Hello" to C:\\test.txt' |
string.Concat("'", s.Replace("'", "'\"'\"'"), "'"); |
| Python† | duplicate bksl; replace newlines with bksl+r and/or bksl+n; replace squo with bksl+squo; wrap with squo (the JS version may also be suitable) | 'joe.o\'brien wrote "Hello" to C:\test.txt' |
string.Concat("'", s.Replace("\\", "\\\\").Replace("\r", "\\r").Replace("\n", "\\n").Replace("'", "\\'"), "'"); |
(where bksl = \; squo = '; dquo = "; r and n are literal)
† Perl and Ruby have alternative syntax, e.g. q'...' and %q'...', but I’m not sure it adds anything here to make dedicated versions; similarly Python has r'...' but I don’t think it can escape a backslash at the end of a value, so is discounted.
A filthy, rough C# proof-of-concept (written in LINQPad) is supplied within the expandable section here...
// regex to find $Tokens$ within dynamic scripts
var tokenOptions = RegexOptions.ExplicitCapture|RegexOptions.CultureInvariant|RegexOptions.Compiled;
var tokenLocator = new Regex(@"[$](?<tok>\w+)(?<quo>[/][Qq])?[$]", tokenOptions);
// some example token values (you would use your existing token resolver)
var tokens = new Dictionary<string, string>(StringComparer.OrdinalIgnoreCase)
{
{"EffectiveUsername", @"joe.o'brien"},
{"EffectivePassword", @"thisIsJoe'sSecretPassword""OK""?!"},
{"Notes", @"This is very likely
to have ""newlines"", \backslashes and 'other' nasty
characters which we should be aware of,
especially at the end of lines,\
withi
n existing words
or even 'at' ""the"" end $of the string\"}
};
// interpreter-specific quotation rules (or, use an abstract function in any base Interpreter class)
var quoters = new Dictionary<string, Func<string, string>>(StringComparer.OrdinalIgnoreCase)
{
{string.Empty, s => s},
{"powershell", s => string.Concat("'", s.Replace("'", "''"), "'") },
{"bash", s => string.Concat("'", s.Replace("'", "'\"'\"'"), "'") },
{"php", s => string.Concat("'", s.Replace("\\", "\\\\").Replace("'", "\\'"), "'") },
{"perl", s => string.Concat("q'", s.Replace("\\", "\\\\").Replace("'", "\\'"), "'") },
{"ruby", s => string.Concat("%q'", s.Replace("\\", "\\\\").Replace("'", "\\'"), "'") },
{"python", s => string.Concat("'", s.Replace("\\", "\\\\").Replace("\r", "\\r").Replace("\n", "\\n").Replace("'", "\\'"), "'") },
// there is probably a more-appropriate JSONSerializer method to catch all nasties; anyway...
{"javascript", s => string.Concat("\"", s.Replace("\\", "\\\\").Replace("\r", "\\r").Replace("\n", "\\n").Replace("\"", "\\\""), "\"") },
};
// function to locate tokens within dynamic scripts and replace with their resolved value
var tokenizer = new Func<string, string, string>((script, lang) => {
// use the right quoter implementation depending on language
var qfunc = quoters.TryGetValue(lang, out var q) ? q : quoters[string.Empty];
// use the regex to replace all instances in a single call
return tokenLocator.Replace(script, m =>
{
// replace tokens.TryGetValue with whatever returns the replacement value, given its token name
if (tokens.TryGetValue(m.Groups["tok"]?.Value, out var tokenValue))
{
if (m.Groups["quo"].Success)
{
tokenValue = qfunc.Invoke(tokenValue);
}
return tokenValue;
}
return m.Value;
});
});
// some sample scripts with as many nasties as I can think of...
var exampleScripts = new Dictionary<string, string> {
////////////////////////////////////////////////////////////////////////////////////////////
{"perl", @"
This is my $EffectiveUsername$. There are many others like it but this one is mine.
If I want to use it as a string literal, I can do so like this:
my $user = $EffectiveUsername/q$;
my $pwd = $EffectivePassword/q$;
See...?
Now, to see how this behaves with a multi-line value like a notes field:
Bare:
$Notes$
As a script element:
my $notes = $Notes/q$;
Did that work out OK...?"},
////////////////////////////////////////////////////////////////////////////////////////////
{"powershell", @"
This is my $EffectiveUsername$. There are many others like it but this one is mine.
If I want to use it as a string literal, I can do so like this:
$user = $EffectiveUsername/q$
$pwd = $EffectivePassword/q$
See...?
Now, to see how this behaves with a multi-line value like a notes field:
Bare:
$Notes$
As a script element:
$notes = $Notes/q$
Did that work out OK...?"},
////////////////////////////////////////////////////////////////////////////////////////////
{"javascript", @"
This is my $EffectiveUsername$. There are many others like it but this one is mine.
If I want to use it as a string literal, I can do so like this:
{""user"": $EffectiveUsername/q$,
""pwd"": $EffectivePassword/q$}
See...?
Now, to see how this behaves with a multi-line value like a notes field:
Bare:
$Notes$
As a script element:
var notes = $Notes/q$;
Did that work out OK...?"},
////////////////////////////////////////////////////////////////////////////////////////////
{"bash", @"
This is my $EffectiveUsername$. There are many others like it but this one is mine.
If I want to use it as a string literal, I can do so like this:
v_user=$EffectiveUsername/q$
v_pwd=$EffectivePassword/q$
See...?
Now, to see how this behaves with a multi-line value like a notes field:
Bare:
$Notes$
As a script element:
v_notes=$Notes/q$
Did that work out OK...?"},
////////////////////////////////////////////////////////////////////////////////////////////
{"python", @"
This is my $EffectiveUsername$. There are many others like it but this one is mine.
If I want to use it as a string literal, I can do so like this:
user = $EffectiveUsername/q$
pwd = $EffectivePassword/q$
See...?
Now, to see how this behaves with a multi-line value like a notes field:
Bare:
$Notes$
As a script element:
notes = $Notes/q$
Did that work out OK...?"}
////////////////////////////////////////////////////////////////////////////////////////////
};
// show the output in LINQPad
exampleScripts.Select(pair => new {
Lang = pair.Key,
Entered = pair.Value,
Result = tokenizer.Invoke(pair.Value, pair.Key)
}).Dump();
Hope this gives you some ideas…