ANSI escape handling and PowerShell

User scenario

Since we've added more ANSI escape sequences for coloring in PS7, if you pipe or redirect that text output, the embedded ANSI escape sequences are sent down the pipeline. This can make log files harder to read. Also, for those wanting to leverage ANSI escape sequences, they are difficult to create/read for scripters and also harder to use formatting presets rather than hardcoding specific colors.

Fundamental philosophy

On Linux, if you use ls --color then different file types use ANSI escape sequences as color indicators. If you pipe this output to less, then you get paging while retaining the color information. If you redirect this output to a file, that file contains the ANSI escape sequences. If you then use the cat command on the file, you see the coloring as the ANSI escape sequences are rendered by the terminal.

On macOS, the similar command is ls -G, however, piping this output to less or redirecting to a file you lose the ANSI escape sequences so it is just plain text. I believe ls on macOS is detecting if output is redirected and turning off coloring automatically.

So for PowerShell, we should have a consistent experience regardless if Windows, Linux, or macOS and interop with native commands that may output ANSI escape sequences in addition to cmdlets/formatting.

Since ANSI escape sequences are de facto standardized, we should not introduce a new intermediate format and should just embed ANSI escape sequences within strings. This will ensure compatibility with other tools that emit or handle ANSI escape sequences.

Proposed technical implementation details

A new automatic variable $PSStyle will be added:

   TypeName: System.Management.Automation.PSStyle

Name            MemberType Definition
----            ---------- ----------
Reset           Property   string AttributesOff {get;set;}
Background      Property   System.Management.Automation.PSStyle+BackgroundColor Background {get;set;}
Blink           Property   string Blink {get;set;}
BlinkOff        Property   string BlinkOff {get;set;}
Bold            Property   string Bold {get;set;}
BoldOff         Property   string BoldOff {get;set;}
Foreground      Property   System.Management.Automation.PSStyle+ForegroundColor Foreground {get;set;}
Formatting      Property   System.Management.Automation.PSStyle+FormattingData Formatting {get;set;}
Hidden          Property   string Hidden {get;set;}
HiddenOff       Property   string HiddenOff {get;set;}
OutputRendering Property   System.Management.Automation.OutputRendering OutputRendering {get;set;}
Reverse         Property   string Reverse {get;set;}
ReverseOff      Property   string ReverseOff {get;set;}
Standout        Property   string Standout {get;set;}
StandoutOff     Property   string StandoutOff {get;set;}
Underline       Property   string Underlined {get;set;}
UnderlineOff    Property   string UnderlinedOff {get;set;}

$PSStyle

The base members return ANSI escape sequences mapped to their names. These are also settable so the user can change bold to underlined, for example. This makes it easier for scripters to author decorated strings with tab completion:

"$($PSStyle.Background.LightCyan)Power$($PSStyle.Underlined)$($PSStyle.Bold)Shell$($PSStyle.Reset)"

$PSStyle.OutputRendering

This is of type System.Management.Automation.OutputRendering which is an enum with the values:

Automatic This is the default. If the host supports VirtualTerminal, then ANSI is always passed as-is, otherwise plaintext
ANSI ANSI is always passed through as-is
PlainText ANSI escape sequences are always stripped so that it is only plain text
HostOnly This would be the macOS behavior where redirected or piped output the ANSI escape sequences are removed

$PSStyle.Formatting

This effectively replaces $Host.PrivateData as the way to read or configure colors for formatting rendering. $Host.PrivateData will continue to exist for backwards compatibility, but is not connected to $PSStyle.Formatting.

   TypeName: System.Management.Automation.PSStyle+FormattingData

Name         MemberType Definition
----         ---------- ----------
Debug        Property   string Debug {get;set;}
Error        Property   string Error {get;set;}
ErrorAccent  Property   string ErrorAccent {get;set;}
FormatAccent Property   string FormatAccent {get;set;}
Verbose      Property   string Verbose {get;set;}
Warning      Property   string Warning {get;set;}

One difference here is that instead of being of type ConsoleColor, these are all strings and doesn't separate foreground and background colors. This means that a single member can have foreground and background colors defined as well as other attributes like bold, underlined, etc...

Scripters can easily leverage this:

"$($PSStyle.Formatting.ErrorAccent)Power$($PSStyle.Formatting.Verbose)Shell$($PSStyle.AttributesOff)"

$PSStyle.Foreground and $PSStyle.Background

These members contain the standard 16 console colors as well as a RGB() methods to specify 24-bit color. For the colors, the values are settable and because they are strings can be any string content and any number of ANSI escape sequences.


   TypeName: System.Management.Automation.PSStyle+ForegroundColor

Name         MemberType Definition
----         ---------- ----------
Rgb          Method     string Rgb(byte red, byte green, byte blue), string Rgb(int rgb)
Black        Property   string Black {get;set;}
Blue         Property   string Blue {get;set;}
Cyan         Property   string Cyan {get;set;}
DarkGray     Property   string DarkGray {get;set;}
Green        Property   string Green {get;set;}
LightBlue    Property   string LightBlue {get;set;}
LightCyan    Property   string LightCyan {get;set;}
LightGray    Property   string LightGray {get;set;}
LightGreen   Property   string LightGreen {get;set;}
LightMagenta Property   string LightMagenta {get;set;}
LightRed     Property   string LightRed {get;set;}
LightYellow  Property   string LightYellow {get;set;}
Magenta      Property   string Magenta {get;set;}
Red          Property   string Red {get;set;}
White        Property   string White {get;set;}
Yellow       Property   string Yellow {get;set;}

PowerShell engine changes

The formatting system, pipelining, and redirection will be updated to respect the value of $PSStyle.OutputRendering.

StringDecorated type

This is an internal type to handle ANSI escaped strings.

Constructor

Single constructor taking a string as input.

`bool IsDecorated` property

Returns if the string contains ANSI escape sequences based on if the string contains ESC or C1 CSI.

`int Length` property

Returns the length of just the text content of the string sans ANSI escape sequences.

`StringDecorated Substring(int contentLength)` method

Returns a substring starting at index 0 up to the contentLength for content that is not part of an ANSI escape sequence. This is needed for table formatting to truncate strings preserving ANSI escape sequences that don't take up printable character space.

`string ToString()` method

Returns the plaintext version of the string.

`string ToString(bool Ansi)` method

Returns the raw ANSI embedded string if Ansi parameter is true, otherwise returns plain text with ANSI escape sequences removed.

`Out-String` update

To enable scripts to easily allow plain text redirection, Out-String will have a -RemoveAnsi switch.

`Select-String` update

By default, it would make sense for Select-String to ignore ANSI escape sequences, perhaps a -IncludeAnsi switch should be added.

$Host.PrivateData

This will be available for legacy scripts/modules that read from it. However, the engine changes (and console host changes) would mean these settings will no longer be observed and instead use the settings from $PSStyle.

Alternate considerations

Original prototype was built on top of System.CommandLine.Rendering, however, that is still a work in progress and is not targeted to be 1.0 in time for 7.1. It is also geared towards C# developers and the user experience is not great for scripts.

Related Issues

#10811
#7744
#3611

Possibly stupid question here: Will I be able to use Export-Clixml on the $PSSyle automatic variable to "round-trip" my thoroughly tricked-out and gorgeous settings to a file that I could use to move them to another environment? Will the ANSI escape codes in the string members get serialized in such a way as to not trip up version control systems? In any case, this seems like a fabulous idea!

Kelly Adams (@adamskt)

Kelly Adams (@adamskt) good question. I don't see any reason why you couldn't serialize $PSStyle to Clixml and deserialize it to overwrite $PSStyle as a way to have the same settings across systems.

So in summary.
(1) A standard, fairly readable way for scripts to do formatting
(2) A simple way to turn off ANSI formatting whether the author used (1) or did it their own way
That solves what I see as the two main problems with ansi codes (wanting to turn it off - sometimes - and ugly scripts) .

I liked the early draft of this idea, and I like this even more. A simple thumbs up wasn't enough :-)

Since common parameters set the preference variables for the local scope, what would your view be on adding a common parameter which sets output-rendering for a single command ? Among other things that would mean if one really hates the authors choice of colors a simple entry in $PSDefaultParamterValues gets rid of it.

I also wonder if more choices should be available so that a user can set their own warning, error, emphasis styles and the author then says "format this as a warning" not "use orange text"

If PowerShell is going to tackle theming I'd really really like to see some extra settings that a TUI might be able to tap into. Taking bootstrap as an example, I'd like to see the styles be under names like:

Primary
Secondary
Success
Danger
Warning
Info
Light
Dark
Muted

Every time I've started making a TUI of any sort, trying to make something that looks nice and also keeps to the spirit of the user's color choices is very challenging.

Patrick Meinecke (@SeeminglyScience)

Re terminology: I wonder if we should use "Virtual Terminal (VT) [Escape] Sequences" rather than "ANSI escape codes" in order to avoid confusion with the "ANSI" character encoding - though, arguably, that confusion is more likely for Windows users and, conversely, Unix users may be less familiar with the term "Virtual Terminal".

FWIW, looking at a few man pages for the various Unix utilities capable of producing colored output suggests that they rarely mention the coloring mechanism and often just talk about "color", and less frequently "escape sequences".

Avoiding "ANSI" would also mean: string ToString(bool Ansi) -> string ToString(bool decorated)

Re output modes (controlling when to use the sequences, $PSStyle.OutputRendering):

Note that GNU ls (and implicitly also macOS/BSD ls) has 3 modes (as do both GNU and macOS/BSD grep):

auto (the default): if stdout is connected to a (VT-enabled) terminal, use color; otherwise (redirected output (pipe, file)) don't.
never: never use color
always: always use color, irrespective of where the output goes.

GNU ls without a color option defaults to auto; using --color is the same as --color=always (the other options obviously being --color=never and --color=auto); note that this default is an unfortunate choice; GNU and macOS/BSD grep more sensibly interpret --color as --color=auto.

macOS/BSD ls, via arguments, offers auto behavior (only) via -G (default is never); environment variable CLICOLOR_FORCE can be set to achieve always behavior.

In short: I think these modes are sufficient and I think they're worth adopting - including the terminology.

Use of environment variables, new CLI parameter, new common parameter to control coloring:

We should consider exposing setting $PSStyle.OutputRendering and the potential theming proposed by Patrick Meinecke (@SeeminglyScience) via environment variables as well, so that there's an easy way for automation environments that call the PowerShell CLI to preset unconditional (always) coloring (e.g., from a POSIX-like shell; name negotiable, but it should be reasonably short: PSSTYLE_MODE=always pwsh -NoProfile -File ...)

Update: #10811 (comment) mentions Jason Shirk (@lzybkr) proposing a new CLI parameter:

Jason Shirk (@lzybkr) had a good suggestion to have a param on pwsh like -color with always, auto, never values.

James O'Neill (@jhoneill) above suggested a new common parameter for cmdlet-individual control (caveat is that unless this parameter is opt-in, conflicts with existing user-defined parameters are possible).

Joel Bennett (@Jaykul)

This already seems kind of similar to what Joel Bennett (@Jaykul) has done with PANSIES, only less flexible and harder to work with. In that module, a PSProvider is added that is capable of generating escape codes for common (named) colors as well as arbitrary hex color values. That can then be used mid-string to insert values in a flexible way.

If users are still going to have to look up the escape sequences for these settings / download a third party module to handle that anyway, I'm not sure this really adds much. Having a base theming framework is a good start, but without some more user-friendly features (VT escapes are not user friendly, period) I don't see it getting a ton of use.

Michael Klement (@mklement0)

Michael Klement (@mklement0) you're right, had the wrong URL in my clipboard at the time. Fixed.

Rain Sallow (/u/ta11ow) (@vexx32)

Rain Sallow (/u/ta11ow) (@vexx32) the $PSStyle variable is explicitly to handle not knowing ESC sequences and allowing for tab-completion in strings. It also has the RGB method for arbitrary 24-bit color.

Patrick Meinecke (@SeeminglyScience)

Patrick Meinecke (@SeeminglyScience) I think those are interesting style names and can see how they may be more future proof. Were you thinking those would replace the ones I have under the Formatting member or in addition? How would you map yours to the existing ones (some are obvious, others not as much like ErrorAccent).

Rain Sallow (/u/ta11ow) (@vexx32)

Rain Sallow (/u/ta11ow) (@vexx32), as implied by Steve Lee (@SteveL-MSFT)'s comment, unless you want to redefine the default colors (which probably isn't a good idea) or redefine the $PSStyle.Formatting properties, you won't have to deal with escape sequences.

However, in order to better support these - advanced - use cases, perhaps we could extend Write-Host to support all the styles supported by $PSStyle and add an -AsDecorated switch that outputs the result as a string with escape sequences.

Patrick Meinecke (@SeeminglyScience)

Patrick Meinecke (@SeeminglyScience) I think those are interesting style names and can see how they may be more future proof. Were you thinking those would replace the ones I have under the Formatting member or in addition? How would you map yours to the existing ones (some are obvious, others not as much like ErrorAccent).

The way I have it in my head is something like this:

class Theme
{
    public Style Default { get; set; }
    public Style Primary { get; set; }
    public Style Secondary { get; set; }
    public Style Success { get; set; }
    public Style Danger { get; set; }
    public Style Warning { get; set; }
    public Style Info { get; set; }
    public Style Light { get; set; }
    public Style Dark { get; set; }
    public Style Muted { get; set; }
}

class Style
{
    public Color Foreground { get; set; }
    public Color Background { get; set; }
    public Decoration Decoration { get; set; }
}

class Color
{
    // Maybe just the RGB value here.

    public bool TryWriteAnsiPrefix(Span<char> buffer, out int amountWritten, bool isBackground = false);

    public bool TryWriteAnsiReset(Span<char> buffer, out int amountWritten, bool isBackground = false);
}

class Decoration
{
    public bool IsUnderlined { get; set; }

    // etc

    public bool TryWriteAnsiPrefix(Span<char> buffer, out int amountWritten);

    public bool TryWriteAnsiReset(Span<char> buffer, out int amountWritten);
}

Where $PSStyle points to Theme.Default maybe with an ETS property called Theme that points to the parent?

_{I realize this is a drastically different design in a few places that aren't relevant, it's just for illustration purposes.}

Ilya (@iSazonov)

Ilya (@iSazonov) The current design would allow for translation of ANSI to HTML if that is required, but for console, ANSI is the de facto standard for authors of tooling

By console, you mean a thin dumb teletype (tty) terminal. But at the same time we want coloring, multiple windows, background tasks, progress bar, power command line editor and so on - we actually want a thick powerful terminal.

I am at a loss that we are spending so much effort to overcome the limitations of tty (here and in PSReadline). (Note that these escape sequences are contrary to the object-based nature of PowerShell.) This approach is counterproductive - we are forced to perform complex string conversions at the application level, then at the terminal level, then at the OS level.

My strong belief is that PowerShell should only offer a generic interface and delegate the rest to the formatting system and host. The formatting system must understand the target is either a console or a file to apply appropriate formatting.
As for the host, when we run PowerShell on Windows, a graphical window actually opens - why would we need a dumb terminal in it if we can use the full power of chromium engine? From the power console we can connect to any remote system (Core, Nano, Linux) and use all benefits of the console.

I would be inclined to agree that expending effort to letting users handle this in code itself manually would be massively wasted if it were not easily and readily accessible to formatters (if not outright designed for that first tbh).

Steve Lee (@SteveL-MSFT)

Might want to add some kind of validation when setting certain sub-properties. For example, see how I was able to set the value of $PSStyle.Formatting.Error = 'hi'. Steve Lee (@SteveL-MSFT)

Using Out-String to strip the ANSI escape sequences seems to work well, however it's adding a new line at the end. I would expect there to be no blank line.

PS /> @("$($PSStyle.Foreground.Blue)Hello$($PSStyle.Reset)", 'string2') | Out-String | Set-content -path test2.txt
PS /> cat ./test2.txt
Hello
string2

With an array containing two string elements, I would expect there to be only two lines in the output.

Trevor Sullivan (@pcgeek86)

Good point about the unwanted trailing newline, Trevor Sullivan (@pcgeek86), but note that - unfortunately - Out-String has always worked this way:

PS> ('foo' | Out-String) -replace '\r?\n', '<newline>'
foo<newline>

Please see #14444.

Thanks for filing the new feedback. Upvoted.

Unfortunately this new feature is exposing a bug in the core tool. Honestly I never had a need to use Out-String until this use case for stripping ANSI escape formatting cropped up.

Trevor Sullivan (@pcgeek86)

Trevor Sullivan (@pcgeek86)
This is the expected behaviour, because both Out‑String and Set‑Content add a trailing newline by default.

You can use the ‑NoNewline switch with Set‑Content to disable adding the extra trailing newline:

@(
	"$($PSStyle.Foreground.Blue)Hello$($PSStyle.Reset)",
	'string2'
) `
	| Out-String `
	| Set-Content -Path test2.txt -NoNewline

Navigation Menu

Repository navigation

ANSI escape handling and PowerShell #13071

User scenario

Fundamental philosophy

Proposed technical implementation details

$PSStyle

$PSStyle.OutputRendering

$PSStyle.Formatting

$PSStyle.Foreground and $PSStyle.Background

PowerShell engine changes

StringDecorated type

Constructor

`bool IsDecorated` property

`int Length` property

`StringDecorated Substring(int contentLength)` method

`string ToString()` method

`string ToString(bool Ansi)` method

`Out-String` update

`Select-String` update

$Host.PrivateData

Alternate considerations

Related Issues

22 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

ANSI escape handling and PowerShell #13071

Description

User scenario

Fundamental philosophy

Proposed technical implementation details

$PSStyle

$PSStyle.OutputRendering

$PSStyle.Formatting

$PSStyle.Foreground and $PSStyle.Background

PowerShell engine changes

StringDecorated type

Constructor

bool IsDecorated property

int Length property

StringDecorated Substring(int contentLength) method

string ToString() method

string ToString(bool Ansi) method

Out-String update

Select-String update

$Host.PrivateData

Alternate considerations

Related Issues

Activity

adamskt commented on Jul 1, 2020

SteveL-MSFT commented on Jul 1, 2020

jhoneill commented on Jul 1, 2020

SeeminglyScience commented on Jul 1, 2020

mklement0 commented on Jul 1, 2020

mklement0 commented on Jul 1, 2020

vexx32 commented on Jul 1, 2020

SteveL-MSFT commented on Jul 1, 2020

SteveL-MSFT commented on Jul 1, 2020

SteveL-MSFT commented on Jul 1, 2020

mklement0 commented on Jul 1, 2020

SeeminglyScience commented on Jul 1, 2020

22 remaining items

SteveL-MSFT commented on Oct 7, 2020

iSazonov commented on Oct 8, 2020

vexx32 commented on Oct 8, 2020

pcgeek86 commented on Dec 15, 2020

pcgeek86 commented on Dec 16, 2020

mklement0 commented on Dec 16, 2020

pcgeek86 commented on Dec 16, 2020

ExE-Boss commented on Jan 15, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

`bool IsDecorated` property

`int Length` property

`StringDecorated Substring(int contentLength)` method

`string ToString()` method

`string ToString(bool Ansi)` method

`Out-String` update

`Select-String` update