Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Why are all newlines gone after PowerShell's Get-Content, Regex, and Set-Content?

Tags:

I want to load a file template into a variable, modify data within the variable and output the modified template to a new location from the variable.

The issue is that PowerShell is removing newlines from my template.

The input file (template file) has Unix line endings which are also required for output since the recipient of the modified version is a Unix-based system.

I have the following code which results into a concatted one-liner:

[String] $replacement = "Foo Bar" [String] $template = Get-Content -Path "$pwd\template.sh" -Encoding UTF8 $template = $template -replace '<REPLACE_ME>', $replacement $template | Set-Content -Path "$pwd\script.sh" -Encoding UTF8 

Having the template input:

#!/bin/sh myvar="<REPLACE_ME>" echo "my variable: $myvar" exit 0 

Resulted into:

#!/bin/sh myvar="Foo Bar" echo "my variable: $myvar" exit 0 

It appears to me that somewhere LF where replaced by one simple whitespace. Finally at the end of the script there is an added CR LF which was not present in the template file.

How do I preserve the line endings and prevent adding further (CR LF) wrong line endings to the final script?

like image 365
burnersk Avatar asked Nov 27 '17 13:11

burnersk


People also ask

How do I get the content of a file in PowerShell?

When you want to read the entire contents of a text file, the easiest way is to use the built-in Get-Content function. When you execute this command, the contents of this file will be displayed in your command prompt or the PowerShell ISE screen, depending on where you execute it.

Does PowerShell support regex?

PowerShell has several operators and cmdlets that use regular expressions. You can read more about their syntax and usage at the links below. PowerShell regular expressions are case-insensitive by default.


2 Answers

For the $replacement variable, you don't really need to specify the type [string], PowerShell will infer that from the assignment.

For the $template variable, [string] is actually wrong. By default, Get-Content will give you an array of strings (i.e. lines) instead of one string.

But in fact you don't even want to split the input into lines in the first place. When Set-Content or Out-File see an array as their input, they will join it with spaces.

Using -Raw makes Get-Content return the entire file as one string, this way also the line endings (like LF for Linux files) will stay the way they are.

$replacement = "Foo Bar" $template = Get-Content -Path "$pwd\template.sh" -Encoding UTF8 -Raw $template = $template -replace '<REPLACE_ME>', $replacement Set-Content -Path "$pwd\script.sh" -Value $template -Encoding UTF8 

PowerShell will save all UTF-8 files with a BOM. If you don't want that, you must use a different utility to write the file:

$UTF8_NO_BOM = New-Object System.Text.UTF8Encoding $False  $replacement = "Foo Bar" $template = Get-Content -Path "$pwd\template.sh" -Encoding UTF8 -Raw $template = $template -replace '<REPLACE_ME>', $replacement [System.IO.File]::WriteAllText("$pwd\script.sh", $template, $UTF8_NO_BOM) 

Notes:

  • PowerShell operators (like -replace) silently operate on arrays. $x -replace "search", "replacement" will perform a replace operation on every member of $x, be that a single string or an array of them.
  • Recommended reading: PowerShell Set-Content and Out-File what is the difference?
like image 174
Tomalak Avatar answered Sep 20 '22 15:09

Tomalak


I think you need to use the -Raw switch with Get-Content in order to load the file as a single string:

[String] $replacement = "Foo Bar" [String] $template = Get-Content -Path "$pwd\template.sh" -Encoding UTF8 -Raw $template = $template -replace '<REPLACE_ME>', $replacement 

To stop the Windows line ending being added to the end of the script, I think you need to use this .NET method for writing the file:

[io.file]::WriteAllText("$pwd\template.sh",$template) 

By default PowerShell attempts to convert your input in to an array of strings for each line in the file. I think because of the Unix line endings its not doing this successfully but is subsequently removing the new line characters.

In PowerShell 3.0 we now have a new dynamic parameter, Raw. When specified, Get-Content ignores newline characters and returns the entire contents of a file in one string. Raw is a dynamic parameter, it is available only in file system drives.

  • https://social.technet.microsoft.com/Forums/windowsserver/en-US/6026b31a-2a0e-4e0a-90b5-355387dce9ac/preventing-newline-with-outfile-or-addcontent?forum=winserverpowershell
like image 25
Mark Wragg Avatar answered Sep 20 '22 15:09

Mark Wragg