The Wiert Corner – irregular stream of stuff

Jeroen W. Pluimers on .NET, C#, Delphi, databases, and personal interests

  • My badges

    Mastodon

  • Twitter Updates

  • My Flickr Stream

    20140508-Delphi-2007--Project-Options--Cannot-Edit-Application-Title-HelpFile-Icon-Theming20140430-Fiddler-Filter-Actions-Button-Run-Filterset-now20140424-VMware-Fusion-6.0.3.-no-reclaimable
    More Photos

  • Pages

  • All categories

  • Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 1,860 other subscribers

Archive for the ‘CommandLine’ Category

« Previous Entries
Next Entries »

Installing Windows software with Chocolatey: a few notes

Posted by jpluimers on 2020/10/28

I will limit myself to software that needs Administrative elevation in order to be installed. This is the default use-case for Chocolatey. It is way way easier than installing software all by hand, but there are a few things you need to know, hence these notes.

Administrative elevation

Since the default use case is installing software that requires Administrative elevation during install, Chocolatey needs to run with Administrative privileges in order to perform these installs.

If you were hoping for a way around this (for instance by having a client/service architecture), then just stop here.

Even though such a structure could technically be created, getting it stable and working it correctly with a truckload of software to be installed (much of which not available as packages during Chocolatey development in the first place) is a task too big.

Think of the size of the Windows Installer team at Microsoft to get installers working in the first place, the extra effort needed by Chocolatey volunteers to get the installers working from the console, then another much more complex layer of getting them running from inside a service and communicating everything back and forth to a non-elevated command prompt would be a nightmare.

I won’t even mention the security steps involved to ensure the non-elevated command prompt has enough rights to send installation instructions to the elevated service.

So the first step is to have an elevated command prompt for Chocolatey.

Being elevated, and Chocolatey needing to download installers requires a local temporary place for them.

By default, that place is %Temp%\chocolatey of the administrative user that elevated the Chocolatey command prompt.

This directory can grow quite big, so dir, so – since there is no choco cleanup yet [WayBack] you need to either:

Install Chocolatey itself

Either the direct one below, or the more secure one (so you can inspect the intermediate [WayBack] install.ps1) at [WayBack] Installation using PowerShell from cmd.exe:

@echo off
SET DIR=%~dp0%
::download install.ps1
%systemroot%\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -ExecutionPolicy Bypass -Command "((new-object net.webclient).DownloadFile('https://chocolatey.org/install.ps1','%DIR%install.ps1'))"
::run installer
%systemroot%\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -ExecutionPolicy Bypass -Command "& '%DIR%install.ps1' %*"

If you want to get rid of it, use [WayBack] Uninstallation.

Besides the one above and below, there are many more [WayBack] Installation: more install options

Output of direct install as Administrator (disclaimers apply):

C:\WINDOWS\system32>powershell -NoProfile -ExecutionPolicy Bypass -Command "[System.Net.WebRequest]::DefaultWebProxy.Credentials = [System.Net.CredentialCache]::DefaultCredentials; iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))" && SET PATH="%PATH%;%ALLUSERSPROFILE%\chocolatey\bin"
Getting latest version of the Chocolatey package for download.
Getting Chocolatey from https://chocolatey.org/api/v2/package/chocolatey/0.10.11.
Downloading 7-Zip commandline tool prior to extraction.
Extracting C:\Users\JEROEN~1\AppData\Local\Temp\chocolatey\chocInstall\chocolatey.zip to C:\Users\JEROEN~1\AppData\Local\Temp\chocolatey\chocInstall...
Installing chocolatey on this machine
Creating ChocolateyInstall as an environment variable (targeting 'Machine')
  Setting ChocolateyInstall to 'C:\ProgramData\chocolatey'
WARNING: It's very likely you will need to close and reopen your shell
  before you can use choco.
Restricting write permissions to Administrators
We are setting up the Chocolatey package repository.
The packages themselves go to 'C:\ProgramData\chocolatey\lib'
  (i.e. C:\ProgramData\chocolatey\lib\yourPackageName).
A shim file for the command line goes to 'C:\ProgramData\chocolatey\bin'
  and points to an executable in 'C:\ProgramData\chocolatey\lib\yourPackageName'.

Creating Chocolatey folders if they do not already exist.

WARNING: You can safely ignore errors related to missing log files when
  upgrading from a version of Chocolatey less than 0.9.9.
  'Batch file could not be found' is also safe to ignore.
  'The system cannot find the file specified' - also safe.
chocolatey.nupkg file not installed in lib.
 Attempting to locate it from bootstrapper.
PATH environment variable does not have C:\ProgramData\chocolatey\bin in it. Adding...
WARNING: Not setting tab completion: Profile file does not exist at 'C:\Users\jeroenAdministrator\Documents\WindowsPowerShell\Microsoft.PowerShell_profile.ps1'.
Chocolatey (choco.exe) is now ready.
You can call choco from anywhere, command line or powershell by typing choco.
Run choco /? for a list of functions.
You may need to shut down and restart powershell and/or consoles
 first prior to using choco.
Ensuring chocolatey commands are on the path
Ensuring chocolatey.nupkg is in the lib folder

Installing packages

Compressing

If you run out of SSD or VM disk space, you can try compress using compact /c /s *.* in these directories:

  • C:\ProgramData\Package Cache
  • C:\ProgramData\Microsoft\VisualStudio\Packages
  • C:\ProgramData\Microsoft\ClickToRun\ProductReleases

Further reading

–jeroen

PS: always watch the output and logs!

Read the rest of this entry »

Posted in Chocolatey, CommandLine, Conference Topics, Conferences, Development, Event, Power User, PowerShell, PowerShell, Scripting, Software Development, Windows | Leave a Comment »

PowerShell: checking minimum version

Posted by jpluimers on 2020/08/13

Nowadays, often your PowerShell code uses features unavailable in older PowerShell versions. When running it on a version that is too old, you usually get an error message, for instance like this:

Unable to find type [Ordered]: make sure that the assembly containing this type is loaded.

Back in the days, this was a new feature introduced in PowerShell 3.0: [WayBack] Use cases of [ordered], the new PowerShell 3.0 feature – Stack Overflow

It is way friendlier to show a message indicating the version is too old in stead of throwing this error.

That’s where the # Requires Version 3.0 directive comes in: [WayBack] about_Requires | Microsoft Docs.

Adding this line to the top of a script gives output like this on a stock Windows 7 SP1 system that has PowerShell 2.0:

# PowerShell -f List-Delphi-Installed-Packages.ps1
The script ‘List-Delphi-Installed-Packages.ps1’ cannot be run because it contained a “#requires” statement at line 1 for Windows PowerShell version 3.0. The version required by the script does not match the currently running version of Windows PowerShell version 2.0.
+ CategoryInfo : ResourceUnavailable: (List-Delphi-Installed-Packages.ps1:String) [], ParentContainsErrorRecordException
+ FullyQualifiedErrorId : ScriptRequiresUnmatchedPSVersion

Note that PowerShell 3.0 is also the minimum version for debugging it in Visual Studio Code (which means you do not have to use PowerShell ISE any more; it is still there , but so far behind as a development tool that many prefer Visual Studio Code):

–jeroen

Posted in CommandLine, Development, PowerShell, PowerShell, Scripting, Software Development | Leave a Comment »

Installing and updating Windows PowerShell – via Microsoft Docs

Posted by jpluimers on 2020/08/11

Since I keep forgetting that PowerShell is part of WMF (Windows Management Framework) and about the compatibility/installation matrix: [WayBack] Installing Windows PowerShell | Microsoft Docs:

The installation package for PowerShell comes inside a WMF installer. The version of the WMF installer matches the version of PowerShell; there’s no stand alone installer for Windows PowerShell.

If you need to update your existing version of PowerShell, in Windows, use the following table to locate the installer for the version of PowerShell you want to update to.

Windows PS 3.0 PS 4.0 PS 5.0 PS 5.1
Windows 10 (see Note1)
Windows Server 2016		 – – – installed
Windows 8.1
Windows Server 2012 R2		 – installed [WayBack] WMF 5.0 [WayBack] WMF 5.1
Windows 8
Windows Server 2012		 installed [WayBack] WMF 4.0  [WayBack] WMF 5.0 [WayBack] WMF 5.1
Windows 7 SP1
Windows Server 2008 R2 SP1		 [WayBack] WMF 3.0 [WayBack] WMF 4.0 [WayBack] WMF 5.0 [WayBack] WMF 5.1

For historic reference:

Downloads

[WayBack] Windows Management Framework 5.1

[WayBack] Windows Management Framework 5.0

To upgrade to WMF 5.0 from 4.0 you need to install .net 4.5 or later on your machine first. Then install WMF 5.0 RTM.

[WayBack] Windows Management Framework 4.0

[WayBack] Windows Management Framework 3.0

–jeroen

Posted in CommandLine, Development, PowerShell, PowerShell, Scripting, Software Development | Leave a Comment »

When NTFS shrink fails, despite using the default settings from the shrink dialog

Posted by jpluimers on 2020/05/25

Sometimes an NTFS shrink still fails, even though you use the built in Windows defragmentation tools, of SysInternals contig tool.

The best you can do is to follow the steps in:

  1. run diskmgmt.msc to try shrinking the disk, then often it is already in the error message: “You can’t shrink a volume beyond the point where any unremoveable files are located see the defrag event in application log for detailed information about the operation when it has completed.”
  2. use eventvwr.exe and look at the Windows Logs for the most recent Application entries that has Source set to defrag

Those defrag entries usually tell about the last file that could not be moved.

You can use wevtutill to query events on the commandline.

Note that contrary to [WayBack] WEVTUTIL – Windows CMD – SS64.com documentation, the option /rd cannot be expanded to /reversedirection , as you will get an error “invalid option reversedirection” – Google Search.

For querying the above defrag event, use this command line (replace /format:XML with /format:text for more readable but also more verbose output):

wevtutil query-events Application /count:2 /format:XML /rd:true /query:"*[System[(EventID=259)]]"

On Windows 10, this is often caused by “System Protection” which locks files under C:\Recovery, but I have also seen $BITMAP, $MFT and $DATA entries.

System protected drives

To view which drives are currently used for system protection (this opens the “System Properties” dialog focussed on the “System Protection” tab):

SystemPropertiesProtection.exe

To disable it for one drive:

Disable-ComputerRestore -Drive "C:"

To enable it for one drive:

Enable-ComputerRestore -Drive "C:"

There seems to be no easy one-command PowerShell way to view the drives have ComputerRestore enabled, as this does not show drive letters:

PowerShell Get-ComputerRestorePoint ^| Format-List

The above gives more detailed output than a plain PowerShell Get-ComputerRestorePoint

Deleting restore points

PowerShell does not have a built-in option to delete restore points, but vssadmin has, but calls them “shadows”.

First list them:

vssadmin list shadows

Then delete them (but be aware this will not prompt for confirmation because of the /quiet):

vssadmin delete shadows /for=C: /quiet

You can also delete them for all drives (this will not prompt for confirmation either):

vssadmin delete shadows /all /quiet

Stopping the volume shadow copy service:

net stop vss

Managing hibernation and page file

Hibernation:

powercfg.exe /hibernate off

powercfg.exe /hibernate on

Page file:

wmic pagefile list /format:list
AllocatedBaseSize=2944
CurrentUsage=0
Description=C:\pagefile.sys
InstallDate=20181018215808.683376+120
Name=C:\pagefile.sys
PeakUsage=0
Status=
TempPageFile=FALSE

wmic computersystem where name="%computername%" get AutomaticManagedPagefile
AutomaticManagedPagefile
TRUE

wmic computersystem where name="%computername%" set AutomaticManagedPagefile=False
Updating property(s) of '\\MYCOMPUTER\ROOT\CIMV2:Win32_ComputerSystem.Name="LAPTOPUW08"'
Property(s) update successful.

wmic computersystem where name="%computername%" get AutomaticManagedPagefile
AutomaticManagedPagefile
FALSE

wmic.exe pagefileset where name="C:\\pagefile.sys" delete
Deleting instance \\MYCOMPUTER\ROOT\CIMV2:Win32_PageFileSetting.Name="C:\\pagefile.sys"
Instance deletion successful.

Sometimes the deletion does not work (see below for workaround):

wmic pagefile list /format:list

AllocatedBaseSize=2944
CurrentUsage=0
Description=C:\pagefile.sys
InstallDate=20181018215808.683376+120
Name=C:\pagefile.sys
PeakUsage=0
Status=
TempPageFile=FALSE

Do not do this:

wmic pagefile delete
Deleting instance \\MYCOMPUTER\ROOT\CIMV2:Win32_PageFileUsage.Name="C:\\pagefile.sys"
ERROR:
Description = Provider is not capable of the attempted operation

wmic pagefileset set name="c:\\pagefile.sys",InitialSize=0,MaximumSize=0
No Instance(s) Available.

Sometimes it still fails, so then you have to use the UI:

  1. Run SystemPropertiesAdvanced.exe
  2. Under Performance, click on Settings
  3. Click the Advanced tab
  4. Under Virtual memory, click the Change button
  5. Ensure Automatically manage page file size for all drives is disabled
  6. Ensure No paging file is selected
  7. Click the Set button
  8. Confirm you really want no page file
  9. Press on the three OK buttons to fully leave the Advanced System Properties dialog.
  10. Reboot

After resizing the disk, reverse the steps:

  1. Run SystemPropertiesAdvanced.exe
  2. Under Performance, click on Settings
  3. Click the Advanced tab
  4. Under Virtual memory, click the Change button
  5. Ensure Automatically manage page file size for all drives is enabled
  6. Confirm you really want no page file
  7. Press on the three OK buttons to fully leave the Advanced System Properties dialog.
  8. Reboot

Bitmap file

Sometimes the file blocking the resize is the NTFS "\$BitMap::$DATA", which few defragmentation tools can move as it is the MFT Bitmap.

Background reading

–jeroen

Posted in CommandLine, Console (command prompt window), Development, Power User, PowerShell, PowerShell, Scripting, Software Development, Windows | Leave a Comment »

Cool hacking stuff…

Posted by jpluimers on 2020/05/06

Boy, it is indeed a game of walls and ladders:

–jeroen

Posted in CommandLine, Development, Power User, PowerShell, PowerShell, Scripting, Software Development, Windows | Leave a Comment »

Mad With PowerShell

Posted by jpluimers on 2020/02/19

This is such a cool blog: Mad With PowerShell (Tim Curwick’s PowerShell blog, tips and tricks, tools and techniques, explanations and explorations).

I bumped into it by finding these two:

What I like most is that it gives great insight on how and why the internals of PowerShell work the way they do, and how to use that to your advantage.

–jeroen

Posted in CommandLine, Development, PowerShell, PowerShell, Scripting, Software Development | Leave a Comment »

Mad With PowerShell: FileInfo and DirectoryInfo objects are not populated upon creation in PowerShell

Posted by jpluimers on 2020/01/16

[WayBack] Mad With PowerShell: FileInfo and DirectoryInfo objects are not populated upon creation in PowerShell.

TL;DR:

  • Apart from the FullName property, the other poprerties of FileInfo and DirectoryInfo are populated later than their instance creation
  • A call to their Refresh method populates or re-populates them
  • That method is implicitly called when you request other properties than FullName

Example: see the [Archive.is]fileinfo.cs: Length property

–jeroen

Posted in .NET, C#, CommandLine, Development, F#, PowerShell, Software Development, VB.NET | Leave a Comment »

ρσℓα¢ķ͌͌͌͌͌͌͌͌͌͌͌͌͌͌ on Twitter : “The PowerShell console history file isn’t just useful for DFIR peeps, red team-ers should check it too! 4096 entries by default, here’s the path…”

Posted by jpluimers on 2020/01/15

Not sure why, but I checked a few of my systems and no  file at %userprofile%\AppData\Roaming\Microsoft\Windows\PowerShell\PSReadline\ConsoleHost_history.txt

Heck: no ConsoleHost_history.txt files on those systems anywhere.

[WayBack] ρσℓα¢ķ͌͌͌͌͌͌͌͌͌͌͌͌͌͌ on Twitter : “The PowerShell console history file isn’t just useful for DFIR peeps, red team-ers should check it too! 4096 entries by default, here’s the path: %userprofile%\AppData\Roaming\Microsoft\Windows\PowerShell\PSReadline\ConsoleHost_history.txt”

–jeroen

Read the rest of this entry »

Posted in CommandLine, Power User, PowerShell, PowerShell | Leave a Comment »

Delphi, decoding files to strings and finding line endings: some links, some history on Windows NT and UTF/UCS encodings

Posted by jpluimers on 2019/12/31

A while back there were a few G+ threads sprouted by David Heffernan on decoding big files into line-ending splitted strings:

Code comparison:

Python:

with open(filename, 'r', encoding='utf-16-le') as f:
  for line in f:
    pass

Delphi:

for Line in TLineReader.FromFile(filename, TEncoding.Unicode) do
  ;

This spurred some nice observations and unfounded statements on which encodings should be used, so I posted a bit of history that is included below.

Some tips and observations from the links:

  • Good old text files are not “good” with Unicode support, neither are TextFile Device Drivers; nobody has written a driver supporting a wide range of encodings as of yet.
  • Good old text files are slow as well, even with a changed SetTextBuffer
  • When using the TStreamReader, the decoding takes much more time than the actual reading, which means that [WayBack] Faster FileStream with TBufferedFileStream • DelphiABall does not help much
  • TStringList.LoadFromFile, though fast, is a memory allocation dork and has limits on string size
  • Delphi RTL code is not what it used to be: pre-Delphi Unicode RTL code is of far better quality than Delphi 2009 and up RTL code
  • Supporting various encodings is important
  • EBCDIC days: three kinds of spaces, two kinds of hyphens, multiple codepages
  • Strings are just that: strings. It’s about the encoding from/to the file that needs to be optimal.
  • When processing large files, caching only makes sense when the file fits in memory. Otherwise caching just adds overhead.
  • On Windows, if you read a big text file into memory, open the file in “sequential read” mode, to disable caching. Use the FILE_FLAG_SEQUENTIAL_SCAN flag under Windows, as stated at [WayBack] How do FILE_FLAG_SEQUENTIAL_SCAN and FILE_FLAG_RANDOM_ACCESS affect how the operating system treats my file? – The Old New Thing
  • Python string reading depends on the way you read files (ASCII or Unicode); see [WayBack] unicode – Python codecs line ending – Stack Overflow

Though TLineReader is not part of the RTL, I think it is from [WayBack] For-in Enumeration – ADUG.

Encodings in use

It doesn’t help that on the Windows Console, various encodings are used:

Good reading here is [WayBack] c++ – What unicode encoding (UTF-8, UTF-16, other) does Windows use for its Unicode data types? – Stack Overflow

Encoding history

+A. Bouchez I’m with +David Heffernan here:

At its release in 1993, Windows NT was very early in supporting Unicode. Development of Windows NT started in 1990 where they opted for UCS-2 having 2 bytes per character and had a non-required annex on UTF-1.

UTF-1 – that later evolved into UTF-8 – did not even exist at that time. Even UCS-2 was still young: it got designed in 1989. UTF-8 was outlined late 1992 and became a standard in 1993

Some references:

–jeroen

Read the rest of this entry »

Posted in Delphi, Development, Encoding, PowerShell, PowerShell, Python, Scripting, Software Development, The Old New Thing, Unicode, UTF-16, UTF-8, Windows Development | Leave a Comment »

PowerShell: fixing `Get-HotFix` having empty `InstalledOn` entries

Posted by jpluimers on 2019/11/12

On some systems, Get-HotFix has many entries with an empty InstalledOn column.

This at least shows there is a date-format difference, but now the Source column is empty.:

Get-HotFix | Select-Object Source,Description,HotfixID,InstalledBy,InstalledOn,@{Name="InstalledOnValue";Expression={$_.psbase.properties["InstalledOn"].Value}} | Out-GridView

I contemplated using Microsoft.Update.Session in the scripts below, but it requires WinRM, the server side implementation of WS-Management – Wikipedia:

[ERROR] [DevMachine] Connecting to remote server DevMachine failed with the following
[ERROR] error message : The client cannot connect to the destination specified in the
[ERROR] request. Verify that the service on the destination is running and is accepting
[ERROR] requests. Consult the logs and documentation for the WS-Management service run
[ERROR] ning on the destination, most commonly IIS or WinRM. If the destination is the
[ERROR] WinRM service, run the following command on the destination to analyze and conf
[ERROR] igure the WinRM service: "winrm quickconfig". For more information, see the abo
[ERROR] ut_Remote_Troubleshooting Help topic.
[ERROR] + CategoryInfo : OpenError: (DevMachine:String) [], PSRemotingTr
[ERROR] ansportException
[ERROR] + FullyQualifiedErrorId : CannotConnect,PSSessionStateBroken

This is a reminder to find and document a proper fix for this.

This at least works:

Get-HotFix | Select-Object Source,Description,HotfixID,InstalledBy,InstalledOn,@{Name="InstalledOnDateTime";Expression={[System.DateTime]::Parse($_.PSBase.Properties["InstalledOn"].Value,[System.Globalization.CultureInfo]::GetCultureInfo("en-US"))}} | Out-GridView

Some links that hopefully help with proper documenting it:

–jeroen

Read the rest of this entry »

Posted in CommandLine, Development, PowerShell, PowerShell, Scripting, Software Development | Leave a Comment »

« Previous Entries
Next Entries »