.NET/C#/PowerShell: building .NET DiscUtils library for virtual disk images
Posted by jpluimers on 2014/03/11
Mono Mac
.NET DiscUtils is an interesting open source .NET library for accessing and manipulating virtual disk images. Since it is entirely written in C# (without the need for P/Invoke), you should even be able to run this on non-Windows machines using mono. Later on, you will see the 0.11.0 build fails this, but it gives good hope it eventually will.
Virtual disk formats supported are DMG, ISO, RAW (IMG/IMA/VFD/FLP/BIF), VDI, VHD, VHDX, VMDK, and XVA, regular disks like Physical, iSCSI and NFS.
There are two ways of getting the .NET DiscUtils tools to run:
- download pre-build binaries (at the time of writing: version 0.10) from via .NET DiscUtils – Home, or
- from the latest source page, click the download button, then build the binaries from the source package. At the time of writing, that version is 0.11.
This post describes the second way, and requires PowerShell to be installed on your system (which probably is, as Windows 7 and Windows Server 2008 R2 include it).
Download, build and run .NET DiscUtil tools
These are the steps to perform:
- Download the latest source page,
- Unzip the downloaded ZIP file,
- Start a command prompt,
- Change the directory to the one where “
build.ps1” resides, - Copy the “
compile.ps1” script below to a new file “compile.ps1” in that directory, - Run the “
compile.ps1” in PowerShell using this command:
PowerShell -File compile.ps1If you get an error like this:
C:\temp.jwp\discutils>PowerShell -File compile.ps1 File C:\temp.jwp\discutils\build.ps1 cannot be loaded because running scripts is disabled on this system. For more information, see about_Execution_Policies at http://go.microsoft.com/fwlink/?LinkID=135170.
+ CategoryInfo : SecurityError: (:) [], ParentContainsErrorRecordException
+ FullyQualifiedErrorId : UnauthorizedAccess C:\temp.jwp\discutils>Then you have to enable PowerShell script execution for the current user by executing this command from Enabling powershell to run unsigned scripts for the current user only:
PowerShell Set-ExecutionPolicy -Scope CurrentUser -ExecutionPolicy RemoteSigned - Copy the “
copyAllUtils.ps1” script below to a new file “copyAllUtils.ps1” in the same directory as “build.ps1” resides, - Run the “
copyAllUtils.ps1” in PowerShell using this command:
PowerShell -File copyAllUtils.ps1It lists the files it has copied, on my systemC:\temp.jwp\discutils\Utils\BCDDump\bin\Debug\BCDDump.exe
C:\temp.jwp\discutils\Utils\BCDDump\bin\Debug\DiscUtils.Common.dll
C:\temp.jwp\discutils\Utils\BCDDump\bin\Debug\DiscUtils.dll
C:\temp.jwp\discutils\Utils\DiscUtils.PowerShell\bin\Debug\DiscUtils.PowerShell.dll
C:\temp.jwp\discutils\Utils\DiskClone\bin\Debug\DiskClone.exe
C:\temp.jwp\discutils\Utils\DiskDump\bin\Debug\DiskDump.exe
C:\temp.jwp\discutils\Utils\ExternalFileSystem\bin\Debug\ExternalFileSystem.exe
C:\temp.jwp\discutils\Utils\FileExtract\bin\Debug\FileExtract.exe
C:\temp.jwp\discutils\Utils\FileRecover\bin\Debug\FileRecover.exe
C:\temp.jwp\discutils\Utils\iSCSIBrowse\bin\Debug\iSCSIBrowse.exe
C:\temp.jwp\discutils\Utils\ISOCreate\bin\Debug\ISOCreate.exe
C:\temp.jwp\discutils\Utils\MSBuildTask\bin\Debug\DiscUtils.MSBuild.dll
C:\temp.jwp\discutils\Utils\NTFSDump\bin\Debug\NTFSDump.exe
C:\temp.jwp\discutils\Utils\OSClone\bin\Debug\OSClone.exe
C:\temp.jwp\discutils\Utils\VHDCreate\bin\Debug\VHDCreate.exe
C:\temp.jwp\discutils\Utils\VHDDump\bin\Debug\VHDDump.exe
C:\temp.jwp\discutils\Utils\VirtualDiskConvert\bin\Debug\VirtualDiskConvert.exe
C:\temp.jwp\discutils\Utils\VolInfo\bin\Debug\VolInfo.exe
running the VirtualDiskConvert on Windows
C:\Users\developer\Desktop\Disk-Images>\bin\DiscUtils\VirtualDiskConvert.exe -outputFormat VMDK-fixed CSBOOT.IMG CSBOOT.vmdk VirtualDiskConvert v0.11.0, available from http://discutils.codeplex.com Copyright (c) Kenneth Bell, 2008-2011 Free software issued under the MIT License, see LICENSE.TXT for details. Progress (100%) |===============================================| 00:00:00.0 C:\Users\developer\Desktop\Disk-Images>dir CSBOOT* ... 04/10/2013 09:57 PM 67,125,248 CSBOOT-flat.vmdk 04/21/2011 06:41 PM 67,125,248 CSBOOT.IMG 04/10/2013 09:57 PM 471 CSBOOT.vmdk
running the VirtualDiskConvert with Mono on a Mac
Jeroens-MacBook-Pro:VM jeroenp$ mono ../Dropbox/bin/DiscUtils/VirtualDiskConvert.exe -of VMDK-fixed ./CSBOOT.IMG CSBOOT.vmdk VirtualDiskConvert v0.11.0, available from http://discutils.codeplex.com Copyright (c) Kenneth Bell, 2008-2011 Free software issued under the MIT License, see LICENSE.TXT for details. Unhandled Exception: System.IO.DirectoryNotFoundException: Could not find a part of the path "/Users/jeroenp/VM\/CSBOOT.IMG".
What probably happens here is that DiscUtils uses the wrong System.IO.Path.DirectorySeparatorChar Field.
compile.ps1 script
Put the compile.ps1 script in the same directory as the DiscUtils.sln solution, then run the script so the solution gets built by the most current version of msbuild.
Based on This is just a little trick I use to find InstallUtil and MsBuild, and make sure I’m using the latest version of them by Joel Bennet.
### http://poshcode.org/896
## Because of Split-Path, I get the "Framework" folder path (one level above the versioned folders)
$rtr = Split-Path $([System.Runtime.InteropServices.RuntimeEnvironment]::GetRuntimeDirectory())
## Then I loop through them in ascending (numerical, but really ascii) order
## each time I find installutil or mdbuild, I update the alias to point at the newer version
foreach($rtd in get-childitem $rtr -filt v* | sort Name) {
if( Test-Path (join-path $rtd.FullName installutil.exe) ) {
set-alias installutil (resolve-path (join-path $rtd.FullName installutil.exe))
}
if( Test-Path (join-path $rtd.FullName msbuild.exe) ) {
set-alias msbuild (resolve-path (join-path $rtd.FullName msbuild.exe))
}
}
msbuild DiscUtils.sln | out-null
if(-not $?)
{
Write-Host "msbuild of DiscUtils.sln Failed"
Exit
}
copyAllUtils.ps1 script
This script uses quite a few tricks, so here are some links for further reading:
PowerShell relies heavily on piping the output of one cmdlet to another. The reason is that the pipeline system does not only support text, but any object or collection of objects.
That the pipelining and filtering system is one of the reasons PowerShell is becoming more and more popular.
Three very important aspects of the pipeline:
- Where-Object (aliased by ? and where) filters the pipeline.
- Inside a pipeline, $_ holds the current active object.
- ForEach-Object receives a collection and unwraps it for you.
Actually, there are both the foreach keyword for use without a pipeline, and the ForEach-Object cmdlet inside a pipeline. They work in a similar way, but there are a few strange things with the foreach keyword. To name a few:
- foreach does not write to the pipeline.
- there can be huge performance differences, see also this post.
- foreach works differently with continue.
- foreach can be more power hungry fan ForEach-Object.
ForEach-Object and foreach are so important that dozens of articles and book fragments have been devoted to it. Read a few of these to get a feel:
- Essential PowerShell: Understanding foreach | Poshoholic.
- Essential PowerShell: Understanding foreach (Addendum) | Poshoholic.
- Microsoft Windows PowerShell 3. 0 First Look – By A. Driscoll.
- Manning: PowerShell in Practice.
- Manning: Windows PowerShell in Action.
Whereas the pipeline does implicitly loop, ForEach-Object loops explicitly. Oh, and there is the difference between the foreach keyword and ForEach-Object cmdlet.
New-Item can create both a new file or a new directory.
When the file or directory exists, it will bail out. There are two alternatives:
- use the -force parameter with New-Item
New-Item $AllUtils -type directory - use the .NET framework
[System.IO.Directory]::CreateDirectory($AllUtils)
Actually, there are 2 more ways of creating directories as described by Learn Four Ways to Use PowerShell to Create Folders – Hey, Scripting Guy! Blog.
Remove-Item can remove a directory or files inside the directory (there even are options -recurse and -exclude, and even -include – which is default). For safety, you can use the -whatif option so it only shows what it would do without the -whatif option.
Get-ChildItem gets you items from a directory. Like Remove-Item, there is are options for -recurse, -exclude and -include.
Since we want to copy both EXE and DLL files we need to pass an array. For this, PowerShell as the @() array subexpression. Some other expressions are grouping and subexpressions. Read more about those at Powershell Concatenation by Akim and at Effective PowerShell Item 10: Understanding PowerShell Parsing Modes | Keith Hill’s Blog.
Though PowerShell is line oriented, you hardly need to use the line continuation character `
Copy-Item copies files to a target. Like New-Item, it has the -force option so you can overwrite existing files. And it supports wildcards too.
By default PowerShell sends the result of an expression to StdOut.
If you want to write your own output, you can for instance use Write-Host.
These posts show various other ways of emitting output:
- Writing Output with PowerShell – Hey, Scripting Guy! Blog.
- windows – How to output something in PowerShell – Stack Overflow.
One of my ideas was to keep a System.Collections.Generic.Dictionary<TKey, TValue> mapping between files to be copied and their target, then only copy the non-duplicate targets.
There are two reasons I didn’t do this:
- Generic dictionaries in PowerShellv1 are an issue, and the v2+ syntax is still a bit ugly
- The current solution is more elegant (only an array with already copied target files with a simple if-check)
For the current solution, I needed to start with an empty array. That is unusual in PowerShell (99% of the times you start with an array that already has values). But it turned out to be remarkably simple as shown by Coretech Blog » Blog Archive » PowerShell: How to create an empty array!.
Arrays in PowerShell are strange anyway, as they
- can contain a plethora of object types
- can be added to (i.e. they behave more like a list of objects than an array of objects)
- are mutable (so you can change values)
Read more about that on the “Hey, Scripting Guy!” blog: Add, Modify, Verify, and Sort Your PowerShell Array – Hey, Scripting Guy! Blog.
PowerShell has many comparison operators, some support regular expressions for matching.
I used the -contains operator for checking if a file was already processed.
## Copy the compiled binaries to the AllUtils directory
$AllUtils = ".\AllUtils"
$AllUtilsDirectory = [System.IO.Directory]::CreateDirectory($AllUtils)
Remove-Item ($AllUtils + "\*.*")
$CopiedFiles = @()
Get-ChildItem ".\Utils\" -recurse -include @( "*.exe", "*.dll" ) |
Where-Object { $_.Directory.BaseName -eq "Debug" } |
Where-Object { $_.Directory.Parent } |
Where-Object { $_.Directory.Parent.name -eq "bin" } |
ForEach-Object {
$fileName = $_.Name.ToUpperInvariant()
if ($CopiedFiles -contains $fileName) {
## Write-Host $_.Name " already copied"
}
else {
Copy-Item $_.FullName $AllUtils -Force
$CopiedFiles += , $fileName
Write-Host $_.FullName
}
}
Exit
Note that you can replace Where-Object with ? and ForEach-Object with %. Some people find that easier to read. I don’t.
## Copy the compiled binaries to the AllUtils directory
$AllUtils = ".\AllUtils"
$AllUtilsDirectory = [System.IO.Directory]::CreateDirectory($AllUtils)
Remove-Item ($AllUtils + "\*.*")
$CopiedFiles = @()
Get-ChildItem ".\Utils\" -recurse -include @( "*.exe", "*.dll" ) |
? { $_.Directory.BaseName -eq "Debug" } |
? { $_.Directory.Parent } |
? { $_.Directory.Parent.name -eq "bin" } |
% {
$fileName = $_.Name.ToUpperInvariant()
if ($CopiedFiles -contains $fileName) {
## Write-Host $_.Name " already copied"
}
else {
Copy-Item $_.FullName $AllUtils -Force
$CopiedFiles += , $fileName
Write-Host $_.FullName
}
}
Exit
–jeroen
via: .NET DiscUtils – Wikipedia, the free encyclopedia.
The Wiert Corner - irregular stream of stuff 
Thanks Jaykul for explaining me a custom PowerShell script that acts like DU (DiskUsage). « The Wiert Corner – irregular stream of stuff said
[…] I should emphasize the importance of the object pipeline even more than I already did. […]