Need to check this out some day: cs.exe compiled from [Wayback] sparse.zip which you can download from [Wayback/Archive] NTFS Sparse Files For Programmers
Archive for the ‘NTFS’ Category
NTFS Sparse Files For Programmers
Posted by jpluimers on 2024/09/25
Posted in C, C++, Development, NTFS, Power User, RoboCopy, Software Development, Visual Studio C++, Windows, Windows 10, Windows 11 | Leave a Comment »
More on empty files
Posted by jpluimers on 2021/10/07
TL;DR: Empty files are indeed of size zero, but there is some disk space involved for their meta-data (like name, permission, timestamps)
Some links (via [WayBack] create zero sized file – Google Search):
- [WayBack] Zero-byte file – Wikipedia
- [WayBack] filesystems – How can a file size be zero? – Super User (thanks [WayBack] phuclv):
Filesystems store a lot of information about a file such as file name, file size, creation time, access time, modified time, created user, user and group permissions, fragments, pointer to clusters that store the file, hard/soft links, attributes… Those are called file metadata. Why do you count those metadata into file size when users do not (need to) care about them and don’t know about them? They only really care about the file content
Moreover each filesystem stores different types of metadata which take different amounts of space on disk. For example POSIX permissions are very different from NTFS permission, and there are also
inodenumbers in POSIX which do not exist on Windows. Even POSIX filesystems vary a lot, like ext3 with 32-bit block address, ext4 with 48-bit, Btrfs with 64-bit and ZFS with 128-bit address. So how will you count those metadata into file size?Take another example with a 100-byte file whose metadata consumes 56 bytes on the current filesystem. We copy the file to another filesystem and now it takes 128 bytes of metadata. However the file contents are exactly the same, the number of bytes in the files are also the same. So displaying file size as 156 bytes on a system but 228 bytes on another is very confusing and counter-intuitive.
- [WayBack] What is the concept of creating a file with zero bytes in Linux? – Unix & Linux Stack Exchange:
touchwill create an inode, andls -iorstatwill show info about the inode:$ touch test $ ls -i test 28971114 test $ stat test File: ‘test’ Size: 0 Blocks: 0 IO Block: 4096 regular empty file Device: fc01h/64513d Inode: 28971114 Links: 1 Access: (0664/-rw-rw-r--) Uid: ( 1000/1000) Gid: ( 1000/1000) Access: 2017-03-28 17:38:07.221131925 +0200 Modify: 2017-03-28 17:38:07.221131925 +0200 Change: 2017-03-28 17:38:07.221131925 +0200 Birth: -Notice that
testuses 0 blocks. To store the data displayed, the inode uses some bytes. Those bytes are stored in the inode table. Look at the ext2 page for an example of an inode structure [WayBack].
Oh and a nice NTFS thing (thanks [WayBack] Paweł Bulwan):
and in case of NTFS, the size of file reported by Windows and most tools is actually the size of the main stream of the file, which we perceive as the content of the file. The file stored on NTFS partition can additionaly have some data stored in alternative data streams, and still have the reported size of 0. It’s a nice filesystem feature to know if you want to have the full picture :)
Related: my really old post command line – create empty text file from a batch file (via: Stack Overflow)
–jeroen
Share this:
Posted in *nix, btrfs, Development, File-Systems, NTFS, Power User, Software Development, Windows | Leave a Comment »
Twitter thread by thread by @0xdade; More unicode shit: zero width space and a zero width nonjoiner in filenames
Posted by jpluimers on 2021/09/22
[WayBack] Thread by @0xdade: Today I learned that you can put zero width spaces in file names on Linux. Have fun. I’m playing with this because punycode/IDN is fascinati…
Today I learned that you can put zero width spaces in file names on Linux. Have fun.I’m playing with this because punycode/IDN is fascinating, and I wanted to know what happened when I started shoving unicode in the path portion of the url, which isn’t part of how browsers try to protect URLs, as far as I can tell
I think it’s more entertaining to have a file that is named *only* a zero width space, but I think using them throughout a filename is better to break tab completion and not stand out too much. A filename that is just blank looks strange in ls output.Thank goodness adduser is looking out for our best interests.Oooh this one is pretty subtle.Just about pissed myself with this one.Not related to the terminal fun, but related to zero width characters:
You can:
– Break url previews https://0xda.de
– @0xdade without tagging
– Make a word like systemd not searchable twitter.com/search?q=from%…Okay but back to command line crap. I really like this one. Create a directory named .[ZWS]One thing that is cool about using zero width spaces is that “ls” has a flag, “-b”, that is meant to escape non-graphic characters. Inserting a newline, for instance, would be escaped to \n. But the zero width space is technically a graphic character, so nothing happens.
Fun.
Have no fear, though. It’s not unbeatable. It’s only fun if the language and LC settings are set to support utf-8. If you set LC_ALL=C or whatever that isn’t utf-8, then it looks like this.Putting a link to this tweet here so that I don’t lose it again in the future.
dade@0xdadeMy god, it is beautiful. I mean except all the whitespace I can’t get rid of before the command lmao.
And these tweets:
- [WayBack] dade в Twitter: “Today I learned that you can put zero width spaces in file names on Linux. Have fun.… “
- [WayBack] @zeratax (fake) Twitterissä: “newline char in filenames is also very “fun”… “
- [WayBack] healthyoutlet on Twitter: “using a zero width space and a zero width nonjoiner as the two possible states of a bit you can hide binary secrets in tweets! … “
- [WayBack] Garth “No CVEs” Mortensen on Twitter: “Quick function to get a zero width space in your clipboard (OS X, use
xselfor linux).zws () { echo -n '\u200D' | pbcopy }… “ - [WayBack] Duncan on Twitter: “Terminal escape sequences are fun too some command line tools on some operating systems will escape them, others don’t.… “
- [WayBack] aaron on Twitter: “lol when I was new at CTF and thought guessctf challenges were fun, I made a “forensics” chal where there is a tarball containing a ton of junk and a single file named a zero-width space with permissions
000. If you cat it, there were\bchars to hide the flag. Super tedious.… “ - [WayBack] َ (@Maxwellcrafter) on Twitter: “I’m pretty sure that this works with Windows too, at least folders, and it messes with stuff sometimes… “
- [WayBack] Uri Granta on Twitter: “Wait till you discover that
Variation Selectors(U+FE00..U+FE0F) are zero-width invisible modifiers with theID_Continueproperty, so can be used inside variable names in languages like Python or C#. And since they’re modifiers, the cursor doesn’t stop when it passes them.… “ - [WayBack] Harri Luuppala on Twitter: “Insert an bell in a filename for your friends. In most terminals, when the Bell character (ASCII code 7,
\ain C) is printed by the program, it will cause the terminal to ring its bell.… “ - [WayBack] J. Neko on Twitter: “Zero width spaces can be very useful. Have also used them in usernames, and to avoid character length rules, and…… “
- [WayBack] ALP on Twitter: “that is nothing. one day i’ve managed to put a backspace in the name of the file! it was more than frustrating trying to delete file that had a hidden typo in its name…… “
- [WayBack] Lucky225🍀 2️⃣ 2️⃣ 5️⃣✸ on Twitter: “I put zero width characters in fucking everything. I’ve found it useful in bypassing a shit ton of shit.… “
- [WayBack] dade on Twitter: “I have more than one “Last name” of zero width.… “
- [WayBack] Lucky225🍀 2️⃣ 2️⃣ 5️⃣✸ on Twitter: “I’ve been using it to avoid having a first name at all in certain places. It’s also fun to put them in passwords. All kinds of fun stuff you can do.… “
- [WayBack] Lucky225🍀 2️⃣ 2️⃣ 5️⃣✸ on Twitter: “This field is required? Okay here’s some ZW chars… “
- [WayBack] dade on Twitter: “I think it’s more entertaining to have a file that is named *only* a zero width space, but I think using them throughout a filename is better to break tab completion and not stand out too much. A filename that is just blank looks strange in ls output.… “
- [WayBack] dade on Twitter: “You could put zw spaces in config files to make it look like it’s reading one file when it’s really reading a different file.… “
- [WayBack] Dylan Katz on Twitter: “This one is my fave: ‘⁄’ (\u2044) or ‘∕’ (\u2215) Allow for this: …”
- [WayBack] dade on Twitter: “Does chrome/firefox’s IDN algorithms not punycode these for lookalike characters? or mixed script confusables? or the unicodeset blacklist?… “
- [WayBack] Dylan Katz on Twitter: “So for the first one they do, for the second one, it strips the user. Also, a lot of this is more on the platform imo. For instance, on mobile devices, users may not be checking the url bar. For instance, I was originally using that trick for QR codes urls… “
- [WayBack] Dylan Katz on Twitter: “(whoops, doubled up on “for instance” there) Basically, depends on the platform, but I feel like creating a convincing link in the app (twitter, slack, whatever) is enough… “
- [WayBack] dade on Twitter: “Well it still shows as a blank line in the ls of the parent directory, so it isn’t flawless.… “
- [WayBack] dade on Twitter: “My god, it is beautiful. I mean except all the whitespace I can’t get rid of before the command lmao.… “
- [WayBack] eater 🥰 on Twitter: “ha yes,, rendering letters that have no width as full width, classic… “
- [WayBack] dade on Twitter: “But it renders the width on the wrong side of the prompt, which is what is particularly interesting. My guess is that has something to do with it being characters that are greater than one byte. It also only does that until I leave that sesssion and go into another one.… “
- [WayBack] Peter Martini on Twitter: “My favorite joke Perl module: … “
[WayBack] Acme::Bleach – For really clean programs – metacpan.org
- [WayBack] dade on Twitter: “Just about pissed myself with this one.… “
- [WayBack] eater 🥰 on Twitter: “ha yes,, rendering letters that have no width as full width, classic… “
- [WayBack] dade on Twitter: “Not related to the terminal fun, but related to zero width characters: You can: – Break url previews
https://0xda.de–@0xdadewithout tagging – Make a word likesystemdnot searchable “
[WayBack] Thread by @Plazmaz: @0xdade Was doing some real fucking around with urls recently: gist.github.com/Plazmaz/565a5c… (was gonna flesh it out more but didn’t find…:
Was doing some real fucking around with urls recently:(was gonna flesh it out more but didn’t find the time)
This one is my fave:
‘⁄’ (\u2044)
or
‘∕’ (\u2215)
Allow for this:
google.com⁄search⁄query⁄.example.com
google.com⁄search⁄query⁄@example.com[WayBack] url-screwiness.md · GitHub:
This is a list of methods for messing with urls. These are often useful for bypassing filters, SSRF, or creating convincing links that are difficult to differentiate from legitimate urls.
And a bit of documentation links:
- [WayBack] Zero-width space – Wikipedia
- [WayBack] Zero-width non-joiner – Wikipedia
- [WayBack] Variation Selectors (Unicode block) – Wikipedia
- [WayBack] Where to find Unicode 5 characters from specific category? – Super User
- [WayBack] Bidirectional text: Overrides – Wikipedia
–jeroen
Share this:
Posted in *nix, .NET, C#, Development, NTFS, Power User, Python, Scripting, Software Development, Windows | Leave a Comment »
Unix and NTFS file systems, hardlinks, inodes, files, directories, dot directories, bugs and implementation details
Posted by jpluimers on 2021/09/21
Lots of interesting tidbits on unix and NTFS file systems.
If you want to blow up your tooling, try creating a recursive hardlink…, which is likely one of the reasons that nx file systems do not support them.
Covered and related topics:
Share this:
Posted in *nix, Development, File-Systems, History, NTFS, Power User, Software Development, Windows, Windows Development | Leave a Comment »
Cipher: a command-line tool to decrypt/encrypt files and directories (een recursively) on Windows
Posted by jpluimers on 2020/07/03
A while ago, I had to mass encrypt a lot of directories and files on Windows for some directories in an existing directory structure.
This helped me to find out which ones were already done (it lists all encrypted files on all drives; the /n ensures the files or encryption keys are not altered):
cipher.exe /u /n /h
This encrypted recursively in one directory B:\Directory:
cipher /D /S:B:\Directory /A
It also has options to wipe data (/W), export keys into transferrable files (/X) and many more.
If you like the Windows Explorer more then to encrypt/decrypt (it is a tedious process): [WayBack] How do I encrypt/decrypt a file? | IT Pro.
Via:
- [WayBack] Microsoft Technet – File Systems: Cipher.exe
- [WayBack] cipher | Microsoft Docs
- [WayBack] encryption – How to list encrypted files in Windows 7? – Super User
- [WayBack] Windows 7 – How to use Cipher.exe – A Critical Win 7 security Tool
- [WayBack] Encrypted Files – Find All on the Local Hard Drives – Windows 7 Help Forums
–jeroen
Share this:
Posted in Encryption, NTFS, Power User, Security, Windows | Leave a Comment »
When storing huge files under NTFS compression, ensure you have twice the disk space
Posted by jpluimers on 2020/01/31
When copying over a 400 gigabyte file over the network to an NTFS compressed folder on a drive with having 600 gigabytes free space, the volume became full after copying ~350 gigabytes.
What I learned is that compressing huge files for later read-only access is fine, but you need about twice the disk space while the copy operation is in progress.
For non-compressed files you can go without this extra reservation.
Background information:
- [WayBack] windows – Why does NTFS compression take up a lot of space? – Super User
- [Archive.is] Understanding NTFS Compression – Ntdebugging Blog
Note there are also issues with NTFS compression and de-duplication. I’m not sure about sparse files. Be careful when you try to compress the system drive where your Windows OS lives on:
- [Archive.is] windows – NTFS compression ate all disk space with no possibility to recover – Super User
- [Archive.is] Shrinking the gap: carving NTFS-compressed files | ForensicFocus.com
–jeroen
Share this:
Posted in NTFS, Power User, Windows | Leave a Comment »
Fixing a broken mirrored Intel Matrix RAID-1 machine
Posted by jpluimers on 2018/04/20
A while ago I had an Intel Matrix RAID-1 pair of drives that got broken. One of them turned “red” and – since both drives were only a few serial numbers apart – the other was giving issues the moment I tried fiddling with it.
These actions failed:
- Windows image backup – would end up with a “blue” screen indicating Windows 10 had a problem and was trying to collect data
- Paragon HDM
- Migrating the OS to a brand new RAID-1 set
- P2V
- Disk2vhd would hang at the 100% completion mark
What had succeeded was a regular Windows backup (a non-image one).
This is what I finally did to get it working again:
- Kill disk2vhd after it hung a few hours at the 100% completion mark
- Verify with
- Mark the VHD file as online using diskpart (first atach the vdisk, select disk, then mark it online)
- Verify with chkdsk that the image was in fact without problems
- Detach the VHD file using diskpart
- Copy the VHD file to a HDD that Paragon HDM would recognise
- Use Paragon HDM to perform a V2P copy
- Paragon expects a .VD file, but if you ask it to use all file types, it does recognise that VHD files contain disk images
–jeroen
References:
- [WayBack] recovery – How to recover data and partition after diskpart clean – Super User
- [WayBack] TestDisk – Partition Recovery and File Undelete
- [Archive.is] Paragon Hard Disk Manager Suite – Overview
- [WayBack] DiskPart Command-Line Options
- [WayBack] DiskPart.exe and managing Virtual Hard Disks (VHDs) in Windows 7 – GES on Windows 7
- [WayBack] Disk2VHD will not end properly – Sysinternals Forums – Page 1
- [WayBack] Disk2vhd
Share this:
Posted in NTFS, Power User, Windows, Windows 10 | Leave a Comment »
17 years ago, C:\nul\nul crashed/BSOD Windows; now $MFT does for Windows < 10
Posted by jpluimers on 2017/05/26
Source:
History repeating itself: [Archive.is] 31607 – C:\nul\nul crashes/BSOD then, now it’s this:
- [WayBack] A bug in NTFS, or hang the entire system / Blog Aladdin RD / Habrahabr
- [WayBack] Баг в NTFS, или как подвесить всю систему / Блог компании Аладдин Р.Д. / Хабрахабр
- [WayBack] In a throwback to the ’90s, NTFS bug lets anyone hang or crash Windows 7, 8.1 | Ars Technica UK
- [WayBack] NTFS bug allows any website to crash Windows 7, 8.1
Via:
- [WayBack] NTFS bug allows any website to crash Windows 7, 8.1Users of older versions of Windows aren’t having the best time of it lately. Last week it was disco… – BetaNews – Google+
- [WayBack] Writing solid code is really hard these days … – Lars Fosdal – Google+
All versions prior to Windows 10 and Windows Server 2016 seem vulnerable.
So add $MFT to this list:
The following device names have been known to render a system unstable: CON, NUL, AUX, PRN, CLOCK$, COMx, LPT1, and CONFIG$.
Oh BTW: history repeated itself this year too. With NUL
In short, Steven Sheldon created a rust package named nul which broke the complete package manager on Windows:
- [WayBack] How I Broke Rust’s Package Manager for All Windows Users – sasheldon.com
- [WayBack] I Broke Rusts Package Manager for Windows Users | Hacker News
- [WayBack] I think a crate called ‘nul’ is causing errors for cargo (windows).
- [WayBack] @bstrie: @sheldonut Hi Steven, Rust team here :) We’re having some technical difficulties with the name of one of your crates… feel free to DM
- [WayBack] Null character – Wikipedia
- [WayBack] IRC Logs for #rust-internals | BotBot.me [o__o] – 2017-04-30 – 84953638 – page 4
- [WayBack] List of software package management systems – Wikipedia: Application-level package managers
- [WayBack] I think the name of this project causes issues for windows users. · Issue #1 · SSheldon/nul
- When cargo updates its crates.io registry, it creates a file for each crate, no extension, just the name.The name
nulis not a valid name in windows 10, so cargo fails to update the registry, and then aborts whatever it was doing (building, searching, ect.).I think this project should be re-published to crates.io under a new name, something likenull-stringsperhaps?
- When cargo updates its crates.io registry, it creates a file for each crate, no extension, just the name.The name
- [WayBack] Fresh install unable to install anything · Issue #3982 · rust-lang/cargo
- Here is what I see with verbose. I get this trying to install clippy or rustfmt. I assume I would get it with other crates.Updating registry
https://github.com/rust-lang/crates.io-index
error: [20/-1] Cannot checkout to invalid path ‘3/n/nul’
- Here is what I see with verbose. I get this trying to install clippy or rustfmt. I assume I would get it with other crates.Updating registry
- [WayBack] IRC Logs for #rust | BotBot.me [o__o] – 2017-04-30 – msg 84953614 – page=19
- [WayBack] What’s the deal with those reserved filenames like NUL and CON? – The Old New Thing
- The solution: [WayBack] Reserve windows crate names. by withoutboats · Pull Request #695 · rust-lang/crates.io
- [WayBack] @withoutboats boats ✨ on Twitter: “I thought Rust was supposed to prevent nul errors! ;-)”
BTW: one of my gripes on learning new languages is that they come with a whole new idiom of their ecosystem: rust, cargo, crates, all sound like being a truck mechanic to me.
–jeroen
Share this:
Posted in Development, Microsoft Surface on Windows 7, NTFS, Power User, Security, Software Development, The Old New Thing, Windows, Windows 10, Windows 7, Windows 8, Windows 8.1, Windows 9, Windows 95, Windows 98, Windows Defender, Windows Development, Windows ME, Windows NT, Windows Server 2000, Windows Server 2003, Windows Server 2003 R2, Windows Server 2008, Windows Server 2008 R2, Windows Server 2012, Windows Server 2012 R2, Windows Server 2016, Windows Vista, Windows XP | Leave a Comment »
The Wiert Corner - irregular stream of stuff 