Lots of interesting tidbits on unix and NTFS file systems.
If you want to blow up your tooling, try creating a recursive hardlink…, which is likely one of the reasons that nx file systems do not support them.
Covered and related topics:
Posted by jpluimers on 2021/09/21
Lots of interesting tidbits on unix and NTFS file systems.
If you want to blow up your tooling, try creating a recursive hardlink…, which is likely one of the reasons that nx file systems do not support them.
Covered and related topics:
Posted in *nix, Development, File-Systems, History, NTFS, Power User, Software Development, Windows, Windows Development | Leave a Comment »
Posted by jpluimers on 2021/08/11
From an interesting twitter thread started by SwitftOnSecurity:
Interesting thought that I need to let sink in for a while before trying it.
Finding out about and fixing Limited User Account bugs:
More to think about:
To be more secure, users should log on with a Limited (or “Least-privileged”) User account (LUA), and use elevated privileges only for specific tasks that require them. Linux/Unix users have understood this for a long time
- The #1 reason for running as non-admin is to limit your exposure.
- My #2 reason for running as non-admin applies to developers. Developing software as User instead of Admin helps ensure that your software will run correctly on end-users’ systems.
- My #3 reason applies just to Microsoft personnel, particularly those of us in customer-facing roles. Hey, y’all! We need to lead by example.
Here’s how I set up home computers for friends and relatives:
- Create a Computer Administrator account called “Admin”. No password. (Read on before you flame.)
- Create a Limited User account for each person who will be using the computer. No passwords.
- Enable the Guest account if it is anticipated that visitors may need to go online.
I instruct all concerned that the Admin account is to be used only for installing software, and to use their individual accounts for all day-to-day use, including web, email, IM, etc. This has worked quite well for everyone I’ve done this for, and don’t get calls anymore about home pages being hijacked, etc. Users generally don’t even have to log out. My 7-year old walks away, the screen saver kicks in, my 3-year old moves the mouse and clicks on his picture (or the frog or whatever it is now) and has his own settings.
[added 2004.06.22]: I also like to make the admin desktop noticeably different from normal user desktops, to help prevent accidental use. For example, use the Windows Classic theme instead of the XP default, set a red background, or a wallpaper that says “For admin use only. Are you sure you need to be here?”
OK, I know you’re bursting already: “No password?!?! Are you insane?!?!” Cool down, now. Starting with Windows XP, a blank password is actually more secure for certain scenarios than a weak password. By default, an account with a blank password can be used only for logging on at the console. It cannot be used for network access, and it cannot be used with RunAs. The user experience of just clicking on your name to log on can’t be beat for simplicity. If you can trust everyone who has physical access to the computer not to log on as someone else or abuse the admin account, this is a great way to go. If not, you can always enable passwords.
The Secondary Logon service was first introduced in Windows 2000, and is in Windows XP and Server 2003. When you start a new process through RunAs, you provide credentials for the account you want the process to run under – for example, the local Administrator account. Assuming the credentials are valid, the Secondary Logon service then causes several things to happen:
- creates a new logon session for the specified account, with a new token;
- ensures that the new process’ token is granted appropriate access to the current window station and desktop (the specifics change somewhat for XP SP2, but aren’t important here);
- creates a new job in which the new process and any child processes it starts will run, to ensure that the processes are terminated when the shell’s logon session ends (correcting a problem with the NT4 Resource Kit’s SU utility).
MakeMeAdmin.cmd invokes RunAs twice, prompting you first for your local admin password, then for your current account password. The bit that runs as local administrator does the following:
- Adds your current account to the local Administrators group (using NET LOCALGROUP, avoiding the problem of needing network credentials to resolve names);
- Invokes RunAs to start a new instance of cmd.exe using your current account, which is at this instant a member of Administrators;
- Removes your current account from the local Administrators group.
The result of the second step is a Command Prompt running in a new logon session, with a brand new token representing your current account, but as a member of Administrators. The third step has no effect on the new cmd.exe’s token, in the same way that adding your account to Administrators does not affect any previously running processes.
In my first MakeMeAdmin post, there’s a section called “Objects created while running with elevated privilege,” the main parts of which I’ll recap here:
Normally, when a user creates a securable object, such as a file, folder, or registry key, that user becomes the “owner” of the object and by default is granted Full Control over it. Prior to Windows XP, if the user was a member of the Administrators group, that group, rather than the user, would get ownership and full control…. Windows XP introduced a configurable option whether ownership and control of an object created by an administrator would be granted to the specific user or to the Administrators group. The default on XP is to grant this to the object creator; the default on Windows Server 2003 is to grant it to the Administrators group….
If I use MakeMeAdmin to install programs, my normal account will be granted ownership and full control over the installation folder, the program executable files, and any registry keys the installation program creates. Those access rights will remain even when I am no longer running with administrator privileges. That’s not what I want at all. I want to be able to run the app, create and modify my own data files, but not to retain full control over the program files after I have installed it.
I concluded by saying:
For this reason, I changed the “default owner” setting on my computer to “Administrators group”.
Today I would like to go further: If you are going to use the same account for admin and non-admin activities (e.g., with MakeMeAdmin), I strongly recommend that you change the “Default owner” setting on your computer to “Administrators group”.
What becomes of all my earlier non-admin tips, tricks and recommendations vis-à-vis RunAs, MakeMeAdmin, PrivBar and their interactions with IE and Explorer? The short answer is that Vista changes just about everything with respect to running with least privilege.
Windows Vista makes running as a standard user (non-admin) much more pleasant, feasible and secure than it was on XP. I’m not going to drill into all those improvements here. Instead, the focus of this post is to update my earlier posts about running on XP as a standard user (the “Running as Admin Only When Required” posts in the Table of Contents) as they pertain to Windows Vista. To save some space, I’ll assume you’ve spent at least a little time running Vista.
…
> On XP/2003, MakeMeAdmin lets you run as a
> standard user, and temporarily elevate your
> standard account to run a selected program
> with administrative privileges.
Right. It doesn’t mean temporarily elevating your administrative account to run elevated, it means temporarily elevating your standard account to run a selected program with administrative privileges in the context of your account.
> Vista gives you the same ability
It does not. Here’s what Vista gives:
> If you are a member of the Administrators
> group on Vista
Exactly. It means temporarily elevating your administrative account to run elevated. It doesn’t help your standard account at all.
> “Run as administrator” serves as a superior
> substitute. With the default settings, a
> member of Administrators can use it as a
> MakeMeAdmin replacement
No, it is not a substitute, it’s different. A member of Administrators can use it to temporarily switch context to an administrative account and run elevated in the administrative account. If the administrator does this to install an application for all users then there’s no real problem, the application gets installed for all users just as it did in XP. But if the administrator wanted to do this to install an application for the standard user, they can’t do it. The administrator gets to install the application for one user’s account, which is going to be the administrator’s account, it’s not going to be the standard user’s account. The standard user doesn’t get the benefit that MakeMeAdmin provided.
Standard users in Vista still need a MakeMeAdmin tool.
Workaround:
Use Ctrl-Break instead.
[Added, March 9, 2005: While this problem occurs on Windows XP, it does not occur on Server 2003 RTM! ]
The bottom line is that the app runs with a “restricted token” that basically has these net effects:
- Group membership: If you were logged in as a member of Administrators, Power Users, or certain powerful domain groups, the app runs without the benefit of those group memberships.
- Registry: The app has read-only access to the registry, including HKEY_CURRENT_USER and HKEY_LOCAL_MACHINE. The app has no access to HKCU\Software\Policies.
- File system (assuming NTFS): The app cannot access the user’s profile directory at all. That includes “My Documents”, “Temporary Internet Files”, “Cookies”, etc.
- Privileges: The app has no system-wide privileges other than “Bypass traverse checking”.
IE works fairly well this way, but with some odd and annoying problems:
- You can’t use SSL (https) at all.
- If you right-click on a hyperlink and choose “Open in New Window”, nothing happens.
- If you enter a URL in the address bar without “http://” in front of it (e.g., “www.msn.com”), you get an error message like “C:\Documents and Settings\aaronmar\Desktop is not accessible. Access is denied.”, before IE goes ahead and loads the site anyway.
- On XP SP2 and on Server 2003, toolbars do not appear where you configured them, if they appear at all. E.g., PrivBar always needs to be re-enabled; “Links” appears (on my machine) in the upper left, to the left of the menu bar. (This wasn’t a problem with XP SP1.)
–jeroen
Posted in Development, Power User, Security, Software Development, Windows, Windows Development | Leave a Comment »
Posted by jpluimers on 2021/07/22
Based on zzz but filters current user, and listener session.
:: https://stackoverflow.com/questions/36715033/how-to-logoff-all-users-on-windows-from-command-line-as-a-domain-administrator :: The findstr bit filters out the current session (starts with ">") and session 65536 (which is the listener) for /f "skip=2 tokens=2,3 delims= " %%a in ('query session ^| findstr /v /b ">" ^| findstr /v "65536 Listen"') DO ( echo %%a|findstr /xr "[1-9][0-9]* 0" >nul && ( logoff %%a ) echo %%b|findstr /xr "[1-9][0-9]* 0" >nul && ( logoff %%b ) ) goto :eof
–jeroen
Posted in Batch-Files, Development, Scripting, Software Development, Windows Development | Leave a Comment »
Posted by jpluimers on 2021/07/07
There are quite a few posts that recommend using SetProcessWorkingSetSize to trim your process working set, usually in the SetProcessWorkingSetSize(ProcessHandle, -1, -1) form:
[WayBack] SetProcessWorkingSetSize function (winbase.h) | Microsoft Docs
Sets the minimum and maximum working set sizes for the specified process.
BOOL SetProcessWorkingSetSize( HANDLE hProcess, SIZE_T dwMinimumWorkingSetSize, SIZE_T dwMaximumWorkingSetSize );The working set of the specified process can be emptied by specifying the value
(SIZE_T)–1for both the minimum and maximum working set sizes. This removes as many pages as possible from the working set. The [WayBack] EmptyWorkingSet function can also be used for this purpose.
In practice you hardly ever have to do this, mainly because this will write – regardless of (dis)usage – all of your memory to the pagefile, even the memory your frequently use.
Windows has way better heuristics to do that automatically for you, skipping pages you frequently use.
It basically makes sense in a few use cases, for instance when you know that most (like 90% or more) of that memory is never going to be used again.
Another use case (with specific memory sizes) is when you know that your program is going to use a defined range of memory, which is outside what Windows will heuristically expect from it.
A few more links that go into more details on this:
SetProcessWorkingSetSize()controls the amount of RAM that your process uses, it doesn’t otherwise have any affect on the virtual memory size of your process. Windows is already quite good at dynamically controlling this, swapping memory pages out on demand when another process needs RAM.By doing this manually, you slow down your program a lot, causing a lot of page faults when Windows is forced to swap the memory pages back in.
SetProcessWorkingSetSizeis typically used to increase the amount of RAM allocated for a process. Or to force a trim when the app knows that it is going to be idle for a long time. Also done automatically by old Windows versions when you minimize the main window of the app.
The only good use case I’ve seen for this call is when you KNOW your process is going to hog a lot of the system’s RAM and you want to reserve it for the duration. You use it to tell the OS “Yes, I’m going to eat a lot of the system RAM during my entire run and don’t get in my way”.
We have found out that, for a GUI application written in Delphi for Win32/Win64 or written in a similar way that uses large and heavy libraries on top of the Win32 API (GDI, etc), it is worth calling
SetProcessWorkingSetSizeonce.We call it with -1, -1 parameters, within a fraction of second after the application has fully opened and showed the main window to the user. In this case, the
SetProcessWorkingSetSize(... -1, -1)releases lots of startup code that seem to not needed any more.
Don’t pinvoke this, just use the
Process.CurrentProcess.MinWorkingSetproperty directly.Very high odds that this won’t make any difference. Soft paging faults are entirely normal and resolved very quickly if the machine has enough RAM. Takes ~0.7 microseconds on my laptop. You can’t avoid them, it is the behavior of a demand_paged virtual memory operating system like Windows. Very cheap, as long as there is a free page readily available.
But if it “blips” you program performance then you need to consider the likelihood that it isn’t readily available and triggered a hard page fault in another process. The paging fault does get expensive if the RAM page must be stolen from another process, its content has to be stored in the paging file and has to be reset back to zero first. That can add up quickly, hundreds of microseconds isn’t unusual.
The basic law of “there is no free lunch”, you need to run less processes or buy more RAM. With the latter option the sane choice, 8 gigabytes sets you back about 75 bucks today. Complete steal.
I had an application which by default would close down entirely but keep listening for certain events. However, most of my code at that point would not be needed for a long time. To reduce the impact my process made, I called
SetProcessWorkingSetSize(-1,-1);. This meant Windows could take back the physical RAM and give it to other apps. I’d get my RAM back when events did arrive.That’s of course unrelated to your situation, and I don’t think you’d benefit.
If your goal is for your application to use less memory you should look elsewhere. Look for leaks, look for heap fragmentations look for optimisations and if you think FastMM is keeping you from doing so you should try to find facts to support it. If your goal is to keep your workinset size small you could try to keep your memory access local. Maybe FastMM or another memory manager could help you with it, but it is a very different problem compared to using to much memory.
…
you can check the FasttMM memory usage via FasttMM calls GetMemoryManagerState and GetMemoryManagerUsageSummary before and after calling API SetProcessWorkingSetSize.
…
I don’t need to use SetProcessWorkingSetSize. FastMM will eventually release the RAM.
To confirm that this behavior is generated by FastMM (as suggested by Barry Kelly) I crated a second program that allocated A LOT of RAM. As soon as Windows ran out of RAM, my program memory utilization returned to its original value.
Yes, it’s a bad thing. You’re telling the OS that you know more about memory management than it does, which probably isn’t true. You’re telling to to page all your inactive memory to disk. It obeys. The moment you touch any of that memory again, the OS has to page it back into RAM. You’re forcing disk I/O that you don’t actually know you need.
If the OS needs more free RAM, it can figure out which memory hasn’t been used lately and page it out. That might be from your program, or it might be from some other program. But if the OS doesn’t need more free RAM, then you’ve just forced a bunch of disk I/O that nobody asked for.
If you have memory that you know you don’t need anymore, free it. Don’t just page it to disk. If you have memory that the OS thinks you don’t need, it will page it for you automatically as the need arises.
Also, it’s usually unwise to call
Application.ProcessMessagesunless you know there are messages that your main thread needs to process that it wouldn’t otherwise process by itself. The application automatically processes messages when there’s nothing else to do, so if you have nothing to do, just let the application run itself.
–jeroen
Posted in .NET, C, C++, Delphi, Development, Software Development, Windows Development | Leave a Comment »
Posted by jpluimers on 2021/07/01
For a very long time, it has been possible to name threads visible in debuggers: How to: Set a Thread Name in Native Code.
In the mean time, under Windows 10, you can both Get and Set the thread name. This brings native applications on par with with .NET where this has always been possible. Chrome uses these new API calls.
Which means I have some reading to do:
–jeroen
Posted in Debugging, Development, Software Development, Windows Development | Leave a Comment »
Posted by jpluimers on 2021/06/24
Interesting takeaway from [WayBack] DCOM calls from thread pool threads
call
CoInitialize*at the start, and callCoUninitializebefore returning. Expensive, but necessary
Related:
–jeroen
Posted in .NET, C, C++, COM/DCOM/COM+, Delphi, Development, Software Development, Windows Development | Leave a Comment »
Posted by jpluimers on 2021/06/22
Windows if full of undocumented gizmo’s, like find alternative for wc -l counting all lines in a file: [WayBack] File Line Count
Use
FINDcommand to count file lines, store line count into a variable.
Description: Running the FIND command with option /v and empty search string will find all lines
Running the FIND command with option /c will output the line count only.
The FOR command with option /f will parse the output, the line count in this case, and the set command put the line number into the cnt variable.Script:
1.
2.
3.
4.
set file=textfile.txt
set /a cnt=0
for /f %%a in ('type "%file%"^|find "" /v /c') do set /a cnt=%%a
echo %file% has %cnt% lines
Script Output:
Script Output
textfile.txt has 50 lines
[WayBack] Stupid command-line trick: Counting the number of lines in stdin | The Old New Thing
Windows doesn’t come with
wc,
but there’s a sneaky way to count the number of lines anyway:some-command-that-generates-output | find /c /v ""It is a special quirk of the
findcommand
that the null string is treated as never matching.
The/vflag reverses the sense of the test,
so now it matches everything.
And the/cflag returns the count.…
The reason dates back to the original MS-DOS
version offind.exe,
which according to the comments appears to have been written
in 1982.
And back then, pretty much all of MS-DOS was written in assembly
language.…
Via: batch file line count – Google Search and [WayBack] windows – How to count no of lines in text file and store the value into a variable using batch script? – Stack Overflow
–jeroen
Posted in Batch-Files, Development, Scripting, Software Development, Windows Development | Leave a Comment »
Posted by jpluimers on 2021/06/01
In addition to [WayBack] c++ – DLL Load Library – Error Code 126 – Stack Overflow while debugging LoadLibrary error 126
Windows dll error 126 can have many root causes. The most useful methods I have found to debug this are:
- Use dependency walker to look for any obvious problems (which you have already done)
- Use the sysinternals utility [WayBack] Process Monitor – Windows Sysinternals | Microsoft Docs from Microsoft to trace all file access while your dll is trying to load. With this utility, you will see everything that that dll is trying to pull in and usually the problem can be determined from there.
Search for the first entry NAME NOT FOUND after your library is being loaded.
It indicates the module that cannot be found which indirectly causes error number 126 (ERROR_MOD_NOT_FOUND in [WayBack] System Error Codes (0-499) – Windows applications | Microsoft Docs ) in [WayBack] LoadLibrary:
ERROR_MOD_NOT_FOUND
- 126 (0x7E)
The specified module could not be found.
This is why I upvoted the very relevant comment:
… when I looked at the rows surrounding my dll being loaded I saw MSVCP140D.dll was giving a result of
NAME NOT FOUND. Turns out the machine that couldn’t load my dll doesn’t have the ‘D’ version of MSVCP140.dll. Everything worked when I built my dll for release! – [WayBack] Pakman.
–jeroen
Posted in Delphi, Development, Software Development, Windows Development | Leave a Comment »
Posted by jpluimers on 2021/05/27
For some Chocolatey installations, I got an error message like this one:
"ERROR: Running ["C:\Users\Developer\AppData\Local\Temp\chocolatey\vmware-tools\10.3.10.12406962\VMware-tools-10.3.10-12406962-x86_64.exe" /S /v /qn REBOOT=R ] was not successful. Exit code was '3010'. See log for possible error messages."
I wish that a Chocolatey install could indicate it is msi based, and Chocolatey would convert this to a soft reboot message, as [WayBack] MsiExec.exe and InstMsi.exe Error Messages – Windows applications | Microsoft Docs indicates it means ERROR_SUCCESS_REBOOT_REQUIRED:
Posted in Chocolatey, Development, InnoSetup, Installer-Development, Power User, Software Development, Windows, Windows Development | Leave a Comment »
Posted by jpluimers on 2021/05/20
In Windows, historically most people approach investigation GUI first. Having turned 50 a while ago, I am no exception.
My real roots however are on the command-line and scripting: roughly 1980s Apple DOS, CP/M, SunOS (yay sh Bourne shell!), MS-DOS, 4DOS, and VAX/VMS (yay DCL shell!), from the 1990s on, some Solaris, a little bit of AIX, HP-UX and quite a bit of Linux, MacOS (né OS/X né Mac OS), and some BSD descendants derivatives (SunOS, AIX and MacOS are based on the Berkeley Software Distribution), and this century a more growing amount of PowerShell).
So I was glad to find out the makers of Process Explorer also made [WayBack] ListDLLs – Windows Sysinternals | Microsoft Docs (via windows get dlls loaded in process – Google Search)
List all the DLLs that are currently loaded, including where they are loaded and their version numbers.
ListDLLs is a utility that reports the DLLs loaded into processes. You can use it to list all DLLs loaded into all processes, into a specific process, or to list the processes that have a particular DLL loaded. ListDLLs can also display full version information for DLLs, including their digital signature, and can be used to scan processes for unsigned DLLs.
Usage
listdlls [-r] [-v | -u] [processname|pid]
listdlls [-r] [-v] [-d dllname]
Parameter Description processname Dump DLLs loaded by process (partial name accepted). pid Dump DLLs associated with the specified process id. dllname Show only processes that have loaded the specified DLL. -r Flag DLLs that relocated because they are not loaded at their base address. -u Only list unsigned DLLs. -v Show DLL version information.
Download: [WayBack] ListDlls.zip.
Now it is much easier to generate a draft deploy list of DLLs (and for Delphi: BPLs) based on a process running on a development machine.
Example output (the -r flags relocation warnings; the first part is the [WayBack] shim that Chocolatey created around the second which is from SysInternals):
Posted in Conference Topics, Conferences, Delphi, Development, Event, History, Software Development, Windows Development | Leave a Comment »