The Wiert Corner – irregular stream of stuff

Jeroen W. Pluimers on .NET, C#, Delphi, databases, and personal interests

  • My badges

  • Twitter Updates

  • My Flickr Stream

  • Pages

  • All categories

  • Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 1,679 other followers

Long pathname support: Watch for MAX_PATH (was: Windows pathname max length problem « Dropbox Forums)

Posted by jpluimers on 2010/06/16

When you want to support long pathnames on Windows, you need to watch for the MAX_PATH limitation.
Some tools, like RoboCopy (developed in C++), Beyond Compare (developed in Delphi) and others get it right.
Getting it right does not depend in your development environment: it is all about calling the right API’s with the right parameters.

Let me take a tool – in this case DropBox, though other tools suffer from the same problem – and investigate how they should do it.

Even though DropBox is cross platform, the Windows version of DropBox limits itself to synchronizing files having less than 260 characters in their path.
This is a big drawback: it is so 20th century having a limitation like this.

It is not a limitation of the Windows API: most Windows API functions don’t suffer from the (DOS-era!) MAX_PATH limitation.
To work around the limitation, just use the wide string API calls and prefix your pathnames with \\?\ (a magic marker that serves a few other purposes too, boy I wish you could use that as a search term on Google <g>).

The limitation is attributed (see links below, DropBox knows about this more than a year) to be caused by the Python version that DropBox uses.

I just verified, and version 0.7110.0 of DropBox uses PYTHON25.DLL with a version of 2.5.4, which was released on 2008-12-23 (yes, that is YYYY-MM-DD).
That is pretty odd: The current Python production versions are Python 2.6.5 and Python 3.1.2.
But I don’t think that version matters much for the DropBox problem.

Therefore I did a MAX_PATH search in the sources of both current production versions: there are not that many MAX_PATH references (see lists below), so it is not clear that Python is indeed the limitation.
I also checked out the MAX_PATH string in the python.org domain, and could not find results that really point to Python as the limiting factor.

Even though the Python people should try to get rid of this MAX_PATH limitation, the solution for DropBox seems to be to select the right calls in Python that support the \\?\ prefix using wide string API calls, and test if everything works.

As a (maybe lame) defense to Python and DropBox: .NET suffers from the same limitation as well (see the StackOverflow question Accessing files beyond MAX_PATH in C#/.NET) and Kim Hamilton wrote a few nice blog entries on how to work around it.

To read more about this, see these two posts in the DropBox forums:

  1. Some files in my DropBox are not updating
  2. Windows pathname max length problem « Dropbox Forums.

This has been a problem in DropBox for
Pitty, as otherwise DropBox is a great tool:

  1. it is one of the few tools that both provide secure storage as well,
  2. allows synchronization between different devices,
  3. supports multiple platforms,
  4. can share folders between different people

–jeroen

As promised above, the MAX_PATH references in the Python sources.

Python 2.6.5:

2.6.5
Lib/ctypes/wintypes.py:MAX_PATH = 260
Lib/ctypes/wintypes.py:                ("cFileName", c_char * MAX_PATH),
Lib/ctypes/wintypes.py:                ("cFileName", c_wchar * MAX_PATH),
Lib/ctypes/wintypes.py:           'LPCWSTR', 'LPOLESTR', 'LPSTR', 'LPVOID', 'LPWSTR', 'MAX_PATH',
Misc/NEWS:  returned a path longer than MAX_PATH. (But It's doubtful this code path is
Modules/posixmodule.c:  char new_path[MAX_PATH+1];
Modules/posixmodule.c:  result = GetCurrentDirectoryA(MAX_PATH+1, new_path);
Modules/posixmodule.c:     than MAX_PATH. */
Modules/posixmodule.c:  assert(result <= MAX_PATH+1);
Modules/posixmodule.c:   since the current directory might exceed MAX_PATH characters */
Modules/posixmodule.c:  wchar_t _new_path[MAX_PATH+1], *new_path = _new_path;
Modules/posixmodule.c:  result = GetCurrentDirectoryW(MAX_PATH+1, new_path);
Modules/posixmodule.c:  if (result > MAX_PATH+1) {
Modules/posixmodule.c:  char namebuf[MAX_PATH+5]; /* Overallocate for \\*.*\0 */
Modules/posixmodule.c:  Py_ssize_t len = sizeof(namebuf)-5; /* only claim to have space for MAX_PATH */

Modules/posixmodule.c:#ifndef MAX_PATH
Modules/posixmodule.c:#define MAX_PATH    CCHMAXPATH
Modules/posixmodule.c:    char namebuf[MAX_PATH+5];
Modules/posixmodule.c:    if (len >= MAX_PATH) {
Modules/posixmodule.c:  char inbuf[MAX_PATH*2];
Modules/posixmodule.c:  char outbuf[MAX_PATH*2];
Modules/posixmodule.c:                  Py_UNICODE woutbuf[MAX_PATH*2], *woutbufp = woutbuf;
Modules/posixmodule.c:                  char modulepath[_MAX_PATH];
Modules/posixmodule.c:    if (ins(module, "maxpathlen",   values[QSV_MAX_PATH_LENGTH])) return -1;
Modules/tkappinit.c:#ifndef MAX_PATH_LEN
Modules/tkappinit.c:#define MAX_PATH_LEN 1024
Modules/tkappinit.c:    char tclLibPath[MAX_PATH_LEN], tkLibPath[MAX_PATH_LEN];
Modules/tkappinit.c:            tclLibPath, MAX_PATH_LEN, 0);
Modules/tkappinit.c:            tkLibPath, MAX_PATH_LEN, 1);
Modules/_hotshot.c:#   ifdef MAX_PATH
Modules/_hotshot.c:#       define PATH_MAX MAX_PATH
PC/bdist_wininst/extract.c:     char pathname[MAX_PATH];
PC/bdist_wininst/install.c:char modulename[MAX_PATH];

PC/bdist_wininst/install.c:char install_script[MAX_PATH];
PC/bdist_wininst/install.c:char python_dir[MAX_PATH];
PC/bdist_wininst/install.c:char pythondll[MAX_PATH];
PC/bdist_wininst/install.c:     char Buffer[MAX_PATH + 64];
PC/bdist_wininst/install.c:     char lpszPath[MAX_PATH];
PC/bdist_wininst/install.c:     WCHAR wszFilename[MAX_PATH];
PC/bdist_wininst/install.c:                         wszFilename, MAX_PATH);
PC/bdist_wininst/install.c:     char fullpath[_MAX_PATH];
PC/bdist_wininst/install.c:     char tempdir[MAX_PATH];
PC/bdist_wininst/install.c:     ini_file = malloc(MAX_PATH); /* will be returned, so do not free it */
PC/bdist_wininst/install.c:char bound_image_dll[_MAX_PATH];
PC/bdist_wininst/install.c:     char fname[_MAX_PATH];
PC/bdist_wininst/install.c:     char vers_name[_MAX_PATH + 80];
PC/bdist_wininst/install.c:     char pathname[_MAX_PATH];
PC/bdist_wininst/install.c:    char prefix[MAX_PATH+1]; // sys.prefix directory.
PC/bdist_wininst/install.c:                                             char install_path[_MAX_PATH];
PC/bdist_wininst/install.c:     char buffer[_MAX_PATH+1];
PC/bdist_wininst/install.c:     char buffer[_MAX_PATH+1];
PC/bdist_wininst/install.c:                     char fname[MAX_PATH];
PC/bdist_wininst/install.c:     char exename[_MAX_PATH];
PC/bdist_wininst/install.c:     char batname[_MAX_PATH];
PC/bdist_wininst/install.c:     static char lastscript[MAX_PATH];
PC/getpathp.c:          TCHAR keyBuf[MAX_PATH+1];
PC/getpathp.c:          DWORD reqdSize = MAX_PATH+1;
PC/getpathp.c:     PLUS Windows itself defines MAX_PATH as the same, but anyway...
PC/os2vacpp/getpathp.c:         char keyBuf[MAX_PATH+1];
PC/os2vacpp/getpathp.c:                                       index, keyBuf, MAX_PATH+1);
PC/os2vacpp/getpathp.c:                                       index, keyBuf,MAX_PATH+1);
PC/VS8.0/kill_python.c:    wchar_t  path[MAX_PATH+1];
PC/VS8.0/kill_python.c:    memset(path, 0, MAX_PATH+1);
PC/VS8.0/kill_python.c:        len = wcsnlen_s(me.szExePath, MAX_PATH) - KILL_PYTHON_EXE_LEN;
PC/VS8.0/kill_python.c:        wcsncpy_s(path, MAX_PATH+1, me.szExePath, len);
PC/_subprocess.c:       TCHAR filename[MAX_PATH];
PC/_subprocess.c:       result = GetModuleFileName(module, filename, MAX_PATH);
PC/_subprocess.c:       filename[MAX_PATH-1] = '\0';
PCbuild/kill_python.c:    wchar_t  path[MAX_PATH+1];
PCbuild/kill_python.c:    memset(path, 0, MAX_PATH+1);
PCbuild/kill_python.c:        len = wcsnlen_s(me.szExePath, MAX_PATH) - KILL_PYTHON_EXE_LEN;
PCbuild/kill_python.c:        wcsncpy_s(path, MAX_PATH+1, me.szExePath, len);
Python/sysmodule.c:     char fullpath[MAX_PATH];

Python 3.1.2:

3.1.2
Lib/ctypes/wintypes.py:MAX_PATH = 260
Lib/ctypes/wintypes.py:                ("cFileName", c_char * MAX_PATH),
Lib/ctypes/wintypes.py:                ("cFileName", c_wchar * MAX_PATH),
Lib/ctypes/wintypes.py:           'LPCWSTR', 'LPOLESTR', 'LPSTR', 'LPVOID', 'LPWSTR', 'MAX_PATH',

Modules/posixmodule.c:  char new_path[MAX_PATH+1];
Modules/posixmodule.c:  result = GetCurrentDirectoryA(MAX_PATH+1, new_path);
Modules/posixmodule.c:     than MAX_PATH. */
Modules/posixmodule.c:  assert(result <= MAX_PATH+1);
Modules/posixmodule.c:   since the current directory might exceed MAX_PATH characters */
Modules/posixmodule.c:  wchar_t _new_path[MAX_PATH+1], *new_path = _new_path;
Modules/posixmodule.c:  result = GetCurrentDirectoryW(MAX_PATH+1, new_path);
Modules/posixmodule.c:  if (result > MAX_PATH+1) {
Modules/posixmodule.c:  char namebuf[MAX_PATH+5]; /* Overallocate for \\*.*\0 */
Modules/posixmodule.c:  Py_ssize_t len = sizeof(namebuf)-5; /* only claim to have space for MAX_PATH */
Modules/posixmodule.c:  if (PyObject_Size(opath)+1 > MAX_PATH) {
Modules/posixmodule.c:#ifndef MAX_PATH
Modules/posixmodule.c:#define MAX_PATH    CCHMAXPATH
Modules/posixmodule.c:    char namebuf[MAX_PATH+5];
Modules/posixmodule.c:    if (len >= MAX_PATH) {

Modules/posixmodule.c:  char outbuf[MAX_PATH*2];
Modules/posixmodule.c:                  Py_UNICODE woutbuf[MAX_PATH*2], *woutbufp = woutbuf;

Modules/posixmodule.c:    if (ins(module, "maxpathlen",   values[QSV_MAX_PATH_LENGTH])) return -1;
Modules/tkappinit.c:#ifndef MAX_PATH_LEN
Modules/tkappinit.c:#define MAX_PATH_LEN 1024
Modules/tkappinit.c:    char tclLibPath[MAX_PATH_LEN], tkLibPath[MAX_PATH_LEN];
Modules/tkappinit.c:            tclLibPath, MAX_PATH_LEN, 0);
Modules/tkappinit.c:            tkLibPath, MAX_PATH_LEN, 1);

PC/bdist_wininst/extract.c:     char pathname[MAX_PATH];
PC/bdist_wininst/install.c:char modulename[MAX_PATH];
PC/bdist_wininst/install.c:wchar_t wmodulename[MAX_PATH];
PC/bdist_wininst/install.c:char install_script[MAX_PATH];
PC/bdist_wininst/install.c:char python_dir[MAX_PATH];
PC/bdist_wininst/install.c:char pythondll[MAX_PATH];
PC/bdist_wininst/install.c:     char Buffer[MAX_PATH + 64];
PC/bdist_wininst/install.c:     char lpszPath[MAX_PATH];
PC/bdist_wininst/install.c:     WCHAR wszFilename[MAX_PATH];
PC/bdist_wininst/install.c:                         wszFilename, MAX_PATH);
PC/bdist_wininst/install.c:     char fullpath[_MAX_PATH];
PC/bdist_wininst/install.c:     char tempdir[MAX_PATH];
PC/bdist_wininst/install.c:     ini_file = malloc(MAX_PATH); /* will be returned, so do not free it */
PC/bdist_wininst/install.c:char bound_image_dll[_MAX_PATH];
PC/bdist_wininst/install.c:     char fname[_MAX_PATH];
PC/bdist_wininst/install.c:     char vers_name[_MAX_PATH + 80];
PC/bdist_wininst/install.c:     char pathname[_MAX_PATH];
PC/bdist_wininst/install.c:    char prefix[MAX_PATH+1]; // sys.prefix directory.
PC/bdist_wininst/install.c:                                             char install_path[_MAX_PATH];
PC/bdist_wininst/install.c:     char buffer[_MAX_PATH+1];
PC/bdist_wininst/install.c:     char buffer[_MAX_PATH+1];
PC/bdist_wininst/install.c:                     char fname[MAX_PATH];
PC/bdist_wininst/install.c:     char exename[_MAX_PATH];
PC/bdist_wininst/install.c:     char batname[_MAX_PATH];
PC/bdist_wininst/install.c:     static char lastscript[MAX_PATH];
PC/getpathp.c:          WCHAR keyBuf[MAX_PATH+1];
PC/getpathp.c:          DWORD reqdSize = MAX_PATH+1;

PC/os2vacpp/getpathp.c:         char keyBuf[MAX_PATH+1];
PC/os2vacpp/getpathp.c:                                       index, keyBuf, MAX_PATH+1);
PC/os2vacpp/getpathp.c:                                       index, keyBuf,MAX_PATH+1);
PC/VS8.0/kill_python.c:    wchar_t  path[MAX_PATH+1];
PC/VS8.0/kill_python.c:    memset(path, 0, MAX_PATH+1);
PC/VS8.0/kill_python.c:        len = wcsnlen_s(me.szExePath, MAX_PATH) - KILL_PYTHON_EXE_LEN;
PC/VS8.0/kill_python.c:        wcsncpy_s(path, MAX_PATH+1, me.szExePath, len);
PC/_subprocess.c:       WCHAR filename[MAX_PATH];
PC/_subprocess.c:       result = GetModuleFileNameW(module, filename, MAX_PATH);
PC/_subprocess.c:       filename[MAX_PATH-1] = '\0';
PCbuild/kill_python.c:    wchar_t  path[MAX_PATH+1];
PCbuild/kill_python.c:    memset(path, 0, MAX_PATH+1);
PCbuild/kill_python.c:        len = wcsnlen_s(me.szExePath, MAX_PATH) - KILL_PYTHON_EXE_LEN;
PCbuild/kill_python.c:        wcsncpy_s(path, MAX_PATH+1, me.szExePath, len);
Python/sysmodule.c:     wchar_t fullpath[MAX_PATH];

6 Responses to “Long pathname support: Watch for MAX_PATH (was: Windows pathname max length problem « Dropbox Forums)”

  1. […] I know those filenames are long, and sometimes you bump into tools that suffer from MAX_PATH pathname length issues, but most built-in Windows functionality is OK with this, and the quality tools are […]

  2. Using \\?\ is like running the gauntlet because not the whole unicode API supports it (as the MAX_PATH msdn doc reads). And there are also some differences between the IO- and ShellAPI so that not all files can be shown in the shell.
    Furthermore there are a lot of structures which are returned by various APIs and contain path arrays like


    PSHFileInfoW = ^TSHFileInfoW;
    {$EXTERNALSYM _SHFILEINFOW}
    _SHFILEINFOW = packed record
    hIcon: HICON; // out: icon
    iIcon: Integer; // out: icon index
    dwAttributes: DWORD; // out: SFGAO_ flags
    szDisplayName: array[0..MAX_PATH - 1] of WideChar; // out: display name (or path)
    szTypeName: array[0..79] of WideChar; // out: type name
    end;

    Just do a search for MAX_PATH in JWA or windows headers.

    And of course, a lot of Libraries and Applications do not understand \\?\

  3. Bertrand said

    What also drives me mad is that Microsoft encourages the use of the desktop and My Documents folder. Nothing wrong with that in principle, but:

    You now have the entire C:\Documents and Settings\[username]\My Documents\ prefix eating into your available file name length, too. Add some fat urls, and you frequently get issues during file transfer.

    • jpluimers said

      I think that Microsoft realized that too.
      Since Windows Vista, it is now X:\Users\[username]\Documents
      Still long, but not as long :-)
      –jeroen

  4. Interesting, but I don’t quite understand how the Python sources are of interest to a Delphi developer.

    Surely of more interest would be the uses of MAX_PATH in the VCL source – a much bigger problem/issue/concern for a Delphi developer hoping to avoid this issue, no?

    And if I understood the point of your post correctly, the results of a Find In Files for MAX_PATH in the VCL source is not very encouraging.

    • jpluimers said

      The post lost some edits, which I now reapplied.

      I knew that Beyond Compare was getting it right (it is written in Delphi) as well as RoboCopy (written in C++).
      That got me thinking it was an API thing.

      My example shows how to track that down for DropBox (written in Pyton) to help their team finding the right solution.
      It shows a pattern that any developer – including Delphi Developers – can use.

      –jeroen

Leave a Reply to Jolyon Smith Cancel reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

 
%d bloggers like this: