The Wiert Corner – irregular stream of stuff

Jeroen W. Pluimers on .NET, C#, Delphi, databases, and personal interests

  • My badges

    Mastodon

  • Twitter Updates

  • My Flickr Stream

    20140508-Delphi-2007--Project-Options--Cannot-Edit-Application-Title-HelpFile-Icon-Theming20140430-Fiddler-Filter-Actions-Button-Run-Filterset-now20140424-VMware-Fusion-6.0.3.-no-reclaimable
    More Photos

  • Pages

  • All categories

  • Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 1,828 other subscribers

Archive for the ‘Scripting’ Category

« Previous Entries
Next Entries »

Mac OS X and python “ValueError: unknown locale: UTF-8”

Posted by jpluimers on 2016/05/03

On Mac OS X, to solve the Python error “ValueError: unknown locale: UTF-8“:

Add some lines to your ~/.bash_profile then re-start bash (or re-login):

export LC_ALL=en_US.UTF-8
export LANG=en_US.UTF-8

–jeroen

via python – Pelican 3.3 pelican-quickstart error “ValueError: unknown locale: UTF-8” – Stack Overflow.

Posted in Development, Python, Software Development | Leave a Comment »

reStructuredText notes (.rst file extension)

Posted by jpluimers on 2016/04/28

Thanks to Eric Grange who asked Which lightweight markup language? I learned about reStructuredText (no cap R!) from a few comments Joseph Mitzen made.

It looks like reStructuredText has been around for much longer than Markdown, has better features (#1 for me: it is unambiguous, #2: native support on GitHub), but isn’t as popular. I think the latter is because finding editors supporting a live preview for it is a bit hard and tools are scattered around the net.

So here are a few notes on how I got reStructuredText to work on my Mac using OS X.

The hardest part was getting the reStructuredText preview for Atom to work:

  1. Verify you have recent apm/npm
  2. Download, then install the latest Mac Pandoc release (filename like “pandoc-*-osx.pkg”).
  3. Open Atom
  4. Menu “Atom” -> “Preferences” to open a “Settings” tab in the Atom user interface
  5. In the list “Settings”, “Keybindings”, “Packages”, “Themes”, “Updates”, “Install”, choose the last one: “Install”
  6. In the “Search packages” textbox, type “language-restructuredtext”, then hit Enter
  7. Wait a few seconds until “language-restructuredtext” appears in the list, then click the “Install”  button next to it
  8. In the “Search packages” textbox, type “rst-preview-pandoc”, then hit Enter
  9. Wait a few seconds until “rst-preview-pandoc” appears in the list, then click the “Install” button next to it
  10. To enable spell checking:
    1. In “Settings”, go to “Packages”
    2. Search for “spell-check”
    3. Click “settings”
    4. Add “gfm.restructuredtext” to the “Grammars” list
    5. Note you get “gfm.restructuredtext” from the “language-restructuredtext” package as described in Spell check in Atom – Atom quick tip #3 – Atom Editor Tips and Tricks.
  11. Restart Atom from the command-line (otherwise it will not find pandoc *)
  12. Open a reStructuredText file
  13. Press Ctrl-Shift-e to show the preview
  • pandoc error:

The error you get when pandoc cannot be found is this one:

‘pandoc’ could not be spawned. Is it installed and on your path? If so please open an issue on the package spawning the process.

It is easy to solve by modifying the Atom startup shell script and then don’t start Atom.app, but start atom from the command-line in a terminal window:

atom

For Windows:

  1. Install Chocolatey Gallery.
  2. Follow the steps at Hello Windows.
  3. Fails horribly: “The term ‘Install-ChocolateyPackage’ is not recognized as the name of a cmdlet”.
    1. Split-Path bug preventing Chocolatey to install a package · Issue #686 · chocolatey/chocolatey · GitHub.

A later try to get Pandoc installed on Windows was much easier: there is a Pandoc for Windows installer now.

Settings changes

I made a few, for instance:

  1. Ensure Tab to expands to spaces: See Soft Tabs and Tab Length under Settings > Editor Settings. via github – Atom – Change indentation mode – Stack Overflow.

Tables

Tables are always a hard thing in any markup. Luckily truben.no/table/ has a good table editor (it’s the same as table-editor.com) and can emit reStructuredText, Markdown, HTML and other formats.

More in the future

Give me some time, and I will post more about using the format and how it compares to my Markdown past.

Note that pandoc does not fully support reStructuredText (for instance not all table features are supported), but docutils rst2html.py does and also gives better warning/error information when parsing.

Here are some links about the reStructured syntax and how they can be rendered by rst2html.py:

For now, I’ll end with the goals of reStructuredText which I really like:

Read the rest of this entry »

Posted in Development, MarkDown, Perl, PHP, Power User, Python, reStructuredText, Scripting, Software Development | Leave a Comment »

findstr as alternative for recursive grep search

Posted by jpluimers on 2016/04/27

Usually I use the old Borland grep.exe that still ships with Delphi. Too bad it is 16-bit app which does not recognise Unicode.

FindStr does. Though much slower and with limited regular expression capabilities, can do recursive searches too:

findstr /spin /c:"string to find" *.*

The /spin is a shortcut for these case insensitive command-line options (the full list of possible options is below):

  /S         Searches for matching files in the current directory and all
             subdirectories.
  /I         Specifies that the search is not to be case-sensitive.
  /N         Prints the line number before each line that matches.
  /P         Skip files with non-printable characters.

Sometimes I leave out the /P to include binary files.

–jeroen

via:

Read the rest of this entry »

Posted in Batch-Files, Development, Power User, RegEx, Scripting, Software Development, Windows, Windows 7, Windows 8, Windows 8.1, Windows NT, Windows Server 2000, Windows Server 2003, Windows Server 2003 R2, Windows Server 2008, Windows Server 2008 R2, Windows Server 2012, Windows Server 2012 R2, Windows Vista, Windows XP | Leave a Comment »

Detecting the use of “curl | bash” server side | Application Security

Posted by jpluimers on 2016/04/20

From the “let people know when they are stupid” department:

Another reason not to pipe from curl to bash. Detecting curl | bash serverside.

Source: Detecting the use of “curl | bash” server side | Application Security

–jeroen

Posted in Development, Python, Scripting, Software Development | 1 Comment »

Cygwin first run and a few aliases

Posted by jpluimers on 2016/04/06

Cygwin first run looks like this:

Copying skeleton files.
These files are for the users to personalise their cygwin experience.

They will never be overwritten nor automatically updated.

'./.bashrc' -> '/home/jeroenp//.bashrc'
'./.bash_profile' -> '/home/jeroenp//.bash_profile'
'./.inputrc' -> '/home/jeroenp//.inputrc'
'./.profile' -> '/home/jeroenp//.profile'

jeroenp@msmxp ~
$

It will copy some default files to your profile so you can modify them.

Since Cygwin will run ~/.bash_profile on logon, and that in turn starts ~/.bashrc (see below), I’ve modified the latter to run ~/.bash_aliases and bring those a bit in sync with my regular Mac machine:


# New user-defined bash aliases file
# Is executed from a modified .bashrc file
# .bashrc is executed from .bash_profile on cygwin
# on other systems, see http://www.joshstaiger.org/archives/2005/07/bash_profile_vs.html
## This is the portion in .bashrc you need to edit in cygwin to enable .bash_aliases:
## Aliases
##
## Some people use a different file for aliases
## if [ -f "${HOME}/.bash_aliases" ]; then
## source "${HOME}/.bash_aliases"
## fi
alias ls-8601='ls -l -T'
alias ls-full-8601='ls -l –time-style=full-iso'
# octal file modes through http://stackoverflow.com/questions/1795976/can-the-unix-list-command-ls-output-numerical-chmod-permissions
alias lsmod='ls -al|awk '\''{k=0;s=0;for(i=0;i<=8;i++){;k+=((substr($1,i+2,1)~/[rwxst]/)*2^(8-i));};j=4;for(i=4;i<=10;i+=3){;s+=((substr($1,i,1)~/[stST]/)*j);j/=2;};if(k){;printf("%0o%0o ",s,k);};print;}'\'''
# silence the progress meter on Cygwin with -sS via http://stackoverflow.com/questions/7373752/how-do-i-get-curl-to-not-show-the-progress-bar
alias whatismyip='curl -sS http://whatismyip.akamai.com && echo'

view raw

.bash_aliases

hosted with ❤ by GitHub

cURL has some idiosyncrasies, for instance Cygwin shows the progress meter by default, but Mac OS X does not. I wanted to disable the cURL progress meter and you heed -sS for that.

ls doesn’t show you octal file modes by default, but chown and umask use them, so I’ve got the lsmod alias through stack-overflow.

Bash initialisation

Getting your bash initialisation right can be tough. There are lengthy discussions about which .bash* files run under what circumstances:

–jeroen

Posted in bash, Development, Scripting, Software Development | Leave a Comment »

stop/start IIS

Posted by jpluimers on 2016/03/03

I know, old knowledge, but I only recently added the below batch files to file collection.

Why? Because since a few Windows versions, the System process uses port 80 because IIS is installed by default in many configurations. And recently I had to do quote a bit of http communication work against a local machine outside the IIS realm.

windows 7 – Why is System process listening on Port 80? – Super User.

Stop IIS:

:: http://stackoverflow.com/questions/22084561/difference-between-iisreset-and-iis-stop-start-command
:checkPrivileges
  net file 1>nul 2>nul
  if '%errorlevel%' == '0' ( goto :gotPrivileges ) else ( goto :getPrivileges )

:isNotAdmin
:getPrivileges
  echo You need to be admin running with an elevated security token to run %0
  goto :exit

:isAdmin
:gotPrivileges
::  net stop w3svc
::  net stop iisadmin
  iisreset /stop

:exit
  ::pause
  exit /b

Start IIS:

:: http://stackoverflow.com/questions/22084561/difference-between-iisreset-and-iis-stop-start-command
:checkPrivileges
  net file 1>nul 2>nul
  if '%errorlevel%' == '0' ( goto :gotPrivileges ) else ( goto :getPrivileges )

:isNotAdmin
:getPrivileges
  echo You need to be admin running with an elevated security token to run %0
  goto :exit

:isAdmin
:gotPrivileges
::  net start w3svc
::  net start iisadmin
  iisreset /start

:exit
  ::pause
  exit /b

–jeroen

Posted in Batch-Files, Development, IIS, Scripting, Software Development | Leave a Comment »

rsync as diff – compare files in two directory on remote server using unix – Stack Overflow

Posted by jpluimers on 2016/03/01

Too bad the accepted answer forgets about deleted files.

Use one of these to compare (but not sync) two directory trees.

For size-only comparision:

rsync -n -avr --size-only --delete /abc/home/sample1/ server2:/abc/home/sample2/

If you want to compare both contents and size:

rsync -n -avrc --delete /abc/home/sample1/ server2:/abc/home/sample2/

–jeroen

via: diff – compare files in two directory on remote server using unix – Stack Overflow.

Posted in *nix, bash, Development, Power User, rsync, Scripting, Software Development | Leave a Comment »

Windows: removing file and directory reparse points (symbolic links, directory links, junctions, hard links)

Posted by jpluimers on 2016/02/02

The interwebs is full of posts telling about how to create file and directory junctions**.

But there is little information about removing them and even less being correct: some suggest to del a directory junction (which just deletes everything in it but the junction).

Finally there is little information about listing all junctions, so lets start with that:

Deleting a link depends on the kind of link, not the kind of source.

Since symlink and hardlinks are for files, and directory symlink and junctions are for directories, this is how:

  • Delete a file symlink or hardlink by using DEL.
  • Delete a directory symlink or junction using RMDIR.

SysInternals – I wrote about them before – has a great junction tool. It can be used to create, delete and (optionally recursively) list reparse points. All usages allow for file and directory junctions.

More about reparse points

This is about the **: actually they are reparse points; for files they are symlinks, for directories mostly junctions, but sometimes symlinks.

And actually the reason I wrote this blog post. As you also have hardlinks. Some combinations of files and directories with these kinds of links fail.

Lets first go to see what kind of links there are on a fresh Windows system.

This is the only directory symlink: C:\Users\All Users and junction will show it like this:

.\\?\C:\\Users\All Users: SYMBOLIC LINK
   Print Name     : C:\ProgramData
   Substitute Name: \??\C:\ProgramData

It is unlike this directory junction C:\Users\Default User which junction will show as this:

\\?\C:\\Users\Default User: JUNCTION
   Print Name     : C:\Users\Default
   Substitute Name: C:\Users\Default

Together with C:\Users\Default and C:\Users\desktop.ini they are hidden, so you need the /AH flag to show them using DIR (as a gist, since WordPress still screws up less than and greater than):


Directory of C:\Users
08/22/2013 04:45 PM <SYMLINKD> All Users [C:\ProgramData]
09/30/2013 06:27 AM <DIR> Default
08/22/2013 04:45 PM <JUNCTION> Default User [C:\Users\Default]
08/22/2013 05:34 PM 174 desktop.ini

view raw

show-hidden-files-in-users.bat

hosted with ❤ by GitHub

When you look at the examples below, it is odd to see that C:\Users\All Users is a SYMLINK and not a SYMLINKD as it points to a directory.

And yes, there are not so and very subtle differences between SYMLINKD and JUNCTION.

Lets show some examples.

The examples are hopefully more complete than the complete guide.

Since symlinks are client side created and not verified until use, you can actually use mklink to create both file and directory symbolic links for a file. DIR shows them as SYMLINK or SYMLINKD.

A SYMLINK to a file actually works, but a SYMLINKD or JUNCTION to a file gives you an Access Denied error. Hardlinks get the attributes of the source (so delete hidden hardlinks using the DEL /AH option).

Example batch file:

view raw

show-mklink-symlink-symlinkd-junction-junction-hardlink-on-file.bat

hosted with ❤ by GitHub

Example output:

view raw

show-mklink-symlink-symlinkd-junction-junction-hardlink-on-file.bat.txt

hosted with ❤ by GitHub

When you try this for directories, you are in for a few small surprises.

A SYMLINK to a directory neither works as file nor as directory. A SYMLINKD or JUNCTION to a directory works. Hardlinks don’t work for directories with reason: limit the risk of cycles.

Example batch file:

view raw

show-mklink-symlink-symlinkd-junction-junction-hardlink-on-file.bat

hosted with ❤ by GitHub

Example output:

view raw

show-mklink-symlink-symlinkd-junction-junction-hardlink-on-file.bat.txt

hosted with ❤ by GitHub

Conclusion

  • symlink and hardlink can be used as files, but not as directories.
  • files referenced through symlinkd and junction behave as empty directories.
  • symlinkd and junction can be used as directories, but not as files.
  • directories referenced as symlink are not usable.
  • directories cannot function as hardlink source.
  • hardlinks to files inherited their attributes.

–jeroen

Posted in Batch-Files, Development, Scripting, Software Development | 2 Comments »

Bash Coding Style · drwetter/testssl.sh Wiki

Posted by jpluimers on 2016/01/31

Awesome read on bash Coding Style · drwetter/testssl.sh Wiki

Posted in bash, Development, Scripting, Software Development | Leave a Comment »

Some more tf.exe related batch files for managing workspaces – via: Find an installed tf.exe, then run it with the command-line parameters specified. « The Wiert Corner

Posted by jpluimers on 2016/01/28

As part of some TFS related posts, I wrote about Find an installed tf.exe, then run it with the command-line parameters specified.

Below are some more batch files related to this. In each batch file, you can replace tf with call "%~dp0tf.bat" so the above batch file is executed first.

Read the rest of this entry »

Posted in Batch-Files, Development, Scripting, Source Code Management, TFS (Team Foundation System) | Leave a Comment »

« Previous Entries
Next Entries »