The Wiert Corner – irregular stream of stuff

Jeroen W. Pluimers on .NET, C#, Delphi, databases, and personal interests

  • My badges

    Mastodon

  • Twitter Updates

  • My Flickr Stream

    20140508-Delphi-2007--Project-Options--Cannot-Edit-Application-Title-HelpFile-Icon-Theming20140430-Fiddler-Filter-Actions-Button-Run-Filterset-now20140424-VMware-Fusion-6.0.3.-no-reclaimable
    More Photos

  • Pages

  • All categories

  • Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 1,854 other subscribers

Archive for the ‘Python’ Category

« Previous Entries
Next Entries »

Python 2.7 Countdown: a year from now it is unsupported

Posted by jpluimers on 2019/01/01

Besides wishing you a happy new year, also a reminder: [WayBack] Python 2.7 Countdown Python 2.7 will retire on januari 1, 2020. Learn more and see the countdown here.

This is indeed a breaking change for Python users, similar as from Perl 4 to Perl 5, and PHP 4 to PHP 5.

It shows two things:

  • how extremely hard it is to evolve a language without breaking things
  • how long it takes for the community at large to digest breaking changes

And indeed porting of complex systems is hard [WayBack] WIP: Port calibre to python 3 by flaviut · Pull Request #870 · kovidgoyal/calibre · GitHub but doable [WayBack] Bug #1714107 “Python 2 is retiring” : Bugs : calibre.

Via: [WayBack1/WayBack2] Python 3 improves in some ways over Python 2, but also makes a bunch of changes that are breaking, but cosmetic (i.e. renaming methods and functions, or… – Kristian Köhntopp – Google+ (with some interesting comments, but also a rant-sequence of someone who would better use that energy to improve Python than to bash it).

–jeroen

Posted in Development, Python, Scripting, Software Development | Leave a Comment »

Python threading: logging exceptions during the `run`

Posted by jpluimers on 2018/12/19

In A few observations on Python while I made my first steps into it, I mentioned the standard threading idiom in Python by wrapping the thread in a function. This has the drawback of having to catch and handle any exceptions in that function.

The higher level [WayBack] threading module has a [WayBack] Thread class with a [WayBack] run() method does not handle exceptions either.

For investigation of threading issues, it’s very convenient to know about the exceptions in a thread and their context.

So I’ve made a small base class that automagically logs any exceptions during a run:

import threading

from log import Log

class ExceptionLoggingThread(threading.Thread):

    def __init__(self):
        threading.Thread.__init__(self)
        self.logger = Log().getLogger()
        self.logger.debug("ExceptionLoggingThread().")

    def run_logic(self):
        self.logger.debug("Thread started.")

    def run(self):
        try:
            self.run_logic()
        except:
            self.logger.exception('Exception in `run`')
            raise

–jeroen

Posted in Development, Python, Scripting, Software Development | Leave a Comment »

Parsing simple html in Python

Posted by jpluimers on 2018/11/29

Was working to get fritzcap to emit a list of interfaces so I could specify which one to capture.

For that I needed to parse the output of http://fritz.box/capture.lua which consists of HTML fragments like below.

What I needed was for each consecutive entries of [WayBack] th and first [WayBack] button tags:

  • content of the th tag
  • content of the value attribute of the button tag having a type="submit" attribute and name=start attribute

So before starting to work on it, I created [WayBack] In order to fix #5, print a list of available interfaces to potentially capture from · Issue #6 · jpluimers/fritzcap

The goal was to get a series of key/value pairs:

4-138 = AP2 (2.4 + 5 GHz, ath1) - Interface 1
4-137 = AP2 (2.4 + 5 GHz, ath1) - Interface 0
4-132 = AP (2.4 GHz, ath0) - Interface 1
4-131 = AP (2.4 GHz, ath0) - Interface 0
4-129 = HW (2.4 GHz, wifi0) - Interface 0
4-128 = WLAN Management Traffic - Interface 0a

So I built a class descending from [WayBack] HTMLParser — Simple HTML and XHTML parser that ships with the [WayBack] Python standard libraries.

If in the future I need more complex HTML parsing, then these links will help me choosing more feature rich parsers:

Back to the HTMLParser descendant in interfaces_dumper.py which can basically be condensed down to the code below.

  • handle_data is called for both start tags and end tags. The th value in data is only present in the start tag (at the time of end tag the data is empty), so you need to keep track of both last_start_tag and last_end_tag.
  • handle_endtag maintains last_end_tag to help handle_data.
  • handle_starttag maintains last_start_tag to help handle_data and also handles the button behaviour.
    • The buttonis only relevant if it has type="submit" and name="start" and a value attribute in that order.
    • Output is in data which is an array of key/value pairs.

Read the rest of this entry »

Posted in Development, Fritz!, Fritz!Box, fritzcap, Internet, Power User, Python, Scripting, Software Development | Leave a Comment »

A few observations on Python while I made my first steps into it

Posted by jpluimers on 2018/11/28

A while back, I made my first steps into Python.

Coming from a mixed language back-ground (including Pascal, Delphi, C#, SQL, batch files, PowerShell, bash, C, Java) it was an interesting experience.

A few observations:

More observations likely to follow.

–jeroen

Read the rest of this entry »

Posted in Development, Python, Scripting, Software Development | Leave a Comment »

Spelling with element symbols from the Periodic table

Posted by jpluimers on 2018/09/11

The [WayBack] Periodic table – Wikipedia contains many symbols.

Combing them allows you to spell word. Not all words, but many of them can be spelled.

So I was glad finding the below article that started with the same fascination I had in chemistry class.

[WayBack] Spelling with Elemental Symbols

It has a great explanation of the algorithm, references to computer science literature and a nice Python implementation.

via: [WayBack] One of the best programming articles I’ve read in a while – This is why I Code – Google+

–jeroen

Read the rest of this entry »

Posted in Algorithms, Development, Fun, LifeHacker, Power User, Python, science, Scripting, Software Development | Leave a Comment »

Just I in case I need to port CombineApacheConfig.py to OpenSuSE properly

Posted by jpluimers on 2018/07/24

I came across a nice tool that combines httpd.conf files:

python CombineApacheConfig.py /etc/apache2/httpd.conf /tmp/apache2.combined.conf

In case I ever need to fully port it to OpenSuSE, I’ve put it in the gist below.

For now it works fine on OpenSuSE when used with the above command. I might make the default depend on the kind of nx it runs on.

via:

–jeroen

Read the rest of this entry »

Posted in *nix, *nix-tools, Apache2, Development, Linux, openSuSE, Power User, Python, Scripting, Software Development, SuSE Linux | Leave a Comment »

I wish the Delphi language supported multi-line strings

Posted by jpluimers on 2018/06/21

Very often, I see people ask for how to embed multi-line strings in a Delphi source file.

The short answer is: you can’t.

The long answer is: you can’t and if you want you have to hack your way around.

The answer should be: just like any of these languages that do support multiline strings:

Many languages support this through a feature called HEREDOC.

Now in Delphi and other languages like Java are building ugly workarounds like for instance this one: [WayBack] RAD Studio Tip: Using resource scripts to organize project dependencies. – Chapman World.

–jeroen

Posted in .NET, C#, Delphi, Development, JavaScript/ECMAScript, Python, Ruby, Scripting, Software Development | 17 Comments »

Ben, blogging: Show the complete apache config file

Posted by jpluimers on 2018/03/20

Quite a while back, I got attended to Ben, blogging: Show the complete apache config file:

If you really want to see all the complete config settings, there is no existing tool for that. This Stack Overflow page  answered this question pretty well: You can use apachectl -S to see the settings of Virtual Host, or apachectl -M to see the loaded modules, but to see all settings, there is no such tool, you will have to go through all the files , starting from familiar yourself with the  general structure of the httpd config files.
… script …

The usage is simple: Run it as python  CombineApacheConfig.py . Since there is no additional parameters given, it will retrieve the default Ubuntu apache config file from  /etc/apache2/apache2.conf and generate the result complete config file in /tmp/apache2.combined.conf. If your config file is in different location, then give the input file and output file location.

Note: Apache server-info page http://127.0.0.1/server-info also provide similar information, but not in the config file format. It is in human readable format. The page works only when it is open from the same computer.

Since I could not find how to post comments there, and it works better for me having a repo, I put it into a gist with attribution to hist post: https://gist.github.com/jpluimers/fd300f3a500cbc78cd862d2a248e7b03
I need to adapt it for OpenSuSE; until then run it as this:
python CombineApacheConfig.py /etc/apache2/httpd.conf /tmp/apache2.combined.conf

–jeroen

 


#!/usr/bin/python2.7
# CombineApacheConfig.py
__author__ = 'ben'
import sys, os, os.path, logging, fnmatch
def Help():
print("Usage: python CombineApacheConfig.py inputfile[default:/etc/apache2/apache2.conf] outputfile[default:/tmp/apache2.combined.conf")
def InputParameter():
if len(sys.argv) <> 3:
Help()
return "/etc/apache2/apache2.conf", "/tmp/apache2.combined.conf"
return sys.argv[1], sys.argv[2]
def ProcessMultipleFiles(InputFiles):
Content = ''
LocalFolder = os.path.dirname(InputFiles)
basenamePattern = os.path.basename(InputFiles)
for root, dirs, files in os.walk(LocalFolder):
for filename in fnmatch.filter(files, basenamePattern):
Content += ProcessInput(os.path.join(root, filename))
return Content
def RemoveExcessiveLinebreak(s):
Length = len(s)
s = s.replace(os.linesep + os.linesep + os.linesep, os.linesep + os.linesep)
NewLength = len(s)
if NewLength < Length:
s = RemoveExcessiveLinebreak(s)
return s
def ProcessInput(InputFile):
Content = ''
if logging.root.isEnabledFor(logging.DEBUG):
Content = '# Start of ' + InputFile + os.linesep
with open(InputFile, 'r') as infile:
for line in infile:
stripline = line.strip(' \t')
if stripline.startswith('#'):
continue
if stripline.lower().startswith('include'):
match = stripline.split()
if len(match) == 2:
IncludeFiles = match[1]
IncludeFiles = IncludeFiles.strip('"') #Inserted according to V's comment.
if not IncludeFiles.startswith('/'):
LocalFolder = os.path.dirname(InputFile)
IncludeFiles = os.path.join(LocalFolder, IncludeFiles)
Content += ProcessMultipleFiles(IncludeFiles) + os.linesep
else:
Content += line # if it is not pattern of 'include(optional) path', then continue.
else:
Content += line
Content = RemoveExcessiveLinebreak(Content)
if logging.root.isEnabledFor(logging.DEBUG):
Content += '# End of ' + InputFile + os.linesep + os.linesep
return Content
if __name__ == "__main__":
logging.basicConfig(level=logging.DEBUG, format='[%(asctime)s][%(levelname)s]:%(message)s')
InputFile, OutputFile = InputParameter()
try:
Content = ProcessInput(InputFile)
except Exception as e:
logging.error("Failed to process " + InputFile, exc_info=True)
exit(1)
try:
with open(OutputFile, 'w') as outfile:
outfile.write(Content)
except Exception as e:
logging.error("Failed to write to " + outfile, exc_info=True)
exit(1)
logging.info("Done writing " + OutputFile)

view raw

CombineApacheConfig.py

hosted with ❤ by GitHub

Posted in *nix, *nix-tools, Apache2, Development, Linux, openSuSE, Power User, Python, Scripting, Software Development, SuSE Linux | Leave a Comment »

pandas/Pandas_Cheat_Sheet.pdf at master · pandas-dev/pandas

Posted by jpluimers on 2018/02/07

pandas – Flexible and powerful data analysis / manipulation library for Python, providing labeled data structures similar to R data.frame objects, statistical functions, and much more

Pandas is on my research list. Some links to get started:

–jeroen

Posted in Development, Python, Scripting, Software Development | Leave a Comment »

How did this shit ever work? – The Isoblog.

Posted by jpluimers on 2018/01/09

Though I like Python, I can feel the pain when maintaining in other’s code: [WayBack] How did this shit ever work? – The Isoblog reporting [WayBack] network.py does not handle interface resets properly · Issue #663 · python-diamond/Diamond.

Via: [WayBack] How did this shit ever work? – Kristian Köhntopp – Google+

Image from Reddit: parodies on O RLY Books: [WayBack] imgur.com/qErVcwA: Life on an Ops team…

–jeroen

Read the rest of this entry »

Posted in Development, Python, Scripting, Software Development | Leave a Comment »

« Previous Entries
Next Entries »