If I ever cross a PHP project, this is my first requirement to pass: [WayBack] Check your PHP project for known security issues – SensioLabs Security Advisories Checker
–jeroen
Archive for the ‘Scripting’ Category
Check your PHP project for known security issues – SensioLabs Security Advisories Checker
Posted by jpluimers on 2020/02/05
Posted in Development, PHP, Scripting, Software Development | Leave a Comment »
ρσℓα¢ķ͌͌͌͌͌͌͌͌͌͌͌͌͌͌ on Twitter : “The PowerShell console history file isn’t just useful for DFIR peeps, red team-ers should check it too! 4096 entries by default, here’s the path…”
Posted by jpluimers on 2020/01/15
Not sure why, but I checked a few of my systems and no file at %userprofile%\AppData\Roaming\Microsoft\Windows\PowerShell\PSReadline\ConsoleHost_history.txt
Heck: no ConsoleHost_history.txt files on those systems anywhere.
[WayBack] ρσℓα¢ķ͌͌͌͌͌͌͌͌͌͌͌͌͌͌ on Twitter : “The PowerShell console history file isn’t just useful for DFIR peeps, red team-ers should check it too! 4096 entries by default, here’s the path: %userprofile%\AppData\Roaming\Microsoft\Windows\PowerShell\PSReadline\ConsoleHost_history.txt”
–jeroen
Share this:
Posted in CommandLine, Power User, PowerShell, PowerShell | Leave a Comment »
Python – list transformation; string formatting – Stack Overflow
Posted by jpluimers on 2020/01/08
Sometimes simple examples are the best: [WayBack] Python – list transformation – Stack Overflow.
Interactive example (note you can run and save at repl.it in either [WayBack] Repl.it – Python 3 or [WayBack] Repl.it – Python 2; you can run but not save it at [WayBack] Welcome to Python.org: interactive Python shell):
# Links the documentation are Python 2, though everything works in Python 3 as well.
x = [1,2,3,4,5,11]
print("x: ", repr(x))
y = ['01','02','03','04','05','11']
print("y: ", repr(y))
# List comprehension https://docs.python.org/2/tutorial/datastructures.html#list-comprehensions
# ... using `str.format()` (Python >= 2.6): https://docs.python.org/2/library/stdtypes.html#str.format and https://docs.python.org/2/library/string.html#formatstrings
y = ["{0:0>2}".format(v) for v in x]
print("y: ", repr(y))
# ... using the traditional `%` formatting operator (Python < 2.6): https://docs.python.org/2/library/stdtypes.html#string-formatting y = ["%02d" % v for v in x] print("y: ", repr(y)) # ... using the format()` function (Python >= 2.6): https://docs.python.org/2/library/functions.html#format and https://docs.python.org/2/library/string.html#formatstrings
# this omits the "{0:...}" ceremony from the positional #0 parameter
y = [format(v, "0>2") for v in x]
print("y: ", repr(y))
# Note that for new style format strings, the positional argument (to specify argument #0) is optional (Python >= 2.7) https://docs.python.org/2/library/string.html#formatstrings
y = ["{:0>2}".format(v) for v in x]
print("y: ", repr(y))
# Using `lambda`
# ... Python < 3 return a list y = map(lambda v: "%02d" %v, x) print("y: ", repr(y)) # ... Python >= 3 return a map object to iterate over https://stackoverflow.com/questions/1303347/getting-a-map-to-return-a-list-in-python-3-x/1303354#1303354
y = list(map(lambda v: "%02d" %v, x))
print("y: ", repr(y))
Output:
Python 3 Python 2
–jeroen
Share this:
Posted in Conference Topics, Conferences, Development, Event, Python, Scripting, Software Development | Leave a Comment »
python – Why does “return list.sort()” return None, not the list? – Stack Overflow
Posted by jpluimers on 2020/01/02
list.sortsorts the list in place, i.e. it doesn’t return a new list. Just write
newList.sort()
return newList
The above answer is goden as performing return list.sort() bytes me often, because Python is usually an environment using the functional approach.
Answer from [WayBack] python – Why does “return list.sort()” return None, not the list? – Stack Overflow
Background information:
- [WayBack] 5. Data Structures — Python 2.7.15 documentation:
list.sort()
list.sort(cmp=None, key=None, reverse=False)- Sort the items of the list in place (the arguments can be used for sort customization, see
sorted()for their explanation).
- [WayBack] 5. Data Structures — Python 3.7.1rc1 documentation:
list.sort()
list.sort(key=None, reverse=False)- Sort the items of the list in place (the arguments can be used for sort customization, see
sorted()for their explanation).
–jeroen
Share this:
Posted in Development, Python, Scripting, Software Development | Leave a Comment »
Using a main check __main__ to call a main function in Python
Posted by jpluimers on 2020/01/01
[WayBack] __main__ — Top-level script environment — Python 3 documentation recommends code like this:
if __name__ == "__main__":
# execute only if run as a script
main()
This has many cool possibilities, including these that I like most as a beginning Python developer:
- having your
def main():function in a separate source file - allowing to
returnprematurely from yourmainfunction (you cannot do this from the main block) - allowing a file to distinguish if it is being loaded as a module, or as a main program
Related:
–jeroen
Share this:
Posted in Conference Topics, Conferences, Development, Event, Python, Scripting, Software Development | Leave a Comment »
How to make a self extracting archive runs your setup.exe, 7zip -sfx
Posted by jpluimers on 2020/01/01
For my link archive step by step instruction on the command-line which can be automated:
- [WayBack] How to make a self extracting archive runs your setup.exe, 7zip -sfx: How to make a self extracting archive that runs your setup.exe with 7zip -sfx switch, 7z -sfx. create self extracting exe 7zip
Via:
- [WayBack] How to create a 7zip sfx executable in batch mode (see the additional answers, the first one refers to the 7zip GUI). – Thomas Mueller (dummzeuch) – Google+
- [WayBack] 7 zip – Create 7-Zip SFX executable – Super User
–jeroen
Share this:
Posted in 7zip, Batch-Files, Compression, Development, Power User, Scripting, Software Development | Leave a Comment »
Delphi, decoding files to strings and finding line endings: some links, some history on Windows NT and UTF/UCS encodings
Posted by jpluimers on 2019/12/31
A while back there were a few G+ threads sprouted by David Heffernan on decoding big files into line-ending splitted strings:
- [WayBack] Having been a little underwhelmed by the performance of TStreamReader when reading huge text files line by line, I attempted to roll my own. I managed … – David Heffernan – Google+ where he compares the speed with Python (which runs circles around equivalent Delphi code)
- [WayBack] I just read this in TEncoding.GetBufferEncoding: function ContainsPreamble(const Buffer, Signature: array of Byte): Boolean; var I: Integer; … – David Heffernan – Google+ (worse: Delphi 209 and up have 3 different implementations of the ContainsPreamble function)
Code comparison:
Python:
with open(filename, 'r', encoding='utf-16-le') as f: for line in f: passDelphi:
for Line in TLineReader.FromFile(filename, TEncoding.Unicode) do ;
This spurred some nice observations and unfounded statements on which encodings should be used, so I posted a bit of history that is included below.
Some tips and observations from the links:
- Good old text files are not “good” with Unicode support, neither are TextFile Device Drivers; nobody has written a driver supporting a wide range of encodings as of yet.
- Good old text files are slow as well, even with a changed SetTextBuffer
- When using the TStreamReader, the decoding takes much more time than the actual reading, which means that [WayBack] Faster FileStream with TBufferedFileStream • DelphiABall does not help much
- TStringList.LoadFromFile, though fast, is a memory allocation dork and has limits on string size
- Delphi RTL code is not what it used to be: pre-Delphi Unicode RTL code is of far better quality than Delphi 2009 and up RTL code
- Supporting various encodings is important
- EBCDIC days: three kinds of spaces, two kinds of hyphens, multiple codepages
- Strings are just that: strings. It’s about the encoding from/to the file that needs to be optimal.
- When processing large files, caching only makes sense when the file fits in memory. Otherwise caching just adds overhead.
- On Windows, if you read a big text file into memory, open the file in “sequential read” mode, to disable caching. Use the FILE_FLAG_SEQUENTIAL_SCAN flag under Windows, as stated at [WayBack] How do FILE_FLAG_SEQUENTIAL_SCAN and FILE_FLAG_RANDOM_ACCESS affect how the operating system treats my file? – The Old New Thing
- Python string reading depends on the way you read files (ASCII or Unicode); see [WayBack] unicode – Python codecs line ending – Stack Overflow
Though TLineReader is not part of the RTL, I think it is from [WayBack] For-in Enumeration – ADUG.
Encodings in use
It doesn’t help that on the Windows Console, various encodings are used:
- Most tools still use Code Page 437 (from the good old DOS days, also known as CP437, OEM-US, OEM 437, PC-8, or DOS Latin US.)
- Unicode aware applications often use UTF-8
- A minority (but growing because of PowerShell: [WayBack] utf 8 – Changing PowerShell’s default output encoding to UTF-8 – Stack Overflow) of tools uses UTF-16 because Windows Unicode support started with UCS-2 with an in-memory little-endian representation (.REG files, MSXML, SQL Server Management Studio)
Good reading here is [WayBack] c++ – What unicode encoding (UTF-8, UTF-16, other) does Windows use for its Unicode data types? – Stack Overflow
Encoding history
+A. Bouchez I’m with +David Heffernan here:
At its release in 1993, Windows NT was very early in supporting Unicode. Development of Windows NT started in 1990 where they opted for UCS-2 having 2 bytes per character and had a non-required annex on UTF-1.
UTF-1 – that later evolved into UTF-8 – did not even exist at that time. Even UCS-2 was still young: it got designed in 1989. UTF-8 was outlined late 1992 and became a standard in 1993
Some references:
–jeroen
Share this:
Posted in Delphi, Development, Encoding, PowerShell, PowerShell, Python, Scripting, Software Development, The Old New Thing, Unicode, UTF-16, UTF-8, Windows Development | Leave a Comment »
Pythonic
Posted by jpluimers on 2019/12/24
When learning Python, one of the terms to get used to is Pythonic, basically shorthand for a loosely defined idiomatic Python way of writing code.
Some links to help you get a feel for this:
- [WayBack] The Idiomatic Way to Merge Dictionaries in Python – Trey Hunner
- Python (programming language) – Wikipedia: Features and philosophy
The language’s core philosophy is summarized in the document The Zen of Python (PEP 20), which includes aphorismssuch as:[51]
- Beautiful is better than ugly
- Explicit is better than implicit
- Simple is better than complex
- Complex is better than complicated
- Readability counts
…
the Python philosophy rejects exuberant syntax (such as that of Perl) in favor of a simpler, less-cluttered grammar. As Alex Martelli put it: “To describe something as ‘clever’ is not considered a compliment in the Python culture.”
…
“there should be one—and preferably only one—obvious way to do it”
- [WayBack] PEP 20 — The Zen of Python
- [WayBack] What is Pythonic? | Secret Weblog
- [WayBack] python – What does pythonic mean? – Stack Overflow
- [WayBack] Code Style — The Hitchhiker’s Guide to Python
- [Webcache] Learning the Pythonic Way
Sometime, I am going to dig into learning how to write Pythonic code for merging and joining dictionaries (preferably those of namedtuple entities). Hopefully these links will help me with that:
- [WayBack] How to use Python left outer join using FOR/LIST/DICTIONARY comprehensions (not SQL)? – Stack Overflow
- [WayBack] python left outer join and right outer join of dictonaries · GitHub
- [WayBack] Built-in Functions — Python 3.7.1rc2 documentation: `zip`
- [WayBack] python – How to merge two dictionaries in a single expression? – Stack Overflow
- [WayBack] python – Merge two dict in separate list by id – Code Review Stack Exchange
- [WayBack] The Idiomatic Way to Merge Dictionaries in Python – Trey Hunner
- [WayBack] python – using `map` to construct multiple named tuples – Stack Overflow
- [WayBack] Map-Reduce, Python and Named Tuples – DZone Web Dev
–jeroen
Share this:
Posted in Development, Python, Software Development | Leave a Comment »
Visual Studio Code: enable Python debugging and selecting the Python version used
Posted by jpluimers on 2019/12/18
A few links and screenshots for my archive (assuming development on MacOS):
Enable Python Debugging
- Start the debugger: key combination Shift-Command-D, or click the debug icon
- Click on the wheel with the red dot in the debugger pane:
, which will generate and open a launch.jsonfile in the current workspace, remote the red dot
and fill the drop down with debug configurations
Via:
- [WayBack] Python debugging configurations in Visual Studio Code Details on configuring the Visual Studio Code debugger for different Python applications
- [WayBack] Python in Visual Studio Code: Debugging
- [WayBack] Running Python scripts from Visual Studio Code – Stève Sfartz – Medium
Selecting the Python version
- Key combination Ctrl-Shift-P
- Type
Select Interpreter
- Select the Python version you want; on my system they were at the time of writing:
Via:
- [WayBack] Configuring Python Environments in Visual Studio Code
- [WayBack] python – How can I debug Python3 code in Visual Studio Code – Stack Overflow
- [WayBack] Use Virtualenv with Python with Visual Studio Code in Ubuntu – Stack Overflow As of September 2016 (according to the Github repo documentation of the extension) you can just execute a command from within vscode that will let you select the interpreter from an automatically
Setting command-line arguments
Commandline arguments are set in the same .vscode/launch.json file:
"args": [
"--quiet", "--norepeat"
],
Though [WayBack] Python debugging configurations in Visual Studio Code: args could have been more clear that you should put that under the Python configuration section you are debugging with, for instance:
{
// Use IntelliSense to learn about possible attributes.
// Hover to view descriptions of existing attributes.
// For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
"version": "0.2.0",
"configurations": [
{
"name": "Python: Current File (Integrated Terminal)",
"type": "python",
"request": "launch",
"program": "${file}",
"console": "integratedTerminal",
"args": [
"--quiet", "--norepeat"
]
},
Setting the startup python program
The page above also has a section on [WayBack] Python debugging configurations in Visual Studio Code: _troubleshooting that you can use to start the same script each time you debug, for instance your integration tests:
{
// Use IntelliSense to learn about possible attributes.
// Hover to view descriptions of existing attributes.
// For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
"version": "0.2.0",
"configurations": [
{
"name": "Python: Current File (Integrated Terminal)",
"type": "python",
"request": "launch",
// "program": "${file}",
"program": "${workspaceFolder}/snapperListDeleteFailures.FileTests.py",
Fazit
I should have read [WayBack] Get Started Tutorial for Python in Visual Studio Code first.
–jeroen
Share this:
Posted in Development, Python, Scripting, Software Development | Leave a Comment »
shell – Should I put #! (shebang) in Python scripts, and what form should it take? – Stack Overflow
Posted by jpluimers on 2019/12/17
It is very important to get the shebang correct. In case of Python, you both need env and the correct Python main version.
Answer
Correct usage for Python 3 scripts is:
#!/usr/bin/env python3This defaults to version 3.latest. For Python 2.7.latest use
python2in place ofpython3.Comment
envwill always be found in/usr/bin/, and its job is to locate bins (like python) usingPATH. No matter how python is installed, its path will be added to this variable, andenvwill find it (if not, python is not installed). That’s the job ofenv, that’s the whole reasonwhy it exists. It’s the thing that alerts the environment (set up env variables, including the install paths, and include paths).
Source: [WayBack] shell – Should I put #! (shebang) in Python scripts, and what form should it take? – Stack Overflow
Thanks GlassGhost and especially flornquake for the answer and Elias Van Ootegem for the comment!
The answer is based on [WayBack] PEP 394 — The “python” Command on Unix-Like Systems | Python.org.
The env is always in the same place, see env – Wikipedia and Shebang (Unix) – Wikipedia.
–jeroen
Share this:
Posted in Development, Python, Scripting, Software Development | Leave a Comment »
The Wiert Corner - irregular stream of stuff 