The Efficient Programmer

Faster Command Execution

Tue, 09 Jul 2019 00:00:00 +0200

I had presented in a previous post how I can easily rerun a previous command on the command line. I used this approach to speed the edit-run cycle for scripting. A better way yet is to use a custom Vim binding to execute the previous command using tmux’s send-keys mechanism. Furthermore, one can directly switch to the tmux window where that command is running.

The necessary tmux commands are the following:

# Send the keys "Up" (arrow up) and "Enter" to the pane <pane> in window <window>.
tmux send-keys -t :<window>.<pane> Up Enter
# Switch to the last (previously active) window.
tmux select-window -t :{last}

If we switch to the last window and then send the keys, we do not even have to give tmux send-keys the target window/pane because by default the currently active pane is used. This results in the following Vim mapping:

" Save current file and rerun previous command in previous tmux window.
map <leader>t :up<CR>:execute 'silent !tmux select-window -t :{last} && tmux send-keys Up Enter'<CR>

This works right after one has executed the command in question once in the other (i.e. non-Vim) tmux window. From that point on, pressing <leader>t results in “instant” script execution.

You can see the desired behavior in the following video. In tmux window 1, a shell script is executed once. In window 0, Vim is used to modify the script. After pressing <leader>t, the script is saved, tmux jumps to back to window 1 and executes the previous command (i.e. the script) again. All in one go.

Run SQL Queries In Background

Fri, 22 Mar 2019 00:00:00 +0100

When working with a large database, some SELECT queries can take really long. And “really long” starts at a minute or two, of course. When designing queries, one often limits the result set because one is not yet actually interested in the final data.

However, you’ll want to have the result eventually. In the meantime, you probably want to play around further with your queries. But you don’t want to keep your current database connection (say, the GUI used for query design) busy. I happen to be in this situation quite often and have written a little script which allows me to run the queries in the background.

I had the following requirements

Flexible parametrization
Measure execution time
Keep results
Allow to reproduce the data easily

The script works as allows: One has to provide a file containing the query to be executed. The input file gets copied (for traceability/reproducibility) and executed. Execution time gets logged and one can provide further arguments like hostname or user/password.

After a couple of iterations, below is what the final script looks like. It is tied to MySQL but that doesn’t matter and can be adapted to your needs, of course.

#!/bin/bash

QUERYFILE=$1
OUTDIR=$2
ARGS=("${@:3}") # everything but the first 2 arguments
if [[ "$#" -le 1 || ! -r "$QUERYFILE" ]] ; then
    echo "Usage: $0 /path/to/query.sql /path/to/existing/outdir [mysql arguments]"
    exit 1
fi
OUTFILEPREFIX=$OUTDIR/$(basename "${QUERYFILE}")_$(date +%Y%m%d_%H%M%S)

echo "Query:"
cat "$QUERYFILE"
echo "Results: $OUTFILEPREFIX*"
echo "Mysql: ${ARGS[*]}"

# Redirect all output to the log file.
# https://serverfault.com/a/103569/234378
exec 3>&1 4>&2
trap 'exec 2>&4 1>&3' 0 1 2 3
exec 1>"$OUTFILEPREFIX.log" 2>&1

set -x # be verbose
cp "$QUERYFILE" "$OUTFILEPREFIX.sql" # copy input file
date
time mysql "${ARGS[@]}" < "${QUERYFILE}" > "${OUTFILEPREFIX}.tsv" # run the query
date

(Download)

Let me explain the different steps in more detail. The only mandatory arguments are the input query file and the output directory. The output directory will contain a copy of the input file, the log file and the results.

The output files get timestamp’ed in order to differentiate between several invocations that use the same (possibly modified) input file. Before running the query, the input parameters are printed to the screen for visual inspection. Then, all further output is redirected to the log file and the input file gets copied. Finally, the query is invoked. The date and time invocations allow to retrospectively study the timings.

Typically, the script is then executed like so

# Input file.
cat query.sql
use mydatabase;
SELECT * FROM mytable;

# Execute the query at server dbhost using username dbuser and providing a
# password at the prompt.
$SCRIPTS/runsql.sh query.sql /outdir -h dbhost -u dbuser -p

# Possible results.
ls -1 /outdir
query.sql_20190322_140308.log
query.sql_20190322_140308.sql
query.sql_20190322_140308.tsv

The input get gets copied oth the output .sql file. The output .tsv will contain the tab-separated result data. If any errors occurred, they can be found in the .log file. In order to see them immediately, I usually run

# Execute the script and show the latest (tail -n 1) log file immediately
# afterwards.
Q=query.sql ; O=/outdir ; $SCRIPTS/runsql.sh $Q $O ; cat $(ls -1 $O/$(basename $Q)*log | tail -n 1)

I use the script regularly to execute a long-running query in a new tmux pane. Sometimes, I even forget about the query. Next day at work, I happily find all the results I need!

How To Make Your Own Paper Wallet

Sat, 06 Oct 2018 00:00:00 +0200

It is common (and secure) practice to save cryptocurrency/blockchain keys on paper. There exists several tools for the various blockchains to generate a public/private key pair and the corresponding the paper wallet at once. In this post, I would like to show you how I created my custom paper wallets.

TL;DR: Generate QR Codes from the public/private keys, paste everything into a simple document and print it.

Whenever I created an account for any blockchain, I made sure I got hold of the private key; or at least of the seed phrase. No matter what the creation method was (website, local wallet software, browser plugin, etc.): I want to have control over my stakes!

Now, what’s a paper wallet? I am gonna assume a paper wallet is simply an offline (physical) document that stores the necessary data to access a blockchain account. Depending on the blockchain technology, this might only be the private key or the seed phrase of an account. Furthermore, the public key as well as some non-mandatory information is handy. For convenience, the hard-to-remember-and-easy-to-mistype strings are often encoded as Quick Response Code.

In practice, my approach is the following: I generate QR code images of the keys and write a simple markdown document with all necessary information. Finally, I use Pandoc to generate a PDF from that file that can be easily printed.

The QR code can be generated with various tools. It is important to do this offline - do not paste any sensitive information to a website that creates QR codes. I did use qrcode for generation in the past which is simply enough.

# Generate QR Codes
qr "WHATEVERTHEPRIVATEKEYIS" > private.png
qr "WHATEVERTHEPUBLICKEYIS" > public.png
qr "SOME RANDOM WORDS FOR THE SEED PHRASE" > seed.png

Then, we’ll make a simple markdown document with all information needed. Note that I only list two blockchain accounts here but more can be used, of course.

# Stellar - XLM
~~~

Account created on 2018-10-06 with
https://www.stellar.org/laboratory/#account-creator?network=public.

Use https://www.stellar.org/account-viewer/#!/ for access.

public: WHATEVERTHEPUBLICKEYIS
private: WHATEVERTHEPRIVATEKEYIS

            public                      private
~~~

![public](./public.png){ width=200px } ![private](./private.png){ width=200px }

# Cardano - ADA

~~~
Account created on 2018-10-06 with Cardano wallet Daedalus v1.1.0.960.
Use https://daedaluswallet.io/ for access.

seed: "SOME RANDOM WORDS FOR THE SEED PHRASE"

            seed
~~~

![seed](./seed.png){ width=200px } \

(Download)

Finally, we’ll simply generate a PDF from that input …

# Generate a PDF from the Markdown file. Tell Pandoc to reduce the default
# (LaTeX) margins.
pandoc -V geometry:margin=1.5cm -s -o paperwallet.pdf paperwallet.md

… giving us our paper wallet:

Less nice than other tools but it does the job! Importantly, this approach makes it possibly to independently create a “paper wallet” for already existing accounts where not printable output has been auto-generated.

Actually, it would be good to add even further hints to the paper wallet on how to access the funds. This would help my clueless inheritors ;)

Endless Command Execution

Mon, 28 May 2018 00:00:00 +0200

I happen to find myself regularly in the situation of writing a throw away BASH script. Often, I need to run the script several times before I have it implemented correctly. This is the typical edit-(compile-)run cycle most programmers know. The trial-and-error workflow must be of course as fast as possible. I am already using tmux so switching between the editor and the console where the script runs is pretty fast. Still, I need to switch to that window, recall the previous command and execute it again. The appropriate tmux/shell key sequence would be <tmux-prefix>4<CAPSLOCK>k<ENTER>. Let’s examine this in detail

<tmux-prefix>4 	Jump to the window (in this case window #4) where the script is run.
<CAPSLOCK>	Enter Normal mode in Zsh. I am using vi-mode in the shell and have mapped CAPSLOCK to ESC
k		Scroll up to the previous command. This is the previously executed script, of course.
<ENTER>		Execute the command.

These are still quite a lot of keystrokes for something that is going to be repeated all the time. Luckily, I have two aliases in my alias file list which allow me to run a command in an endless while loop.

# While True Start
alias wts='while true ; do echo WHILE BEGIN $(date); '
# While True End
alias -g wte=' ; echo WHILE EXIT $?; echo WHILE END $(date); read ; done'

The usage is pretty simple:

$ wts echo "runme" wte
WHILE BEGIN Mon May 28 15:05:01 CEST 2018
runme
WHILE EXIT 0
WHILE END Mon May 28 15:05:01 CEST 2018

When I press Enter now, I get the same command repeated again and again. So my edit-run cycle keystrokes are now <tmux-prefix>4<ENTER>. Plus, I don’t have to retype the while loop every time I am in the edit-run cycle.

This works of course only due to the -g feature in Zsh’s alias built-in which allows to define global aliases, i.e. aliases that are expanded not only at the beginning of the command. Typing two distinct aliases is a bit cumbersome. A more intuitive solution would be a shell function:

wtse () {
        while true
        do
                echo WHILE BEGIN $(date)
                $@
                echo WHILE EXIT $?
                echo WHILE END $(date)
                read
        done
}

However, this gets in your way when using more “advanced” shell magic. Watch:

$ wts ls | tee my.output wte
# vs
$ wtse ls | tee my.output

In the second case, the file (my.output) will also contain the additional information that was printed by the function. This was arguably not intended and so keeping the original while loop aliases makes sure you do not have to worry about redirections & pipes.

The date commands serve two purposes. Firstly, they make it visible that the command has run and finished; just in case the execution didn’t output anything. Finally, they allow to get a feeling how long the command took to execute.

There are further ways to shorten the edit-run cycle. One could setup a trigger in the editor to run the command in the background or send the appropriate tmux keys to run it in another window/pane. I haven’t tested these options (yet). I think the trigger-in-the-editor approach is cumbersome to setup because one has to “configure” the editor every time so that it knows what to run. Redirection/pipes might also be a problem.

By “sending the tmux keys” I mean that one could alternatively instruct the editor to tell tmux to run the command in another window/pane. This would of course assume that the other window/pane is prepared for that. I.e. one would have had run the command at least once to be able to send the generic instruction “run again”. If that is the case, a simple tmux send-keys -t "<window>.<pane>" Escape k Enter should do. This assumes that the shell uses vi bindings and <ESC>k<ENTER> scrolls up and executes the last command. Again, here I feel the “burden” of the setup (finding the window/pane or always use the preset ones). Furthermore, it is less convenient to change the actual command being run.

Some Grep Tips

Sun, 11 Mar 2018 00:00:00 +0100

In this post, I am going to present some simple tips when using grep.

Default Grep Command

I am using my custom grep command grr by default whenever I would run grep. It is defined as follows:

alias grr="grep --recursive --line-number \
    --exclude-dir=.git --exclude-dir=.svn \
    --exclude='*.o' --exclude='*.pyc' --exclude=.ctags"

The command is called grr for “GRep Recursive”. The --recursive switch is self-explanatory. The --line-number option makes grep prepend each match with the line where it was found.

The --exclude switches can be changed depending on the actual needs. There are tools dedicated specifically to search in source code folders. However, the above alias can be used on any machine - not only within the personal development environment.

A word of caution: Do not set an alias for grep without changing the name, i.e. don’t do alias grep="grep --recursive ...". This may bite you in shell scripts where the author “blindly” uses grep and thus your custom grep is called.

Use Colors And Pipes

I have already written about G as a better alias for grep when piping. Defining …

alias -g G='| grep'

… allows to run:

some_command G mypattern

An improved version of this alias keeps the color code from the input:

alias -g G='| grep --color=always'

This is really handy when piping grep into grep like so:

$ cat input.txt
aa
ab
ac
bb
bc
cc

$ cat input.txt | grep a | grep -v b
aa
ac

With color=alwyas set up grep will keep the input color codes arriving at stdin. I really like to pipe grep when it is easy to spell out the search pattern with multiple greps (“search for lines containing this but not that”). Without that option, only the last grep would colorize - whatever the last pattern than was.

A Grep Wrapper

At work, there is a database dump containg lots of customer information. The file roughly looks like this:

customer1|somedata|otherdata|whateverdata|...
customer1|evenmoredata|otherdata|dododata|...
customer2|somereallystrangedata|otherdata...
customer2|evenmoredata|otherdata|...
customer3|somedata|otherdata|...
customer3|areyoustillreadingthis?|otherdata|...
...

I often find myself to search for data for a specific customer. For various reasons it is actually easier to grep for the customer in the dump file instead of querying the database. For this very end I have defined a dedicated shell function:

grc () { # GRep Customer
    FILE="/path/to/dump.txt"
    ls -hl $FILE
    grep --color $@ $FILE
}

This function is obviously just a wrapper around a grep with preset options. I call it like so: grc "customer1". The ls invocation is a hint to the age of the dump.

This could be achieved in other ways, too. The point is that it is very handy to type and always accessible from the command line (where I spent my day). This is not specific to grep but rather a common pattern: A couple of commands belong together? A couple of command line arguments are always used in conjunction? Make a shell function or alias and automatize tedious work away.

How To Link A C Library In Haskell

Wed, 01 Nov 2017 00:00:00 +0100

In this blog post, I am going to show you how to link a C library in Haskell. Each step is kept as simple as possible in order to focus on the overall goal.

I will:

develop a small C program
generate two static C libraries from the C program
merge both libraries into one
develop a small Haskell program
let GHC link to the merged C library
use the functions exported by the C library within Haskell

The code snippets can be found here.

The C program

The C program is going to be extremely simple. The main.c is calling a library function and printing the result:

// main.c

#include <stdio.h>
#include "liba.h"

int main(int argc, char *argv[]) {
	printf("main.c: main()\n");
	double a = liba_func(1.0); // 42
	printf("main.c: a = %f\n", a);
}

The library function liba_func() is provided by, who guessed it, liba.c. The latter refers to libb_func() in order to carry out its work.

// liba.c

#include <stdio.h>
#include "libb.h"

double liba_func(double i) {
	printf("liba.c:liba_func() side effects!\n");
	return i + 1.0 + libb_func(); // return i + 1 + 40
}

// libb.c

#include <math.h>

double libb_func() {
	return fmod(5, 2) + 39.0; // return 40
}

The corresponding (educational) Makefile:

# gcc -Wall:    Enable all warnings
# gcc -c:       Do not link
# gcc -g:       Add debug symbols
# gcc -lm:      Link to libm(ath)
# gcc -o:       Output file name
#
# Makefile $@:  The file name of the target of the rule
# Makefile $^:  The names of all the prerequisites

main: main.c liba.o libb.o
        gcc -g -Wall -o $@ $^ -lm

liba.o: liba.c
        gcc -g -Wall -c -o $@ $^

libb.o: libb.c
        gcc -g -Wall -c -o $@ $^

clean:
        rm -rf *.o main

The static libraries

We can convert lib*.o into static libraries using ar. This tool creates a so called archive file which is basically just a collection of object files. In our case, both archives each contain only one object file. For completeness, we’ll let gcc link again the static libraries instead of the object files for main, too.

 #
 # Makefile $@:  The file name of the target of the rule
 # Makefile $^:  The names of all the prerequisites
+#
+# ar -c:        Create the archive
+# ar -r:        Update existing files in an archive
+# ar -s:        Add or update an archive index

-main: main.c liba.o libb.o
+main: main.c liba.a libb.a
        gcc -g -Wall -o $@ $^ -lm

...

+liba.a: liba.o
+       ar -csr $@ $^
+
+libb.a: libb.o
+       ar -csr $@ $^

One can inspect the archive content with nm <file.a>. We see that libb.a defines the function (“symbol”) libb_func and expects the symbol fmod. The latter comes from math.h and the actual function is linked by gcc (see -lm in the Makefile’s main target).

$ nm libb.a

libb.o:
         U fmod
00000000 T libb_func

Merging the libraries

The obvious way to merge both static libraries is to create a single archive from both object files (i.e. ar -csr libab.a liba.o libb.o). However, I am going to assume the libraries come as is and the object files are not available. This allows me to show a trick with ar. The issue with ar is that you cannot simply create an archive by listing archive files instead of object files on the command line. ar will happily archive the input files without looking into them and thus create a nested archive. The inner archives are not accessible anymore:

$ ar -csr libab.a liba.a libb.a # create a new archive from the previous ones
$ nm libab.a
nm: liba.a: File format not recognized
nm: libb.a: File format not recognized

We can however ask ar to add the contents of one archive to the current one using the ADDLIB directive from GNU ar’s “librarien” compatibility mode. In this mode, ar is controlled with a script which we can also simply pipe from the command line.

$ echo 'CREATE libab.a\nADDLIB liba.a\nADDLIB libb.a\nSAVE\nEND' | ar -M
$ nm libab.a

libb.o:
         U fmod
00000000 T libb_func

liba.o:
00000000 T liba_func
         U libb_func
         U puts

This time, inspecting the archive yields the exported symbols (functions) as expected. The necessary Makefile changes:

 libb.a: libb.o
        ar -csr $@ $^

+libab.a: liba.a libb.a
+       echo 'CREATE $@\nADDLIB liba.a\nADDLIB libb.a\nSAVE\nEND' | ar -M
+

The Haskell program

The Haskell program is going to be straight forward. I’ll let stack create the a simply project for me

$ stack new haskell simple

The initial Main.hs:

module Main where

main :: IO ()
main = do
  putStrLn "hello world"

Linking to the C library

stack generates a Cabal file within the project. This file is kind of like the Makefile of a Haskell app. We can define GHC options within that file. In order to tell GHC to link to an external library, we have to give a library path and a library name (pretty similar to GCC). This is done by -L<path> and -l<libname> respectively. So we could simply use stack’s ghc-options parameter to add these. However, it is recommended to use extra-lib-dirs and extra-libaries instead. This is how we need to adapt the cabal file:

 executable haskell
   hs-source-dirs:      src
+--  ghc-options:         -L../c -lab -- don't do this
+  extra-lib-dirs:      ../c
+  extra-libraries:     ab
   main-is:             Main.hs
   default-language:    Haskell2010
   build-depends:       base >= 4.7 && < 5

Of course, ../c is the path to the aforementioned C program which contains libab.a. The file name’s prefix is striped so that libab becomes ab only.

Using the C library

We are going to make a wrapper Haskell library (file) that exposes the functions of the C library. Calling the actual C functions is done trough Haskell’s Foreign Function Interfarce. The source code is simple and listed below. You can google around to find lots of information about suing the FFI in more detail.

{-# LANGUAGE ForeignFunctionInterface #-} -- enabling the FFI

module LibAb where -- declaring the module

foreign import ccall "liba_func" c_liba_func :: Double -> Double -- importing the C function

liba_func :: Double -> Double
liba_func = c_liba_func -- wrapping the C function inside a Haskell function

Now that we have the Haskell function liba_func at our disposal, we can go ahead and call in Main.hs:

 module Main where

+import LibAb
+
 main :: IO ()
 main = do
-  putStrLn "hello world"
+  putStrLn "Please enter a number:"
+  d <- fmap (read::String->Double) getLine
+  putStrLn $ "The answer is: " ++ show (liba_func d)

This is the outcome:

$ stack build --exec haskell
... 
Please enter a number:
1.1
liba.c:liba_func() side effects!
The answer is: 42.1

Note how the pure Haskell function liba_func :: Double -> Double produces side effects! This is quite expected. The Haskell compiler cannot look into the compiled C functions and cannot differentiate pure from impure code. It merely links the foreign functions where asked to. The programmer needs to provide this information. In this case, it should have been liba_func :: Double -> IO Double to accommodate for the printf in the C function.

 module LibAb where -- declaring the module

-foreign import ccall "liba_func" c_liba_func :: Double -> Double -- importing the C function
+foreign import ccall "liba_func" c_liba_func :: Double -> IO Double -- importing the C function

-liba_func :: Double -> Double
+liba_func :: Double -> IO Double
 liba_func = c_liba_func -- wrapping the C function inside a Haskell function

 main = do
   putStrLn "Please enter a number:"
   d <- fmap (read::String->Double) getLine
-  putStrLn $ "The answer is: " ++ show (liba_func d)
+  a <- liba_func d
+  putStrLn $ "The answer is: " ++ show a

Done! You just learned how to combined C & Haskell! Well, at least some part of it.

You can retrace all steps by looking into the commit history of this Git repo.

Simple MySQL Data Migration

Mon, 24 Jul 2017 00:00:00 +0200

I find myself more often than I think needing to shuffle around MySQL data. Usually this is about dumping one table to another database. For example, I sometimes want to get some production data into a test or development database to reproduce an issue. On my local box I often need to recreate an existing database structure for development purposes.

The task happened often enough that I took the time to write a little helper script for that. I wanted to have it as general as possible so I came up with the following command line interface.

$ migrate.pl --from srchost:srcuser:srcpass:srcdb \
    --to dsthost:dstuser:dstpass:dstdb \
    --dump-table table1 --dump-table table2 \
    --rename-table tableold:tablenew

The --from and --to arguments are mandatory and define the source and destination database. Since the databases could be everywhere (read: on different hosts) one specifies the database’s host and credentials, too.

The actual table(s) to be copied are set via --dump-table which can provided multiple times. Likewise, --rename-table will rename a table in the target database. The script does not dump & rename within one step.

Using a script with a nice command line interface has several advantages.

It’s usually faster than any GUI MySQL client.
Rerunning the same command or a slightly different (dumper another table) is simple and efficient.
Less error-prone compared to manually typing the MySQL commands and/or clicking around in the GUI.

Here is the script’s code:

#!/usr/bin/perl
#
# Shuffle some MySQL data around.

use strict;
use warnings;
use Getopt::Long;

sub main {
    my %opts = ();
    GetOptions(
        \%opts,
        "from=s",
        "to=s",
        "dump_tables|dump-table=s@",
        "rename_tables|rename-table=s@",
        "no_data|no-data",
    ) or die("Error in command line arguments!");
    my $from = $opts{from};
    my $to = $opts{to};
    if (!defined($from) || !defined($to)) {
        usage(1);
    }

    # Getting the access credentials:
    my ($from_host, $from_user, $from_pass, $from_db) = split(/:/, $opts{from});
    my ($to_host, $to_user, $to_pass, $to_db) = split(/:/, $opts{to});

    # Do some sanity checking:
    print "Data from: $from_user\@$from_host.$from_db\n";
    print "Data to  : $to_user\@$to_host.$to_db\n";
    print "Continue? [y/N] ";
    my $input = <STDIN>;
    chomp($input);
    if ($input ne "y" && $input ne "Y") {
        exit 0;
    }

    # Export using the least intrusive database access:
    my $mysql_cmd_export = "mysqldump --single-transaction --lock-tables=false --skip-add-locks";
    $mysql_cmd_export .= " -h $from_host -u $from_user -p$from_pass";
    if ($opts{no_data}) { # Only copy table structure
        $mysql_cmd_export .= " --no-data";
    }

    # The Import is a simply mysql client reading from stdin:
    my $mysql_cmd_import = "mysql";
    $mysql_cmd_import .= " -h $to_host -u $to_user -p$to_pass";

    # Dump all specified tables:
    for my $table (@{$opts{dump_tables}}) {
        my $cmd = "$mysql_cmd_export $from_db $table | $mysql_cmd_import $to_db";
        print "Dumping $from_db.$table to $to_db.$table ...\n";
        system($cmd);
    }

    # Rename all specified tables:
    for my $oldcolonnew (@{$opts{rename_tables}}) {
        my ($old, $new) = split(/:/, $oldcolonnew);
        my $cmd = "$mysql_cmd_import $to_db -e \"RENAME TABLE $old TO $new;\"";
        print "Renaming $to_db.$old to $to_db.$new ...\n";
        system($cmd);
    }

    exit(0);
}

sub usage {
    my ($ret) = @_;
    print "Mandatory:\n";
    print "$0 --from 'host:user:pass:db' --to 'host:user:pass:db'\n";
    print "\n";
    print "Optionial:\n";
    print "This will completely dump the named table from \$from to \$to.\n";
    print "--dump-table <tablename>\n";
    print "\n";
    print "This will rename named table from \$old to \$new in \$to.\n";
    print "--rename-table <oldname>:<newname>\n";
    exit($ret);
}

main();

(Download)

The code is pretty forward I believe. I’d like to highlight some points, though.

Using the password on the command line is insecure. Anyone with read access to the process list (or your history) can read the password. This script is for development purposes and I believe it is thus okay to leave it like so.

There is a --no-data switch which allows to simply copy a table’s structure.

I am using mysqldump’s –single-transaction, –lock-tables=false & –skip-add-locks switches to make the dumping process faster. This will make sure the database is not blocked (for other applications) while the dump is going on (pretty important for a shared databases). The downside is that the dumped data may not be consistent (when multiple tables are dumped) but I never experienced any problems with that.

Before actually doing anything, the script prints the source & target databases and asks the user to confirm. This is the last resort before overwriting any existing data. It’s kind of funny: The less steps you have to do manually, the faster you can do stupid things…

Test Deployments With Git Tags

Thu, 13 Jul 2017 00:00:00 +0200

Deploying an app to production typically happens using some tool like Jenkins. The deployment job usually takes the Git tag as input parameter. The parameter is used to checkout the expected version of the app, build & deploy it.

Now, the tag to be used must obviously be assigned in advance. The developers probably follow a specific development/release procedure (Git Flow anyone?) and tag releases when they are “done”.

I sometimes want to deploy a version of the software which is not ready yet. A typical scenario is that I cannot reproduce a bug in my local development environment but can see it in production or staging. In this case, I’d like to deploy (to staging!) a version of the app with some debugging statements included.

The deployment tool wants a tag for deployment though. I surely just need to tag this debug version and deploy it! I tend to forget how to do this and I also want the process to be as smooth as possible.

Here’s what I defined in my shell:

function tag_debug_version () {
    # Tag the current HEAD and publish it.
    TAG=test_tag_rolf
    git tag -d $TAG # delete previous tag
    git tag $TAG # make a new tag on HEAD
    git fetch --tags origin # update other people's tags
    git push -f origin $TAG # publish
}

Plain and stupid but it does the job. So the workflow for the described scenario is:

$ git checkout -b tmp-debug master # create the debug branch

# add some tracing and commit
$ git commit -m "Add debug output"

$ tag_debug_version # tag & push the debug version

# deploy version "test_tag_rolf"

Using the same tag has the advantage of not polluting the repo with pretty useless debug versions. Furthermore, the browser’s form cache will remember it so you don’t have to type it out again during deployment (assuming deployment happens via a web interface).

Comparing Directories Effectively

Tue, 23 May 2017 00:00:00 +0200

I often want to compare directories. I need to know which files only exist in one of them and which files differ. If any files differ, I sometimes want to see the actual change so I need an efficient way to do so.

For starters, I am using this drq alias to diff directories:

$ ls -1 A B # show both folder's content with ls
A:
a
aonly
b
contentmatters

B:
a
b
bonly
contentmatters

$ drq A B # diff both folders
Only in A: aonly
Only in B: bonly
Files A/contentmatters and B/contentmatters differ

The alias drq expands to diff -r -q which executes a recursive (-r) diff that only lists changes (-q, i.e. quiet).

In order to inspect file changes, I use vimdiff. By default, one could simply run vimdiff file1 file2. However, when I don’t know in advance whether the files do have differences I don’t want to actually fire up Vim unless they do. Thus, my ~/.functions contains the following definition:

vd () {
    diff $@ > /dev/null
    if [[ $? -eq 1 ]]
    then
        vimdiff -c 'windo set syntax=off' $@
    fi
}

This simply first diffs the files in question and opens Vim only if diff finds any differences . Running vd file1 file1 will just return without starting vim. This approach causes the two files to be diff’ed twice but I rarely experience any waiting times. (I turn off the syntax in diff view to not get distracted with windo set syntax=off.)

Finally, I have another seemingly strange function defined:

Files () {
    vd $1 $3
}

Files is now a function which runs my previously mentioned vd on its first and third argument. Go back to drq and guess ;-).

Found it? Well, once the recursive diff told me that two files differ and I want to compare them, I can simply copy the whole line with the two files, paste it and execute it as a new command. The Files function will forward its “arguments” to vimdiff as needed.

Using Tmux in visual mode, this looks like this:

After executing the drq A B, I have to type <prefix>[k<Space>0<Enter><prefix>] (prefix is the Tmux prefix, of course):

         Move     Jump to
         up       Line Start  Paste
         |        |           |
<prefix>[k><Space>0<Enter><prefix>]
    |        |       |
Open Tmux    Start   Copy
Visual Mode  Selection

This is much faster than you would expect. Hitting Enter at this point will open up Vim and show the changes. Instead of pressing k, you can also jump higher with 3k for example.

You might wonder why Files uses my custom vd instead of plain vimdiff. The only reason for that is that sometimes the files in questions have changed after the recursive diff’s execution and this makes sure Vim is not started for no reason.

PEP8 Standard Deviation

Fri, 21 Apr 2017 00:00:00 +0200

Official coding standards ease reading of other people’s code and (usually) kill useless debates. When I code in Python, I follow PEP8. Being a VIM user, Syntastic & flake8 are my tools for the job.

There is one thing in the PEP8 coding style which I don’t comply with: line lengths. The standard says: “Limit all lines to a maximum of 79 characters.” This is really to hard to follow. A typical situation which is easy to imagine is:

class MyClass:
    def my_function():
        if cond1:
            ...
            if cond2:
                ...
                call_this_func_with_long_name(argument1, argument2, argument3, argument4)

This looks made up but happens very often in practice. The standard specifies multiple ways to deal with such a scenario. Luckily it also says “For […] a team that can reach agreement on this issue, it is okay to increase the […] maximum length to 99 characters”. 25% more helps a lot!

However, there is one situation where it is really awkward to follow the style guide: Using the str.format() method with a lot of arguments within a “long” string. Consider:

# not okay according to PEP8
log.warn('The {} of {} failed. Please check {} or {}'.format(method_name, step_name, logfile_name, db_entry)

# fine but ugly
log.warn('The {} of {} failed. Please check {} or {}'.
         format(method_name, step_name, logfile_name, db_entry)

The second version gets nasty once the wrapped format itself exceeds the 79 (or 99) characters, especially when using keywords within the replacement field, i.e. 'The answer is {answer}'.format(answer=42). Furthermore, diff‘ing commits where only some string content changed may yield multiple changed lines instead of a single line change (which it is semantically). I have to say that I also spend (lost?) quite some time in the past making the format invocation comply with the standard. For theses reasons, I decided to deviate from the PEP8 maximum line length when formatting strings. Obviously, I want to have Vim/Syntastic/flake8 help me with this …

The code linter flake8 can be configured per project or globally. The latter is done in the file ~/.config/flake8. In order to increase the maximum line length, you simply add the following to the file:

[flake8]
max-line-length = 99

Once this option is set, flake8 (invoked within Vim by Syntastic) will not grump anymore for lines lengths up to 99 characters. I have found flake8-putty to be an easy way to disable the annoying E501: line too long error specifically for string formatting. The following two regexps tell flake8-putty to ignore E501 for every line which contains the literal .format( or every line ending with a single quote.

[flake8]
max-line-length = 99
putty-ignore =
    /\.format\(/ : E501
    /'$/ : E501

Today’s screens are wide enough to view two Python files side by side even if they both have line lengths of 99 characters so I believe it is okay to slightly exceed PEP8’s 79 characters. All very long lines “only” contain strings and it is usually not the stuff someone is interested in when reading code.

Caveats: It may cause problems when the editor autowraps long lines but I don’t do that. Any CI system (or even Git hooks) enforcing PEP8 needs to be configured properly, too.