git latexdiff usage: visually highlight changes in version controlled Latex files

October 2, 2017 Leave a comment

This is a follow-up post to latexdiff-git, which is outdated by now.

latexdiff is a powerful tool that uses two Latex files to generate a third Latex file in which differences between the first two files are visually highlighted. latexdiff is especially useful when comparing an old with a new version of the same Latex file. However, latexdiff does not account for any version control on its own. This means that if you want to visually highlight the differences between two versions of a version controlled Latex file, you are required to manually checkout the two different versions ahead of comparing them with latexdiff. In case you are using git as your version control system, this is where git-latexdiff comes into play. It accounts for checking out the different version of a Latex file as well as comparing them with latexdiff in a single command: you only need to specify which latex file and versions should be used for the comparison.

Preconditions

Ensure you have git, a latex distribution (e.g. texlive-full), and latexdiff installed and available in the path of your CLI.

git-latexdiff installation

Clone git-latexdiff:

git clone https://gitlab.com/git-latexdiff/git-latexdiff#

Follow instructions in the README. This will involve a:

sudo make install

in most cases on an Ubuntu system. If all went well “git latexdiff” should be available in your CLI (it should print the help message):

user@machine:~$ git latexdiff

fatal: Please, provide at least one revision to diff with.
Usage: git latexdiff [options] OLD [NEW]
 git latexdiff [options] OLD --
 git latexdiff [options] -- OLD
Call latexdiff on two Git revisions of a file.

OLD and NEW are Git revision identifiers. NEW defaults to HEAD.
If "--" is used for NEW, then diff against the working directory.

Options:
 --help this help message
 --help-examples show examples of usage
 --main <file> name of the main LaTeX, R Sweave,
 or Emacs Org mode file.
 The search for the only file containing 'documentclass'
 will be attempted, if not specified.
 For non-LaTeX files, a reasonable `prepare` command
 will be used unless explicitly provided
 --no-view don't display the resulting PDF file
 --latex run latex instead of pdflatex
 --bibtex, -b run bibtex as well as latex
 (pdflatex,bibtex,pdflatex,pdflatex)
 --biber run BibLaTex-Biber as well as latex
 (pdflatex,bibtex,pdflatex,pdflatex)
 --view view the resulting PDF file
 (default if -o is not used)
 --pdf-viewer <cmd> use <cmd> to view the PDF file (default: $PDFVIEWER)
 --no-cleanup don't cleanup temp dir after running
 --no-flatten don't call latexpand to flatten the document
 --cleanup MODE Cleanup temporary files according to MODE:

- keeppdf (default): keep only the
 generated PDF file

- none: keep all temporary files
 (may eat your diskspace)

- all: erase all generated files.
 Problematic with --view when the
 viewer is e.g. evince, and doesn't
 like when the file being viewed is
 deleted.

--latexmk use latexmk
 --latexopt pass additional options to latex (e.g. -shell-escape)
 -o <file>, --output <file>
 copy resulting PDF into <file> (usually ending with .pdf)
 Implies "--cleanup all"
 --tmpdirprefix where temporary directory will be created (default: /tmp).
 Relative path will use repository root as a base
 --verbose, -v give more verbose output
 --quiet redirect output from subprocesses to log files
 --prepare <cmd> run <cmd> before latexdiff (e.g. run make to generate
 included files)
 --ln-untracked symlink uncommited files from the working directory
 --version show git-latexdiff version.
 --subtree checkout the tree at and below the main file
 (enabled by default, disable with --whole-tree)
 --whole-tree checkout the whole tree (contrast with --subtree)
 --ignore-latex-errors keep on going even if latex gives errors, so long as
 a PDF file is produced
 --ignore-makefile ignore the Makefile, build as though it doesn't exist
 -* other options are passed directly to latexdiff
 --bbl shortcut to flatten a bbl file of the same name as the project
 --latexpand OPT pass option OPT to latexpand. Use multiple times like
 --latexpand OPT1 --latexpand OPT2 to pass multiple options.
 --latexdiff-flatten use --flatten from latexdiff instead of latexpand

Unrecognized options are passed unmodified to latexdiff.

git-latexdiff usage

The main CLI usage of git-latexdiff is:

git latexdiff --main YOURFILE OLD_HASH [NEW_HASH]

A bunch of useful options are available for git-latexdiff. For example:

-v # good in case you run into any errors
--cleanup keeppdf # delete the checked-out 'old' and 'new' folders but keep the pdf
--cleanup all # delete all generated files afterwards
--output FILE # copy pdf before deleting it to FILE (disables automatically viewing the pdf). good in combination with --cleanup all
--tmpdirprefix ./FOLDERNAME/ # alternative to the above: specify where temporary stuff is stored, makes it easier to access diff files. good in combination with --cleanup keeppdf
--bibtex # also run bibtex
--biber # also run biber

Therefore, the command you might want to run therefore might be similar to this:

git latexdiff --main MYFILE.tex --bibtex --output git-latexdiff.pdf --cleanup all OLD_HASH NEW_HASH

…where you can specify

--

as NEW_HASH, if you want to run git latexdiff against the current (possibly unstaged/uncommited) version of the file, or where you can leave out NEW_HASH completely, if you want to run it against the last commit. Or your command might look similar to this:

mkdir git-latexdiff; git latexdiff --main MYFILE.tex --tmpdirprefix ./git-latexdiff/ --cleanup keeppdf OLD_HASH

 

Savitzky-Golay Filters: Approximating Time Series using Polygons with an Example in R

August 6, 2017 Leave a comment

Continuous data streams (“time series data”) are usually smoothed before data processing is applied on them. For this purpose both the running mean filter (also called moving/rolling mean/average) and the related running median filter are frequently used. Both have the disadvantage of “cutting off” peaks. This is a side effect of not trying to approximate a given signal in the best way. That is, they apply a simple function without incorporating the error they introduce on the signal during the approximation.

As alternative to such approaches a Savitzky-Golay filter can be used. It tries to approximate a given signal using a sliding window approach and a low degree polynomial to model data within that window. In contrast to running mean/median it also incorporates the introduced error in the approximation process using linear least squares. This leads to not “simply cutting off peaks” but modeling them in the best way possible, just as the rest of the data.

Here’s a simple example of a Savizky-Golay filter in comparison to running mean/median in R on an excerpt of the beaver data:

library(signal)
matplot(data.frame( beaver1[,3], # original data
runmed(beaver1[,3], k = 11), # with running median filter
filter(filt = sgolay(p = 5, n = 11), x = beaver1[,3]) # with SG filter
), type='l', lwd=2, lty=1, ylab='')
legend('topleft', legend=c('original', 'runmed', 'Savitzky–Golay'), col=1:3, lty=1, lwd=2)

 

Savitzky-Golay Filter Example

Savitzky-Golay Filter Example

Install R from source: a simple way to compile R from source and use it in RStudio or on servers without X

Sometimes we need to use special versions of R (e.g. the latest version of R is not available in the repositories of the machine we need to use). This post describes a simple way of compiling R from sources and using it in RStudio or on servers without X.

Get latest R source

Get the source of your desired R version from https://www.r-project.org/. You want to obtain a file named R.x.y.z.tar from there and untar it using tar -xf R-x.y.z.tar.gz. Change into the untar-ed folder then.

More details on this are available at https://cran.r-project.org/doc/manuals/r-release/R-admin.html#Getting-and-unpacking-the-sources

Build R from source for RStudio

Built R using the following commands. Be aware that there are a number of prerequisites to this compilation, like having gcc and g++ installed.

./configure --enable-R-shlib
make
make check
make check-all

 

Passing the checks creates the R and Rscript frontends in ./bin/ — those are what you most likely need. You can call these directly, link to them, or use them in RStudio (see below). If you forgot to add the –enable-R-shlib parameter to configure you need to delete and re-extract the tar before redoing the process — otherwise make will fail.

More details on this are available at https://cran.r-project.org/doc/manuals/r-release/R-admin.html#Simple-compilation

Define the R to use with RStudio

RStudio uses the R defined by which R. Hence we can add the following to the ~/.profile file to define which R is used by RStudio:

# export RSTUDIO_WHICH_R=/path/to/your/R # this is what RStudio usually uses
export RSTUDIO_WHICH_R=/usr/local/bin/R  # this is what we use
. ~/.profile # reloads the profile without requiring us to logout+login again

More details on this are available at https://support.rstudio.com/hc/en-us/articles/200486138-Using-Different-Versions-of-R

Build R for environments without X/without Java

Do the same steps as above, but use this configure instead:

./configure --with-x=no --disable-java

More details on this are available at https://cran.r-project.org/doc/manuals/r-release/R-admin.html#Essential-programs-and-libraries

Hint for “package not available for R version x.z.y…” errors

If package installations fail because your repositories don’t contain the required SW, try the Berkeley mirror: from our experience, they host lots of packages in many versions. For example:

install.packages("ggplot2", repos="http://cran.cnr.berkeley.edu")

Alternatively, the URL to the source archive can be specified directly:

packageurl <- "http://cran.r-project.org/src/contrib/Archive/XXXX/XXXX_A.B.C.tar.gz"
install.packages(packageurl, contriburl=NULL, type="source")

More details on this are available at http://stackoverflow.com/questions/17998543/what-to-do-when-a-package-is-not-available-for-our-r-version

Yet another option is to use devtools to fetch code e.g. directly from GitHub:

devtools::install_github("hadley/lineprof")

More details on this example can be found at http://adv-r.had.co.nz/Profiling.html

 

Categories: Data Analysis Tags: , , , , , , ,

Rhythmbox is silent: reset pulseaudio configuration

Rhythmbox is silent?

After playing around with different Linux Shells/UIs/window managers (including i3, Gnome Shell, etc) and some frequent restarts, I noticed that for some reason my music player Rhythmbox had stopped playing any sounds. Though it was still “playing” songs, no sounds were actually made by speakers, headphones, etc. In contrast, all other sounds were OK, including OS sounds like info/error messages, or other players like VLC. Changing sound setting, volumes, as well as changing audio devices did not have any effect on this.

Resetting pulseaudio configuration

Turns out that the pulseaudio configuration was messed up. The solution to this problem is to move/delete the old configuration, then let it be recreated (the original problem description and solution to this is here):

mv ~/.config/pulse/ ~/.config/pulse_old/ # move/delete the old config
pulseaudio -k # restart pulseaudio

After doing this and restarting Rhythmbox its sound should be back to normal, – though you might need to restart the OS if you run into problems with external speakers/headphones.

i3 window manager: quick setup and configuration

February 5, 2017 Leave a comment

i3_window_manager_screenshot

i3 example [Wikipedia]

The i3 window manager is a tiling window manager: it enables power users to operate/arrange/manage application windows in a very fast way in tiles, without any requirements to use a mouse. Alex Booker provides an excellent introduction and overview to the i3 window manager, which is divided into 3 videos: general introduction, configuration, and (visual) fine tuning. In this post we summarize the i3 configuration we use atm for a quick lookup (which is partially based on Alex Booker’s videos).

Installation


sudo apt-get install i3

Logout, then login to i3. The i3 configuration wizard starts automatically if there is no i3 configuration file yet (we recommend using Super as the mod key at this point, because Alt is used by a huge amount of applications for other tasks). You can always remove the i3 config file at ~/.config/i3/config and start the wizard again using i3-config-wizard.

Basic usage

We strongly recommend to go through all the shortcuts in the official i3 reference card (it just takes a moment and is the core of how you will operate i3 afterwards). Be aware that if you are using an alternative keyboard layout, i3 automatically detects which keys are located at the positions that i3 would normally use – so you will be able to use the same physical keys, even if you are using a completely different layout.

After being aware of the basic usage of i3, skimming through the official i3 user’s guide will make sense. This should highlight the possibilities with i3 (in both how to operate i3 and how to configure it), which will make later configuration tasks easier by far.

i3 config file

The main i3 config is located in ~/.config/i3/conf. This file is created by i3-config-wizard at first and can be modified as wished afterwards.

Be aware that changing any i3 config at least requires i3 to be reloaded (“restarted in place”) with Mod+Shift+R. Some configuration also require the underlying application that is started/controlled by i3 (e.g. compton) to be restarted. If i3 restarts those applications on each reload depends on its configuration (that’s the difference between exec and exec_always in the config file). If the underlying application was started with exec, restarting the application by hand or logging out/loging back in is required.

(Re)starting i3 from command line

To start i3 from a native tty command line:

DISPLAY=:0 # or :1 etc...
export DISPLAY
# killall i3 # required in case i3 is running
i3 -c .config/i3/config

To instead restart a running i3 from a native tty command line:

i3-msg reload
i3-msg restart

Media buttons etc.

Most likely i3 does not recognize your media buttons correctly from the beginning. You at first need playerctl to enable control over (parts) of the media buttons, of which you can get different packages (including .deb) here. The i3 FAQ then provides a config snippet that can be copied (appended) to your i3 config file to map keys to the media buttons. Be aware that the number of the pulse audio device may be different (in the script it’s 0, but for others it might e.g. be 1). If e.g. the buttons cause the currently focused window to flicker, but the volume does not change, this might be an indication for using the wrong device. If you want to specify the player that you control with the play/pause/next/previous buttons via playerctl buttons, you use the –player parameter, e.g. –player=rhythmbox. Be sure that the player provides an MPRIS D-Bus interface (e.g. vlc, audacious, spofity, etc). With Rhythmbox, for example, the corresponding plugin has to be enabled first.

Set wallpaper

You need e.g. feh to set a wallpaper in i3:


sudo apt-get install feh

By adding the following to your i3 config file you always set the wallpaper when i3 is started/reloaded:


exec_always feh --bg-scale /path/to/your/wallpaper.jpg

For different wallpaper scaling options, check out the man page of feh (it provides e.g. –bg-center, –bg-fill, –bg-max, –bg-scale, –bg-tile, …)

Launch applications on specified workspaces

At first, you need to find out the xprop window class of your target application. For this, start xprop and click on the application window. The 2nd string under WM_CLASS(STRING) is the string you need to know in the next step: add the following line to your i3 config file (and replace CLASS with the just determined string):


assign [class="CLASS"] 10

10 in this case indicates the nr. of the workspace the application window should be positioned.

Autostart applications

To automatically start applications, add one of the following lines to your i3 config file:


exec APPLICATION_COMMAND


exec_always APPLICATION_COMMAND

exec only executes the command when i3 is started, and exec_always executes the command each time i3 is reloaded. For example, starting your web browser is only useful for each fresh i3 start, but setting the wallpaper is useful to be done each time i3 is reloaded.

Monitor configuration

An easy way to configure the monitor setup using an UI is with arandr (UI for randr):


sudo apt-get install arandr

If you want to make a certain monitor configuration permanent you can a) configure monitors as wished in arandr, then b) save the configuration to a file, and c) add “exec_always CONFIG_FILE” to your i3 config file.

Wifi controls

i3 does not come with a built in UI for to control your wifi. To avoid configuring it via config files (which is still pretty cumbersome in Linux after all those years) you can use e.g. the graphical frontend wicd-gtk:


sudo apt-get install wicd-gtk
wicd-gtk

Change system fonts

One possible cosmetic improvement is to use different system fonts in i3. Alex recommends the Yosemite San Francisco Font: you can download it from its github page (under “manual install”), then change the font entry in the i3 config file to:


font pango:System San Francisco Display 10

Further, changing the font for the used gtk is recommended. You can do this is the gtk configuration file (in ~/gtkrc-2.0 and/or .config/gtk-3.0/…) by changing the font line to e.g.:


gtk-font-name="System San Franzisco 10"

Another option is to use the lxappearance config UI (but it currently seems buggy as not detecting all installed fonts correctly, as long they are not already in use with the system):


sudo apt-get install lxappearance
lxappearance

De-uglify i3

When you previously had Gnome Shell installed, you can use gnome-tweak-tool to configure your look-and-feel and add the following line to your i3 config file to “de-uglify” i3 at startup:

exec --no-startup-id gnome-settings-daemon

For further/other configuration or if the above approach does not work for you you can try lxappearance, gtk-chtheme, and qt4-config:

sudo apt-get install lxappearance gtk-chtheme qt4-qtconfig

Replace dmenu with rofi


sudo apt-get install rofi

In the i3 config file, replace the dmenu-shortcut with e.g.


bindsym $mod+a exec rofi -show run -lines 20 -eh 1.3 -opacity "85" -bw 0 -font "System San Francisco Display 10"

or, to have different colors:


set $bg-color&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; #2f343f
set $text-color #f3f4f5
bindsym $mod+a exec rofi -show run -lines 20 -eh 1.3 -opacity "90" -bw 0 -bc "$bg-color" -bg "$bg-color" -fg "$text-color" -hlbg "$text-color" -hlfg "$bg-color" -font "System San Francisco Display 10"

Transparency and fading effects

i3 does not provide for transparency or fading effects itself (it has less purpose in doing so, as e.g. windows don’t overlap in a tiling arrangement). To add those, use e.g. compton:


sudo apt-get install compton

and add “exec compton” to your i3 config file. The configuration is for compton is done via ~/.config/compton.conf, which you first have to copy from the example file location. With Ubuntu, this is:


cp /usr/share/doc/compton/examples/compton.sample.conf ~/.config/compton.conf

Afterwards, you can change settings there to your needs. We usually do the following:

  • Increase window fading speed, which is specified in milliseconds between fading steps: “”fade-delta = 5;”
  • Remove transparency from unfocused windows and menues (increases their readability). To do so, comment out “inactive-opacity” and “menu-opacity”.

Replace the i3status bar (bottom/top bar) with i3blocks


sudo apt-get install i3blocks

Copy the example i3blocks config file:


cp /etc/i3blocks.conf ~/.config/i3/i3blocks.conf

And change the value of the “status_command” from “i3status” with “i3blocks -c ~/.config/i3/i3blocks.conf” in your i3 config file.

Then you can customize i3blocks in your ~/.config/i3/i3blocks.conf file. For example:

  • Set the interval of date/time to 1 to have a 1s time update interval.
  • For volume add “command=/usr/share/i3blocks/volume 5 pulse” and set the interval 1 to use the pulse audio volume level with 1s interval.
  • Replace the text of labels with character/utf-8 icons as you like – this saves space in the bar. Alex has some nice examples for this in his own config file on github.

Update notification in i3blocks

The following script checks and prints if updates are available and if a reboot is required after updates:

#!/bin/bash                                                                                                        

# count how many updates we have got
ups=`/usr/lib/update-notifier/apt-check --human-readable | head -1 | awk '{print $1;}'`

# print the results
if [ "$ups" -eq "1" ]
then
  echo "There is 1 update"
elif [ "$ups" -gt "1" ]
then
  echo "There are $ups updates"
elif [ -f /var/run/reboot-required ]; then
     echo 'Reboot required'
else
  echo "Up to date"
fi

This script can be included in i3blocks by adding the following to in the i3blocks config file:

# Update status
[updates]
label=
interval=3600
command=/path/to/update/status/script/from/above

Workspaces on predefined screens

For users using a multi-monitor setup configuring to which screen specific workspace automatically go might be interesting:


# put workspaces on specific screens
workspace 1 output HDMI-1
workspace 2 output DVI-I-1
...

More i3 shortcuts

  • Auto back-and-forth: after switching from workspace A to workspace B, pressing the shortcut for workspace B again to switch back to workspace A. Add “workspace_auto_back_and_forth yes” to enable this feature.

Application configuration

Usability of some applications benefits from some more configuration:

  • The new mail attention add-on for Thunderbird makes i3 raise the urgent notification flag for new mails, so that the corresponding workspace is highlighted.
  • Enabling the “Message Notification” add-on in Pidgin makes i3 raise the urgent notification flag for new messages, so that the corresponding workspace is highlighted.

More hints

As i3 does not come with a built in network configuration UI: to configure networks like VPNs you can use the following (parts of the network-manager and network-manager-gnome packages):

  • nm-connection-editor allows editing network settings
  • nmtui allows for activating/deactivating networks over the command line

i3 does also not come with a built in UI for configuring computer mice. The mouse pointer speed can be adapted with xinput:

  • use “xinput –list –short” to determine the name of your mouse device, then
  • “xinput –set-prop “MOUSE DEVICE NAME” “Device Accel Constant Deceleration” 1.8″ to change the mouse pointer speed. The latter can be added to the i3 config file to automatically keep the speed after a restart of i3.

Backup your i3 configuration

Most (if not all) i3 configuration is located in those 3 files, which consequently are the only ones that need to be backed up:


~/.config/i3/config
~/.config/i3/i3blocks.conf
~/.config/compton.conf

Categories: Linux, Misc

How I learned to type the Neo2 keyboard layout: a review after the first 1.5 years

December 27, 2016 Leave a comment

neo-iconA couple of years back, I was asking myself: should I switch my keyboard layout to a more effective one? If yes, which new layout should I learn? For obvious reasons I searched for reports of people that did such a switch before. And this is why I try to give such a report myself now, after typing Neo2 for about 1.5 years – which seems to be a good point for summarizing my success story with Neo2 so far.

But first: some details about my typing characteristics before I learned typing the Neo2 layout. I…

  • …was “successfully” using the German QWERTZ keyboard layout for something like 12 years with a 10-finger-system.
  • …was typing with somewhere around 450 characters per minute for normal text.
    …did software development for a living and especially disliked the access to special characters my back-then keyboard layout.
  • …didn’t like the idea of using an ineffective keyboard layout in general. The layout I was using back then was a) not optimized for today’s hardware and b) I used it all day as it is the number 1 tool used in my profession, so investing in improvements should really pay off on the long run.

Overview of alternative keyboard layouts

There are (and were back in the day) many alternatives to the layout I used. For some examples, see e.g.:

I looked a bit into heuristically optimized keyboard layouts. Some of them use the same higher layers as Neo2 and just have layer 1 and 2 rearranged heuristically, such as the bone layout (German: http://wiki.neo-layout.org/wiki/Bone). Why didn’t I chose such a layout? Those had/still have less support, as they are not used so widely. This brings additional effort when configuring/using different PCs/hardware.

Additionally, for something so important for me and my daily business I preferred to use something already proven to work well, and would therefore bring less problems with it.

Why use Neo2 instead of other alternatives?

Neo was designed back in 2004 and was optimized for the German language. Due to German being very similar to English from key frequency point of view the “being optimized to German” did not make a big difference to me (and still does not make any, you can rest assured that it’s very comfortable to type English with it). Neo2 has a very good heatmap (optimized for key position and typing interleaving with left/right keys:

keyboard_heatmap_for_german_neo

Though there are arrangements that are a little more effective and further optimized (e.g. heuristically),  when it comes to key frequencies, having such a heatmap still is an amazing thing.

To me, even more important was that access to special characters is solved very cleverly with Neo2. There are 6 layers in total:

  • Layer 1+2 are lower and upper case characters – those are the ones you are probably used to on your keyboard. The “modifier” keys are the Shift keys.
  • Layer 3 are special characters (right below your fingers, e.g. in the home row!). The “modifier” keys are the CapsLock button on the left side and the “#” key on the right side.
  • Layer 4 are other special characters and special keys
  • Layer 5+6 are Greek letters.

The special characters on layer 3 make accessing special characters for programming very easy. And layer 4 is something that is easily underestimated: it features:

  • Keys to control cursor position like up/down/left/right, page up/down, delete left/right, home/end, escape, tab, enter, etc. – all under your left hand.
  • All numbers in the form of a keypad – all under your right hand.

From this point of view, Neo2 is more effective for programmers than e.g. Dvorak, Colemak, or similar keyboard layouts (still, those are pretty cool too). But besides the Neo2 layout having a lively community and already being proven to working well, this was the main reason for learning Neo2 instead of any other layout.

Learning and switching to Neo2

How not to do it

Maybe 3 years before I already had tried Neo2 – for very short time. Back then I set my goal similar to “I want to type in Neo!”, printed a Neo2 layout onto an A4 sheet of paper, switched my keyboard layout to Neo2 and just started typing it. In short: it did no work out. I had to search for every key using the printout, and of course got frustrated over the extremely low speed of typing (which was to be expected) very fast – and quit within hours.

How to do it

After the first try, the goal of switching keyboard layout remained. I wanted to use a more effective keyboard layout/Neo2 someday, as it would just be so much more effective for my everyday usage. The second time (maybe 2 years after the first try) I was better prepared to really perform a switch:

  1. I used KTouch with Neo2 to train single keys – one after another (the KTouch “beginner mode”). I did this is my free time, at work I continued to use my QWERTZ layout. Further, I made sure that I typed all keys with the correct fingers (I had had a slight mash-up with the QWERTZ layout before).
  2. In KTouch I used standard settings, which are minimum 180 chars per minute and 98% accuracy to pass each lesson and unlock the next one. This ensures you don’t try to learn  too much at once, which was the problem with my first try.
  3. Using KTouch I trained about 1/3 of the Neo2 lessons provided. Those were all about introducing and training new character keys. It was no problem to write with my QWERTZ layout at work at the time (switching was smooth).
  4. After finishing about 1/3 of the Neo2-lessons provided with KTouch (within 1.5 months as far as I remember), I did the “all in” thing: I switched all my layouts to Neo2 and typed all texts with it. This included my keyboard at work as well as e.g. my on-screen keyboard on my mobile phone. Of course it was very slow at the beginning again (first few days), but now I learned very fast. After about 2 to 3 weeks of typing, my speed was OK, while I was still learning at a fast rate without requiring any extra training time. This was when typing the old QWERTZ layout became more difficult for me.
  5. Since then I’ve just continued to write Neo2. Once in a while I did some more lessons in KTouch – but these were just for training and comparing typing speeds. But overall, using a typing trainer like KTouch definitely was the “enabler” of learning a new keyboard layout for me.

Progress report after the first 1.5 years

I started learning Neo2 around end of February 2015. Within about 1.5 months (after successfully completing about 1/3 of the lessons in KTouch) I made the “all in” move and switched all my layouts to Neo2. My progress since then:

  • Mid of April (~2 months): about 190 chars/min, 95.5% accuracy in the 1000 English words test from KTouch Neo2.
  • End of April: feeling more comfortable every day.
  • Mid of May (~3 months): about 250 chars/min, 95.5% accuracy in the German/English words test from KTouch Neo2.
  • Mid of June (~4 months): 286 char/min, 98.5% accuracy (09:02 in the “repeat many words test” in KTouch Neo2).
  • End of September (~7.5 months): 286 char/min, 96.3% accuracy (21:13 in the 1000 most frequently used English words test in KTouch Neo2) and 289 char/min, 97.2% accuracy (08:57 in the “repeat many words test” in KTouch Neo2).
  • Beginning of February (~1 year): 337 char/min, 96.8% accuracy (6:50 in the KTouch Neo2 train comma lesson).
  • Mid of February 2016 (~1 year): 330 char/min, 97.2% accuracy (18:20 in the 1000 most frequently used English words test in KTouch Neo2) and 311 char/min, 96.4% accuracy (8:20 in the KTouch Neo2 “repeat many words test”).
  • April 2016 ~1 year 2 months: 318 chars/min with 97.2 accuracy with the KTouch Neo2 congratulation text for finishing Neo2 training. Programming: 241char/min with 96.5% accuracy in KTouch Neo2 shell text (special characters, numbers, …). This pretty much is what I wanted in the beginning – it made me writing code on my keyboard more effectively. Additionally: have the feeling that I need to think less about how I do things, and more about what I want do to. Specially the position keys under left hand at level 4 (move cursor up/down/left/right, page up/down, delete left right, home/end) have proven extremely effective – much more than I initially thought – to quickly move the cursor around when programming.
  • July 2016 ~1 years 5 moths: 333 chars/min with 98.0% accuracy (18:09 in the 1000 most frequently used English words test in KTouch Neo2).

This is an example of how your progress for learning Neo2 in KTouch could look like:

  • Left: progress for one single lesson: usually, you will always increase in speed and accuracy when repeating the same lesson (which is why you are able to learn a new keyboard layout relatively fast if you take it seriously).
  • Right: progress for different/all lessons: usually, you increase in speed and accuracy for a specific lesson, and when you move on to the next lesson your speed and accuracy will be lower for it at the beginning – which is because you are just learning completely new things there. I can scroll far to the left in this view – and it’s a good feeling to see speed and accuracy continuously increasing over time there 😉

 

 

 

 

The final (and frequently asked) questions

Is my typing speed still increasing? Yes, definitely. Was it worth the effort? Would I do it again? Yes, definitely, to both. Though, at the moment I’m still a bit slower in typing regular texts than I was before, programming is definitely faster already. Typing feels less cumbersome: especially complex combinations of symbols/characters are easier to type, and I don’t need to move my right hand anymore for reaching e.g. arrow keys, delete/insert, or similar. Besides making a large number small tasks a little faster (which sums up to a lot in the end), this also just feels more satisfactory.

So, bottom line: in my opinion, especially when your job involves something like programming, learning Neo2 really pays off in the end.

Weka on Android: load precomputed model and predict new samples

November 27, 2016 Leave a comment

Weka on Android

Imagine you want to use machine learning on Android to predict the value for your target variable for new samples. Usually, neither the storage capabilities (for storing the training data) nor the computational power for training such models is available on mobile devices – assuming that you use a medium or large dataset and thorough evaluation of different model types and model parametrizations with a subsequent model selection. In such a scenario, the way to go therefore is:

  1. Train, evaluate, and select the optimal model for your machine learning prediction offline, e.g. using a desktop PC or server hardware.
  2. Export the ready trained model (“model file”, which usually tends to be relatively small) and ship it with your Android application.
  3. The “application case”/”production case” on the mobile device simply involves loading and using the model to predict new, yet unseen samples. Thereby, the prediction on the mobile device requires much less computational power and storage capabilities than the previous training of the model.

Based on the previous post about loading and using Weka models with JavaSE, this example demonstrates how to load and use a model Android which was previously trained with Weka “offline”, aka on a desktop PC. The core points you have to look out for when setting up the Android project:

  1. Use the exact same version of Weka you used for offline training the model also on Android.
  2. Copy the weka.jar to libs/, then select “Add as library…”.
  3. Ensure that “compile files(’libs/weka.jar’)” is present in build.gradle.
  4. Do a clean build: at the first time, this is going to require some CPU power and take some time.
  5. Don’t do anything with Weka that is UI related: Weka uses the JavaSE UI, which is not available on Android.
  6. Follow the details in the example below:
    1. Structure the new, yet unseen instance you want to predict the target variable so that Weka can work with it.
    2. Load the classifier object.
    3. Predict the target variable for the new instance.

The complete Android example project containing the code below can be downloaded here:

  • Download (7z): Android Weka Example.
  • SHA256sum: aaf8594954d3561218dc0dd621740f3184a171efa7016a2efe4b743a19845ffb
package at.fhooe.mcm.ml.androidwekaexample;

import android.content.res.AssetManager;
import android.os.Bundle;
import android.support.v7.app.AppCompatActivity;
import android.support.v7.widget.Toolbar;
import android.util.Log;
import android.view.View;
import android.view.Menu;
import android.view.MenuItem;
import android.widget.TextView;
import android.widget.Toast;

import java.io.IOException;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.Random;

import ml.mcm.fhooe.at.androidwekaexample.R;
import weka.classifiers.Classifier;
import weka.core.Attribute;
import weka.core.DenseInstance;
import weka.core.Instances;

public class MainActivity extends AppCompatActivity {

    private static final String WEKA_TEST = "WekaTest";

    private Random mRandom = new Random();

    private Sample[] mSamples = new Sample[]{
            new Sample(1, 0, new double[]{5, 3.5, 2, 0.4}), // should be in the setosa domain
            new Sample(2, 1, new double[]{5.6, 3, 3.5, 1.2}), // should be in the versicolor domain
            new Sample(3, 2, new double[]{7, 3, 6.8, 2.1}) // should be in the virginica domain
    };

    private Classifier mClassifier = null;

    TextView mTextViewSamples = null;

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_main);
        Toolbar toolbar = (Toolbar) findViewById(R.id.toolbar);
        setSupportActionBar(toolbar);

        // show samples
        StringBuilder sb = new StringBuilder("Samples:\n");
        for(Sample s : mSamples) {
            sb.append(s.toString() + "\n");
        }
        mTextViewSamples = (TextView) findViewById(R.id.textViewSamples);
        mTextViewSamples.setText(sb.toString());

        Log.d(WEKA_TEST, "onCreate() finished.");
    }

    @Override
    public boolean onCreateOptionsMenu(Menu menu) {
        // Inflate the menu; this adds items to the action bar if it is present.
        getMenuInflater().inflate(R.menu.menu_main, menu);
        return true;
    }

    @Override
    public boolean onOptionsItemSelected(MenuItem item) {
        // Handle action bar item clicks here. The action bar will
        // automatically handle clicks on the Home/Up button, so long
        // as you specify a parent activity in AndroidManifest.xml.
        int id = item.getItemId();

        //noinspection SimplifiableIfStatement
        if (id == R.id.action_settings) {
            return true;
        }

        return super.onOptionsItemSelected(item);
    }

    public void onClickButtonLoadModel(View _v) {
        Log.d(WEKA_TEST, "onClickButtonLoadModel()");

        AssetManager assetManager = getAssets();
        try {
            mClassifier = (Classifier) weka.core.SerializationHelper.read(assetManager.open("iris_model_logistic_allfeatures.model"));

        } catch (IOException e) {
            e.printStackTrace();
        } catch (Exception e) {
            // Weka "catch'em all!"
            e.printStackTrace();
        }
        Toast.makeText(this, "Model loaded.", Toast.LENGTH_SHORT).show();
    }

    public void onClickButtonPredict(View _v) {
        Log.d(WEKA_TEST, "onClickButtonPredict()");

        if(mClassifier==null){
            Toast.makeText(this, "Model not loaded!", Toast.LENGTH_SHORT).show();
            return;
        }

        // we need those for creating new instances later
        // order of attributes/classes needs to be exactly equal to those used for training
        final Attribute attributeSepalLength = new Attribute("sepallength");
        final Attribute attributeSepalWidth = new Attribute("sepalwidth");
        final Attribute attributePetalLength = new Attribute("petallength");
        final Attribute attributePetalWidth = new Attribute("petalwidth");
        final List<String> classes = new ArrayList<String>() {
            {
                add("Iris-setosa"); // cls nr 1
                add("Iris-versicolor"); // cls nr 2
                add("Iris-virginica"); // cls nr 3
            }
        };

        // Instances(...) requires ArrayList<> instead of List<>...
        ArrayList<Attribute> attributeList = new ArrayList<Attribute>(2) {
            {
                add(attributeSepalLength);
                add(attributeSepalWidth);
                add(attributePetalLength);
                add(attributePetalWidth);
                Attribute attributeClass = new Attribute("@@class@@", classes);
                add(attributeClass);
            }
        };
        // unpredicted data sets (reference to sample structure for new instances)
        Instances dataUnpredicted = new Instances("TestInstances",
                attributeList, 1);
        // last feature is target variable
        dataUnpredicted.setClassIndex(dataUnpredicted.numAttributes() - 1);

        // create new instance: this one should fall into the setosa domain
        final Sample s = mSamples[mRandom.nextInt(mSamples.length)];
        DenseInstance newInstance = new DenseInstance(dataUnpredicted.numAttributes()) {
            {
                setValue(attributeSepalLength, s.features[0]);
                setValue(attributeSepalWidth, s.features[1]);
                setValue(attributePetalLength, s.features[2]);
                setValue(attributePetalWidth, s.features[3]);
            }
        };
        // reference to dataset
        newInstance.setDataset(dataUnpredicted);

        // predict new sample
        try {
            double result = mClassifier.classifyInstance(newInstance);
            String className = classes.get(new Double(result).intValue());
            String msg = "Nr: " + s.nr + ", predicted: " + className + ", actual: " + classes.get(s.label);
            Log.d(WEKA_TEST, msg);
            Toast.makeText(this, msg, Toast.LENGTH_SHORT).show();
        } catch (Exception e) {
            e.printStackTrace();
        }
    }

    public class Sample {
        public int nr;
        public int label;
        public double [] features;

        public Sample(int _nr, int _label, double[] _features) {
            this.nr = _nr;
            this.label = _label;
            this.features = _features;
        }

        @Override
        public String toString() {
            return "Nr " +
                    nr +
                    ", cls " + label +
                    ", feat: " + Arrays.toString(features);
        }
    }
}
Categories: Data Analysis