Weka on Android: load precomputed model and predict new samples

November 27, 2016 Leave a comment

Weka on Android

Imagine you want to use machine learning on Android to predict the value for your target variable for new samples. Usually, neither the storage capabilities (for storing the training data) nor the computational power for training such models is available on mobile devices – assuming that you use a medium or large dataset and thorough evaluation of different model types and model parametrizations with a subsequent model selection. In such a scenario, the way to go therefore is:

  1. Train, evaluate, and select the optimal model for your machine learning prediction offline, e.g. using a desktop PC or server hardware.
  2. Export the ready trained model (“model file”, which usually tends to be relatively small) and ship it with your Android application.
  3. The “application case”/”production case” on the mobile device simply involves loading and using the model to predict new, yet unseen samples. Thereby, the prediction on the mobile device requires much less computational power and storage capabilities than the previous training of the model.

Based on the previous post about loading and using Weka models with JavaSE, this example demonstrates how to load and use a model Android which was previously trained with Weka “offline”, aka on a desktop PC. The core points you have to look out for when setting up the Android project:

  1. Use the exact same version of Weka you used for offline training the model also on Android.
  2. Copy the weka.jar to libs/, then select “Add as library…”.
  3. Ensure that “compile files(’libs/weka.jar’)” is present in build.gradle.
  4. Do a clean build: at the first time, this is going to require some CPU power and take some time.
  5. Don’t do anything with Weka that is UI related: Weka uses the JavaSE UI, which is not available on Android.
  6. Follow the details in the example below:
    1. Structure the new, yet unseen instance you want to predict the target variable so that Weka can work with it.
    2. Load the classifier object.
    3. Predict the target variable for the new instance.

The complete Android example project containing the code below can be downloaded here:

  • Download (7z): Android Weka Example.
  • SHA256sum: aaf8594954d3561218dc0dd621740f3184a171efa7016a2efe4b743a19845ffb
package at.fhooe.mcm.ml.androidwekaexample;

import android.content.res.AssetManager;
import android.os.Bundle;
import android.support.v7.app.AppCompatActivity;
import android.support.v7.widget.Toolbar;
import android.util.Log;
import android.view.View;
import android.view.Menu;
import android.view.MenuItem;
import android.widget.TextView;
import android.widget.Toast;

import java.io.IOException;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.Random;

import ml.mcm.fhooe.at.androidwekaexample.R;
import weka.classifiers.Classifier;
import weka.core.Attribute;
import weka.core.DenseInstance;
import weka.core.Instances;

public class MainActivity extends AppCompatActivity {

    private static final String WEKA_TEST = "WekaTest";

    private Random mRandom = new Random();

    private Sample[] mSamples = new Sample[]{
            new Sample(1, 0, new double[]{5, 3.5, 2, 0.4}), // should be in the setosa domain
            new Sample(2, 1, new double[]{5.6, 3, 3.5, 1.2}), // should be in the versicolor domain
            new Sample(3, 2, new double[]{7, 3, 6.8, 2.1}) // should be in the virginica domain
    };

    private Classifier mClassifier = null;

    TextView mTextViewSamples = null;

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_main);
        Toolbar toolbar = (Toolbar) findViewById(R.id.toolbar);
        setSupportActionBar(toolbar);

        // show samples
        StringBuilder sb = new StringBuilder("Samples:\n");
        for(Sample s : mSamples) {
            sb.append(s.toString() + "\n");
        }
        mTextViewSamples = (TextView) findViewById(R.id.textViewSamples);
        mTextViewSamples.setText(sb.toString());

        Log.d(WEKA_TEST, "onCreate() finished.");
    }

    @Override
    public boolean onCreateOptionsMenu(Menu menu) {
        // Inflate the menu; this adds items to the action bar if it is present.
        getMenuInflater().inflate(R.menu.menu_main, menu);
        return true;
    }

    @Override
    public boolean onOptionsItemSelected(MenuItem item) {
        // Handle action bar item clicks here. The action bar will
        // automatically handle clicks on the Home/Up button, so long
        // as you specify a parent activity in AndroidManifest.xml.
        int id = item.getItemId();

        //noinspection SimplifiableIfStatement
        if (id == R.id.action_settings) {
            return true;
        }

        return super.onOptionsItemSelected(item);
    }

    public void onClickButtonLoadModel(View _v) {
        Log.d(WEKA_TEST, "onClickButtonLoadModel()");

        AssetManager assetManager = getAssets();
        try {
            mClassifier = (Classifier) weka.core.SerializationHelper.read(assetManager.open("iris_model_logistic_allfeatures.model"));

        } catch (IOException e) {
            e.printStackTrace();
        } catch (Exception e) {
            // Weka "catch'em all!"
            e.printStackTrace();
        }
        Toast.makeText(this, "Model loaded.", Toast.LENGTH_SHORT).show();
    }

    public void onClickButtonPredict(View _v) {
        Log.d(WEKA_TEST, "onClickButtonPredict()");

        if(mClassifier==null){
            Toast.makeText(this, "Model not loaded!", Toast.LENGTH_SHORT).show();
            return;
        }

        // we need those for creating new instances later
        // order of attributes/classes needs to be exactly equal to those used for training
        final Attribute attributeSepalLength = new Attribute("sepallength");
        final Attribute attributeSepalWidth = new Attribute("sepalwidth");
        final Attribute attributePetalLength = new Attribute("petallength");
        final Attribute attributePetalWidth = new Attribute("petalwidth");
        final List<String> classes = new ArrayList<String>() {
            {
                add("Iris-setosa"); // cls nr 1
                add("Iris-versicolor"); // cls nr 2
                add("Iris-virginica"); // cls nr 3
            }
        };

        // Instances(...) requires ArrayList<> instead of List<>...
        ArrayList<Attribute> attributeList = new ArrayList<Attribute>(2) {
            {
                add(attributeSepalLength);
                add(attributeSepalWidth);
                add(attributePetalLength);
                add(attributePetalWidth);
                Attribute attributeClass = new Attribute("@@class@@", classes);
                add(attributeClass);
            }
        };
        // unpredicted data sets (reference to sample structure for new instances)
        Instances dataUnpredicted = new Instances("TestInstances",
                attributeList, 1);
        // last feature is target variable
        dataUnpredicted.setClassIndex(dataUnpredicted.numAttributes() - 1);

        // create new instance: this one should fall into the setosa domain
        final Sample s = mSamples[mRandom.nextInt(mSamples.length)];
        DenseInstance newInstance = new DenseInstance(dataUnpredicted.numAttributes()) {
            {
                setValue(attributeSepalLength, s.features[0]);
                setValue(attributeSepalWidth, s.features[1]);
                setValue(attributePetalLength, s.features[2]);
                setValue(attributePetalWidth, s.features[3]);
            }
        };
        // reference to dataset
        newInstance.setDataset(dataUnpredicted);

        // predict new sample
        try {
            double result = mClassifier.classifyInstance(newInstance);
            String className = classes.get(new Double(result).intValue());
            String msg = "Nr: " + s.nr + ", predicted: " + className + ", actual: " + classes.get(s.label);
            Log.d(WEKA_TEST, msg);
            Toast.makeText(this, msg, Toast.LENGTH_SHORT).show();
        } catch (Exception e) {
            e.printStackTrace();
        }
    }

    public class Sample {
        public int nr;
        public int label;
        public double [] features;

        public Sample(int _nr, int _label, double[] _features) {
            this.nr = _nr;
            this.label = _label;
            this.features = _features;
        }

        @Override
        public String toString() {
            return "Nr " +
                    nr +
                    ", cls " + label +
                    ", feat: " + Arrays.toString(features);
        }
    }
}
Categories: Data Analysis

Weka in Java: Predict new samples using a precomputed and exported model object

October 2, 2016 1 comment

weka

Weka allows for exporting/saving computed models into a binary model file (usually having a “.model” file extension). On other machines (that e.g. don’t have access to training data or would lack the computational power for training the model themselves), those model files can be loaded and used to predict new samples. The following example in Java highlights two aspects of this process, which are a) loading such a model object and b) how to format a new, yet unclassified instance so that its target variable can be predicted using the loaded model:

import java.util.ArrayList;
import java.util.List;

import weka.classifiers.Classifier;
import weka.core.Attribute;
import weka.core.DenseInstance;
import weka.core.Instances;

public class Main {

	public static void main(String[] args) {
		new Main().main();
	}

	public void main() {

		// we need those for creating new instances later
		final Attribute attributeSepalLength = new Attribute("sepallength");
		final Attribute attributeSepalWidth = new Attribute("sepalwidth");
		final Attribute attributePetalLength = new Attribute("petallength");
		final Attribute attributePetalWidth = new Attribute("petalwidth");
		final List<String> classes = new ArrayList<String>() {
			{
				add("Iris-setosa");
				add("Iris-versicolor");
				add("Iris-virginica");
			}
		};

		// Instances(...) requires ArrayList<> instead of List<>...
		ArrayList<Attribute> attributeList = new ArrayList<Attribute>(2) {
			{
				add(attributeSepalLength);
				add(attributeSepalWidth);
				add(attributePetalLength);
				add(attributePetalWidth);
				Attribute attributeClass = new Attribute("@@class@@", classes);
				add(attributeClass);
			}
		};
		// unpredicted data sets (reference to sample structure for new instances)
		Instances dataUnpredicted = new Instances("TestInstances",
				attributeList, 1);
		// last feature is target variable
		dataUnpredicted.setClassIndex(dataUnpredicted.numAttributes() - 1); 

		// create new instance: this one should fall into the setosa domain
		DenseInstance newInstanceSetosa = new DenseInstance(dataUnpredicted.numAttributes()) {
			{
				setValue(attributeSepalLength, 5);
				setValue(attributeSepalWidth, 3.5);
				setValue(attributePetalLength, 2);
				setValue(attributePetalWidth, 0.4);
			}
		};
		// create new instance: this one should fall into the virginica domain
		DenseInstance newInstanceVirginica = new DenseInstance(dataUnpredicted.numAttributes()) {
			{
				setValue(attributeSepalLength, 7);
				setValue(attributeSepalWidth, 3);
				setValue(attributePetalLength, 6.8);
				setValue(attributePetalWidth, 2.1);
			}
		};
		
		// instance to use in prediction
		DenseInstance newInstance = newInstanceSetosa;
		// reference to dataset
		newInstance.setDataset(dataUnpredicted);

		// import ready trained model
		Classifier cls = null;
		try {
			cls = (Classifier) weka.core.SerializationHelper
					.read("/home/pirius/iris_model_logistic_allfeatures.model");
		} catch (Exception e) {
			e.printStackTrace();
		}
		if (cls == null)
			return;

		// predict new sample
		try {
			double result = cls.classifyInstance(newInstance);
			System.out.println("Index of predicted class label: " + result + ", which corresponds to class: " + classes.get(new Double(result).intValue()));
		} catch (Exception e) {
			e.printStackTrace();
		}
	}
}
Categories: Data Analysis

Ubuntu Gnome 16.04: Some Useful Configuration

August 12, 2016 Leave a comment

Ubuntu Logo      Ubuntu Gnome Logo

A lot has changed in Ubuntu Gnome in the past 2 years. With the release of Ubuntu Gnome 16.04 a few months ago we take the chance to do a follow-up of our 2014 post and summarize the configuration we usually apply to a fresh installation of Ubuntu Gnome 16.04.

To consider for backup before installation

Just for completeness: you might want to backup some things beside private documents, music, videos etc, e.g.:

  • ~/.bashrc contains path variables, changes to bash behaviour etc.
  • ~/.bash_aliases contains aliases, other additions to bashrc.
  • ~/.conkyrc contains configuration for conky (if you use conky).
  • ~/.bash_profile “The personal initialization file, executed for login shells ” (see bash man page).
  • ~/bin/ may contain scripts and links to executables (as ~/bin is usually added to PATH).
  • ~/.config: contains configurations for different applications (e.g ~/config/compton.conf or ~/.config/i3/ if compton or i3 they are used).
  • ~/.gnupg/: contains PGP keys (used e.g. via Thunderbird enigmail).
  • ~/.local/share/rhythmbox if you use Rhythmbox, this folder contains your local playlists.
  • ~/.mozilla/: contains Firefox configuration, history etc.
  • ~/.purple/: contains purple based messaging configuration (e.g. pidgin), including possibly used OTR keys.
  • ~/.ssh/ contains ssh private and public keys, authorized keys for ssh login and ssh fingerprints seen in the past.
  • ~/.thunderbird/ contains Thunderbird configuration and mails.
  • GTK bookmarks: contains GTK based bookmarks (used e.g. by Nautilus, Nemo, Thunar, PCManFM etc). With Ubuntu 14.04 an newer they are located at ~/.config/gtk-3.0/bookmarks.
  • /etc/fstab contains automatic mounted partitions and folders.
  • cron jobs can be found on multiple locations (root-level /etc/crontab, root-level anacron and periodics /etc/cron*, user-level /var/spool/cron/crontabs)

Gnome Shell

In case you are new to Gnome Shell we strongly recommend reading a short intro, as this will probably save you lots of time later: Gnome shell cheat sheet one and two (they are not always updated to the newest…). Some examples for of useful features of Gnome Shell that might not be intuitive if you were using other UIs before:

  • The main menu of Gnome Shell style windows is hidden in the black area right next to the “Activities” button (top left) and gets shown only on click.
  • The message tray (lists messages from apps like Skype, Pidgin, Thunderbird etc.) is hidden as well and accessible via the shortcut Super+M.
  • The suspend button is hidden in the top right user menu under the power button and becomes visible on pressing Alt.
  • You can restart Gnome Shell with executing “r” in the Alt+F2 box without closing your applications etc. Useful to developers as well as to “soft reset” stuff in case of window freezes or other bugs. This is btw. equivalent to calling gnome-shell –replace directly (in case you do that from a tty without X server you need to specify the display – which will usually be 0 as you will typically only have one X server running: gnome-shell –replace -d :0).
  • Executing “lg” in Alt+F2 brings up the Gnome Shell log, which is useful in case of problems with Gnome Shell and its extensions.

Gnome Shell search providers

  • Type “search” in the activities view to bring up the search configuration. You might want to disable and/or reorder some choices here.
  • Type “privacy” in the activities view to bring up the privacy settings, which might be worthwhile to review as well.

Gnome Shell tweak tool

The tweak tool provides access to a number of tuning options. It should come preinstalled already, but in case it is not there yet:

sudo apt-get install gnome-tweak-tool

For example:

  • Changing the system font size (we usually change it from 11 to 10).
  • Changing the cursor theme (we usually change it to the dmz cursor theme:
     sudo apt-get install dmz-cursor-theme
  • Change workspace behavior, so that workspaces don’t only affect the primary screen but all screens.
  • Change how modal dialogs are attached to their parent windows.
  • Window selection model, if you’re used to mouse-over window selection (as I’m personally). Here’s the difference between mouseover and sloppy in case you wonder. I personally don’t like windows to be raised to the front automatically after the mouse pointer standing still for a certain time in it. This can be changed by turning off “automatically raise windows” in tweak tool or “auto-raise” in org–>gnome–>desktop–>wm–>preferences, or by adapting “auto-raise-delay” next to it. And in case you want windows to immediately get focus as you hover the mouse over them deselect “focus-change-on-pointer-rest” in org–>gnome–>shell–>overrides.

Gnome Shell extensions

We typically install some Gnome Shell Extensions, including the following ones:

  • AlternateTab: don’t group windows of same application in Alt+Tab, can be configured to show only the application icon instead of the window preview.
  • Dash to Dock: easy way to set maximum dock icon size + disabling “require pressure to show dock” to achieve nearly-normal dock behaviour.
  • Icon Hider: allows for individually showing/hiding icons in the top bar.
  • gTile: a window tiling utility – supports aligning windows to tiles spanning your screen (e.g. 2×2, 3×2, 4×4).
  • Native windows placement: more intuitive placing of windows in the activities window.
  • Notification Alerts: makes the user’s menu blink on new messages in the message tray.
  • Recent Items: adds list of recently used items to the top bar.
  • Search recently used files: integrated into the Gnome Shell activities view – this seems to work no longer as it’s not compatible with Gnome Shell >= 3.14. We’re still investigating how to best include recently used files in the activities search (frankly neither the Gnome Shell search providers “files” and “documents” nor other extensions seem to do this trick).
  • TopIcons Plus: one successor of TopIcons that nicely integrates into Gnome Shell and shows legacy application “tray” icons on the right side of the shell top bar.
  • User themes: enables loading of themes from ~/.themes/, which can then be selected in the gnome tweak tool to customize Gnome Shell.
  • WindowOverlay Icons: shows application symbol above windows in the activities overview. With centred and enlarged icons (~150px) helps pinpointing windows fast in our opinion.

Window button layout: put buttons left

We usually change the button layout so that the buttons are arranged on the left side (like with MAC systems) – as this saves you mouse distance. For Gnome Shell the needed tool is dconf-editor:

sudo apt-get install dconf-editor

In dconf-editor changes have to be applied to “button-layout” in  to org.gnome.desktop.wm.preferences in Ubuntu 16.04. Our usual button layout is “close,minimize,maximize”, all on the left side.

More look and feel: themes, icons, …

If you feel the need for changing the look and feel of your Gnome Shell even more, you can add different themes, icons sets, etc to Gnome Shell (all in the tweak tool). We sometimes use these:

  • Super Flat Remix Gnome Shell theme – “super lightweight”. If you use conky you might want to adapt the opacity to fit this theme (its at about 15%).
  • Elementary Icon set – the old one from 2013, “super clean”.
  • Depending on your needs you might also consider installing a dock shell extension (e.g. Dash to Dock, which features a bit more options for configuration than e.g. SimpleDock).
  • If you are instead up for a good looking and nicely working MAC oriented theme, consider the Gnome Shell/GTK+ theme OS X 10.11 El Capitan.

Some further configurations

Custom window shortcuts

Window specific shortcuts are mostly in dconf-editor in org–>gnome–>desktop–>wm–>keybindings. Additionally to adding assignments you might need to remove some to get your configuration working. An example of what we often use:

  • maximize: []
  • maximize-horizontally: [‘<Super>Page_Up’]
  • minimize: [‘<Super>Down’]
  • switch-to-workspace-up: [‘<Control><Alt>Up’]
  • toggle-maximized: [<Super>’Up’]
  • unmaximize: []

Custom command shortcuts

User specific commands can be defined in “Keyboard–>Shortcuts”. We typically create a shortcut to open a file manager at the data partition, such as Ctrl+Alt+E for nemo /media/data or nautilus /data/media, depending on your file manager.

Show all startup applications

Since Ubuntu 12.04 some startup applications are hidden, therefore not shown in gnome-session-properties. To show those applications use:

sudo sed -i "s/NoDisplay=true/NoDisplay=false/g"; /etc/xdg/autostart/*.desktop

Special Keyboard Buttons with MS Ergonomic Keyboard

If you happen to use a MS Ergonomic Keyboard (e.g. the Natural Ergonomic Keyboard 4000) some buttons might not work for you by default. It might be helpful to change the following:

  • For multimedia forward/backward: in Keyboard->Shortcuts->Sound and Multimedia: assign “Forward” and “Backward” to the previous/next track actions.
  • To use the zoom slider for scrolling add the following in etc/udev/hwdb.d/61-keyboard-local.hwdb
    keyboard:usb:v045Ep00DB*
     KEYBOARD_KEY_0c022d=pageup
     KEYBOARD_KEY_0c022e=pagedown

    Then recreate the hwdb:

    sudo udevadm hwdb --update

    Those might require to replug the keyboard or reboot.

Tooltip background color

The default tooltip background color is black – which is horrible for programs using dark fonts (like Eclipse). To change the tooltip background color install gnome-color-chooser and in Specific–>Tooltips change Foreground to black and Background to pale yellow, as Nick Andrik suggested here.

Application Configuration

File browser

  • Though Nautilus file browser has become better with the latest releases some might still not be convinced by it. One of the disadvantages it brings for power users is the missing tree view feature in the main area (the right one, not the left tree or places view). Options for replacing Nautilus include: Nemo, Thunar, PCManFM, Dolphin, and many more.
  • In case you stick with Nautilus:
    • If you dislike the recursive-search-as-you-type feature: you can disable it (and enable the old interactive search) by enabling org.gnome.nautilus.preferences.enable-interactive-search with the dconf-editor.
    • Most file browsers support the creation of files from templates in the context menu (right click -> “new document” -> select one of your templates). The location you need to put template files that should show up in this menu is ~/Templates/. If you notice that with Nautilus the “new documents” entry in the context menu is missing entirely, try creating an empty file in ~/Templates/, which should make it visible again.
    • Add custom actions with nautilus-actions-config-tool, such as running Terminator from the current location as stated in detail here.

Bash

We usually adapt Bash behavior to immediately log commands of all opened terminals.

Terminal emulation

Usually we also replace the default terminal emulation if it’s not very powerful (split screens, all hotkeys configurable etc.). Our current choice for Ubuntu is terminator, which features split screens and a detailed hotkey configuration – but of course there are many others out there with similar features.

Firefox

Besides enabling the “do not track me” flags in browsers other measures might also help reducing the traceability. In Firefox–>Properties–>Privacy (actual names might differ slightly, but the meaning is clear):

  • Enable “Firefox will record custom history”: discard third party cookies when Firefox closes
  • Enable “Clear history on Firefox close”: check “empty cache”, “discard cookies” and “discard active logins”

Firefox addons

Some Firefox addons we find useful, therefore tend to install straight away as well:

Thunderbird

We usually install some Thunderbird addons directly via Thunderbird->Tools->Addons. Addons we find useful are:

Disabling unnecessary notifications

With Gnome Shell, notifications tend to get persisted in the message tray. This is a good thing, as we won’t miss e.g. that we’ve been pinged in a chat, even if we were not at the PC when the ping happened. But we also don’t want unnecessary messages to show up there, as those tend to be distracting. Usually, we disable the following notifications (and alike in other programs):

  • Notifications from contacts coming online/offline in Pidgin, Skype, etc.
  • Notifications of music players playing the next track (e.g. Rhythmbox: disabling the “Notifications” plugin).
  • Notifications from any all-time-running terminal, like Yakuake or Guake.

Finally: root partition backup

Discussions about backups are omnipresent, so we won’t discuss the problem again here. In case there is no automatic backup having a “clean” – but properly configured system right after installation might be a good time to do a backup. That might save you lots of time in case users mess up the system. An option for quick backups is using fsarchiver from a live media (the installation media might come in handy). For a quick start with fsarchiver we’d recommend their quick start page. We assume that your installation was not split across multiple partitions in our example. The fsarchiver-command for backing up a partition on /dev/sda1 to /media/data/backups/datetime.fsa using 2 cores (-j2), splitting archives to multiple files of 4000MB (-s4000), standard compression (-z3) and verbose output (-v) from a live media would be:

sudo fsarchiver savefs -j2 -s4000 -z3 -v /media/data/backups/datetime.fsa /dev/sda1

The fsarchiver-command for restoring partition /dev/sda1 from /media/data/backups/datetime.fsa from a live media would be:

sudo fsarchiver restfs -j2 -v /media/data/backups/datetime.fsa id=0,dest=/dev/sda1

 

Categories: Linux

Einstein Riddle/Zebra Puzzle in Prolog

einsteinIn this post we solve another logic puzzle deriving rules from the puzzle and stating them in Prolog. And this time it is a rather famous puzzle one: the Zebra puzzle (also called Einstein riddle/puzzle).

The puzzle

The setting is as follows: there is a group of 5 people. Each person lives in exactly one house, has exactly one nationality and exactly one pet, drinks exactly one drink and smokes exactly one brand of cigarettes. Different goals are possible: a famous one is “who drinks water and who owns the zebra”. But one could instead also define the goal to derive for all houses and people: who lives is which house, has which nationality and which pet, drinks which drink and smokes which brand of cigarettes – given the fact, that each of these attributes is unique among the people in our group. This is the question we are going to answer. The following hints are provided (from Wikipedia):

  1.     There are five houses.
  2.     The Englishman lives in the red house.
  3.     The Spaniard owns the dog.
  4.     Coffee is drunk in the green house.
  5.     The Ukrainian drinks tea.
  6.     The green house is immediately to the right of the ivory house.
  7.     The Old Gold smoker owns snails.
  8.     Kools are smoked in the yellow house.
  9.     Milk is drunk in the middle house.
  10.     The Norwegian lives in the first house.
  11.     The man who smokes Chesterfields lives in the house next to the man with the fox.
  12.     Kools are smoked in the house next to the house where the horse is kept.
  13.     The Lucky Strike smoker drinks orange juice.
  14.     The Japanese smokes Parliaments.
  15.     The Norwegian lives next to the blue house.

Puzzle solution in Prolog

We now need to write the knowledge from above as Prolog rules and ask for a solution fulfilling these rules:

% ####################################################################
% Zebra puzzle solver. A house is a list with 6 attributes:
% 1. Color
% 2. Nationality
% 3. Drink
% 4. Cigarettes
% 5. Pet
% 6. The housenumber (fixed by hand to avoid redundant solutions)
%
% Steps to obtain the solution:
%
% 1) Read definition to prolog:
%   ['zebra_puzzle.pl'].
%
% 2) Ask for solutions:
%   solve(H1,H2,H3,H4,H5).
%
% Rainhard Findling
% 09/2015
% ####################################################################

all_members([H],L2) :- member(H,L2).
all_members([H|T],L2) :- member(H,L2), all_members(T, L2).

% ####################################################################
% RULES
% ####################################################################

% rule 1 (5 houses) is implicit for our implementation

rule2(H1,H2,H3,H4,H5) :-
    member(H, [H1,H2,H3,H4,H5]),
    H = [red,england,_,_,_,_].

rule3(H1,H2,H3,H4,H5) :-
    member(H, [H1,H2,H3,H4,H5]),
    H = [_,spain,_,_,dog,_].

rule4(H1,H2,H3,H4,H5) :-
    member(H, [H1,H2,H3,H4,H5]),
    H = [green,_,coffee,_,_,_].

rule5(H1,H2,H3,H4,H5) :-
    member(H, [H1,H2,H3,H4,H5]),
    H = [_,ukraine,tea,_,_,_].

rule6(H1,H2,H3,H4,H5) :-
    member(HL,[H1,H2,H3,H4,H5]),
    member(HR,[H1,H2,H3,H4,H5]),
    HL = [white,_,_,_,_,NrL],
    HR = [green,_,_,_,_,NrR],
    NrR-NrL =:= 1.

rule7(H1,H2,H3,H4,H5) :-
    member(H, [H1,H2,H3,H4,H5]),
    H = [_,_,_,altemgold,snails,_].

rule8(H1,H2,H3,H4,H5) :-
    member(H, [H1,H2,H3,H4,H5]),
    H = [yellow,_,_,kools,_,_].

rule9(_,_,H3,_,_) :-
    H3 = [_,_,milk,_,_,3].

rule10(H1,_,_,_,_) :-
    H1 = [_,norway,_,_,_,1].

rule11(H1,H2,H3,H4,H5) :-
    member(HL,[H1,H2,H3,H4,H5]),
    member(HR,[H1,H2,H3,H4,H5]),
    HL = [_,_,_,chesterfield,_,NrL],
    HR = [_,_,_,_,fox,NrR],
    abs(NrL-NrR,1).

rule12(H1,H2,H3,H4,H5) :-
    member(HL,[H1,H2,H3,H4,H5]),
    member(HR,[H1,H2,H3,H4,H5]),
    HL = [_,_,_,kools,_,NrL],
    HR = [_,_,_,_,horse,NrR],
    abs(NrL-NrR,1).

rule13(H1,H2,H3,H4,H5) :-
    member(H, [H1,H2,H3,H4,H5]),
    H = [_,_,orange,luckystrike,_,_].

rule14(H1,H2,H3,H4,H5) :-
    member(H, [H1,H2,H3,H4,H5]),
    H = [_,japan,_,parliament,_,_].

rule15(H1,H2,H3,H4,H5) :-
    member(HL,[H1,H2,H3,H4,H5]),
    member(HR,[H1,H2,H3,H4,H5]),
    HL = [_,norway,_,_,_,NrL],
    HR = [blue,_,_,_,_,NrR],
    abs(NrL-NrR,1).

% ####################################################################
% solve the puzzle
% ####################################################################

solve(H1,H2,H3,H4,H5) :-
    % bind houses to position to avoid multiple solutions with switched variables
    H1 = [H1_col,H1_nat,H1_drink,H1_cig,H1_pet,1],
    H2 = [H2_col,H2_nat,H2_drink,H2_cig,H2_pet,2],
    H3 = [H3_col,H3_nat,H3_drink,H3_cig,H3_pet,3],
    H4 = [H4_col,H4_nat,H4_drink,H4_cig,H4_pet,4],
    H5 = [H5_col,H5_nat,H5_drink,H5_cig,H5_pet,5],
    % the rules
    rule2(H1,H2,H3,H4,H5),
    rule3(H1,H2,H3,H4,H5),
    rule4(H1,H2,H3,H4,H5),
    rule5(H1,H2,H3,H4,H5),
    rule6(H1,H2,H3,H4,H5),
    rule7(H1,H2,H3,H4,H5),
    rule8(H1,H2,H3,H4,H5),
    rule9(H1,H2,H3,H4,H5),
    rule10(H1,H2,H3,H4,H5),
    rule12(H1,H2,H3,H4,H5),
    rule13(H1,H2,H3,H4,H5),
    rule14(H1,H2,H3,H4,H5),
    rule15(H1,H2,H3,H4,H5),
    % for all variable ensure that all values exist
    all_members([white, green, red, blue, yellow], [H1_col,H2_col,H3_col,H4_col,H5_col]),
    all_members([spain, japan, england, ukraine, norway], [H1_nat,H2_nat,H3_nat,H4_nat,H5_nat]),
    all_members([orange, coffee, milk, tea, water], [H1_drink,H2_drink,H3_drink,H4_drink,H5_drink]),
    all_members([luckystrike, parliament, altemgold, chesterfield, kools], [H1_cig,H2_cig,H3_cig,H4_cig,H5_cig]),
    all_members([dog, zebra, snails, horse, fox], [H1_pet,H2_pet,H3_pet,H4_pet,H5_pet]).

If executed, two possible solutions are printed:

['zebra_puzzle.pl'].
solve(H1,H2,H3,H4,H5).

H1 = [yellow, norway, water, kools, zebra, 1],
H2 = [blue, ukraine, tea, chesterfield, horse, 2],
H3 = [red, england, milk, altemgold, snails, 3],
H4 = [white, spain, orange, luckystrike, dog, 4],
H5 = [green, japan, coffee, parliament, fox, 5] ;

H1 = [yellow, norway, water, kools, fox, 1],
H2 = [blue, ukraine, tea, chesterfield, horse, 2],
H3 = [red, england, milk, altemgold, snails, 3],
H4 = [white, spain, orange, luckystrike, dog, 4],
H5 = [green, japan, coffee, parliament, zebra, 5] ;

That’s it, puzzle solved! 😉

SSH Proxy: Server and Client Side of Using an SSH User Without Shell as Proxy Server

April 30, 2016 Leave a comment

Accessing services on an internal network over a proxy reachable from the Internet.

Imagine you want one of your machines to become a proxy for external users to be able to access local resources or the internet as if they were on that machine. You could run a dedicated proxy server for that, but if the machine provides SSH and you want an easy solution, you can use SSH as well – without risking any shell-related issues (like this user also having access to the local file system).

Why would you want stuff like that in the first place? That’s what proxy servers (and most frequently VPNs) are made for: to access country-, company-, or university-internal resources from outside of that network just as you would from the inside.

Server side

The server side is rather easy. Let’s assume we call that user “sshproxy”. The important thing is to not give your proxy user a shell, which we do here during creation of the user:

sudo useradd -m -d /home/sshproxy -s /bin/false sshproxy # create user and home directory, disable shell

The user’s password is disabled by default (you can check that there’s a ! in /etc/shadow for the sshproxy user). If you don’t want to use passwords but private-public keys (which I would recommend): in /home/sshproxy create the .ssh/ folder and the .ssh/authorized_keys file and ensure they’re readable for the sshproxy user. There you need to add the ssh public keys of people that should be allowed to use the ssh proxy. The cool thing about it is that different real life users can make use of the same sshproxy user: you just need to add to manage the keys of real life users for the sshproxy user. Further, you likely don’t want the sshproxy user to be able to change the authorized keys, therefore make the file read only for that user (e.g. root owned + writeable by owner only). If you have an AllowUsers section in your /etc/ssh/sshd_config, you need to add the sshproxy user there and restart ssh. That’s it on the server side.

User side

If you’re one of the users that need to generate their ssh keypair, you can have a look at:

https://help.ubuntu.com/community/SSH/OpenSSH/Keys#Generating_RSA_Keys
(very easy to understand)

https://stribika.github.io/2015/01/04/secure-secure-shell.html
(details that are good to know if you want to have more secure keys = not so easy to crack)

If you use RSA then use at least 2048 bit key length (4096 can’t hurt…), e.g. with ssh-keygen with the “-b 4096” parameter. If you’re on a Windows machine: you can do the same using e.g. Putty. Don’t forget to use a good password to protect your private key file (usually the “id_rsa” file) and never disclose it – servers only need your public key (usually “id_rsa.pub”).

Using the proxy via Linux shell

Everything’s built in, just use the following command.

ssh -D 8080 -N sshproxy@YOUR-IP-OR-URL -p YOUR-PORT

This command opens the local port 8080 (on the client machine) for proxy tasks. YOUR-PORT-URL and YOUR-PORT correspond to the SSH server running on the server machine. If everything worked fine you won’t get any response in the shell – just leave the terminal open. Make sure your private key file is in ~/.ssh/id_rsa or provide it explicitly with “-i”. If you can’t access the server try removing the “-N”. Then you should see that the server logs you in and out immediately (this means everything works fine from the SSH, proxy and tunnel side – if you add -N again, you should be fine therefore). You can check your local opened ports with nmap (“nmap localhost”). If port 8080 is not in the list before login and opened after login your tunnel works.

Using the proxy via Putty

  • address: YOUR-IP-OR-URL (where SSH server is running)
  • port: YOUR-PORT (where SSH server is running)
  • user: sshproxy
  • Enable the check box “Don’t start a shell or command at all” in “Connection-SSH”
  • Specify your private key in “Auth”
  • Tunnels: add a “dynamic tunnel” on “local port 8080”, leave destination open (should state “D8080” in Putty after you apply)

Sending data to your local port

As you now have a tunnel on port 8080 on your local machine open, you need to send request to this port, instead of your standard gateway. E.g. for your browser you can achieve that configuring the browser to use a proxy (e.g. Firefox: FoxyProxy AddOn: set address “localhost” and port “8080”).

Btw: you can do other cool things with that proxy as well, such as reverse port tunnelling, circumventing firewalls etc. But you should ensure that this is allowed in the company/country you’re in before you get yourself into troubles.

Repair PDFLaTeX generated pdf using GhostScript (Adobe Acrobat Reader error 131)

February 28, 2016 Leave a comment

I tend to do presentations using LaTeX and Beamer, while working on Linux and using TeXLive as LaTeX distribution – which all work fine. But sometimes I need to share these PDFLaTeX compiled presentations with people using Windows and Adobe Acrobat Reader as their pdf viewer. The feedback I usually get back: your pdf is broken, error 131. And frankly, that seems to be true.

Opening PDFLaTeX generated pdf files with Windows Adobe Acrobat Reader results in error 131 – more precisely in the displayed error message “There was a problem reading this document (131)”. Other pdf viewers don’t complain about the pdf, just Adobe Acrobat Reader. A quick solution to repair the pdf file is by using GhostScript:

gs -dSAFER -dBATCH -dNOPAUSE  -sDEVICE=pdfwrite -sOutputFile=output.pdf input.pdf&amp;lt;/code&amp;gt;&amp;lt;/pre&amp;gt;

Voilà, the pdf file can be opened using Adobe Acrobat Reader – although, the source of the problem still exists of course (generating a pdf file not adhering the standard in the first place).

Git security: enabe fsckobjects in ~/.gitconfig:

February 3, 2016 Leave a comment

In order to prevent possible tampering with code in git repositories you work with (e.g. malicious manipulation of objects during clone, fetch, push…), check if these lines exist in your ~/.gitconfig and add them, if they don’t:

[transfer]
fsckobjects = true
[fetch]
fsckobjects = true
[receive]
fsckObjects = true

These enable git checking transferred objects for their integrity using their computed hashes.

Original idea from here: https://groups.google.com/forum/#!topic/binary-transparency/f-BI4o8HZW0
(and the corresponding bug on Debian here: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=813157)