git-difftool: an easy way to highlight changes in git-managed text files

In two previous posts (post one, post two) we have shown how to use git latexdiff and latexdiff to visually highlight the differences of a Latex file in between its different versions in the form of a Latex-compiled diff file. For comparing arbitrary textfiles instead of Latex files, no Latex-based visual comparison is available. However, git itself ships a nice little tool to compare two different versions of the same textfile and highlight its differences: git difftool. It basically does two things for you:

  • Checkout the versions of the file to compare
  • Fire up your diff program to show you/highlight the differences

All you need to do is specify the file and the versions you want to compare.

Configuration

git_icon

git difftool is part of the git installation. However, it needs to know which diff program you want to use when calling git difftool. You can specify this by adding the following snippet to your ~/.gitconfig:

[diff]
    tool = MY-DIFF-TOOL

where MY-DIFF-TOOL could be vimdiff, kdiff3, or similar (ensure the tool you specify is installed and available in the PATH on your machine). Now git difftool should be ready for usage:

user@machine:~$ git difftool -h
usage: git difftool [-t|--tool=] [--tool-help]
                    [-x|--extcmd=]
                    [-g|--gui] [--no-gui]
                    [--prompt] [-y|--no-prompt]
                    [-d|--dir-diff]
                    ['git diff' options]

Usage

git difftool has a simple call syntax (in fact the same as git diff). The command you might want to compare files might be similar to this:

git difftool COMMIT1 COMMIT2 FILE

If you use

--

as COMMIT1 and leave out COMMIT2 you will see the differences between the file in its current, possibly unstaged/uncommited stage and its last commit. If you specify a hash for COMMIT1 and use

--

as COMMIT2 you will see the differences of the current file to the specified commit instead. If you specify hashes for both COMMIT1 and COMMIT2 you will see the differences between those two versions of the file.

DNS problems with Ubuntu? Disable dnsmasq with NetworkManager

Ubuntu LogoOn Ubuntu, if you suddenly cannot resolve DNS addresses anymore, though your network connection is up, you might just have run into a problem with dnsmasq (a local DNS server) that is used by NetworkManager. This post is about disabling dnsmasq and using the DNS servers advertised by your network directly instead.

Is it a DNS problem?

A good indication that you are facing a DNS problem on your machine is when you are connected to a network (meaning you still have an IP address assigned), but your internet connection suddenly stops working, and you are unable to ping DNS addresses like www.google.com:

ping www.google.com # unknown host

but you are still able to ping IP-addresses like 8.8.8.8:

ping 8.8.8.8 # works fine

This looks like you are not able to resolve DNS addresses any more – but you are still correctly connected to your network and to the internet.

What could cause this problem on Ubuntu?

Ubuntu uses NetworkManager, which in turn uses dnsmasq: a local DNS server running on your machine. As dnsmasq is started locally, redirecting DNS requests to the local address 127.0.0.1 would be fine in such a setup. You can see this being used by looking at your /etc/resolv.conf file: if it shows the following line, a local DNS server is in use:

nameserver 127.0.0.1

However, with certain setups you might run into problem when using this way of resolving DNS addresses. This might be the case e.g. when you are using a different program in parallel to manage your connections, which is not cooperating with dnsmasq well.

What could be the solution?

An easy solution is to try if your machine works fine without using dnsmasq. In this case DNS with be resolved not using a local dnsmasq DNS server but using the DNS server advertised via DHCP. To disable dnsmasq with NetworkManager, comment out the dns=dnsmasq line in /etc/NetworkManager/NetworkManager.conf:

# dns=dnsmasq

This will prevent NetworkManager from starting a dnsmasq instance, hence will prevent it from locally resolving DNS lookups. DNS lookups will then be done using the DNS servers advertised to your machine from DHCP. You can check that this actually worked as follows: after a reboot (clears all possible cached DNS servers info on your machine) and connecting to a network you should see actual DNS servers showing up in /etc/resolv.conf (with actual IP-addresses of course) in the place of the former 127.0.0.1 one:

nameserver x.x.x.x
nameserver y.y.y.y
nameserver z.z.z.z

Reduce pdf file size with GhostScript pdf compression under Linux/Unix

March 19, 2018 Leave a comment

We frequently need to mail pdf files that are too big for regular mail services, such as a 40MB pdf file with a maximum 10MB send restriction. In such situations quick and effective pdf compression comes in handy that does not reduce the quality to a level of the file becoming unusable.

GhostScript

GhostScript LogoUnder Linux and Unix-like systems GhostScript is one of the most powerful tools (probably the most powerful one) to manipulate files like pdf, ps, etc. If you are on a Linux/Unix-like system and need a job with pdf files done check out the “How to use GhostScript” site. It’s not unlikely that GhostScript already has a built in solution to your problem. Consequently it also features a way of effectively compressing pdf files with different options and settings.

GhostScript pdf compression

Effective pdf compression is possible with GhostScript using a single command (adapted from here and here):

gs -sDEVICE=pdfwrite -dCompatibilityLevel=1.4 -dPDFSETTINGS=/screen -dNOPAUSE -dQUIET -sOutputFile=outputfile.pdf inputfile.pdf

-dPDFSETTINGS= specifies the quality level of the pdf file. This effects embedded pixel graphics (also adapts embedded color profiles) and is the main option for controlling the compression level, thereby the resulting file size:

  • -dPDFSETTINGS=/screen (72 dpi images)
  • -dPDFSETTINGS=/ebook (150 dpi images)
  • -dPDFSETTINGS=/printer (300 dpi images)
  • -dPDFSETTINGS=/prepress (300 dpi images, color preserving)
  • -dPDFSETTINGS=/default

Other switches: the output is written as pdf (-sDEVICE=pdfwrite), the pdf compatibility level is set to 1.4 (-dCompatibilityLevel=1.4), the process does not require user interaction (-dNOPAUSE and -dQUIET), and GhostScript skips the startup message (-dQUIET).

Categories: Linux, Misc Tags: , , , , , , , ,

apt-get: install software/packages in specific version

February 5, 2018 Leave a comment

Sometimes there is a need to install a specific version of software for it to be compatible to other software, already have certain bug fixed, or because we need specific features. And sometimes there are multiple versions of the software available in the repos of your Linux machine, which allows for selecting the version that you want to have instead of just installing the default version. With apt this is possible – however, be careful to not install incompatible versions of software that causes conflicts with other things on your machine.

We are now going to demonstrate installing a specific version of fish, the user friendly interactive shell. At the time of writing there is only version 2.2.0.3 available in the Ubuntu 16.04.3 repositories, but we need at least version 2.3.x, as this is where fish became compatible to fzf, the fuzzy command-line finder (which is a very helpful tool btw). At the time of writing the latest version of fish in the Ubuntu PPA of fish 2.x is 2.7.x. We therefore at first need to add its PPA to make the version we need available to apt:

sudo apt-add-repository ppa:fish-shell/release-2
sudo apt-get update
sudo apt-get install fish

Now, to see all versions of the desired package available for installation on your machine with apt execute:

apt-cache madison fish

In our example we see:

fish | 2.7.0-1~xenial | http://ppa.launchpad.net/fish-shell/release-2/ubuntu xenial/main amd64 Packages
fish |    2.2.0-3 | http://fi.archive.ubuntu.com/ubuntu xenial/universe amd64 Packages

We can also look at some more details with:

apt-cache policy fish

In our example we see:

fish:
  Installed: (none)
  Candidate: 2.7.0-1~xenial
  Version table:
     2.7.0-1~xenial 500
        500 http://ppa.launchpad.net/fish-shell/release-2/ubuntu xenial/main amd64 Packages
     2.2.0-3 500
        500 http://fi.archive.ubuntu.com/ubuntu xenial/universe amd64 Packages
        100 /var/lib/dpkg/status

where 2.7.0-1~xenial is the exact identifier of the version we want to install for our scenario. We can now install this version using apt by setting the additional SW parameter:

sudo apt-get install SW=version

which in case of our example is:

sudo apt-get install fish=2.7.0-1~xenial

Done!

Redshift: protect your eyes during long nights in front of your PC

January 20, 2018 Leave a comment

For the night owls amongst us: looking at our screens during long night sessions – thereby at the whole spectrum of visible light from blue to red – means accepting a bunch of unhealthy drawbacks. For example: as the blue part of the spectrum of visible light is usually only visible during daytime our bodies have adapted to using it as a “clock” mechanism: while we see blue light we i.e. release hormones that keep us awake. That can e.g. make falling asleep more difficult/can cause insomnia when we have worked in front of a monitor for a couple of hours until late in the night. Therefore, besides dimming monitors (reduces the contrast to the dark surroundings), reducing the amount of blue light during night sessions and causing a tinge of red on our monitors is a good thing – not only for our eyes, but also for our bodies as a whole.

How to cause a redshift on your screen

Luckily for the night owls amongst us this problem has been recognized and addressed for different platforms already: for example, even many mobile phones provide for the functionality to redshift screen colors nowadays. For Linux, applications that do this job for us include redshift, openlux, and f.lux (the latter seems to be the original thing but is closed source). My current personal recommendation would be to use redshift, as it’s open source, included in the repositories of all major distributions, easy configurable, and does the job with a single command on your terminal. One word for the curious: from the technical perspective redshift relies on an X server extension to function.

Redshift logo

Redshift project logo

In terms of features redshift automatically adjusts screen colors to better match what the natural light would be – which would be caused by a sun that has probably already set hours ago, meaning there should be no blue light at all. redshift uses your location and time to adjust the screen colors. It provides for deriving the location automatically, but I personally like providing it by hand (see example below). Next to the location the second important setting is what you want your screen colors to be during daytime and nighttime (the color temperature in K to be exact). Both settings can be provided as parameters when calling redshift from the terminal:

redshift -t DAYTEMPERATURE:NIGHTTEMPERATURE -l LAT:LON

For example, I use those color temperatures and coordinates (for the latter providing exact data is not as important as it might seem):

redshift -t 6500:3200 -l 48:14

One more hint: of course you can autostart redshift with i3 by adding this line to your ~/.i3/config/i3.config:

exec --no-startup-id redshift -t 6500:3200 -l 48:14

 

git-latexdiff errors? Workaround: using latexdiff manually allows fixing broken things

December 4, 2017 Leave a comment

In a previous post we have highlighted git-latexdiff, which visually highlights the changes between two versions of the same latex file managed with git – without you being required to checkout any versions of that file yourself. In certain cases git-latexdiff might terminate with errors instead of producing a visual diff file. For example, it might struggle with broken latex code, broken figures, certain usage of subfiles, etc. In such cases, it might help to manually do what git-latexdiff would do fully automatically for you. Here’s a simple step-by-step explanation of how you could achieve visual highlighting even with broken Latex files:

  • Have the repository R2 available, which contains the file F in the newer version you want compared. Copy the repository R2 to a different folder R1 (you could do this on other ways too, but with copying you can hardly mess things up). In the copied folder checkout the version of the repository which contains F in the older version you want compared. Therefore, R2 contains F in the new version and R1 contains F in the old version now.
  • You can now fix both versions of F  should they contain broken latex code. F essentially needs to compile in both R2 and R1.
  • If you run into troubles with figures (also in subsequent steps) it might be helpfull to add the “draft” parameter to the documentclass. This can be done in both F files or in the diff.tex generated later. It disables figures in the documents, which causes less problems later in case figures are part of what prevents latexdiff, pdflatex, or one of the other tex tools from doing their jobs. This might also the case if figures have both been added and removed, or if figure paths have changed between those versions.

After those preparations, in your terminal run the following command from somewhere outside both repositories:

latexdiff --append-safecmd=subfile R1/path/to/F R2/path/to/F --flatten > diff.tex

The option

--apend-safecmd=subfile

makes the subfile command safe to use within the scope of the “\DIFadd” or “\DIFdel” commands, which latexdiff uses to mark differences between versions. The option

--flatten

replaces “\input” and “\include” commands within body by the content of the files in their argument. This makes all content of the document appear in your highlighted diff in the end. If the latexdiff-command succesfully generated a diff.tex file, copy this file into R2 or R1, right next to where F is located. You should now be able to compile and show the pdf file:

pdflatex diff.tex && biber diff && pdflatex diff.tex && evince diff.pdf

Fixing even more problems

In case you run into errors during compilation using the method above you might be able to manipulate the diff.tex file. If this does not help and if you are just interested in differences in the plain text there is an even stronger workaround you can try: you can copy the important parts of the content of your main .tex files in both R1 and R2 into newly created – and therefore clean – dummy.tex files. The important part of the text will usually be textual paragraphs, and more importantly for big documents, includes of subfiles. The dummy.tex files need to be of the same document class (e.g. a book class document will need a clean book again to work).  Remember to do this in an identical way in both R1 and R2. Afterwards you can compare the dummy.tex files using latexdiff (instead of the original main files). Doing so can circumvent many different types of problems that might be introduced due to code in custom Latex .cls or .sty files that is either unsupported by latexdiff and/or is broken in some way.

git latexdiff usage: visually highlight changes in version controlled Latex files

October 2, 2017 2 comments

This is a follow-up post to latexdiff-git, which is outdated by now.

latexdiff is a powerful tool that uses two Latex files to generate a third Latex file in which differences between the first two files are visually highlighted. latexdiff is especially useful when comparing an old with a new version of the same Latex file. However, latexdiff does not account for any version control on its own. This means that if you want to visually highlight the differences between two versions of a version controlled Latex file, you are required to manually checkout the two different versions ahead of comparing them with latexdiff. In case you are using git as your version control system, this is where git-latexdiff comes into play. It accounts for checking out the different version of a Latex file as well as comparing them with latexdiff in a single command: you only need to specify which latex file and versions should be used for the comparison.

Preconditions

Ensure you have git, a latex distribution (e.g. texlive-full), and latexdiff installed and available in the path of your CLI.

git-latexdiff installation

Clone git-latexdiff:

git clone https://gitlab.com/git-latexdiff/git-latexdiff#

Follow instructions in the README. This will involve a:

sudo make install

in most cases on an Ubuntu system. If all went well “git latexdiff” should be available in your CLI (it should print the help message):

user@machine:~$ git latexdiff

fatal: Please, provide at least one revision to diff with.
Usage: git latexdiff [options] OLD [NEW]
 git latexdiff [options] OLD --
 git latexdiff [options] -- OLD
Call latexdiff on two Git revisions of a file.

OLD and NEW are Git revision identifiers. NEW defaults to HEAD.
If "--" is used for NEW, then diff against the working directory.

Options:
 --help this help message
 --help-examples show examples of usage
 --main <file> name of the main LaTeX, R Sweave,
 or Emacs Org mode file.
 The search for the only file containing 'documentclass'
 will be attempted, if not specified.
 For non-LaTeX files, a reasonable `prepare` command
 will be used unless explicitly provided
 --no-view don't display the resulting PDF file
 --latex run latex instead of pdflatex
 --bibtex, -b run bibtex as well as latex
 (pdflatex,bibtex,pdflatex,pdflatex)
 --biber run BibLaTex-Biber as well as latex
 (pdflatex,bibtex,pdflatex,pdflatex)
 --view view the resulting PDF file
 (default if -o is not used)
 --pdf-viewer <cmd> use <cmd> to view the PDF file (default: $PDFVIEWER)
 --no-cleanup don't cleanup temp dir after running
 --no-flatten don't call latexpand to flatten the document
 --cleanup MODE Cleanup temporary files according to MODE:

- keeppdf (default): keep only the
 generated PDF file

- none: keep all temporary files
 (may eat your diskspace)

- all: erase all generated files.
 Problematic with --view when the
 viewer is e.g. evince, and doesn't
 like when the file being viewed is
 deleted.

--latexmk use latexmk
 --latexopt pass additional options to latex (e.g. -shell-escape)
 -o <file>, --output <file>
 copy resulting PDF into <file> (usually ending with .pdf)
 Implies "--cleanup all"
 --tmpdirprefix where temporary directory will be created (default: /tmp).
 Relative path will use repository root as a base
 --verbose, -v give more verbose output
 --quiet redirect output from subprocesses to log files
 --prepare <cmd> run <cmd> before latexdiff (e.g. run make to generate
 included files)
 --ln-untracked symlink uncommited files from the working directory
 --version show git-latexdiff version.
 --subtree checkout the tree at and below the main file
 (enabled by default, disable with --whole-tree)
 --whole-tree checkout the whole tree (contrast with --subtree)
 --ignore-latex-errors keep on going even if latex gives errors, so long as
 a PDF file is produced
 --ignore-makefile ignore the Makefile, build as though it doesn't exist
 -* other options are passed directly to latexdiff
 --bbl shortcut to flatten a bbl file of the same name as the project
 --latexpand OPT pass option OPT to latexpand. Use multiple times like
 --latexpand OPT1 --latexpand OPT2 to pass multiple options.
 --latexdiff-flatten use --flatten from latexdiff instead of latexpand

Unrecognized options are passed unmodified to latexdiff.

git-latexdiff usage

The main CLI usage of git-latexdiff is:

git latexdiff --main YOURFILE OLD_HASH [NEW_HASH]

A bunch of useful options are available for git-latexdiff. For example:

-v # good in case you run into any errors
--cleanup keeppdf # delete the checked-out 'old' and 'new' folders but keep the pdf
--cleanup all # delete all generated files afterwards
--output FILE # copy pdf before deleting it to FILE (disables automatically viewing the pdf). good in combination with --cleanup all
--tmpdirprefix ./FOLDERNAME/ # alternative to the above: specify where temporary stuff is stored, makes it easier to access diff files. good in combination with --cleanup keeppdf
--bibtex # also run bibtex
--biber # also run biber

Therefore, the command you might want to run therefore might be similar to this:

git latexdiff --main MYFILE.tex --bibtex --output git-latexdiff.pdf --cleanup all OLD_HASH NEW_HASH

…where you can specify

--

as NEW_HASH, if you want to run git latexdiff against the current (possibly unstaged/uncommited) version of the file, or where you can leave out NEW_HASH completely, if you want to run it against the last commit. Or your command might look similar to this:

mkdir git-latexdiff; git latexdiff --main MYFILE.tex --tmpdirprefix ./git-latexdiff/ --cleanup keeppdf OLD_HASH