Packaging an executable Python script nicely in Nix for NixOS

05/11/19 — sshow

This is a cross-post from blog.stigok.com

I was making a Python script as a backend to an Arduino project I did. As usual I got the idea of upping the ante in my hobby project, this time by packaging and running that backend on NixOS.

This was a great way for me to learn more about NixOS, but also a way to discover the difficulties of getting into it from scratch. I want to share the .nix files I ended up with, and how I incorporated it into my NixOS config, in the hope that I can speed things up for someone else getting to know Nix.

I will not go deep into how the language Nix works, but rather comment a tiny bit on what I’ve done that I feel is noteworthy for the new citizens of NixOS.

Building a custom Python package in NixOS

Python project

My Python project consists of just a few files, of which the below are the only ones that matter for the sake of this post.

├── tests
│   ├── test_data.json
│   └── test_ruterstop.py
├── requirements.txt
├── ruterstop.py
└── setup.py

The setup.py file is a small file that leans on the Python 3 built-in library distuils, which is used to define an installable Python package. Most importantly, it defines what scripts that should be added to the PATH, i.e. executable scripts.

# setup.py
from distutils.core import setup

setup(
    name='ruterstop',
    version='0.0.1',
    scripts=['ruterstop.py',],
)

This file will let you “install” the package/script by running python setup.py install, which is also how the Nix function python37Packages.buildPythonPackage builds Python packages by default.

Since the script will be installed as an executable, I have to define a suitable hashbang at the top of ruterstop.py:

#!/usr/bin/env python3

I will not be locking down my python library dependencies to specific versions right now, but rather just pull in the (latest) versions from the Nix package repository. The contents of my requirements.txt are:

bottle
requests

You will see these being defined explicitly in the Nix function buildPythonPackage later on. They will not be installed using pip.

Before continuing, you should verify that running python setup.py install works as intended. Maybe you have to run pip install -r requirements.txt on your development system first.

package.nix

You can start out in a new directory wherever you’d like.

Let’s start out with the default.nix file. This file will describe the package itself, where and how to get its sources, what dependencies to inject and how to actually build it.

# Below, we can supply defaults for the function arguments to make the script
# runnable with `nix-build` without having to supply arguments manually.
# Also, this lets me build with Python 3.7 by default, but makes it easy
# to change the python version for customised builds (e.g. testing).
{ nixpkgs ? import <nixpkgs> {}, pythonPkgs ? nixpkgs.pkgs.python37Packages }:

let
  # This takes all Nix packages into this scope
  inherit (nixpkgs) pkgs;
  # This takes all Python packages from the selected version into this scope.
  inherit pythonPkgs;

  # Inject dependencies into the build function
  f = { buildPythonPackage, bottle, requests }:
    buildPythonPackage rec {
      pname = "ruterstop";
      version = "0.0.1";

      # If you have your sources locally, you can specify a path
      #src = /home/stigok/src/ruterstop

      # Pull source from a Git server. Optionally select a specific `ref` (e.g. branch),
      # or `rev` revision hash.
      src = builtins.fetchGit {
        url = "git://github.com/stigok/ruterstop.git";
        ref = "master";
        #rev = "a9a4cd60e609ed3471b4b8fac8958d009053260d";
      };

      # Specify runtime dependencies for the package
      propagatedBuildInputs = [ bottle requests ];

      # If no `checkPhase` is specified, `python setup.py test` is executed
      # by default as long as `doCheck` is true (the default).
      # I want to run my tests in a different way:
      checkPhase = ''
        python -m unittest tests/*.py
      '';

      # Meta information for the package
      meta = {
        description = ''
          Realtime stop info for public transport in Oslo, using the EnTur JourneyPlanner API
        '';
      };
    };

  drv = pythonPkgs.callPackage f {};
in
  if pkgs.lib.inNixShell then drv.env else drv
  • You can see if your package file compiles with nix-instantiate --eval default.nix
  • You can build it and look at the resulting package with nix-build default.nix. A symlink results will be created in your working directory.

I want to run my service on boot with systemd. Next section takes on defining that service file.

service.nix

Create this file right next to default.nix

{ config, lib, pkgs, ... }:

let
        # The package itself. It resolves to the package installation directory.
        ruterstop = pkgs.callPackage ./default.nix {};

        # An object containing user configuration (in /etc/nixos/configuration.nix)
        cfg = config.services.ruterstop;

        # Build a command line argument if user chose direction option
        directionArg = if cfg.direction == ""
                          then ""
                          else "--direction=${cfg.direction} ";
in {
    # Create the main option to toggle the service state
    options.services.ruterstop.enable = lib.mkEnableOption "ruterstop";

    # The following are the options we enable the user to configure for this
    # package.
    # These options can be defined or overriden from the system configuration
    # file at /etc/nixos/configuration.nix
    # The active configuration parameters are available to us through the `cfg`
    # expression.

    options.services.ruterstop.host = lib.mkOption {
        type = lib.types.str;
        default = "0.0.0.0";
        example = "127.0.0.1";
    };
    options.services.ruterstop.port = lib.mkOption {
        type = lib.types.int;
        default = 4000;
    };
    options.services.ruterstop.stop-id = lib.mkOption {
        type = lib.types.str;
        example = "6013";
    };
    options.services.ruterstop.direction = lib.mkOption {
        type = lib.types.str;
        default = "";
        example = "inbound";
    };
    options.services.ruterstop.direction = lib.mkOption {
        type = lib.types.str;
        default = "";
        example = "inbound";
    };
    options.services.ruterstop.extraArgs = lib.mkOption {
        type = lib.types.listOf lib.types.str;
        default = [""];
        example = ["--debug"];
    };

    # Everything that should be done when/if the service is enabled
    config = lib.mkIf cfg.enable {
        # Open selected port in the firewall.
        # We can reference the port that the user configured.
        networking.firewall.allowedTCPPorts = [ cfg.port ];

        # Describe the systemd service file
        systemd.services.ruterstop = {
            description = "Et program som viser sanntidsinformasjon for stoppesteder i Oslo og Akershus.";
            environment = {
                PYTHONUNBUFFERED = "1";
            };

            # Wait not only for network configuration, but for it to be online.
            # The functionality of this target is dependent on the system's
            # network manager.
            # Replace the below targets with network.target if you're unsure.
            after = [ "network-online.target" ];
            wantedBy = [ "network-online.target" ];

            # Many of the security options defined here are described
            # in the systemd.exec(5) manual page
            # The main point is to give it as few privileges as possible.
            # This service should only need to talk HTTP on a high numbered port
            # -- not much more.
            serviceConfig = {
                DynamicUser = "true";
                PrivateDevices = "true";
                ProtectKernelTunables = "true";
                ProtectKernelModules = "true";
                ProtectControlGroups = "true";
                RestrictAddressFamilies = "AF_INET AF_INET6";
                LockPersonality = "true";
                RestrictRealtime = "true";
                SystemCallFilter = "@system-service @network-io @signal";
                SystemCallErrorNumber = "EPERM";
                # See how we can reference the installation path of the package,
                # along with all configured options.
                # The package expression `ruterstop` expands to the root
                # installation path.
                ExecStart = "${ruterstop}/bin/ruterstop.py --server --host ${cfg.host} --port ${toString cfg.port} --stop-id ${cfg.stop-id} ${directionArg}${lib.concatStringsSep " " cfg.extraArgs}";
                Restart = "always";
                RestartSec = "5";
            };
        };
    };
}

Adding the package to the system

Now the service can be registered in the system configuration. Since the package itself is defined by the service file, it’s not necessary to import default.nix.

Open /etc/nixos/configuration.nix and import the service.nix file. Somewhere up at the top is the imports section. Add the path to the service file there.

imports =
  [
    ./hardware-configuration.nix
    /path/to/your/ruterstop/service.nix # <--
  ];

Then, further down the file, where appropriate for your taste, enable the service and service options.

services.ruterstop.enable = true;
services.ruterstop.stop-id = "6013";
services.ruterstop.direction = "outbound";
services.ruterstop.extraArgs = ["--debug"];

Now, try to rebuild the system and switch to the new configuration immediately by using the switch argument.

# nixos-rebuild switch

If everything went right, that should have started the service and enabled it to start on boot. Check the service log to see if it runs alright.

# journalctl -u ruterstop.service

That should be it! Comments are very welcome!

References

Solution to SquareCTF 2019 - Talk To Me

19/10/19 — capitol
Name:

Talk To Me

Category:

general - ruby

Points:

100

Writeup

ruubies

We got a telnet interface to a small chat bot, but it didn’t understand anything we said to it at first.

Sending it characters that wasn’t in [A-Za-z] gave a more interesting result:

(eval):1: syntax error, unexpected end-of-input, expecting ')'
(%=;10-90*1.500/1.1002+9,|'")
                             ^
/talk_to_me.rb:16:in `eval'
/talk_to_me.rb:16:in `receive_data'
/var/lib/gems/2.5.0/gems/eventmachine-1.2.7/lib/eventmachine.rb:195:in `run_machine'
/var/lib/gems/2.5.0/gems/eventmachine-1.2.7/lib/eventmachine.rb:195:in `run'
/talk_to_me.rb:31:in `<main>'

So obviously they ran the input through eval. This led us down a rabbit hole of trying to get a reverse shell working, but that was blocked.

And then one of us noticed that the chat bot asked us to greet it as it had greeted us, so the problem was reduced to how to send it something that could pass the first check for [A-Za-z] and then be evaluated to the string ‘Hello!’.

Since none of us where ruby programmers this took quite a while to figure out. The solution we came up with in the end was:

$ (sleep 1; echo "('' << 72)+('' << 101)+( '' << 108)+( '' << 108)+('' << 111)+('' << 33)") | nc -v talk-to-me-dd00922915bfc3f1.squarectf.com 5678

The flag was flag-2b8f1139b0726726.

Release of Ripasso version 0.2.0

03/10/19 — capitol

ripasso-cursive

I have just released the first version of ripasso, a password manager that lets you control the level of risk that you expose your passwords too.

Ripasso aims to be filesystem compatible with pass, and this enables you to use the same password store across all your devices.

Passwords in ripasso are encrypted with one or more public pgp keys and optionally added to a git repository.

This gives you a lot of flexibility on how securely you want to manage the passwords. Here are some examples use cases:

  • You can encrypt the passwords with all the people on your team’s gpg keys and have a common passwords in a shared git repository.
  • You can store your passwords locally without using git, that way there’s no history of your old passwords to leak.
  • You can use your password store on your phone (if you use git as a backend), but have a separate gpg key for your computer and your phone. That way if you lose your phone, your gpg identity isn’t lost.

Ripasso is written in rust and so far packaged in nix and arch.

Install instructions

Arch linux

yay install ripasso-git

Nix

nix-env -iA nixpkgs.ripasso-cursive

General

git clone git@github.com:cortex/ripasso.git
cd ripasso
cargo build -p ripasso-cursive

Credits

  • Joakim Lundborg - developer
  • Alexander Kjäll - developer
  • Stig Palmquist - NixOS packager
  • David Plassman - Arch packager

Also a big thanks to everyone who contributed with bug reports and patches.

Mojolicious: Executing code with url_escape()

08/04/19 — sgo

failraptor

Let’s look at the url_escape function in the Mojolicious web framework for Perl 5, and how it can be used to evaluate code through the second argument of the function.

TLDR;

use Mojo::Util qw(url_escape);
url_escape('some-stuff', '\w](?{die()})|[a'); # Dies!

The url_escape function

This function is a part of the very useful Mojo::Util library.

It accepts a list of two arguments:

  1. A string to escape $str
  2. An optional pattern that determines what characters to escape $pattern

Some usage examples:

url_escape('foo/bar.baz'); # Returns "foo%2Fbar.baz"
url_escape('foo/bar.baz', '/.'); # Returns "foo%2Fbar%2Ebaz"

This is the implementation in Mojolicious v8.13:

sub url_escape {
  my ($str, $pattern) = @_;

  if ($pattern) {
    unless (exists $PATTERN{$pattern}) {
      (my $quoted = $pattern) =~ s!([/\$\[])!\\$1!g;
      $PATTERN{$pattern}
        = eval "sub { \$_[0] =~ s/([$quoted])/sprintf '%%%02X', ord \$1/ge }"
        or croak $@;
    }
    $PATTERN{$pattern}->($str);
  }
  else { $str =~ s/([^A-Za-z0-9\-._~])/sprintf '%%%02X', ord $1/ge }

  return $str;
}

When url_escape is called with a $pattern it hasn’t seen before, it will generate, string eval and cache a function to handle this specific pattern. Subsequent calls with the same $pattern will re-use the generated code.

So, an input parameter to the function is interpolated into a string which is evaled… Interesting! Let’s try to inject some code. :D

Quoting

(my $quoted = $pattern) =~ s!([/\$\[])!\\$1!g;

The $pattern input value is quoted with the expression s!([/\$\[])!\\$1!g; and stored in $quoted before it’s interpolated into the string of code to be evaled, preventing us from ending the pattern part of the string substitution. Damn… So close!

Code Subpattern

It doesn’t appear that we can (easily) break out of the substitution expression, so let’s try something inside the expression instead, like (?{ code }).

code subpattern: A regular expression subpattern whose real purpose is to execute some Perl code—for example, the (?{…}) and (??{…}) subpatterns.

Very nice, but this can be a bit dangerous, as the perlre doc points out:

[..] for reasons of security, use re ‘eval’ must be in scope. This is to stop user-supplied patterns containing code snippets from being executable. In situations where you need to enable this with use re ‘eval’ , you should also have taint checking enabled. Better yet, use the carefully constrained evaluation within a Safe compartment. [..]

Normally, adding (?{ code }) to a pattern through string interpolation would result in a fatal error, unless use re 'eval' is set. But since the code we want to tamper with is string evaled, these restrictions do not apply.

Crafting the $pattern argument

This is what we want to execute:

die()

Let’s wrap die() in a code subpattern and add some stuff to both sides of it to make it match and compile.

'\w](?{die()})|[a'

Passing this $pattern to url_escape makes it generate code that looks like this:

sub { $_[0] =~ s/([\w](?{die()})|[a])/sprintf '%%%02X', ord $1/ge }

So we’re left with:

url_escape('some-stuff', '\w](?{die()})|[a'); # Dies!

Is this a vulnerability?

The Mojolicious team reviewed a PoC before this post and concluded that it was not a security vulnerability.

This behaviour is not exploitable unless the function is used incorrectly. It could for example allow for vulnerability chaining if a bug is found that allows an attacker to control the $pattern argument.

How to create vulnerable code:

  1. The most obvious way to create a vulnerability is to expose the second $pattern argument of url_escape directly to user input. Game Over.

  2. A more subtle way is by having another function return a list an attacker can control as arguments to url_escape.

    use Mojo::Util qw(url_escape);
       
    sub good { return 'some-stuff' }; # returns 1 string
    sub evil { return ('some-stuff', '\w](?{die()})|[a') }; # returns 2 strings
       
    url_escape( good() ); 
    url_escape( evil() ); # Dies!
    

    Variations of this belong to a class of vulnerabilities in Perl web applications that are generally well known.

Takeaways

Regular expressions in Perl are very powerful, functions that return lists can be scary, and library functions can have hidden features.

Solution to UTCTF 2019 - Jacobi's Chance Encryption

16/03/19 — capitol
Name:

Jacobi’s Chance Encryption

Category:

crypto

Points:

750

Writeup

red herrings

We got this challenge text and an implementation of a strange crypto system that we needed to break:

Public Key 569581432115411077780908947843367646738369018797567841

Can you decrypt Jacobi’s encryption?

def encrypt(m, pub_key):
    bin_m = ''.join(format(ord(x), '08b') for x in m)
    n, y = pub_key

    def encrypt_bit(bit):
        x = randint(0, n)
        if bit == '1':
            return (y * pow(x, 2, n)) % n
        return pow(x, 2, n)

    return map(encrypt_bit, bin_m)

And also the encrypted flag.

Looking at the implementation it does some really strange things. It loops over every bit in the plaintext and gets a large random number x. Then it checks if the bit is 1 or 0 and encodes that information as either y * x2 or x2 in the congruence class of y.

There is also a public key involved that’s just two primes multiplied together, but it’s more of a red herring.

This is actually a kind of neat algebra problem. We want to know if a number was a square of itself before it got reduced by modulo n or not, and we know that n and the number are coprime.

As with all math problems, someone has solved this in sage already. The name of the function is kronecker, so all we had to do was to write a small sage program to reverse the crypto function.

file = open("flag.enc", "r")
data = file.readline()

data = data.split(",")

pubkey = list(factor(569581432115411077780908947843367646738369018797567841))

i = 0
str = ""
sol = ""
for b in data:
    if b == "":
        continue

    if kronecker(int(b, 16), pubkey[1][0]) == 0:
        str += "1"
    else:
        str += "0"

    i += 1

    if i % 8 == 0:
        sol += chr(int(str, 2))
        str = ""

print(sol)

The flag was utflag{did_u_pay_attention_in_number_theory}.