Using systemd services of Type=notify with Watchdog in C

15/03/18 — sshow

systemd watchdog

A systemd service of Type=notify waits for the executable program to send a notification message to systemd before it is considered activated. Up until the service is active, its state is starting. systemctl start <svc> will block until the service is active, or failed.

Similarly, a service which has WatchdogSec set will expect to receive a notification message no less than at every specified time interval. If no message has been received, systemd will kill the process with SIGABRT and place the service in a failed state.

Here is an example of how a service can be configured to automatically be restarted if it hangs.

[Unit]
Description=Watchdog test service

[Service]
ExecStart=/home/sshow/watchdog-service-test
Type=notify
WatchdogSec=15
Restart=on-failure
RestartSec=10

[Install]
WantedBy=default.target

Our example process is written in C.

The manpage for sd_watchdog_enabled recommends that the daemon sends a notification message every half of the time of WatchdogSec. The messages themselves are sent using sd_notify with the contents READY=1 when the process is done initialising, and WATCHDOG=1 for keep-alive notifications.

#include <stdio.h>              // printf
#include <stdlib.h>             // getenv
#include <unistd.h>             // sleep
#include <time.h>               // usleep
#include <systemd/sd-daemon.h>  // sd_notify


int main()
{
    setbuf(stdout, NULL);

    // Detect if expected to notify watchdog
    uint64_t watchdogNotifyIntervalUsec = 0;
    int watchdogEnabled = sd_watchdog_enabled(0, &watchdogNotifyIntervalUsec);

    // man systemd.service recommends notifying every half time of max
    watchdogNotifyIntervalUsec = watchdogNotifyIntervalUsec / 2;
    printf("Watchdog status: %d\tWatchdog notify interval (divided by 2): %ld\n", \
        watchdogEnabled, watchdogNotifyIntervalUsec);

    // Just to illustrate that `systemctl start` blocks until notified
    printf("Waiting five seconds before notifying ready state...\n");
    sleep(5);

    // Notify systemd service that we're ready
    sd_notify(0, "READY=1");
    printf("called sd_notify READY\n");

    int i = 0;
    while (1) {
        printf("iteration <%d>\n", i);

        if (watchdogEnabled) {
            // Notify systemd this service is still alive and good
            sd_notify(0, "WATCHDOG=1");
            printf("called sd_notify WATCHDOG\n");
        }

        i++;

        usleep(watchdogNotifyIntervalUsec);
    }

    return 0;
}

Our specific usage

We have a vending machine in our space that is running a card reader daemon for a NFC reader. The reader some times stops reading, or hangs unexpectedly. Now we can send watchdog notification messages whenever the NFC poll function runs, and if it doesn’t, systemd will kill the process, restart the service and (hopefully) help it start reading cards again.

References

  • man systemd.service
  • man sd_notify
  • man sd_enable_watchdog
  • https://stackoverflow.com/a/1157217/90674
  • https://stackoverflow.com/a/35653394/90674

Announcing the integration library between Struts 1.3 and spring 5.0

10/02/18 — capitol

The swedish word for ostrich is struts

Ageing java enterprise developers, look here!

Are you still maintaining an aging enterprise beast that you don’t have the budget to rewrite to modern micro-services?

Have your company spent too much money into an codebase so that you can never throw it away?

Do you still want to have experience with the latest and greatest toolset so that you stay relevant on the job market and can get a higher salary when you switch jobs?

If the answers to the above questions are yes, look no further!

Introducing the integration library between Struts 1.3 and Spring 5.0

Now you can use the latest spring release together with your old enterprise application.

The code was resurrected from the spring 3 code base and ported forward to spring 5.

Add this dependency to your struts project

    <dependency>
        <groupId>no.hackeriet</groupId>
        <artifactId>struts1-spring5</artifactId>
        <version>1.0.0</version>
    </dependency>

and just replace org.springframework.web.struts with no.hackeriet.struts1Spring.struts.

Written because sometimes it’s easier to write code than navigate politics.

By the way, don’t forget to patch the security holes in struts 1 yourself!

Happy hacking!

Running tomcat with systemd

08/01/18 — capitol

tomcat

The tomcat server’s documentation suggests using a custom compiled manager daemon called jsvc from the commons-daemon project.

Most modern linux systems uses systemd to manage it’s server processes and it has roughly the same capabilities as jsvc and much more.

To run tomcat on my machines I use a simple systemd service file that starts the service as the tomcat user and sets some basic java settings.

[Unit]
Description=Apache Tomcat Web Application Container
After=syslog.target network.target

[Service]
Type=forking

Environment=JAVA_HOME=/usr/lib/jvm/java-8-oracle/
Environment=CATALINA_PID=/opt/apache/apache-tomcat/temp/tomcat.pid
Environment=CATALINA_HOME=/opt/apache/apache-tomcat
Environment=CATALINA_BASE=/opt/apache/apache-tomcat
Environment='CATALINA_OPTS=-Xms512M -Xmx1024M -server -XX:+UseParallelGC'
Environment='JAVA_OPTS=-Djava.awt.headless=true -Djava.security.egd=file:/dev/./urandom'

ExecStart=/opt/apache/apache-tomcat/bin/startup.sh
ExecStop=/bin/kill -15 $MAINPID

User=tomcat
Group=tomcat
UMask=0007
RestartSec=10
Restart=always

[Install]
WantedBy=multi-user.target

Binding to port 80 or 443

It’s also possible to give tomcat permission to bind to ports below 1024 without running it as root by adding this line in the [Service] section

AmbientCapabilities=CAP_NET_BIND_SERVICE

And also change the port="8080" or port="8443" setting in server.xml.

Limiting memory, cpu or I/O

Systemd gives you control over how much cpu, memory and I/O tomcat can use, which can be useful if you run multiple micro-services on the same server and want to isolate them from each other.

This setting for example limits the amount of cpu available to 20% of one processor:

CPUQuota=20%

All options are described in the manual here.

Systemd uses the cgroups system in the linux kernel in order to control resource usage.

Security capabilities

Systemd also have a lot of other capabilities to lock down the service and reduce the effects if your application gets hacked. You can

  • Isolating services from the network
  • Service-private /tmp
  • Making directories appear read-only or inaccessible to services
  • Taking away capabilities from services
  • Disallowing forking, limiting file creation for services
  • Controlling device node access of services

as explained here.

Visiting Xil.se hackerspace in Malmö

06/01/18 — capitol

logo

Before 34c3 this year I visited Malmö for a couple of hours. And while I was there I managed to squeeze in a visit to the Xil.se hackerspace.

entrance

They have the most anonymous entrance I have ever seen, but after some guidance over the phone I managed to find it.

They are located in a residential part of Malmö, but are close to both public transports and some restaurants.

The space is located in a basement, and they are working on renovating it, but the places that they have finished are cosy.

main_workbench1

They do a lot of hardware and soldering, so most of the space is optimized for that.

solder_station

Having a microscope makes it super easy to solder tiny parts.

main_workbench2

The other rooms are not as furnished, but at least they managed to get some servers up

work-in-progress

And 3d-printers and games.

workbench

When they don’t do hardware, they play a lot of ctf’s and we have played a few events together.

We are looking forward towards visiting when it’s not in the middle of winter.

real-logo

Solution to nc3 Tys Tys

31/12/17 — capitol

acab

name:

Tys Tys

category:

reverse

points:

n/a

Writeup

The last problem in the danish police CTF was Tys Tys and we got a stripped elf binary named tys_tys and this danish text:

Fredag morgen kl. 7 mødte professor Tournesol ind på laboratoriet, hvor han arbejdede. Bag 
arbejdsbordet med softice-centrifugen fandt han en stærkt blødende professor Lennardo. Lennardo 
var såret, og gerningsmanden var flygtet. Før Lennardo udåndede, nåede han at remse en hemmelig 
cifferkode op: 3,2,0,4,2,2,5,0,6,7,10,9,11,2,1,11,0,7,10,6,10,8,9,0,6,7,10,9,11,2,1,11. Og med 
sit livs sidste kræftanstrengelser pegede han mod skuffe 3, hvor et USB-medie lå gemt. I skuffen 
lå ligeledes en seddel med teksten: "Jeg har for en sikkerhedsskyld nullet programmet."

På USB-mediet fandt Tournesol et program, og programmet gemte på en besked(flaget), der kunne 
redde verdenen!

Decompiling this in IDA gave us the following c code:

/* This file has been generated by the Hex-Rays decompiler.
   Copyright (c) 2007-2015 Hex-Rays <info@hex-rays.com>

   Detected compiler: GNU C++
*/

#include <defs.h>


//-------------------------------------------------------------------------
// Function declarations

int init_proc();
// ssize_t recv(int fd, void *buf, size_t n, int flags);
// int putchar(int c);
// char *strcpy(char *dest, const char *src);
// uint16_t htons(uint16_t hostshort);
// ssize_t send(int fd, const void *buf, size_t n, int flags);
// int snprintf(char *s, size_t maxlen, const char *format, ...);
// int close(int fd);
// struct hostent *gethostbyname(const char *name);
// void *malloc(size_t size);
// __int64 ptrace(enum __ptrace_request request, ...);
// void bzero(void *s, size_t n);
// int connect(int fd, const struct sockaddr *addr, socklen_t len);
// int __fastcall _cxa_finalize(_QWORD); weak
// int socket(int domain, int type, int protocol);
int sub_930();
int sub_9C0();
int sub_A00();
__int64 sub_A30();
void *sub_A9C();
_BYTE *sub_AEB();
const char *sub_BF0();
__int64 __fastcall main(__int64 a1, char **a2, char **a3);
void __fastcall init(unsigned int a1, __int64 a2, __int64 a3);
void term_proc();
// int ITM_deregisterTMCloneTable(void); weak
// int _gmon_start__(void); weak
// int Jv_RegisterClasses(void); weak

//-------------------------------------------------------------------------
// Data declarations

_UNKNOWN unk_E20; // weak
__int64 (__fastcall *off_201D68[3])() = { &sub_A00, &sub_A30, &sub_9C0 }; // weak
__int64 (__fastcall *off_201D78)() = &sub_9C0; // weak
_UNKNOWN unk_201D80; // weak
void *off_202008 = &off_202008; // weak
char byte_202010; // weak
_UNKNOWN unk_202017; // weak
// extern _UNKNOWN __cxa_finalize; weak


//----- (0000000000000858) ----------------------------------------------------
int init_proc()
{
  int (**v0)(void); // rax@1

  v0 = &_gmon_start__;
  if ( &_gmon_start__ )
    LODWORD(v0) = _gmon_start__();
  return (unsigned __int64)v0;
}
// 20205C: using guessed type int _gmon_start__(void);

//----- (0000000000000900) ----------------------------------------------------
#error "906: positive sp value has been found (funcsize=3)"

//----- (0000000000000930) ----------------------------------------------------
int sub_930()
{
  int (**v0)(void); // rax@1

  v0 = (int (**)(void))(&unk_202017 - (_UNKNOWN *)&byte_202010);
  if ( (unsigned __int64)(&unk_202017 - (_UNKNOWN *)&byte_202010) > 0xE )
  {
    v0 = &ITM_deregisterTMCloneTable;
    if ( &ITM_deregisterTMCloneTable )
      LODWORD(v0) = ITM_deregisterTMCloneTable();
  }
  return (unsigned __int64)v0;
}
// 202010: using guessed type char byte_202010;
// 202058: using guessed type int ITM_deregisterTMCloneTable(void);

//----- (00000000000009C0) ----------------------------------------------------
int sub_9C0()
{
  int result; // eax@4

  if ( !byte_202010 )
  {
    if ( &__cxa_finalize )
      _cxa_finalize(off_202008);
    result = sub_930();
    byte_202010 = 1;
  }
  return result;
}
// 8E8: using guessed type int __fastcall _cxa_finalize(_QWORD);
// 202008: using guessed type void *off_202008;
// 202010: using guessed type char byte_202010;

//----- (0000000000000A00) ----------------------------------------------------
int sub_A00()
{
  if ( unk_201D80 && &Jv_RegisterClasses )
    Jv_RegisterClasses();
  return 0;
}
// 202060: using guessed type int Jv_RegisterClasses(void);

//----- (0000000000000A30) ----------------------------------------------------
__int64 sub_A30()
{
  __int64 result; // rax@3
  signed int v1; // [sp+Ch] [bp-4h]@1

  v1 = 0;
  if ( !ptrace(0, 0LL, 1LL, 0LL) )
    v1 = 2;
  result = ptrace(0, 0LL, 1LL, 0LL);
  if ( result == -1 )
    result = (unsigned int)(3 * v1);
  return result;
}

//----- (0000000000000A9C) ----------------------------------------------------
void *sub_A9C()
{
  return &unk_E20;
}

//----- (0000000000000AEB) ----------------------------------------------------
_BYTE *sub_AEB()
{
  _BYTE *buf; // ST10_8@1
  struct hostent *v1; // ST18_8@1
  int fd; // ST08_4@1
  _BYTE *result; // rax@1
  __int64 v4; // rcx@1
  struct sockaddr addr; // [sp+20h] [bp-20h]@1
  __int64 v6; // [sp+38h] [bp-8h]@1

  v6 = *MK_FP(__FS__, 40LL);
  buf = malloc(3uLL);
  v1 = gethostbyname("45.63.119.180");
  fd = socket(2, 1, 0);
  addr.sa_family = 2;
  *(_WORD *)&addr.sa_data[0] = htons(0x115Cu);
  *(_DWORD *)&addr.sa_data[2] = **(_DWORD **)v1->h_addr_list;
  bzero(&addr.sa_data[6], 8uLL);
  connect(fd, &addr, 0x10u);
  send(fd, "?\n", 2uLL, 0);
  buf[(signed int)recv(fd, buf, 3uLL, 0)] = 0;
  close(fd);
  result = buf;
  v4 = *MK_FP(__FS__, 40LL) ^ v6;
  return result;
}

//----- (0000000000000BF0) ----------------------------------------------------
const char *sub_BF0()
{
  const char *result; // rax@1
  __int64 v1; // rdx@1

  result = "PRATEZ";
  v1 = *MK_FP(__FS__, 40LL) ^ *MK_FP(__FS__, 40LL);
  return result;
}

//----- (0000000000000C7C) ----------------------------------------------------
__int64 __fastcall main(__int64 a1, char **a2, char **a3)
{
  const char *v3; // rax@1
  const char *v4; // rax@1
  const char *v5; // rax@1
  __int64 result; // rax@4
  __int64 v7; // rcx@4
  signed int i; // [sp+0h] [bp-50h]@1
  char dest; // [sp+7h] [bp-49h]@1
  char v10; // [sp+Ah] [bp-46h]@1
  char v11; // [sp+Dh] [bp-43h]@1
  char s[13]; // [sp+13h] [bp-3Dh]@1
  __int64 v13; // [sp+20h] [bp-30h]@1
  __int64 v14; // [sp+28h] [bp-28h]@1
  __int64 v15; // [sp+30h] [bp-20h]@1
  __int64 v16; // [sp+38h] [bp-18h]@1
  __int64 v17; // [sp+48h] [bp-8h]@1

  v17 = *MK_FP(__FS__, 40LL);
  v3 = (const char *)sub_A9C();
  strcpy(&dest, v3);
  v4 = sub_AEB();
  strcpy(&v10, v4);
  v5 = sub_BF0();
  strcpy(&v11, v5);
  snprintf(s, 0xDuLL, "%s%s%s", &dest, &v10, &v11);
  v13 = 0LL;
  v14 = 0LL;
  v15 = 0LL;
  v16 = 0LL;
  for ( i = 0; i <= 31; ++i )
    putchar(s[*((_BYTE *)&v13 + i)]);
  putchar(10);
  result = 0LL;
  v7 = *MK_FP(__FS__, 40LL) ^ v17;
  return result;
}
// C7C: using guessed type char s[13];

//----- (0000000000000D90) ----------------------------------------------------
void __fastcall init(unsigned int a1, __int64 a2, __int64 a3)
{
  __int64 v3; // r13@1
  signed __int64 v4; // rbp@1
  __int64 v5; // rbx@2

  v3 = a3;
  v4 = &off_201D78 - off_201D68;
  init_proc();
  if ( v4 )
  {
    v5 = 0LL;
    do
      ((void (__fastcall *)(_QWORD, __int64, __int64))off_201D68[v5++])(a1, a2, v3);
    while ( v4 != v5 );
  }
}
// 201D68: using guessed type __int64 (__fastcall *off_201D68[3])();
// 201D78: using guessed type __int64 (__fastcall *off_201D78)();

//----- (0000000000000E04) ----------------------------------------------------
void term_proc()
{
  ;
}

#error "There were 1 decompilation failure(s) on 12 function(s)"

Some manual analysis gave us that the function sub_A9C always returns the string “_L3”.

The server at 45.63.119.180:4444 that was contacted returned the string “WND” when a ? was sent in, and “Forkert” for everything else, the server also crashed after the challenge had been running for a hour.

The last part was produced from the function sub_BF0 that returned “PRATEZ”.

Using the professors list of numbers as indexes into the string we could produce the flag:

<?php

$key =
array(3,2,0,4,2,2,5,0,6,7,10,9,11,2,1,11,0,7,10,6,10,8,9,0,6,7,10,9,11,2,1,11);

$str = "_L3WNDPRATEZ";

for($i = 0; $i < count($key); $i++){
        echo $str[$key[$i]];
}
?>

flag was: W3_N33D_PRETZ3LZ_REPEAT_PRETZ3LZ