C++: Using the Luhn Algorithm to Validate the CheckSum in the Personnummer Supplied in User Input

The Swedish Personnummer uses a variant of the Luhn Algorithm to generate a checksum (technically, a check digit) value that is is appended to the end of the personnummer. So, forexample, the nine-digit number YYMMDD-SSS would become YYMMDD-SSSC with the tenth digit being the checksum.

The model for generating the checksum is created by multiplying the alternating digits by a 2 or a 1 and, then, adding the sum of the products. In the nine-digit series, this means 212121212. So, for example, the number 010214-010 would become (0x2) + (1×1) + (0x2) + (2×1) + (1×2) + (4×1) + (0x2) + (1×1) + (0x2). When the product is greater than nine, the sum of those numbers is added. So, for example, 2×7 = 14 –> 1 + 4 = 5 and, thus, our result would be five for that single instance of digit.

The last number in the sum is then subtracted from 10 to create the checksum number. So, for example, 46 –> 6 and 10-6 = 4, and 4 becomes our checksum value.

Below is an C++ program to validate the checksum for the provided personnumer, given in the user-input. PersonnummerSet is merely a struct of ints that I’ve created in a header file, so that it can be referenced by other libraries (either by direct assignment to type or pointer). StringMagic is another header file that uses std::string.erase/std::remove to remove the ‘-‘ character from the input (Swedish Personnummers are always written as YYMMDD-SSSC.)

#include <cstring> // Replacing the old and busted with the new hotness.
#include <iomanip>
#include <iostream>
#include <cstdio> // Replacing the old and busted with the new hotness.
#include "Personnummerset.h"
#include "StringMagic.h"

class Personnummer
{
public:
    /**
     *  The PersonNumber is a property of type PersonnnummerSet, which is used to process the digits
     *  from the personnummer provided.
     */
    PersonnummerSet PersonNumber {};

    /**
     *  Initializes a new instance of the <see cref="Personnummer" /> class.
     * @param number - The converted long from the user's input.
     */
    explicit Personnummer(long number)
    {
        // We separate the three digit serial to future-proof for deriving sex of the individual in future checks.
        this->PersonNumber.Year = static_cast<int>(number / 100000000 % 100);   // First 2 integers     (0&1)
        this->PersonNumber.Month = static_cast<int>(number / 1000000 % 100);    // Second 2 integers    (2&3)
        this->PersonNumber.Day = static_cast<int>(number / 10000 % 100);        // Third 2 integers     (4&5)
        this->PersonNumber.SerialNumber = static_cast<int>(number / 100 % 100); // Fourth 2 integers    (6&7)
        this->PersonNumber.Sex = static_cast<int>(number / 10 % 10);            // First 1 integer      (8)
        this->PersonNumber.CheckSum = static_cast<int>(number % 10);            // Second 1 integer     (9)
    }

public:
    /**
     *  Generates the Luhn Algorithm checksum and compares that from the checksum given from the user's input.
     * @return A boolean indicating of the calculated checksum matches the provided checksum.
     */
    bool ValidateCheckSum()
    {
        return this->PersonNumber.CheckSum == this->GenerateChecksum();
    }

private:
    /**
     * This is a variation of the Luhn Algorithm, which is used to generate the checksum from the
     * first nine digits of the personnummer. Each value in the first nine digits is multiplied by
     * either a 2 or a 1. See: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2773709/figure/Fig1/
    */
    int GenerateChecksum()
    {
        int firstCheck;
        int fifthCheck;
        int seventhCheck;
        int ninthCheck;
        int firstYear = PersonNumber.Year > 9 ? PersonNumber.Year / 10 % 10 : 0;
        int secondYear = PersonNumber.Year%10;
        int firstMonth = PersonNumber.Month > 9 ? PersonNumber.Month / 10 % 10 : 0;
        int secondMonth = PersonNumber.Month%10;
        int firstDay = PersonNumber.Day > 9 ? PersonNumber.Day / 10 % 10 : 0;
        int secondDay = PersonNumber.Day%10;
        int firstSerial = PersonNumber.SerialNumber/10%10;
        int secondSerial = PersonNumber.SerialNumber%10;

        int firstYearDigit = firstYear * 2;
        if (firstYearDigit > 9)
        {
            int firstSum = firstYearDigit/10%10;
            int secondSum = firstYearDigit%10;
            firstCheck = firstSum + secondSum;
        }
        else
        {
            firstCheck = firstYearDigit;
        }

        int secondCheck = secondYear * 1;
        int thirdCheck = firstMonth * 2;
        int fourthCheck = secondMonth * 1;
        int firstDayCheck = firstDay * 2;
        if(firstDayCheck > 9)
        {
            int firstSum = firstDayCheck/10%10;
            int secondSum = firstDayCheck%10;
            fifthCheck = firstSum + secondSum;
        }
        else
        {
            fifthCheck = firstDayCheck;
        }

        int sixthCheck = secondDay * 1;
        int firstDigitRandom = firstSerial * 2;
        if (firstDigitRandom > 9)
        {
            int firstSum = firstDigitRandom/10%10;
            int secondSum = firstDigitRandom%10;
            seventhCheck = firstSum + secondSum;
        }
        else
        {
            seventhCheck = firstDigitRandom;
        }

        int eighthCheck = secondSerial * 1;
        int sexCheck = PersonNumber.Sex * 2;
        if(sexCheck > 9)
        {
            int firstSum = sexCheck/10%10;
            int secondSum = sexCheck%10;
            ninthCheck = firstSum + secondSum;
        }
        else
        {
            ninthCheck = sexCheck;
        }

        int sum = firstCheck + secondCheck + thirdCheck + fourthCheck + fifthCheck + sixthCheck + seventhCheck + eighthCheck + ninthCheck;
        int lastOfSum = sum%10;
        int checkSum = 10 - lastOfSum;
        return checkSum;

    }
};

int main()
{
    std::string user_input;
    std::cout << "Enter Personnummer (YYMMDD-SSSC): " << std::endl;
    std::cin >> user_input;

    // Necessary or the long conversion treats '-' or '+' as a delimiter.
    // NOTE: People aged over 99 will always have '+' in their personnummer.
    if(user_input.find('-') != std::string::npos)
    {
        StringMagic::RemoveDashFromString(user_input);
    }
    else if (user_input.find('+') != std::string::npos)
    {
        StringMagic::RemovePlusFromString(user_input);
    }

    // Smart-pointers are the future...
    std::unique_ptr<Personnummer> personNumber (new Personnummer(std::stol(user_input)));
    std::cout << std::boolalpha;
    std::cout << "CheckSum Passes: " << personNumber->ValidateCheckSum() << std::endl;

    personNumber.reset();
}

The above code can be used to validate almost any personnummer – save for those individuals whom are over 99 years of age, as the separator changes from ‘-‘ to ‘+’.

…and that’s about the extent of fun with Swedish Personnummers in C++ that this post will cover.

Thanks for coming to this NERDTalk™ and happy programming!

Ubuntu: Changing the Crypt (Plymouth) Login Screen

So, with all of this customisation of my Ubuntu machine going on, there had to be something else that I could configure, right? Well, there’s a lot, actually, but from the purely aesthetical point of view, the crypt screen was the next logical choice.

For those of you who may not be aware, and haven’t implemented encryption via Yubikey and LUKS, you can encrypt your volume and are forced to enter it on boot to get into the system. Think of it like BitLocker™ for Windows™®, only better.

celtic-come-at-me-bro

Oh, yeah, that’s right: I said it…

Anyways, the first thing that you’ll want to do is find a Plymoth theme. There’s actually a site full of them, here, and there’s – arguably – hundreds (if not thousands) more that you could find across the internet.

So, first things first, we download the theme. In my case it was the Paw Ubuntu/Mint Floral theme, just for the sake of proof-of-concept and the aesthetic appeals to me. YYMV. However, this is the them being used in this haphazard “how-to”.

Once the file is downloaded, extract it. (NOTE: Make sure that the script’s name matches the folder name; otherwise, the later regex will fail.) Move the contents of the inner parent to the shared directory on Ubuntu:

$ sudo mv Paw-Ubuntu-Floral /usr/share/plymouth/themes/

Now that it’s copied, we need to tell Ubuntu to build a new boot screen with the one we just copied over. This is fairly easy to do (kind of) in all of three commands:

$ sudo update-alternatives --install /usr/share/plymouth/themes/default.plymouth default.plymouth /usr/share/plymouth/themes/Paw-Ubuntu-Floral/paw-ubuntu-floral.plymouth 100 $ sudo update-alternatives --config default.plymouth
Det finns 2 val för alternativet default.plymouth (som tillhandahåller /usr/share/plymouth/themes/default.plymouth).

  Val          Sökväg                                                                 Prioritet  Status
------------------------------------------------------------
* 0            /usr/share/plymouth/themes/ubuntu-logo/ubuntu-logo.plymouth               100       automatiskt läge
  1            /usr/share/plymouth/themes/Paw-Ubuntu-Floral/paw-ubuntu-floral.plymouth   100       manuellt läge
  2            /usr/share/plymouth/themes/ubuntu-logo/ubuntu-logo.plymouth               100       manuellt läge

Tryck  för att behålla nuvarande val[*], eller ange urvalsnummer: 1       
update-alternatives: använder /usr/share/plymouth/themes/Paw-Ubuntu-Floral/paw-ubuntu-floral.plymouth för att tillhandahålla /usr/share/plymouth/themes/default.plymouth (default.plymouth) i manuellt läge

$ sudo update-initramfs -u
update-initramfs: Generating /boot/initrd.img-4.15.0-46-generic

If you receive the following error, you probably didn’t verify the file name of the script, like I suggested in the section above:

W: plymouth module (/usr/lib/x86_64-linux-gnu/plymouth//.so) missing, skipping that theme.

If so, go back and change the name of the script and restart this entire process over from the beginning. (Yes, I know, which is why I cautioned you about it, before…)

Alright, assuming all went well, all you need to do, now, is to reboot and you’ll see your newfound plymouth theme when you’re prompted to input your credentials for the disk’s encryption.

Thanks for coming to this NERDTalk™ and happy Ubuntuing!

C++: Learning About Virtual Classes

Over the weekend, I was given something to learn for a potential job (in Sverige) and that was Virtual Classes in C++. If you’re familiar with Partial Classes the idea kind of related, only it’s better. I’m sure how much of a pain it will be to debug but that’s the next thing that I want to test.

Anyways, Virtual Classes work off of inheritence and just like inheritence, you’re “wrapping” one object around another but with virtual classes, the exposure is seemless.

Take these two classes for a principle example:

#include iostream
#include cstring // Replacing the old and busted with the new hotness.

class LabEquipment
{
    public:
        explicit LabEquipment(int number)
        {
            std::cout << "Lab Equipment: " << number << "\n";
        }

        virtual void LabWrite(std::string message)
        {
            std::cout << "Writing " << message << " from LabEquipment.\n";
        }
};

class Tubes : virtual public LabEquipment
{
    public:
        Tubes(int tubes, int number)
                : LabEquipment(number)
        {
            std::cout << "Tubes: " << tubes << "\n";
        }

        virtual void TubesWrite(std::string tubesMessage)
        {
            std::cout << "Writing " << tubesMessage.c_str() << " from Tubes.\n";
        }
};

I can then write the following to leverage both Tubes and LabEquipments classes:

// Object
Tubes tubes(2, 3);
tubes.TubesWrite("hello");
tubes.LabWrite("hello, again, ");

// Pointer
Tubes * pTubes = new Tubes(3,4);
pTubes->TubesWrite("hello");
pTubes->LabWrite("hello, again,");

If I debug this in CLion (on Ubuntu 18.04LTS), then I get what I would expect:

Lab Equipment: 3
Tubes: 2
Writing hello from Tubes.
Writing hello, again,  from LabEquipment.
Lab Equipment: 4
Tubes: 3
Writing hello from Tubes.
Writing hello, again, from LabEquipment.

Note that I didn’t have to do anything else but declare the class’ inheritence for the methods to be exposed. None of that this stuff is necessary for it for it to be exposed, as it’s purely abstracted from the child to the parent.

In this way, C++ affords more versatility that .NET, in that, if I truly need to change any part of a class, there needn’t be any further changes – save on the caller.

Now, suppose that I declare the same method twice:

class ShowMessage
{
    public:
        explicit ShowMessage{};
    
        void virtual OutMessage(std::string message)
        {
            std::cout << "Writing " << message << " from WriteMessage.";
        }
};

class DisplayMessage : virtual public ShowMessage
{
    public:
        DisplayMessage()
                : ShowMessage()
        {};
        
        void OutMessage(std::string message)
        {
            std::cout << "Writing " << message << " from DisplayMessage";
        }
};

Our compiler, if it’s anything worwhile, should immediately alert us that we hiding the method; which, in this case, happens.

Markering_027

The IDE telling us that we’ve errored and suggesting ways to fix it.

If we override the method we can just drive on, as if nothing ever happened. We can also delete the method in DisplayMessage (which is what we should really do, as it’s redundant code already accomplished from within the ShowMessage class).

Now, let’s suppose we look at properties. Here’s a basic set of classes with int properties:

class FirstCount
{
    public:
        int Forsta;

    public:
        explicit FirstCount(int count)
        {
            Forsta = count;
        };
};

class SecondCount : virtual public FirstCount
{
    public:
        int Andra;

    public:
        SecondCount(int secondCount, int firstCount)
                : FirstCount(firstCount)
        {
            Andra = secondCount + Forsta;
        };
};

We can then construct an object that references them.

SecondCount secondCount(240, 547);
int first = secondCount.Forsta;
int second = secondCount.Andra;

When we debug, we can see that the value is what we’d expect (without having to have had used this).

Markering_028

You can see that the vlaue for second/Andra is the sum of 240 and 547, the 547 having been passed through from the FirstCount class.

…And that’s about as far as I’ve gotten, really… There’s a whole diamond thing that we could get into but I can’t be arsed to get that complicated with it, yet.

Anyways, thanks for coming to this NERDTalk and happy programming!

Ubuntu 18.04 (LTS): Disk Encryption with YubiKey

(WARNING: If you don’t read this, very carefully, you’ll shoot your eye out! If you do, might I suggest Bacon Pancakes to cheer you up? Or Christopher Walken dancing?)

The suggested soundtrack for this post comes from MASTER BOOT RECORD. The genre is called “chiptune” and a big shout-out to my friend, Wes, on keying me into them.

In one of my previous posts, I covered using the YubiKey to lock the machine when it’s removed. I intend to do one on configuring the machine to require the YubiKey for login but that’ll be for another day (if I get around to it). For now, the last remaining piece is to tie the disk’s encryption to the Yubikey, which is what we’ll be doing in this post.

The first thing that we’ll want to do is add the PPA for PrivacyIdea:

sudo add-apt-repository ppa:privacyidea/privacyidea

If your device is configured like mine, adding a PPA will automatically launch apt-get update. If not, then – of course – have apt-get update the list of available packages:

sudo apt-get update

Next, we’ll install the YubiKey-LUKS package that we need to tie the LUKS encryption to the YubiKey.

sudo apt-get install yubikey-luks

Since I’ve already used Slot 2 in the YubiKey for login, I’m going to use Slot 1 for the disk encryption. You’ll need to plug-in your YubiKey for this.

(BE FOREWARNED: You will nuke whatever key is currently in that slot. This is an unrecoverable action.):

ykpersonalize -1 -ochal-resp -ochal-hmac -ohmac-lt64 -oserial-api-visible

Then, we’ll need to add a new key to Slot 7, which will be used when you input your password to the challenge from Yubikey. (Read: It’s not actually your password being stored there.)

sudo yubikey-luks-enroll -d /dev/sda3 -s 7

As soon as you do this, Ubuntu should prompt you to reboot. Do so and verify that your new password works. If it doesn’t, your old password should still work to get into the drive.

Now that we’ve set it up and verified that it’s working, let’s delete the original password because we’re concerned with security, yeah?

First, let’s get a list of the slots and see which one is open:

sudo cryptsetup luksDump /dev/sda3
[sudo] lösenord för [REDACTED]: 
LUKS header information for /dev/sda3

Version:       	1
Cipher name:   	aes
Cipher mode:   	xts-plain64
Hash spec:     	sha256
Payload offset:	4096
MK bits:       	512
MK digest:     	[REDACTED] 
MK salt:       	[REDACTED] 
MK iterations: 	105194
UUID:          	a806d67c-9357-4860-a296-948be2090293

Key Slot 0: ENABLED
	Iterations:         	1683106
	Salt:               	[REDACTED] 
	Key material offset:	8
	AF stripes:            	4000
Key Slot 1: DISABLED
Key Slot 2: DISABLED
Key Slot 3: DISABLED
Key Slot 4: DISABLED
Key Slot 5: DISABLED
Key Slot 6: DISABLED
Key Slot 7: ENABLED
	Iterations:         	1583950
	Salt:               	[REDACTED] 
	Key material offset:	3536
	AF stripes:            	4000

Now that we know Slot 0 was the O.G. key holder (because we added to Slot 7 and the rest of the slots are disabled), we can target it for removal.

To do that, we’ll run the following command.

(BE FOREWARNED: You will deleting the only passphrase that doesn’t require the YubiKey to be present in the system. If you haven’t rebooted the machine to verify the passphrase+YubiKey combination, I strongly urge you not to do this step.):

sudo cryptsetup -q luksKillSlot /dev/sda3 0

To verify, let’s dump the LUKS Slots again (redacted for brevity):

Key Slot 0: DISABLED
Key Slot 1: DISABLED
Key Slot 2: DISABLED
Key Slot 3: DISABLED
Key Slot 4: DISABLED
Key Slot 5: DISABLED
Key Slot 6: DISABLED
Key Slot 7: ENABLED

And, now, the disk is only protected by the password+YubiKey combination. So, if the device gets stolen and the YubiKey isn’t in it (say, when I’m travelling), then the device is pretty useless – that is, unless you reformat it and start with new operating system.

Thanks for coming to this NERDTalk™ and I hope it helps you in your security endeavours in the future. 🙂

How I Shot My Own Self in the Foot with Semaphores….

…and the lessons I’ve learned. (Seemed like it needed a continuation, yeah?)

So, one of my many (now backlogged) personal projects is to write a Windows Service that copies files from a source directory to a target directory (or directories) and then encrypts the files via the LSA account (if so configured in the app config file). The idea was bourne out of two premises:

  1. As someone who doesn’t currently have a cloud provider, I still need disaster recovery and, if I’m in this position, surely there’s at least one other person in the world in the same boat. Maybe.
  2. This is, perhaps, the more crucial part. We take for granted that we have pretty fat pipes with which to fit all of the cats that we could ever want into the tubes and still have room left over. Other areas of the world still don’t have this luxury. What opportunities are available to them, now that we’re in a mobile-first, cloud-first world?

So, since we’re dealing with disks, I wanted to prevent it being an attack surface and/or prevent disk queuing or thrashing. If you’ve ever seen my project for the Windows Service to download the Bing Images, then you’re already familiar with my preferred route of semaphore use, which is the SemaphoreSlim.

So, I write my service, compile it (via Visual Studio), and install on my machine (via InstallUtil). I fire it up and drop a file into the source folder and, when the timer hits, it copies and encrypts the files to the destinations. O.k., so far, so good. I add a second file and nothing happens. Wait a few more minutes, still nothing.

O.k., I shot myself in the foot, somewhere, but nowhere I need to solve the crime, where I’m also the murderer; which isn’t, necessarily, the easiest task in the world.

So, I check task manager. The service is still running, so we didn’t hit a second-chance exception and the service terminated. O.k., I’m not a total idiot, at least (I hope).

I check the Wait Chain Traversal in Task Manager and it says the process is running fine, nothing is blocked.

O.k…. That makes even less sense. Time to debug it. I create a dump of the process in task manager and crack it open in WinDbg.

I’m using a tool called PDE, which you can get from the public OneDrive shared by Andrew Richards here. I’ll be using it to look for a thread with the most amount of frames. Why? Well, just because nothing is happening, it doesn’t mean that nothing is actually happening.

And, wouldn’t you know it, it looks like a dead-lock:

0:000> !pde.deep 50

6 TID:0718 kb kbn kbnL kn knL kpn kPn
# Child-SP RetAddr Call Site
00 00000040`6d9fd158 00007ff8`a198c7ee ntdll!NtWaitForMultipleObjects+0x14
01 00000040`6d9fd160 00007ff8`9720264e KERNELBASE!WaitForMultipleObjectsEx+0xfe
02 00000040`6d9fd460 00007ff8`972024a8 clr!WaitForMultipleObjectsEx_SO_TOLERANT+0x62
03 00000040`6d9fd4c0 00007ff8`972022a1 clr!Thread::DoAppropriateWaitWorker+0x1e4
04 00000040`6d9fd5c0 00007ff8`971ed336 clr!Thread::DoAppropriateWait+0x7d
05 00000040`6d9fd640 00007ff8`9717703f clr!CLREventBase::WaitEx+0xc4
06 00000040`6d9fd6d0 00007ff8`9717700c clr!Thread::Block+0x27
07 00000040`6d9fd700 00007ff8`97176db1 clr!SyncBlock::Wait+0x19d
08 00000040`6d9fd840 00007ff8`8a05bd08 clr!ObjectNative::WaitTimeout+0xe1
09 00000040`6d9fd9c0 00007ff8`8a05bac9 mscorlib_ni!System.Threading.SemaphoreSlim.WaitUntilCountOrTimeout(Int32, UInt32, System.Threading.CancellationToken)+0x68
0a 00000040`6d9fda10 00007ff8`8a05b872 mscorlib_ni!System.Threading.SemaphoreSlim.Wait(Int32, System.Threading.CancellationToken)+0x149
0b 00000040`6d9fdaa0 00007ff8`37a41a12 mscorlib_ni!System.Threading.SemaphoreSlim.Wait()+0x12
0c 00000040`6d9fdad0 00007ff8`8a071333 Felsokning_Services_ShadowService!Felsokning.Services.ShadowService.ShadowService+<>c__DisplayClass35_2.b__4(System.String)+0x3d2
0d 00000040`6d9fdc40 00007ff8`8a060680 mscorlib_ni!System.Threading.Tasks.Parallel+<>c__DisplayClass17_0`1[[System.__Canon, mscorlib]].b__1()+0x293
0e 00000040`6d9fdd00 00007ff8`8a12a540 mscorlib_ni!System.Threading.Tasks.Task.InnerInvokeWithArg(System.Threading.Tasks.Task)+0x20
0f 00000040`6d9fdd30 00007ff8`896a5ab3 mscorlib_ni!System.Threading.Tasks.Task+<>c__DisplayClass176_0.b__0(System.Object)+0x110
10 00000040`6d9fdda0 00007ff8`896a5944 mscorlib_ni!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)+0x163
11 00000040`6d9fde70 00007ff8`896ef565 mscorlib_ni!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)+0x14
12 00000040`6d9fdea0 00007ff8`896eec93 mscorlib_ni!System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef)+0x215
13 00000040`6d9fdf50 00007ff8`896f3011 mscorlib_ni!System.Threading.Tasks.Task.ExecuteEntry(Boolean)+0x73
14 00000040`6d9fdf90 00007ff8`896f2eee mscorlib_ni!System.Threading.Tasks.ThreadPoolTaskScheduler.TryExecuteTaskInline(System.Threading.Tasks.Task, Boolean)+0x31
15 00000040`6d9fdfe0 00007ff8`8a05f8d7 mscorlib_ni!System.Threading.Tasks.TaskScheduler.TryRunInline(System.Threading.Tasks.Task, Boolean)+0x19e
16 00000040`6d9fe080 00007ff8`8a06c938 mscorlib_ni!System.Threading.Tasks.Task.InternalRunSynchronously(System.Threading.Tasks.TaskScheduler, Boolean)+0xa7
17 00000040`6d9fe110 00007ff8`8a06dbb7 mscorlib_ni!System.Threading.Tasks.Parallel.ForWorker[[System.__Canon, mscorlib]](Int32, Int32, System.Threading.Tasks.ParallelOptions, System.Action`1, System.Action`2, System.Func`4, System.Func`1, System.Action`1)+0x428
18 00000040`6d9fe1f0 00007ff8`8a06d46e mscorlib_ni!System.Threading.Tasks.Parallel.ForEachWorker[[System.__Canon, mscorlib],[System.__Canon, mscorlib]](System.Collections.Generic.IEnumerable`1, System.Threading.Tasks.ParallelOptions, System.Action`1, System.Action`2, System.Action`3, System.Func`4, System.Func`5, System.Func`1, System.Action`1)+0xd7
19 00000040`6d9fe2b0 00007ff8`37a4155a mscorlib_ni!System.Threading.Tasks.Parallel.ForEach[[System.__Canon, mscorlib]](System.Collections.Generic.IEnumerable`1, System.Action`1)+0x9e
1a 00000040`6d9fe350 00007ff8`8a071333 Felsokning_Services_ShadowService!Felsokning.Services.ShadowService.ShadowService.b__35_3(System.String)+0xea
1b 00000040`6d9fe3e0 00007ff8`8a060680 mscorlib_ni!System.Threading.Tasks.Parallel+<>c__DisplayClass17_0`1[[System.__Canon, mscorlib]].b__1()+0x293
1c 00000040`6d9fe4a0 00007ff8`8a12a540 mscorlib_ni!System.Threading.Tasks.Task.InnerInvokeWithArg(System.Threading.Tasks.Task)+0x20
1d 00000040`6d9fe4d0 00007ff8`896a5ab3 mscorlib_ni!System.Threading.Tasks.Task+<>c__DisplayClass176_0.b__0(System.Object)+0x110
1e 00000040`6d9fe540 00007ff8`896a5944 mscorlib_ni!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)+0x163
1f 00000040`6d9fe610 00007ff8`896ef565 mscorlib_ni!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)+0x14
20 00000040`6d9fe640 00007ff8`896eec93 mscorlib_ni!System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef)+0x215
21 00000040`6d9fe6f0 00007ff8`896f3011 mscorlib_ni!System.Threading.Tasks.Task.ExecuteEntry(Boolean)+0x73
22 00000040`6d9fe730 00007ff8`896f2eee mscorlib_ni!System.Threading.Tasks.ThreadPoolTaskScheduler.TryExecuteTaskInline(System.Threading.Tasks.Task, Boolean)+0x31
23 00000040`6d9fe780 00007ff8`8a05f8d7 mscorlib_ni!System.Threading.Tasks.TaskScheduler.TryRunInline(System.Threading.Tasks.Task, Boolean)+0x19e
24 00000040`6d9fe820 00007ff8`8a06c938 mscorlib_ni!System.Threading.Tasks.Task.InternalRunSynchronously(System.Threading.Tasks.TaskScheduler, Boolean)+0xa7
25 00000040`6d9fe8b0 00007ff8`8a06dc4f mscorlib_ni!System.Threading.Tasks.Parallel.ForWorker[[System.__Canon, mscorlib]](Int32, Int32, System.Threading.Tasks.ParallelOptions, System.Action`1, System.Action`2, System.Func`4, System.Func`1, System.Action`1)+0x428
26 00000040`6d9fe990 00007ff8`8a06d46e mscorlib_ni!System.Threading.Tasks.Parallel.ForEachWorker[[System.__Canon, mscorlib],[System.__Canon, mscorlib]](System.Collections.Generic.IEnumerable`1, System.Threading.Tasks.ParallelOptions, System.Action`1, System.Action`2, System.Action`3, System.Func`4, System.Func`5, System.Func`1, System.Action`1)+0x16f
27 00000040`6d9fea50 00007ff8`37a41376 mscorlib_ni!System.Threading.Tasks.Parallel.ForEach[[System.__Canon, mscorlib]](System.Collections.Generic.IEnumerable`1, System.Action`1)+0x9e
28 00000040`6d9feaf0 00007ff8`896a5ab3 Felsokning_Services_ShadowService!Felsokning.Services.ShadowService.ShadowService.StartShadowCopyWork(System.Object)+0x246
29 00000040`6d9fec00 00007ff8`896a5944 mscorlib_ni!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)+0x163
2a 00000040`6d9fecd0 00007ff8`896db0ba mscorlib_ni!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)+0x14
2b 00000040`6d9fed00 00007ff8`896daeb7 mscorlib_ni!System.Threading.TimerQueueTimer.CallCallback()+0xba
2c 00000040`6d9fed60 00007ff8`89695b05 mscorlib_ni!System.Threading.TimerQueueTimer.Fire()+0x87
2d 00000040`6d9fedd0 00007ff8`97046da3 mscorlib_ni!System.Threading.TimerQueue.FireNextTimers()+0x75
2e 00000040`6d9fee50 00007ff8`97046c58 clr!CallDescrWorkerInternal+0x83
2f 00000040`6d9fee90 00007ff8`9704751d clr!CallDescrWorkerWithHandler+0x4e
30 00000040`6d9feed0 00007ff8`97049d14 clr!MethodDescCallSite::CallTargetWorker+0xf8
31 00000040`6d9fefd0 00007ff8`97047ce1 clr!AppDomainTimerCallback_Worker+0x34
32 00000040`6d9ff0c0 00007ff8`97047c50 clr!ManagedThreadBase_DispatchInner+0x39
33 00000040`6d9ff100 00007ff8`97047b8d clr!ManagedThreadBase_DispatchMiddle+0x6c
34 00000040`6d9ff200 00007ff8`97047d1f clr!ManagedThreadBase_DispatchOuter+0x75
35 00000040`6d9ff290 00007ff8`97049cb6 clr!ManagedThreadBase_FullTransitionWithAD+0x2f
36 00000040`6d9ff2f0 00007ff8`97049c16 clr!AppDomainTimerCallback+0x7c
37 00000040`6d9ff350 00007ff8`9704975f clr!ThreadpoolMgr::AsyncTimerCallbackCompletion+0x7e
38 00000040`6d9ff3a0 00007ff8`9704835c clr!UnManagedPerAppDomainTPCount::DispatchWorkItem+0x1bc
39 00000040`6d9ff440 00007ff8`97048105 clr!ThreadpoolMgr::ExecuteWorkRequest+0x64
3a 00000040`6d9ff470 00007ff8`971fefef clr!ThreadpoolMgr::WorkerThreadStart+0xf5
3b 00000040`6d9ff510 00007ff8`a43181f4 clr!Thread::intermediateThreadProc+0x86
3c 00000040`6d9ffbd0 00007ff8`a538a251 kernel32!BaseThreadInitThunk+0x14
3d 00000040`6d9ffc00 00000000`00000000 ntdll!RtlUserThreadStart+0x21

Threads: 1 of 13
Frames : 50
Command: knL

Right, so what’s the state of the SemaphoreSlim, itself?

0:006> !mdt 000001f45a4f3200
000001f45a4f3200 (System.Threading.SemaphoreSlim)
    m_currentCount:0x0 (System.Int32)
    m_maxCount:0x7fffffff (System.Int32)
    m_waitCount:0x6 (System.Int32)
    m_lockObj:000001f45a4f3240 (System.Object)
    m_waitHandle:NULL (System.Threading.ManualResetEvent)
    m_asyncHead:NULL (System.Threading.SemaphoreSlim+TaskNode)
    m_asyncTail:NULL (System.Threading.SemaphoreSlim+TaskNode)

The properties that we care about are the m_currentCount and the m_waitCount. We can look at the source code for SemaphoreSlim and see that these signify what the problem is:

m_currentCount – The semaphore count, initialized in the constructor to the initial value, every release call increments it and every wait call decrements it as long as its value is positive otherwise the wait will block. Its value must be between the maximum semaphore value and zero.

m_waitCount – The number of synchronously waiting threads, it is set to zero in the constructor and increments before blocking the threading and decrements it back after that. It is used as flag for the release call to know if there are waiting threads in the monitor or not.

So, here’s what happened: I leveraged Parallel.ForEach and for each disk action I required the caller to request wait on the semaphore, until it completes and releases the semaphore. The Semaphore’s initial value was stupid-low: 10. So, as soon as the second file was added, we created a condition in which multiple threads (because of parallelization) were requesting locks on the Semaphore and starving it out of being available for any further work.

In theory, this would be 1 lock for the source directory, 2 locks for the two target directories, 1 lock per file (so 2x2x2)… You can quickly see why this was a bad approach, even though it had the best intentions, at heart.

So, now, I’m refactoring the code to leverage threadpooling (or async tasks) and have definitely learned my lesson on resource starvation. 🙂

Thanks for coming to this TEDTalk and happy debugging!

Debugging One of the World’s Worst Error Messages

So, for a project for a friend with cancer, I was setting up something leveraging NextCloud (which another friend set up an account for me on their local instance for). It isn’t important who the friends were or what the project was but it’s slight backdrop into why I didn’t just use another program.

So, I installed the NextCloud desktop client on a Win10 (x64) machine and went to fire it up. Then came this gloriously (and mundanely) generic error message:

Markering_022

Very descriptive and actionable, yeah? (hint: LoadLibrary actually hints as to what’s going on, if you’ve run into assembly referencing before, but we’ll cover that later in this post.)

Well, the first thought is to go to the Event Viewer to see what happened. Only problem is: No events were logged. Anywhere. It never happened. It never even existed.

O.k., time to get medieval on this. Since the window never renders for the application and this seems to happen on initialisation, we’ll have to hook into the application to catch the event[s] in question.

Enter ADPlus. ADPlus has been around for hot-minute and isn’t, precisely, my go-to for debugging puposes; however, it’s definitely handy when you want to catch things happening on start-up.

So, we need to tell adplus to start and to monitor for an instance of the nextcloud application to loaded/referenced/scheduled by the kernel. (Littletid-bit here: The load happens before it’s reference is available in the process list, which is why we get the results you’ll see later-on.)

To tell adplus to “listen” for an instance of this application, we’ll do something like the following:

PS C:\Program Files (x86)\Windows Kits\10\Debuggers\x64> .\adplus.exe -Crash -pmn "nextcloud.exe" -o "C:\Dumps\NextCloud\"
Starting ADPlus
********************************************************
* *
* ADPLus Flash V 7.01.007 08/11/2011 *
* *
* For ADPlus documentation see ADPlus.doc *
* New command line options: *
* -pmn  - process monitor *
* waits for a process to start *
* -po  - optional process *
* won't fail if this process isn't running *
* -mss  *
* Sets Microsoft's symbol server *
* -r   *
* Runs -hang multiple times *
* *
* ADPlusManager - an additional tool to facilitate *
* the use of ADPlus in distributed environments like *
* computer clusters. *
* Learn about ADPlusManager in ADPlus.doc *
* *
********************************************************

Logs and memory dumps will be placed in C:\Dumps\NextCloud\\20190303_105927_Crash_Mode
Starting to monitor the following processes:
NEXTCLOUD.EXE
Press Enter to stop Monitoring...

Now, that we’ve set adplus to monitor, we fire-up nextcloud.exe. Adplus will attach and create a dump file for the first-chance exception (you can configure it for second-chance only).

So, now we have our dump file. We crack it open in Windbg and let’s look at the stack:

0:000> kncL
# Child-SP RetAddr Call Site
00 000000e2`704fdcc8 00007ff8`5c056db8 ntdll!NtTerminateProcess+0x14
01 000000e2`704fdcd0 00007ff8`5b35d3ba ntdll!RtlExitUserProcess+0xb8
02 000000e2`704fdd00 00007ff8`39a04e48 kernel32!ExitProcessImplementation+0xa
03 000000e2`704fdd30 00007ff8`39a0526e atig6pxx+0x4e48
04 000000e2`704fdef0 00007ff8`1289866c atig6pxx!DrvValidateVersion+0x16
05 000000e2`704fdf20 00007ff8`12898a7d opengl32!pgldrvLoadAndAllocDriverInfo+0x23c
06 000000e2`704fdfb0 00007ff8`128b2ca9 opengl32!LoadAvailableDrivers+0x39d
07 000000e2`704fe630 00007ff8`128b2055 opengl32!wglDescribePixelFormat+0x139
08 000000e2`704fe760 00007ff8`584d4c79 opengl32!wglChoosePixelFormat+0x85
09 000000e2`704fe810 00007ff8`1588eeb0 gdi32full!ChoosePixelFormat+0x39
0a 000000e2`704fe840 00007ff8`1588d540 qwindows+0x3eeb0
0b 000000e2`704fe990 00007ff8`1588eba6 qwindows+0x3d540
0c 000000e2`704fec50 00007ff8`1585c48b qwindows+0x3eba6
0d 000000e2`704fecf0 00007ff8`1585cdba qwindows+0xc48b
0e 000000e2`704fed50 00007ff8`1585ba5c qwindows+0xcdba
0f 000000e2`704fed80 00007ff8`11724d20 qwindows+0xba5c
10 000000e2`704fedf0 00007ff8`019b7770 Qt5Gui!QOpenGLContext::create+0x30
11 000000e2`704fee20 00007ff8`12d012da Qt5WebEngineCore!QtWebEngineCore::initialize+0x120
12 000000e2`704fee70 00007ff8`116ebd4d Qt5Core!QCoreApplicationPrivate::init+0x52a
13 000000e2`704fef30 00007ff8`131035cf Qt5Gui!QGuiApplicationPrivate::init+0x3d
14 000000e2`704ff190 00007ff8`130ff8bb Qt5Widgets!QApplicationPrivate::init+0xf
15 000000e2`704ff1c0 00007ff7`b1db1aa7 Qt5Widgets!QApplication::QApplication+0x5b
16 000000e2`704ff1f0 00007ff7`b1d039d4 nextcloud+0xc1aa7
17 000000e2`704ff300 00007ff7`b1cf1878 nextcloud+0x139d4
18 000000e2`704ff630 00007ff7`b1dc6524 nextcloud+0x1878
19 000000e2`704ff880 00007ff7`b1dc59c2 nextcloud+0xd6524
1a 000000e2`704ff910 00007ff8`5b3581f4 nextcloud+0xd59c2
1b 000000e2`704ff950 00007ff8`5c05a251 kernel32!BaseThreadInitThunk+0x14
1c 000000e2`704ff980 00000000`00000000 ntdll!RtlUserThreadStart+0x21

Normally, you might have to go frame by frame to understand what’s going on but since we don’t have the symbols for nextcloud or qwindow, we can kind of wing it and still figure out what’s going.

As the frames are read from bottom to top (in terms of processing order), the frames of interest to us will be 09 to 02 (which are in bold above). This shows us that GDI and/or OpenGL are looking for a driver that maps to a specific pixel format. We hit a frame in the ATI driver code that we can’t resolve to a method but then we see the call to terminate the process; so, we can probably discern that this is where the error screen is shown (the one with the LoadLibrary error).

So, if we going perusing around the interwebs, we should be able to confirm our suspicions and we’ll find this on MSDN, which has the following notation: “If the function fails, the return value is NULL. To get extended error information, call GetLastError.” Since this happens in native code and we’re unable to ascertain the method called in the ATI driver, it’s safe for us to assume that the ATI driver attempts to load a specified assembly and that assembly doesn’t exist. Also, since the exception is trapped and throw into the window, this is why the .excr record doesn’t exist on the stack. (Yay for native and scopes.)

Easiest solution: Install the nextcloud client on a different machine and do what I need to do or attempt to update the ATI video driver and hope that it resolves the issue. The former is a much more assured way to resolve it, though, so that’s what I did.

When developing programs, it’s important to consider that exceptions might occur in third-party dependencies. This mightn’t be the most productive error messages for users to figure out what’s going on. In fact, I whole-heartedly believe that if I hadn’t ever dealt with loading libraries or debugging, I’d have no clue as to why the program was broken.

This makes the error message one of the world’s crappiest error messages simply because it is inactionable by a user, if a user hasn’t the necessary tools and experience explained above.

So, then, what is the point of the error message?

I get the point, don’t get me wrong, in that lets someone know something went wrong. However, the general nature of the error message, itself, leads much to be desired. What library was it trying to load? Why did it fail? I’m sure I could eventually find a mapping for the NT Status errors that relate to this but why should I, as an end-user, have to resort to that level of depth just to figure out the problem?

The problem is shared between the application writers (nextcloud), GDI, OpenGL, and ATI. Each layer should expect an exception to be possible and to trap and raise it with the requisite information to make it actionable.

This is just… “Something happened. Good luck!”

Thanks for coming to this TEDTalk and I hope that it hasn’t bored you to death, despite it’s brevity.

Ubuntu 18.04LTS: Auto-Locking When YubiKey is Removed

It took me a while to piece this together, so I figured that I would write a handy “how-to”, in order to prevent someone else from losing the man-hours that it took to piecemeal this together.

The first thing that we’ll need to do is to create the monitoring rule. This rule is specifically used to key-off of the event that is trigged when the YubiKey is removed; however, we’ll need some information for that before we can write the rule.

You can use the below command to obtain the device-specific information:

udevadm monitor --environment --udev

With the command running, remove the YubiKey device from the system. I would suggest copying all of the properties that you see in the output to a text file. We’ll use some of these to define your rule.

Right. So, let’s create the rule. Run this command to start the text editor:

sudo nano /etc/udev/rules.d/85-yubikey.rules

In the text editor, you’ll add the rule. Below is mine, for an example:

# Yubikey Udev Rule: running a bash script in case your Yubikey is removed 
ACTION=="remove", ENV{ID_VENDOR_ID}=="1050", ENV{ID_MODEL_ID}=="0401", RUN+="/usr/local/bin/gnome-screensaver-lock"

The most specific piece that you’ll copy, verbatum, is the overloaded “Run” section.

Press Ctrl+X in the text editor window, when you’re done. Press your key for confirmation (‘Y’ in English, ‘J’ in Swedish, etc.). Then press ‘Enter’ (a.k.a.: carriage return) to confirm the path we specified when we started the text editor.

You should now be back at the terminal.

Now that the rule is in place, we need to define the action. We’ve already declared where it will go to fetch the action definition (see the overloaded “Run” declaration above) and now we need to create that file.

Run this command to start the text editor:

sudo nano /usr/local/bin/gnome-screensaver-lock

Add the following code in the editor:

#!/bin/bash 
# Double checking if the Yubikey is actually removed
if [ -z "$(lsusb | grep Yubico)" ]
then
        logger "YubiKey Removed or Changed"
        sessionids=`/bin/loginctl list-sessions | grep <userAccount> | awk '{print $1}'`
        for id in $sessionids
                do
                        logger "Locking session id:" $id
                        /bin/loginctl lock-session $id
                done
fi

(Be sure to change <userAccount> to your actual user account.)

Again, press Ctrl+X to start exiting the text editor. Press the confirmation key. Press the enter key to confirm the path.

The script that we just made must be made an executable for udev to be able to leverage it:

sudo chmod +x /usr/local/bin/gnome-screensaver-lock

Now that we have the rule and the actions defined, we need to tell the udevd service to reload the rules, so it’ll work. Run the following in the terminal:

sudo udevadm control --reload-rules
sudo service udev reload

Now, we should be able to remove the YubiKey device and Ubuntu should auto-lock.

If it doesn’t, you can try to monitor the udev service:

sudo service udev status
 systemd-udevd.service - udev Kernel Device Manager
   Loaded: loaded (/lib/systemd/system/systemd-udevd.service; static; vendor preset: enabled)
   Active: active (running) since Mon 2019-02-25 23:11:28 CET; 4 days ago
     Docs: man:systemd-udevd.service(8)
           man:udev(7)
 Main PID: 482 (systemd-udevd)
   Status: "Processing with 24 children at max"
    Tasks: 1
   CGroup: /system.slice/systemd-udevd.service
           └─482 /lib/systemd/systemd-udevd

mar 02 22:39:58 Tradgardsforeningen mtp-probe[15598]: checking bus 1, device 30: "/sys/devices/pci0000:00/0000:00:14.0/usb1/1-3"
mar 02 22:39:58 Tradgardsforeningen mtp-probe[15598]: bus: 1, device: 30 was not an MTP device
mar 02 22:43:17 Tradgardsforeningen systemd-udevd[15742]: Process '/usr/local/bin/gnome-screensaver-lock' failed with exit code 2.
mar 02 22:43:17 Tradgardsforeningen systemd-udevd[15749]: Process '/usr/local/bin/gnome-screensaver-lock' failed with exit code 2.
mar 02 22:43:20 Tradgardsforeningen mtp-probe[15759]: checking bus 1, device 31: "/sys/devices/pci0000:00/0000:00:14.0/usb1/1-3"
mar 02 22:43:20 Tradgardsforeningen mtp-probe[15759]: bus: 1, device: 31 was not an MTP device
mar 02 22:48:00 Tradgardsforeningen mtp-probe[15901]: checking bus 1, device 32: "/sys/devices/pci0000:00/0000:00:14.0/usb1/1-3"
mar 02 22:48:00 Tradgardsforeningen mtp-probe[15901]: bus: 1, device: 32 was not an MTP device
mar 02 22:49:26 Tradgardsforeningen mtp-probe[16019]: checking bus 1, device 33: "/sys/devices/pci0000:00/0000:00:14.0/usb1/1-3"
mar 02 22:49:26 Tradgardsforeningen mtp-probe[16019]: bus: 1, device: 33 was not an MTP device

The above snippet is from when the executing script looked quite different and, thus, it was failing to execute. Keep in mind that you may need to define different parameters in your monitoring rule, for example, if your device has a different vendor or model id and the removal isn’t creating the logging event:

mar 02 23:00:15 Tradgardsforeningen root[16941]: YubiKey Removed or Changed

If you can see the event being logged, then verify that you specified the correct username in the script.

If all else fails, there’s Ask Ubuntu. 🙂

Thanks for coming to this TEDTalk and I hope that it helps prevent someone losing the hours that I lost. 🙂