Planet HantsLUG

May 19, 2016

Steve Kemp

Accidental data-store .. is go!

A couple of days ago I wrote::

The code is perl-based, because Perl is good, and available here on github:

..

TODO: Rewrite the thing in #golang to be cool.

I might not be cool, but I did indeed rewrite it in golang. It was quite simple, and a simple benchmark of uploading two million files, balanced across 4 nodes worked perfectly.

https://github.com/skx/sos/

May 19, 2016 06:38 PM

May 18, 2016

Steve Kemp

Accidental data-store ..

A few months back I was looking over a lot of different object-storage systems, giving them mini-reviews, and trying them out in turn.

While many were overly complex, some were simple. Simplicity is always appealing, providing it works.

My review of camlistore was generally positive, because I like the design. Unfortunately it also highlighted a lack of documentation about how to use it to scale, replicate, and rebalance.

How hard could it be to write something similar, but also paying attention to keep it as simple as possible? Well perhaps it was too easy.

Blob-Storage

First of all we write a blob-storage system. We allow three operations to be carried out:

  • Retrieve a chunk of data, given an ID.
  • Store the given chunk of data, with the specified ID.
  • Return a list of all known IDs.

 

API Server

We write a second server that consumers actually use, though it is implemented in terms of the blob-storage server listed previously.

The public API is trivial:

  • Upload a new file, returning the ID which it was stored under.
  • Retrieve a previous upload, by ID.

 

Replication Support

The previous two services are sufficient to write an object storage system, but they don't necessarily provide replication. You could add immediate replication; an upload of a file could involve writing that data to N blob-servers, but in a perfect world servers don't crash, so why not replicate in the background? You save time if you only save uploaded-content to one blob-server.

Replication can be implemented purely in terms of the blob-servers:

  • For each blob server, get the list of objects stored on it.
  • Look for that object on each of the other servers. If it is found on N of them we're good.
  • If there are fewer copies than we like, then download the data, and upload to another server.
  • Repeat until each object is stored on sufficient number of blob-servers.

 

My code is reliable, the implementation is almost painfully simple, and the only difference in my design is that rather than having an API-server which allows both "uploads" and "downloads" I split it into two - that means you can leave your "download" server open to the world, so that it can be useful, and your upload-server can be firewalled to only allow a few hosts to access it.

The code is perl-based, because Perl is good, and available here on github:

TODO: Rewrite the thing in #golang to be cool.

May 18, 2016 06:49 PM

Debian Bits

Imagination accelerates Debian development for 64-bit MIPS CPUs

Imagination Technologies recently donated several high-performance SDNA-7130 appliances to the Debian Project for the development and maintenance of the MIPS ports.

The SDNA-7130 (Software Defined Network Appliance) platforms are developed by Rhino Labs, a leading provider of high-performance data security, networking, and data infrastructure solutions.

With these new devices, the Debian project will have access to a wide range of 32- and 64-bit MIPS-based platforms.

Debian MIPS ports are also possible thanks to donations from the aql hosting service provider, the Eaton remote controlled ePDU, and many other individual members of the Debian community.

The Debian project would like to thank Imagination, Rhino Labs and aql for this coordinated donation.

More details about GNU/Linux for MIPS CPUs can be found in the related press release at Imagination and their community site about MIPS.

by Laura Arjona Reina at May 18, 2016 07:30 AM

May 16, 2016

Debian Bits

New Debian Developers and Maintainers (March and April 2016)

The following contributors got their Debian Developer accounts in the last two months:

  • Sven Bartscher (kritzefitz)
  • Harlan Lieberman-Berg (hlieberman)

Congratulations!

by Ana Guerrero Lopez at May 16, 2016 10:10 PM

May 15, 2016

Debian Bits

What does it mean that ZFS is included in Debian?

Petter Reinholdtsen recently blogged about ZFS availability in Debian. Many people have worked hard on getting ZFS support available in Debian and we would like to thank everyone involved in getting to this point and explain what ZFS in Debian means.

The landing of ZFS in the Debian archive was blocked for years due to licensing problems. Finally, the inclusion of ZFS was announced slightly more than a year ago, on April 2015 by the DPL at the time, Lucas Nussbaum who wrote "We received legal advice from Software Freedom Law Center about the inclusion of libdvdcss and ZFS in Debian, which should unblock the situation in both cases and enable us to ship them in Debian soon.". In January this year, the following DPL, Neil McGovern blogged with a lot of more details about the legal situation behind this and summarized it as "TLDR: It’s going in contrib, as a source only dkms module."

ZFS is not available exactly in Debian, since Debian is only what's included in the "main" section archive. What people really meant here is that ZFS code is now in included in "contrib" and it's available for users using DKMS.

Many people also mixed this with Ubuntu now including ZFS. However, Debian and Ubuntu are not doing the same, Ubuntu is shipping directly pre-built kernel modules, something that is considered to be a GPL violation. As the Software Freedom Conservancy wrote "while licensed under an acceptable license for Debian's Free Software Guidelines, also has a default use that can cause licensing problems for downstream Debian users".

by Ana Guerrero Lopez at May 15, 2016 08:55 PM

May 06, 2016

Andy Smith

Using a TOTP app for multi-factor SSH auth

I’ve been playing around with enabling multi-factor authentication (MFA) on web services and went with TOTP. It’s pretty simple to implement in Perl, and there are plenty of apps for it including Google Authenticator, 1Password and others.

I also wanted to use the same multi-factor auth for SSH logins. Happily, from Debian jessie onwards libpam-google-authenticator is packaged. To enable it for SSH you would just add the following:

auth required pam_google_authenticator.so

to /etc/pam.d/sshd (put it just after @include common-auth).

and ensure that:

ChallengeResponseAuthentication yes

is in /etc/ssh/sshd_config.

Not all my users will have MFA enabled though, so to skip prompting for these I use:

auth required pam_google_authenticator.so nullok

Finally, I only wanted users in a particular Unix group to be prompted for an MFA token so (assuming that group was totp) that would be:

auth [success=1 default=ignore] pam_succeed_if.so quiet user notingroup totp
auth required pam_google_authenticator.so nullok

If the pam_succeed_if conditions are met then the next line is skipped, so that causes pam_google_authenticator to be skipped for users not in the group totp.

Each user will require a TOTP secret key generating and storing. If you’re only setting this up for SSH then you can use the google-authenticator binary from the libpam-google-authenticator package. This asks you some simple questions and then populates the file $HOME/.google_authenticator with the key and some configuration options. That looks like:

T6Z2KSDCG7CEWPD6EPA6BICBFD4KYKCSGO2JEQVII7ZJNCXECRZPJ4GJHD3CWC43FZIKQUSV5LR2LFFP
" RATE_LIMIT 3 30 1462548404
" DISALLOW_REUSE 48751610
" TOTP_AUTH
11494760
25488108
33980423
43620625
84061586

The first line is the secret key; the five numbers are emergency codes that will always work (once each) if locked out.

If generating keys elsewhere then you can just populate this file yourself. If the file isn’t present then that’s when “nullok” applies; without “nullok” authentication would fail.

Note that despite the repeated mentions of “google” here, this is not a Google-specific service and no data is sent to Google. Google are the authors of the open source Google Authenticator mobile app and the libpam-google-authenticator PAM module, but (as evidenced by the Perl example) this is an open standard and client and server sides can be implemented in any language.

So that is how you can make a web service and an SSH service use the same TOTP multi-factor authentication.

by Andy at May 06, 2016 04:34 PM