Keystone. Synchronize users from AD to OpenLDAP [python script]

With OpenStack i need to sync users from corporate Microsoft Active Directory to OpenLDAP managed by Keystone.

As input i have list of user DNs. Script syncs only particular user parameters, such as

• objectclass
• mail
• cn
• givenName
• sn
• uid
• userPassword
• carLicense

These parameters a minimum enough for Keystone.

Usage:
1) Make sure you have "python-ldap" installed
2) Prepare "users" file with list of DNs in source AD

CN=user1,OU=USERS,DC=EXAMPLE,DC=com

CN=user2,OU=OTHER,OU=USERS,DC=EXAMPLE,DC=com




3) Start script below

./sync_users.py --users_file users \

                --src_ldap_uri ldap://192.168.0.11:389 \

                --src_ldap_user 'DOMAIN\administrator' \

                --src_ldap_pass 'xxx' \

                --dst_ldap_uri 'ldap://10.0.0.11' \

                --dst_ldap_user 'cn=admin,dc=domain,dc=tld' \

                --dst_ldap_pass xxx \

                --dst_ldap_tree 'ou=Users,dc=domain,dc=tld'

#!/usr/bin/python

import ldap

import ldap.modlist as modlist

import argparse

 

 

parser = argparse.ArgumentParser()

parser.add_argument('--users_file', default='users')

parser.add_argument('--src_ldap_uri', required=True)

parser.add_argument('--src_ldap_user', required=True)

parser.add_argument('--src_ldap_pass', required=True)

parser.add_argument('--dst_ldap_uri', required=True)

parser.add_argument('--dst_ldap_user', required=True)

parser.add_argument('--dst_ldap_pass', required=True)

parser.add_argument('--dst_ldap_tree', required=True)

args = parser.parse_args()

 

 

USERS_FILE = args.users_file

SRC_LDAP_URI = args.src_ldap_uri

SRC_LDAP_USER = args.src_ldap_user

SRC_LDAP_PASS = args.src_ldap_pass

DST_LDAP_URI = args.dst_ldap_uri

DST_LDAP_USER = args.dst_ldap_user

DST_LDAP_PASS = args.dst_ldap_pass

DST_LDAP_USER_TREE = args.dst_ldap_tree

 

 

def prepare_attrs(src_attrs):

    attrs = {}

    attrs['objectclass'] = ['top','inetOrgPerson','organizationalPerson','person']

    attrs['mail'] = src_attrs["mail"]

    attrs['cn'] =  src_attrs["cn"]

    attrs['givenName'] = src_attrs["givenName"]

    attrs['sn'] = src_attrs["sAMAccountName"]

    attrs['uid'] = src_attrs["sAMAccountName"]

    attrs['userPassword'] = '{SASL}%s' % src_attrs["mail"]

    attrs['carLicense'] = 'TRUE'

    return attrs

 

src_ldap = ldap.initialize(SRC_LDAP_URI)

src_ldap.simple_bind_s(SRC_LDAP_USER, SRC_LDAP_PASS)

 

 

dst_ldap = ldap.initialize(DST_LDAP_URI)

dst_ldap.simple_bind_s(DST_LDAP_USER, DST_LDAP_PASS)

 

 

with open(USERS_FILE) as f:

    users_to_sync = f.read().splitlines()

 

 

for user_dn in users_to_sync:

    user_cn = user_dn.split(',')[0]

    user_tree = user_dn.split(',', 1)[1]

    user = src_ldap.search_s(user_tree, ldap.SCOPE_ONELEVEL, user_cn)

    if user:

        dn = "cn=%s,%s" % (user_cn, DST_LDAP_USER_TREE)

        ldif = modlist.addModlist(prepare_attrs(user[0][1]))

        try:

            dst_ldap.add_s(dn,ldif)

            print "Synced user %s" % user[0][0]

        except ldap.ALREADY_EXISTS:

            print "User %s already exists" % user[0][0]

 

 

src_ldap.unbind_s()

dst_ldap.unbind_s()

ssh-agent via systemd

Main issue if you're not using desktop environment, how and when to start "ssh-agent" to store your ssh identity info. In case of full DE you have special daemon, like "gnome-keyring".

On the other hand there are a lot of custom hacks like "how to use ssh-agent with screen" or "using with xinitrc", etc.

Finally i've found that using ssh-agent with systemd is pretty useful. Example from arch wiki is not working for me (i assume for others also). I'm not using separate systemd socket and use additional SuccessExitStatus=2 parameter.

1) Create "~/.config/systemd/user/ssh-agent.service"

mkdir -p ~/.config/systemd/user
cat << EOF > ~/.config/systemd/user/ssh-agent.service
[Unit]
Description=ssh-agent

[Service]
ExecStart=/usr/bin/ssh-agent -d -a %t/ssh_auth_sock
SuccessExitStatus=2

[Install]
WantedBy=default.target
EOF

2) Start and enable ssh-agent service

systemctl --user daemon-reload
systemctl --user enable ssh-agent
systemctl --user start ssh-agent

3) Check that it works properly

ssh-add -l

OpenStack Horizon good catch

TL;DR

Wanna share my joy, i always want to finish _debugging_ with awesome oneline fix...this is it.

We'r doing multi-regional OpenStack across at least 7 datacenters. PKI tokens are really good for such kind of deployments.

- No token storage required. (except non-critical token hashes)
- Token verification without keystone

Unfortunately cons of this approach that token are pretty big ~8Kbytes. At least full catalog packed into token. Finally we'r using PKIZ (compressed PKI) to decrease the size (Backported to Icehouse from Juno).

All works like a charm...before today after adding one more region. We've noticed that Horizon stopped to show "Project" tab nor any additional tenants for user. From CLI all works perfectly.

Error:  Request attribute token must be less than or equal to 8192.

Finally found that we'r affected by this bug. Horizon while getting project list tries to send _full_ token instead of id (md5 hash) that's why token not fits into 8K.

The simplest workaround (not the solution) was to increase keystone "max_token_size". My challenge was to not finish on this easy workaround and continue to dig deeper.

P.S.: My uncompressed token was 8.5K (88 endpoints)
P.S.S.: Large OpenStack community has it's benefits: in 80% cases someone already plunged into your issue and filled a bug .

Updating BIOS with USB stick on Thinkpad (Linux)

Basically Lenovo supports two ways of upgrading BIOS: Windows upgrade utility or burning ISO to CD.
I don't have nor Windows nor CD-ROM on my laptop.

Idea is to prepare bootable USB stick from the ISO.

1) Get the BIOS Update Bootable CD from http://support.lenovo.com (In my case for T530)
2) Convert ISO to img. This is most important step because. Based on manual from thinkwiki.org i'd used
"geteltorito" script for convertation. This script supports special harddisk emulation (BootMediaType=4) ISO format.

wget http://userpages.uni-koblenz.de/~krienke/ftp/noarch/geteltorito/geteltorito
chmod +x geteltorito
./geteltorito -o bios.img ~/Downloads/g4uj25us.iso

3) dd image onto USB stick. ( CAREFUL: Use real path to USB instead of /dev/sdX )

sudo dd if=bios.img of=/dev/sdX bs=512K

4) Boot from USB and follow the instructions. Also please read official readme before doing any flashing. (i.e. for T530)

NOTES:
- Make sure USB booting enabled in BIOS. "Config --> USB --> USB UEFI BIOS support" should be "Enabled"
- After updating firmware i've plunged into issue with not workable "brightness control" keys. Solution: add kernel paramenter acpi_osi="!Windows 2012"

Webex on Arch (64bit)

The main aim of this post just to help with installing webex on 64bit Arch by this method. Unfortunately package is outdated. Latest [Nov 12] version of archive could be found here.

So to build from sources:

wget https://www.dropbox.com/s/jsrbs07t5545x03/firefox32-20141112-1.src.tar.gz?dl=1 -O firefox32-20140714-1.src.tar.gz
tar xf firefox32-20140714-1.src.tar.gz && cd firefox32
sudo makepkg -s
sudo pacman -U firefox32-20140714-1-x86_64.pkg.tar.xz

To start 32-bit firefox:

firefox32

AAAhhh! Recovering after "dd"!!!

dd if=/dev/zero of=/dev/sda bs=1M

Yeeeah, i did it on my laptop (F**n copy-pasting):

Ctrl+C after 1sec and ~700MB zeroed. Perfect evening!!!

But system is still alive. So no problem, let's start:

#0 I'm lucky, because 700MB equal "sda1 (/boot) + part of sda2 (old Ubuntu partition, i don't need it). My root (sda5 with Arch) not affected. I need to recover partition table itself and "/boot".

#1 Old partition table still in memory
cat /proc/partitions                                                                                                                                                              [23:32:52]
major minor  #blocks  name

 179        0    1966080 mmcblk0
 179        1    1965952 mmcblk0p1
   8        0  488386584 sda
   8        1     194560 sda1
   8        2  195221504 sda2
   8        3     976896 sda3
   8        4          1 sda4
   8        5  104856576 sda5
   8        6  187133976 sda6
 254        0   36700160 dm-0
 254        1   31457280 dm-1
 254        2   31457280 dm-2
 254        3   20971520 dm-3

#2 Recreating same partition table with fdisk
fdisk /dev/sda

"o"   create a new empty DOS partition table
"n"   add a new partition. For Last sector use +<BLOCKs from /proc/partitions>

After double (or more) checking:
"w"   write table to disk and exit

#3 recovering /boot
unmount /boot
mkfs.ext2 /dev/sda1
mount /boot
WARN: Special for Arch only (https://wiki.archlinux.org/index.php/syslinux)
pacman -U /var/cache/pacman/pkg/linux-3.13.8-1-x86_64.pkg.tar.xz
pacman -U /var/cache/pacman/pkg/syslinux-6.02-8-x86_64.pkg.tar.xz
syslinux-install_update -i -a -m
vim /boot/syslinux/syslinux.cfg <-- Set right root partition to sda5

That's it. And KISS for everyone.

KVM storage caching modes perfomance comparison

In my everyday work i need perfomance rich KVM VMs on my workstation. I already use virtio drivers for network and storage. But what about tune it more?

First, I've moved my VMs from qcow images to raw LVM volumes.

Next, I've switced disk cache mode to 'unsafe' as most fastest. I don't care about data integrity if smth goes wrong.

But decided to compare different modes. Detailed description about cache modes could be found here.

Cache modes differ mostly by used or not "page cache" of host operating sytem and used or not "write cache" of host hardware disk (See table 1).

Table 1

	Disk Write Cache (Host)	Page Cache (Host)
writethrough	×	✔
none	✔	×
writeback	✔	✔
unsafe	Used, but ignores transfer operations	Used, but ignores transfer operations

I've done disk perfomance testing with different cache modes on raw LVM volume. Used bonnie++ benchmark. 5 tests for every cache mode. Full results here.

Table 2

	Write (sequental block), MBs	Read (sequental block), MBs
writethrough	35	2337 (page cache)
none	49	103
writeback	45	2381 (page cache)
unsafe	54	2257 (page cache)

Most fastest mode is "unsafe". But do not use "unsafe" mode if you need data integrity.
Optimal mode is "writeback", it has good write perfomance and used page cache, that increases read perfomance a lot.
Safest mode is writethrough(by default in libvirt), but slowest.

Resume:
unsafe cache mode gave me +57% disk perfomance.

Specs:
Disk: Seagate Barracuda 500G (ST500DM002-1BC142)
MB: ASUS P8H67-M EVO
CPU: Intel(R) Core(TM) i5-2500K CPU @ 3.30GHz
OS: ARCH linux 3.11.6-1-ARCH x86_64 (libvirt 1.1.4-1, qemu 1.6.1-2)