The first question we asked ourselves was: What exactly do we want to find out? As we worked a lot on optimizing the LDAP resolver, we first wanted to know if privacyIDEA 2.19 by itself processes authentication requests faster than privacyIDEA 2.18.1. The next question was whether performance can be further improved by enabling the new user cache feature of privacyIDEA 2.19. Here, we wanted to differentiate between a worst-case scenario with an empty cache and a best-case scenario with an already-populated cache. For each scenario, we wanted to get an idea of the time that privacyIDEA needs to handle an incoming authentication request. Having cleared our objective, the next step was creating a suitable lab environment that resembles the real world as closely as possible to run our benchmarks in.

Our lab environment

We created a lab environment as follows. First, we set up an Univention Corporate Server and added 1000 users to its directory, which we simply called user000 to user999, as well as a privacyIDEA service account. In the same network, we prepared two Ubuntu 16.04 virtual machines and installed privacyIDEA 2.18.1 on the first and privacyIDEA 2.19-dev5 on the second machine. On both instances, we added a realm with a LDAP resolver connecting to the Univention Corporate Server via LDAPS.

In order to keep our environment as close to the real world as possible, we would need to enroll HOTP or TOTP tokens for our 1000 users. However, we ultimately decided against it. For one, we suspected that the time spent calculating and checking the next OTP value on the server is relatively small in comparison to the time spent communicating with the LDAP server. Enrolling real OTP tokens would also require us to keep track of secrets and counter values on our benchmark client, which would complicate our setup a lot. To keep things simple, we instead decided to enroll one simple password (SPASS) token for each user.

Our benchmarking approach

Now, we had set up two privacyIDEA instances with 1000 tokens. The next question was: How exactly do we now measure the performance of one authentication request? We decided to settle on the following approach: One benchmark consists of 2000 successful authentication requests, performed one after another for the users user000 to user999. This means that each user is authenticated twice during one benchmark. For each authentication request, we measured the time from sending the request until receiving the response using a simple benchmarking script in Python based on python-requests. We copied the script to the virtual machines and performed all authentication requests against https://localhost in order to exclude the network delay from our measurements. Running the script then produces 2000 measurements of response time, of which we computed the median response time.

We decided to measure the response times for the following scenarios:

• Scenario #1: Authentication against privacyIDEA 2.18.1
• Scenario #2: Authentication against privacyIDEA 2.19, with the user cache feature disabled
• Scenario #3: Authentication against privacyIDEA 2.19, with the user cache enabled and initially empty
• Scenario #4: Authentication against privacyIDEA 2.19, with an already-populated user cache. This means
  that the user cache contains valid 1000 entries, one for each user from user000 to
  user999.

The scenarios 2, 3 and 4 were carried out on the privacyIDEA 2.19 machine. For scenarios 3 and 4, we enabled the user cache with a timeout of one day (corresponding to 86400 seconds).

Our results

Now, we had everything in place to start our benchmarks! In total, running the benchmark for all four scenarios took roughly one hour and we obtained the following results.

Interesting! According to our measurements, an update to privacyIDEA 2.19 alone seems to reduce the median response time by roughly 57% (Scenario #2), even without enabling the user cache. This speedup can probably be attributed to some performance improvements in the LDAP resolver (see issues 655 and 664).

Furthermore, if the user cache is enabled and fully populated (Scenario #4), the median response time is reduced by another 33%. In comparison to privacyIDEA 2.18.1, this corresponds to a reduction by 71%. Of course, this models a best-case scenario in which the LDAP server does not need to be queried any more at all. This may not be the case in the real world, e.g. if an otppin=userstore policy is enabled.

Scenario #3 is quite interesting, as the user cache is initially empty and is subsequently populated during the first 1000 authentication requests. For the second round of 1000 authentications, privacyIDEA can rely on the user cache instead of querying the LDAP server. We can also observe this if we plot our measurements:

The horizontal axis denotes our measurements and the vertical axis gives the median response time in milliseconds. We can observe that the median response time drops dramatically after the thousandth measurement, from which we can conclude that the user cache does have a measurable positive impact on the performance of authentication requests.

Of course, our benchmark is not perfect and leaves room for improvement due to multiple reasons. Firstly, the measurements also include the round-trip time between LDAP server and privacyIDEA instance, which significantly depends on the network setup. Secondly, we have only enrolled SPASS tokens and no real OTP tokens. Thirdly, we have not performed any concurrent requests and cannot, for example, say anything about the maximum number of authentication requests per second. Finally, two authentication requests by the same user are several minutes apart. If the same user sends two authentication requests during the timespan configured by the cache timeout option of the LDAP resolver (which defaults to 2 minutes), privacyIDEA queries an in-memory cache, which may be even faster than the query to the local database performed by the user cache.

However, we believe that our benchmark shows that privacyIDEA 2.19 improves the performance of authentication requests quite significantly even without the user cache. Additionally, enabling the user cache may bring significant performance improvements in case a large number of users are expected to send authentication requests over a large timespan. Finally, we noticed that the LDAP connection in our test setup is quite fast (a LDAP search takes just unter 30 milliseconds), so the user cache may provide an even better speedup in case of slower LDAP servers or connections. You are welcome to try it out for yourself! If you have any further questions, pleask ask them on our community site.

Today we released privacyIDEA 2.15. In this release privacyIDEA command line client supports the initialization and enrollment of the Nitrokey. The Nitrokey is an open USB devices that acts as authentication device and password safe. It can hold your PGP keys but also provides several OTP slots. privacyIDEA now can initialize these OTP slots, so that you can use your own key material and use the Nitrokey as an open and trusted authenticator. This way you get the maximum trust and transparency by running open source software, using open and standardized algorithms and open hardware.

Arbitrary User Attributes and Client Overview

With privacyIDEA 2.15 the administrator now can edit arbitrary user attributes. These user attributes can be included in the authentication response and the new privacyIDEA FreeRADIUS plugin can map these user attributes to any RADIUS response attribute.

In the Web UI the administrator now also gets an overview of all authenticating clients. This may help him to keep track of the connected applications.

Download

You can download privacyIDEA via github, the python package index or the Ubuntu Launchpad repository. privacyIDEA is also available as privacyIDEA Enterprise Edition from NetKnights providing additional downloads for CentOS or the Univention Corporate Server.

Changelog

• Client Overview. Display the type of the requesting   authenticating clients (#489)
• Support for NitroKey OTP mode (admin client)

Enhancements

• You can edit arbitrary user attributes in privacyIDEA.
• Such user attributes can be mapped to any RADIUS attribute.
• Performance enhancements using Caching singletons for Config, Realm, Resolver and Policies
• Allow configuration of the registration email text (#494)
• Return SAML attributes only in case of successful authentication (#500)
• Policy “reset_all_user_tokens” allow to reset all  failcounters on successful authentication (#471)
• Client rewrite mapping also checks for X-Forwarded-For (#395, #495)

Fixes

• Fixing RemoteUser fails to display WebUI (#499)
• String comparison in HOSTS resolver (#484)

Starting with privacyIDEA 1.2 a debian package for Ubuntu 14.04 is availble.

Please note, that you will not be able to install it on 12.04, since there would be missing requirements. privacyIDEA depends on the following packages:

 python-setuptools python-pylons python-qrcode python-netaddr python-ldap python-pyrad python-yaml python-configobj python-repoze.who python-httplib2 python-crypto python-docutils python-repoze.who-plugins

There are two PPA repositories available on launchpad: privacyidea/privacyidea-dev and privacyidea/privacyidea. The -dev repository is for development releases and testing. The privacyidea/privacyidea repo should contain stable releases.

Add the repository to your system and install privacyIDEA

To add the repository to your system run the following command:

add-apt-repository ppa:privacyidea/privacyidea

Fetch information on new content:

apt-get update

Install it:

apt-get install privacyidea

The package creates an SQLite database at /var/lib/privacyidea/token.sqlite. Of course you can use any other database backend. But using sqlite gets you up and running quickly.

Moreover the package contains a start-script /etc/init.d/privacyidea, that is running privacyidea in python-paster, a simple, lightweight webserver.

To start privacyidea run:

service privacyidea start

privacyIDEA is now listening on port 5001.

Create your admin account

Finally you need to create a first admin account to log in to the management interface:

privacyidea-create-pwidresolver-user -u admin -p test -i 1000 >> /etc/privacyidea/admin-users

Instead of using the weak password test, you should make up a cooler one.

Now you can login at https://yourserver:5001/ with the username “admin@admin” and the password you created.

Some performance data

The paster is a small webserver. The SQLite is not a state-of-the-art database.

So I always would recommend running privacyIDEA with Apache. This is describe in this post.

I was wondering what this paster and sqlite could do, So I created a realm containing the local users from /etc/passwd and assigned a simple pass token to one of these users.

Now I was able to issue an authentication request by calling the API like this:

 https://myserver:5001/validate/check?user=man&pass=test

Now I used ApacheBench to call this URL:

% ab -n 1000 -c 10 -s 5 https://172.16.200.139:5001/validate/check?user=man\&pass=test
This is ApacheBench, Version 2.3 <$Revision: 1528965 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 172.16.200.139 (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests


Server Software:        PasteWSGIServer/0.5
Server Hostname:        172.16.200.139
Server Port:            5001
SSL/TLS Protocol:       TLSv1.2,AES256-GCM-SHA384,2048,256

Document Path:          /validate/check?user=man&pass=test
Document Length:        135 bytes

Concurrency Level:      10
Time taken for tests:   41.964 seconds
Complete requests:      1000
Failed requests:        0
Total transferred:      373000 bytes
HTML transferred:       135000 bytes
Requests per second:    23.83 [#/sec] (mean)
Time per request:       419.643 [ms] (mean)
Time per request:       41.964 [ms] (mean, across all concurrent requests)
Transfer rate:          8.68 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        2   11  14.4      9     218
Processing:   121  407 270.7    332    3340
Waiting:      120  406 270.8    332    3340
Total:        126  418 272.0    342    3350

Percentage of the requests served within a certain time (ms)
  50%    342
  66%    401
  75%    456
  80%    499
  90%    678
  95%    948
  98%   1318
  99%   1632
 100%   3350 (longest request)

This was done on a Virtual Machine running in VirtualBox with 2 processors and 2GB of RAM. The host machine is an Intel i7-4702MQ CPU @ 2.20GHz.

24 Authentications per seconds, no failed requests look rather good to me.

So why not give it a try?