We are pleased to announce the release of privacyIDEA 3.0.1 today. In this release, we did not implement any new features, but instead focused on stability.

Our last major release privacyIDEA 3.0 came with several big changes. Most notably, it introduced a new PUSH token class. With this token class, a login request sends a push notification on the user’s smartphone that can be accepted or declined. You can read more about the PUSH token here. Thanks to a lot of feedback from the community, we were able to fix several issues with our initial implementation:

• Add logic checking to setup of PUSH token (#1592)
• Remove double enrollment notification of PUSH token in WebUI (#1598)
• Fix to allow spaces in Firebase configuration (#1599)
• Add support for iOS Firebase configuration (#1608)
• Fix to allow PUSH token enrollment, even with Label-policy (#1589)
• Fix to mark PUSH token challenge answered in the database (#1584)

privacyIDEA 3.0.1 also comes with usability improvements on the frontend side:

• Beautify the vertical alignment in the Web UI top menu (#1559)
• Fix user cache configuration read – defaults to 0 (#1596)
• Remove links in audit log for normal users (#1497)
• Fix placeholder in realm dropdown in login dialog (#1498)
• Allow the usage if “browserLanguage” in custom templates (#1620)
• Open all accordions when searching for policy action (#1558)
• Fix to hide support links also in menu (#1626)

Finally, we corrected some inconsistencies and bugs in the backend, and improved on the compatibility with Python 3:

• Fix the validity period of the registration token (#1587)
• Check UI rights for user resolvers (#1496)
• Fix enckey creation in Python 3 (#1594)

privacyIDEA 3.0.1 is available via the Python package index, and can be installed on Ubuntu 16.04 and 18.04 using our community repositories. Detailed installation instructions can be found in our documentation. Before upgrading your existing installation, be sure to read our update notes.

If you have any questions or suggestions, you are welcome to join our community forum. In case you require enterprise support, we refer to the privacyIDEA Enterprise Edition offered by NetKnights GmbH.

The first question we asked ourselves was: What exactly do we want to find out? As we worked a lot on optimizing the LDAP resolver, we first wanted to know if privacyIDEA 2.19 by itself processes authentication requests faster than privacyIDEA 2.18.1. The next question was whether performance can be further improved by enabling the new user cache feature of privacyIDEA 2.19. Here, we wanted to differentiate between a worst-case scenario with an empty cache and a best-case scenario with an already-populated cache. For each scenario, we wanted to get an idea of the time that privacyIDEA needs to handle an incoming authentication request. Having cleared our objective, the next step was creating a suitable lab environment that resembles the real world as closely as possible to run our benchmarks in.

Our lab environment

We created a lab environment as follows. First, we set up an Univention Corporate Server and added 1000 users to its directory, which we simply called user000 to user999, as well as a privacyIDEA service account. In the same network, we prepared two Ubuntu 16.04 virtual machines and installed privacyIDEA 2.18.1 on the first and privacyIDEA 2.19-dev5 on the second machine. On both instances, we added a realm with a LDAP resolver connecting to the Univention Corporate Server via LDAPS.

In order to keep our environment as close to the real world as possible, we would need to enroll HOTP or TOTP tokens for our 1000 users. However, we ultimately decided against it. For one, we suspected that the time spent calculating and checking the next OTP value on the server is relatively small in comparison to the time spent communicating with the LDAP server. Enrolling real OTP tokens would also require us to keep track of secrets and counter values on our benchmark client, which would complicate our setup a lot. To keep things simple, we instead decided to enroll one simple password (SPASS) token for each user.

Our benchmarking approach

Now, we had set up two privacyIDEA instances with 1000 tokens. The next question was: How exactly do we now measure the performance of one authentication request? We decided to settle on the following approach: One benchmark consists of 2000 successful authentication requests, performed one after another for the users user000 to user999. This means that each user is authenticated twice during one benchmark. For each authentication request, we measured the time from sending the request until receiving the response using a simple benchmarking script in Python based on python-requests. We copied the script to the virtual machines and performed all authentication requests against https://localhost in order to exclude the network delay from our measurements. Running the script then produces 2000 measurements of response time, of which we computed the median response time.

We decided to measure the response times for the following scenarios:

• Scenario #1: Authentication against privacyIDEA 2.18.1
• Scenario #2: Authentication against privacyIDEA 2.19, with the user cache feature disabled
• Scenario #3: Authentication against privacyIDEA 2.19, with the user cache enabled and initially empty
• Scenario #4: Authentication against privacyIDEA 2.19, with an already-populated user cache. This means
  that the user cache contains valid 1000 entries, one for each user from user000 to
  user999.

The scenarios 2, 3 and 4 were carried out on the privacyIDEA 2.19 machine. For scenarios 3 and 4, we enabled the user cache with a timeout of one day (corresponding to 86400 seconds).

Our results

Now, we had everything in place to start our benchmarks! In total, running the benchmark for all four scenarios took roughly one hour and we obtained the following results.

Interesting! According to our measurements, an update to privacyIDEA 2.19 alone seems to reduce the median response time by roughly 57% (Scenario #2), even without enabling the user cache. This speedup can probably be attributed to some performance improvements in the LDAP resolver (see issues 655 and 664).

Furthermore, if the user cache is enabled and fully populated (Scenario #4), the median response time is reduced by another 33%. In comparison to privacyIDEA 2.18.1, this corresponds to a reduction by 71%. Of course, this models a best-case scenario in which the LDAP server does not need to be queried any more at all. This may not be the case in the real world, e.g. if an otppin=userstore policy is enabled.

Scenario #3 is quite interesting, as the user cache is initially empty and is subsequently populated during the first 1000 authentication requests. For the second round of 1000 authentications, privacyIDEA can rely on the user cache instead of querying the LDAP server. We can also observe this if we plot our measurements:

The horizontal axis denotes our measurements and the vertical axis gives the median response time in milliseconds. We can observe that the median response time drops dramatically after the thousandth measurement, from which we can conclude that the user cache does have a measurable positive impact on the performance of authentication requests.

Of course, our benchmark is not perfect and leaves room for improvement due to multiple reasons. Firstly, the measurements also include the round-trip time between LDAP server and privacyIDEA instance, which significantly depends on the network setup. Secondly, we have only enrolled SPASS tokens and no real OTP tokens. Thirdly, we have not performed any concurrent requests and cannot, for example, say anything about the maximum number of authentication requests per second. Finally, two authentication requests by the same user are several minutes apart. If the same user sends two authentication requests during the timespan configured by the cache timeout option of the LDAP resolver (which defaults to 2 minutes), privacyIDEA queries an in-memory cache, which may be even faster than the query to the local database performed by the user cache.

However, we believe that our benchmark shows that privacyIDEA 2.19 improves the performance of authentication requests quite significantly even without the user cache. Additionally, enabling the user cache may bring significant performance improvements in case a large number of users are expected to send authentication requests over a large timespan. Finally, we noticed that the LDAP connection in our test setup is quite fast (a LDAP search takes just unter 30 milliseconds), so the user cache may provide an even better speedup in case of slower LDAP servers or connections. You are welcome to try it out for yourself! If you have any further questions, pleask ask them on our community site.