We will use the privacyIDEA logging facility to let the server not only store its system logs and audit locally but also feed them to a remote Logstash server. We will also show how to use the new Logging module of the Event Handler, introduced in privacyIDEA 3.3, to customize the logged information. The usual path of information is displayed in the following picture.

Setup the Base System

As the installation Logstash, Elasticsearch and Kibana is documented at the vendors website, we will not go into detail here. In any case, you need a java runtime environment. For Ubuntu you can use the package default-jre. Once you have the elastic stack up and running, turn towards privacyIDEA.

The installation of privacyIDEA is documented at privacyidea.readthedocs.io. For a quick start, there is a community package repository for Ubuntu 18.04 LTS available. Install the privacyIDEA server and become a little familiar to the WebUI, which is the primary management interface.

The base configuration of privacyIDEA is set in the configuration file pi.cfg and the dedicated logging configuration file logging.cfg or logging.yml. In the default Ubuntu 18.04 package installation, those are located in /etc/privacyidea/. To be able to view the audit logs in the WebUI and send them at the same time to the python logger, the ContainerAudit module is used.

# /etc/privacyidea/pi.cfg
PI_AUDIT_MODULE = 'privacyidea.lib.auditmodules.containeraudit'
PI_AUDIT_CONTAINER_WRITE = ['privacyidea.lib.auditmodules.sqlaudit','privacyidea.lib.auditmodules.loggeraudit']
PI_AUDIT_CONTAINER_READ = 'privacyidea.lib.auditmodules.sqlaudit'
PI_AUDIT_LOGGER_QUALNAME = 'pi-audit'
PI_LOGCONFIG = '/etc/privacyidea/logging.cfg'

Note that we use a custom audit logger name “pi-audit” in the above configuration. See the documentation of the Logger Audit.

Send privacyIDEA logs to Logstash

The logging module privacyidea.lib.auditmodules.loggeraudit sends the audit messages to the python logging system and makes it available to the configuration by logging.yml. To send both the privacyIDEA server logs and the audit log to Logstash, the module python-logstash-async comes in handy. It can be installed through pip by

~$ pip install python-logstash-async

The module can be used in a logging.cfg or logging.yml in YAML and INI format respectively. Minimal examples for the configuration of the logstash-async module are found on Github Gist. A more detailed YAML configuration file is also available, which provides a good basis for this test case.

Restart privacyidea for the changes to have effect. If you used the extended configuration from gist, you should now see the audit log in /var/log/privacyidea/audit.log.

Receive privacyIDEA logs with Logstash

On the other end, Logstash is configured to listen on port 5959 and to forward the logs to Elasticsearch using different indices for the qualnames pi-audit, pi-eventlog and all the rest (privacyidea.*).

# /etc/logstash/conf.d/privacyidea_elasticsearch.conf
# privacyIDEA input is logged by the python-logstash-async module
input {
   tcp {
      port => 5959
      codec => json
      tags => ["privacyidea"]
   }
}
# filter adds metadata field according to logger to
# separate the privacyIDEA audit log from the rest
filter {
   if [extra][logger_name] == "pi-audit" or [extra][logger_name] == "privacyidea.lib.auditmodules.loggeraudit" {
      mutate { add_field => { "[@metadata][indexPrefix]" => "pi-audit" } }
   } else if [extra][logger_name] == "pi-eventlog" {
      mutate { add_field => { "[@metadata][indexPrefix]" => "pi-eventlog" } }
   } else {
      mutate { add_field => { "[@metadata][indexPrefix]" => "privacyidea" } }
   }
}
# Logs are sent to elasticsearch using the indexPrefix
output {
   elasticsearch {
      index => "%{[@metadata][indexPrefix]}-%{+YYYY.MM.dd}"
   }
   # additional output to syslog
   stdout {
      codec => rubydebug
   }
}

Restart logstash afterwards. The output section contains an additional part for logging to stdout. On a systemd-driven system (check ~$ ps -p 1), it can be viewed by

~# journalctl -f -u logstash

Once you interact with the privacyIDEA server, you should see the incoming audit log messages in json format in the journalctl ountput on the logstash machine. The example below is the audit message for viewing the audit log in the privacyIDEA WebUI:

{
          "extra" => {
                    "logger_name" => "pi-audit",
                   "process_name" => "MainProcess",
                    "thread_name" => "MainThread",
                           "line" => 85,
         "logstash_async_version" => "1.6.4",
                           "path" => "/opt/privacyidea/privacyidea/lib/auditmodules/loggeraudit.py",
                      "func_name" => "finalize_log",
                    "interpreter" => "/opt/privacyidea/venv/bin/python",
            "interpreter_version" => "3.6.9"
     },
          "level" => "INFO",
        "program" => "/opt/privacyidea/pi-manage",
           "port" => 47962,
            "pid" => 10047,
      "logsource" => "myhost",
     "@timestamp" => 2020-03-25T15:32:42.748Z,
       "@version" => "1",
           "type" => "python-logstash",
        "message" => "{'success': True, 'serial': '', 'user': '', 'realm': '**', 'resolver': '', 'token_type': '', 'client': '127.0.0.1', 'client_user_agent': 'firefox', 'privacyidea_server': 'localhost:5000', 'action': 'GET /audit/', 'action_detail': '', 'info': '', 'administrator': 'admin', 'policies': '', 'timestamp': datetime.datetime(2020, 3, 25, 15, 32, 42, 748526)}",
           "host" => "henning-t470"
 }

Display privacyIDEA logs with Kibana

The logs received by Logstash are sent to Elasticsearch which talks to the Kibana instance. The Elasticsearch indices should appear in Kibana’s index management, available from the home screen.

The privacyIDEA indices will look like shown below.

Note: the yellow health status is due to a default index setting "index.number_of_replicas": "1". Changing it to zero will result in a green status. Under “Data Views” create a new data view with the index-pattern “privacyidea*,pi-audit-*,pi-event*”.

In the Logs view, select the privacyIDEA data view you can select the log columns (“selected fields”) to be shown. The privacyIDEA log messages are now nicely displayed.

Don’t forget to save!

Event-based logging from privacyIDEA to Logstash

New in privacyIDEA 3.3 is the Event Handler module “Logging”. With this module, custom logging messages can be bound to any event. This opens the door to a whole new world of monitoring possibilities in privacyIDEA. To demonstrate the feature, we simply log whenever a token is disabled — a silly example, of course. The Event Handler is created as shown below.

For the Logging module, only one action is available. The log level, the name of the logger and a custom message are required. The message field supports variables known from the user notification module (see documentation). Note, that it depends on the context of the RESTful API event if a certain variable is available or not.

The chosen logger name has to be added as a logger in logging.yml to send it to Logstash.

loggers:
  pi-eventlog:
    handlers:
      - logfile
      - logstash_async
    level: DEBUG

Restart privacyIDEA to apply the changes in the config file logging.yml. After triggering the event by disabling a token in privacyIDEA, Kibana shows the notification.

Of course, you can use the logging event handler for more reasonable purposes like not to send the full audit log to logstash but to single-out the important validate-check events. You can even apply some more conditions, if you like making use of the powerful condition properties of the privacyIDEA Event Handlers. This not only spares bandwidth and storage space but prevents important information to be buried by other data. The message field in privacyIDEA can be used for a custom log message with contextual information. The configurable logger name (e.g. pi-validate-check) provides an additional identifier. In the case of suspicious behavior or a security incident, all the information is there to quickly track down the threat.

Conclusion

With this demonstration of the logging facility, privacyIDEA proves again to be extraordinarily scalable. It integrates well with logging systems like Logstash and Splunk since the privacyIDEA server builds on the standard python logging library. For Logstash, this article showed the detailed steps how to integrate privacyIDEA via the loggeraudit and a small third-party python module called python-logstash-async.

In privacyIDEA 3.3 the logging capabilities have been further extended by an Event Handler module which enables to conditionally log arbitrary events to the python logging system. We showed that also these messages can easily be passed to Logstash and open a vast playground custom logging.

The solution shown here is only one possible approach. Since privacyIDEA is available as open source an licensed under the AGPL, another possibility would be of course to write your very own logger module to do whatever you want. privacyIDEA is and will be always open source and therefore it will always stay in your hands.

If you would like to have a custom logger module, but have no time to implement it yourself, you can always request a quote from Netknights, the company which drives the privacyIDEA innovation via Github. They also provide professional support for privacyIDEA, including enterprise repositories for Ubuntu and CentOS/RHEL containing the server and a number of additional components and tools.

First, we setup three machines with Ubuntu Server 18.04 and provide similar /etc/hosts files to each of them. For a proper setup, Kolab requires a fully qualified domain name. We choose kolab.netknights.it.

127.0.0.1      localhost
127.0.1.1      kolab
192.168.56.200 kolab.netknights.it    kolab
192.168.56.201 pi.netknights.it       pi
192.168.56.202 keycloak.netknights.it keycloak

To put the system into action, one would have to configure DNS and NAT properly so that the server is reachable from the internet. DNS record of type A, AAAA and MX are crucial to do this. You may read about this requirement in the Kolab docs “preparing the system“. For this proof of concept we will not use any logical volumes nor discuss firewall setups or SSL transport layer security.

After updating the initial Ubuntu Server systems, we install privacyIDEA, Keycloak and Kolab following their general installing instructions. We start with Kolab, since in this scenario, we would like to attach the ds-389 LDAP directory delivered by Kolab to our backend, i.e. Keycloak and privacyIDEA.

Kolab 16 Installation

The primary OS supported by Kolab is CentOS, which is also supported by privacyIDEA. However, there are also Kolab and privacyIDEA packages available for Ubuntu 18.04. We will use these for our setup. The install instructions can be found at docs.kolab.org. We summarize them here for convenience

# Add repositories for apt to /etc/sources/ 
~$ echo 'deb http://obs.kolabsys.com/repositories/Kolab:/16/Ubuntu_18.04/ ./ deb-src http://obs.kolabsys.com/repositories/Kolab:/16/Ubuntu_18.04/ ./' \ 
| tee /etc/apt/sources.list.d/kolab.list 
# Add signing key 
~$ wget -q -O- https://ssl.kolabsys.com/community.asc | apt-key add - 
~$ echo -e 'Package: *\nPin: origin obs.kolabsys.com\nPin-Priority: 501' \ 
| tee /etc/apt/preferences.d/kolab 
~$ apt-get update
~$ apt-get install kolab

We let the postfix SMTP service be configured as “Internet with smarthost”. Outbound email will be relayed to another (trusted) mail server, e.g. the one of your ISP. Inbound mail will reach postfix on port 25 if your DNS records are configured correctly. After installation the configuration script is called via

~$ setup-kolab

It will ask for some information and several passwords. The password for the directory manager will be used to first login, so remember it. Also passwords for a the cyrus-imapd administrator, a kolab-service user, the mysql database root user and several database passwords are needed. Note, that all of them are stored in the /etc/kolab/kolab.conf file and only very few are needed for interactive logins.

Kolab comes without a predefined admin user. Only the directory admin is defined which should not be used for user administration. So we login as “cn=Directory Manager” and define a new admin user. In our default setup, the UID is generated from the surname (check the “System” tab), so we choose kolab-admin to differentiate from pi-admin and keycloak-admin later.

On the System tab we set the predefined kolab-admin role for the user to grant him access for user management. Also set a password.

Hit submit to complete the process. Next add another ordinary user to test the second factor login later on. We call this user test-user.

privacyIDEA 3.x Installation

Now, we setup privacyIDEA. Install instructions for the most recent version can be found at readthedocs.privacyidea.io. We install the official Ubuntu packages, specifically the privacyidea-apache2 package. The installation only takes few minutes. After creating the admin user, here called pi-admin, with

~$ pi-manage admin add pi-admin -e pi-admin@localhost

we login to the UI. privacyIDEA needs to access the LDAP directory provided by Kolab, so we create an LDAP resolver and use the Kolab service account created above via “Config->Users->New ldapresolver”. The dn and password for the kolab-service account can be checked in the file /etc/kolab/kolab.conf. We use the OpenLDAP preset given by privacyIDEA but change the UID type to “dn”. The resolver test buttons help to avoid typos and to check the connection. For the test, we leave TLS to be configured later.

The resolver is added to a new realm kolab_realm at “Config->Realms->Create Realm”.

You should now be able to see the users within privacyIDEA.

We will enroll an HOTP token for test-user with the privacyIDEA App, available from Google Play Store. You may alternatively use the Google Authenticator. Install the app and proceed with the enrollment as given below. The privacyIDEA UI auto-completes the username as you type. The generated QR code must be scanned with the App to complete the enrollment.

The OTP token is now assigned to the user test-user and the privacyIDEA app on your phone should display a six-digits OTP code.

To issue trigger challenges asking for an OTP key on user login, privacyIDEA needs an authorization. Since we do not want our pi-admin password to flow through the wire all the time, we create another, unpriviledged admin user on the privacyIDEA terminal.

~$ pi-manage admin add trigger-admin -e trigger-admin@localhost

The trigger-admin needs a superuser policy to restrict the access. In “Config->Policies”, first create a default superuser policy using the “superuser” template. Add only the pi-admin to the admin field. Then add another policy without template. Name it trigger_admin, select the scope admin, add the action triggerchallenge and add the created trigger-admin to the admin field. Now, we have two admin users. pi-admin has the default superuser access and trigger-admin has only very limited access allowing to trigger the challenge.

As privacyIDEA is now up and running, we proceed with the Keycloak server, which will act as the central element in the SSO environment.

Keycloak 9 Installation

Start the Keycloak installation by downloading the Keycloak standalone server from keycloak.org. General install information is found in the “getting started” guide. A detailed guide how to integrate Keycloak with systemd on Ubuntu Server 18.04 LTS is found here. The necessary steps are summarized below.

~$ sudo apt-get update
~$ sudo apt-get install default-jre-headless
# Install Keycloak 9
~$ mkdir -p /opt/keycloak /etc/keycloak
~$ wget https://downloads.jboss.org/keycloak/9.0.0/keycloak-9.0.0.tar.gz
~$ tar -xvzf keycloak-9.0.0.tar.gz
~$ mv keycloak-9.0.0.tar.gz /opt/keycloak
# Add keycloak user
~$ groupadd keycloak
~$ useradd -r -g keycloak -d /opt/keycloak -s /sbin/nologin keycloak
~$ chown -R keycloak: /opt/keycloak
~$ chmod o+x /opt/keycloak/bin/
# Place config file
~$ cp /opt/keycloak/docs/contrib/scripts/systemd/wildfly.conf /etc/keycloak/keycloak.conf
# Setup systemd files
~$ sed 's/wildfly/keycloak/' /opt/keycloak/docs/contrib/scripts/systemd/launch.sh \
| tee /opt/keycloak/bin/launch.sh
~$ chown keycloak: /opt/keycloak/bin/launch.sh
~$ sed 's/wildfly/keycloak/g' /opt/keycloak/docs/contrib/scripts/systemd/wildfly.service \
| tee /etc/systemd/system/keycloak.service
# Enable and start the daemon
~$ systemctl daemon-reload
~$ systemctl enable keycloak
~$ systemctl start keycloak
~$ systemctl status keycloak
# Add admin user
~$ /opt/keycloak/bin/add-user-keycloak.sh -r master -u keycloak-admin -p <password> 
~$ systemctl restart keycloak

Now you should be greeted by Keycloak at http://192.168.56.202:8080. Login with your created keycloak-admin user. As Keycloak should validate the user logins it has to have access to the user store. In “User Federation”, add an LDAP provider with the following settings. The kolab-service account is used as an unpriviledged bind and again we disable TLS for the test setup.

Hit “Synchronize all users” to pull the users from LDAP to Keycloak. You may enable the periodic sync to keep the Keycloak user store up-to-date.

Important: The keycloak-admin should not be required to provide a second factor to prevent locking the configuration while testing. For this purpose, define a no2fa group in “Groups” and add the keycloak-admin to that group in “Users”.

Next, privacyIDEA is integrated with Keycloak. Following our earlier article on the integration of Django with Keycloak and privacyIDEA, we download the two files PrivacyIDEA-Provider.jar and privacyIDEA.ftl of the most recent release of the privacyIDEA keycloak-provider and install it to Keycloak.

~$ wget https://github.com/privacyidea/keycloak-provider/releases/download/v0.3/PrivacyIDEA-Provider.jar
~$ wget https://github.com/privacyidea/keycloak-provider/releases/download/v0.3/privacyIDEA.ftl
~$ cp PrivacyIDEA-Provider.jar /opt/keycloak/standalone/deployment/
~$ cp privacyIDEA.ftl /opt/keycloak/themes/base/login/

In Keycloak, the authentication is managed in so called “Authentication Flows”. Copy the default browser-based flow below and rename it to PrivacyIDEA.

Add an execution to “PrivacyIDEA Forms” and choose the installed plugin called PrivacyIDEA from the list.

Delete the unnecessary items in the flow (or set them to disabled), so that only “PrivacyIDEA Forms” and “Cookie” remain. The authentication flow should now look like this:

We set PrivacyIDEA to REQUIRED here, which means that additionally to username and password, the second factor is required for all users. We have to configure the plugin to reach our privacyIDEA server at https://192.168.56.202. We disable SSL-verification for the self-signed certificate here, which you must not do in a productive environment. Members of the no2fa group, defined above will not be asked for their second factor. For issuing the trigger challenge a service account is needed. We use the trigger-admin account created in privacyIDEA earlier.

Set the edited authentication flow as default browser flow in “Bindings”.

So privacyIDEA is now configured to challenge the second factor for every user. The last step is to enable OpenID Connect logins in roundcubemail.

Installation of the Kolab SSO plugin

For the OIDC, Kolab provides the kolab_sso plugin for Roundcubemail which is available on git.kolab.org. Clone the repository and copy the plugin to the Roundcubemail directory to install it.

~$ git clone https://git.kolab.org/diffusion/RPK/roundcubemail-plugins-kolab.git
~$ cp -r roundcubemail-plugins-kolab/plugins/kolab_sso/ /usr/share/roundcubemail/plugins/

Place the default configuration file.

~$ cp /usr/share/roundcubemail/plugins/kolab_sso/config.inc.php.dist /etc/roundcubemail/kolab_sso.inc.php

Apache should redirect host.roundcube/sso to host.roundcube/?_task=login&_action=sso, since keycloak does not support parameters in urls. It will display “Invalid parameter: redirect_uri”. Add the redirect as follows to /etc/apache2/sites-enabled/roundcubemail.conf.

RewriteEngine On
RewriteCond %{REQUEST_URI} ^/roundcubemail
RewriteRule "^sso" "/roundcubemail/?_task=login&_action=sso" [L,QSA]

We proceed on the Keycloak machine and add Roundcubemail as a new OpenID Connect client as given below.

Save the form to access the “Credentials” tab. We will soon need the generated secret again.

The kolab_sso plugin needs the certificate of the Keycloak server during the OpenID Connect authentication process. It is available from the Keycloak management console in the “Realm Settings”.

Add the key and the client secret alongside the token_uri and auth_uri to the kolab_sso.inc.php configuration file. Make sure that the public key copied from keycloak is properly formatted. The IMAP, SMTP and LDAP credentials in the top part of the file are required for accessing the mailbox, sending emails and accessing the server addressbooks. Configure them accordingly.

After successfully testing the ordinary password login with Roundcubemail at http://192.168.56.200/roundcubemail, you may disable the password login to allow only OpenID Connect by setting

 $config['kolab_sso_disable_login'] = true;

in kolab_sso.inc.php.

Test the login

We are now ready to test the OpenID Connect login at Roundcubemail. Navigate once again to http://192.168.56.200/roundcubemail to test the login. You may monitor some log files during the process.

/var/log/privacyidea/privacyidea.log
/var/log/roundcubemail/*
/opt/keycloak/standalone/log/*

Conclusion

We demonstrated the integration of privacyIDEA with Keycloak to provide a solid basis to secure your applications with a second factor in a single sign-on (SSO) environment. For maximum flexibility, the system relies on standard protocols such as SAML or OpenID Connect (OIDC). The privacyIDEA keycloak-provider is designed to perfectly fit the two components together, uniting the rich identity management capabilities of Keycloak and the powerful multi factor management of privacyIDEA.

We chose the Roundcubemail webmailer of the Kolab Collaboration Server as an example application. The kolab_sso plugin provided the necessary interface to connect via OIDC to easily enhance security by adding a second factor managed by privacyIDEA. The setup of other popular open collaboration platforms such as Tine 2.0 or Open-Xchange work similarly.

Including additional applications in this setup is very easy as long as they support at least one SSO protocol. These applications do not even to be hosted on your own servers. Nowadays, most cloud-based applications offer both, the possibility to use an external identity provider and to use OIDC. Thus, you can also use remote services with your own user base, defining access-rules to fit your needs.