Streaming data is a commodity. Thanks to all sorts of streaming data sources we can build reactive systems whereby an event occuring in one corner of the system triggers events somewhere else. Streaming data speeds up processes because businesses can react to events instead of proactively having to ask for “what’s new”. IAM is no exception. Actually, an IAM system should be a source of critical information shared with other systems because IAM events are core to everything else happening within a modern, data-driven organization.
Why? Here are some reasons:
- A sign-up triggers a welcome email and a follow-up email campaign.
- A customer modifying their profile propagates new data to a CRM or an HR system.
- A large number of login attempts triggers alerts or flag an account with suspicious activity.
- A membership change reconfigures external systems, for example, database permissions or Kafka ACLs.
- It’s a general auditing mechanism.
- Data from other systems can be actively fused directly back to IAM in response to selected events.
You can surely come up with other reasons within your business domain.
Various IAM solutions offer different notification methods, webhooks are the most common. Systems like Auth0 or Ory offer webhook extension points. An action, for example, a sign in triggers a preconfigured URI with some payload. When triggered, the endpoint can further pass the received information, and sometimes modify it.
The problem with this approach is that we are often faced with a biased choice of events we can subscribe to. This choice might be for several less and more nefarious reasons. Maybe the original architect of the IAM solution didn’t think of a particular use case? Or specific events would enable users to implement what the provider offers in a higher-paid plan. Sometimes we are lucky and can contribute additional notifications, but there’s often friction. Open-source is excellent, but many maintainers suffer feature creep and hesitate to take in one-off features serving a limited number of users because every additional feature comes with a maintenance burden.
An event bus is a better solution. All events flow through the event bus, there is a universal way to subscribe to the event bus. Events contain raw information, they can be observed and reacted on. Many modern infrastructure components come with an event bus. Docker and Kubernetes come to mind. Surprisingly, Kafka doesn’t have one.
§Keycloak Event Listener SPI
Keycloak has one, too, and it comes in the form of an Event Listener SPI. An implementation of an Event Listener SPI plugs in directly into the core of Keycloak and receives all events happening inside of Keycloak deployment. Keycloak comes with a default implementation - its logging mechanism. There are one hundred and four events in Keycloak 18.0.1 we can act on. That’s pretty significant. There are some very useful event types defined in there. User-related, for example:
- Login, logout.
- OAuth client login and logout.
- Account linking.
- Consent grant, update, and rejection.
- Various account information changes.
But also internal Keycloak changes, the most significant one being CLIENT_UPDATE because it carries all information modified in OAuth clients, even event authorization services!
§but I said streaming
Let’s face it. Tailing and parsing logs isn’t the most convenient way of working with streaming data. The best would be to have a method to consume structured data. Like JSON. Or protocol buffers. Well, the good news is that Event Listener is a Keycloak SPI. We can write one. More than 2.5 years ago, I wrote one and put it somewhere on GitHub, then forgot about it. I rediscovered it recently while cleaning up old GitHub repositories. I have updated it for Keycloak 18 and made it available on GitHub.
What is it?
- Keycloak protobuf SPI[2] streams protobuf Keycloak events in real-time to a gRPC server.
- Keycloak protobuf event server[3] is a reference implementation of the gRPC server.
- Keycloak protobuf SPI compose[4] contains a reference Docker Compose environment with instructions required to run Keycloak behind an Envoy proxy with the event listener SPI, all TLS-enabled.
This SPI shows how to turn Keycloak into a streaming IAM source. I have chosen gRPC, I can’t remember the exact reason, but nothing is preventing us from streaming those events directly to Apache Kafka or any other system.
§how: try it out
If you have Docker and Docker Compose available, you can try it now, and it shouldn’t take more than 10 minutes of your time. Simply follow the instructions from the Keycloak protobuf SPI compose4 repository readme.
The Docker Compose configuration in that example builds on my previous articles showing how to run local Keycloak with TLS behind an Envoy proxy, so there is a little friction related to setting up TLS. The following sections of one of the previous articles about Keycloak contain relevant details:
- Keycloak 17.0.0 with TLS in Docker compose behind Envoy proxy: envoy configuration[5]
- Keycloak 17.0.0 with TLS in Docker compose behind Envoy proxy:certificates5
Once running and configured, you can see what’s going on by looking at the logs of the gRPC server:
|
|
Open the browser and go to https://idp-dev.gruchalski.com, click on the Administration Console link to navigate to Keycloak login screen. The username is admin, password is admin. In the left menu, click on Events.
Next, click on the Events Config tab and focus the Event Listeners field, you will see a drop-down with the new SPI listed:
Select the new SPI, switch all possible options to On, and click Save.
Look at the log of the event server:
2022-06-18T11:00:33.359Z [INFO] keycloak-protobuf-event-server: OnAdminEvent: admin-event="time:1655550032713 realmId:{value:\"a3149aad-f5bf-4213-b771-004c95322ce9\"} authDetails:{realmId:{value:\"a3149aad-f5bf-4213-b771-004c95322ce9\"} clientId:{value:\"1d0bdadd-b82b-49dd-9193-f18a2e9e461c\"} userId:{value:\"cff2f608-293b-4cac-b640-fb399009274c\"} ipAddress:{value:\"172.18.0.4\"}} resourceType:{value:\"REALM\"} operationType:UPDATE resourcePath:{value:\"events/config\"} representation:{value:\"{\\\"eventsEnabled\\\":true,\\\"eventsListeners\\\":[\\\"jboss-logging\\\",\\\"keycloak-protobuf-spi-event-listener\\\"],\\\"enabledEventTypes\\\":[\\\"LOGIN\\\",\\\"LOGIN_ERROR\\\",\\\"REGISTER\\\",\\\"REGISTER_ERROR\\\",\\\"LOGOUT\\\",\\\"LOGOUT_ERROR\\\",\\\"CODE_TO_TOKEN\\\",\\\"CODE_TO_TOKEN_ERROR\\\",\\\"CLIENT_LOGIN\\\",\\\"CLIENT_LOGIN_ERROR\\\",\\\"FEDERATED_IDENTITY_LINK\\\",\\\"FEDERATED_IDENTITY_LINK_ERROR\\\",\\\"REMOVE_FEDERATED_IDENTITY\\\",\\\"REMOVE_FEDERATED_IDENTITY_ERROR\\\",\\\"UPDATE_EMAIL\\\",\\\"UPDATE_EMAIL_ERROR\\\",\\\"UPDATE_PROFILE\\\",\\\"UPDATE_PROFILE_ERROR\\\",\\\"UPDATE_PASSWORD\\\",\\\"UPDATE_PASSWORD_ERROR\\\",\\\"UPDATE_TOTP\\\",\\\"UPDATE_TOTP_ERROR\\\",\\\"VERIFY_EMAIL\\\",\\\"VERIFY_EMAIL_ERROR\\\",\\\"VERIFY_PROFILE\\\",\\\"VERIFY_PROFILE_ERROR\\\",\\\"REMOVE_TOTP\\\",\\\"REMOVE_TOTP_ERROR\\\",\\\"GRANT_CONSENT\\\",\\\"GRANT_CONSENT_ERROR\\\",\\\"UPDATE_CONSENT\\\",\\\"UPDATE_CONSENT_ERROR\\\",\\\"REVOKE_GRANT\\\",\\\"REVOKE_GRANT_ERROR\\\",\\\"SEND_VERIFY_EMAIL\\\",\\\"SEND_VERIFY_EMAIL_ERROR\\\",\\\"SEND_RESET_PASSWORD\\\",\\\"SEND_RESET_PASSWORD_ERROR\\\",\\\"SEND_IDENTITY_PROVIDER_LINK\\\",\\\"SEND_IDENTITY_PROVIDER_LINK_ERROR\\\",\\\"RESET_PASSWORD\\\",\\\"RESET_PASSWORD_ERROR\\\",\\\"RESTART_AUTHENTICATION\\\",\\\"RESTART_AUTHENTICATION_ERROR\\\",\\\"IDENTITY_PROVIDER_LINK_ACCOUNT\\\",\\\"IDENTITY_PROVIDER_LINK_ACCOUNT_ERROR\\\",\\\"IDENTITY_PROVIDER_FIRST_LOGIN\\\",\\\"IDENTITY_PROVIDER_FIRST_LOGIN_ERROR\\\",\\\"IDENTITY_PROVIDER_POST_LOGIN\\\",\\\"IDENTITY_PROVIDER_POST_LOGIN_ERROR\\\",\\\"IMPERSONATE\\\",\\\"IMPERSONATE_ERROR\\\",\\\"CUSTOM_REQUIRED_ACTION\\\",\\\"CUSTOM_REQUIRED_ACTION_ERROR\\\",\\\"EXECUTE_ACTIONS\\\",\\\"EXECUTE_ACTIONS_ERROR\\\",\\\"EXECUTE_ACTION_TOKEN\\\",\\\"EXECUTE_ACTION_TOKEN_ERROR\\\",\\\"CLIENT_REGISTER\\\",\\\"CLIENT_REGISTER_ERROR\\\",\\\"CLIENT_UPDATE\\\",\\\"CLIENT_UPDATE_ERROR\\\",\\\"CLIENT_DELETE\\\",\\\"CLIENT_DELETE_ERROR\\\",\\\"CLIENT_INITIATED_ACCOUNT_LINKING\\\",\\\"CLIENT_INITIATED_ACCOUNT_LINKING_ERROR\\\",\\\"TOKEN_EXCHANGE\\\",\\\"TOKEN_EXCHANGE_ERROR\\\",\\\"OAUTH2_DEVICE_AUTH\\\",\\\"OAUTH2_DEVICE_AUTH_ERROR\\\",\\\"OAUTH2_DEVICE_VERIFY_USER_CODE\\\",\\\"OAUTH2_DEVICE_VERIFY_USER_CODE_ERROR\\\",\\\"OAUTH2_DEVICE_CODE_TO_TOKEN\\\",\\\"OAUTH2_DEVICE_CODE_TO_TOKEN_ERROR\\\",\\\"AUTHREQID_TO_TOKEN\\\",\\\"AUTHREQID_TO_TOKEN_ERROR\\\",\\\"PERMISSION_TOKEN\\\",\\\"DELETE_ACCOUNT\\\",\\\"DELETE_ACCOUNT_ERROR\\\"],\\\"adminEventsEnabled\\\":true,\\\"adminEventsDetailsEnabled\\\":true}\"} error:{noValue:{}}"
This is a string-serialized protobuf object printed to the log. What can we infer from this event?
- It’s an admin event:
OnAdminEvent: admin-event="time:1655549196131. - Modified entity was a realm:
resourceType:{value:\"REALM\"}. - We have opted-in for admin events:
\\\"adminEventsEnabled\\\":true. - We hvae opted-in for regular events:
\\\"eventsEnabled\\\":true.
Any action executed in the Master realm will now be logged. For example, log out from the realm and observe the log:
2022-06-18T11:03:16.875Z [INFO] keycloak-protobuf-event-server: OnEvent: event="time:1655550196794 type:LOGOUT realmId:{value:\"a3149aad-f5bf-4213-b771-004c95322ce9\"} clientId:{noValue:{}} userId:{value:\"cff2f608-293b-4cac-b640-fb399009274c\"} sessionId:{value:\"241a5172-107d-48e0-b5c2-d6b83966648f\"} ipAddress:{value:\"172.18.0.4\"} error:{noValue:{}} details:{key:\"redirect_uri\" value:\"https://idp-dev.gruchalski.com/admin/master/console/#/realms/master/events-settings\"}"
Log back in:
2022-06-18T11:03:34.235Z [INFO] keycloak-protobuf-event-server: OnEvent: event="time:1655550214221 realmId:{value:\"a3149aad-f5bf-4213-b771-004c95322ce9\"} clientId:{value:\"security-admin-console\"} userId:{value:\"cff2f608-293b-4cac-b640-fb399009274c\"} sessionId:{value:\"5ccb5161-c3b1-4dee-935a-6c67bb8400da\"} ipAddress:{value:\"172.18.0.4\"} error:{noValue:{}} details:{key:\"auth_method\" value:\"openid-connect\"} details:{key:\"auth_type\" value:\"code\"} details:{key:\"code_id\" value:\"5ccb5161-c3b1-4dee-935a-6c67bb8400da\"} details:{key:\"consent\" value:\"no_consent_required\"} details:{key:\"redirect_uri\" value:\"https://idp-dev.gruchalski.com/admin/master/console/#/realms/master/events-settings\"} details:{key:\"username\" value:\"admin\"}"
2022-06-18T11:03:34.741Z [INFO] keycloak-protobuf-event-server: OnEvent: event="time:1655550214737 type:CODE_TO_TOKEN realmId:{value:\"a3149aad-f5bf-4213-b771-004c95322ce9\"} clientId:{value:\"security-admin-console\"} userId:{value:\"cff2f608-293b-4cac-b640-fb399009274c\"} sessionId:{value:\"5ccb5161-c3b1-4dee-935a-6c67bb8400da\"} ipAddress:{value:\"172.18.0.4\"} error:{noValue:{}} details:{key:\"client_auth_method\" value:\"client-secret\"} details:{key:\"code_id\" value:\"5ccb5161-c3b1-4dee-935a-6c67bb8400da\"} details:{key:\"grant_type\" value:\"authorization_code\"} details:{key:\"refresh_token_id\" value:\"2a19e586-bf5c-417f-b314-75df22335b33\"} details:{key:\"refresh_token_type\" value:\"Refresh\"} details:{key:\"scope\" value:\"openid profile email\"} details:{key:\"token_id\" value:\"ce75b467-73e3-46b7-b97b-a259780b12ce\"}"
And so on.
Of course, in a real implementation, you’d be streaming those events directly to something like Kafka instead of printing them to the screen, so other apps in your company could react to those.
Event listeners are registered on a per-realm basis. To observe events in another realm, you need to repeat the installation steps on every realm for which you want to observe events.
§get in touch
Are you integrating Keycloak into your company and need a partner who can help you take it beyond where you thought it was possible? Reach out to me at radek.gruchalski@klarrio.com.