Copyright 2020 OpenStack Foundation

This work is licensed under a Creative Commons Attribution 3.0
Unported License.
http://creativecommons.org/licenses/by/3.0/legalcode

Central Authentication Service

https://storyboard.openstack.org/#!/story/2007604

Now that OpenDev is entirely distinct from the OpenStack project, it’s a great time to revisit the single sign-on and central authentication topic in a new light. We’ve rehashed and debated this so very many times in the past 5+ years, but never really officially documented what we want, our operating constraints, and what options we seem to have.

Problem Description

Our Web-based services (currently at least Gerrit and StoryBoard, but probably also MediaWiki, Zanata, maybe Askbot, RefStack, LimeSurvey, and perhaps soon the Zuul dashboard, Gitea, Mailman3/Hyperkitty or others) need logins, and we need to be able to associate accounts across different services with the same individual. We have traditionally used Launchpad/Ubuntu OpenID for this. With the addition of non-OpenStack projects, requiring people to have a UbuntuOne and OpenStackID accounts to use OpenDev services is less than ideal. However, it’s also important for OpenStack that some individuals can be connected to a corresponding OSF profile for affiliation and CCLA tracking.

  • We want a central single sign-on system for OpenDev services

  • We may want distinct realms for different Zuul tenants

  • We do not want it to directly handle authentication credentials

  • We want the SSO infrastructure operated within OpenDev

  • We want OpenStackID to be one of the available federated IDPs

  • We need to be able to migrate current Gerrit and StoryBoard IDs

Once we’re at the finish line in the future, the system would look like:

  • User wants to log into an OpenDev service

  • Service login redirects them to something like opendevid.org

  • The opendevid.org interface presents them a choice of identity providers

  • Selecting one of these options bounces them through the corresponding IDP to authenticate

  • Once authenticated, opendevid.org redirects the user back to the original service

  • A user can always go to a URL such as opendevid.org/account and associate additional identities with their account, so that they’re not limited to just a single external IDP

  • Optionally, OIDC tokens could be used for role/group claims to avoid ACL management within some services (this could come in handy for managing things like StoryBoard teams)

It could be argued that an authentication service supporting a local account database offers additional flexibility so that users still have an avenue to log in if their external IDP(s) of choice are suddenly defunct, but we’ve gotten by for nearly a decade relying on external IDPs for our services (a mix of Launchpad/UbuntuOne SSO and OpenStackID), and would prefer not to incur the operational and legal overhead of securing a database full of user credentials. We expect to recommend to our users that they affiliate more than one external IDP with their OpenDev identity, to ensure continuity. Failing that, we can work with them individually on a best-effort basis to reestablish access for their accounts when necessary. We also get pushback from users already to the effect, “Why do I need yet another account for your services? I should just be able to authenticate with my existing provider of choice.”

Proposed Change

Infra will run a central authentication service, which we’ll call OpenDevID. It will serve as the basis of SSO for all OpenDev services. OpenDevID will provide both OpenId and OAuth acting as a single source of authentication for other OpenDev services. The term OpenDevID and associated opendevid.org domains are used here as placeholders. If possible, we should attempt to come up with a more catchy name, one which is ideally also less likely to cause confusion with OpenStackID.

OpenDevID will be a pure federated system and not a primary source of identity management itself. That means that a user will log into OpenDevID using another credential their choice: raw OpenID, Github, Google, Twitter, Ubuntu, OpenStackID, whatever. One will also be able to associate as many other systems as one desires. This will allow a user to log in to the OpenDevID service using an account, such as Launchpad/UbuntuOne, then associate accounts from other systems, such as OpenStackID, as desired.

Over the years we’ve evaluated a number of options, some promising at the time but later left fallow or discovered to be harboring previously unforeseen impediments for our use case. The greatest challenge across these has been support for Launchpad/UbuntuOne, due to its reliance on OpenID v1 which has support in basically none of the options still available to us. This specification covers deploying Keycloak for our OpenDevID, with SimpleSAMLphp as a shim for handling legacy Launchpad/UbuntuOne integration (at least temporarily for purposes of transition) unless Keycloak gains upstream support for OpenID v1.

Alternatives

Some of the options we’ve explored over the years are enumerated here.

Keystone

Keystone is working on becoming a suitable standalone identity broker, but this is still in progress and likely won’t support OpenID v1.

Extend an Existing Solution

Add/write a Launchpad/UbuntuOne specific OpenID driver for the broker of our choice, perhaps Dex. Preseed its database with the OpenID mappings we already have, and generate the appropriate reverse mapping to put into Gerrit. Then configure the broker to allow people to add other IDP identities to their account, including OpenStackID, and tell people they have six months to associate another IDP with their OpenDevID account before we shut off Launchpad/UbuntuOne as a viable IDP so that we’re not stuck maintaining it forever.

This has the benefit of being all within our control, but also involves us writing a chunk of code which may or may not be in a language we find interesting. It also requires updating Gerrit et al to authenticate with the broker over something other than OpenID v1, but this is something we should be doing regardless once we’re off of Launchpad/UbuntuOne. OpenID v1 is pretty dead these days. For example, if we go with Dex, there is a Dex-specific plugin for Gerrit already; the OAuth2 plugin for Gerrit also has extensive support for many identity providers.

A challenge with the coreos/golang broker was that adding new providers was not too bad, but there weren’t generic providers other than the openid-connect broker. If we’re adding OpenID v1 support along with a provider the amount of effort is higher. That said, the fact that there are also provider specific drivers means we could just take the easy route and implement a Launchpad/UbuntuOne driver and hardcode things. OpenID itself is pretty simple, so maybe it wouldn’t be that terrible. So far the most promising suggestion for the general OpenIDv1 problem has been to use simplesamlphp to make an OpenIDv1 shim (we have contributors who have already done basically that in the past).

OpenStackID as a Proxy

Work with the OpenStackID maintainers to add the ability to associate Launchpad/UbuntuOne IDs with OpenStackIDs. Have OpenStackID require a Launchpad/UbuntuOne ID be associated with an OpenStackID account if the referrer is opendevid.org. Then write a tiny service that takes an OpenstackID+Launchpad/UbuntuOne ID pair as an input, that will write that info to the Gerrit and StoryBoard databases. Add code to OpenStackID which hits that service when someone logs in from opendevid.org so that any time someone identifies we collect the mapping and update the user account. After some time, allow other IDPs than OpenStackID. We won’t need to switch off openid v1 in gerrit if we go this route but we may still want to.

This is super hacky, and puts OpenStackID in the critical path for a period of time, but has the benefit of the development work being shared with OpenStackID manitainers (other than the tiny little DB update service). The mapping work for OpenStackID to Launchpad/UbuntuOne may already be something the OpenStackID maintainers were looking at, as one of the things they want out of this is to be able to make that association and initially thought OpenStackID would do it. If this solution seems appealing, we should ask them for more info just to be sure this isn’t already accounted for.

Map Identities via ETL

Do a behind the scenes OpenID mapping exchange with OpenStackID based on user E-mail. Pre-generate a set of OpenStackID identities for each account based on E-mail address. Put those accounts into the backend DB of opendevid.org. Go ahead and add other IDPs. If someone logs in from one of the other IDPs and it comes back with a known E-mail address we have associated with an OpenStackID, make them log into OpenStackID too proving they are that person, which will then just add the association. If they log in with not-OpenStackID, it’ll just create a new account.

In this case we would map Launchpad/UbuntuOne to OpenStackID at a point in time, then rely on the OpenStackID E-mail matching any other accounts from that point forward. This would work with existing brokers because we don’t need to OpenID v1, and allows us to just do a hard cutover using, for example, the generic openid-connect support in Dex. The biggest question here is, do we believe that E-mail address matching for the intial mapping will be good enough that followup support issues will be reasonable to handle?

This has the benefit of not needing to write any Launchpad/UbuntuOne OpenID support code, but has a drawback of the initial database mapping being potentially incomplete/inexact, so there might be rectifications that need to be done.

Help Improve Launchpad/UbuntuOne

Development seems to have stalled years ago, so we would likely be left carrying a fork best case. Also, while technically open source, running our own rebranded version of this would be next to impossible. On top of that, it’s a source of truth for identity, and likely unsuitable as an IDP broker, so this still sticks us with maintaining a databse of user credentials and doesn’t allow users to bring their own external identities either.

Reuse or Rebrand OpenStackID

We’re really looking for a service which delegates to other identity providers, but OpenStackID is itself a source of truth. Not relying solely on OpenStackID actually puts us in a good position to get OpenStackID adoption faster than we would otherwise, since there will be a legitimate path from Launchpad/UbuntuOne to OpenStackID. There is no intent to prevent people from pushing changes for review without first associating an OpenStackID. The purpose of this to better support non-OpenStack projects, and requiring an “OpenStack ID” sounds pretty antithetical to that.

Independent IDPs Across Services

We can’t just have a place for people to write down all their accounts that may be used across services and then let services use arbitrary OpenID providers. Not all of our services support the same protocols, nor do all IDPs, so the intersection of these may be empty. We’d rather not require users to have more than one identity and have to remember which one to use for what service.

Note: this is essentially status quo, it’s the situation we’re already in today with some services using Launchpad/UbuntuOne and others using OpenStackID.

Implementation

Bootstrapping

We currently have a mapping of Launchpad/UbuntuOne OpenID accounts with E-mails used for Gerrit. We can use this to pre-populate the OpenDevID database so that all of our existing users will pre-exist in the system. We can then ask people to log in and then associate their OpenStackID account.

Options

Five options for implementation have been evaluated:

  • Write our own

  • Ipsilon

  • Dex

  • Hydra

  • Keycloak (current consensus choice)

Write our own

We probably shouldn’t write our own from scratch, unless no other options are remotely viable. We have enough bespoke software we’re already maintaining, and this is also how OpenStackID came into the picture.

Ipsilon

Ipsilon is awesome, but seems to be more focused on having a user database (be that FreeIPA, OpenLDAP, et cetera) so can’t serve as a mere aggregation broker for external IDPs. Otherwise it fits the majority of our criteria.

https://ipsilon-project.org/doc/intro.html

We tried our own proof of concept deployment around five years ago, but it has come a long way since then

Dex

Dex, from CoreOS, https://github.com/coreos/dex seems like a decent fit, in that it’s designed to provide OAuth2 and can be set up to delegate auth to another configured provider:

https://github.com/coreos/dex/blob/master/Documentation/oidc-connector.md

https://github.com/coreos/dex/blob/master/Documentation/github-connector.md

Downside to Dex is it’s a bit more complex and is focused on being run in a Kubernetes, so we might have to tease apart some things to run it not in Kubernetes. There is also already Gerrit integration written.

It doesn’t seem to support OpenID v1 (at least not after a quick skim of the docs) so we would have to add that (and hope upstream takes it) in order to continue supporting Launchpad/UbuntuOne.

Hydra

Hyrdra seems simpler and not Kubernetes focused,

https://www.ory.am/run-oauth2-server-open-source-api-security

https://github.com/ory/hydra/tree/master/docs

but might involve us needing to write some Java code for Gerrit integration.

This too doesn’t seem to support OpenID v1 so we’d have to implement it and hope upstream is receptive to those changes as part of supporting Launchpad/UbuntuOne.

Gluu ( https://www.gluu.org/ ) is another option we looked into, though it would need OpenID support added.

Keycloak

Keycloak looks promising, and directly supports identity brokering:

https://www.keycloak.org/docs/latest/server_admin/index.html#_identity_broker

It does not support OpenID for authentication or brokering, so we would potentially need to add this. It’s also a large Java application, but then again we have a fair amount of experience running these lately anyway (Gerrit, Zanata, Zookeeper…). A solution discussed at the virtual PTG in April 2020 was to use https://simplesamlphp.org/ as a go-between identity broker to provide a SAML authentication proxy for Launchpad/UbuntuOne.

Based on the above evaluation, Keycloak is the consensus front-runner.

Gerrit Integration

We currently use OpenID 1.0 in SSO mode for Gerrit. There is an OAuth2 provider:

https://github.com/davido/gerrit-oauth-provider/tree/master/src/main/java/com/googlesource/gerrit/plugins/oauth

One of the options it already supports is CoreOS Dex, so if we go that route we should be able to integrate with Gerrit. If we go Keycloak, Gluu, Ipsilon or Hydra, we might need to work with gerrit-oauth-provider maintainers to make a more generic OAuth2 driver or something, though this probably works already or would at worst be straightforward to add. Gerrit’s OAuth plugin supports keycloak starting in the 2.14 version:

https://gerrit.googlesource.com/plugins/oauth/+/refs/heads/stable-2.14/README.md

It’s worth noting, StoryBoard already contains OIDC support (and has been tested with the OIDC implementation in OpenStackID), so this is less of a concern for it.

Assignee(s)

Primary assignee:

None

Gerrit Topic

Use Gerrit topic “central-auth” for all patches related to this spec.

git-review -t central-auth

Work Items

  • Stand up a Keycloak proof of concept deployment

  • Work out an UbuntuOne auth proxy for Keycloak using SimpleSAMLphp

  • Test the Gerrit OAuth driver’s Keycloak support

  • Test StoryBoard’s OIDC support with the Keycloak PoC

  • Write up a migration workflow plan for evaluation

  • TODO: additional steps once the above has been knocked out and we know more

Repositories

Probably one for our SimpleSAMLphp proxy, at a minimum (and perhaps others).

Servers

At least the ID broker service will need to run somewhere and, due to its sensitivity, almost certainly on its own server isolated from everything else.

Basically any of our Web-based services which have public authentication enabled will also be affected as we switch them to use this for their single source of identity.

DNS Entries

Based on prior discussions, we should give the identity broker service a distinct domain rather than putting it in a subdomain of opendev.org, to minimize risk from cross-site request forgery and similar sorts of attacks which are easier to pull off between sites in the same parent domain.

Documentation

At a minimum, the new account creation documentation in the OpenDev Manual will need updating for our new workflow.

Security

This is essentially the core of our security for user accounts on public Web services we operate, so pretty much any security consideration you can think of applies here.

Testing

We should integrate the new service into our deployment testing and exercise it with services whose deployments we already test.

Dependencies

TODO: Too early to say, will definitely require deployment automation and the like.