Avatar
by Tom Hodgson on 20 Aug 2018

At SaleCycle, microservices process data from thousands of web sites in real time. Microservices are great but life becomes difficult when we need to change a service’s API. With 5000 chunky pieces of data flowing into the system every second we can’t just swap the new API in, the existing consumers will break, we’ll lose data, and probably money (I don’t deal with the finances).

Why we use microservices

All of our services talk to each other using JSON and REST APIs. Modifying them is relatively risk free. A suite of Runscope tests ensures that nothing changes from the consumers point of view each time we make a change. This is in much the same way unit tests allow us to change code without breaking anything. We’re entirely cloud based, and all our deployments are automated. We use AWS ECS to run services in Docker containers sitting behind an API Gateway. We can swap out a Node.js server for one written in Go simply by merging a pull request.

The current API

A particular microservice stores our client data. It exposes the following endpoint:

GET /clients

which returns a list of clients:

[{
    "id": "95dafe9f-a98b-4240-9c6e-6b4e73bd37d9",
    "clientMarketingConfig": {
        "enabled": true
    },
    ...
}, ..., {
    "id": "5f0745c3-cbc2-459e-bef4-d64653cc2e05",
    "clientMarketingConfig": {
        "enabled": false
    },
    ...
}]

there are several sub resources e.g.

GET /clients/95dafe9f-a98b-4240-9c6e-6b4e73bd37d9/campaigns

which returns a list of sub resources, each with a reference to its parent, through the clientId property.

[{
    "id": "0e9bb8af-ab24-49c7-9d9c-7e7d760dbee7",
    "clientId": "95dafe9f-a98b-4240-9c6e-6b4e73bd37d9",
    ...
}, ..., {
    "id": "d6c3b7ca-b14d-4b20-a9e8-9e9a09d0d4a6",
    "clientId": "95dafe9f-a98b-4240-9c6e-6b4e73bd37d9",
    ...
}]

The new API

Up to now there has been a one-to-one mapping between a client and its website, but we want to change this. It turns out, up to now, whenever we have been using the word ‘client’ we have meant ‘website’. We decide its time to change the APIs to reflect this. We want the new API to be

GET /sites

which returns a list of sites (note the change in the property names):

[{
    "id": "95dafe9f-a98b-4240-9c6e-6b4e73bd37d9",
    "siteMarketingConfig": {
        "enabled": true
    },
    ...
}, ..., {
    "id": "5f0745c3-cbc2-459e-bef4-d64653cc2e05",
    "siteMarketingConfig": {
        "enabled": false
    },
    ...
}]

and the sub resources e.g.

GET /sites/a-client-id/campaigns

to return JSON with a parent ‘siteId’ instead of ‘clientId’

[{
    "id": "0e9bb8af-ab24-49c7-9d9c-7e7d760dbee7",
    "siteId": "95dafe9f-a98b-4240-9c6e-6b4e73bd37d9",
    ..
}, ..., {
    "id": "d6c3b7ca-b14d-4b20-a9e8-9e9a09d0d4a6",
    "siteId": "95dafe9f-a98b-4240-9c6e-6b4e73bd37d9",
    ...
}]

The plan

So the plan is:

  1. Deploy the new API alongside the old one.
  2. Migrate all the consumers of the API to the new format
  3. Delete the old API.

The current implementation

At SaleCycle we use ‘the right technology for the task’. We have microservices running in Go, Kotlin, Java, Node, and other technologies. They are deployed in docker or AWS serverless architectures. This one happens to be in Java using the vert.x framework, chosen to support a high level of concurrent connections and to simplify its dependency on Apache Ignite (also Java).

the model classes currently look like:

    public class Client extends Entity implements Resource {
        private Guid id;
        private ClientMarketingConfig clientMarketingConfig;
        public ClientMarketingConfig getClientMarketingConfig() {
            return clientMarketingConfig;
        }
        public void setClientMarketingConfig() {
            this.clientMarketingConfig = clientMarketingConfig;
        }
        ...
    }


    public class Campaign extends Entity implements Resource {
        private Guid id;
        private Guid clientId;
        public Guid getClientId() {
            return clientId;
        }
        public void setClientId(Guid clientId) {
            this.clientId = clientId;
        }
        ...
    }

I’m personally not a fan of getters and setters on data objects (I much prefer immutability) but in Java it’s often the way to go. It can help external libraries play nicely, especially those that depend on reflection e.g. for parsing and validation etc.

With VertX it is simple to bind an endpoint to a RequestHandler and (omitting a couple of details) the code looks something like this:

    router.get("/clients/:clientId/campaigns").handler(api.getCampaigns())
    router.get("/clients").handler(api.getClients())
    ...

where for this example API would look something like:

public interface API {
    public Collection<Campaign> getCampaigns();
    public Collection<Client> getClients();
}

Failed Attempt 1: ‘The crafty hack’

We duplicate the endpoints e.g:

    router.get("/clients/:clientId/campaigns").handler(api.getCampaings())
    router.get("/sites/:siteId/campaigns").handler(api.getCampaigns())
    router.get("/clients").handler(api.getClients())
    router.get("/sites").handler(api.getClients())
    ...

and we rename the entities, rename clientId to siteId, and add the new getters and setters. Finally we deprecate the old client getters and setters and modify them to use the new siteId backing field:

Client.java becomes:

    public class Site extends Entity implements Resource {
        private final Guid id;
        private SiteMarketingConfig siteMarketingConfig;
        public SiteMarketingConfig getSiteMarketingConfig() {
            return siteMarketingConfig;
        }
        public void setSiteMarketingConfig() {
            siteMarketingConfig
        }
        @Deprecated
        public SiteMarketingConfig getClientMarketingConfig() {
            return siteMarketingConfig;
        }
        @Deprecated
        public void setClientMarketingConfig() {
            this.clientMarketingConfig = siteMarketingConfig;
        }
    }

and Campaign.java is modified to:

    public class Campaign extends Entity implements Resource {
        private final Guid id;
        private final Guid siteId;
        public Guid getSiteId() {
            return siteId;
        }
        public void setSiteId(Guid clientId) {
            this.siteId = siteId;
        }
        @Deprecated
        public Guid getClientId() {
            return siteId;
        }
        @Deprecated
        public void setClientId(Guid clientId) {
            this.siteId = siteId;
        }
    }

On the face of it this looks quite cunning, we’ve hardly changed the code base and we have both APIs running side by side. The returned JSON (serialised automatically from the Java classes by Jackson) contains both the before and after fields e.g. for the case of a client, the returned JSON would be:

{
    "id": "95dafe9f-a98b-4240-9c6e-6b4e73bd37d9",
    "clientMarketingConfig": {
        enabled: true
    },
    "siteMarketingConfig": {
        enabled: true
    },
    ...
}

However we updated the first consumer and realised it wasn’t going to work and heres why: the first consumer was a TypeScript/React/Redux UI that is used to manage the campaigns and clients. It’s still consuming the old /clients API. The UI doesn’t know anything about the siteMarketingConfig but it will be present on any data it consumes. Imagine we disable marketing, because we only know about the clientMarketingConfig property we will only change that field. We will modify the object retrieved from the /clients endpoint and the update request will look something like:

PUT /clients/95dafe9f-a98b-4240-9c6e-6b4e73bd37d9
{
    "clientMarketingConfig": {
        enabled: false
    },
    "siteMarketingConfig": {
        enabled: true
    },
}

Now when the server receives the updated entity and attempts to deserialise the Java entity classes, one of the clientMarketingConfig or siteMarketingConfig is going to override the other. We could try and be clever by defining an order or ignore unknown fields etc, but because we need to support consumers of both the old and new API we are going to fail. Perhaps if we’d used PATCH to perform updates we wouldn’t have this problem, but we didn’t.

Failed Attempt 2: Prison work

After wasting time trying to be clever, we decided to bite the bullet and code two separate APIs.

We created the new entities, keeping the old ones:

    package com.salecycle.models.backwards.compatible

    @Deprecated
    public class Client extends Entity implements Resource {
        private Guid id;
        private ClientMarketingConfig clientMarketingConfig;
        public ClientMarketingConfig getClientMarketingConfig() {
            return clientMarketingConfig;
        }
        public void setClientMarketingConfig() {
            this.clientMarketingConfig = clientMarketingConfig;
        }
    }

    package com.salecycle.models.backwards.compatible

    @Deprecated
    public class Campaign extends Entity implements Resource {
        private Guid id;
        private Guid clientId;
        public Guid getClientId() {
            return clientId;
        }
        public void setClientId(Guid clientId) {
            this.clientId = clientId;
        }
    }

    package com.salecycle.models

    public class Site extends Entity implements Resource {
        private final Guid id;
        private SiteMarketingConfig siteMarketingConfig;
        public SiteMarketingConfig getSiteMarketingConfig() {
            return siteMarketingConfig;
        }
        public void setSiteMarketingConfig() {
            siteMarketingConfig
        }
    }

    package com.salecycle.models

    public class Campaign extends Entity implements Resource {
        private final Guid id;
        private final Guid siteId;
        public Guid getSiteId() {
            return siteId;
        }
        public void setSiteId(Guid siteId) {
            this.siteId = siteId;
        }
    }

We could have subclassed the common parts, but I’m going to be deleting the deprecated entities after the migration, so I just copied and pasted. We put them all in a single package and deprecated them so it would be relatively simple to delete them all later.

We duplicated the endpoints again, but handled the requests differently e.g:

    router.get("/clients/:clientId/campaigns").handler(api.getCampaings())
    router.get("/sites/:siteId/campaigns").handler(api.getOldCampaigns())
    router.get("/clients").handler(api.getClients())
    router.get("/sites").handler(api.getSites())
    ...

where the signature of API looks like:

public interface API {
    public Collection<Campaign> getCampaigns();
    public Collection<com.salecycle.backward.compatible.Campaign> getOldCampaings();
    public Collection<Site> getSites();
    public Collection<com.salecycle.backward.compatible.Client> getClients();
}

The problems with this approach became apparent once we started to implement this interface. As we duplicated the existing getClients to create getSites function we needed to duplicate its dependencies. Our stacks are all cloud based hosted in AWS and we use DynamoDB to store our entities. We have a DynamoDbService to retrieve these entities:

interface class<T extends Entity> DynamoDBService {
    Collection<T> get();
    void put(T)
}

We have convertor classes to create our entities:

public interface Converter<T extends Entity> {
    T toEntity(Map<String, AttributeValue> map);

    Map<String, AttributeValue> toMap(T entity);
}

We have other interfaces to represent Resources, Validatable, not to mention all our own business logic. Now we have a whole new hierarchy of types to support and implement and I find myself duplicating the entire codebase, so at this point I stop again.

Failed attempt 3: Design patterns

The idea here was to duplicate the entity classes (as in the previous attempt), but simply decorate the endpoint to create the legacy endpoint:

    router.get("/sites").handler(api.getSites())
    router.get("/clients").handler(siteConvertor.convertToClients(api.getSites()))
    ...

where

interface SiteConvertor {
    Collection<Client> convertToClients(Collection<Site> sites)
}

This was pretty much a non starter we have lots of entities, each with lots of properties, I didn’t really fancy writing the conversion logic (and the tests). It was a better solution than the previous attempt, but I needed something better.

At the 4th attempt…

Maybe if it hadn’t been quite such a significant change, (the root of all our resources) any of the first three attempts would have been fine. The previous attempt was on the right line we just needed a nicer way of implementing it. We realised that we could update all the code and entities to the new API and simply use a different serialiser for the old endpoints.

So we only have the new style entities:

    public class Site extends Entity implements Resource {
        private final Guid id;
        private SiteMarketingConfig siteMarketingConfig;
        public SiteMarketingConfig getSiteMarketingConfig() {
            return siteMarketingConfig;
        }
        public void setSiteMarketingConfig() {
            siteMarketingConfig
        }
    }


    public class Campaign extends Entity implements Resource {
        private final Guid id;
        private final Guid siteId;
        public Guid getSiteId() {
            return siteId;
        }
        public void setSiteId(Guid siteId) {
            this.siteId = siteId;
        }
    }

Then we can expose both endpoints but override the serialiser/deserialiser for the legacy endpoint. The API service already used a Jackson ObjectMapper to do the JSON serialisation so it required very little code to make it overridable for the legacy call.

    router.get("/sites").handler(api.getSites())
    router.get("/clients").handler(api.getSites(createbackwardsCompatibleObjectMapper()))

Finally we just needed to use Jackson mixins to create an object mapper that would convert to and from the legacy form of the entities during serialisation (and deserialisation).

    public static ObjectMapper createBackwardsCompatibleMapper() {
        ObjectMapper mapper = new ObjectMapper();
        mapper.setVisibility(mapper.getSerializationConfig()
                .getDefaultVisibilityChecker()
                .withFieldVisibility(JsonAutoDetect.Visibility.ANY)
                .withGetterVisibility(JsonAutoDetect.Visibility.NONE)
                .withSetterVisibility(JsonAutoDetect.Visibility.NONE)
                .withCreatorVisibility(JsonAutoDetect.Visibility.NONE));
        mapper.addMixIn(Site.class, SiteMixin.class);
        mapper.addMixIn(Campaign.class, EnityMixin.class);
        return mapper;
    }

The mapper.setVisibility is just boilerplate. The calls to mapper.addMixIn specify the functionality you wish to override during serialisation/deserialisation. You can apply Jackson annotations to any fields in the mixin class, and they will override the default behaviour determined by the original class. In our case we want to change the name of the JSON properties so we use the @JsonProperty annotation.

public class SiteMixin {
    private SiteMarketingConfig siteMarketingConfig;

    @JsonProperty("clientMarketingConfig")
    public SiteMarketingConfig getSiteMarketingConfig() {
        return siteMarketingConfig;
    }

    @JsonProperty("clientMarketingConfig")
    public void setSiteMarketingConfig() {
        siteMarketingConfig
    }
}

and

public class EntityMixin {
    private String siteId;

    @JsonProperty("clientId")
    public String getSiteId() {
        return siteId;
    }

    @JsonProperty("clientId")
    public void getSiteId(String siteId) {
        this.siteId = siteId;
    }
}

and with only those changes we have the old and new APIs running alongside each other. There is no pollution of the codebase with legacy concepts. The mixins and extra route registrations can all be defined in separate files which can be simply be deleted when the migration is complete e.g. in an open closed manner.