Automation

Connecting systems: API, file exchange or message queue

Every Monday morning someone pulls a file out of one system and retypes the lines into another. It works, until that person is off sick, until a line gets skipped, or until two people process the same batch. You want the systems to do it themselves, and you want to be able to see that it went well.

Connecting systems is not, in the first place, a technical problem. There are three proven shapes for moving the data, all three well documented. The hard part sits elsewhere: agreeing what a customer, an order or an hour actually means on both sides of the line, and settling in advance what happens when the other side does not answer. An integration that only works when everything goes right is not an integration, it is a demo.

When you need an integration

The question does not start with the technology. It starts with one of these situations.

The same data is keyed in twice, into two systems, by two different people.
An export and an import go back and forth daily or weekly, by hand, and nobody notices when a step is skipped.
A customer or supplier asks whether you have an interface they can deliver data into.
An integration already exists, but you only find out it has stopped when someone rings.
The figures in two systems drift apart and nobody can point to where it starts.

What the work covers

Building it is the smallest part. This is the rest.

Choosing the shape. An API integration when the data has to be right immediately; file exchange when it is large volumes on a fixed rhythm and you want to be able to read the file back later; a message queue when both sides have to be able to run independently. I also set out why the other two are not the answer here.
The data definition. What each field means, which values are allowed, and which system wins when the two disagree. This is the heaviest part of the work, and the only part technology cannot settle for you.
Error handling. What happens on a rejected record, a time-out, or a counterparty that is unreachable for a day. Messages wait rather than vanish, with retries at increasing intervals and a limit on how many.
Repeatability. The same message may arrive twice without producing a second order. That calls for an identifier on every message and a check on the receiving side.
Monitoring and access. Making it visible that the integration ran and how much went through, with an alert to a human when it does not. Plus how the two sides identify themselves, and who replaces the keys when someone leaves.
Testing on your own data, including the messy records. An integration trips over exceptions, not over the example in the manual.

What you get

An integration is only finished when someone else can run it. That means working software, plus the paperwork needed to understand what happens when it goes wrong.

A working integration in your own environment, with the source code and configuration in your hands.
An interface document: which data crosses, in which direction, in which format, and what the fields mean.
Monitoring you can look at yourself, with an alert when throughput stops or the error queue builds up.
A short incident guide: what to do when the other side was unreachable, how to resubmit, and how to check that nothing was processed twice.
A handover to your own people or your support partner, so that carrying on does not depend on me.

What it is not

Not a replacement for either system. An integration repairs a process, not a badly set up system.
Not a one-off data migration. Related work, different risks, a separate decision.
Not a reporting or dashboard project. Moving data is a different job from opening data up for analysis.
Not an ongoing support contract with an out-of-hours rota. I deliver the integration and the monitoring; who answers the phone at night is agreed separately.
Not an integration that works around the other party. If the counterparty will not offer an interface, that is a conversation first, not a build.

Common questions

API, file exchange or message queue: which one do I need?

Two questions settle it: how fresh does the data have to be, and what may happen when the other side does not respond. If a change must be visible within seconds, an API integration is the simplest answer. If it is large volumes at a fixed moment, file exchange is sturdier and easier to check: the file is still there to look at. If both sides have to run independently of each other, or peaks arrive faster than the receiver can absorb them, put a queue in between: the sender delivers and is done, the receiver works through it at its own pace. Two shapes side by side in one landscape is not a design fault.

The other system has no API. Is it still possible?

Yes, just by a different route. We look for another way out: a scheduled export to a secure folder, a report in a fixed format, read access to a database. What changes is not feasibility but frequency and error handling. What I advise against is an integration that drives a system's screens: it breaks at the next update, and nobody sees it coming.

What happens if the other system is down for a day?

That is a design decision, not an accident. A message should wait rather than disappear: it stays put, retries follow at increasing intervals, and after an agreed number of attempts it moves to a separate pile with an alert attached. When the other side comes back, the backlog drains by itself. What you do not want is an integration that carries on after an outage as if nothing had been left behind.

What if the same message arrives twice?

Then nothing may happen twice. That is what idempotency means: running the same instruction twice gives the same result as running it once. Concretely, every message carries its own identifier and the receiving side keeps track of which identifiers it has already processed. It is the difference between an integration you can safely switch back on after an outage, and one nobody dares restart.

How long does it take, and what drives the price?

Building is not what sets the lead time. What does: the data definition (how many fields, how many exceptions, how many parties have to agree), whether the counterparty has a usable interface and documentation, whether a test environment exists, and whether personal data or financial transactions pass through it. A single integration between two systems that both have a decent interface is a short engagement of a few days. Once the interface still has to be designed, or three parties are at the table, it becomes a project. The shape is agreed up front: time and materials with a ceiling I will not exceed without checking first, or a fixed price for a defined result when the scope is sharp.

Who runs the integration afterwards?

Your own people or your support partner. I hand over the source code, the configuration, the monitoring and the incident guide to whoever will be running it. If I stay involved after that, it is a separate arrangement and not a condition. An integration only its builder can maintain is a risk you get thrown in for free.

More within this service

Two systems that still do not talk to each other?

Sketch briefly which systems are involved and what currently moves back and forth by hand. In a first conversation I will say which shape fits, and whether an integration is genuinely the answer.

Book an intro call Send a message