We live in an increasingly digital world where the ability to discover, connect and share data is becoming ever more essential. Today, we have the potential for four-nation NHS analysis, cross-EU health data spaces, and cross-cohort longitudinal research. But to unlock these opportunities, we must first understand how best to harmonise the data.
Harmonising data means making datasets from different sources comparable in terms of structure, labels, and coding — for example, how ethnic groups or gender categories are defined. Through harmonisation, information from multiple sources can be transformed into a unified, coherent format. This not only makes analysis easier by allowing direct comparison and consistent application of statistical tests, but also enables meaningful meta-analyses across platforms, regions and countries. Crucially, it allows larger sample sizes to be achieved by combining analyses of datasets, without moving the underlying data. For data providers, harmonisation ensures their datasets are more accessible, usable and valuable to customers.
Harmonising data is much like translating between languages — the goal is to preserve meaning without distortion. When starting a new dataset or data collection, it may be possible to agree in advance on codes, categories, and structures. Retrospective harmonisation, however, is often far more challenging. For example, one dataset may record ethnicity (“White”), while another records nationality (“British”). These are fundamentally different concepts and cannot simply be merged. Similarly, valuable detail is lost when broader categories replace specific ones — for instance, “Born in Pakistan, India, China, Japan” might all be collapsed into a single “Asian” category.
Another risk is false equivalence: data items may appear harmonised because they share a label, but in reality reflect different definitions. For example, “depression” might mean self-reported symptoms in one dataset, but prescribed medication in another. The central challenge in data harmonisation is therefore finding an appropriate “translation” that preserves meaning as faithfully as possible.
Here are some suggestions to guide effective data harmonisation:
Tip 1: Know your data
You cannot harmonise data you don’t understand — just as you cannot translate a language you don’t speak. Understanding your data means recognising what is genuinely comparable across datasets and what is fundamentally different. Differences may stem from the time of collection, the populations involved, or the policies and record keeping systems in place. In such cases, it may be better to keep datasets separate rather than force them together. For example, should primary care data from the UK and the Republic of Ireland be combined, even if it seems technically possible?
The best way to understand data is to talk to the people who collected it and the stakeholders who will use it. These conversations should start early, because harmonisation is an iterative process that takes time. For historical datasets, this can be particularly challenging as institutional memory fades.
It’s also vital to recognise that while all data has limitations, harmonisation introduces new ones. Important details may be lost, and sometimes apparent “findings” may be artifacts of the harmonisation process itself. For instance, if most people in a dataset coded as “Asian” are from India, you may wrongly generalise results to all “Asian” groups, such as those from China.
In short: to harmonise and interpret data properly, you need to understand it. At the very least, you need confidence in your own dataset — otherwise it becomes nearly impossible to agree on mappings with other data owners.
Tip 2: Be transparent about your decisions
The way data is harmonised should be transparent and repeatable. That means documenting why certain choices were made, how the process was carried out, and which experts were involved in reviewing and approving it. For example, if diagnosis codes were harmonised, was the process reviewed and validated by health professionals?
Equally important is acknowledging the limitations introduced by harmonisation — what insights remain valid and what can no longer be interpreted with confidence. Since there are often multiple ways to harmonise data, it is good practice to also provide access to the original data and the documentation of how mapping has been performed. This helps users see how meaning may have shifted during the process and decide for themselves whether to rely on the harmonised version. This is particularly valuable when external factors — such as governance rules or the need to reach a minimum sample size — have shaped the mapped.
Transparency not only builds trust but also helps the wider community converge on shared best practices and develop standardised tools for specific types of data.
In the end, harmonisation is more art than exact science. There may never be a single “right” way to do it — but being clear about the approach is key to helping people understand, evaluate and trust the results.
Tip 3: Plan your metadata carefully
Harmonisation needs to be flexible and adaptable to change. Don’t bury yourself or your users with endless documentation. Instead, document the process once and then record updates in a step-by-step, git-style system. This avoids a confusing trail of disconnected versions (v1, v2 … v100) and instead creates a logical progression that builds on previous work.
It’s also worth making your metadata as future-proof as possible. For example, marriage records were once defined as Bride and Groom but are now more often recorded as Person 1 and Person 2. Using slightly broader or less rigid categories can help your harmonisation remain relevant as standards evolve.
A smart strategy is to start with the straightforward, uncontroversial parts and release those first. Basic fields like date of birth or date of death are generally consistent across datasets, as are clear relationships such as mother and child. Begin where everyone can agree, then tackle the more complex or contested areas.
Tip 4: Share the Knowledge
Data harmonisation is a form of translation, and too often it relies on the knowledge of a single individual. If that person leaves, valuable context about the process can be lost. Good metadata helps, but it is not enough on its own.
Harmonisation should be a team effort, with decisions documented, discussed and shared across the group. This ensures continuity, reduces dependence on any one person, and builds collective expertise that strengthens both the process and the results.
Authors: Bandyopadhyay A, Barbour J, Boyd A, Brophy S, Difei S, Edwards M, Fallesen P, Ha N, Hamilton N, Harron K, Mendez A, Morales A, Opoku R, Palmer J, Robinson L, Seaborn M, Toft H.