BMJ.

The first step was to establish the pattern for content ingestion, mapping how to pull in the different types of content from the five different data silos.

• Pipeline modelling

To start with, the Nordcloud team focused on one of the silos: journal content. The team defined the event-driven data pipeline and mapped a range of data processing tasks, then replicated this model across the four other systems.

• Data validation

Inherently, scientific articles are difficult to pull metadata from, or to easily group. Complex components and files from each article need to be indexed. This required additional validation, and custom data models to incorporate all the possible formats of content from each article: images, tables and charts, histology slides, plus the copy.

• File packaging

To complete all the processing tasks, a packaging system was designed to order the files, and replicated across all articles to ensure each and every component was in place and able to be indexed. 

Once the content ingestion pipeline was in place, the next step was to index the data using Elasticsearch, then create an API to enable cross-product search and retrieval.

• Elasticsearch

The raw data from each storage system is ingested through the content data pipeline solution and into Elasticsearch where it is indexed. 

• Complete search capability

This enables BMJ to search across the five data sources. This function is a massive enabler, allowing unprecedented flexibility and opportunity for pulling together bespoke content in real time. This is key for a modern publisher that has content going live continuously.

• API gateway

Combining the above with a custom API allows the BMJ to write applications to use the index to search all its content and be linked back to its original source – journals; best practices; learnings and education; videos; and websites.

The Nordcloud team also reviewed BMJ’s existing DevSecOps practices, along with data and tooling, to identify areas that could be improved.

• More than a process

DevSecOps isn’t just a one-off exercise. It’s an approach to culture, automation, and AWS platform design that integrates security as a shared responsibility throughout the entire IT lifecycle.

• Optimisation

The review provides documentation to help measure the effectiveness of a company’s AWS security controls and bring recommendations to promote continuous improvement.

• Future-proofing

BMJ welcomed this project, challenging its own team to enforce best practices and fill capabilities gaps – beyond the scope of the content pipeline project – to support BMJ’s future AWS efforts.