Artificial Intelligence (AI) is fast becoming the cornerstone of business analytics, allowing companies to generate value from the ever-growing datasets generated by today’s business processes. At the same time, the sheer volume and velocity of data demand high-performance computing (HPC) to provide the power needed to effectively train AIs, do AI inferencing, and run analytics. According to Hyperion Research, HPC-enabled AI, growing at more than 30 percent, is projected to be a $3.5 billion market in 2024.

This rising confluence of HPC and AI is being driven by businesses and organisations honing their competitive edge in the global marketplace as digital transformation is accelerated and brought to the next level through IT transformation processes.

“We’re seeing HPC-enabled AI on the rise because it extracts and refines data quicker and more accurately. This naturally leads to faster and richer insights, in turn enabling better business outcomes and facilitates new breakthroughs and better differentiation in products and services while driving greater cost savings,” said Mike Yang, President at Quanta Cloud Technology, better known as QCT.

While HPC and AI are expected to benefit most industries, the fields of healthcare, manufacturing and higher education and research (HER) and Finance stand to gain perhaps the most due to the high-intensity nature of the workloads involved.

Application of HPC-enabled AI in the fields of next-generation sequencing, medical imaging and molecular dynamics have the potential to speed drug discoveries and improve patient care procedures and outcomes. In manufacturing, finite element analysis, computer vision, electronic design automation and computer-aided design are facilitated by AI and HPC to speed product development, while analysis generated from Internet-of-Things (IoT) data can streamline supply chains, enhance predictive maintenance regimes and automate manufacturing processes. HER utilises technology to explore fields such as dynamic structure analysis, weather prediction, fluid dynamics and quantum chemistry in an ongoing quest to solve global problems like climate change and achieve breakthroughs and deeper exploration through cosmology and astrophysics.    

Optimising HPC and AI Workloads

The AI and Machine Learning (ML) algorithms underlying these business and scientific advances have become significantly more complex, delivering faster yet more accurate results, but at the cost of significantly more computational power. The key challenge now facing organisations is building HPC, AI, HPC-enabled AI, and HPC-AI converged workloads—while shortening project implementation time. Ultimately, this will allow researchers, engineers, and scientists to concentrate fully on their research.

IT support would also need to actively manage their HPC and AI infrastructure, leveraging the right profiling tool for optimisation of HPC and AI workloads. Optimised HPC/AI infrastructure should deliver the right resources at the right time for researchers and developers to accelerate computational processes.

In addition, understanding workload demands and optimising performance helps IT avoid additional workload and extra labour for finetuning, significantly reducing the total cost of ownership (TCO). To optimise HPC and AI workloads effectively and quickly, organisations can consider the following steps:

Identify key workload applications and data used by the customer, as well as the customer’s expectations and pain pointsDesign infrastructure and building the cluster, ensuring that hardware and software stack can support the workloadsContinue the process of always adjusting and finetuning

QCT leverages Intel’s profiling tool Intel Granulate gProfiler to reveal the behaviour of the workload before tapping its deep own deep expertise to analyse the behaviour and design a fine-tuning plan to help with optimisation. Through this process, organisations can ensure rapid deployment, simplified management, and optimised integrations—all at cost savings.

AI continues to offer transformational solutions for businesses and organisations, but the growing complexity of datasets and algorithms is driving greater demand on HPC to enable these power-intensive workloads. Workload optimisation effectively enhances the process and, at the heart of it, enables professionals in their fields to focus on their research to drive industry breakthroughs and accelerate innovation.

To discover how workload profiling can transform your business or organisation, click here.

Artificial Intelligence, Digital Transformation, High-Performance Computing

Since the premier of the wildly popular 1993 dinosaur cloning film Jurassic Park, the sciences featured in the film, genetic engineering and genomics, have advanced at breathtaking rates. When the film was released, the Human Genome Project was already working on sequencing the entire human genome for the first time. They completed the project in 2003 after 13 years and at a cost of $1 billion. Today, the human genome can be sequenced in less than a day and at a cost of less than $1,000.

One leading genomics research organization, The Wellcome Sanger Institute in England, is on a mission to improve the health of all humans by developing a comprehensive understanding of the 23 chromosomes in the human body. They’re relying on cutting edge technology to operate at incredible speed and scale, including reading and analyzing an average of 40 trillion DNA base pairs a day.

Alongside advances in DNA sequencing techniques and computational biology, high-performance computing (HPC) is at the heart of the advances in genomic research. Powerful HPC helps researchers process large-scale sequencing data to solve complex computing problems and perform intensive computing operations across massive resources.

Genomics at Scale

Genomics is the study of an organism’s genes or genome. From curing cancer and combatting COVID-19 to better understanding human, parasite, and microbe evolution and cellular growth, the science of genomics is booming. The global genomics market is projected to grow to $94.65 billion by 2028 from $27.81 billion in 2021, according to Fortune Business Insights. Enabling this growth is a HPC environment that is contributing daily to a greater understanding of our biology, helping to accelerate the production of vaccines and other approaches to health around the world.

Using HPC resources and math techniques known as bioinformatics, genomics researchers analyze enormous amounts of DNA sequence data to find variations and mutations that affect health, disease, and drug response. The ability to search through the approximately 3 billion units of DNA across 23,000 genes in a human genome, for example, requires massive amounts of compute, storage, and networking resources.

After sequencing, billions of data points must be analyzed to look for things like mutations and variations in viruses. Computational biologists use pattern-matching algorithms, mathematical models, image processing, and other techniques to obtain meaning from this genomic data.

A Genomic Powerhouse

At the Sanger Institute, scientific research is happening at the intersection of genomics and HPC informatics. Scientists at the Institute tackle some of the most difficult challenges in genomic research to fuel scientific discoveries and push the boundaries of our understanding of human biology and pathogens. Among many other projects, the Institute’s Tree of Life program explores the diversity of complex organisms found in the UK through sequencing and cellular technologies. Scientists are also creating a reference map of the different types of human cells.

Science on the scale of that conducted at the Sanger Institute requires access to massive amounts of data processing power. The Institute’s Informatics Support Group (ISG) helps meet this need by providing high performance computing environments for Sanger’s scientific research teams. The ISG team provides support, architecture design and development services for the Sanger Institute’s traditional HPC environment and an expansive OpenStack private cloud compute infrastructure, among other HPC resources.

Responding to a Global Health Crisis

During the COVID-19 pandemic, the Institute started working closely with public health agencies in the UK and academic partners to sequence and analyze the SARS-COV-2 virus as it evolved and spread. The work has been used to inform public health measures and to help save lives.

As of September 2022, over 2.2 million coronavirus genomes have been sequenced at Wellcome Sanger. They are immediately made available to researchers around the world for analysis. Mutations that affect the virus’s spike protein, which it uses to bind to and enter human cells, are of particular interest and the target of current vaccines. Genomic data is used by scientists with other information to ascertain which mutations may affect the virus’s ability to transmit, cause disease, or evade the immune response.

Society’s greater understanding of genomics, and the informatics that goes with it, has accelerated the development of vaccines and our ability to respond to disease in a way that’s never been possible before. Along the way, the world is witnessing firsthand the amazing power of genomic science.

Read more about genomics, informatics, and HPC in this white paper and case study of the Wellcome Sanger Institute.

***

Intel® Technologies Move Analytics Forward

Data analytics is the key to unlocking the most value you can extract from data across your organization. To create a productive, cost-effective analytics strategy that gets results, you need high performance hardware that’s optimized to work with the software you use.

Modern data analytics spans a range of technologies, from dedicated analytics platforms and databases to deep learning and artificial intelligence (AI). Just starting out with analytics? Ready to evolve your analytics strategy or improve your data quality? There’s always room to grow, and Intel is ready to help. With a deep ecosystem of analytics technologies and partners, Intel accelerates the efforts of data scientists, analysts, and developers in every industry. Find out more about Intel advanced analytics.

High-Performance Computing

Blockchain is one of the great blindsides in the history of technology.  Major trends around cloud technology, virtualization, and mobile you could see coming, but a novel distributed computing model based on public key cryptography?  That came almost completely out of the blue. 

When the Nakamoto whitepaper dropped in 2008 it unceremoniously set off a ferment of innovation that continues growing to this day.  It wasn’t entirely unexpected if you were watching the digital currency scene—at the time an even nerdier backwater of the already highly nerdy cryptography frontier.  The paper itself gives a tip of the hat to several prior artists like Adam Back’s HashCash whitepaper

In that view, Bitcoin looks like a sensible progression.  However, even as a natural outcome of compounding creativity the Bitcoin paper is extraordinary. 

The proof of work (PoW) solution to the double-spend problem is non-obvious even knowing the prior art.  And that idea led to an unpredictable result: the possibility of decentralized, permissionless virtual machines. 

The first shoots in this spreading revolution were issued by Vitalin Biturik in the Ethereum whitepaper.  The basic idea of leveraging blockchain to build Turing machines was introduced there.  Once you have established the viability of a permissionless, compute-enabled network, you get what you might expect: a horde of smart computer scientists and engineers leaping into the space to find new ways of taking advantage of and improving upon the possibilities.

In short, we have seen an outpouring of genius.  Obviously there have been some blind alleys and questionable characters at work.  None of that discredited the real groundbreaking work that has been and is being done in the space.  After the Ethereum virtual machine’s introduction, several promising avenues of progress have been proposed and implemented.  Here’s a look at some of the most prominent.

Ethereum and the virtual machine

If Bitcoin is the root from which the web3 tree has grown, Ethereum is the trunk from which the branches have spread.  Ethereum took the conceptual step of saying with a system in hand for verifying transactions are valid, maybe we can build a virtual machine.  There are devils in the detail here—implementing such a system is a substantial challenge—but not only is it possible, it also opens up applications with unique characteristics.

In general, these applications are known as dApps, or distributed applications.  dApps are comprised of smart contracts that run on-chain and the traditional web apps that interface with them.

The concept of a smart contract is perhaps the best concept to use as a lens in understanding Ethereum’s basic innovation.  Smart contracts are so called because they represent a contract on the network—they specify what are valid and binding contracts for the participants.  (Participants are cryptographically bound to these contracts via their private keys).  In a sense, the blockchain structure enables code to describe a variety of verifiable agreements between people and systems.  

Smart contracts are “smart” because they are autonomous.  This is the characteristic that really distinguishes them from conventional applications: no intervention by outside entities is necessary for their action.

With a generally available network that can enforce contracts, in which participants can join in a permissionless way, many traditional business models are potentially vulnerable to disruption.  As this article describes in detail, finance is potentially just the tip of the iceberg.  Also inherent in Ethereum basic design is the possibility of decentralized governance, or DAO (distributed autonomous organizations). 

Many practical realities must be overcome in realizing the promise of blockchain, and many of the innovations in subsequent projects are targeted at doing just that.

Peercoin and proof of stake

The consensus mechanism is the means by which nodes in the network come to agreement on what are valid transactions.  Bitcoin originated PoW consensus, which uses the cryptographically demonstrable work done in mining certain values.  This works but suffers from being energy intensive and acts as a performance bottleneck. The Peercoin whitepaper introduced an alternative mechanism known as proof of stake (PoS). 

The wealth of projects that have since been built using the PoS model is a wonderful testament to its efficacy, but perhaps the greatest testament is Ethereum itself moving to a PoS model with Ethereum 2.  Proof of stake opens up possibilities by streamlining the overall operations of blockchain networks.  

Proof of stake works by ensuring validator nodes are vested in the network.  In general, that means establishing that validators hold a certain amount of the crypto token used by the platform to denote value. At the very highest level, you can say proof of stake works by creating an incentive for nodes to behave correctly.  Compromising the network by means of a Byzantine network attack, like a Sybil attack or a 51% attack, will devalue the very coins held by the malicious nodes.  This increases the cost of attacks and is a disincentive.  It’s simpler and more lucrative to simply participate in good faith.

PoS eliminates the high energy cost on validator nodes.  It also reduces the minimum time required by nodes to process transactions.  That is because the nature of PoW is doing difficult computations, something that depends upon time and electricity.

PoS is not without drawbacks.  Nevertheless, it represents a real innovation and opened up not just new implementations, but also caused people to begin thinking creatively about proof of consensus and other fundamentals in “layer 1” technology.

Solana and proof of history

Another breakthrough in blockchain thought is Solana’s proof of history (PoH) mechanism.  It’s whitepaper describes a system wherein a verifiable delay function (VDF) is applied to the network, enabling validator nodes to largely ignore the question of transaction ordering.

A verifiable delay function is one that, similar to a mining function, establishes that it has run by executing a cryptographic function.  This function outputs a hash that then proves it has run and a certain amount of time has elapsed.  This is like a cryptographic clock.

By devising an architecture that allows validators to share a single VDF server, the entire Solana network boasts radically faster block verification times.  It’s important to note that PoH by itself is a performance optimization, not a validation mechanism.  It must be combined with a consensus mechanism.  In Solana’s case, its token (SOL) adopts PoS. 

Avalanche and validation neighborhoods

The Avalanche whitepaper introduces an ingenious approach to consensus.  It proposes that nodes can agree upon valid transactions by sampling a small set of the network.  You can think of this as validation proceeding against a ‘flash’ subnet.  As the paper says, “Each node polls a […] randomly chosen set of neighbors, and switches its proposal if a supermajority supports a different value.”

This simple-seeming idea is well suited to the conditions of a distributed network.  It obtains lower overhead for nodes because they don’t have to refer to the entire network to be assured they have a valid copy of the blockchain state.  At the same time, the interconnected operation of all the different validator neighborhoods means the overall network state always moves towards valid consensus. 

Avalanche’s whitepaper is also notable for explicit and clear statements of two other principles that have gained traction.  The first is the idea of creating a “network of networks” wherein the underlying network enables a variety of networks that can operate independently or when desired interdependently via the chain’s token (AVAX).  The example given in the paper is of one subnetwork that handles gold contracts and another that handles real estate.  The two operate independently unless someone wants to buy real estate using their gold, at which point AVAX becomes the medium of exchange.

The second idea Avalanche puts forth well is self-governance.  In short, it has built in its protocol the ability of nodes to alter the parameters of its operation.  In particular, the member nodes have the ability to control staking timeframes, fee amounts, and minting rates. 

Internet computer and partial synchrony

The internet computer project is founded on a whitepaper that introduces a new mechanism for obtaining beneficial characteristics from both conventional and blockchain networks, thereby “obtaining the efficiency of a permissioned protocol while offering many of the benefits of a decentralized PoS protocol.”

This is done by considering the overall network as a collection of somewhat independent subnets.  Each subnet operates in terms of liveness as a permissioned network dependent upon a centralized PKI (public key infrastructure).   However, the context of these networks is run by a DAO.  That is, the protocol, network topology and PKI are all in the control of the decentralized network.

This enables efficiency of transaction processing without sacrificing safeness.  This is called partial synchrony in the paper, the idea being that each subnet operates for a defined period as a synchronous network.  This allows the subnets to rapidly proceed.  The subnets then participate in asynchronous collaboration to confirm the validity of the progress.  This operates on the explicit assumption that less than ⅓ of the network may be participating in a Byzantine style attack—a common threshold in distributed computing.  The overall asynchronous network is then tuned to preserve safety and resilience in harmony with the subnets being tuned to maximize throughput.

Ongoing innovation

While we’ve covered a lot of ground here, there are other intriguing whitepapers, and more being proposed.  It’s a fascinating space to watch, with some daring and mind-expanding thinking going on.

Blockchain, Emerging Technology