{"id":7027,"date":"2021-06-21T13:28:03","date_gmt":"2021-06-21T11:28:03","guid":{"rendered":"https:\/\/tekmart.co.za\/t-blog\/?p=7027"},"modified":"2021-06-21T13:30:06","modified_gmt":"2021-06-21T11:30:06","slug":"how-long-until-cloud-becomes-the-preferred-environment-to-run-hpc-workloads","status":"publish","type":"post","link":"https:\/\/tekmart.co.za\/t-blog\/how-long-until-cloud-becomes-the-preferred-environment-to-run-hpc-workloads\/","title":{"rendered":"Running HPC workloads?-how long until cloud becomes the preferred environment."},"content":{"rendered":"Reading Time-approximately:<\/span> 5<\/span> minutes<\/span><\/span>\n

The high-performance computing market is growing, but enterprises that have already made large investments in on-premise HPC will need to weigh up whether a move to the cloud would be worth their while.<\/strong><\/h2>\n\n\n\n
\"Sooraj<\/figure>\n\n\n\n

By Sooraj Shah<\/a><\/p>\n\n\n\n

Enterprise spending on high-performance computing (HPC) has skyrocketed over the past decade. According to Intersect360 Research, a 10-year growth streak from 2009 to 2019 culminated in global HPC market revenue reaching $39bn in 2019.<\/p>\n\n\n\n

The Covid-19 pandemic is expected to have stunted the market\u2019s growth somewhat, with projections suggesting spending will have contracted by 3.7% once the final figures for 2020 are in.<\/p><\/blockquote><\/figure>\n\n\n\n

But according to MarketsandMarkets, this is not expected to last. It predicts the market will grow to $49.4bn by 2025, at a compound annual growth rate (CAGR) of 5.5%, fuelled by increasing demand from enterprises for more efficient and scalable computing resources. It also says there is an emerging need for high-speed data processing with accuracy, and highlights the adoption of HPC in the cloud as a driver for uptake.<\/p>\n\n\n\n

It remains to be seen, however, whether enterprises are investing in cloud-based or on-premise HPC, in terms of both upgrades and migrations.<\/p>\n\n\n\n

Companies thinking about moving their HPC workloads to the cloud may have concerns similar to those that enterprises had 10 years ago when weighing up whether or not to move their more conventional workloads off-premise \u2013 especially if they had already invested heavily in building out their on-premise HPC infrastructure estate.<\/p>\n\n\n\n

The difference in cost for on-premise HPC compared with on-premise IT infrastructure is huge. Enterprises could spend up to $200m on their HPC infrastructure, and regular updates mean costs can continue to mount after the initial deployment.<\/p>\n\n\n\n

The dilemma they face is whether they should continue investing in an environment on which they have already spent a lot of money, or make use of cloud computing in conjunction with their existing environment. Or should they switch to a fully cloud-enabled HPC environment?<\/p>\n\n\n\n

Cloud switching costs too high for AstraZeneca<\/strong><\/h3>\n\n\n\n

For\u00a0pharmaceutical giant AstraZeneca, switching to a cloud-based HPC system is off the cards for the foreseeable future for cost reasons, despite the company tapping into cloud for some of its high-performance workloads.<\/p>\n\n\n\n

\u201cOur HPC platforms are on-premise, but we do use HPC in the public cloud, especially if we\u2019ve got short, sharp workloads that need GPU [graphics processing unit] or FGPA [field programmable gate array] and so we do use those kinds of workload in the public cloud as well, simply because sometimes you need a bigger scale than we are able to offer within our own locations,\u201d says Scott Hunter, the company\u2019s global infrastructure services director.<\/p><\/blockquote>\n\n\n\n

In other words, AstraZeneca has opted for \u201caugmenting on-premise with cloud\u201d in its HPC environment. It extends its\u00a0Kubernetes\u00a0platform out to various public clouds to do this, says Hunter, which gives it consistency while allowing it to take advantage of what the cloud has to offer.<\/p>\n\n\n\n

\u201cIf we wanted to run those in GPUs on A100s from Nvidia, then we\u2019d use Google or Oracle for that, and if we wanted to use FPGA, then it would be [Microsoft] Azure or AWS [Amazon Web Services], and if we wanted to get the ultimate scale, we would launch all-in on Azure because even though its capability might not be the fastest, it seems to have the largest amount of available GPU,\u201d says Hunter.<\/p>\n\n\n\n

A recent HPC user survey by Intersect360 declared AWS the top-rated cloud computing platform in every category but one \u2013 including technical and operational impression, future outlook and likeliness to use in two years.<\/p>\n\n\n\n

However, AstraZeneca\u2019s intention is not to move to a cloud-based HPC, purely for cost reasons.<\/p>\n\n\n\n

\u201cOn-premise is between five and seven times cheaper than doing a similar type of activity in the cloud,\u201d says Hunter.<\/p>\n\n\n\n

The reason why there is such a high cost associated with cloud-based HPC for AstraZeneca is that it works with a lot of double-precision chemistry workloads, which tend to be operational 24 hours a day, 365 days a year.<\/p>\n\n\n\n

\u201cThat means that unless we get a cracking deal from one of the major public cloud vendors or Oracle, it makes sense to run on-premise because of the cost,\u201d says Hunter.<\/p>\n\n\n\n

Formula One team\u00a0McLaren Racing\u00a0is also currently using the model of on-premise HPC with cloud usage for certain situations.<\/p>\n\n\n\n

\u201cCloud is pretty important, but performance and latency are equally important, so while we spin up vast amounts of services in well-known cloud provider environments, we have to rely on HPC systems that are on-premise, so there\u2019s a balance for us between both,\u201d says Ed Green, principal digital architect at McLaren Racing.<\/p>\n\n\n\n

The suggestion is that high-performance workloads will work more effectively running on the on-premise systems built for them. But that doesn\u2019t mean McLaren Racing will not look at using more cloud-based HPC, particularly when cars are on the track.<\/p>\n\n\n\n

\u201cThe workloads are fairly static during a race week and then the minute the cars come out on the track, you start seeing a lot more data coming through and workloads start to increase as well,\u201d says Green. \u201cSo it can be really interesting going forward looking at new models in those areas.\u201d<\/p>\n\n\n\n

From on-premise to cloud-based HPC<\/strong><\/h3>\n\n\n\n

While many CIOs are content with their on-premise HPC environment, this is not necessarily the case in every organisation.<\/p>\n\n\n\n

Elaine Neeson, CIO at the Children\u2019s Cancer Institute in Australia, says that when the organisation hired a biomathematician, he required the use of high-performance computing. Neeson and her team decided to build this on-premise with Dell.<\/p>\n\n\n\n

\u201cWe realised that it was never going to work on-premise as we wanted it to, and we would have to keep sinking quite a lot of funds into it, which wasn\u2019t really feasible,\u201d she says.<\/p>\n\n\n\n

As the institute had already been using Microsoft Azure for virtualisation and Office 365, it decided to pilot the use of Azure for the high-performance workloads, with a particular focus on ensuring that costs would not spiral as a result of switching to the cloud.<\/p>\n\n\n\n

\u201cWe didn\u2019t want the team spinning up huge resources and having them on 24\/7 if they weren\u2019t using them because the cost would have just blown up,\u201d says Neeson. \u201cSo the six-month pilot gave us a taste of how we could use the cloud efficiently.\u201d<\/p>\n\n\n\n

The biomathematician\u2019s team now carries out 90% of its work in the cloud.<\/p>\n\n\n\n

Migrating from Dell\u2019s on-premise HPC to Microsoft Azure was not an easy switch to make \u2013 and a big reason for that is internal resourcing and skills, which also hindered the institute with the on-premise HPC initially.<\/p><\/blockquote>\n\n\n\n

\u201cWe\u2019re a Windows shop and most of our laptops are Windows, although our researchers use Macs, but HPC is all run off Linux and that was a huge challenge for us \u2013 we still struggle to get those specialised skills internally,\u201d says Neeson.<\/p>\n\n\n\n

She puts this down to the competitive market in Australia, where the institute would have to compete with the likes of Macquarie Bank and other huge corporates for skilled staff.<\/p>\n\n\n\n

\u201cLinux was one of the biggest upskilling programmes we had to do internally because we couldn\u2019t find a skilled Linux person to come in,\u201d she says. \u201cThe cloud actually made that a little bit easier because it\u2019s all very much plug and play in a lot of regards, so moving to get off-premise did have a lot of advantages.\u201d<\/p>\n\n\n\n

The Children\u2019s Cancer Institute still has not turned off the on-premise HPC, but its end of life is in February 2021.<\/p>\n\n\n\n

\u201cAt the moment, there is still a little bit of work the researchers need to do on a couple of the analysis pipelines they use to enable them to be used in the cloud,\u201d says Neeson. \u201cSo they\u2019re working through that now and until they get that switched over, we will keep on-premise as a backup.\u201d<\/p><\/blockquote>\n\n\n\n

Neeson acknowledges that she pushed for her team to work quickly to migrate to cloud-based HPC, but this is not always possible for other enterprises, particularly those that have invested even more into on-premise HPC, or those with far bigger workloads.<\/p>\n\n\n\n

Cloud-based HPC accelerating<\/strong>.<\/h3>\n\n\n\n

It is clear the overall HPC market has plenty of room to grow \u2013 and is being fuelled by demand for both on-premise and cloud deployments.<\/p>\n\n\n\n

A report by Hyperian Research says the on-premise HPC market was worth $24bn in 2020 and will grow at 7% a year until 2024, while the HPC cloud market was worth $4.3bn in 2020 and will grow at 17% a year until 2024.<\/p>\n\n\n\n

This suggests that, over time, the HPC cloud market will overtake the on-premise HPC market, but that shift will take time.<\/p>\n","protected":false},"excerpt":{"rendered":"

Reading Time-approximately:<\/span> 5<\/span> minutes<\/span><\/span>The high-performance computing market is growing, but enterprises that have already made large investments in on-premise HPC will need to weigh up whether a move to the cloud would be worth their while. By Sooraj Shah Enterprise spending on high-performance computing (HPC) has skyrocketed over the past decade. According to Intersect360 Research, a 10-year growth streak from 2009 to 2019<\/p>\n

Read More<\/a><\/p>\n","protected":false},"author":112,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[155,38,218,17,221,142,35,8,4,30,3,84],"tags":[],"class_list":["post-7027","post","type-post","status-publish","format-standard","hentry","category-batting-for-tech-in-the-covid-19-times","category-best-practices-for-data-center-operations","category-ceo-quick-reads","category-cio-resources","category-computer-storage-hardware","category-covid-19-organization-mitigation-responses","category-data-center-facilities","category-data-center-hardware","category-datacenter-news","category-expert-advise-and-opinion","category-industry-news-and-expert-advise","category-trends-datacenter-hardware-and-solutions"],"_links":{"self":[{"href":"https:\/\/tekmart.co.za\/t-blog\/wp-json\/wp\/v2\/posts\/7027","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/tekmart.co.za\/t-blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/tekmart.co.za\/t-blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/tekmart.co.za\/t-blog\/wp-json\/wp\/v2\/users\/112"}],"replies":[{"embeddable":true,"href":"https:\/\/tekmart.co.za\/t-blog\/wp-json\/wp\/v2\/comments?post=7027"}],"version-history":[{"count":2,"href":"https:\/\/tekmart.co.za\/t-blog\/wp-json\/wp\/v2\/posts\/7027\/revisions"}],"predecessor-version":[{"id":7030,"href":"https:\/\/tekmart.co.za\/t-blog\/wp-json\/wp\/v2\/posts\/7027\/revisions\/7030"}],"wp:attachment":[{"href":"https:\/\/tekmart.co.za\/t-blog\/wp-json\/wp\/v2\/media?parent=7027"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/tekmart.co.za\/t-blog\/wp-json\/wp\/v2\/categories?post=7027"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/tekmart.co.za\/t-blog\/wp-json\/wp\/v2\/tags?post=7027"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}