A summary of our discussion during our DoKC Town Hall held in November.
Data on Kubernetes Community (DoKC) Town Halls are an opportunity to bring the community together to meet one another, share end-user journey stories, discuss DoK-related projects and technologies, and stay up-to-date on community events and ways to participate.
The following provides a brief overview, video and transcript for the DoKC Town Hall that was held November 2023.
Batch workloads in multi-tenant environment with Apache YuniKorn
Presented by:
Sunil Govindan, Senior Engineering Manager, Cloudera & member of Apache YuniKorn PMC
Wilfred Spiegelenburg, Principal Software Engineer, Cloudera & member of Apache YuniKorn PMC
In this hands-on workshop, Sunil and Wilfred will introduce Apache YuniKorn, an open-source resource scheduler that redefines resource scheduling on Cloud.
This Town Hall covered the following:
- Enhancements to core scheduling logic to bring high performance, efficient resource sharing, and multi-tenancy-oriented capabilities to the scheduling Spark jobs;
- Management of resource quotas, resource sharing and auto-scaling;
- Optimizations for running Apache Spark workloads on Kubernetes;
- First-hand experiences of running large-scale Apache Spark and other K8s workloads with Apache YuniKorn.
Ultimately, the presentation explains how you can schedule large scale Apache Spark jobs efficiently on Kubernetes in the cloud.
Watch the Replay
Read the Transcript
Download the PDF or scroll to the bottom of this post.
Ways to Participate
To catch an upcoming Town Hall, check out our Meetup page.
Let us know if you’re interested in sharing a case study or use case with the community.
Data on Kubernetes Community
Website | Slack | LinkedIn | YouTube | Twitter | Meetups
Operator SIG
#sig-operator on Slack | Meets every other Tuesday
Transcript
Speaker 1: We are excited to have some special guests from the Apache Unicorn community with us to talk about Apache Unicorn. Apache Unicorn is being used by folks like Lyft, apple, Alibaba, Pinterest, Datadog. So quite a lot of people are using this already. So I’m really excited to have them dig in and tell us all about it. And I will turn it over to you, Sunil and Wilfred.
Speaker 2: So good morning, good evening everyone. It’s so nice to see all of you. I’m Sunil Goen from Cloudera. I lead the Cloudera data Engineering Spark team and the UN unicorn scheduling team for private cloud. So I’m in this big data space for over a decade working on many projects. I contribute to Apache Unicorn in various roles at this point of time. So along with me, I have Wil Wil Fred, how are you?
Speaker 3: Good morning. I’m Wil Spielberg. I’m based in Melbourne, Australia, working on Apache Unicorn, and before that I worked on yarn and spark in the Apache Hadoop environments.
Speaker 2: Thank you, Wil Fred. So in today’s session we’ll be covering the various aspects of create scheduling and how that ecosystem is ready at run various big data workloads in a multi-tenant fashion and how unicorn is actually helping that aspect. So let’s start the session. So I’ll start with the data and its relevance first, the data explosion, it’s basically one of the fundamental piece and around 75 percentage of the data are not quite structured. And Cloudera is managing around 25 exhibits of data in its old HDFS form factor and around 200 plus trillion objects are in S3 as well. Next point please. And along with that, it’s clear that it’s not about the data as well, right? Everyone needs access to the data as well. So we are seeing about 50 x increase in the consumer end and all types of application, be it AI or non-AI, they all need access to this data, but storing and managing data is enough.
Speaker 2: I’ll say no. The reason is that you really get the best value proposition out of the data, otherwise it is meaningless. So for that, in the past decade there were a lot of powerful big data engines that were helping. And in the stock we are going to talk a bit about some of those engines and then more about platform. In the next slide, I would like to take you guys through a detailed use case that we are seeing in the normal data ecosystem. So from the point of view of a data scientist or a data engineer. So they will be building pipelines and this data will be sourced from various destinations and it’ll be able to process this data as well. And finally to give to some kind of a machine learning model. But what matters is the first piece, like you’ll be connecting to various data sources and the scale of that will be enormous.
Speaker 2: And then you’ll be applying some kind of a delta and some updates onto the data that actually you got it. And you’ll be naturally doing a lot of asset transactions updates and deletes and et cetera. And finally you may need to take some kind of backups as well. So auditing and snapshots are very crucial at this point of time. And then some periodic historical analysis. To do that, you will need some kind of time travel kind of feature sets. And finally on this dataset, you will do some predictive analysis using your machine language models and et cetera. So this is a classical use cases that we often see from many of our customers in various phase and these kind of use cases we’re running on the big data platform on-prem, majorly powered by Hadoop pian. And it’s a set of other projects along with that.
Speaker 2: So if you go to the next slide, we will be able to see to use these for these kind of use cases. These are the engines that were helping us. So I’m just picking up very few engines starting with iceberg, which will be helping us to do the time travel and the ies. And underneath for the ingestion we were using NiFi and Flink for some data processing for the classical SQL data warehouse operation. We were using Impala. And of course park was majorly for all the data processing that the transformations and the compaction that you want to do. So these were some of the engines that were helping us to achieve these use cases. And there are many more engines and the customers will use different types of engines as they go along. Now, if these are the engines that are in play, I think let’s look at the past decade and how we actually evolved.
Speaker 2: It started off with the map reviews and eventually on top of that, many, many engines came along and the churn of these engines are so big they’re happening in every other year. And we are actually in a phase where we are seeing L element, right? I think LLM also requires a lot of data processing as well. So these engines are there and some of these engines definitely died of, but majority of the engines are still alive and they’re actually very crucial for processing data from that first slide that I mentioned, that it is still crucial. So the key takeaway, I think what I want to bring it is the data is constant here. Engines changes, platform changes, but the data is constant and usually the new engine, whenever it comes, the platform has to be flexible enough to make sure that they can actually adopt it, right?
Speaker 2: So it was yarn or Mesos kind of ecosystem in the past decade. It moved to Kubernetes towards the end of the last decade and 20 16, 20 17 onwards. And Kubernetes is one of the powerful form factor at this point of time, which helps to make sure that all these different types of mixed workloads can run in a very efficient manner and the maintenance of the DataOps DevOps will become much more easier in the cloud native ecosystem. And the challenges or the actions for us is making sure that we embrace this change of new engines or the changing dynamics of the platform, but at the same time, keeping them open is extremely crucial. If you go and actually put anything back to any specific vendor, it’s very difficult to operate after that. So a key takeaways that create us is very crucial and it is playing a very strong advocate in terms of running these engines in a powerful manner.
Speaker 2: And with Kubernetes, what we are seeing, these are some of the critical advantages. One is like we are able to move from a server architecture to microservice architecture, which makes the maintainability of the services and the engines much easier and it becomes much more modular because earlier it was a monolith and now we are able to cut it into pieces and disaggregated storage and compute. That is one of the key pieces that we felt that it’s the right approach to move towards for a classic hybrid topology. Because in the past, in the old form factor, we were relying on co-located storage and compute and access to the data was always costly because the IOPS was very heavy or network was very costly back in the elastic decade. So we usually send the compute to those storage and do the computation there and process the data, then do the aggregation oversend.
Speaker 2: We no longer need to do that because IOPS is far better with SSDs and network. These pipes are way more wider. There is one of the advantages that we are seeing and more than that we are seeing a good TOC or what I call it as the cost benefits because you can scale these clusters independently, both compute cluster and storage cluster based on the exact demand you can do it. So we are seeing a lot of benefits with that and Kubernetes ecosystem is extremely good in actually onboarding new applications. The cost of bringing a new application was very big in the past. Now it’s very seamless and towards end I think it’s all about upgrades, upgrade of an application. That’s what I mean, it’s very easy to do in Kubernetes. High availability and zero downtime, those kind of aspects are seamless. So with that, I want to ask that one fundamental question here.
Speaker 2: Are the data engines ready to adapt the Kubernetes ecosystem and the cloud native architecture? So this is a fundamental piece because we are seeing definitely a lot of benefits by moving to Kubernetes In the past, the way we are able to now move to a microservice architecture seamlessly, we can now make sure that the dependencies can be easily managed. You can run different versions of the application, which was not quite possible because you may have a Python three dependency, Python two dependency on some application and in one of the node when you run everything together, it is very difficult. You have the Python run times and et cetera, but it was not quite easy to manage other dependencies like say for example, product of 2.5, product of 2.8 or three, et cetera. So it is becoming much easier to manage them without any doubt and we are able to manage the applications in a seamless way with Kubernetes.
Speaker 2: But at the same time, are there any kind of new challenges that we are seeing? So that is one of the concerns that we were having or one, we learned these challenges over time. It took a good amount of time for us to actually unblock some of these and solve it in a way that it has to be solved. Most of the changes that we are done, it is actually available in the open source community. And some of the projects that we started were basically to make sure that these gaps that we identified in the Kubernetes is actually dealt with. So I’ll start with few of them. If you go to the next slide, we wanted to talk about some of the critical challenges that we saw. The first one was do we have the flexibility to manage the resources in a hierarchical queues? And the reason is many different organizations, they have a larger cluster and they actually have different departments and different departments will run different type of use cases and they want to make sure that the resources that are available to each of these queues are properly managed to run maybe a Spark job or maybe a hive job or maybe a Flink or so if there are multiple users, they will make sure that the users can share the cluster sources available for that specific use case.
Speaker 2: The question is, is it possible to do the same thing in Kubernetes, right? And second one, I think I already kind of mentioned about the multi-tenancy aspects of it. How can I attend different or a hundred different Spark users can run different type of data engineering pipeline without conflicting with others and making sure that they get the dedicated quota at any point of time. The easy answer, what we already saw or what we already heard from different customers or users is that yes, it is cloud, just do auto scaling, but at the end of the day it comes with the cost, right? The dollar is always attached to that. So it is not always possible to put a controller on different use cases and manage them independently. The cost of managing and maintaining such a dynamic toggling or the capacity cap was not always easy.
Speaker 2: So that also was actually one of the challenges that we saw. And third one is all about the SLA part. Since it was running with a co-located compute and storage, it was easy to the maximum performance out. So once we desegregated the storage and compute, that means I can access some data from S3 or A BFS or I can access the data from an on-prem object store like Apache. Also on that I click, we open sourced a while back and keep the compute somewhere else. So when we move to this architecture, is it possible to get back the same type of SLA or the performance? And similarly how much I can scale up. It’s not about actually running a 10,000 node cluster or a 5,000 node cluster, but it matters is when I’m running a huge data processing job by the end of the quarter to do some kind of a quarterly business report, I need to go through the N India dataset.
Speaker 2: And that point of time, maybe the data jobs like Spark or Flink jobs will bring up maybe 50,000 containers and that is nothing but 50,000 ports. So how do I able to scale and make sure that each of those ports will do its on same task and then shut it down and pull back the data and then give a successful output back to the job submitter maybe after nine hours or 12 hours if the job is failing and coming back because I cannot actually run hundreds of containers, then it’s a problem. So these kind of scale issues were also a concern. And finally, the cost effective factor that I mentioned, is it possible to manage under the same dollar amount that they were spending in the past? So with these challenges, we felt that most of these are converging to the scheduling area and we felt that that’s an area which needs a lot of effort and emphasis to make sure that it can perform the way that it was performing in the previous ecosystem that is yarn or meso and still run the same engines and get the best out of the Kubernetes while it is giving us new other opportunities.
Speaker 2: So if you could go to the next slide. I’ll just hit on the core part. These are some of the scheduling challenges that we were seeing in Kubernetes and the quotas, they were quite add-on, they’re not part of the scheduler, so we were not able to quite set the quota just like the way I mentioned earlier. We want to set quotas for each queue and that is not quite possible. And the enforcements where at the resource creation, that means if you are not able to find the NF quota for your job, your job will fail. But we want them to be queued in the big data ecosystem. Job queuing is extremely essential because many of the pipelines will be queued up and then you need to run them and the quotas and limits are at the namespace level. That is not enough for us because we are looking for a hierarchical queuing system and the scheduling side.
Speaker 2: Wilford will give a detailed outlook in the upcoming slides, but I will give a very higher level overview and usually in the default scheduler they see the cluster as one large queue. They take all the ports comes in, they do one type of sorting and then they started scheduling it. But when you’re running a multi-tenant, big data type of a workload on a Kubernetes ecosystem and to control queues and quotas and different scheduling policies, we never had the flexibility of getting those kind of options from Kubernetes. And we are seeing that as one of the largest gap today. And so with that, I would like to hand it out Wilfred, who will take us through the unicorn scheduler and how it’s solving majority of these use cases that I mentioned over to you Alfred.
Speaker 3: Thank you Sunil for that one. So based on all the remarks that were given by Sunil, we started working on a unicorn scheduler to do what we thought we needed to get into Kubernetes to move data workflows and data processing seamlessly onto Kubernetes. So the basis that we started off with were saying, okay, what we want to do is we want to make sure that if we run data workloads and batch workloads, that it works seamlessly with Apache Unicorn, even if you run other things on your Kubernetes cluster. So it doesn’t matter if you are running Apache Spark, hi TensorFlow Ray jobs on your cluster, if you also want to do some web servicing or running other things on that same Kubernetes cluster, you don’t want to split the things up and make a break between scheduling data services or data processing and services. The other basic point that we said we want a really simple integration part, we don’t want to go and talk to all these different engines and say, can you make some changes in the engine so that you can work with Apache Unicorn and can get scheduled through us? So we want to say just pick your job up, put it in there, put a couple of extra annotations or labels on whatever you put into Kubernetes and it can be scheduled by unicorn and we will get all the advantages of unicorn.
Speaker 3: Since we came from the yarn side of things, we all said, look, we’ve seen a lot of advantages of what we are doing on the yarn. There’s a lot of people that want to say, I’ve got all these groups. Everybody needs their own quotas. Everybody wants to get their own bits of guaranteed resources on a cluster, so can always get some stuff. But combine that with some people say first in first out, whatever I submit first needs to finish first. Other people say, no, no, everything needs to start and if it’s busy on the cluster, it needs to be shared fairly between the different things that we are running. So we want these different scheduling policies to pop up and that again into the hierarchy that we set. So big thing, workload queuing, data processing is bursty. Sometimes you process petabytes of data in a day and if it’s a Saturday you might not process that much data that same day.
Speaker 3: So you want to have a workload queuing instead of denying access to the cluster if the cluster is full. The other thing is a request that we’ve seen often and we are using his Spark application as an example only because that could be anything is a gang scheduling whatever you run, you know what you want and it can only run if you get a certain amount of resources to start with. So for the Spark application for instance, to run, you must have a driver and a certain amount of executors to run the same with a tensor flow job or if you do array cluster on Kubernetes, you need a service, the array service and you need the work notes for the array to be up. So all of these things are again scheduling. So you want to ask for these resources upfront and be scheduled as give me all or nothing and go on that.
Speaker 3: Hooking back to that first in first out, fair sharing of the clusters. We want to be able to sort all these different applications in different ways based on the location of where they’re running and that needs to be flexible, not one option for the whole cluster. So taking all these things into account, we looked at unicorn and we’ve integrated this into a hierarchical model around quotas and queuing for people that are aware of Mesos and yarn and things like that. There’s similar concepts in all those resource managers and we’ve taken those same things and put them into Kubernetes using unicorn. So resources are managed via queues and what we can do is we can create those queues automatically for you if you don’t want to configure everything from start or you can say no, I want to set up predefined static queues that work and are available all the time.
Speaker 3: Then looking at how we process the data and where the workloads are, the resources are always located in that leave and the hierarchy that sits on top of all those leaves is used to aggregate all the resource usage towards the root of the cube. So pots and other things run in the leaf, the usage of that leave gets aggregated into the parent into the root, and if you’ve got a deeper hierarchy, you can make it as deep as you want. It works the way through all these levels in the hierarchy and move through that. There’s two kinds of things because we already said we’ve got workload queuing, so requests are pending the allocations that are really assigned to the note and we keep track of both of these things and we use them during the scheduling cycle to make sure that we either do it, share fairly, do first in first out or give everybody what they want on the system.
Speaker 3: Limiting what’s running is also part of that same hierarchical structure. So a quota that I set up is defined on whatever level you want within that hierarchy. So I can set a quota at the leave, I can set a quota on a parent or I can combine quotas being set on both leave and the parent. And that can be done not just purely at the queue level, but we can even split it out as far as saying we’ve got one user that runs information in one leave and it’s that user is limited in that leave to a certain amount of quota or that same user can be defined at the parent and say no, everything that it runs there is limited. So again, from a multi-tenancy perspective and from a multi-user perspective, we can allow you to run all these things and make sure that everybody is sticking within its quota and within its setup, everything completely enforced throughout that whole hierarchy.
Speaker 3: So let’s see how that would work. In a simple setup that I’ve got, I’ve got two groups within my company, within a single company development and production. I’ve got two teams under development and a team in sales. My first team runs a couple of pots and under development a couple of salespeople are working and doing some data processing. So we’re running all that stuff. The usage of all these teams gets aggregated into the parents and overall I use a certain amount of resources at the root of that queue and the root could be the company and above that we could have multiple other layers again to get you your multi-tenancy setups and then your multi-tenant clusters created. So I’m going to set some resource limits on different points. The root we don’t limit because that’s the size of the cluster, so that’s automatically done for you, but the development team is never allowed to use more than 60 gigabyte and four CPUs.
Speaker 3: So they’re already using 20 and now the second team comes up and they want to start up a huge bot that is going to use more than the quota that is available because the user usage is already accounted for. We’ve got a resource limit and since the port is larger than the quota that is available to the team, it just sits there and waits up until the quota is coming available. So it doesn’t get denied, the job gets accepted, it just gets queued up, up until the point that we need to go and really schedule it.
Speaker 3: That all works nicely, but there’s still a number of other things that we need to get to make this workable in a multi-user, multi-tenant kind of setup. So on a Kubernetes level we have preemptions, we have priorities and preemption and priority is used by the scheduler to make sure that things run when they should be running and that’s the only thing that you’ve got. Priority defines the scheduling cycle. You set a priority on a PO and whatever has the highest priority gets scheduled first, but there’s already a remark within the Kubernetes community even itself it says if you’ve got a rogue user it could create a pot with the highest priority and it would get scheduled before anything else. There’s nothing you can do about that unless you start doing really deep inside of the Kubernetes things and start working your way through all kinds of special settings to even get a little hold of that.
Speaker 3: Preemption is the other part of that same coin. Priorities are used to schedule things, but it’s also used in preempting and making, which is used to make room for high priority Other things, scheduling is an opt-in during the scheduling cycle and you say, is this bot allowed to preempt some other things that are running in the cluster to make room? It doesn’t look at what does it cost if I kill that other thing in the cluster. And if you are for instance looking at spark jobs or other jobs that have got a manager kind of a setup, if you kill the manager of the job killing one pot has got real big flow on effect that is far larger than whatever killing that one pot costs you. So that impact, that knowledge is not there. The scheduler just says priority ranking, high priority and you can’t make sure that a pot is not preempted if it’s already running. It’s just not possible. So why is preemption mission critical in batch workloads and why does that little bit in Kubernetes not work for batch processing or data load processing?
Speaker 3: So there’s multiple things there. What we’ve done is we’ve said, okay, we are looking at when are we want to use preemption and how is that going to affect our data processing workloads. So we want to use preemption to make room for things when we’ve got elastic queues because we’ve borrowed something because somebody has started up a huge data processing workload and nobody else was using the cluster. So it’s taken way more than it should have been. It’s still on its max, but it’s taken way more than what it should have been using under normal workloads.
Speaker 3: So in that case you want to PMT to get it back if other people want to use the cluster, get it back to whatever is normal for that workload to run. So in the case of over subscription, we want to be able to burst out but go back into that guaranteed amount of resources that we’ve got. So with static use, we can’t do that. Kubernetes thinks of a cluster as being one big static queue. So it’s always having a problem with under utilization and if you have got your quotas and everything set up on a Kubernetes level to not have under utilization, your cluster will be far larger than what you need or your cost could spiral all the time. So elastic use gives you the best of both things, keeping your cost within checks and balances but still being able to burst out. But we need preemption for that. And when you look at batch workloads, if you look at something that doesn’t use preemption and I start up an application one, it uses everything in the cluster that is there and the next data processing job comes up. Application two, it has to wait up until the first application releases the resources and then it can run up until the end.
Speaker 3: If you do use preemption, then you can say, oh, that application one can burst out of and use the whole cluster for as long as there’s nobody else asking for anything. But then when the second application comes up, we’ll start sharing fairly and will take the resources down to whatever is guaranteed or what should be guaranteed to be available for that. Yeah, it might lengthen the runtime for application one slightly, but in the end you’ll get your processing sorted and if you’ve got more complex situations that you’ve got multiple queues, multiple applications, you’ll get a better throughput and not as blocked every single time that you’re sticking with one application done second application. So it’s not going to be a single flow through, but you’ll have multiple things going through the same time. So from a preemption side on the unicorn we looked at what does data processing needs.
Speaker 3: So we want to be able to say a will limit you. You can never go above a certain limit, but we will also give you a guaranteed amount of resources so that if you run your normal processing which should fit within your guarantee, then you will always have that available and everything else will be shifted aside for you to make sure that you can stick to your SLAs. So my guarantee on my queues help me to stick to my SLAs in the long run. Preemption helps us to make sure that if something has burst over, its guaranteed that it will be brought back towards it guaranteed resources. On the other hand, if something pops up that didn’t get its guaranteed, it’ll help that workload get towards its guaranteed. So we’ll do that within the hierarchy. But as we see in the next slide, it’s also important to make sure that we shield bits and pieces from this.
Speaker 3: We do that in two ways. We allow you to do fencing and we do allow you to do some priority adjustments in there. The other thing that we also said with preemption is we need to make sure that we are application aware. So take the example from Spark, we’ve got a driver with executors or TensorFlow, which has got a master and workers, then we need to make sure that we do not kill that master or the driver kind of setup. So let’s see how that would look like in a slightly larger setup with a bit more of an hierarchical organization. So what we’ve said is if we’ve got workloads running within a certain part of that structure, we want to be able to fence that off from the rest. So that means that whatever happens within sales, it isn’t allowed to go outside of that fence and pick resources from marketing or from the system and whatever.
Speaker 3: We say the same thing, but we don’t want developers to take resources from the production kind of setup. So preemption can’t go outside of these fences. Similar setup we do for the marketing side of things. So we don’t want marketing to pick resources from sales and again that not from product, but on the other end we’ve got things that are running and that are system critical which might need to be used for finance or something else. We want to be able to have those workloads pick up anything from other places. So preemption works on the guaranteed resources. I’ve got simple setup over guaranteed, below guaranteed. And let’s see what happens if I now submit a high priority workload that needs to do some sales figures analysis in production and I put that into the production work queue and then we say, oh, because that queue is under, its guaranteed, it will get some resources out of the dev group because they are using more than what they should be doing and we preempt whatever was lower priority and we give it to that production job to make sure that it runs on time.
Speaker 3: And because this is fenced within sales, I can’t go and pick up workloads from the marketing side of things even though that queue is using more than it’s guaranteed, I can’t go and pick that in those resources from that side. So that’s done. And now the developers on the marketing side support a high priority workload because they said, oh, we need to do this for our new release. We don’t have our guaranteed workload, so let’s see what we can get. And because the development group is fenced and it can’t take resources from the production team says, no, sorry, I can’t preempt anything for you. It means that you just have to wait up until resources become available in the long run. And now last option, we submit something high priority workload on the system side of things and because it’s under guaranteed, it will go and find something somewhere that it’s over. It’s guaranteed we’ve already done some PMing here, so that’s all done. But now it says, okay, give me some of the data or the resources that are there on the marketing side. So the redistribution without problem, and this can be extended again and not just being sales and marketing. It could be company one or company two within the same cluster. So your multi-tenancy will be covered by this use case.
Speaker 3: What we’ll show in a little bit of live demo, I think I’m got two minutes or so for that and that should be enough. We’ve got unicorn running in a kind cluster. We’ve got simple three notes, hierarchical queues, and what we’ll see is we’ll see some quota quotas and p tion working in that setup. So in the background, I’ve already started my unicorn cluster and this is the unicorn ui. I’ve created two Qs root high and rule low. I currently don’t have any usage in the queues. It’s completely empty the cluster. I’ve got no applications running and I’ve got the three nodes as defined before that.
Speaker 3: Okay, let’s start the setup. So this is what a configuration looks like. It’s all yaml, it’s config maps, it’s what we’ve said before, light touch. We don’t need operators, we don’t need CRDs, we don’t need anything like that. It’s all a config map with simple configurations in there. So q config with the specific properties for priorities and preemption created, we’ve created some priority classes and again, we do the light touch, we use the standard priority classes from Kubernetes. We put an annotation on there. So instead of having to create our own Kubernetes objects, we leverage the existing Kubernetes objects to do all this. And what we’re going to submit is a really simple job that will start up 10 containers. They’ll all each use their resources and we’ll distribute them over the two different queues. We even specify some user information which we could use later on to show and look at quotas per user. And so let’s make the first application, so this is just running a cube control with a standard pot creation in the background as you can see in here or refresh the ui.
Speaker 3: We’ve now got one application in running. We’ve got eight bots running because what was happening is we’ve got the queues set up and we had a quota set up, a maximum quota on the queue of eight CPUs and we’re using 7.2 CPUs so we can’t start up more pods because the rest of the pods that are available and weigh and pending on this application are all using 900 milli CPUs. So they can’t be started. These are allocated and we’ve got resources pending. We’ve got two pots sitting there and pending because we submitted 10 pots, but only eight have been allocated.
Speaker 3: If I now submit a second application with a high priority ports, I will see that I’ve got some usage now in the root high because there was still a little bit of space left in the quota, but currently I’m getting stuck at these two pots that are running, but I haven’t reached my guarantee quota yet. That’s 6,006 CPUs and I’m only using 1.8 CPUs, so the rest is still waiting there. Over time, we should be seeing this change because we say, okay, let’s not pm everything immediately from the scratch. Let’s wait a little bit and see what happens if we get resources that become available. If it takes too long we’ll start preempting, but if for, that’s about 30 seconds. So I think I’ve talked for more than enough or 30 seconds. And what you now see is that we are running from way more than the two pots, so we’re running more pots, we’re running six, which gives us five and a half CPUs, which is pretty close to what the guaranteed was that we’ve had before. And what you’ll also see is the low queue has gone down and the HighQ has come up. So we are now approximately to where we should be with our guaranteed, but we haven’t gone over it.
Speaker 3: How high did we go up? We went up as high as we could on the low Q side. We have gone down from using more than seven to only three point something CPUs. We stuck there because we wouldn’t want to go further than that. If I just submit another 10 pots in that same low priority queue, I will see that refresh better. I’ll see that my pending resources have gone up, although there’s way more pending than what my maximum resources would allow. I still accept all the pots, everything gets into the system and whenever the time comes that there’s enough resource available, we’ll get through and things will start. I think that’s it from our side. So if you want to get in contact with us, we’ve got part of the Apache Foundation. Talk to us visitors on the website. We’ve got a email channel and we’ve got a unicorn slack. All this information is also available on unicorn, the apache.org. Reach out to us. We are open, we welcome everybody. We try to be responsive. The whole community tries to be responsive within hours of asking questions and things like that. Thank you.
Speaker 1: Yeah, thank you both. That was fantastic. Thank you all for joining. We really appreciate it. Thanks again to Sunnel and Wilford for joining. We really appreciate the information about Apache Unicorn. Hope to see you all again.
