Apache Software Foundation Announces Apache® YuniKorn™

Wilmington, DE, May 16, 2022 (GLOBE NEWSWIRE) — The Apache Software Foundation (ASF), the all-volunteer developers, stewards, and incubators of more than 350 open source projects and initiatives, today announced Apache® YuniKorn™ in as a High Level Project (TLP).

Apache YuniKorn is a standalone, cloud-native big data and machine learning resource scheduler for batch jobs and long-running services on large-scale distributed systems. The project was originally developed at Cloudera in March 2019, entered the Apache incubator in January 2020, and graduated as a high-level project in March 2022.

“The Apache YuniKorn community works together to solve cloud resource planning challenges,” said Weiwei Yang, vice president of Apache YuniKorn. “It’s really great to see the Apache Way shine through YuniKorn’s incubation process. We’re lucky to have such an open, collaborative and diverse community that is friendly and cares about everyone’s success. motivates us to continue to evolve and improve.” every day.”

Apache YuniKorn natively supports big data application workloads and mixed workloads, and provides a unified, cross-platform scheduling experience. Features include:

  • Cloud native — runs on-premises and in a variety of public cloud environments; maximizes resource elasticity with better throughput.
  • Hierarchical resource queues — efficiently manage cluster resources; provides the ability to control resource consumption for each tenant.
  • Application-aware scheduling — recognizes users, applications, and queues; schedules based on submission order, priority, resource usage, etc.
  • Job scheduling — built-in robust scheduling capabilities; supports inter-queue preemption based on fairness, hierarchies, pluggable node sorting policies, preemption, etc.
  • Central management console: monitors the performance of different tenants; the single dashboard tracks resource usage for managed nodes, clusters, applications, and queues.
  • Efficiency – reduces resource fragmentation and proactively triggers scaling; the elasticity of the cloud reduces overall operational costs.

Additionally, the project announced the release of Apache YuniKorn v1.0, the fifth update since entering the Apache incubator. Improvements include:

  • Decreased memory and CPU usage
  • Extensive information on metrics and diagnostics
  • New deployment model supporting future upgrades
  • Technical overview of the plugin deployment mode

Optimized to run Apache Spark on Kubernetes (open source software container orchestration system), Apache YuniKorn’s performance makes it an optional replacement for the default Kubernetes scheduler. Apache YuniKorn excelled in benchmark tests with other schedulers in the categories of resource sharing, resource fairness, preemption, gang scheduling, and binpacking, with throughput exceeding 610 allocations per second over 2000 knots.

YuniKorn is used at Alibaba, Apple, Cloudera, Lyft, Visa, and Zillow, among others.

“We are delighted to see this offer come to fruition. Apache YuniKorn powers Apache Spark workloads for Cloudera Data Engineering (CDE), a key Kubernetes-based service supporting the Cloudera Data Platform,” said Vinod Kumar Vavilapalli, Senior Director of Engineering at Cloudera and former PMC president of Apache Hadoop. “As part of Cloudera’s public and private cloud offerings, Apache YuniKorn adds tremendous flexibility and control when performing large-scale analytics, allowing customers to better optimize the performance and value of their deployments. “

“Apache YuniKorn is an essential infrastructure service for bringing big data/ML workloads to the cloud,” said Chunde Ren, engineering manager at Alibaba Big Data Open-source team. “YuniKorn brings better scheduling, performance, elasticity, and ease of use capabilities for running workloads on Kubernetes, especially for Big Data and Machine Learning workloads, which benefits many users on the cloud. It is a great pleasure for us to have participated in the YuniKorn community since its inception and to see it grow into a high-level project.”

“Apache YuniKorn is becoming a popular choice for those looking to run big data workloads on Kubernetes, with more and more use cases in development,” Yang added. “We invite anyone who wants to join the YuniKorn community and work with us to solve these difficult problems.”

See Apache YuniKorn in action at Kubernetes Batch + HPC Day Europe (May 17, 2022 in Valencia, Spain https://sched.co/10F0t) and Spark AI Summit 2022 (June 27-30 in San Francisco and online https: //databricks.com/dataaisummit/north-america-2022/agenda/?sessionid=1388).

Availability and supervision

Apache YuniKorn software is released under the Apache v2.0 license and is overseen by a self-selected team of active contributors to the project. A Project Management Committee (PMC) guides the day-to-day operations of the project, including community development and product launches. For downloads, documentation, and ways to get involved with Apache YuniKorn, visit https://yunikorn.apache.org/ and https://twitter.com/YuniKorn_Sched .

About Apache Incubator

The Apache Incubator is the primary path of entry for projects and codebases wishing to be part of the Apache Software Foundation’s efforts. All code donations from external organizations and existing external projects come into ASF through the incubator to: 1) ensure that all donations comply with ASF legal standards; and 2) develop new communities that adhere to our guiding principles. Incubation is required for all newly accepted projects until further review indicates that the infrastructure, communications, and decision-making process have stabilized in a manner consistent with other ASF projects successful. Although the incubation status does not necessarily reflect the completeness or stability of the code, it does indicate that the project has not yet been fully approved by the ASF. For more information visit http://incubator.apache.org/

About the Apache Software Foundation (ASF)

Founded in 1999, The Apache Software Foundation is the world’s largest open source foundation, managing over 227 million lines of code and providing over $22 billion worth of software to the public at 100% free of charge. The ASF all-volunteer community has grown from 21 original founders overseeing the Apache HTTP Server to over 820 individual members and 200 project management committees who have successfully led over 350 Apache projects and initiatives in collaboration with over 8 400 members through ASF’s meritocratic process known as “The Apache Way”. Apache software is an integral part of nearly all end-user computing devices, from laptops and tablets to mobile devices in businesses and mission-critical applications. Apache projects power most of the Internet, manage exabytes of data, perform teraflops of operations, and store billions of objects in virtually every industry. The commercially friendly and permissive Apache v2 license is an open-source industry standard, helping launch billion-dollar companies and benefiting countless users around the world. ASF is an American nonprofit 501(c)(3) charitable organization funded by individual donations and corporate sponsors, including Aetna, Alibaba Cloud Computing, Amazon Web Services, Anonymous, Baidu, Bloomberg, Capital One, Cloudera , Comcast, Confluent, Didi Chuxing, Facebook, Google, Huawei, IBM, Indeed, LINE Corporation, Microsoft, Namebase, Pineapple Fund, Red Hat, Replicated, Talend, Target, Tencent, Union Investment, VMware, Workday and Yahoo!. For more information visit http://apache.org/ and https://twitter.com/TheASF

© The Apache Software Foundation. “Apache”, “YuniKorn”, “Apache YuniKorn” and “ApacheCon” are either registered trademarks or trademarks of Apache Software Foundation in the United States and/or other countries. All other trademarks and registered trademarks are the property of their respective owners.

# # #