Fair Scheduler features and conversion details

Certain Fair Scheduler properties cannot be auto-converted by the fs2cs conversion utility. Review the list of these properties and if they are supported in Capacity Scheduler and by Queue Manager UI to learn how you can configure them.

Migrating to CDP Private Cloud Base 7.1.6 or higher

Table 1. Queue resource-quota related features
Property	Description	Conversion information
`<minResources>`	Minimum resources the queue is entitled to.	Partially supported in Capacity Scheduler. Ignored by the fs2cs conversion utility. Not supported by Queue Manager UI.
`<maxResources>`	Maximum amount of resources that will be allocated to a queue.	There is an equivalent feature in Capacity Scheduler. Ignored by the fs2cs conversion utility. For each queue, max-capacity will be set to 100%. Supported by Queue Manager UI. For more information, see Set user limits within a queue.
`<maxChildResources>`	Maximum amount of resources that can be allocated to an ad hoc child queue.	There is an equivalent feature in Capacity Scheduler. Ignored by the fs2cs conversion utility. Its value can be two distinct percentages (vcore/memory) or an absolute resources, but the leaf-queue-template only accepts a single percentage. Supported by Queue Manager UI.

Table 2. Global scheduling settings
Property	Description	Conversion information
`yarn.scheduler.fair.max.assign`	Dynamic maximum assign	There is an equivalent feature in Capacity Scheduler. Fine-tuning of the following three properties are required: `yarn.scheduler.capacity.per-node-heartbeat.multiple-assignments-enable` `yarn.scheduler.capacity.per-node-heartbeat.maximum-container-assignments` `yarn.scheduler.capacity.per-node-heartbeat.maximum-offswitch-assignments` Not supported by Queue Manager UI.
`yarn.scheduler.fair.user-as-default-queue`	User as default queue	Relative mode:A placement rule needs to be created. Create a queue, such as "root.users" and enable Dynamic Auto Child Creation for it (make it a Managed Parent Queue). Create the following placement rule: `{ "type":"user", "matches":"*", “parentQueue”: “root.users”, "policy":"user", “create”: true, "fallbackResult":"skip" }` The following limitations apply: It is not possible to have "root" as a parent for dynamically created queues. "root.users" queue cannot have static leafs. Those are queues that always exist and are created manually. For information about weight mode see Auto-converted Fair Scheduler properties. Supported by Queue Manager UI.

Table 3. Preemption
Property	Description	Conversion information
`yarn.scheduler.fair.preemption.cluster-utilization-threshold`	The utilization threshold after which preemption kicks in.	There is an equivalent feature in Capacity Scheduler: `yarn.resourcemanager.monitor.capacity.preemption.max_ignored_over_capacity`. It specifies the resource usage threshold over its configured capacity that a queue must meet before it is eligible for preemption. Supported by Queue Manager UI.
`minSharePreemptionTimeout`	The number of seconds the queue is under its minimum share before it will try to preempt containers to take resources from other queue.s	Not supported in Capacity Scheduler.
`fairSharePreemptionTimeout`	The number of seconds the queue is under its fair share threshold before it will try to preempt containers to take resources from other queues.	Partially supported in Capacity Scheduler. This can be achieved by using the following configurations together: `yarn.resourcemanager.monitor.capacity.preemption.natural_termination_factor` `yarn.resourcemanager.monitor.capacity.preemption.max_wait_before_kill` Supported by Queue Manager UI.
`fairSharePreemptionThreshold`	The fair share preemption threshold for the queue.	Partially supported in Capacity Scheduler. This can be achieved by using the following configurations together: `yarn.resourcemanager.monitor.capacity.preemption.natural_termination_factor` `yarn.resourcemanager.monitor.capacity.preemption.max_wait_before_kill` Supported by Queue Manager UI.

Table 4. Placement rules
Fair Scheduler placement rules	Description	Conversion information
`create="false" or "true"`	Disable or enable creating a queue dynamically in YARN. This option can be specified on all rules.	Partially supported in Capacity Scheduler. Use the Capacity Scheduler Dynamic Queue Mappings policies: `u:%user:[managedParentQueueName].[queueName]` `u:%user:[managedParentQueueName].%user` `u:%user:[managedParentQueueName].%primary_group` `u:%user:[managedParentQueueName].%secondary_group` Supported by Queue Manager UI.
`<rule name="specified"/>`	If a user has submitted the application by specifying a queue name (other than the “default” queue), then this rule will be successful. Hence the remaining set of rules won't be executed.	Not supported in Capacity Scheduler.
`<rule name="primaryGroupExistingQueue"/>`	If submitted user’s(userA) primary group name (groupA) exists, submit to groupA.	There is an equivalent placement rule in Capacity Scheduler: `<value>u:%user:%primary_group</value>` Supported by Queue Manager UI.
`<rule name="secondaryGroupExistingQueue"/>`	If submitted user’s(userA) secondary group name (groupA) exists, submit to groupA.	There is an equivalent placement rule in Capacity Scheduler: `<value>u:%user:%secondary_group</value>` Supported by Queue Manager UI.
`<rule name="nestedUserQueue">`	Depending on the nested rule, this places the job to the following queues: `root.[primaryGroup].[userName]` `root.[secondaryGroup].[userName]` `root.[queuePath].[userName]`	Not supported in Capacity Scheduler.
`<rule name="default" queue=”qName”/>`	Fall back policy by which rule will fall back to queue named in the property ‘queue’ or the “default“ queue if no queue property is specified (if all matches fail).	There is an equivalent placement rule in Capacity Scheduler: `<value>u:%user:default</value>` Supported by Queue Manager UI.

Migrating to CDP Private Cloud Base 7.1.5 or lower

Table 5. Queue resource-quota related features
Property	Description	Conversion information
`<minResources>`	Minimum resources the queue is entitled to.	Partially supported in Capacity Scheduler. Ignored by the fs2cs conversion utility. Not supported by Queue Manager UI.
`<maxResources>`	Maximum amount of resources that will be allocated to a queue.	There is an equivalent feature in Capacity Scheduler. Ignored by the fs2cs conversion utility. For each queue, max-capacity will be set to 100%. Supported by Queue Manager UI.
`<maxChildResources>`	Maximum amount of resources that can be allocated to an ad hoc child queue.	There is an equivalent feature in Capacity Scheduler. Ignored by the fs2cs conversion utility. Its value can be two distinct percentages (vcore/memory) or an absolute resources, but the leaf-queue-template only accepts a single percentage. Supported by Queue Manager UI.
`<schedulingPolicy>`	Scheduling policy of a queue (for example, how submitted applications should be ordered over time). .	There is an equivalent feature in Capacity Scheduler. Manual fine tuning might be necessary. note if DRF is used anywhere in Fair Scheduler, then the converted configuration utilizes DRF everywhere and it is not possible to place a queue with “Fair” policy under one which has “DRF” enabled. Supported by Queue Manager UI.

Table 6. Global scheduling settings
Property	Description	Conversion information
`<user name="..."> <maxRunningApps>...</maxRunningApps></user>`	Maximum running apps per user	There is an equivalent feature in Capacity Scheduler. Fine-tuning of the following three properties are required: Maximum apps per queue User limit percent User limit factor Supported by Queue Manager UI. For more information about user limits, see Set user limits within a queue.
`<userMaxAppsDefault>`	Default maximum running apps	Not supported in Capacity Scheduler.
`yarn.scheduler.fair.max.assign`	Dynamic maximum assign	There is an equivalent feature in Capacity Scheduler. Fine-tuning of the following three properties are required: `yarn.scheduler.capacity.per-node-heartbeat.multiple-assignments-enable` `yarn.scheduler.capacity.per-node-heartbeat.maximum-container-assignments` `yarn.scheduler.capacity.per-node-heartbeat.maximum-offswitch-assignments` Supported by Queue Manager UI.
`yarn.scheduler.fair.user-as-default-queue`	User as default queue	There is a very similar feature in Capacity Scheduler. Perform the following steps: Create a queue, such as `root.users` and enable the `suto-create-child-queue` setting for it. Use the following placement rule: `"u%user:%user"` The following restrictions apply: It is not possible to have `root` as a parent for dynamically created queues. `root.users` cannot have static leafs, that is, queues that are defined in the `capacity-scheduler.xml` file. For more information, see the Placement Rules table. Supported by Queue Manager UI.

Table 7. Preemption
Property	Description	Conversion information
`yarn.scheduler.fair.preemption.cluster-utilization-threshold`	The utilization threshold after which preemption kicks in.	There is an equivalent feature in Capacity Scheduler: `yarn.resourcemanager.monitor.capacity.preemption.max_ignored_over_capacity`. It specifies the resource usage threshold over its configured capacity that a queue must meet before it is eligible for preemption. Supported by Queue Manager UI.
`minSharePreemptionTimeout`	The number of seconds the queue is under its minimum share before it will try to preempt containers to take resources from other queue.s	Not supported in Capacity Scheduler.
`fairSharePreemptionTimeout`	The number of seconds the queue is under its fair share threshold before it will try to preempt containers to take resources from other queues.	Partially supported in Capacity Scheduler. This can be achieved by using the following configurations together: `yarn.resourcemanager.monitor.capacity.preemption.natural_termination_factor` `yarn.resourcemanager.monitor.capacity.preemption.max_wait_before_kill` Supported by Queue Manager UI.
`fairSharePreemptionThreshold`	The fair share preemption threshold for the queue.	Partially supported in Capacity Scheduler. This can be achieved by using the following configurations together: `yarn.resourcemanager.monitor.capacity.preemption.natural_termination_factor` `yarn.resourcemanager.monitor.capacity.preemption.max_wait_before_kill` Supported by Queue Manager UI.

Table 8. Placement rules
Fair Scheduler placement rules	Description	Conversion information
`create="false" or "true"`	Disable or enable creating a queue dynamically in YARN. This option can be specified on all rules.	Partially supported in Capacity Scheduler. Use the Capacity Scheduler Dynamic Queue Mappings policies: `u:%user:[managedParentQueueName].[queueName]` `u:%user:[managedParentQueueName].%user` `u:%user:[managedParentQueueName].%primary_group` `u:%user:[managedParentQueueName].%secondary_group` Supported by Queue Manager UI.
`<rule name="specified"/>`	If a user has submitted the application by specifying a queue name (other than the “default” queue), then this rule will be successful. Hence the remaining set of rules won't be executed.	Not supported in Capacity Scheduler.
`<rule name="primaryGroupExistingQueue"/>`	If submitted user’s(userA) primary group name (groupA) exists, submit to groupA.	There is an equivalent placement rule in Capacity Scheduler: `<value>u:%user:%primary_group</value>` Supported by Queue Manager UI.
`<rule name="secondaryGroupExistingQueue"/>`	If submitted user’s(userA) secondary group name (groupA) exists, submit to groupA.	There is an equivalent placement rule in Capacity Scheduler: `<value>u:%user:%secondary_group</value>` Supported by Queue Manager UI.
`<rule name="nestedUserQueue">`	If submitted the embedded rule, all rules are allowed except for the reject rule, is executed to generate a parent queue and the user’s (userA) name is created as a child of the parent.	Not supported in Capacity Scheduler.
`<rule name="default" queue=”qName”/>`	Fall back policy by which rule will fall back to queue named in the property ‘queue’ or the “default“ queue if no queue property is specified (if all matches fail).	There is an equivalent placement rule in Capacity Scheduler: `<value>u:%user:default</value>` Supported by Queue Manager UI.