1 Introduction

Homa [1] [2] shows how efficient a datacenter RPC can be when the transport knows each message’s size and the receiver decides who sends next. Its Linux implementation delivers short-message tail latencies 7 to 83 times lower than TCP and DCTCP on the same hosts [2]. As used here, tail latency is the 99th percentile: the time within which 99 of every 100 messages complete. The metric fits RPC because a request may fan out to many servers¹, and all further processing waits until every reply is received.

Homa’s price for the lower latency is a new transport. Every server and matching client application must be converted to a message-based transport instead of today’s prevalent TCP streams [3], and datacenter switches must map Homa’s priorities to their egress queues. Given the thousands of applications and microservices a datacenter typically hosts, we felt it important to look for an alternative that moves the burden from the application layer back down to the network fabric.

Our target was to stay within 1.5 times (§3) Homa’s tail latency with stock TCP. Each ToR node runs PFQD [4], a per-flow queueing discipline that we created for shared internet links. This paper extends it to the datacenter. We set up our simulator to duplicate the topology of [2], with PFQD on the hosts and the ToR and TCP as the transport. We then configured and ran simulations using the stock Linux CCs, CUBIC [5] and BBRv3 [6], as well as our WarpRATE [7] (§7).

Initial findings showed that stock PFQD with TCP, under any CC, was insufficient. PFQD at the ToR beat FIFO and DCTCP in all runs, but a short message still waited a full round of every active flow at the receiver’s port, and its tail latency remained several times Homa’s. This is because Homa’s advantage comes from its receiver knowing how much of each message remains: the first packet carries the message length, and the receiver uses it to schedule senders. We gave PFQD a similar insight without resorting to a comparable initial exchange.

We observed that a scheduling queue at an egress port can play the receiver’s role at the source and that each TCP sender can indicate, in each packet, how many bytes the flow has remaining; given that, all that remained was to encode a remaining data indication in a packet’s DSCP field. We then added a Completion Tier (CT) to PFQD. The CT is a set of bands ordered by estimates to completion that serve first the flows closest to finishing.

The resulting simulated performance, with stock CUBIC senders, fell well within our target and in several instances performed significantly better. Across every workload and load, TCP’s short-message tail stayed within 1.19 times Homa’s as the median of five seeds, and on W5, where large messages carry nearly all the bytes, the short messages’ tail was 40 to 53% below Homa’s. Additionally, mid-size messages completed faster than under Homa, and when fifteen senders replied to one receiver at the same moment, no packet was lost, even though the receiver, unlike Homa’s, never told the senders when to send (§5). The benefit grows even larger in cases where Homa does not manage the queue: an elephant into a receiver’s port dominated the port over Homa but was held to its share by PFQD, and a busy uplink between racks, which Homa’s grants cannot see, left Homa’s large messages with tails up to 22 times worse (§6).

The results need real-world confirmation, however they are encouraging enough to report. For the results, we used

WarpSIM [8], which was built to reproduce real-world behavior, and a Homa model written from the current HomaModule code [9]. §2 summarizes Homa and PFQD, and §3 defines the four experiments and their workloads and how we ensured the comparisons are fair. §4, §5 and §6 presents the results with stock senders and §7 with WarpRATE. §8 weighs the results against the success criteria and covers what we learned, §9 describes what configuration PFQD and WarpRATE need and how a datacenter would deploy them, and §10 closes.

2 Overview of Homa and PFQD

2.1 Homa

Homa² is a message transport. A client makes a request and the receiver schedules the arrival of every message addressed to it. In Homa, a message of at most 40 KB is sent immediately, without requesting a grant, at a priority set by its length. A larger message announces itself and sends only what the receiver grants. A receiver issues grants to at most eight messages at once, favoring the one with the fewest bytes left. The ToR port enforces the priorities, so bytes in flight toward a receiver are bounded by its issued grants and small messages overtake large ones.

Figure 1 traces an exchange: a short request travels unscheduled; the longer response is announced, then sent at the granted priority, with new grants sent as packets arrive so the sender never waits after the first round trip. The response acknowledges the request, and Homa keeps no connection once it is delivered.

2.2 PFQD

PFQD [4] is a queueing discipline that manages egress for a physical network port. It needs nothing from its peers, instead, identifying each flow by its quad address. It keeps one FIFO per flow and gives priority to control traffic, connection setup and pure acknowledgments. After the control tier flows segregate into a high priority tier and a low priority tier. A round-robin cycle serves a control tier allowance, one packet of every queued high priority flow, and one packet of the next queued low priority flow, and runs as long as traffic remains queued. After each cycle the next low priority flow is advanced so that no flow starves. Flows that stay backlogged past a demotion age move to the low tier, where they share the tier’s single packet turns. Congestion signals are per flow: a flow whose queue exceeds its share of a sojourn budget is CE-marked if it is ECN-capable and loses its oldest packet if it is not.

PFQD assumes nothing about who else is in the physical port queue. Round-robin is processor sharing: a short message arriving behind thirty active flows waits for thirty packets before egress, and at 25 Gbps with 3 KB packets that is about 30 us, which is easily several times a short message’s entire unloaded completion time.

3 Validation Methodology

3.1 Workloads

We use the Homa workloads defined in [1]. The workloads are organized into five message-size distributions sourced from Facebook, Google, and the web-search cluster of the DCTCP study [10]. The Linux Homa evaluation [2] uses W2 to W5 with messages truncated at 1 MB, and omits W1 as nearly identical to W2; we do the same and choose sizes consistent with HomaModule’s load generator [9]. Figure 1 of [1] plots the distributions. W2 and W3 messages are a few hundred bytes; and, in W4 and W5, messages over 40 KB carry 98% of the bytes. In each run, every host sends messages, with no reply, to randomly chosen hosts at random times, so that on average it uses 50% or 80% of its 25 Gbps link. To match the Linux evaluation, we also run W2 at 3.2 Gbps per host.

3.2 Experiments

Each experiment changes one thing about where congestion forms and who controls it (Table 1). E1 repeats the cluster evaluation benchmark from [2], and E2 adapts the incast experiment of [1] to ask whether receiver grants are needed to protect a port. E3 to E5 are new: E3 and E4 move congestion where a Homa receiver cannot schedule it, and E5 adds flows that leave the datacenter.

3.3 Metrics

A message’s slowdown is defined as its completion time³ divided by its time on an idle network, such that 1 is ideal⁴. Messages are sorted by size into

Table 1: Experiments

	Name	From	Question	Setup
E1	Homa’s home case	Linux evaluation	Can TCP match Homa where Homa is designed to win?	16 hosts on one ToR, every host running the transport under test; W2-W5 at 50% and 80% load
E2	Synchronized fan-in	Homa incast, adapted	Are receiver grants needed to protect the receiver’s port?	N of the other 15 hosts send an S-byte response to one receiver at the same instant, rounds spaced to keep its link half busy
E3	Elephant at the port	New	Who keeps the port when a flow the receiver did not admit arrives?	E1 at 50% load plus a 17th host sending UDP at 10 or 25 Gbps, or a CUBIC bulk transfer, into one receiver
E4	Shared uplink	New	What happens when congestion forms where no receiver sees it?	Two racks of eight joined by a 100 or 60 Gbps uplink; W4 and W5 at 50% and 80% load
E5	Traffic leaving the datacenter	New	Does a datacenter-sized ECN budget harm flows that leave the datacenter?	W4 and W5 at 80% load plus four internet hosts at 4 ms RTT, each with one inbound and one outbound bulk CUBIC flow

ten deciles, and each decile is summarized by its median (P50) and 99th percentile (P99)⁵. Where a table compares a configuration with Homa, it gives that configuration’s P99 divided by Homa’s for the same decile and seed: 1.2 means its slowest 1% took 20% longer, and below 1 means it fared better.

3.4 Simulated Homa Model

All results are produced using WarpSIM [8], a discrete-event simulator whose TCP follows the Linux stack: SACK, RACK, TLP, F-RTO, CUBIC [5], BBRv3 [6] and DCTCP [10]. The WarpSIM Homa model fully implements Homa including unscheduled messages up to 40 KB, announced longer messages, the receiver’s grant scheduler over eight messages, the sender’s pacer, recovery by RESEND on a 1 ms timer, and eight strict-priority levels at the switch.

All runs use a 3,000-byte MTU. Each run simulates traffic for 60 ms (20 ms for W2 and W3) and then drains for up to 300 ms which exceeds the Kernel’s 200 ms minimum retransmission timeout. The runs send between 3,900 to 1.1 million messages. Unfinished messages are counted separately rather than dropped; every reported run finished every message unless we state otherwise.

Table 2 validates the model against the evaluation’s loads on 16 hosts at 25 Gbps rather than 40. The order of the three transports holds in every workload and DCTCP’s distance from Homa falls within the paper’s range. TCP’s distance is larger because our Homa has no host software overhead, which favors Homa.

Table 2: Homa model against ATC 2021

Short-message ratio	ATC 2021	Model, W3 / W4 / W5
P99, TCP	19-72	127 / 123 / 86
P99, DCTCP	7-83	16 / 26 / 15
P50, TCP	3.5-7.5	3.3 / 23 / 23
P50, DCTCP	2.7-3.8	1.3 / 3.3 / 3.7

3.5 Ensuring A Fair Comparison

Whenever a choice decided who wins, we chose in favor of Homa. TCP clients spread each host pair over 32 connections, as the evaluation’s client does [9], so independent messages do not wait in one ordered stream. TCP hosts run fq_codel, and both transports see the same 5us NIC transmit ring. TCP receivers acknowledge within 8us, as receive offload on a datacenter NIC does, with 50us as a sensitivity case (§4); Homa’s receiver grants without delay. Every switch port models a 1 MB buffer, and Homa’s priority switch shares it by dynamic threshold [11].

3.6 Trial Configurations and Success Criteria

Every experiment compares Homa with a series of TCP configurations, each adding one step to the last: TCP through ordinary switches, as most datacenters run it today; DCTCP, the datacenter variant of TCP, as the Linux evaluation ran it; TCP through PFQD; PFQD with the Completion Tier; a larger first flight, so that a 40 KB message leaves in one round trip as it does under Homa; and finally ECN marking at the ToR. The last is the configuration a datacenter would deploy. The configurations with the larger first flight, with and without ECN, are also run with BBRv3 and with WarpRATE as the sender. §9 describes each mechanism.

We defined the success criteria before any result, and they have two parts. First, on E1, TCP’s tail for short messages, those up to 40 KB, must stay within 1.5 times Homa’s on every workload. Second, when an elephant or a busy uplink is added (E3, E4), TCP’s short-message tail must match or beat Homa’s. Passing both supports the claim. If TCP misses the first but passes the second, the claim narrows to Homa wins only where it controls every host and every port; if it misses both, the claim fails. For fan-in (E2), we would conclude that Homa’s receiver grants are needed only if TCP repeatedly overflowed the switch buffer or left unacceptable tails.

Table 3: Short-message P99 relative to Homa, 8 us receive coalescing

CELL	FIFO	DCTCP	PFQD	+CT	+40KB	+ECN
W2 3.2G	0.97	0.97	0.95	0.98	0.99	0.98
W2 50%	5.03	3.87	1.63	1.19	1.22	1.19
W3 50%	7.28	4.68	1.71	1.07	1.08	1.07
W4 50%	8.37	4.81	1.43	1.02	1.03	1.01
W5 50%	3.00	1.45	1.12	0.47	0.48	0.47
W3 80%	22.0	7.58	6.28	1.19	1.19	1.19
W4 80%	21.8	6.79	3.83	1.05	1.05	1.05
W5 80%	10.6	3.57	4.09	0.59	0.60	0.60

4 Experiment E1: Homa’s Base Case

E1 is the experiment Homa was built for: every host runs the transport under test, nothing else shares the fabric, and congestion forms at the receivers’ ports. Table 3 gives, for each workload and load, the worst ratio of a TCP configuration’s 99th-percentile slowdown to Homa’s over the size deciles Homa sends unscheduled. It is computed as the median of five trial seeds with each seed compared with Homa using the same seed. The columns are CUBIC through FIFO switches, DCTCP, PFQD without CT, PFQD with CT, CT with the 40 KB first flight, and that configuration with ECN. Figure 2 shows the distributions at 80% load.

The baselines behave as the Linux evaluation found. FIFO switches leave TCP’s short messages up to 22 times behind Homa at the tail once the load is real, and DCTCP’s marking narrows that to 1.4 to 7.6 times. PFQD alone narrows it further and meets the 1.5 criterion in three of the eight cells, all at light load; elsewhere it stays 1.7 to 6.3 times behind. It removes the standing queue that FIFO builds by confining every flow to its share, but a short message still waits a round behind every active flow at the receiver’s port.

With CT on the hosts and the ToR the ratio falls to between 0.47 and 1.22 in every cell. The tested configuration, with ECN, ranges from 0.47 to 1.19 with zero drops. The same configuration without ECN lost up to 27,000 packets per five seeds at 80% load. On W5, whose messages are the largest, CT’s short-message tail is 40 to 53% below Homa’s, because Homa’s short messages wait behind the unscheduled bursts of other messages at the same priority level, which CT separates by bytes remaining. Furthermore, no trial of any CT configuration exceeded 1.24 and the success criterion holds everywhere for short messages.

The mid-size messages favor CT as well. At W5 and 80% load, messages of 51 to 255 KB complete at a median slowdown of 1.8 to 2.1 under Homa and 1.7 to 1.8 under the tested configuration, and a 99th percentile of 18 to 25 under Homa against 2.9 to 4.3. The clear improvement is due to the fact that a grant-driven receiver lets a mid-size message wait while it serves smaller ones and CT’s bounded share does not. Homa wins the largest decile, 255 KB to 1 MB, at a median of 2.3 against 4.6 for CT with ECN and 2.9 without it.

Finally, receive coalescing decides how much the 40 KB first flight matters. With acknowledgments generated within 8us, the second round trip of a 30 KB message is short. When receivers coalesce over 50us, as on the internet clients the simulator was calibrated on, CT alone stayed at 0.95 and 1.17 of Homa’s on W5 while the 40 KB first flight brought it to 0.49 and 0.60, and every other cell kept its ratio within 1.22 of Homa’s. We further note that the first flight changes nothing when acknowledgments are fast and halves W5’s tail when they are slow.

5 Experiment E2: Fan-in Without Grants

Homa’s receiver grants have a second job besides ordering messages: they bound the bytes in flight toward a

Table 4: Fan-in operation P99 and port drops, 8 us receive coalescing

Sender	15x64K	8x512K	15x512K
Homa	1.01 / 0	1.00 / 0	1.00 / 0
CUBIC	0.99 / 0	0.99 / 1,083	0.99 / 856
CUBIC 4MB	0.99 / 0	0.99 / 0	0.99 / 0
CUBIC ECN	1.65 / 0	1.24 / 0	1.12 / 0
BBRv3	0.99 / 0	1.01 / 17,718	1.00 / 24,011
BBRv3 ECN	1.05 / 0	1.04 / 10,191	1.03 / 7,630
WarpRATE	0.99 / 0	0.99 / 0	0.99 / 0

receiver, so a burst of responses cannot overrun the receiver’s port. Whether a datacenter needs that bound is the question of whether it needs receiver admission at all, and it is answered by fan-in, where many senders answer one receiver at once. In each round of this experiment N senders, chosen at random from the other fifteen hosts, start an S-byte message to the same receiver at the same instant, and rounds are spaced so that the receiver’s link is half busy. Each message is an independent request; each round is also a partition-aggregate operation that completes with its last response, and we compare its completion with the time the receiver’s link needs to deliver all N responses.

Table 4 gives, for the three tests that separate the configurations, the operation’s 99th-percentile slowdown and the drops at the receiver’s port; every TCP row is CT with the 40 KB first flight, and a row marked 4MB runs with 4 MB switch ports instead of 1 MB. Operations complete at the ideal time under Homa and under CT without ECN, from four senders of 16 KB to fifteen senders of 512 KB; the ratios of 0.99 come from the idle-network time counting Homa’s larger headers (§3). Individual responses are split by size. Fifteen 16 KB responses completed at a median slowdown of 5.6 under CT against Homa’s 9.6, because Homa sends every small response unscheduled at the same priority and they share the port in arrival order, while CT orders them by bytes remaining; for 64 and 512 KB responses Homa’s grants did better, 7.3 to 7.9 against 9.9 to 13.7.

Homa never let more than about 400 KB queue at the receiver’s port. TCP, sending its first flight from every sender at once, filled the 1 MB port once responses reached 512 KB. CUBIC lost 0.6 to 1.3% of the packets the responses needed and its operations still completed at the ideal time; BBRv3, which does not back off on loss in the same way, lost 15 to 21%.

By the criterion we fixed in advance, that is repeatable buffer overflow, which the criterion counts as evidence for receiver admission; completion tails stayed at the ideal. We then tested three remedies short of admission. The first solution is increase the port buffer space: with 4 MB ports no configuration lost a packet and every operation stayed at the ideal, but CUBIC’s port peaked at 2.7 MB and BBRv3’s at 2.8 MB. The second solution is: mark before the buffer is full. Using

Table 5: Short-message P99 at the elephant’s port, 8us receive coalescing

Cell	Homa	FIFO	DCTCP	+CT	+40KB	+ECN
W4 10G	4.2	104	27.1	6.8	4.3	4.2
W4 25G	87.4	96K	108K	8.2	4.3	4.4
W5 10G	9.8	60.9	12.1	7.7	3.1	3.2
W5 25G	8.4	67K	67K	7.8	3.1	3.1
W4 TCP	n/a	36.2	26.5	7.0	4.3	4.2
W5 TCP	n/a	21.8	11.4	7.1	3.0	3.1

the 20us ECN budget of the tested configuration: CUBIC lost nothing and its port peaked at 591 KB. The price was that completion times increased to 1.68 times the ideal across the nine tests. The third solution is to change the sender: Using WarpRATE, which paces each flow at the rate the path delivers, each trial completed every operation at the ideal time with no drop and a peak of at most 948 KB of port buffer.

6 Experiments E3 to E5: When Homa Does Not Own the Queue

Homa’s guarantees rest on its receivers knowing everything that arrives at their ports. The final three experiments take that away by introducing: an uninvited flow, a congested link that no receiver manages, and flows that leave the datacenter through the same ports.

6.1 Experiment E3: An Elephant at the Receiver’s Port

We add a seventeenth host that sends an elephant into one receiver’s ToR port while the rack runs W4 or W5 at 50% load.⁶ Table 5 gives the 99th-percentile slowdown of short messages addressed to that receiver, the median of three trial seeds. The last two columns show CT with the 40 KB first flight, with and without ECN.

PFQD held the elephant to its share of the port, about 12 of the 25 Gbps, and the receiver’s messages kept a 99th-percentile slowdown of 3.0 to 4.4 for short messages and 5.0 to 9.6 for the largest, with ECN at the upper end of that range. Homa has no way to do the same. The UDP flood sat at the lowest priority, where Homa also places a lone scheduled message, so the receiver’s large messages queued behind it; a line-rate elephant took 20 to 22 Gbps of the port, and those messages reached a 99th-percentile slowdown of 160 to 1,169, against 38 to 77 with the 10 Gbps elephant. FIFO and DCTCP collapsed whenever the elephant ignored their congestion signals. A TCP elephant is not possible against Homa receivers, which carry no TCP; under PFQD it was confined like the UDP one.

Both designs trust a field the hosts write, and we tested what a host that lies can do. An elephant that marks itself with Homa’s highest priority took the port from Homa entirely, raising the 99th-percentile slowdown of short messages to 18,000 to 27,000. The same elephant carrying a forged band 0 did the same to CT, to 22,000

Table 6: Cross-rack P99, 8 us receive coalescing

Cell	Homa	+ECN	PFQD+CT	DCTCP
W4 50%	3.1 / 6.5	3.1 / 8.0	4.2	13.7
W5 50%	5.4 / 13.3	2.4 / 9.0	6.4	7.4
W4 80%	3.2 / 46.8	3.2 / 25.4	4,362	18.5
W5 80%	5.2 / 39.2	2.5 / 26.1	52.2	16.3
W4 60G	13.9 / 144	3.1 / 16.8	12.4	15.0
W5 60G	11.6 / 332	2.4 / 14.9	11.0	9.6

to 46,000 without ECN, because band 0 wins eight of every nine serves. Plain PFQD, which reads neither field, was untouched. In a datacenter the operator runs every host stack and has no reason to forge either mark, so we treat this as the trust boundary for DSCP in general rather than as a defect of either design.

6.2 Experiment E4: A Shared Uplink

The second variation splits the hosts into two racks of eight joined by one uplink, so that about half of all messages cross it. At 100 Gbps and 50% load the uplink is about half busy; at 80% load, or at 60 Gbps and 50% load, it is close to full. Table 6 gives the 99th-percentile slowdown of messages that cross it, the median of three seeds, with the uplink at 100 Gbps except in the two 60G trials, which run at 50% load. For Homa and the deployed configuration each entry reports short messages, then the largest; the PFQD+CT and DCTCP columns report only short messages.

When the uplink had room the designs were close. When it was busy, Homa’s large cross-rack messages degraded to a 99th-percentile slowdown of 39 to 332, because a receiver grants bytes against its downlink and cannot see that the uplink is where they queue. The tested configuration kept the same messages at 15 to 26, up to 22 times better, and lost no packet at the uplink, where Homa lost up to 4,400 per run; CT without ECN kept them at 14 to 26 but lost up to 7,900. Its short cross-rack messages matched Homa’s in the two W4 cells with a 100 Gbps uplink and beat them by two to five times in the other four. The busy uplink broke only plain PFQD without CT and without ECN, whose CUBIC senders, acknowledged quickly and never marked, overran the uplink, lost 44,000 packets per run on W4 at 80% load and left its short messages with a 99th-percentile slowdown in the thousands, and 52 on W5.

6.3 Experiment E5: Traffic That Leaves the Datacenter

The fifth experiment puts internet traffic on the same ToR. Four hosts behind 10 Gbps links, each a 4 ms round trip away, send one bulk CUBIC flow into a server and receive one from it while the rack runs W4 or W5 at 80% load in the deployed configuration for 200 ms. The inbound senders run no PFQD and are not ECN-capable, as most internet senders are not.

Table 7: Datacenter and internet flows on one ToR (in, out: Gbps per internet flow)

Budget	Short P99	DC drops	In	Out
W4 20 us	4.35	0	0.03	0.34
W4 5 ms	4.39	29,451	0.64	2.64
W4 class	4.38	0	2.28	3.36
W5 20 us	4.58	0	0.03	0.35
W5 5 ms	3.36	28,709	0.53	2.28
W5 class	4.63	0	1.85	2.95

Table 8: WarpRATE and CUBIC with ECN, 8 us receive coalescing

Measure	Homa	CUBIC	WarpRATE
E1 short P99 / Homa	1	0.47-1.19	0.47-1.20
E1 large P99, W4 80%	20.0	25.7	17.2
E1 large P99, W5 80%	21.5	26.4	19.3
E2 op P99, worst cell	1.05	1.68	1.04
E2 peak, 15x512K, KB	396	591	948
E4 large P99, W4 60G	144	16.8	12.5
E4 large P99, W5 80%	39.2	26.1	22.2
E4 drops, W5 80%	1,286	0	1,471

Table 7 compares three ways of setting the ToR’s ECN budget by the datacenter’s short-message tail and reports drops and each internet flow’s goodput.

A flat 20 us budget kept the datacenter lossless but held every internet flow near the 64 KB floor of its per-flow limit: inbound flows, head-dropped at that depth, ran at 20 to 50 Mbps, and outbound ones, marked at 0.2 to 0.5 Gbps. A flat 5 ms budget freed the internet flows but let the datacenter’s flows lose about 29,000 packets per run, since at 25 Gbps it marks almost nothing; on W5 it also gave the lowest short-message tail. With the RTT class of §9 the datacenter stayed lossless, its short-message tail within 1% of a run without internet traffic, and the internet flows ran faster than under 5ms, 3.5 times inbound and about 30% outbound at the median, since the datacenter’s short queues left them room.

7 WarpRATE as the Sender

WarpRATE [7] paces every flow at a rate it measures. We ran it in the deployed configuration, with the DSCP stamp, CT, the 40 KB first flight, ECN, and with the measurement-interval rule described in §9. Table 8 compares it with CUBIC in the same configuration and with Homa.

WarpRATE met every criterion CUBIC met and removed most of what ECN costs CUBIC. Its short-message tail matched CUBIC’s in every E1 cell. Its largest messages stayed below Homa’s tail on W4 and W5 at 80% load. Its fan-in operations completed at the ideal time in every cell, 1.04 at worst, where CUBIC with ECN took up to 1.68, and its port never overflowed, though it filled to 948 KB. Across a busy uplink its largest messages reached a 99th-percentile slowdown 15 to 34% below CUBIC’s in five of six cells; in the sixth, W5 over

the 60 Gbps uplink, it reached 34.3 against CUBIC’s 14.9, from 9.0 to 38.3 across seeds. WarpRATE paces each flow at the rate the path delivers rather than filling a window until a queue marks it, so it needs the marks less often and pays less when they come.

WarpRATE dropped packets where CUBIC with ECN dropped none: in E1 at most 1.4 per 1,000 packets, and across a busy uplink 1.8 per 1,000, about the same as what Homa lost. Under the elephant carrying a forged band 0 in E3 it finished every message, as CUBIC did, with a short-message tail of 1,200 to 32,000 against CUBIC’s 650 to 84,000 and Homa’s 18,000 to 27,000.

8 Discussion and Limits

8.1 Exceeding the criteria

By the criteria we fixed in advance (§3), the claim holds: in the tested configuration TCP’s short-message tail stayed within 1.5 times Homa’s in every trial, with at most 1.19 as the median over seeds and 1.21 in the worst single seed. With an elephant or a shared uplink the deployed configuration beat Homa on the same measure in seven of ten trials and matched it within 3% in the other three; Homa won none. WarpRATE in place of the stock senders met the same criteria (§7).

In fan-in the criterion counts repeatable buffer overflow as evidence that receiver admission is needed, and without ECN, CT’s TCP senders overflowed a 1 MB port in every fan-in cell with 512 KB responses, though every operation still completed at the ideal time. The remedies we then tested, larger ports, ECN at a 20us budget and a pacing sender, each remove the overflow without a receiver scheduling its senders. We interpret this as showing that admission is not required for fan-in, while granting that Homa’s grants deliver the combination of ideal completion and a small, bounded port occupancy that none of the three alternatives matched.

8.2 What Homa does better

Homa orders messages by exact size at the receiver, and that is worth something at the top of the size range. With receivers acknowledging within 8us, CT’s mid-size messages complete at a lower median than Homa’s and with a far lower tail, but the largest decile, up to 1 MB, finishes faster under Homa at the median, 2.3 against 2.9 without ECN and 4.6 with it, because six bands and a bounded share approximate shortest-remaining-first rather than implement it. ECN, which keeps the deployed configuration’s ports free of loss, is also what raises its largest messages’ tail above Homa’s on W4 and W5 at 80% load. Homa’s grants also cap the bytes a receiver’s port must hold, below half of a 1 MB port in every fan-in cell.

Homa’s message semantics are the third difference, and they are not an advantage in themselves. Homa delivers each message independently, so a short message never waits behind a large one to the same peer; conversely, TCP delivers a connection’s bytes in order. When the short message depends on context the large one builds, that order is what the application needs, and over Homa it would have to be restored above the transport.

8.3 Loss and bulk data

Loss separates the two designs further. Homa has no acknowledgments: a receiver finds a missing packet only after its message has gone quiet for 5ms, and it delivers nothing until the whole message is present, so a single loss costs a message milliseconds. TCP repairs a hole from SACK within about a round trip and delivers every byte before it at once. On the shared uplink, where Homa’s grants could not see the queue, Homa lost up to 4,400 packets per run and its large cross-rack messages reached tails up to 22 times the deployed configuration’s; CT without ECN lost more, up to 7,900, and still kept those tails at 14 to 26 (§6), consistent with a loss costing TCP a round trip and Homa 5ms. For Homa to recover as quickly, its grants would have to report which bytes arrived, and its unscheduled messages would require acknowledgments; it would essentially need to rebuild the loss recovery TCP already supports.

Homa’s messages are capped at 1 MB, so bulk data travels as many RPCs. A 1 GB response becomes about a thousand requests and responses that the application must issue, reorder and reassemble, since RPCs to one server may complete in any order. Each piece is held whole until it completes and carries its 5ms exposure to a loss. Thus, a transfer that TCP carries as one stream repairing holes within a round trip becomes a thousand chances to stall.

9 PFQD-DC Final Design

This section describes the design tradeoffs behind the results: the completion stamp, CT, the first flight, ECN and the RTT class that keeps ECN timely without harming flows that leave the datacenter.

9.1 The completion stamp

The host knows what the ToR does not. When TCP transmits a segment, the socket already holds every byte the application has written, so the number of bytes queued behind that segment is the difference between the socket’s write sequence and the segment’s end. We call its bucket the segment’s band and write it into the DSCP field. Five segmentations at 3, 12, 48, 192 and 768 KB give six bands, which fit into three bits, and band b is carried as DSCP (b << 2) | 3⁷. A deployment must reserve those codepoints across its fabric, as it would any locally defined DSCP.

The band is per packet and computed once, when the packet enters the host’s qdisc. It measures the burst the application has handed over, not the whole transfer, so a response streamed in small writes looks like a series of small bursts, which is what the socket knows. It needs nothing from the application or the receiver, and it is independent of the congestion

controller, since it reads only the socket’s sequence numbers. A host that does not stamp looks like ordinary traffic.

Four implementation details matter. A TSO packet is stamped once, from its first segment, and the NIC copies that header to every segment it transmits, so the later segments of a large write carry a band up to one TSO packet larger than exact. A retransmission is banded using the socket’s current write sequence⁸, so it reflects the backlog at the time it is resent. The write sequence is read without the socket lock, which is safe because it only grows: a stale read can only understate the backlog, by what the application is writing at that moment. The update rewrites only the six DSCP bits, keeping the ECN field and fixing the IP checksum, and a DSCP the application set itself is left alone.

9.2 The Completion Tier

PFQD keeps its one FIFO per flow, and CT, its set of completion queues ordered by band, decides which flow’s FIFO to serve next. A flow is listed under the band of the packet at its head, and after the control tiers the scheduler serves the flow whose head carries the smallest band, round-robin among flows in the same band. Because the head decides, a later packet in a flow never overtakes an earlier one, and a flow whose large transfer is nearly finished competes with a new small message on equal terms.

Strict band order alone would let a steady supply of small bursts starve everything else, so the share is bounded. After eight banded serves the next opportunity goes to PFQD’s ordinary round-robin over all flows, which bounds any flow’s wait whatever its band. Banded serves count toward PFQD’s service cycle, so the allowance PFQD gives acknowledgments each cycle keeps its cadence.

9.3 Host and ToR

The same qdisc runs on the sending host’s NIC and on the ToR, and both places matter. At the ToR, CT decides between flows from different hosts. At the host it decides between the host’s flows. The latter matters when a host sends a mid-size response while it is also streaming a large transfer: with fq_codel on the host, W5’s 7 to 31 KB messages waited up to 93us at the 99th percentile behind the host’s large transfers, a wait no switch can undo. A host whose DSCP marks must not leave it, such as a server facing the internet, can run CT for its queue without stamping the wire.

9.4 The first flight

Homa sends up to 40 KB of a message before the receiver responds. TCP sends its initial window, ten segments by default [13], and Linux receivers advertise a first window of about 29 KB whatever the MTU, which caps any sender’s first flight. A 30 KB message therefore needs a second round trip under stock TCP and not under Homa. Linux lets an operator set both the initial congestion window and the initial receive window per route, so the deployment sizes both from a 40 KB target and the route’s MTU: fourteen 2,960-byte segments on our 3,000-byte fabric, twenty-nine at 1,500 bytes. No code changes; a helper applies the two route metrics when the module loads.

9.5 ECN

PFQD marks only traffic it forwards; a host never marks its packets. The ToR marks a flow whose queue exceeds its share of a sojourn budget, and the deployment sets that budget to 20us, against PFQD’s default of 5ms, which suits internet paths, but at 25 Gbps is deeper than the port buffer. Each flow’s limit is floored at 64 KB. Hosts only need to negotiate ECN, and the same helper that sets the first flight marks its routes ECN-capable, so every connection the host opens asks for ECN whatever its congestion controller and whatever the host-wide default.

9.6 The RTT class

CT depends on timely ECN. With the 5ms budget PFQD uses on internet paths, a 25 Gbps port marks almost nothing, and the fan-in and busy-uplink results lose their lossless ports. But the ports that carry datacenter RPCs also carry flows that leave the datacenter: a server answers local peers and internet clients through the same NIC and ToR port, and internet traffic arrives on the same downlinks. A switch cannot tell which flows those are, and a 20us budget applied to them holds each near the 64 KB floor, far below a long-RTT window (§6). Homa never meets this, because only Homa endpoints speak Homa and every Homa flow stays inside the datacenter. A TCP fabric needs each flow’s RTT in its packets.

In our design the DSCP stamp carries it. The sending host sets one more bit of the DSCP value, the RTT class, from the socket’s minimum RTT: short below 1 ms, long at or above it. Every PFQD hop gives short-class packets a 20us budget and long-class packets 5ms per packet and with no per-flow state, so a flow whose packets or directions take different paths is treated the same at every hop. Internet traffic arrives unstamped, and the datacenter edge, already the DSCP trust boundary, stamps it long class on entry; an edge facing a trusted upstream that stamps its traffic keeps those stamps instead. Unstamped traffic inside the datacenter keeps the ToR’s budget, 20us. The long budget never exceeds an internet router’s 5ms, so every PFQD router, inside the datacenter or not, can honor the class safely, and the edge leaves the codepoints on outbound packets for the routers beyond it.

Specifically, the class and band share pool 2. Bands 0 to 5 of the short class are b = 0 to 5, of the long class b = 8 to 13, and b = 7 and b = 15 are the two classes without a band, the latter being what the edge stamps; a PFQD-DC deployment must reserve these fourteen codepoints.

9.7 WarpRATE’s measurement interval

WarpRATE sets its pacing rate from what each measurement interval

delivers, and an interval closes on the first acknowledgment after one round trip of the path: the minimum round trip plus a margin for its spread, a margin that falls to zero once the connection sees a CE mark, since a mark shows the extra delay is a queue. An interval restarts whenever a connection sends again after idling, so idle time between messages never enters a rate; on a pooled RPC connection, an interval that spanned the idle time would read a burst delivered at line rate as a trickle and pace the connection there. A long transfer never idles and is unaffected. WarpRATE’s recovery timer, a heartbeat that fires only when acknowledgments stop, is re-armed by the same event, so a burst lost whole after an idle period is retransmitted within one heartbeat period, rather than after the pinned retransmission timeout.

9.8 Deployment

All of the pieces exist as code. The stamp and CT are part of sch_pfqd 2.5.0, off by default and switched host-wide by two module parameters, stamp and completion, with per-interface overrides through tc; the ToR runs CT with the 20us budget for internal traffic and the per-class budgets, and the edge stamps traffic arriving from the internet. A helper applies the 40 KB first flight and ECN as route metrics when the WarpRATE module loads, at boot and again whenever routes change. So the hosts change too: they stamp, run PFQD on their NIC and carry the route metrics, and their clients spread traffic over a pool of connections per peer, as RPC frameworks already do and as the Linux Homa evaluation’s client did. What stays unchanged is TCP, its congestion control and every application interface.

9.9 Related designs

pFabric [14] showed that switches serving the packet with the least remaining work approach ideal flow completion, given hosts that write the remaining size into every packet; PIAS [15] approximated that order on commodity switches without size information, and NDP [16], like Homa, moved scheduling to the receiver. Our stamp is closest to pFabric’s, but it indicates only what the socket already knows.

10 Conclusion

PFQD-DC is PFQD with its Completion Tier running on the data center hosts and the ToR switches. It utilizes ECN marking at every PFQD hop, and an RTT class, driven by one DSCP value per packet. Every hop orders completions by the band and takes the flow’s ECN budget, 20us or 5ms, from the RTT class, so datacenter flows are marked within microseconds while flows that leave the datacenter keep their typical windows.

Within the limits of simulation, this design kept short-message tail latency within 1.2 times Homa’s on every workload. It completed mid-size messages faster than Homa, completed synchronized fan-in without loss and without receiver grants, and maintained its latency where Homa underperformed, behind an elephant or across a busy uplink. Furthermore, flows leaving the datacenter through the same ports maintained their throughput because every hop sized its marking to each flow’s RTT.

Those are the properties that latency-sensitive traffic like short message floods and many-to-one bursts require. Homa retains an edge for servicing the largest messages, where ordering by exact size pays off, and its grants hold a port’s occupancy lower than any alternative we tested. However, WarpRATE as the sender closed most of these gaps at the cost of a little bit of packet loss with retransimssion. We also note that wherever loss occurs, TCP wins.

Thus, Homa asks for a new transport on every host and an RPC layer built around its message interface. What we tested asks for a queueing discipline on the hosts and the ToR and three route metrics, all the while keeping TCP and every application and every RPC framework as they are. A formal testbed has to confirm it, but a datacenter that wants Homa-class latency may find it possible using the protocols it already supports.

11 Statement on AI Use

Models and tools provided by Anthropic were used to create simulation harnesses, organize data sets, and validate implementations against specifications.

References

• [1] B. Montazeri, Y. Li, M. Alizadeh, and J. Ousterhout, “Homa: A Receiver-Driven Low-Latency Transport Protocol Using Network Priorities,” in Proceedings of ACM SIGCOMM, 2018, pp. 221–235.
• [2] J. Ousterhout, “A Linux Kernel Implementation of the Homa Transport Protocol,” in Proceedings of the USENIX Annual Technical Conference (ATC), 2021, pp. 773–787.
• [3] X. Lu and Z. Zhang, “Leveraging Homa: Enhancing Datacenter RPC Transport Protocols,” in Netdev 0x17, the Technical Conference on Linux Networking, 2023.
• [4] D. Vadura, “Practically Fair Queueing: A Per-Flow Scheduling Discipline for Shared Internet Links,” technical report, 2026.
• [5] L. Xu, S. Ha, I. Rhee, V. Goel, and L. Eggert, “CUBIC for Fast and Long-Distance Networks.” RFC Editor, Aug. 2024. doi: 10.17487/RFC9438.
• [6] N. Cardwell, Y. Cheng, C. S. Gunn, S. H. Yeganeh, and V. Jacobson, “BBR: Congestion-Based Congestion Control,” Communications of the ACM, vol. 60, no. 2, pp. 58–66, Feb. 2017, doi: 10.1145/3009824.
• [7] D. Vadura, “WarpRATE: A Rate-Paced CC Solution,” technical report, Mar. 2026.
• [8] D. Vadura, “WarpSIM: A Picosecond-Fidelity Discrete-Event Simulator for Modern Congestion Control,” technical report, Mar. 2026.
• [9] PlatformLab, “HomaModule: A Linux Kernel Implementation of Homa.” 2026.
• [10] M. Alizadeh et al., “Data Center TCP (DCTCP),” in Proceedings of the ACM SIGCOMM 2010 Conference (SIGCOMM '10), ACM, Aug. 2010, pp. 63–74. doi: 10.1145/1851182.1851192.

• [11] A. K. Choudhury and E. L. Hahne, “Dynamic Queue Length Thresholds for Shared-Memory Packet Switches,” IEEE/ACM Transactions on Networking, vol. 6, no. 2, pp. 130–140, 1998.
• [12] K. Nichols, S. Blake, F. Baker, and D. Black, “Definition of the Differentiated Services Field (DS Field) in the IPv4 and IPv6 Headers.” 1998.
• [13] J. Chu, N. Dukkipati, Y. Cheng, and M. Mathis, “Increasing TCP's Initial Window.” RFC Editor, Apr. 2013. doi: 10.17487/RFC6928.
• [14] M. Alizadeh et al., “pFabric: Minimal Near-Optimal Datacenter Transport,” in Proceedings of ACM SIGCOMM, 2013, pp. 435–446.
• [15] W. Bai, L. Chen, K. Chen, D. Han, C. Tian, and H. Wang, “Information-Agnostic Flow Scheduling for Commodity Data Centers,” in Proceedings of USENIX NSDI, 2015, pp. 455–468.
• [16] M. Handley et al., “Re-architecting Datacenter Networks and Stacks for Low Latency and High Performance,” in Proceedings of ACM SIGCOMM, 2017, pp. 29–42.