End-System Aware Peak Link Utilization Transport for Big Data Transfer over Dedicated Connections

Abstract

High-performance networks provisioning 10/40-Gbps dedicated channels have been rapidly developed and increasingly deployed to support big data transfers on the order of terabytes or petabytes in large-scale network-intensive applications. However, end users have not seen corresponding increase in application throughput mainly because traditional end-to-end transport methods are not optimized for such connections. New congestion or flow control mechanisms are desirable to meet the challenges brought to transport protocol design by dedicated connections. The advent and proliferation of multi-core processors make it now possible to improve application throughput by providing multiple processing and network resources to a single data transfer. Based on the existing PLUT method, we proposed a new dynamic buffer management scheme and an automatic parallelism tuning mechanism to achieve the maximal bottleneck rate. The performance superiority of PLUT is justified by the experimental results collected over a local 10Gbps dedicated connection.