SENSE: Intelligent Network Services for Science Workflows

Original paper

Layer2/3 Services, Full Lifecycle, Multi-Domain, Multi-Resource, Interactive, End-to-End

SENSE Team

The Software-defined network for End-to-end Networked Science at Exascale (SENSE) research project is building intelligent network services to accelerate scientific discovery in the era of big data driven by Exascale, cloud computing, machine learning and artificial intelligence.

SENSE includes a model-based orchestration system which operates between the SDN layer controlling the individual networks/end-sites, and science workflow agents/middleware. In addition,  SENSE has developed a “Network Resource Manager” and “End-Site Resource Manager” which enable advanced features in the areas of multi-resource integration, real time responsiveness, and user interactions.

The SENSE system defines the mechanisms needed to dynamically build end-to-end deterministic and policy-guided Layer 2/3 network services. An intent-based interface allows applications to express service requirement in a high-level domain science specific context, with mechanisms for interactive and full-service lifecycle coordination with workflow automation systems. The SENSE system has two key objectives:

  • Facilitate science workflow related provisioning and life-cycle management for a variety of end-to-end network services.
  • Provide the intelligence and control to enable more optimal and efficient use of network infrastructures.

The impact of these capabilities includes an ability for science applications to manage the network as a first-class schedulable resource akin to instruments, compute, and storage. This will enable workflow driven optimization of resources spread across a vast  geographic  footprint  such  as  those  used  in  energy sciences.

Demonstration Activities

The SENSE SC19 Network Research Exhibition (NRE) demonstration will show the status of ongoing work to integrate SENSE services with domain science workflows. This will include early work and vision for integration of SENSE services with the Large Hadron Collider File Transfer Services, DOE Superfacility, Big Data Express, and distributed compute infrastructures such as the Pacific Research Platform/National Research Platform. A common vision for these integrations is the provisioning of SENSE Layer 2 and Layer 3 services based on knowledge of current and planned data transfers. SENSE allows workflow middleware to redirect traffic at granularities ranging from a single flow, specific end-system, or an entire end-site onto the desired SENSE provisioned service. The SENSE Layer 2 services provide deterministic end-to-end resource guarantees, including the network and Data Transfer Node (DTN) elements. The SENSE Layer 3 service provides the mechanisms for directing desired traffic onto specific Layer 3 VPN (L3VPN) for policy and/or quality of service reasons.

Resources

A SENSE testbed consisting of network and end- system resources has been deployed across DOE Laboratories, Universities facilities, and ESnet. To control network resources, the SENSE system interacts with production provisioning system of Energy Sciences Network (ESnet) and other regional and site networks. To control end-systems, SENSE software is deployed at the end-sites. For production  DTNs,  limited  access  is  provided  by tailoring the set of SENSE based dynamic configurations to match local site polices. This approach to use a mix of production and research resources enables experience with various real- world site deployments and considerations. This testbed is utilized to develop and test the SENSE software, as well as test with domain science use cases.

Figure 1. SENSE SC19 Demonstration Topology

The SENSE demonstration at SC19 utilizes this persistent testbed and adds resources on  the exhibit showroom. Figure 1 shows the SENSE SC19 demonstration topology with a focus on the SENSE Layer 3 VPN service. This demonstration will utilize several wide area links into the SC19 Exhibit Floor from ESnet, Internet2, and other external locations. The SC19 local resources for this demonstration will be in the Caltech Booth 543.

The SENSE project is also collaborating with other NRE demonstrations including the following:

  • SC19-NRE-019 – Global  Petascale  to Exascale Workflows for Data Intensive Science
  • SC19-NRE-020 – LHC Multi-Resource, Multi- Domain Orchestration via AutoGOLE, SENSE
  • SC19-NRE-011 – Big Data Express
  • SC19-NRE-023 – International Data Transfer over AmLight Express and Protect (ExP)

SENSE Team