Skip to main content
  • New Internet Architecture Board, IETF Trust, IETF LLC and Internet Engineering Task Force Leadership Announced

    Members of the incoming Internet Architecture Board (IAB), the IETF Trust, the IETF Administration LLC (IETF LLC) Board of Directors, and the Internet Engineering Steering Group (IESG)—which provides leadership for the Internet Engineering Task Force (IETF)—have been officially announced, with new members selected by the 2021-2023 IETF Nominating Committee.

      13 Feb 2023
    • Informing the community on third-party correspondence regarding the W3C

      In accordance with our policy of transparency, this blog post is being published in order to keep the community informed about recent correspondence with lawyers acting on behalf of the Movement for an Open Web.

      • Lars EggertIETF Chair
      8 Feb 2023
    • Six Applied Networking Research Prizes Awarded for 2023

      Six network researchers have received Internet Research Task Force Applied Networking Research Prize (ANRP), an award focused on recent results in applied networking research and on interesting new research of potential relevance to the Internet standards community.

      • Grant GrossIETF Blog Reporter
      9 Jan 2023
    • Travel grants allow Ph.D. students to participate at IETF meeting in-person

      Sergio Aguilar Romero and Martine Sophie Lenders, both Ph.D. students in technology fields, attended and participated in the IETF 115 meeting in London with assistance through travel grants from the Internet Research Task Force.

      • Grant GrossIETF Blog Reporter
      7 Jan 2023
    • Impressions from the Internet Architecture Board E-Impact Workshop

      The IAB ran an online workshop in December 2022 to begin to explore and understand the environmental impacts of the Internet. The discussion was active, and it will take time to summarise and produce the workshop report – but the topic is important, so we wanted to share some early impressions of the outcomes.

      • Colin PerkinsIAB Member
      • Jari ArkkoIAB Member
      6 Jan 2023

    Filter by topic and date

    Filter by topic and date

    Automating interoperability testing to improve open standards for the Internet

    • Marten Seemann
    • Jana Iyengar

    8 Sep 2020

    While interoperability testing has been one of the hallmarks of open standards and Internet protocol development for more than three decades, the process itself has been limited by manual and inconsistent testing.

    Automated QUIC Interoperability testing

    To address these shortcomings, we created QuicInteropRunner (QIR), a framework that enables implementers to maintain and update their own endpoints, making it easier for them to set up and run interoperability tests using carefully constructed and repeatable network scenarios. We have also set up continuous testing with QIR and a page where the results are available.

    As implementations of the emerging QUIC protocol being developed in the Internet Engineering Task Force (IETF) began to mature, interoperability testing became increasingly important to ensure that independently-developed implementations worked as expected. However, the existing approach to testing was limited in a number of ways, including:

    • the number of implementations that could be tested, since there are a quadratic number of combinations to test;
    • the number of features that could be tested—a problem exacerbated by the complexity of protocols built for the modern Internet; and 
    • the range of network conditions under which the protocol could be repeatedly tested (which meant that the wide variety of network conditions under which the protocol would be expected to perform well on the global Internet could not be replicated). 
    QUIC badge

    As a result, identifying issues with the protocol and implementations took longer than it should have, and the existing tests could miss scenarios likely to be found once they were widely in use.

    The QuicInteropRunner (QIR) is our attempt at overcoming these limitations. Using simple container orchestration and network simulation, QIR makes it possible to meet the constraints of implementers while automating this process for on-demand, continuous, and repeatable interoperability and performance testing. A more complete explanation how QIR uses Docker containers and the ns-3 network simulator is available in our paper, which was recently presented at the EPIQ ‘20 conference.

    Currently, 13 QUIC implementations (two of which only implement a QUIC server and one which only implements a QUIC client) are included in QIR. Importantly, any implementer can include their implementation in QIR by building a compatible container image, making it publicly available, and adding it to the list of implementations.

    To enable continuous interoperability testing, we have set up a QIR instance that continuously pulls the implementation images and runs all the tests in the framework. The results of this continuous testing are posted on this webpage.

    As a network of networks, the global Internet we know today works because independently deployed and managed components work well together through open protocols. Internet interoperability testing traces its history back to at least 1986 and remains a key part of new protocol development and deployment. While QIR was developed for testing QUIC implementations, we believe the central ideas, components, and even code can be repurposed for testing other protocols that will help the Internet evolve and work better.

    Share this page