AUP has for many years provided fast, reliable internet connectivity to the campus community in all buildings. In early 2023, our wireless system was still providing speeds that met and exceeded the requirements of the campus community. However, the equipment being used on-campus was soon to be discontinued by its vendor, meaning that AUP would need to transition to a new Wi-Fi system by the end of the year to avoid security issues (possible security holes that would no longer be resolved by the Wi-Fi system's manufacturer) and reliability issues (lack of availability of replacement parts). As such, IT Services initiated the process of searching among the available Wi-Fi solutions on the market to begin the process of finding a suitable replacement Wi-Fi system.
Although the existing Wi-Fi system (based on what is known as "Wi-Fi 5" technology) was meeting the current needs of the community, we looked for opportunities to implement a cutting-edge system that would meet not only the current needs of the community, but that would meet the needs of the equipment and applications that would be used over the upcoming years. In addition to the features that our current system offered, we therefore wished to add:
Having determined the specifications we needed to provide our community with an excellent Wi-Fi experience, we contacted four market-leading vendors, detailed our needs, looked over the solutions that were offered. We pursued conversations with three vendors and ended up getting equipment from two of those vendors that we could test.
Tests were performed in the busiest location on campus during the summer semester, the Amex Café, allowing us to see how the access points would behave with a high amount of users. After working through some technical issues with some of the test devices, we found a solution that performed very well and suited our needs.
With the assistance of AUP's internal maintenance team, IT Services replaced the entire Wi-Fi system in mid-August, mounting over 80 access points throughout the campus's seven buildings. After initial tests, everything seemed to be working very well: speeds and connection quality were excellent.
When students and faculty arrived on campus, we were thrilled to be able to offer a new, upgraded Wi-Fi experience, after having tested the new equipment in a busy location during the summer semester and having no problems before the arrival of students. However, after the arrival of the entire student body on-campus, we encountered two issues:
We opened a support request with the equipment supplier to ask why the access points were restarting. After all, this was a brand-new system, and even a high amount of stress on the equipment shouldn't make the device restart. The vendor was slow to respond, so we did investigations on our own, looking into what might cause this:
We provided ongoing updates to the vendor, but their support was not helpful. Their support did confirm, after several weeks of data collection, that they discovered that the software in the access points was access points was crashing on their own, and that it was not a fault in our network that caused the issue. Although this provided welcome reassurance that we had not incorrectly configured the system, it didn't fix this annoying problem.
We received complaints of bad connectivity in many areas (mainly in the Quai d'Orsay and Combes buildings), and although we would occasionally see the issues ourselves when walking around the buildings, we could not reliably recreate them, and usually the students and staff who raised these issues with us could not recreate the issues on demand. We noticed that issues mostly disappeared when few people were in the buildings, but that even with lots of people present, speeds were often excellent - when things worked. We opened a second ticket with the Wi-Fi equipment vendor, but they also had difficulty discovering the origin of the problem, which seemed to pop up and disappear at any moment. However, it was clear that there was an issue, due to the numerous complaints we received about the new system.
The vendor's support team investigated possible issues in our existing network backbone that might cause the problems about which we were hearing, but did not find anything wrong, leaving us in a situation where even the equipment's manufacturer confirmed that nothing was set up incorrectly yet could not find the origin of the issues.
After over a month of receiving disappointing non-responses on both of the support requests that we'd raised, the issue was finally escalated to someone working for the equipment vendor who was able to push their engineering team to work on identifying the issues. After a few weeks, they were able to find what was causing the four wireless access points to crash (and we later found, what was also causing the general performance issues).
Devices on the internet need a way to speak with each other, which they do using the Internet Protocol version 4 (IPv4) and an accompanying "IP address". Traditionally, each device was assigned a temporary or permanent IPv4 address consisting of four numbers ranging from 0-255, separated by decimals: for example, 193.49.37.66 is an IPv4 address that could be used by a server on the internet. However, because IPv4 addresses by nature only allow for a maximum of just over 4 billion unique addresses, which seemed like a lot in the 1980s but is entirely insufficient for the growth of the modern Internet, a newer Internet Protocol called IPv6 was introduced, with a new, longer address. An IPv6 address is represented as 32 hexadecimal characters (0-9, a-f) divided into groups of four that are separated by colons: 2001:067c:2ebc:abcd:face:1967:b33f:cafe, for example, would be an example of a valid IPv6 address. This new format, allowing for 16^32 (16 to the power of 32) addresses, or about 340 undecillion addresses, provides plenty of room for the internet to grow. However, until almost every device on the internet can use IPv6, new devices with IPv6 addresses generally need to use both older IPv4 addresses and newer IPv6 addresses to communicate. We are currently partway through a decades-long transition period where online services are made available using both IPv4 and IPv6, as Internet providers and corporate/education networks slowly activate IPv6, and eventually will deactivate IPv4.
Major services such as Google, Facebook, and LinkedIn have been available using both the older IPv4 addresses and the newer IPv6 addresses for over a decade. Around 70 percent of French internet connections support IPv6 (the third-highest ranking in the world in the world), and more than two-thirds of Internet traffic on IPv6-enabled networks (where both IPv6 and IPv4 addresses are available) is routed using IPv6. However, corporate and educational networks are notorious for being very slow to implement this new IPv6 connectivity and continue to rely only on the older IPv4 protocol, often due to a lack of resources or training. As a result, equipment that is provided by major networking companies is often better-tested with IPv4 than with IPv6, even if the equipment is certified to work with both.
AUP activated IPv6 connections on its network in the first part of 2023, providing every non-wireless device with both IPv4 and IPv6 addresses. This allowed our Internet connection to be more reliable, decreased latency, and allowed us to double our available bandwidth to the campus. IPv6 was not activated on our Wi-Fi network at the time, because our old Wi-Fi system did not support IPv6 addresses. When we installed the new Wi-Fi system, IPv6 became instantly available on Wi-Fi, allowing us to offer complete IPv6 and IPv4 connectivity to our community. Over the course of the summer no issues were reported, and our tests showed that the system was working as expected.
After almost two months of applying pressure on the Wi-Fi equipment vendor, they notified us that they had made progress. They noticed that something about IPv6 (reminder, a protocol that is not yet widely implemented with their customers) was causing the Wi-Fi access points to crash and have performance issues. They then created a network in their lab that was similar to ours, and started transferring large amounts of data over it using the IPv6 protocol, until they found a configuration where their test equipment started crashing the way that ours did. The culprit? A piece of software in the device, tasked with converting IPv6 "multicast" packets to IPv6 "unicast" packets, was simply not able to keep up with the high IPv6 traffic level on the network that started when students arrived on campus for the Fall semester. Given that this software is essential to our IPv6 network configuration and cannot be disabled, their suggestion was to temporarily block IPv6 traffic on the Wi-Fi network, to effectively work around the issue in their Wi-Fi system.
After temporarily disabling IPv6 traffic on our Wi-Fi access points, as the vendor recommended:
Because virtually all online services are still offered using the old IPv4 protocol, we made the decision that until the vendor fixes their devices' handling of IPv6, we will leave IPv6 deactivated, offering a reliable, fast connection using only IPv4. This is the current state of the network, and it has been reliable: an IPv4-only network, similar to what we had prior to summer of 2023, but with better performance and connection stability than our old Wi-Fi system that had been replaced.
Because implementing IPv6 has technical advantages, we continue to hold conversations with the vendor regarding the development of a permanent solution, and have had regular meetings with their US-based Director of Product Management to receive updates on their progress. As the vendor works to resolve this issue, we will keep IPv6 connectivity turned off on our Wi-Fi networks, to maintain a high quality of service using the IPv4 protocol. When the vendor provides us with a technical solution that will allow us to re-activate IPv6 on the Wi-Fi network, we will deploy it slowly, with extensive testing, until we are satisfied that they have fixed the issue in such a way that we can reactivate IPv6 campus-wide.
We would like to thank the AUP community for their patience as we worked to identify the problem and to resolve the issues that people have experienced. We encourage all members of the community who have any questions or concerns relating to Wi-Fi at AUP to visit the IT Services office in room Q-A201, or to e-mail itservicesaup.edu.