Tomorrow belongs to
those who prepare for it today.
Tomorrow belongs to
Breaking Blog Silence
I’ve purposely been avoiding posts for a while as our Prairie Line Express initiative would have likely been the elephant in the room. As we wind down the first phase of that initiative, it’s time to start posting again.
KanREN is a week and a half away from our 2017 Annual meeting. My post this morning is a reflection on where we are as I head into preparing for the meeting. In no particular order:
- Our FY18 budget, approved back in January finally began to provide some financial relief to our universities in the form of service volume scaling.
- We’re very comfortable with a 100Gbps/10Gbps hybrid backbone now… oh yeah, with increased reliability too.
- We’ve started a project to build a “sub-ring” in Kansas City for growing member needs there.
- We’re in the middle of an initiative (with a pilot service) to make gigabit service affordable to typical KanREN members.
- We are leading the nation with no-cost inclusion of comprehensive DDoS mitigation and scrubbing for FY18.
- About to unwrap substantially improved internal/external network monitoring and visualization tools.
- Finishing a complete overhaul of our business documents and processes around them (thank you Finance Committee!!!).
- Have been working hard to stay committed to longer-term objectives and measuring your progress.
- And have seen more desire to serve on our board of directors from the membership than I’ve seen in many, many years.
- …This may be my favorite. The Capital planning/reserve program started many years ago have now been fully utilized for a complete cycle of backbone and CPE replacement. Without assessments to our members.
I don’t want to say it’s all coming up roses at KanREN, but as I move into annual meeting presentation mode, I think even in these uncertain times for education in Kansas, we’re as strong as ever — We owe that to a dedicated staff and an involved membership.
Attending a Technical Meeting
I had the rare opportunity today to attend a meeting with several of our biggest members and some of our staff. This is a technical planning meeting about some “new stuff” we’re looking to do in partnership with them.
The meeting was really good, and I think we’ve got a great solution (thank you Brad Fleming). But the meeting ended early, and we all stayed to talk. I started listening to the member technical folks talking to each other about a number of things they’re doing.
I was amazed at how often the word “KanREN” came up in their discussions. I’ll give one example here that really makes the point. They were talking about phones, and I listened to more than one of them talk about how KanREN picked up their voice traffic on a different port, transported it to the SIP trunk provider as priority traffic, etc.
In every case, I know this is something we just did for them. We didn’t charge anything, nothing one-time, nothing recurring. We just did it. Then I remembered some conversations with another member who did something similar, but didn’t work with us. I know that they ended up paying for a dedicated circuit from a 3rd party carrier with a multi-year commitment… You get the point.
Throughout the entire conversation, I kept hearing things like this come up. It’s nice reassurance that we are doing our job, and these members are getting a huge amount more than “Internet” from us. The best part of all was that every time KanREN’s part of the picture came briefing into focus, it was alway just a given…. Yeah, no problems because KanREN just did that for us.
This makes me feel good. I think about our staff mostly — the people who are actually interfacing with these people all of the time — and that they’re doing a great job making sure we keep true to our cause and do what I keep promising we’re doing.
Fees, Budget, Performance
As I wrap up the FY2018 budget and associated fee structure, it’s never more clear to me what KanREN is here for. I hope that sounds odd, because that’s how I want to hook you into continuing to read this post.
KanREN, like every organization is a balancing act. Ours has a very tight closed loop control system. Our members are the recipients of our services, form our organizational governance, determine what services we will offer and determine what they will cost. When I prepare an annual budget with our board Finance Committee, it’s ever so clear to me that what we’re doing and how we’re doing it really is member driven.
It also reminds me how much we do that cannot be quantified on a rate sheet. In particular, the number of things we provide as included (DDoS mitigation, large IP address blocks, technical support, etc.) because that’s how you run an R&E network focused on it’s members — and how different that is than a typical telecommunications carrier. Nothing against the carriers, but their business is making money for their shareholders.
This year, I believe our Finance Committee challenged me with the toughest budget ever. The committee clearly identified several priorities for me… and they’re often competing priorities. For example, it’s hard to reduce per-unit fees, but increase our operating reserves, all the while ensuring our ability to support our members is not impacted. I’m pleased with the outcome. By listening to what’s important, and keeping that clearly in mind, I have a target to hit, and know where to strike the balance.
I’m proud of the work we’ve done this fall, and excited to present the FY2018 budget to the entire membership in a few weeks. Once again, I’m convinced that we continue to meet and exceed expectations of our membership in delivering high-quality services, with the best support anywhere, and at a price that’s fair and competitive for what we’re offering.
Sometimes We Nail It!
It has been a while, I need to get back into the swing of making posts here. Several of my blog posts have centered on the effort we pay to the quality of connectivity, not the quantity. In fact, twice in the last six months I’ve given external presentations centered on this theme.
As long as I can remember, KanREN has been so closely tied to high-performance networking that this obsession with quality connectivity is in our very DNA. This week, Internet2 is holding it’s annual TechEx (Technology Exchange) conference in Miami, FL. This is a gathering of the nerdiest nerds in the community, and is purely about technical items. Our A.D. for Technology, Brad Fleming is attending.
Internet2 Sr. Director of Strategic Projects, Eric Boyd, presented on the next generation of the perfSONAR measurement infrastructure. In his presentation, he listed “top deployments” of perfSONAR worldwide. This list of eleven deployments included KanREN.
It is often easy to get bogged down in the state’s budget woes, our own financial planning, day to day business and network operations, etc. and forget about one of the most important reasons KanREN is here. This stuff is important. And after I’ve preached the importance of accurate network characterization and how “gigabit” really isn’t always gigabit, why end-user experience is the real target, not a number with “bps” at the end — even sometimes in conflict with respected advisors — I sometimes start to feel like I need to take the tin foil hat off. Then something like this comes around and I realize, we’re doing it right. What we’re learning and how it affects our members is important.
Just recently I tried to explain to some of our largest members that while we have to watch the “industry standard usage stats”, because that’s how we get billed, and part of making sure we’re fairly cost-allocating. But traffic graphs, at their very best, are still an “average” view of what’s going on — even very, very fast updating ones. Making sure everyone is paying their fare share is a business necessity, but trying to ensure that there aren’t tail drops, packets aren’t delayed in queues and latency stays stable (and low) — that’s how you make end users happy, and that’s where we really spend the majority of our time working.
I often talk about the things that “aren’t my job anymore”. For those new to KanREN, this month celebrates my 17th year at KanREN. I started on the technical side as a UNIX systems admin (not Linux in those days), and worked my way through just about every position here before my “job-jumping” stopped at Executive Director in 2008. As an “engineer by nature”, I often comment about the technical work that I used to really enjoy… I’m sure my memory has somewhat romanticized the “fun” that I had when I was one of the network guys. I digress…
KanREN is a small shop, and we all have many responsibilities here. The last purely technical role I have here is as KanREN’s “electrician”. When sizing power circuits, PDUs, breakers, and in particular -48VDC power; I’m still the person who usually gets called on to do the work, or at least to consult or review specifications.
But even as Executive Director, being the “power guy” has a benefit. I try hard to stay close to the staff and their work. While putting on a t-shirt and shorts and heading over to the Internet2 colo to wire up -48VDC to a new Juniper MX960 is actually fun and refreshing for me, it also gives me a chance to be “just one of the guys”. It helps me keep perspective. It makes me a better leader.
Last week I had just such an opportunity — twice actually. The first time, Brad and I got ready to start doing the work before we realized we’d brought the wrong gage wire. The second time we were successful. I sometimes struggle with the value to the organization in me taking a day off from my normal duties for an activity like this. But when I do, I always end up thinking it was well worth it. This time, as expected, the work gave me perspective as a leader, let me have a little fun doing something I love to do, and gave Brad and I a chance to spend some time together as co-workers. Getting to work with the staff, just like anyone else, is really the best part.
Dyn’s Baker’s Dozen
I just saw Dyn’s (formerly Renesys) “Baker’s Dozen” article for 2015. It’s one of those little things that reminds me that our obsession with quality connectivity really does make a difference. The article describes Dyn’s view of top Internet providers based on their own Internet Intelligence – Transit product. Two of the providers that KanREN uses for three of our six upstream connections, Level(3) (KC and Wichita) and Telia Carrier (KC) were ranked #1 and #2 respectively. And a third, Hurricane Electric, made the cut in the Baker’s Dozen.
KanREN has a total of six commodity upstream connections spread over three of our backbone POPs. It’s something most of our members probably don’t think much about — go get some Internet, the carriers are all the same, right? Wrong. They’re not. KanREN staff are constantly evaluating Internet providers. We balance several things in making our selections. We look at reliability, reachability, reputation, routing table analysis, applicability to our particular member base and of course cost. We work hard to balance all of these (and more) aspects in creating our portfolio of upstream providers to create the best possible end-user experience, while controlling costs.
By building our backbone into key facilities in Kansas City and Wichita State University, KanREN is able to leverage direct connections with nearly any provider out there — something you can’t do unless you’re in these key locations. Together, through KanREN, our members have access to a portfolio of Internet carriers that wouldn’t be possible for them on their own. I’m proud of our work in building our portfolio. This is likely another thing our members don’t think much about — that’s great, they don’t need to, because we’ve got them covered!
It’s Not About Bandwidth
As my time at the Internet2 Global Summit winds down, I think about some things I’ve been predicting. I’m more convinced the future is coming fast. And it’s not about a number with bps behind it.
Sure, you have to have bandwidth, but soon, it won’t be the end-all be-all of connectivity. Really it already isn’t. For large institutions and R&E networks, the days of drawing most of our “upstream” bandwidth from commercial Internet connections may be numbered.
End-to-End performance is the name of the game, and once you’ve got bandwidth, what do you do next? You eliminate the middle man and all of his foibles. Already we’re starting to see this happen. KanREN is working on a number of paths through our neighboring R&E networks and Internet2 that get us closer to content and cloud providers. I announced our intent to begin moving in this direction at the KanREN annual meeting last month. This week at the I2 Global Summit, I see even more evidence this is coming.
There’s been a fair amount of discussion about cloud and content provides, how we create environments within our national backbone to carry their traffic. Phrases like “deterministic service” got used a lot. Leveraging our infrastructures to create “express paths” to content and cloud providers. That’s what we’re going to be doing.
We are already seeing cloud providers and carriers offering, for a fee, more direct connections to critical cloud services. It’s done with things called “circuits”, and you might have heard of them. In the old days, circuits to content or application service providers were a lot more common. Then came the Internet, and we could just get everywhere with that one connection to it. But now these services are more critical, and we need to ensure performance… full circle.
This will be a major focus for the R&E community. In a way, it’s what research networking has always been about. It’s why Internet2 was created; to build a network that looked a lot like the Internet, but provided more stable, robust and… dare I say “deterministic” performance for science and big data. Now we have another use. Now it’s time to bring the same concepts to bear for production content and applications services (i.e. the Cloud).
Last year, the LHC; next year, Microsoft.
Relevance of R&E Networking
This week I’m attending the Internet2 Global Summit (read: member meeting) in Chicago, IL. This is a particularly good meeting for me as I’ve spent the last month mired in Services Memorandums, Membership Agreements and coming year logistics like HR policy updates, etc.
While attending a meeting specifically for R&E network CEOs, I characterized what we, as R&Es, do in a few words, “Our goal is to provide a superior end-user experience; whether end-user is a researcher or a student watching Netflix. We are interested in anything that helps us provide that experience.”
This was in response to Internet2’s questions about certain initiatives and how we, as the regionals (slang term for the state and regional R&E networks), would prioritize them. Simple right? Sounds like it.
KanREN is under constant pressure to justify our expense from our members. Routinely, sometimes even from our largest, founding members, KanREN is called in to explain why we’re not the cheapest Internet they can get. I always struggle to describe why comparing us to commercial ISPs isn’t apples to apples. It isn’t easy, but it should be. KanREN has implemented what our members have asked for: performance, reliability, flexibility, transparency.
This is why R&E networks are and will remain relevant — which is a common question. I believe that as long as our focus is on our members, we continue to meet their needs on time, at scale, we will naturally evolve with our members, our own relevance mirroring theirs.
In the next post: What I think the tangible changes for R&E networking will be in the next 2-3 years.
State of the Consortium – Internet of the Future
I’ve made no blog posts for about three weeks now. Our annual meeting, and most importantly the state of the consortium presentation, really take that much time to prepare for.
Preparing the “state of the consortium” is a time of reflection for me. It’s when I pull together all of the numeric data I can, all of the anecdotes, freeze what’s on the table with our board and in my head and paint a picture of where KanREN is right now, the ground we’ve covered in the last year, and where I think we’re headed.
Right now, I think we will see a lot of changes in the next few years. KanREN has always struggled with quality vs. price when attracting new members. Sometimes I wonder if we already have all of the institutions as members who really are willing to pay more to get more, or have specialized needs that only an R&E network can provide.
All of that aside, it’s clear that we need to work on ways to become more cost-effective… Which has been a constant struggle as long as I’ve been the executive director.
The “Internet” continues to evolve. It was once a vast “any to any” world where content came from nearly any endpoint. Today, it’s becoming a much more tame place in many respects. The majority of content comes from more easily defined and located content delivery networks. The goal is to find more ways to get to that content that are less expensive and have better performance than commodity Internet providers. I can see a future where “upstream” is mostly coming from connections into exchanges and CDNs, and the actual “Internet” we buy is minimized… And when you think about that, think big. How does it impact things like security and DDoS when the majority of content isn’t coming over those generic connections anymore? The next few years will be an exciting time indeed!
Defensible Networks – The New Security
“Security”. A broad topic. Historically it’s largely been out of KanREN’s wheelhouse. So much of the traditional security paradigm has been aligned with individual institutional policy that it’s hard for an R&E network (within the network itself) to meet everyone’s needs. Simple things like BCP38, blocking Windows file services, etc. have been the bulk of what we do — very important things, just not very flashy ones to be sure.
In recent months, while gathering with peers at national level events, I’ve started to see a new paradigm form where R&E networks can take a more effective, direct role in the security puzzle. The Denial of Service Attack (DoS) — and in particular, the Distributed kind (DDoS) are eclipsing many traditional security related events in their impact. Now that being online is absolutely critical, taking an organization off-line is a remarkably juicy target.
Sure, there are commercial alternatives for scrubbing and delivering clean traffic, but the cost is absolutely exorbitant. I believe R&E networks, thanks to their close relationship with their members, will develop a great new set of layered approaches that extend from their member organizations through upstream providers to help mitigate denial of service. I call this, “Building Defensible Networks”. By layering solutions, including that exorbitant scrubbing, black-hole routing, re-alignement of institutional practices with NAT, combining open source and commercial solutions, etc., our community is uniquely positioned to build an adequate defense, at a more affordable price. Why? Because the farther “upstream” you can push denial of service mitigation and remediation, the more effective it is — and with nearly any security solution, implementing it in layers is more effective and efficient.
My prediction is that within the next 12-18 months, making a network defensible will be just as important as making it low loss or low latency. There is already a great deal of collaborative effort happening within the Quilt and Internet2 communities around these solutions. A new metric in measuring performance is upon us, and it’s about defense. KanREN is participating in the effort, and in collaboration with our community will meet the coming challenges!
Why Gigabit Connections Might Not Be
I often hear people talk about why they can get “gigabit Internet” for their homes for (insert low, low price here), but their schools, etc. cannot, and they ask me why. I’ve been struggling for years with an explanation for this… It’s not easy. Without going through all of the dynamics, the TCP protocol (what most Internet traffic uses) has some limitations. To illustrate this I conducted a simple test. I set up two Linux machines to communicate directly with each other over a simulated (and very generous) 1Gpbs connection and tested it with iPerf.
Not surprisingly, the direct achieved a Gbps as expected:
0.0-10.0 sec 1.31 GBytes 1.12 Gbits/sec
But in the real world, there are delays. Let’s see what it looks like when you add 10ms of latency (really, pretty small, and very conservative):
0.0-10.0 sec 670 MBytes 560 Mbits/sec
Ok, that’s not as great… Now, I’ve never seen a DIA contract with a school that didn’t specify something like “< 1% packet loss”. Ok, let’s throw .1% (1 TENTH of a percent) of loss at it too:
0.0-10.1 sec 61.6 MBytes 51.2 Mbits/sec
Finally, let’s get really ugly. Not likely to happen very often, but can on a bad day. Let’s now look at 20ms of latency and 1% of packet loss:
0.0-10.2 sec 16.9 MBytes 13.8 Mbits/sec
Basic testing reveals that just because you have a “gigabit” connection, really doesn’t mean you’re going to move a gigabit/sec. But when you put several thousand connections from a school district behind it, even with latency and loss, you’ll still fill the pipe. Service providers know what I’ve show above. That’s why you can buy a “gigabit” for your home, for a LOT less than a gigabit for your school.
The Quilt meeting with USAC
The Quilt recently sent a delegation to Washington, DC to discuss issues in our community with key organizations, USAC chief among them. USAC’s unwillingness to work with R&Es under e-Rate is well known.
Chris Henderson, USAC’s relatively new CEO has clearly indicated a desire for USAC to become a “facilitator and not a regulator” in the process. The group discussed many common problems R&E’s have had working with USAC and it’s clear that the key will be getting to higher levels in the organization when problems arise.
Chris also indicated that internal performance metrics within USAC will change to incentivize successful consortium applications.
University HCF e-Link Applications Submitted
This week, the first round of Arkansas e-Link applications for HCF funding have been submitted by our public universities. KU, KUMC, KSU, WU, ESU, PSU, FHSU have now applied for funding for the current fiscal year.
If successful, this represents a 65% savings on all, or nearly all of the annual KanREN billing to these institutions. Cocktail napkin savings, if all who have currently applied are approved, will be about $1.3 million dollars in rebates!
MACE pre-conference presentation
Melinda and I presented on the general topic of “future ready infrastructure” this week at the MACE pre-conference meeting for K12 tech directors. As I hoped, it was really more of a discussion than a presentation (read: informal and organic).
While our session was poorly attended, and the pre-conference meeting was about 1/3 to 1/2 of the expected participation, I gained additional insight onto our challenges growing membership in K12 space.
KanREN continues to be viewed as a vendor. Our position as a membership consortium and 501(c)(3) simply isn’t understood by the K12 community. Even the pre-conference organizer, who is a great KanREN proponent, referred to us as a vendor 3 times in as many hours. I believe that this is one of the most pervasive issues surrounding K12 growth for KanREN. But I’ve come to realize that it is an issue that we cannot solve alone; our members must clearly articulate what we are when they talk with their peers.