Building a Distributed Team

I recently saw a post on Twitter that claimed something along the lines of “Remote work is mostly BS”, which inspired me to write this blog.

When we started Apstra, we stated an explicit goal to provide the ability for remote workers to join our team, and be productive, happy, and successful — even in a startup environment. It was a bet because we had mixed success in past experiences in achieving this. In those situations, most of the work still happened at headquarters, and while some team members worked well remotely, many remote workers felt disenfranchised and ultimately either left or did not deliver to their full potential.

We believed we could do better and achieve the goal of enabling remote work, and having a truly distributed company. I thought two factors would have a determinant impact in accomplishing this:

  1. Having the right practices, rooted in the right values and culture

  2. Using the right technology

Looking back over these past few years, I’m proud that we have largely succeeded in our goals. We have team members in various states in the US, as well as Canada, Europe, and Asia. From individuals working from home, to groups collocating at coworking spaces, it’s worked quite well across all departments, including Engineering and Marketing, and job functions, including managers and executives. It has helped us tap into the best talent, wherever that talent happens to be, and to scale the company.

In this blog, I’d like to dig a bit more into the two factors that helped us achieve this.

Having the right practices, rooted in the right values and culture

We have found that to enable quality remote work, it is critical to have a culture that embodies a set of practices, enabled by the right set of values. Here are some examples of such practices:

  • All team members, including those working remotely, have to explicitly block the times in their calendars when they’re unavailable; and make it clear when they are available. This makes it clear to others when they can schedule meetings with them, or reach them for questions or queries.

  • When team members make it clear through their calendars that they are indeed available, then they should make themselves available for when their colleagues reach out to them, e.g. on messaging tools, or for a video session or a phone call. Done well, this is even more efficient than having to walk to someone’s desk or to a conference room to meet in person.

  • When remote employees join Apstra, they usually start spending a few days at our headquarters, for onboarding. During that time, they’re assigned a coach or a set of coaches to answer any questions that they may have, to show them around and to introduce them to the rest of the team. This way, the new employee gets immersed in the company and feels part of the team before they get back home away from headquarters.

  • We recognize that there is a human element to meeting and interacting in person, and personal connections need to be maintained. To achieve that, the company encourages, and pays for all the travel expenses for remote folks to spend time at the Apstra headquarters at a regular cadence, usually every month.

  • Every meeting includes a web and video conference link along with a physical meeting room. In the same vein, all meeting rooms include a video/web conferencing system. That is, meetings are “location” agnostic — team members can participate from wherever they choose: they can choose to join the room in person, join from their desk at home, or a coworking space close to their home. The result is that all team members, independently of location feel like “equal citizens” and participate equally independent of location.

  • At Apstra, we have developed a clear development process so everyone knows where we are in developing a feature and what the priorities are without having to hang around the water cooler. As part of this, design, implementation, test docs are compiled from all the design decisions into a coherent whole, to keep everyone aligned, whether they are in the next office or the next continent.

There are other more subtle rules that we put in place to improve the experience of folks working remotely. One such rule is that in meetings that include participants that are remote, remote folks are given preemptive priority when they speak. This gives confidence to the remote employees that their voices will be heard even if they’re not present in the room.

Using the right technology

These practices depend on having the right tools. Indeed, using the right technology is critical in enabling remote work. At Apstra, we’re big users of collaboration tools. One can reach anyone, at any time using our messaging tools. Messaging tools are available not only on team members’ laptops, but also on their phones. These technologies enable us to video call anyone, or video conference any set of people at any time.

We also heavily use collaboration tools that enable folks to collaborate, in real time, over documents, spreadsheets, or presentations. In fact, the level of collaboration over these documents has become so powerful that meetings are no longer necessary to finalize a plan or any type of collateral.

Apstra also uses project management, wiki, and group chat tools that are built to enable collaboration. We did also try other tools, such as the robots that remote workers can control remotely with less success, except for fun and as a recruiting tool!  

Note that many companies use those types of tools. What is critical here is to tie the use of these tools specifically to support our best practices. So we are thoughtful about the reasons why we deploy the tools. Otherwise, we run the danger of being “tool-happy”, and deploy tools for the sake of deploying tools, which could create confusion and negatively affect productivity. In that same vein, the tools we deploy are just there to enable collaboration and cooperation, and are not allowed to interfere with connections between team members.

Building a Distributed Company

As an Intent-based networking company, our intent was to build a distributed company where remote employees can network with other employees wherever they are. As with the intent-based networking we provide to our customers, achieving this intent required instituting best practices in the company to allow efficient distributed operations; and using the appropriate tools for the job. As a result, we are well on our way to build a company where remote employees can contribute from wherever they are.

Also, the same tools and our company’s culture have also paid dividends and benefited our employees who are based in headquarters. They can enjoy the same flexibility to easily work from home or temporarily from a remote location when needed. So our efforts to strive at being a great distributed organization have significantly helped us achieve our goals of being a great company overall.

So independently of where you are located — if you’re interested in joining the company that pioneered Intent-Based Networking; if you want to be part of our mission to radically automate infrastructure; and if you believe you have the intellect, values, and skills required to join our talented team, then please contact us, we’d love to hear from you!

* This article was originally published here

Intent-based Networking and Threat Mitigation

In my previous blog I discussed Intent-Based Networking and security and how the adoption of Intent-Based Networking allows users to greatly enhance the security posture of their network. In this blog we dig deeper and see how an Intent-Based Network can help with threat mitigation.

Intent-Based Data Center Automation

Enterprises are implementing Intent-Based Networking to realize efficiency, agility, and significant OpEx savings. The technology has matured and is now widely deployed across various industries as described here. Intent-Based Data Center Automation leverages Intent-Based Networking technology to achieve a new level of data center transformation.  I’ll use Intent-Based Data Center Automation to show how Intent-Based Networking can help with threat mitigation.

So, what is an Intent-Based Data Center? Data centers have evolved significantly over the years; the following figure shows the evolution of the data center over the past few decades.

This white paper describes in detail the evolution of the data center and the following figure depicts some of the capabilities required to implement an Intent-Based Data Center.

The functionality outlined in the previous figure needs to be well supported, and allow users a choice of hardware and switch OS, choice of workload (e.g. bare metal, virtual machines, containers and microservices), and choice of cloud (e.g. Microsoft Azure, Amazon AWS, Google GCP, etc.).

Intent-Based Networking Solutions

As I mentioned, Intent-Based Networking solutions are widely deployed in production networks. As these solutions mature, it is important to qualify the maturity and capabilities of an Intent-Based Networking solution. Sasha Ratkovic describes the maturity of various Intent-Based Networking solutions in this excellent blog, and as depicted in the following figure.

In order to leverage Intent-Based Networking for threat mitigation, the solution must be at Level 2 or higher as explained in the Intent-Based Networking taxonomy blog. Real-time change validation is a key capability and a fundamental requirement for an Intent-Based Networking solution to be able to effectively tackle threat mitigation.

Data Center Security

As we see adoption of Intent-Based Networking technologies to help drive efficiency, agility, and OpEx savings, Enterprises, Cloud Service Providers, and telcos should also think about the security posture of their data center and leverage Intent-Based Networking constructs to enhance the security posture of their data center. As the latest trends are discussed this week during the RSA Conference in San Francisco. it will be interesting to see where Intent-based Networking falls into the fold.

Data center security has two major elements:

  1. Perimeter security: Securing North-South traffic, which is typically handled by a next-generation firewall.

  2. Securing East-West traffic (traffic inside the data center, typically server-to-server traffic between tiers, containers, and/or microservices).

There is a lot of innovation in the industry to secure East-West traffic. Application architectures have evolved drastically, shifting most of the traffic in modern data centers to East-West, rather than North-South. In this blog we will explore how Intent-Based Networking can help mitigate the threat to East-West traffic inside the perimeter and complement other solutions to help drive efficiency in security operations for the data center.

Threats to East-West Traffic

In order to secure East-West traffic there are a few key challenges: there is limited visibility as the traffic is typically high speed (40G, 100G or even 400G Ethernet) and it is several orders of magnitude larger in volume than North-South traffic. This is a very conducive environment for an adversary to conduct an attack with minimal risk of detection. Attacks can occur can in a variety of ways. For example let’s look at two typical methods of attack used by adversaries and how an Intent-Based Networking system can help mitigate the impact. We will analyze several real-world examples.

Administrator Credential Compromise

In this mode of attack a network administrator is a victim of a phishing attack, or the adversary is a malicious insider. With this type of compromise the adversary can operate with the privileges of the administrator. In this mode, the attack is from the inside and the perimeter-based security solutions will not be effective as the attack is happening behind the firewall. In this scenario the adversary will typically use lateral movement to find vulnerable spots inside the data center and then figure out how to establish Command and Control (C2) communication to further the attack. In order to mitigate the attack, it is necessary to detect the lateral movement and the C2 communications.

An approach could be to look at various metrics and analyze “behaviors” of various elements in the data center and use that to detect suspicious behaviors and attribute the same to a given threat and mitigate the impact of the threat. This is not straightforward as the the system needs to collect metrics across various nodes and run queries and analytics on collected data and determine patterns (“behaviors”) that are considered normal (good) behavior and then detect suspicious behavior. Note that there is a lot of innovation in this area with concepts like Artificial Intelligence (AI) and Machine Learning (ML) being utilized to detect these behaviors. We will explore how an Intent-Based Networking system can help with detection of suspicious behaviors.

System Vulnerabilities

Another typical attack vector is when adversaries exploit system vulnerabilities to gain unauthorized access to the network. We see news about various vulnerabilities discovered and impacting systems, and software vendors working diligently to provide patches and mitigation steps for detected vulnerabilities. We see a huge adoption of open source software, which is excellent as an innovation driver. However, the fact that the source code for open source is available by definition, it makes the software vulnerable to exploits.

A security system has the responsibility to enable users to leverage open source software securely. In these types of attacks the adversary will use a valid authenticated connection allowed by the perimeter firewall as an authenticated user and then leverage a known vulnerability in a given network OS, server OS, or a server’s hosted application and use techniques like privilege escalation to get administrator (or root) level access to systems. Once this happens the attack approach is similar to the credential compromise and mitigation starts with detecting suspicious behavior.

Intent-Based Networking Systems and Threat Mitigation

With Intent-Based Networking, the system is designed to provide the ability to reason about various metrics along with their context (e.g. role of the element, and its relation to other elements in the network) in real time. What I mean by real time is that you do not necessarily need to collect and store all the network state data and correlate it by querying a datastore offline. Intent-Based Networking solutions offer the ability to validate your intent (expected behavior) with network operational state as the system processes the data.

If intent is set up to detect suspicious behavior, the system will provide alerts when a deviation from expected behavior is seen in the network. This is known as continuous real-time validation and is a key differentiator in how Intent-Based Networking systems can greatly optimize how you detect and mitigate threats.

Based on a declarative specification of intent, the system knows what network state needs to be collected and analyzed and it can do this for every iteration of network telemetry. In addition, sophisticated Intent-Based Networking systems can collect interesting data for further analysis and use the concept of data pipelines for more advanced processing like trends, baselines, averages, standard deviations, or even the ability to apply custom functions to relevant and context-enriched data.

Detecting an Attack — a Real World Example

Let’s take the example of an attack called “DNSpionage” where DNS request/response payloads were used by adversaries to exchange infected system and C2 information and further the attack.

To counter this, an enterprise could have had in place a set of data processing pipelines for proactively detecting this type of activity. The proactive monitoring should look for the following:

  1. Monitor the DNS request/response payload sizes and report a deviation in payload sizes.

  2. Detect new behavior in systems related to startup folder changes (or task scheduler or cron job changes) and alert if there is a deviation from normal behavior. A security administrator can program certain known suspicious patterns and trigger these detections.

  3. A security researcher can also program “known” suspicious processes and the system will report any instance of these processes or even process command lines if there are observed and as soon as they are observed.

Note each of the above “behaviors” in isolation could be normal operation, but when they happen in conjunction with each other and are detected on systems related to each other, it could indicate an attack in progress, This is an illustration of how an Intent-Based Networking solution can be used to detect anomalous behavior, alert security operators, and greatly reduce their steps to help detect and catch new and existing variants of known threats or an attack in progress.

Note that this approach can be very effective if the Intent-Based Networking system can detect several such “behaviors” – also known as “indicators of compromise” (IOCs) – in the order of several hundreds thousands (or even millions) in real time. The system should also be flexible enough for security researchers to be able add or update new “behaviors” without major disruption and as a part of normal operation. In traditional systems this would mean collecting logs from various elements and using logic outside the core system to correlate various events by using data analytics on large amounts of data.

Summary

Cyber attacks are becoming increasingly sophisticated and use automation and other methods to create variants of previous techniques to generate new day-zero threats. It is a constant battle to negate the threat of cyber attacks. A solid approach is to have security baked into normal network orchestration, automation, and operation; in order to do that one has to start with an Intent Based Networking approach and then leverage the “composition” of an Intent-Based Network to be able to detect anomalous behavior. The figure below shows how an Intent-Based Networking system can effectively leverage network context and help with threat mitigation.

Intent-Based Analytics (part of Intent-Based Data Center Automation) provides a powerful threat detection pipeline to map attack signatures to indicators of compromise.  Intent-Based Data Center Automation is then able to do correlation in real-time to determine based on the indicators of compromise present, which attack signature is being used, and where the threats are detected, which can then be used to take action to remediate the threat. All of this is only possible with a true Intent-Based Networking solution.

* This article was originally published here