EDR — Benefits, Concerns and Issues
Published 17 July 2018 – ID G00319345 – 23 min read
Security and risk management leaders increasingly look for detailed visibility, actionable insight and tailored remediation endpoint capabilities. But misunderstanding and overestimating the capabilities of EDR offerings and the effort needed to leverage them can cause more issues than they solve.
Endpoint detection and response (EDR) solutions remain very complex to operate.
For all the vendor and industry talk of AI and machine learning, EDR solutions continue to rely primarily on the oversight of highly skilled humans to identify and resolve issues.
Typical organizations that face normal budget and staffing challenges are ill prepared to leverage and maximize the benefits of EDR solutions by themselves.
Organizations with low maturity endpoint maintenance and management programs experience higher EDR workloads.
Detecting and responding to incidents or events caused by vulnerable applications or operating systems reduce the value of having an EDR solution vis-a-vis a vulnerability-scanning platform.
Managed EDR solution provider capabilities vary dramatically among vendors and regions.
SRM leaders who are weighing the deployment of an EDR solution must:
Establish well-defined security operations and incident response programs with mature vulnerability and patch management processes already in place.
Focus on post-event analysis and response capabilities rather than active hunting, detection and response; this is especially true for Type B and Type C organizations.
Deploy EDR as an active detection and response platform and plan to incorporate a managed EDR solution to complement their internal capabilities.
Shortlist providers that offer technical assistance in incident response to supplement staffing.
Organizations have long had the ability to look at detailed log and forensics data from their network and perimeter solutions. Operational data from firewalls, gateways, proxies, networks and other sources have been part of the routine post-event forensics analysis process for many years, with organizations often leveraging security information and event management (SIEM) solutions as their central repository and analysis platform.
Until the advent of EDR, the traditional approach of collecting forensic data from endpoints has been on a reactive basis, where a forensics tool would be deployed to target post-event endpoints and the data collected would depend on what the operating system logged. EDR provides organizations deep granular endpoint data that they have been accustomed to getting from network and perimeter solutions.
Table 1: EDR — An Overview of Principal Benefits, Concerns and Issues
Recording of context-rich endpoint event and state information.
Pricing of EDR solutions remains at a premium.
Incident data collection and analysis occur post-event with limited incident response automated capabilities.
Option to store collected data on endpoints themselves, centralized servers, the cloud or as a hybrid of these.
Requires the installation, management and updating of yet another agent.
EDR provides very limited to no contextual insight outside of the endpoint data it collects, requiring manual intervention to correlate data with such external sources as firewalls, CASB, etc.
Data retention periods can support the operational needs of different organizations.
Support of EDR capabilities varies by platforms and versions of operating systems.
Knowledgeable staffs with EDR experience are extremely difficult to find and come at a premium.
Ability to search collected data to identify issues on one or many endpoints at a time.
Requires staff with strong knowledge of endpoint operations to obtain benefits.
Vendors and managed service providers offer staff augmentation, but capabilities and costs vary dramatically.
Currently available solutions now appeal to a broad segment of organizations with differing technical abilities.
AI and machine learning remain mostly marketing terms rather than actual product capabilities.
Contrary to many clients’ understanding of the products, EDR does not resolve fundamental security and operational issues within organizations, nor does it eliminate the need for basic hygiene and patching.
Source: Gartner (July 2018)
EDR agents are akin, in their most basic form, to flight data recorders, or “black boxes,” on airplanes. Black boxes record all of the technical and operational data of aircraft including heading, speed, altitude; positioning of the landing gear, ailerons, flaps; weight, center of gravity; plus much other technical data including pilot conversations. Black boxes do not record passenger conversations.
EDR solutions record all of the technical and operational data of endpoints including IP, MAC, DNS data, connected USB device information, network connections and ports, running processes, device drivers, threads and their related metadata, windows services, loaded DLLs, CMD and PowerShell command history and memory contents and much more. EDR solutions do not record such application data as what is typed in a Word document or email, although they may scan files for malicious macros. EDR solutions can store all of this data or only the most critical elements either on on-premises-located servers, on endpoints themselves, in the cloud or as a hybrid of them depending of vendor solutions.
This data is typically stored for a period ranging from a few days to several months. EDR solutions provide organizations with the ability to analyze and search such detailed endpoint data by using filters and Indicators of Compromise (IOC) along with other data sources and search parameters.
Organizations can use EDR solutions to search for traces of malicious software and activity, patching data and other endpoint-related activities and can even help answer such day-to-day operational questions as how often a particular application has been used in the past month on a single endpoint or on all the endpoints in a department or across the organization. The questions that can be answered with an EDR solution are quite boundless, but most organizations use EDR specifically to address security-related questions, because that is where EDR solutions provide some of their unique visibility and insights — and ultimate value.
Most, but not all, EDR solutions provide capabilities that can manually or automatically remediate or trigger remediation processes, alert conditions on endpoints either from within or as parts of an integration with such third-party tools as system patching and updating solutions. Levels of capability vary dramatically between vendor offerings. One example of an automated remediation is one in which — on the detection of ransomware activity on an endpoint — the network drivers for that endpoint are disabled to prevent the spread of the ransomware.
EDR Appeal Crossing Organizational Types
EDR solutions have become more broadly available from both next-generation vendors and traditional endpoint protection platform (EPP) providers. As a result, EDR solutions have transitioned their appeal from being the sole purview of Type A or lean forward or leading-edge organizations to Type B and even Type C organizations.
Type A organizations represent the smallest group of organizations. They adopt new technologies very early in the adoption cycle and have the budgeting and staffing resources to configure and implement new technologies and solutions rapidly within their environment. These organizations tend to focus on best-of-breed solutions that best address their business, technology and security needs and have the capacity to integrate, develop and build custom-made components as required. They see the use of technology as a competitive differentiator. Their tolerance for operational risk is high and their approach to technology change is to run projects in parallel by tasking multiple teams to work on technology and business changes simultaneously.
Type B organizations represent the largest group of organizations. They typically experience budgeting and staffing resource constraints and, as a result, focus on overall value by weighing the risks of the early use of new technology against the benefits. Their goal is to stay relatively current on technology without getting too far ahead of or behind their competition and focus on technology deployments that improve their organization’s productivity, product quality, customer service and security. Type B organizations typically wait for a technology to become mainstream before considering implementation. They tend to be moderate in their approach, frequently using benchmarks within their industry to justify their investments in technology. Type B organizations balance innovation with reasonable caution when selecting new solutions. This is the highest growth market for EDR at this time.
Type C organizations represent the second-largest group. They typically view technology as an expense or operational necessity and use it as a means to reduce costs. These organizations experience severe budgeting and staffing resource constraints and, as a result, prefer simple-to-deploy and -use integrated solutions with managed service add-ons that can best complement their minimal staff. These organizations wait for technologies to become absolutely stable and for costs to acquire and operate to reach the lowest quartile before committing to purchase. They focus on prevention rather than on detection and response capabilities and on solutions that are integrated and offer a complement of managed services. EDR is typically deployed in Type C organizations when available in conjunction with an EPP solution. This market is one that demonstrates very slow growth for EDR.
EDR solutions provide enhanced capabilities over traditional endpoint security solutions and can create a force multiplier of staff, but these capabilities have their drawbacks.
EDR Capabilities Come at a Significant Cost
While product costs have on average dropped by roughly 35% per year over the past four years, products remain priced at a premium versus other endpoint solutions even today. They typically range from one to three times the cost of a traditional full EPP suite.
Many of the renewal quotes that Gartner has reviewed over the past 18 months do not always show pricing reductions that are in step with the market. This means that organizations that are renewing an EDR solution originally acquired three years earlier often have to put in significant effort to push pricing down to today’s market price averages (typically seen in new deployment quotes). The initial quote offered for a renewal is often only slightly reduced or perhaps offered at exactly the same or even slightly higher pricing than what was negotiated in the initial purchase several years earlier.
An additional cost to consider is the distribution of other agent endpoint software. While most EDR agents are relatively small and represent minimal impact on system memory and CPU resources, they do represent yet another component that needs to be distributed and managed on the endpoint. While there have been minimal reports of agent issues due to updates of endpoint software components or the operating system itself, from time to time clients have reported issues that have temporarily locked systems until refreshed.
Significant capability differences also exist between EDR agents available by vendors for Windows 10, 8, 7, XP (if available); Windows Server version; Mac and Linux. Mobile device agents are currently not available or offer very elementary capabilities. Some EDR agents can record only some of the endpoint activities on some operating systems and not on others. Other agents have limited or no prevention or remediation capabilities on some platforms. This can result in a patchwork of security solutions that are inconsistent across organizational assets.
Finally, EDR solutions can only monitor systems that have the EDR agent installed. That can limit visibility in an environment containing populations of BYOD where the EDR agent has not been deployed. Plus EDRs for cloud workloads like containers and Internet of Things (IoT) devices are currently not available, which limits visibility into critical operational components.
Perceived Versus Actual Implementation
A simple way to explain the perceived versus the actual implementation of an EDR solution is by way of an analogy. I enjoy fishing. My young son also enjoys fishing. Our idea of father-and-son fishing is quite simple: My son gets his movie-character-themed fishing rod, we buy a small container of worms and we visit my friend at his lakefront cottage. We fish right off the dock. Within 10 minutes, my son usually has caught nearly a dozen fish — admittedly very small fish — but the excitement and energy are at a high peak. After that engaging means of activity, he is pretty much done fishing for the day. As far as fishing is concerned, we accomplished our goal with minimal effort and maximized our fun in the process. Success!
Most organizations expect their EDR solutions to operate in a very similar way to my son’s experience of fishing. Open up the console, have just about anyone enter “ransomware” or some other generic search term and all of the key events will be triaged and organized from severe to benign with a pull-down list of automated and contextualized remediation conveniently available right beneath their fingertips. All that is left to do is to click away and all the organization’s security problems will be solved. Unfortunately, the reality is quite different.
While it is true that many EDR solutions now provide simple guided search operations, most organizations still do not know what they really need to search for. Also, the work of reviewing or even obtaining some form of a basic understanding of what a particular event means entails that triaging and assigning a severity and then determining the best course of action remain the responsibilities of the console operator.
Continuing with the fishing analogy, operating an EDR solution is in fact much more like my experience of fishing with my friend. He is by all accounts a truly expert fisherman. He could easily have his own TV show if only he had better jokes. When I go fishing with him, it is a lot of work for me. It turns out that fishing is serious business after all — and it requires a lot of planning.
The first question he always asks me is, “Which fish do you want to catch today?” My answering “the one that lives in the water” is never a good reply and puts a serious damper on the start of our day. So I have learned over the years to turn the tables around and use his expert knowledge to start things off in a better way to help me determine what fish we should be fishing for that day. I start by asking him questions like: “Which fish can we find in this lake?” “Which of these fish would be most active based on the time of the day we will be going out?” “Which fish would be most active based on the temperature, position of the sun, the wind, etc.?”
In fact, I am using my friend to guide me down the assessment process to identify our target fish. Once we have determined the fish we are looking to catch, I then use my friend to guide me down the next set of decisions, such as where we will go to catch this fish, which rod, line, lure, etc. we will use, at what depth we will cast our lines and so on. He is my expert coach and without his help I would never have any hope of actually catching the fish we had decided was our target for that day.
While EDR solutions are being sold and deployed in more typical Type B and some Type C organizations, the unfortunate truth is that, even with all the marketing emphasis and industry talk of AI and machine learning being applied within EDR solutions, AI and ML are still at a very early stage of maturity, and EDR vendors still expect your organization to have talented experts operating the console.
AI and ML Gone Missing
Today, EDR solutions do not come with an EDR version of my friend bundled in like an “analyst in the box.” They do not come with a coach to guide you through various analysis or decision trees within their products directly unless they are directly bundled with a managed detection and response offering, which is a fancy way of saying that they will provide talented staff to help you with your EDR deployment.
AI and ML are overhyped and overused marketing terms that unfortunately do not have any standardized connotations regarding actual capabilities within EDR solutions. As a result, each vendor claim must be thoroughly vetted to ensure that the organization’s understanding of the capabilities provided by the solution is in fact realized in the product.
The unfortunate reality is that operating EDR for most organizations is more like my going fishing without my friend and expecting to catch the target fish with zero experience, knowledge or the proper tools: essentially relying on just plain luck. EDR provides very rich and very complex data that requires advanced knowledge, understanding and experience to analyze and understand.
This is why most Type B and Type C organizations — often after several months of frustration — tend to eventually reconsider their EDR deployment as an incident-response-focused solution rather than as a platform by which they are guided in their efforts to conduct active threat hunting, detection and response, because they lack those capabilities.
Using an EDR solution as a post-event endpoint data analysis tool is the way the majority of organizations end up using their EDR deployment. However, this is not usually what organizations had in mind when they originally purchased their EDR solutions.
Cloud or On-Premises
As noted previously, EDR solutions can store all or only the most critical data elements it collects either on an on-premises server, on the endpoint itself, in the cloud or as a hybrid of them depending on the vendor solution. The typical concern over storing data in the cloud relates to the disclosure of sensitive data about the day-to-day operations of endpoint software to a third-party outside the organization. While most organizations have embraced cloud-based solutions for many of their IT and security workload needs, some types of clients in specific verticals still prefer to maintain their data on-premises or within specific geographies when using the cloud.
Most vendors cannot accommodate such specific geographic requirements as hosting both data collection and analysis outside the U.S. This can impact data compliance requirement within regions. But the main benefits of cloud storage include lowered complexity in deploying solutions, elimination of on-premises server hardware/software and maintenance, ease of scaling to larger or smaller workloads and access to data even when an endpoint is off or is compromised. This comes at a cost.
Cloud storage requires that organizations decide on their retention periods upfront. Retention periods can be from a few days all the way up to six months. The longer the retention period, the more visibility into past events and also typically the higher the cost for storing data. The upload of the endpoint data to the cloud can have an impact on outbound data throughput. While some solutions offer compressed data streams or a form of load balancing of data upload over longer periods of time, large environments with restricted networks or chokepoints can experience bursting issues.
Cloud-based solutions can also pose challenges in the integration of security and operational data from such other existing solutions as directory and inventory services, network devices, perimeter solutions and SIEMs as well as in creating workflows with ticketing services, update and patching. They may require opening additional connections and ports on the perimeter to support uni- or bidirectional communications.
EDR solutions provide visibility into how an event occurred and, as a result, can tell an endpoint’s overall story. These findings can be used to help determine the overall condition of the endpoint, the potential root cause and also if other endpoints within the environment exhibit similar symptoms. A remediation can be put into action using EDR and other solutions. This part is a good side of EDR.
Getting to the Root of Problems
In a typical incident-response-focused deployment, this analysis, or creating the narrative of the story line, is conducted at some period of time after a situation has taken place and may have already spread. The trigger of the investigation is often when a user reports experiencing an issue with the system or perhaps the operations team notices a degradation of service. In this manner, EDR is used to review the events leading up to the issue and assisting in determining the root cause.
EDR does speed up this investigative process, but there is still a high level of skill involved in performing the investigation. Given enough time, even a poorly staffed EDR solution can successfully search the collected endpoint data and resolve some issues because it is limited to the investigation of a clearly identified target. While this approach resolves issues and does provide value, it rarely elevates an organization’s overall security posture, as it is a very reactive and inconsistent approach to security. It also does not provide for the proactive detection and containment of threats in real time, which means an organization will remain vulnerable to evolving threats.
Most EDR solutions provide very limited note taking within events, workflow tracking, ticketing (internal or external) or even basic role-based access control (RBAC) to assign specific administrative and oversight entitlements to EDR operations staff or a managed service provider. This lack of capabilities results in a poor experience when investigating events that require multiple analysts to resolve, such as after-hours investigations, leveraging a managed service or third-party incident response provider or when there is a need to create an action that is outside the EDR solution itself, such as when an update or patch is required on an endpoint.
Third-party integration, when available, is conducted through APIs and typically requires knowledgeable staff to code the integration or a consulting engagement with a third party to build the component. Report generation is usually focused on the technical aspects of incidents that are difficult to communicate to other stakeholders within the organization, such as line of business leaders and senior management.
EDR solutions rarely incorporate such asset critical data as “this system belongs to the CEO or has PCI data” or activity data sourced from other solutions in the organization, such as active directory information, network and firewall logs and other data sources to help prioritize events. EDR operators often have to connect to multiple consoles to pull this asset and any operational and risk-related data and have to use external systems to keep track of their investigations. And although user and entity behavior analytics (UEBA) have become integrated with many security solutions, EDR has yet to leverage this innovative and potentially valuable source of data analysis.
Do You Patch?
Type B and Type C organizations often struggle with system management, patching and updating. This results in environments that have limited protections even against well-known vulnerabilities and threats.
Organizations deploying EDR solutions in such environments can expect to experience significantly increased strains on their operations staff and systems responsible for endpoint management because many of the resolutions to issues identified by EDR are to remove malicious software, patch or update an application or service, or perhaps even reimage an entire systems in situations where no other option is possible, which can result in data loss if the system was not backed up.
Using EDR to catch basic threats that should be blocked by baseline security hygiene measures is the wrong use of EDR. Doing so will ultimately not result in a better security posture for the organization.
Can You Staff?
Many Type B organizations struggle with finding operational budgets to adequately staff an EDR deployment and have difficulty in finding qualified individuals with the depth of knowledge and experience required to operate an EDR solution even on a basic level. While organizations are typically capable of finding perimeter security or network security staff at reasonable market rates, the skills required to do perimeter or network analysis tasks are not easily transferable to endpoints.
Perimeter and network event data differ greatly from endpoint software operations collected by an EDR solution and, as a result, perimeter or network staff require significant training to become proficient in understanding, analyzing and remediating endpoint issues. Endpoint experts with experience with EDR deployments remain rare commodities.
Augmenting Your Staff
Managed security services (MSS) have been part of the security outsourcing landscape for many years, taking care of the day-to-day operations of IT and IT security solutions within their client organizations. A new breed of MSS that offer managed EDR has evolved over the past few years to address skills and staffing shortages in this market. These solution providers often offer one or more tiers of services with different SLAs and capabilities.
One example is that of a very high-level and low-touch model, where the role of the service provider is to act more like a backup or supplement to an already-staffed EDR operations team. In this capacity, they do not perform day-to-day activities but rather offer additional oversight and reporting and can complement the existing client’s team during incidents. This form of managed EDR is typically inexpensive and includes retainer fees when additional assistance is required by the client, such as during the response to an incident.
Another example, at the other extreme, is that of a low-level very high-touch model where the solution provider, from a remote office, actively investigates security threats using data collected by the EDR and other security solutions and programmatically contains or mitigates threats using the elements that make up the security technology stack in the client’s environment. In this capacity, the solution provider is an integrated extension of a client organization’s existing capabilities. This form of managed EDR is typically significantly more costly and can run many times the cost of the EDR solution itself, depending on the capabilities required.
Managed EDR solutions have become more widely available over the past 18 months, with some EDR vendors providing their own capabilities themselves or via their reseller or system integrator network. However, the quality and availability of the detective, investigative and remediative services vary dramatically between vendors and regions.
Vendor Lock-In and Vendor Risk
Over time, EDR solutions become intertwined with security and operations teams and it becomes difficult to switch out to another vendor, especially when a lot of customized scripts for responses and workflow have been created due to the amount of work required to re-create them. While this isn’t necessarily bad, there are currently too many vendors in this market and many will not survive long term. Also, there are limited paths to exit for small vendors because all of the existing incumbent EPP vendors have created their own EDR solutions, which is traditionally an exit path for small vendors.
This means that vendors who have traction currently either have IPO ambitions or are opting to go for additional series of venture-capital-backed funding to fuel growth. Vendors who have not secured market share or a niche of client deployments are at risk. Clients using these vendors should consider establishing plans in the event that their vendor disappears.
EDR Does Not Mean Protection Is Improved
Organizations need to consider all of the factors highlighted in this research when contemplating an EDR solution to ensure that their EDR deployments meet their operational and security ambitions. Deploying an EDR solution in and of itself does not eliminate the need to deploy other security solutions, nor does it imply that security will improve without significant effort or cost.