This article was published in the Communications of the ACM and in the ACM Queue issue on static analysis.

Correctly harnessing static-analysis techniques in the design of developer tools relies on understanding human intuition. Research advances in the field continue to open up possibilities about the type of information that can be derived about a program’s runtime behavior. Distilling this information into a valuable set of insights for developers depends on asking not only what is computationally feasible, but also what is intuitive and useful in the context of solving complex problems.

Several outcomes are made possible through static analysis, including but not limited to the detection of anomalies, security vulnerabilities, dead code, and other possible optimizations. Understanding how such outputs can be translated into an effective interface—one that enhances the experience of software engineering, its ergonomics, and usability—directly impacts the successful application of static-analysis research in future software development.

The tools and processes used to create software can be radically transformed by applying several techniques that fall under the broad umbrella of static program analysis. Considering the net impact software has on our world, improving developer tools can accelerate the pace of human progress, but only if done based on a thoughtful understanding of human psychology.

Given the aggregate economic, social, political, and environmental footprint of software tooling, abstractions, and language design, the opportunity cost of advancing developer-facing static-analysis applications without being grounded in human-centered methods is too high to ignore. Developer tools facilitate human decision making. Creating effective tools is predicated on having a lucid idea of users and their needs, challenges, and blind spots.

While user behavior is typically not central to advancing static analysis, applications developed without using human-centered methods run the risk of reproducing, if not exacerbating, the limitations of current tools. Increased clarity enhances the technical rigor of code, leading to focus being shifted more toward the ideas underlying the code and the speed with which these ideas are circulated. Reducing the cognitive load imposed by tedious tools or inefficient workflows frees humans to engage in higher-order thinking and reasoning. Therefore, advances in the applications of static analysis should be aimed at supporting human decision making.

Software Engineering Has Growing Complexity

Programming languages have evolved tremendously since their inception in the early 1950s and, alongside their development, an entire discipline dedicated to the creation of software has emerged. Software engineering continues to mature, not just as a practice, but also in terms of its tools, processes, state of the art, professionalism, economics, and culture. At this point, it is apparent just how complex modern software engineering has become. This complexity is the result of many factors and, more broadly, a trend in the industry that favors aggressive release cycles over a thoughtful focus on anticipating and preventing consequences to end users.

In the context of building developer-facing tools, where end users are software engineers, the rate of new tools and technologies entering the ecosystem can be high. Such tools often introduce specific micro-optimizations to workflows in exchange for higher overall complexity. The explosion of services, off-the-shelf APIs, inconsistent standards, and language boundaries that developers must work across daily has made modern development workflows untenable.

Ironically, the growing number of tools and services that attempt to assist developers often contribute to more noise, clutter, moving parts, and complexity. Worse yet, the complexity is not intellectually rich. Substantial coordination between tools is required to perform simple operations. This busywork draws from a finite pool of cognitive resources and has been shown to result in greater decision fatigue when confronted with problems of greater substance.28

The risk of complexity extends far beyond the impact it has on working conditions for software engineers. Safety-critical systems designed to operate at scale can have unintended consequences caused by code complexity. A lawsuit filed against Toyota in 2003 exposed how an unmaintainable, untestable code base resulted in failsafe defects that caused loss of throttle control during vehicle operation, resulting in unintended acceleration.27 This outcome arose because of several anti-patterns in the code, such as the use of 10,000 global variables, and a single point of failure. System complexity made the code not only difficult to debug, but also impossible to regulate. The National Highway and Traffic Safety Association was unable to investigate the system because of its complexity.

Litigation is evolving, however. More recently a court ruled that a defendant has the right to examine the source code used to produce evidence against the defendant.5 As legal ramifications grow, there is greater pressure for companies to be responsible stewards of their code.

Developer tools can be enhanced using static-analysis applications to manage this complexity. Here are some questions to be answered: Does a program have security vulnerabilities? What is the program’s worst-case execution time, and can it be further optimized? Are there any common runtime errors, such as division by zero or numeric overflow?

Static analysis provides answers to questions about the behavior of computer programs at runtime, for all possible inputs, without running them.

Human-Centered Design and Understanding Program Behavior

More information about code is useful, but what humans need is an interface that matches their intuition. Consumer-facing applications have long embraced a practice of catering as closely as possible to native human intuition. To build intuitive applications, developers commonly draw on research and best practices in the fields of user experience (UX), usability, user-interface (UI) design, human-computer interaction (HCI), and cognitive science. Although these disciplines have evolved considerably over the past few decades, the dearth of design thinking is noticeable in the landscape of developer-facing tools. This is often a result of deprioritized investment in understanding how humans think, which is arguably the most important prerequisite to leveraging static analysis correctly.

Each of these disciplines informs design differently. User experience defines the overall perceptions, emotions, and judgments that result from an individual’s interaction with a system.23 Such perceptions are influenced by many aspects, such as how delightful it is, as well as its usability: how easily a system affords achievement of its intended purpose. Usability is defined as a measure of effectiveness, efficiency, and satisfaction.13 It is typically evaluated through a set of heuristics, such as Jakob Nielsen’s 10 usability heuristics for interface design.22 User interface refers to anything the user may interact with, such as the visual elements, keyboard buttons, sounds, and lights. To that end, UX includes, but is not limited to, usability and UI. Human-computer interaction is an academic research domain that applies empirical study to better understand UX by incorporating psychology, cognitive sciences, and human factors.14 Understanding the differences between these fields ensures relevant techniques from each discipline are applied appropriately.

The intersection of HCI and static analysis is important because software engineering requires an understanding of program behavior. As the reach of software grows, so do standards for code to be fault-tolerant, reliable, and high quality. What high-quality software even means, however, remains a hotly debated topic. Given the breadth of languages, problem domains, programs, and varying degrees of complexity associated with each, the definition of quality can vary greatly, and so can the tools and processes thought best to achieve it. A myriad of “best practices” floods the software engineering canon to keep pace with the rapid emergence of new technologies and languages that gain adoption and often contradict one another.

Given the fluidity of what is right and the uncertainty of what can go wrong, programmers must exercise ongoing vigilance to ensure immediate priorities are correctly identified, and to determine the best way to translate those priorities into an implementation that balances factors underlying quality, such as performance, readability, size, cost, maintainability, security, and other competing trade-offs. To do this, programmers must understand the implications of each decision, which requires an understanding of program behavior. Understanding program behavior, however, continues to be difficult for a variety of reasons.

Context Switching Is an Obstacle to Understanding Program Behavior

One obstacle preventing developers from understanding program behavior is the large amount of context switching required to be productive. Software engineers spend more time building mental models and checking assumptions than writing code, but they often have inadequate tooling to support them in this process. The information needed to decide can be difficult to identify, let alone access. When the programmer does know exactly what information is relevant, it is often scattered across a variety of disconnected sources that must be consulted and fused together in a way that best maps to the problem.

Context switching is known to increase error rates and slow down productivity, as it can create dangerous interruptions in a coherent flow of thoughts. Task switching has been shown to reduce performance on a primary task, resulting in higher error rates, greater time taken to complete the task, or both. Divided attention has a damaging effect on encoding memory, as well as on retrieval.29 Not only does modern software engineering require significant context switching to achieve anything meaningful, but this information-finding journey is often littered with distractions and fruitless rabbit holes into which even those most seasoned in their craft can fall.

To consider the impact of context switching, the accompanying table shows the amalgamation of the digital and physical tools used, sometimes simultaneously, to facilitate software development.

image

At the end of this information-finding journey, even if the programmer is somehow successful at maintaining a coherent train of thought while jumping across several contexts, the information retrieved may itself be incorrect or insufficient. This process repeats until enough information is available to build a reasonable understanding of what is going on and what must be done next. Lacking adequate tools, however, the programmer may use this information to make decisions that are improperly evaluated. Complex code paths, unreliable validation systems, flaky tests, and tedious processes leave much to be desired when the programmer just wants to visualize control flow or assess impact across dependencies.

Poor interfaces supporting the creation of a mental model also widen the gap between the source code produced and the labor required to generate that result. Visibility of state and control flow are additionally reduced over problems that span platforms and languages, as each language is supported by its own set of tools. While Language Server Protocol (LSP) is one attempt to provide a lingua franca for communicating between language compilers, few editors support integrations that unlock LSP’s cross-platform capabilities, because of the required overhead.

This problem of scattered or insufficient information is exacerbated in the absence of documentation. Documenting everything is often too onerous, so it becomes an activity secondary to writing code. In this situation, justifications for decisions remain unclear. Newcomers to a codebase amass information less efficiently than they could if learning was better facilitated by existing tools.

Context switching has increased, especially as high-frequency collaboration has become normalized. While asynchronous communication tools are designed to help connect workers, they are often used synchronously for real-time conversations. This has led to workplace cultures where visible activity is often conflated with productivity. A constant stream of notifications not only creates an artificial sense of urgency but is damaging to the deep focus demanded by programming. Research consistently shows that individuals experienced significantly more stress, anxiety, and irritability when tasks were interrupted.21 Studies also show that after being interrupted, workers don’t resume their primary task 41% of the time.24 On average, trying to resume a primary task after an interruption took 25 minutes longer.19 Correctness and security guarantees naturally suffer when the cognitive load of context switching is high.

Well-Designed Abstractions Improve HCI

The goal of any well-designed system, not just software, is to ensure relevant knowledge is available at the appropriate time and place. Throughout its history, computer science has addressed this objective through the design and implementation of abstractions. Abstractions help distribute vast amounts of information into subsystems, where each constituent subsystem has a dedicated purpose. Layers of abstractions fill the chasm between human and machine, with each layer translating conceptual ideas into ones and zeroes.

Well-designed abstractions and languages can help elucidate program behavior. Abstractions provide focus at the layer most relevant to the problem domain, hiding information that is assumed to be unimportant. Unfortunately, identifying what is important is just as controversial as the topics of “quality” and “best practices.” Because of this, the design of abstractions is often more an art than a science. Creating the right boundaries and expressing appropriate information at each layer is critical to providing the most useful information that precisely befits the context. Abstractions can also backfire. They can be too opaque, and to raise the signal of what is most important, they end up hiding too much.

“The purpose of abstraction is not to be vague, but to create a new semantic level in which one can be absolutely precise.”— Edsger W. Dijkstra, The Humble Programmer, 1972 ACM Turing Lecture

The tensions underlying abstraction design are like those associated with using static analysis to build developer tools. It is important to surface an appropriate level of detail. While certain contexts demand more granular precision, greater detail is not always conducive to navigating and inspecting vast amounts of code more easily. On one hand, users may be inundated with overwhelming metrics that can paralyze decision-making or guide attention toward less meaningful data that leads to yak shaving and micro-optimizations. On the other hand, there is the risk of concealing important information or introducing automation that yields undesirable outcomes. The tension between detail and generality is unique to each problem and must be negotiated as such.

Why Is HCI Deprioritized?

While it is difficult to pinpoint why exactly human-centered methods are given less importance in the design of developer-facing tools, the following speculations explore cultural aspects that may play a role.

Skewing toward compilers and programming languages. Academics focusing on static analysis tend to be compiler researchers and programming language theorists whose interests and expertise slant toward studying language capabilities, expressive algebras, interpreters, and algorithmic design. Because of their specialization, discussion of usability and HCI is often limited. Furthermore, ideas on static analysis have existed in computer science literature for a lot longer than the time researchers have had to explore their limits in modern systems. Therefore, there is a disconnect between research focusing on what insights can be extracted from code and the UX challenges such insights could mitigate in the wild.

While an increasing amount of active research is concerned with using HCI to help people better understand code, this research is still mostly disconnected from static-analysis applications.10 Research in task analysis, while capable of characterizing procedural knowledge to measure the efficacy of various interfaces, also remains disconnected from the implementation of static-analysis techniques in developer tools.

Engineers who are less likely to be design savvy. Unlike front-end engineers who may have more expertise in design, or product engineers who are skilled in building a business case justifying their work, engineers concerned with bringing static-analysis tools to market typically come from corners of academia and tend to be hired into roles more narrowly focused on algorithm design, scale, and infrastructure. Given their expertise, they may not prioritize a design process that draws on empirical user research or usability to validate their assumptions, leading to less intuitive implementations.

Because of their specialization, they also prioritize technical correctness over communicating the business value proposition to leaders and stakeholders, communication that could lead to more organizational support and design resources. A functioning body of code in the form of a proof of concept is pointless unless its value to users and the business is clearly communicated. Even if significant engineering investment is poured into a prototype, skepticism around the usefulness of such work grows, and such projects may become orphaned by a lack of external support necessary to carry them into production. Since most static-analysis techniques are also costly and difficult to scale, the justification for this work dwindles.

Lack of consistency and standards. Modern software engineering offers unique debugging challenges that stem from a lack of consistency and standards. While open source software was historically blamed for the lack of standardization, proprietary software contributes just as much to inconsistencies. This is because standards emerge organically when they are favored by market forces, not through systematic development of protocols by standards committees.2