This paper presents data that describe the effects on software development performance from both the production methods of software development and the social processes of how software developers work together. The premise behind this paper is that the production methods of software development--such as methodologies, techniques, and tools--become available, then disappear at anever-increasing rate while the social processes of the people developing software change more slowly. For example, there has been a stream of software development methodologies (e.g., object-oriented, clean-room, and rapid-prototyping methodologies), techniques (i.e., participatory design, requirements engineering), and software development tools (such as computer-aided software engineering [CASE] tools) that reflect a production focus on software development. At the same time, our knowledge of how software developers work together is growing more slowly (see, e.g., References 1-3).

Since software development is, at the least, partly a social process means that understanding how people work together to build software is critical, since the importance of software in our society is matched by the difficulty encountered in its development. For example, about 40 percent of U.S. corporate capital expenditures are directed toward software.⁴ The U.S. government is also committing billions of dollars to supporting research in computer hardware and software.⁵ Large-scale failures of software underscore the difficulty in its development.⁶ For example, the U.S. Internal Revenue Service (IRS) has spent billions of dollars to replace the present tax system with little to show as a result of trying, and no replacement is yet available.⁷ Further, corporate-level failures of information systems are routinely reported in the popular press (see, e.g., References 8-10).

In this paper we draw on data collected from 40 software development teams at one U.S. site of a large, global, software and hardware manufacturing company. (We have named the company "Compuco" for the purpose of confidentiality.) These data are used to explore the relationships of both the production methods and social processes to software development performance. Specifically, in this paper we address two questions:

1. What are the contributions of production methods to software development performance?
2. What are the contributions of the social processes of software developers working together to software development performance?

To address these two questions the paper continues in four parts. In the first part we highlight issues of developing software and how they can be addressed. In the second and third parts we highlight our data collection and present our analysis and findings. We then conclude with implications and suggestions for software developers and for software development researchers.

A multiperspective view on developing software

In this part we outline general issues with software development and discuss particular issues at the research site used in this study. This discussion is premised on the observation that the two most common responses to the current difficulties with creating software are for software development organizations to: (1) establish and follow more formalized production methods for building software products and (2) use teams of software development specialists and the potential positive synergy that arises from their interactions. Typically, software development teams are brought together to make new, or enhance existing products. A team is two or more software developers who are engaged in building a defined product to be delivered within a certain time frame. A team relies on the collective skills of its members because of the scope of the effort, the inherent complexity of the effort, and the number of tasks needed to develop modern software that normally exceeds the ability of any one developer.

The team's members rely on methodologies, techniques, and tools to support software production. A methodology represents the set of tasks and their ordering that defines the processes of production. For example, rapid application development (RAD),^11,12 clean-room,¹³ and object-oriented¹⁴ and formal¹⁵ methodologies are currently popular. Techniques are sets of actions taken to complete a particular task. Examples include Joint Application Development (JAD)¹⁶ and structured walkthroughs.¹⁷ Typically, a methodology draws on many techniques and a tool provides automation or structure to a technique.¹⁸ This is the premise behind CASE tools: they are not a methodology by themselves, but they support the use of techniques and can enable methodologies.^19,20

This production perspective highlights the present emphasis for much of the current research on software development. Production methods describe how individuals should work and the focus is on the project. The team's goal is to follow the proper method and to use the proper techniques and tools in support of that project.²¹ A substantial body of writing focuses on the role of methodologies, techniques, and tools (see, e.g., References 22, 23). The production perspective seeks to underscore the repetitive, predictable, and manufacturing-like aspects of software development (see, e.g., References 24-26). Reducing the variations in the development process, making the development process more routine, developing specific guidelines for how to best address common tasks, and making tools available to support these efforts are seen as central to the maturation of software development as a profession and as a more certain contributor to society.^27-29

A second perspective on software development is as a social process.^30-33 This perspective focuses on how software developers work together to produce software. The trend to using teams to produce software magnifies the issues of working together to build software.³⁴ For example, the social issues of working together to build software include coordination and communication breakdowns^1,2,34,35 and the positive and negative effects of such social processes as intragroup conflict management.^36-38 Recently, the culture of software development in the U.S. has been seen as a unifying force among developers, providing them a commonality of focus that enables software development.^39,40 This implies that another issue for software development teams is the degree of shared norms that tie software developers together.

Social processes of software development encompass both informal communication and intragroup coordination activities such as discussing how activities will be performed, finding and taking the time to talk with other team members, and the sharing of ideas and information.^41,42 Social processes also include resolving the conflicts that arise in the course of working together, and encompass issues such as reducing the level of irritation and frustration among the team, getting along well with one another, and resolving differences in a timely manner.^43,44 There are norms of loyalty and supportiveness that are part of the growing culture of software developers as they learn how to work together. This includes caring for other members in the group (the occurrence of helping behaviors between team members), being inspired by being a member, and seeing the team as distinct from the larger organization.

A third way to view software development is from an individual perspective. This perspective focuses on the unique contributions of individuals to the team. Issues from an individual perspective can include, for example, the degree of experience and skill that each developer contributes.³⁸ The individual perspective also encompasses the internal motivations for participating (such as ego gratification or social power accumulation) and other personal motives.¹

In this multiperspective approach to understanding software development, distinguishing between the production, social, and individual levels of a software development team's work is analytically useful, but the actions of the three perspectives occur in a tightly woven, interdependent process.^45,46 For example, including formal coordination mechanisms as a part of the production process, while informal communication and coordination mechanisms are considered part of the social processes, is an analytic distinction. In both formal and informal meetings, team members socialize. They talk, they encourage (or discourage) supportive feelings toward the team, they respond to (or ignore) potential intragroup conflicts, and they do (or do not) communicate and coordinate. Further, the team members jockey for social power, they worry about the effects of embarrassing questions, and they try to impress their peers. Thus, the formal coordination factor reflects an intersection of production, social, and individual processes. However, for this study, we focus on the team-level issues of production and social perspectives, leaving the individual perspective for future work. We return to this issue in our discussion of the findings at the end of the paper.

A fourth perspective on software development is contextual . This perspective suggests that issues such as the competitiveness of the company, the industry in which the company operates, the degree of managerial skill, the level of resources, and other extra-organizational factors affect software development performance.^34,47 For example, these factors contribute to the difficulty in comparing data about software development teams who focus on making custom-designed embedded software for the U.S. Department of Defense (DoD) with teams building DOS and Windows**-based packaged products for the personal computer market. That is, the differences in customers, markets, and goals of the software products are so diverse that comparisons are often difficult.⁴⁷ Because of the potential for contextual issues to overshadow the goals of this study, we focused on collecting data from one organization. This provides some control (albeit imperfect) of the potential effects of organizational and environmental context. It also reduces (but does not eliminate) the potential effects of intradepartmental differences.

The setting: Developing software at Heartland. The data and analysis we discuss in this paper are drawn from a field study at one U.S. software development site (named Heartland for the purpose of confidentiality) of Compuco. Heartland creates subsystem software such as database products and languages. These are sold as commercial packages, often in combinations that can provide for integrated solutions. Packaged software is also known as commercial, shrink-wrapped, and commercial-off-the-shelf software. As a packaged software vendor, Compuco licenses their products for use by others. These products may have thousands, or even millions, of licensees. This implies that development of these products is done for a distant user who is typically not a member of the organization, making extensive user and developer interaction difficult.⁴⁸ Typically, at Heartland developers are involved with one product for a lengthy time, participating in the developmental and production cycle.^40,49 This means that the developers become quite knowledgeable about both the features and the development history of the product. Finally, and unlike most custom development efforts, Heartland's developers have no hand in the products in which their subsystem is embedded.⁴⁷

Packaged software development, here exemplified by Heartland, is an expanding and important aspect of the software industry. While DoD and other large institutions may always demand a certain level of custom-built software, the market for packaged software is both large and growing.^4,39 For example, IBM has been a dominating force in commercial software with products such as IMS* (Information Management System) and DB2* (DATABASE 2*). Microsoft Corporation's rise as a corporate power is built on the sales of its packaged software.^40,50

Three reasons made Heartland a suitable research site. First, they develop products for the commercial market. As noted, this is an important, expanding, and relatively under-studied area of software development. Second, they are large enough to support enough teams to provide adequate data for the study. Finally, the software developers and managers were motivated to participate. For example, like other commercial software developers, Heartland's products face increasing competition. Product time-to-market issues and product quality are often competing pressures. The market life of each commercial software product being developed is also shortening. Even so, long-term maintenance demands increase with each new release. This combination led to difficulties in maintaining existing products and slowed the pace of new product development at the host research site. Concurrently, developers were increasingly unhappy with the way their work was structured and with the workplace pressures they faced. They were working harder, spending more time at work, and seeing fewer results. Senior managers at Heartland were also under tremendous pressure from the Compuco corporate executives to create new products and extend revenue streams on existing lines.

To respond to the intertwined influences of market pressure, product pressure, workplace pressure, and corporate pressure, Heartland's senior development managers responded by refocusing development to be team-based. This was also a move away from the functional and project matrix management structure they had relied on for more than 20 years. As a part of that effort, this research represents a joint effort between the authors and the developers at Heartland to better understand the software development team processes and performance effects at their site. However, management and team structures remained unchanged during the course of this study. The current team structure is based on having a team leader, a technical leader, and a relatively small number of developers to comprise the entire team. Each team's leader reports, in turn, to a product manager.

The production aspects. Heartland's software development organization exemplifies the production perspective of software development. The site follows a rigorous, well-defined software development method that is based on structured analysis and design approaches and has been evolving for nearly 20 years. This methodology is both well-known and heartily supported by senior development managers. There is extensive training in the core software development methodology and base techniques provided to all developers. As part of this method, personnel have developed and integrated a number of automated software development tools (some are locally developed and some are commercial products) into the development methodology. This methodologic rigor is, in part, one of the reasons why Heartland has been lauded for their process quality (having won both several internal-to-the-company and industry-wide quality awards in the past few years). However, even at a single site, where developers have been trained in one set of techniques and share a common set of tools, the way these aspects of production are used varies.

To capture the use of, and variations in, the production aspects of software development at Heartland, we collected data on (1) the level of formal coordination mechanisms use, (2) the level of formal methods use, and (3) the levels of tool use. Formal coordination mechanisms include formal project meetings, formal client meetings, and required documentation and deliverables. Thus, this measure of formal coordination acts as a surrogate of project management. The other two factors represent the use of a software development methodology. For instance, the use of version control procedures and management of code libraries suggests a reliance on a defined software development methodology. High levels of tool usage reflect the use of standard development techniques that allows for automation.

The social processes. The second perspective on software development is as a social process. This focus is on how the software developers work together to produce software. Heartland's development organization also recognizes the importance of the social processes of software development. For example, there are numerous group dynamic, conflict management, and listening-skills seminars, extensive rewards for superior team performance, and both formal and informal acknowledgments that teamwork is critical. Further, development team members meet on a regular basis (typically weekly) and normally reside near each other.

The social processes of software development that we measured include how team members perceive their level of informal communication and coordination, their ability to resolve intragroup conflicts, and the degree of supportiveness and loyalty they felt for the other members of their team. The level of informal communication and coordination means the amount, and value, of communication both with team members and with key people who are not part of the team. The ability to resolve intragroup conflicts includes both surfacing differences and negotiating acceptable shared agreements and compromises. The degree of supportiveness includes how team members feel toward other members of their team, and how they perceive other team members feel toward them.

Software development performance. In order to assess the value of production and social processes, a measure of software development performance is required. This is problematic for at least two reasons. First, the performance of software development teams has been measured in many different ways. Second, most of these measures have significant measurement problems such as limited relationships to business value and questionable validity.^51,52

Our approach is to view software development as an activity intended to produce a product that will affect the behavior of one or more stakeholders. We further constrain our definition of stakeholders to be individuals who are not team members but who can affect design activities and who can be affected by the resulting information systems.^53-55 While the development team members certainly have a stake in the product, our focus is on the external-to-the-team stakeholders. These might be user managers, senior development managers, or senior customers, and they assess the team's performance based on their knowledge of the organization's needs, experience with previous and ongoing software development projects, and their expectations for quality. Thus, we share the view of Seidler⁵³ who finds that perceptual assessments of performance provided by such knowledgeable managers (i.e., stakeholders) have a high level of convergence with other objective measures of performance.

Given these issues, we characterize software development performance as multidimensional and include three attributes: product quality, team efficiency, and team effectiveness. Stakeholders provide the product quality assessment. Both the team effectiveness and efficiency measures are also assessed by stakeholders and combined into an aggregated measure of team performance. The team performance factor is also drawn from the developers on each of the teams. This provides a self-reported measure of team performance in terms of team effectiveness and team efficiency. Thus, three performance factors are measured: stakeholder-rated product quality, stakeholder-rated team performance, and self-reported (by the developers) team performance.

Studying Heartland's software development teams

This part describes the research methods used to assess the effects of the production and social processes of software development teams on software development performance. To do this, we used multiple data collection methods: direct observation of their work as it occurred, quantifiable data on use and performance (drawn from two surveys), and a synthesis of the anecdotes and stories of software development. Beginning in early 1993, we spent 18 months collecting data on Heartland's software developers doing their work, in their native environment. Using interviews and surveys is sensible since we are collecting data on their perceptions about how their teams rely on the various production process and social process factors. The individual responses to the survey are aggregated to the team level for analysis.⁵⁶ Analysis of the survey data is done using multiple regression and correlation techniques.^57-60 This means that each performance factor is assessed for the contribution of both the production and social process factors (represented as the amount of variance explained). Table 1 presents these factors and the question areas to help define the factors. Appendix A presents their zero-item correlations, means, and standard deviations.

Table 1 Factors and scale questions

Factor, Factor Reliability*	Scale Questions (Areas That Help Define the Factor)
Informal Communication and Coordination, .70	The amount of communication with team members
	The amount of communication with key people who are not part of the team
	The amount of coordination with team members
	The amount of coordination with key people who are not part of the team
Supportiveness, .71	The support that team members give to other members of their team
	The loyalty that team members give to the team
	The coverage given to each team member as needed
Conflict Management, .71	The ability to surface differences among the group
	The ability of the group to negotiate acceptable shared agreements
	The ability of the team members to reach compromises
Formal Coordination, .82	The use of scheduled meetings
	The value of scheduled meetings
	The use of project management documents
	The value of project management documents
	The level of information sharing required as part of development methodology
	The value of information sharing required as part of development methodology
Method Use, .88	The use of prescribed methods or patterns
	The value of prescribed methods or patterns
	The use of formal project reviews
	The value of formal project reviews
	The use of required documentation in developing software
	The value of required documentation in developing software
Automated Tool Use, .71	The use of automated development software
	The value of automated development software
	The use of shared code repositories/libraries
	The value of shared code repositories/libraries
	The use of tools for common access to work products when developing software
	The value of tools for common access to work products when developing software
Stakeholder-rated Product Quality, .91	The quality of the system produced by the project team
	The number of defects in the system
	The extent to which the system adds value to our firm
	The extent to which the system adheres to organizational standards
Stakeholder-rated Team Performance, .91	The efficiency of project team operations
	The adherence to schedules during the project
	The amount of work the project team produced
	The ability of the project team to meet the goals of the project
	The extent to which the users' business needs are reflected in the system
	The contribution of the system to the performance of the firm
Self-Reported Team Performance, .95	The efficiency of project team operations
	The adherence to schedules during the project
	The amount of work the project team produced
	The ability of the project team to meet the goals of the project
	The extent to which the users' business needs are reflected in the system
	The contribution of the system to the performance of the firm
*Note: Factor reliability is a measure of the degree to which responses to these questions covary. This is measured using Chronbach's alpha. If the alpha is greater than 0.70, the scale comprising a set of indicators is considered reliable for exploratory work. Several scales are much higher, suggesting that the use of previously validated scales improves the value of these factors.

With the support of the team members and their managers (including senior managers), we gathered data from 45 software development teams. We formally interviewed ⁵⁶ people. Many of these people spoke with us again, often several times, during our observation period. We also met informally, or in organized response sessions (such as debriefings of teams), with another 153 developers and managers from the 45 teams following our survey data collection. The topics of these informal meetings varied, but the issues and questions were driven by the ongoing analysis of both survey data and previous interviews. Most of the informal sessions were not taped. Instead, these were documented with post hoc field notes.⁶¹

Survey-based data were gathered from 40 teams (and 128 respondents) at the site. This represents more than 30 percent of the project teams and 10 percent of the developers at the site. Five of the 45 teams were not surveyed: three declined and two were performing software support and not development. The surveys were developed from existing scales (see Appendix B) and pilot-tested at the site.⁶² We used these surveys to gather data about how the team members interacted with one another, how they produced software, self-reported performance, and demographic data about the teams and team members. Survey questions were posed at the team level, as this is the level of analysis. This is also the level of measurement and the level of theory.⁶³

We also collected data about performance from stakeholders for each of these teams. This was done with a structured survey in a phone-based interview employing the scale developed and used by Henderson and Lee⁵⁵ in their study of software developers. In the debriefing (offered to all teams⁶⁴) that followed the major data collection and analysis phases, we returned to the site and asked additional questions to reflect on and amplify the findings from the initial data analysis.

Some additional data on product quality were gathered by the site and provided to us. This archival data on product quality is at the product level. This cannot be directly linked to the individual modules. This means that product quality data cannot be associated with the 40 teams in this analysis. However, product quality data can be compared at the product level and that is discussed in the final part of this paper.

Interview data are used to amplify findings from the statistical analysis. Self-reported team performance data are drawn from the same survey that is used to collect the predictors of that performance. This is a form of method bias that typically results in over-exaggerated relationships.⁶⁰ Thus, the regressions based on self-reported team performance may be more useful as an indicator of the existence of a relationship than as a measure of the strength of that relationship. Stakeholder data are used to develop the team performance and product quality factors. These data are drawn from separate surveys and are not as susceptible to this method bias.

As we noted at the beginning of this paper, the development teams in this study build software for commercial sale. These commercial software products are quite large (typically more than one million lines of code) and some have been in the market for three decades. The teams involved in this study contribute to one of four products. Each team is typically charged with one module--a distinct part of the overall product--that must be integrated together and work seamlessly in the final assembled product. What this means is that, while the modules may be distinct segments and identifiable at some level, as a product they are tightly integrated into one system. During development, this means that teams are often highly dependent on one another and these dependencies are not sequential. That is, two modules may pass data back and forth when used in the product. This means the two teams must work closely together as both customer and producer. Team size ranges from 4 to 14 people, with the average being 9 members. On average, the team members have been together for nearly two years and they change team leadership every year. Table 2 includes more information about these developers.

Table 2 Team and individual characteristics (without age and gender data)

Characteristics	Years (Mean)
Individual professional experience	12.0
Individual as an employee	4.2
Individual management experience*	4.5
Individual in present position	2.3
Individual as a team member	1.5
Team with 75% of present team membership	1.9
Team with same team leader	1.0
	Number (Mean)
Individual project experience	9.8
Team size	9.0
	Education (%)
College degree	89
Master's degree/Ph.D.	34
*Note: Reflects responses from 27 percent of sample, representing those having previous management experience but who are not presently managing.

The level of experience and product knowledge are viewed as critical measures of a team's competence. For example, developers on the 40 teams have a mean of more than nine years in software development, more than four years with the company, and nearly two years with their present teams. The individual respondents have a mean of 9.8 project's worth of experience (where a project is a previous release of a product). Of the respondents, 89 percent have a college degree; 34 percent also have a master's degree or doctorate. Many of the original developers have stayed with these products since inception and they provide a wealth of product knowledge and perspective to the teams. Further, an espoused belief at the site is that it takes several years for junior developers to become fully aware of product intricacies.

Effects of production and social processes

To address the two questions posed at the beginning of the paper, this part presents the analysis and findings in four subsections: The first describes the effects of production factors on software development performance. The second presents the added effects of the social process factors on software development performance. The third describes the moderating influence of production factors such as methodology and tool use. The fourth presents issues with the analysis.

The effects of production processes.Tables 3 and 4 present analyses that assess the contributions to software development performance due to production methods. Table 3 presents the contributions of the two methodology factors to each of the three performance factors. This analysis shows that both the formal method and tool use factors provide no significant contribution to any of the three performance factors. That is, variations in the use of formal methods and the use of automated development tools provide no explanation of the variance in stakeholder-rated product quality, stakeholder-rated team performance, or self-reported team performance.

Table 4 presents the contribution of the two methodology factors and the formal coordination factor to each of the three performance factors. The formal coordination factor accounts for a significant portion of the variance in both the stakeholder-reported and self-reported team performance. However, none of these production factors provides any significant explanation of the variation in stakeholder-rated product quality.

This analysis indicates at least two findings. First, the methodology factors provide little direct explanatory value. Second, the use of formal coordination mechanisms helps to explain the most variance in two of the three performance factors. What this also suggests is that these production factors may have an indirect, or moderating, effect on performance.^65,59 This means that the use of formal methods or automated software development tools may influence the relationship between the social process factors and the performance factors. We discuss this influence in the next several subsections.

The effects of production and social processes. To assess the contributions to software development performance due to the social processes of software developers working together, Table 5 presents results of the analysis combining both the production and social process factors in assessing their contribution to each of the three performance factors. Data show that higher levels of formal coordination, higher levels of informal communication, more intragroup conflict management, and higher levels of supportiveness explain 69 percent of the variance in self-reported team performance. The data in Table 5 also show that higher levels of formal coordination, higher levels of informal communication, and more conflict management account for 23 percent of the variance in stakeholder-rated team performance. These factors also account for nearly 20 percent of the variance in the quality of the team's software product.

The moderating influence of methodology and tool use. Table 6 presents additional analysis into the role of production factors in explaining variations in the three performance factors used. This was suggested by the findings reported in Table 4. To conduct this analysis, we split the sample of teams into two subsets. The first subset rated lower than the aggregated mean of the use of formal methods and automated software development tools. The other subset rated higher than the aggregated mean of both factors. Regressions for each of the three performance factors using the remaining four factors for both subsets produced no significant models of the teams with high formal method and tool use.

Two significant models were found for teams in the low formal method and tool use subset. That is, for the 20 teams in the subsample that had lower levels of methodology use, the levels of formal and informal coordination and conflict management are significant contributors, explaining 25 percent of the variance in product quality. All four factors are significant contributors and account for 62 percent of the variance in self-reported team performance. This implies that, for the teams who make minimal use of both formal methods and automated software development tools, low levels of use moderate the relationship between the three social process factors (plus formal coordination) and two of the three performance factors used in this analysis. Further, the performance of teams who used both formal methods and automated software development tools the least, are not significantly different from the high use teams.

Similar split-sample analysis using the social process factors and formal coordination produced no significant models. This suggests that these social process factors and formal coordination do not have any moderating influence on the use of formal methods and automated software development tools. Since two of the performance factors are based on the perceptions of stakeholders, it is important to note that these stakeholders were unlikely to know much about the social processes of the teams. However, they would be aware of any formal coordination mechanisms that required the team--or its leaders--to contact the stakeholder (via either meetings or deliverables).

Analysis issues. There are at least two issues with this analysis that merit additional discussion. The first issue regards the use of multiple regression as the basic analysis technique. That is, analysis based on multiple regression is susceptible to multicollinearity among the independent variables.⁵⁷ Multicollinearity is the tendency for supposedly independent factors to be related to each other, possibly invalidating certain assumptions behind using this statistical technique. Potential multicollinearity issues are minimized if the tolerance between any two factors remains below 0.700.60 In this analysis no tolerance exceeded this upper limit.

A second issue with this analysis regards the sources of data. Since all data are collected at one software development site, there may be some concern with the representative distribution of the data. Appendix A provides the means and standard deviations for all the factors used in this study. The normal distributions (indicated by the standard deviations) suggest that the use of one site for data collection did not constrain the range of responses used to construct the factors used in this analysis.

The roles of production factors in supporting the social processes

Data indicate that the level of both formal methods and automated tool use do not aid in predicting software development performance. However, the role of formal coordination, and several of the social process factors, can account for some of the variation in the software development performance we measured. For example, the combination of production and social process factors accounts for 25 percent of the variance between the teams with the highest and lowest levels of stakeholder-rated product quality.

In this section we address four issues suggested by this analysis. The first two issues are (1) rethinking the relationship between the production and social factors, and (2) potential limitations due to the interesting sample demographics. The last two issues go beyond the data from the current study to present discussions of (3) the potential effects to software development practice suggested by these findings, and (4) the question of the unexplained variance in stakeholder-rated team performance and product quality.

Rethinking the relationship between production and social process factors. Data from this study show that of the three production factors measured, the two that focus on assessing the methodologic aspects--level of method and tool use--are not significant predictors of software development performance. Furthermore, the analysis of the moderating effects of both method and tool use in this analysis suggests that higher levels of method and tool use are even less valuable than lower levels in helping to predict software development performance.

One reason this site was selected for the research is that they have a well-established software development methodology. This is accompanied by extensive training and the provision of automated tools to support steps of the software development methodology. Data presented in Figure 1 indicate that many of the method aspects are not used extensively and are seen as having marginal value. Further, while tool use varies, it provides no predictive value for performance. Other data, provided by the site on quality at the product level, provides little additional information. For example, the distribution of the teams, relative to all three measures of software team performance, does not differ across the five products to which these 40 teams contribute.

Figure 1

The findings suggest that one explanation for the importance of the formal coordination factor is it reflects actions that bring the production aspects of software development into a social context. That is, this factor represents the use of meetings, documents, and required interactions that bring the developers together. In this way, the formal coordination aspects of a methodology are valuable since they provide for an occasion to socialize. Observations of meetings and the anecdotes of participants suggest that it is the process of socializing at these meetings that provides value, not the occurrence of the meetings or the requirement of the methodology being followed to have those meetings occur. This finding also implies that a value of project management, for which this formal coordination factor serves as a surrogate, may be that most project management techniques require team members to interact.

If team-level software development methodologies serve primarily as a means to require socialization, it may be that these methods are important primarily because they help to teach team members when, and what, to discuss. In that sense, teaching software development methods and tool usage may have the unintended effect of guiding "valued" social processes. For example, Vessey and Sravanapudi⁶⁶ find that CASE tools serve as effective collaboration devices. This is one way to interpret the value of lower levels of method and tool use: they reflect an effective source of guidance without dogmatic (and socially counter-productive) effects that may arise from over-adherence to formal methods or tool use.

The explanatory value of both social processes and the role of formal coordination highlight the importance of nonproduction aspects of the development process. While this has been widely recognized, this analysis underscores the extent to which the factors influencing software development performance are still poorly understood.^6,27 Further, this finding suggests that allowing for "people factors" in the discussion of new software methodologies (see, e.g., References 49, 67) may be placing the emphasis on the wrong aspect: "putting the cart before the horse." Perhaps software development methods should be developed to explicitly encourage socialization among developers--a behavior-centered process. This is the gist of DeMarco's⁶⁸ point that most software developers, when asked to work with another on a project, never ask, "in what language?" They ask, "with whom?" Refocusing software development methods from production (or engineering)-centered to socially (or behaviorally)-centered methods seems appropriate given that the data show that software development production aspects are both secondary to social aspects of software development and most valuable if used sparingly.⁶⁹

This emphasis reversal, where production methods are designed to support the social processes of software development teams, is well-represented in the discussions of software development at Microsoft.^3,40,50,70 They contend that Microsoft's development processes are based on individual interaction and flexible processes based on fixed team meetings. Work by Zachary^3,40 suggests that the social interactions of dominant team members shape the development processes more than do production methods. Sawyer, Farber, and Spillers³³ find that, when allowed, software teams will adapt tool use to fit their own, emergent needs. Curtis⁶⁹ asserts that any software development effort that does not explicitly account for how people work together is likely to be unsuccessful. Data from this study support these assertions and findings.

In discussing the limited effect of methods and tool use on software development team performance, it is imperative to realize that our focus is at the team-level uses of tools and methods. Thus, the steady evolution and improvement of programming languages, compilers, debuggers, and other elements of individual-level software development do not contradict these findings. In fact the improvements in software development at the individual level have been substantial. The issue raised by this analysis is that the improvements in individual productivity due to individual-level tools, which do help software developers, are only indirectly linked to team performance. The same is not as certain for methods that focus on organizing the group (see, i.e., Reference 71). These data suggest that this absence of affect arises because current methods are not developed around the social interactions of developers, focusing instead on producing software. Since the three levels of software development--individual, social, and production--are tightly tied together, a myopic focus on production actually decreases the potential value that software development methodologies are to provide.

Interesting sample demographics. One issue with generalizing the findings from this study is the potential uniqueness of the sample. First, these developers produce packaged software and this is different from traditional information systems development.⁴⁷ Second, data in Table 1 suggest that three characteristics of the developers at this site are unusual: level of formal education, years of professional experience, and team stability (as measured by time in the same job and time as a member of the same team). This combination of extensive formal education, professional software development experience, and team stability suggests that, while the production and social aspects of software development provide some predictive value, the major differentiating factors may be at the individual level.

These data imply that, even with well-developed communication, coordination, conflict-management skills, and a strong sense of team supportiveness, the performance of software development teams hinges on individual talent. This scenario lends additional credence to Boehm's⁷² work with software cost drivers. He suggested that individual programmer differences are three times the power of the team's effect on software costs. Weinberg¹ posits that the difference between the best and worst programmers may be a factor of 10. One commonly held belief is that software gurus and tremendous individual contribution are the basis for commercial software success (see, e.g., References 3, 40). The current study provides data to support this assertion. What is not clear is the extent to which the social aspects of software development mitigate or enhance individual skills.

Paradoxes for practice. Generalizing to a broader population based on the data from this small, and possibly unique, sample must be seen as speculation. That said, we speculate that the findings of this study suggest two paradoxes. The first point is the need for teams of software developers to provide diverse perspectives versus the use of software development methods that are designed to minimize variance. That is, one of the premises in establishing software methods and formal production processes is to remove some of the variance that working together in the highly dynamic, conflict-oriented, pressure-filled workplace that characterizes commercial software development seems to demand.

In contrast to the cross-functional, heterogeneous basis of teams, formal software processes are designed to remove individual variability. This is done by focusing on how deliverables occur, implicitly ignoring who makes them. The premise behind using teams in software development is that the production losses due to the need for people to work together are offset, it is believed, by the production gains of this group effort.⁷³ This means that a defined software development process is designed to reduce the individual variability that the use of teams comprising multiple software development specialists is supposed to enhance. The data in this study indicate that, while a defined software development method may not directly contribute to software development performance, its absence heightens the effects of the social processes.

A second point is that the skills surrounding the social process issues of coordination, communication, conflict management, and supportiveness are typically not being provided to developers as part of their formal education. So, these skills are either developed on the job or the developers are limited by the ability to use only their technical knowledge in the social world that is software development. Given the increasing technical demands and the quick-changing nature of computing technology, commercial software developers are often caught between maintaining and expanding their technical skill base while also being required to learn and use these "soft skills." Since software developers have relatively low levels of social needs,^74,75 this increased emphasis on socialization may be producing part of the stress that Heartland's developers are experiencing.

This skills paradox suggests that embedding mechanisms to improve the social processes between developers can aid in developing both sets.³³ Curtis⁶⁹ asserts that there is still too little attention paid to developing methods of software development based on the socialization patterns and needs of the developers (though Microsoft's synch-and-stabilize approach does so implicitly^50,70). Certainly practicing developers understand the need for interaction and communication at an informal level.^1,6 As Pressman⁷⁶ observes about the obvious, "A process that stifles creativity will encounter resistance."

A third point is suggested by the data in Appendix A. The correlations among the three social process factors and formal coordination factor suggest that these four factors may represent facets of a higher-level factor. That is, these four factors may be representing a higher-level factor such as "software development team culture." This nebulous concept is both acknowledged in practice and not easily measured.^39,47 In that context, this research represents a small step toward a better understanding of the norms, behaviors, myths, and rituals that make up the culture of software development, a culture that extends beyond the realm of production.^39,77

The unexplained variance. We set out to explain the performance variations in software development due to both production and social factors. Analysis of the survey data from the 40 software teams at Heartland suggests that other, unmeasured factors must account for the unexplained variance. We focused on the social and production levels of software development teams. Findings from this study suggest that we should now include individual-level factors such as motivation, experience, and knowledge (see, e.g., References 1, 6, 38). At the social level, we could also expand the analysis to include other factors. For example, we did not account for team leadership and influence,^78,79 other aspects of group process such as boundary spanning,⁸⁰ intragroup control,^41,55 and social power,⁸¹ and more of the cultural aspects of development.^39,82

From a production perspective, additional analysis might benefit from looking at the specific subcycles of software development tasks and focus in more detail on the use of specific techniques and tools. And, other measures of software development performance (such as lines of code or function points) may be more useful (i.e., Susskind¹⁸ used a combination of reported and measured factors). Further, while there will always be random chance, we believe that it plays a smaller role than, for example, the 75 percent for stakeholder-rated product quality. The better accounting for variance using the self-reported (versus stakeholder-rated) team performance measure is, as stated, due in part to the use of one instrument to collect both process and outcome data. Finally, extending the analysis to allow comparisons across organizational boundaries may help to explore contextual factors that might influence software development. Given all these issues, what this research suggests is that we are still unsure of many of the key contributors to explaining the variance in software development performance.

Acknowledgments

The assistance of the Heartland developers, Bob Spillers, Joel Farber, Harry Campbell, Dave Tolleson, Mike Pauser, and Mike Dockter have made this paper possible. Comments on earlier drafts from Bob Benjamin, Lisa Covi, Kevin Crowston, Bob Heckman, Chatpong Tangmanee, Ping Zhang, and three anonymous reviewers have substantially improved this paper.

Appendix A: Factor correlations

Appendix B: Scale development

To develop the measures of social process we draw on two sources. For the ability of the team to manage intragroup conflict we use a scale developed and validated by Green and Taber.⁸⁴ To measure the level of informal coordination, communication, and feelings of supportiveness, we draw on a scale developed by Hackman⁸⁵ for use in evaluating group process behaviors. To depict the three factors used to assess the production processes, we draw on a measurement scale for technology that focuses on technology use⁸⁶ that is tailored for the software development domain.⁸⁷ The scale is adapted from Kraut and Streeter.⁸⁸

We chose a set of performance measures that encompass a multiattribute view of software performance. The three measures--product quality, team efficiency, and team effectiveness--are drawn from a scale used to assess software development team performance.^19,55,83 Self-reported performance data were collected from the developers. This scale draws on the work of Guinan, Cooprider, and Sawyer.¹⁹ Scales were pretested prior to use and the entire scale was used to assess the factor for analysis (see the note in Table 1).

*Trademark or registered trademark of International Business Machines Corporation.

**Trademark or registered trademark of Microsoft Corporation.

Cited references

Accepted for publication May 1, 1998.