Software Development: The Process

Software development and delivery process is the backbone of business for a professional organization in software industry. Software development is a team activity involving more than one professional with defined set of roles and responsibilities. The involvement of multiple people in the process requires a well defined work flow to be maintained. Apart from human resources the software development process (SDP) typically requires development software tools, platforms and hardware infrastructure and most importantly a strong collaboration mechanism within the team involving the artifacts of the delivery. Ideally a software development and delivery process (as per the water falls model) involves certain technical steps or phases as briefly described below. Phases including financial, business and management activities such as scoping, estimation, planning are not included here.

Requirement Study & Analysis: Where a person (Requirement Analyst) interacts with the user or customer to understand the process and requirement that the software (to be developed) should implement. The information captured in this phase is generally captured in Software Requirement Specification (SRS). Test cases to validate the system (after development) are also defined following this phase.

Architecture & Design: Specifying and documenting the architecture and design for the software system to visualize the basic structure or model which should be followed to construct the software system. This phase can have two primary steps such as -

· Defining the business architecture or functional design by covering the business processes and logic

· Defining the technical architecture or system design explaining the structure of system components required to implement the business processes.

Design and Architecture specifications are documented in this phase.

Development & Coding: In this phase, different software components are built or developed according to the design specified in the previous phase and tested. The outcome of this phase is source code and executable code.

Deployment & System Testing: All components developed are put together and tested to evaluate the ability of the system to provide the functionalities specified during the requirement capturing phase. Test cases defined during the requirement capturing phase are executed in this phase and test results are captured and defects found during testing are fixed.

User Acceptance: Once system testing is completed the software is submitted to the customer for final acceptance. This phase ideally have customer testing involvement. Defect fixing activities (if any) are also involved in this phase.

Operation & Maintenance: Once the software is accepted by the customer, it is finally delivered and installed for final use. In case any change in requirement takes place or any problems are identified by the customer after the delivery is done, necessary changes to implement new requirement or to fix problems are implemented in the system. In this case a new waterfall (starting with Requirement Analysis phase) can be followed for each new requirement or defect. Please refer to the waterfall model at http://en.wikipedia.org/wiki/Waterfall_model.

Monitoring & Tracking: Overall monitoring and tracking activities for the entire project are performed by a project manager (PM) throughout the phases of a software development lifecycle (SDLC).

The steps mentioned above are depicted in figure 1 below.

Figure 1: Software development lifecycle phases

Pain Areas in the Process

Æ Majority of employees in a software development organization (SDO) are involved in the process of software development for customers. The development activities (projects) are performed by forming project teams consisting of different people. Typically a lead or manager is involved to monitor and track a project. Apart from the project team members (PTM) other stakeholders such as reviewers, users and customer also get involved throughout the lifecycle of a project. Nowadays most of the stakeholders come from different geographic locations even with different time zones. This is the reason why a project requires a strong collaboration mechanism to be in place.

Æ Usage of standard practices, software tools and frameworks increases the efficiency of the software under development by reducing defects and improving performance, flexibility, extensibility etc. Enforcing standards and best practices over a team of people can be a challenge without a centralized control and strong collaboration.

Æ The process of requirement study & analysis does not only involve human interactions, it can also involve several documents and pre-existing software systems as well. Sometimes existing source code from the previously existing system can also be taken as the input for requirement. Recovering the design and logic from an existing system or source code might need a reverse engineering mechanism to be in place.

Æ Proper event tracking technique always helps in getting everyone in a team notified about the latest updates and help them to react promptly.

Æ The next problem commonly faced is increasing gap in communication which often results in a big difference between the requirement and the development software system. Figure 2 illustrates the same with a popular cartoon. Usually this problem leads to wastage of investment and following rework.

Figure 2: Gap between requirement and implemented system

Æ Misunderstandings in certain areas of requirement between a person with and without software background create major problems in communication as illustrated by the famous Dilbert Cartoon in figure3. Ideally a predefined flow of communication and precisely designated requirement capturing interface with relevant questions and clarifications can help in reducing such gaps. Educating the stakeholder (without software background) a little also helps here.

Figure 3: Misunderstanding in requirement capturing

Æ Communication Gaps can exist between the requirement analyst (who understands the requirement from the customer or user) and the architect or designer which can be minimized by recording requirement with all necessary information which can help the architect or designer. Similar gaps can exist between designer and developer which lead to defects introduced during development phase.

Æ Sometimes manual effort spent in build, deployment and installation procedures leads to human dependency. This process can be made automatic and time-driven with the help of a Continuous Integration (http://www.thoughtworks.com/continuous-integration) technique.

Æ Rather than putting the test cases in document only and executing manually during the testing phase, Test Driven Development (TDD) Tools can be used to record the use cases and execute them automatically by integrating the same with Continuous Integration tools. Martin Fowler’s article (http://www.martinfowler.com/articles/continuousIntegration.html) on Continuous Integration explains this more clearly.

Æ Overall a central repository with version control mechanism is a very important component to keep all artifacts which are referred (e.g. customer supplied documents) and produced (e.g. SRS, Design Document, Source Code etc.) during a project secured.

Goals to Achieve

The primary goal of software development process is to produce deliverables according to requirement with expected quality. The artifacts produced by a project which can be treated as deliverables are usually the SRS, architecture document, design document or source code etc. The final goal of a software development project should ideally be a working piece of software meeting customer’s requirement.

During SDLC, injection of defects is inevitable. Defects are introduced in different phases of a software development project such as Requirement Analysis, Design or Development. The impact of defects on delivery process and cost to fix them is directly proportional with time. The bugs discovered at the end of the SDLC would cost more to get fixed. So it is better, to find problems in the initial state of SDLC and fix them at the earliest. Based on a study (https://buildsecurityin.us-cert.gov/articles/best-practices/code-analysis/business-case), it has been found that, the savings potential from early defect detection is huge; nearly 60 percent of all defects usually exist by design phase. Figure 4 shows how the cost to fix a defect increases with time.

There are certain techniques such as Pair Programming (PP), Continuous Integration (CI), and Test Driven Development (TDD) etc. which are found to be highly effective in catching the bugs early. While we think of an integrated software development environment (ISDE) as a solution, these techniques need to be considered as well. So we are assuming that, our solution will be equipped to facilitate such techniques. For this article, let’s assume a name for the solution as Integrated Software Development Factory (iSDF).

Figure 4: Increasing Cost to Fix a Defect with time

Integrated Software Development Factory (iSDF): In Search of a Pain Killer

A discussion on so many pain areas together is enough to get a strong headache for anyone. So if we quickly look for a pain killer, a solution to address all primary pain areas in developing a software solution or system, we need to map the resolutions of the pain areas as use cases to the pain killer. Let’s name this pain killer as Integrated Software Development Factory (iSDF). Typical use cases for iSDF deduced from the pain areas described above are the following.

· Submit Requirement: The software end-user or customer should be able to enter his / her requirement in our iSDF using a graphical user interface. The customer should also be able to submit associated documents supporting the requirement. The requirement will be stored in system and later referred by other involved in the project team. Once a requirement is submitted, all relevant stakeholders such as requirement analyst, PM and architect should be notified to take necessary actions to process the same.

· Study & Analyze Requirement: The requirement analyst should have an interface to view the requirement statements and supporting documents entered by the customer for analysis. The analyst should also be able to interact with the end-user / customer for clarifications on the requirement. The analyst should be able to produce and submit an SRS document according to pre-defined format or template. Once the SRS is submitted, required stakeholders such as PM and architect should be notified to take necessary actions to process the same. iSDF should capture requirement in predefined format such as use case or user story so that other PTMs can refer to the same following a specific standard and paradigm (e.g. Object-Oriented Paradigm). Specially, the designer should be able to refer to the use cases or user stories by using a design tool integrated with iSDF. This will help in reducing the interpretation gap between requirement and design.

· Define Architecture: iSDF should provide an interface for the Architects to view SRS and define the architecture for the software application based on the SRS submitted by the analyst. Relevant project members such as PM and architecture reviewer should be notified while architecture is defined and submitted in iSDF.

· Review Architecture: Once the architecture is produced, a technical reviewer should be able to review the architecture and provide comments on the same. The architect will incorporate changes suggested by the reviewer (if any) and update the architecture document. Proper notification mechanism (to PM and Architect) should be in place on review comments submission.

· Define Test Cases: iSDF should provide an interface for the Testers to view requirement and define required test cases for software application based on the SRS submitted by the analyst. Relevant project members such as PM, analyst and customer (if required) should be notified while test cases are identified and submitted in iSDF.

· Review Test Cases: Once test cases are produced, a reviewer (ideally analyst or business users) should be able to review the test cases and provide comments on the same. The test case creator will incorporate changes suggested by the reviewer (if any) and update the test cases. Proper notification mechanism (to PM, analyst and other concerned members) should be in place on review comments submission.

· Define Design: Once the architecture is finalized after review, the designer should be notified to start producing the software design. iSDF should provide an interface for the designer to view the requirement (in form of use case or user story), architecture and supporting documents such as SRS and architecture document. The designer should be able produce the software application design by using some design tools integration with iSDF and submit. Once submitted iSDF will store the design and relevant documents and notify the PM and design reviewer.

· Review Design: Once the design is created, a technical reviewer should be able to review the design and provide comments on the same. The designer will incorporate changes suggested by the reviewer (if any) and update the design document. Proper notification mechanism (to PM and designer) should be in place on review comments submission.

· Develop & Build: The next phase to design is development and build, which primary includes the generation of source code in desired programming languages, customization and configuration. The developers should be provided with proper interface (in iSDF) to be able to build or develop software applications to work in this phase.

Integrated development environments (IDE) like Eclipse and NetBeans are few examples of development tools which can be integrated with iSDF so that, design models (e.g. UML) can be viewed and referred using the IDE. Code generation options can also be integrated to generate code from the design models whenever required. The source code, configuration files and other associated artifacts produced during this phase should be stored in a centralized repository secured by a version control system such as Git, SVN and CVS etc. so that everyone in a development team should be able to share and update the source code as required.

An automatic build tool can be integrated with iSDF so that, the build process can be scheduled in a periodic manner or based on an event (such as on any update in source code repository). Usually this feature is available in typical Continuous Integration (CI) tools like Hudson, Apache Gump and CruiseControl etc. Once triggered by a time-driven or a event-driven approach, the scope of activities for a CI tool should be to check out the source code and configuration files from version controlled repository, compile and build the same, deploy the application on an execution platform e.g. an application server for a test tool to start testing.

Technical reviewers should be notified on any code change or repository update so that, the new code change can be reviewed.

Figure 5: iSDF use cases

· Review Code: Once a new code or modified piece of code or other artifacts are submitted in repository, a notification should be send out to assigned reviewer to review the new or modified artifact and provide comments or approve the change. Once the comments or approval is posted the concerned developer or changer should be notified by iSDF notification mechanism. For any clarification, an integrated collaboration mechanism can be used by the reviewer and changer if required. To support required activities during the review, a code review interface, a notification, a collaboration mechanism and workflow should be in place.

· Test: This is an important phase in a SDLC. iSDF should have an interface for the testers to produce test cases associated with the requirement of a project during the requirement analysis phase. Once the development is completed, testers should be able to configure the test cases in automated testing tools integrated with iSDF. For example, Fitness is a very useful wiki based tool to implement Test Driven Development (TDD). Such tools can be integrated with the CI layer (within iSDF) so that, build, deployment and testing can be automated removing human dependencies. Once test results are produced, a notification mechanism can circulate the results to concerned parties (e.g. developer, project manager and tester etc.) for further actions.

· Defect Tracking: Defects can be recorded and monitored using a Defect Tracking tool such as Bugzilla, Jira, Trac, Traxis and Mantis etc. In order to do this, iSDF should be integrated with a Defect Tracking Tool. While defects are introduced concerned parties can be notified by the system for further actions.

· Deploy & Install: Different versions of application execution platforms or environments for development, testing, staging and production can be set up and integration with iSDF so that, once the application is developed and tested is can be easily deployed or installed on a designated execution environment. Developers can use a development environment to deploy and perform unit testing on their code. On completion of development and code review, individual code changes can be integrated and deployed on a testing environment for the testers to start testing. Once the testing is over, the tested application can be deployed on staging environment for customer’s acceptance testing or review.

· Acceptance by Customer: Our iSDF solution should also provide an interface for the customers or business users to test and review the completed application and provide final acceptance or sign off. Typically, a staging environment and an interface to report defects are provided for customer acceptance testing. The defect reporting interface should have notification capabilities to keep the concerned parties informed.

· Monitor & Track SDLC: Apart from an event based notification mechanism, iSDF should also provide a facility so that the entire SDLC followed in a project should be monitored and tracked by a project manager and other concerned stakeholders on a regular basis. Usually the progress of a project, issues and defects are the primary items of interest for monitoring.

· Administration: Capabilities like User management, user authentication, authorization, project creation, project member allocation can be implemented as administrative activities required for iSDF.

Recipe of a Pain Killer: The Solution

In order to implement the abovementioned use cases certain technical capabilities will be necessary for our iSDF. These capabilities (listed below) together can be considered as a good recipe for iSDF.

iSDF should have a Web based Graphical User Interface or portal to be able to provide all user interfaces required for requirement submission, requirement analysis, collaboration and document management.

A Workflow Management Framework will be required to manage workflows and human interactions in terms of requirement analysis and reviews etc.

A Software Design Tool or interface should be integrated with iSDF so that, the requirement documents or SRS can be referred by the designer and design the software accordingly. Ideally a use case model prepared by the requirement analyst should be captured and shared among relevant users within iSDF in a predefined format so that the design tool can interpret the same as the starting point of design. Apart from preparing design models (e.g. class diagram, sequence diagrams and activity diagrams etc.), the designer can also prepare a design document and store the same in centralized repository of iSDF. The centralized repository can be controlled by version control systems to secure the documents.

A Code Generation feature can be added to iSDF to generate initial code structure from design. For example, once the designer completes class diagrams for an application and executes the code generator on the same, the code generator should be able to read the class diagrams and generate initial source codes in desired programming languages (e.g. Java, C++, C# etc.). Once the initial source code structure is generated, they can be stored (checked) in a version controlled repository for reference later by the developers.

Development Tools or environments can be integrated with iSDF, where developers can check out the existing source codes from version controlled repository, modify the code, compile and check in the modified code back in the repository. Once there is an update in source code repository, an automatic event or notification should be generated in iSDF that should trigger the build process (through a CI tool).

A Reverse Engineering facility of iSDF can help in generating design models from code (in case existing source codes are referred as requirement artifacts). For example, class diagrams can be generated from Java source code.

A Reusable Component Repository (containing standards, frameworks, reusable code etc.) should be available with iSDF for the designer, architects and developers to refer to and reuse during architecture definition, design and development.

An Email Notification mechanism to notify members involved in a project or administrator whenever there is a change in repository. For example, the PM and requirement analyst should be notified whenever the customer or business user submits a requirement. Similarly architects, designers should be notified whenever the SRS is finalized and code reviewers should be notified once there is an update on source code.

A Collaboration Framework for the geographically distributed members to collaborate within each other during a review, analysis and general discussions. The mode of communications can be implemented through chat, email, desktop sharing and video chatting etc.

A Continuous Integration platform or tool to automate build, testing and deployment processes should be implemented in iSDF. This is make the process of compiling, building source codes and deploying application packages on desired execution platforms e.g. application or web server etc. Once an update is made on source codes by the developers and automatic event will be generated which will trigger the CI tool to start the build and deployment process. Figure 6 depicts a typical continuous integration process.

A Build Tool should be provided as an integrated feature of iSDF. This tool should ideally be integrated with the CI component so that, whenever there is an update in code repository this tool can be automatically invoked based on an event. On invocation, the tool should be able to check out the latest source code changes from the code repository, compile, build and deploy the same on an execution platform (e.g. application server) for execution. Some popular build tools used nowadays are Apache Ant, Apache Buildr, Apache Maven, Cabal, Gradle, Bamboo etc.

Integrated Testing Tools for software application testing should be provided as an important feature of iSDF. The testing tools which can implement a test driven development (TDD) process such as Fitnesse. It should provide the feature to configure the test cases and test data in it. The testing tool should be integrated with the continuous integration tool or platform so that, on each update in source code or application configuration, testing can be executed automatically and the results can be shared (using email) with appropriate stakeholders. Ideally when the application is built and deployed on the execution platform (e.g. Application Server), the testing tool should start executing the test cases which were already pre-configured in it.

Figure 6: Continuous Integration Process

A Defect or Bug Tracker tool can be integrated with iSDF to manage and track defects. These tools can be integrated with iSDF to register and track bugs or defects found during testing. Monitoring and tracking of defect life cycle and maintaining a defect repository for further analysis are important aspects of this.

A User Authentication & Authorization mechanism to authenticate and authorize users (preferably directory service integration will be a modern technique for this) should be integrated with iSDF. This will enable security and restrict user access based on authority.

Reporting Tool is a significant component for any application software. For iSDF, the customer, project manager and other designated stakeholders can monitor and track the software development process by using desired web based reporting interfaces or dashboards.

Figure 7 shows the entire iSDF solution as conceptualized in this article.