Report R4: Midterm Report

Define a process for analyzing your PSP data and producing a report. Use this process to produce report R4. You will update this process to produce the final report at the end of the course.

Requirements

 

Task 1-- Develop a process for analyzing the data and creating a report on programs 1A through 6A. This should include the results listed below under Task 3. This process must include a planning phase, the task performance phases, and a postmortem phase. Produce and submit a process script and planning forms for enacting this process.

Task 2-- Plan and enact the process defined in Task 1. Use the planning form to record the planned time for this work and track and record the actual time you spend. Submit the planned and actual process data together with the analysis report.

Task 3-- Analyze the data for programs 1 through 6. Spreadsheet analyses are suggested and graphical summaries and presentations are encouraged. At a minimum, produce the following:

  1. An analysis of LOC and development time estimating accuracy and how it has evolved during the programs developed to date.

  2. An analysis of the defect types injected and removed for the programs developed to date. These data should be shown in a format similar to that in table D23.

  3. An analysis of the defect types found by the compiler. These data should be shown in a format similar to that in Table D24.

  4. An analysis of the defect fix times, using the format used in report R3 (Table D22).

  5. Develop a design review checklist to find the most-frequent design defects in a design review.

  6. Develop a code-review checklist to find the most-frequent coding defects in a code review.

Result: Use the GQM method to draw conclusions and set personal improvement goals. Submit the required analyses, tables, and checlists, together with a brief written report. Use graphs wherever possible.

Table 7-1. D23: Percentage of defects injected and removed by phase

 Number Injected Percentage Injected Number Removed Percentage Removed 
TypeDesignCodeDesignCodeCompileTestCompileTest

Table 7-2. D24: Percentage of Defects Found in Compile

Defect TypeNumber of defects at compile entryNumber of defects found in compilePercentage of type found by the compiler
 
--[Humphrey95] 

Report Process

Without much guidance, I'll be flailing a bit on this, but I'll base this process "script" off Humphrey's format for the PSP scripts.

Table 7-3. Report R4 Development Script

Phase NumberPurposeTo guide the reporting of PSP data
 Entry Criteria

  1. Requirements statement

  2. Time and defect logs in .ppl format

  3. Project summary logs for all programs, listing LOC estimates and actual LOC.

  4. Tools for analysis and graphing (evalpplog and gnuplot)

  5. For gnuplot, the gnuplot scripts generate_loc_graph.gnu, generate_loc_accuracy_graph.gnu, and generate_time_graph.gnu.

1Planning

  1. Verify you have the data necessary to fulfil the requirements

  2. Make an estimate of the time necessary to produce the required report

  3. Complete the time recording log

2Data Collation

  1. Enter rough-guess estimated LOC vs actual LOC in a file named "loc.plt", in a format suitable for analysis by GnuPlot (ie rows of "x y" pairs).

  2. Enter PROBE-estimated LOC vs actual LOC in a file named "loc_accuracy.plt"

  3. Enter estimated time and actual time in a file named "time.plt", in a format suitable for analysis by GnuPlot.

  4. Create error data for LOC and time by using the equation:

    Error% = 100 * (actual - estimate)/estimate

    The script "make_error_numbers.py" can be used to automate this, as well as the use of makefile targets (see lesson 7 Makefile.am); these will be files consisting of single numbers, each to their own line

3Data analysis

  1. Generate graphs for LOC estimation, LOC estimation accuracy, and time estimation accuracy, using the commands gnuplot 'commandfile.gnu'; use the script files generate_loc_accuracy_graph.gnu, generate_loc_error_graph.gnu, generate_loc_graph.gnu, generate_time_error_graph.gnu, and generate_time_graph.gnu. The use of Makefile targets is encouraged.

  2. Use evalpplog to gather and collate time and defect log statistics. The evalpplog tool can collate data from multiple logs. If your logs are all named "cpp*ppl", and are located under the current directory tree somewhere, the command line find . -name "cpp*ppl" | xargs evalpplog will analyze all the logs.

4Report generation

  1. Using the graphs from phase 3 and the historical data, write an analysis of LOC and development time estimation accuracy and how it has evolved.

  2. Using the format in the requirements, use the composite error log generated in phase 3 to analyze the defect types injected and removed for the programs.

  3. Using the composite error log, create an analysis of defect types found by the compiler.

  4. Using the composite error log, create an analysis similar to report R3 on defect types.

  5. Using the composite error log, analyze design-injected defects to update or create a design review checklist to find the most-frequent design defects in a design review.

  6. Using the composite error log, update or create a code review checklist to find the most-frequent code-injected defects in a code review.

5PostmortemComplete a report summary with actual time data
 Exit criteria

  1. Completed report summary with actual and estimated time data.

  2. Completed time logs.

  3. LOC/Time estimation accuracy report.

  4. Defect type injection/removal report

  5. Defect types found by compiler report

  6. Defect fix times report

  7. Updated design review checklist

  8. Updated code review checklist

Report R5

LOC and time estimation accuracy (C++ only)

LOC data is taken from programs 2A through 6A; first graph is a depiction of rough-guess estimated LOC vs actual program LOC, showing the best-fit linear regression line through the data points. As the graph shows, the data is not particularly accurate, and the "best fit" approximation is not particularly good.

rough-guess estimated LOC vs actual LOC

Rough-guess estimated new/changed LOC vs actual program new/changed LOC; with these four data points, the fit is fairly bad.

A more "calibrated" comparison is the comparison of PROBE- generated estimates vs actual LOC; the "actual" numbers are the same as the last graph, but the estimates are those generated from the full PROBE algorithm. A "perfect fit" line is plotted to show how the PROBE estimates varied from a perfect 1-to-1 match

PROBE estimates vs actual LOC

PROBE estimates vs actual data. The samples are still very scattered and not particularly accurate-looking.

In fact, it doesn't look like we're doing well on accuracy at all, until we take a closer look at accuracy by using the "percent error" equation on the numbers.

%error in LOC estimation by program

Percent error in each estimation. Note the downward trend.

A look at the time figures reveals some similar trends:

Estimated dev. time vs actual dev.time

Estimated development time vs actual development time. Aside from an obvious outlier, this has actually been fairly regular.

Percent error in time estimation by program

Percent error in each time estimation; note the similarity to the LOC percent error graph.

...and a similar analysis of the time estimation error reveals an extremely similar curve. This makes sense, since the time estimates are made directly from size estimates; they should form similar curves. Still, it's good to see that they are similar-- and that our estimation abilities seem to be getting better over time. Note the very large spike in percent error near the beginning of the graphs; this is program 2a, which was significantly larger than predicted and certainly took a great deal of time.

Defect types injected/removed

Table 7-4. Percentage of defects injected and removed by phase -- C++

 Number Injected Percentage Injected Number Removed Percentage Removed 
TypeDesignCodeDesignCodeCompileTestCompileTest
md (missing design)15110.60.7120.94.3
me (missing error-handling)0100.70 1 0 0.7
wn (wrong name)1190.713.515 6 10.6 4.3
mi (missing implementation)0100.70 1 0 0.7
wa (wrong algorithm)482.85.70 13 0 9.2
wc (wrong condition)201.400 2 0 1.4
ch0100.71 0 0.7 0
wt (wrong type)0402.83 1 2.1 0.7
we (wrong expression)211.40.70 3 0 2.1
is (interface specification)0100.70 1 0 0.7
ic (interface capability)17512.13.52 4 1.4 2.8
sy (syntax)043030.543 0 30.5 0
id (interface description)110.70.71 0 0.7 0
ma (missing assignment)0100.70 1 0 0.7
iu (interface use)0805.75 4 3.5 2.8
Eiffel        
mc (missing call)0505.5414.41.1
md (missing design)202.20111.11.1
wn (wrong name)013014.310311.03.3
mi (missing implementation)0202.2020.02.2
wa (wrong algorithm)010011191.19.9
wc (wrong condition)0303.3030.03.3
sy (syntax)031034.131134.11.1
wt (wrong type)0101.1101.10
is (interface specification)0101.1101.10
ic (interface capability)252.25.5515.51.1
ma (missing assignment)0808.8444.44.4
iv (invariant)0101.10101.1

Defect types found by the compiler

Table 7-5. D24: Percentage of Defects Found in Compile -- C++

Defect TypeNumber of defects at compile entryNumber of defects found in compilePercentage of type found by the compiler
md3133
me100
wn211571
mi100
wa1300
wc200
wc200
ch11100
wt4375
we300
sy4343100
ic6233
is100
id11100
ma100
iu9555
Eiffel   
mc5480
md2150
wn131077
mi200
wa10110
wc300
sy323197
wt11100
ic6583
is11100
ma8450
iv100

Defect fix times

Table 7-6. Defect Fix Times

C++   
 Defects found in compileDefects found in testTotal Defects found
Defects injected in design
Total Fix Time:0.9
Total defects:1
Average fix time:0.90
Total Fix Time:108.3
Total defects:11
Average fix time:9.8
Total Fix Time:109.2
Total defects:12
Average fix time:9.1
Defects injected in code
Total Fix Time:31.2
Total defects:69
Average fix time:0.5
Total Fix Time:155.2
Total defects:25
Average fix time:6.2
Total Fix Time:186.4
Total defects:94
Average fix time:1.98
Total defects injected
Total Fix Time:32.1
Total defects:70
Average fix time:0.45
Total Fix Time:263.5
Total defects:36
Average fix time:7.31
Total Fix Time:295.6
Total defects:106
Average fix time:2.79
Eiffel   
 Defects found in compileDefects found in testTotal Defects found
Defects injected in design
Total Fix Time:2.7
Total defects:2
Average fix time:1.3
Total Fix Time:4.7
Total defects:1
Average fix time:4.7
Total Fix Time:7.4
Total defects:3
Average fix time:2.5
Defects injected in code
Total Fix Time:44.6
Total defects:56
Average fix time:0.8
Total Fix Time:80.0
Total defects:24
Average fix time:3.3
Total Fix Time:124.6
Total defects:80
Average fix time:1.55
Total defects injected
Total Fix Time:47.3
Total defects:58
Average fix time:82
Total Fix Time:84.7
Total defects:25
Average fix time:3.4
Total Fix Time:132
Total defects:83
Average fix time:1.6

It's interesting to note that defects found in test seem to be substantially larger, or at least require longer fix times. The too-simple conclusion is that errors found later in the development cycle "cost more" than errors found earlier. This is probably true, but it's also true that the compiler is catching very simple errors automatically-- errors which couldn't be caught during test, because the program simply won't compile enough to test without them.

Design Review Checkliste

The largest categories of design-injected failures are "missing design" and "interface capability" (which is essentially a lesser manifestation of "missing design"). Less frequent are "wrong name", "wrong algorithm", "wrong expression" (a lesser manifestation of wrong algorithm), and "wrong expression" (a boolean manifestation of wrong expression).

It's tough to know where to start with this. These are all fairly fuzzy things; I'll start with Humphrey's design review checklist on [Humphrey95] page 705, and attempt to expand it as necessary. The problem is, there isn't much necessary-- the "missing design" errors are covered by the "complete" category, and and "wrong algorithm" errors are covered mostly by the "logic" section. Still, I've added a few minor items, shown by emphasis.

Table 7-7. Design Review Checklist

PurposeTo guide you in conducting an effective design review 
General

  1. As you complete each review step, check off that item in the box to the right.

  2. Complete the checklist for one program unit before you start to review the next.

 
Complete

Ensure that the requirements, specifications, and high-level design are completely covered by the design:

  1. All specified outputs are produced

  2. All needed inputs are furnished

  3. All required includes are stated

  4. All appropriate data features have accessors/mutators

  5. Classes have make/constructor when appropriate

  6. C++: Classes with virtual functions have virtual destructor

 
Logic

  1. Verify that program sequencing is proper:

    1. Stacks, lists, and so on are in the proper order

    2. Recursion unwinds properly

  2. Verify that all loops are properly initiated, incremented, and terminated

  3. Ensure complex algorithms are factored into easily-computable parts where possible

 
Special Cases

Check all special cases:

  1. Ensure proper operation with empty, full, minimum, maximum, negative, zero values for all variables.

  2. Protect against out-of-limits, overflow, underflow conditions

  3. Ensure "impossible" conditions are absolutely impossible

  4. Handle all incorrect input conditions

 
Functional Use

  1. Verify that all functions, procedures, or objects are fully understood and properly used

  2. Verify that all externally referenced abstractions are precisely defined

 
Names

Verify the Following:

  1. All special names and types are clear or specifically defined

  2. The scopes of all variables and parameters are self-evident or defined

  3. All named objects are used within their declared scopes

 
ContractsCheck/create require/ensure/check/invariant contracts for all methods and classes 
StandardsReview the design for conformance to all applicable design standards 

Code Review Checklist

Big items injected in the coding phase are wrong name and syntax, with interface use, wrong algorithm, interface capability, and wrong type finishing up. With C++, wrong type is very sneaky, because it's good at automatically converting without any input from the programmer.

Humphrey gives us a C++ code review checklist, so I'll supplement it with emphasized lines (and modifications). I'll try and merge the Eiffel code review checklist with this, noting the obvious places.

Table 7-8. Code Review Checklist

PurposeTo guide an effective code review 
General

  1. As you complete each review step, check off that item in the box to the right

  2. Complete the checklist for one program unit before you start to review the next unit

 
CompleteVerify that the code covers all the design 
Includes (ignore for Eiffel)Verify that includes are complete 
Initialization

Check variable and parameter initialization:

  1. At program initiation

  2. At start of every loop

  3. At function/procedure entry

  4. In object construction

 
Calls

Check function call formats:

  1. Pointers (C++ only)

  2. Parameters

  3. Use of '&' (C++ only)

 
Names

Check name spelling and use

  1. Is it consistent?

  2. Is it within the declared scope?

  3. Do all structures and classes use '.' reference? (C++ only)

 
Pointers (C++ only)

Check that

  1. Pointers are initialized NULL

  2. Pointers are deleted only after new

  3. New pointers are always deleted after use

 
Output format

Check the output format:

  1. Line stepping is proper

  2. Spacing is proper

 
Block closureEnsure that blocks are properly opened/closed (brace matching in C++, proper closure in Eiffel) 
Logic operators/assignment

  1. Verify proper use of == vs =

  2. Verify proper use of = vs :=

  3. Verify proper boolean conditions; simplify where appropriate

  4. Check logic for proper parentheses nesting

  5. Check complex operations for possible refactorings

 
Line-by-line check

Check every LOC for

  1. Instruction syntax

  2. Proper punctuation

 
Standards/prettify

  1. Ensure code conforms to coding standards

  2. Prettify code (indent for C++, pretty for Eiffel)

 
File open/close

Verify that all files are

  1. Properly declared

  2. Opened

  3. Closed

 
Special interest items

  1. Ensure loop indices are updated (Eiffel)

  2. Ensure header-file/implementation-file parity, with special emphasis on const (C++)

 

Postmortem

Time preparing this report was spent as follows:

Table 7-9. Time Recording Log

Student:Victor B. PutzDate:000117
  Program:R4
StartStopInterruption TimeDelta timePhaseComments
000117 14:07:57000117 14:31:55023estimation accuracy 
000117 14:31:58000117 14:48:23016defect injection/removal 
000117 14:49:10000117 14:55:3406defects found by compiler 
000117 14:55:38000117 15:06:13010defect fix times 
000117 15:08:17000117 15:41:27033design review checklist 
000117 15:41:29000117 15:57:34016code review checklist 
      

I'm not doing an extensive postmortem on this because I don't feel the time involved would justify the reward. If I continue using the PSP, this section will have to be heavily automated, and will differ so much from the current hand-drawn method that it will no longer be recognizable.