FY09 SARP Research Topics Topic 1 Requirements Need

fy09 sarp research topics topic: 1. requirements =============== need: 1.1. automated requirements tracing tool -----

FY09 SARP Research Topics
Topic:
1. Requirements
===============
Need:
1.1. Automated requirements tracing tool
----------------------------------------
Automated requirements tracing and certification (to NPR 7150.2) tool.
A solution or tool to help develop and maintain traceability to
requirements for certification purposes. This tool would be one that
automates the process of certification of a software system to meet
NPR 7150.2 Software Engineering Requirements, Constellation software
engineering/development requirements, and Ground Operations Project(s)
software engineering/development requirements.
Constellation and Ground Operations and Project level documents are
being created. These documents are being warehoused in Cradle and
Windchill. Individual documents may have traceability matrices in the
appendices of the documents that serve to trace back to parent
requirements. Need bi-directional traceability in one tool. The
projects at KSC are concerned with meeting NPR 7150.2 as well as the
Constellation Program requirements for 1) successful product
development and 2) meeting the NPR 7150.2 requirement for CMMI
assessment purposes.
Relevant domain(s):
Ground Operations projects, implementing Constellation Program
requirements.
Project(s) that would use the proposed tool/solution:
Ground Operations projects
Current tool/solution in use and its shortcomings:
Cradle and Windchill is the configuration management tool for the
requirements and the documents. These two tools are used for all of
Constellation, and apparently do not offer the ability to create
traceability matrices or tables between the documents.
Constraints that might influence the research work plan:
Timeline that the proposed tool/solution must support:
ASAP
Language requirement:
English... actually SQL Server would be recommended as the database
development tool.
Size or scale requirements:
An enterprise level database server other than MS Access.
Required deliverables:
A tool to help develop and maintain traceability to requirements for
certification purposes, whether it be for CMMI or proof that Agency or
Program level software engineering requirements are met.
Other useful information:
Whoever is the PI for this would have to have intimate knowledge of
NPR 7150.2, the Constellation Software Engineering requirements, the
Ground Operations software Engineering requirements etc.
Topic:
1. Requirements
Need:
1.2. Ontologies for standards and requirements
----------------------------------------------
Relevant domain(s):
Project(s) that would use the proposed tool/solution:
Current tool/solution in use and its shortcomings:
Constraints that might influence the research work plan:
Timeline that the proposed tool/solution must support:
Language requirement:
Size or scale requirements:
Required deliverables:
Other useful information:
Topic:
1. Requirements
Need:
1.3. Formal methods for hardware and software specifications
------------------------------------------------------------
Relevant domain(s):
Project(s) that would use the proposed tool/solution:
Current tool/solution in use and its shortcomings:
Constraints that might influence the research work plan:
Timeline that the proposed tool/solution must support:
Language requirement:
Size or scale requirements:
Required deliverables:
Other useful information:
Topic:
2. Resource estimation
======================
Need:
2.1. Assurance cost/benefit analysis tool
-----------------------------------------
Software estimation tools: for both software and complex electronics.
A tool to provide a risk-based assessment of required assurance level.
Some people associate software class with level of risk, but many
people don't make that association. We've tried to research acceptable
risk levels for the potential loss of missions of various sizes and
the potential loss of people and property. NASA guidelines for
acceptable levels of risk exposure.
Relevant domain(s):
Projects at the concept phase can benefit from providing good
estimates of required level of effort. Help identifying the
appropriate level of assurance would primarily affect the projects
that do not have IV&V.
SMA organization, Software Engineers, Project Manager, software
community.
Project(s) that would use the proposed tool/solution:
Projects at the concept phase can benefit from providing good
estimates of required level of effort. Help identifying the
appropriate level of assurance would primarily affect the projects
that do not have IV&V.
All projects with SA support.
Constellation and its sub-projects.
Current tool/solution in use and its shortcomings:
Unofficial rough risk exposure diagrams have been created to identify
likelihood of loss versus value. Acceptable loss is based on range
safety limits for casualty expectations and rough NASA assurance
guidelines. Risk exposure diagrams are a way to compare one project
against another to see if assurance levels are consistent for similar
levels of risk exposure.
There are many cost and resource estimating tools but none
specifically designed for software assurance tasks which include
software safety, software reliability, software quality, software
verification and validation, and software independent verification and
validation. No tools cover complex electronics. Tool should also allow
for different software efforts based on software classification.
Constraints that might influence the research work plan:
Unfortunately it is difficult to relate loss of human life to a dollar
value to compare safety-critical to mission-critical levels. It is
also difficult to assign a dollar value to NASA's reputation. It would
be helpful to identify in one place how much NASA is willing to risk
human life, how much NASA is willing to risk the loss of X dollar
property, how much NASA is willing to risk loss of an X dollar
mission, and how much NASA is willing to risk its reputation. Not
everything needs to be 99.9999% reliable at a 95% confidence level.
The difficulty is identifying the appropriate level and making it
consistent across a variety of projects.
Timeline that the proposed tool/solution must support:
Language requirement:
Standard desktop application
Size or scale requirements:
Required deliverables:
Estimating tool
Other useful information:
CMMI v1.2
Complex Electronics Guidebook
Topic:
3. Model-based engineering
==========================
Need:
3.1. Architecture tools & techniques
------------------------------------
(1) architecture frameworks, (2) product line architectures, (3)
architecture description languages and modeling, and (4) reference
architectures.
A tool to analyze a software architecture to develop a claims
-evidence diagram would be helpful. Everyone seems to want to see a
list of the project-specific evidence/artifacts required to say
assurance is complete. The lowest level of the evidence diagram should
be a set of required artifacts. The diagram provides a scope of
effort. This information would also help assurance personnel explain
what projects are buying and why.
Relevant domain(s):
Architecture
Project(s) that would use the proposed tool/solution:
NASA flight software projects
Software assurance personnel would likely make use of software
assurance claims-evidence diagrams throughout the project life cycle,
as a road map.
Current tool/solution in use and its shortcomings:
Architecture Design and Analysis Language (AADL) and Rational Software
Architect (RSA).
Constraints that might influence the research work plan:
Timeline that the proposed tool/solution must support:
Language requirement:
Size or scale requirements:
Required deliverables:
A claims-evidence diagram template compliant with NASA standards that
can be tailored would be the delivered product. An interactive version
that facilitates project-specific modifications would be excellent
(but not expected).
Other useful information:
Topic:
3. Model-based engineering
Need:
3.2. Interoperability of frameworks and models
----------------------------------------------
Ensure interoperability and validity of the suite of frameworks and
models adopted for Orion flight software development and assurance, so
that several types of models are integrated, unambiguous and
consistent. Identify technical and process deficiencies that make it
difficult to design, implement and evaluate software with the suite of
tools.
Relevant domain(s):
Orion flight software development
Project(s) that would use the proposed tool/solution:
Orion flight software engineering
Current tool/solution in use and its shortcomings:
The problem is that there are a variety of tools and modeling
approaches: Model Driven Architecture, Object Oriented Analysis and
Design, Unified Modeling Language, System Reference Models and the
Department of Defense Architecture Framework (DoDAF). The tools and
their representations need to be evaluated for overlaps and gaps,
mappings and incompatibilities.
Constraints that might influence the research work plan:
Timeline that the proposed tool/solution must support:
Preliminary release by Orion CDR (Aug, 2009), follow up report and
release by end of Spiral 6 (June 2010)
Language requirement:
C++ and Matlab
Size or scale requirements:
Medium to large scale systems
Required deliverables:
A method and associated prototype tools
Other useful information:
Data access and tool access are constraints.
Topic:
3. Model-based engineering
Need:
3.3. Assurance of model-based auto-generated code
-------------------------------------------------
Practices, requirements, and guidance need to be developed for the
assurance of model-based software. Software being developed by first
generating a model and subsequently using an auto-coder to generate
the software based upon the model is becoming more common. The
traditional software assurance approach must be updated to account for
these changes. For instance, it is not effective to manually review
the large quantities of code that may be generated?
Relevant domain(s):
Flight software assurance
Project(s) that would use the proposed tool/solution:
Various Constellation projects are utilizing this technique.
Current tool/solution in use and its shortcomings:
Potential conflicts with existing software assurance requirements
Constraints that might influence the research work plan:
Timeline that the proposed tool/solution must support:
No specific timeline. However, model-based development is currently in
use and becoming more popular
Language requirement:
UML is the most common
Size or scale requirements:
Required deliverables:
A guidebook or standard, perhaps followed by procedures and checklists
Other useful information:
Topic:
3. Model-based engineering
Need:
3.4. State analysis
-------------------
State Analysis was originally developed at JPL for the MDS project and
perhaps has overtaken reference architecture and is a stand-alone
approach to model-based engineering. Of course, there are several
languages out there that can be used to perform model-based
engineering; State Analysis may be the most overlooked approach.
Consider using SpecTRM as a tool for capturing State Analysis
artifacts.
Relevant domain(s):
Model-based engineering
Project(s) that would use the proposed tool/solution:
Current tool/solution in use and its shortcomings:
Constraints that might influence the research work plan:
Timeline that the proposed tool/solution must support:
Language requirement:
Size or scale requirements:
Required deliverables:
Other useful information:
Topic:
3. Model-based engineering
Need:
3.5. VV&A of models and simulations
-----------------------------------
Relevant domain(s):
Project(s) that would use the proposed tool/solution:
Current tool/solution in use and its shortcomings:
Constraints that might influence the research work plan:
Timeline that the proposed tool/solution must support:
Language requirement:
Size or scale requirements:
Required deliverables:
Other useful information:
Topic:
3. Model-based engineering
Need:
3.6. UML quality metrics
------------------------
A tool to analyze a software architecture to develop a claims
-evidence diagram would be helpful. Everyone seems to want to see a
list of the project-specific evidence/artifacts required to say
assurance is complete. The lowest level of the evidence diagram should
be a set of required artifacts. The diagram provides a scope of
effort. This information would also help assurance personnel explain
what projects are buying and why. UML models do not follow the
standard software development model. Checklists and metrics are needed
to measure the quality of the UML model.
Relevant domain(s):
SMA organization, Software Engineers and project manager
Project(s) that would use the proposed tool/solution:
The current Constellation program includes its projects and
sub-elements (Ex. Ares/CLV/CEV etc…)
Current tool/solution in use and its shortcomings:
Currently there are no standard metrics available. Solutions are
achievable but resources are limited.
Benefits are better use of SA limited resources and better planning
for supporting in the areas of Mission Assurance.
Constraints that might influence the research work plan:
Timeline that the proposed tool/solution must support:
Language requirement:
Size or scale requirements:
Required deliverables:
Other useful information:
Topic:
4. Standards compliance
=======================
Need:
4.1. Software safety case approach and method
---------------------------------------------
Developers and reviewers need a way to evaluate software-intensive
system safety by looking directly at the arguments supporting safety
claims, not by confirming that the correct development process was
followed. For example, a systems engineer needs to ensure that the
argument that a flight abort decision is correct and is based on a
correct analysis of the probabilities of false positives and false
negatives from the abort detection system (among other things). In
European aerospace applications, this kind of assurance for safety
critical systems is often provided by safety cases, but our practice
does not include them.
The concept of a “safety case” (a subset of the more general
“dependability case”) has been in wide use in, especially, European
safety-critical industries (power, aviation, rail, etc), but has not
made inroads into the US, in particular, has not been adopted by NASA
and its contractors. As a result, there is understandable widespread
reluctance to commit to their use within NASA. In fact, while there
has been much discussion of Safety Case in CxP, it has generated
considerable controversy and no work has been done by NASA on the
concept; it remains without even a single case study here despite its
widespread adoption abroad. Safety cases (in the form of dependability
cases) were extremely controversial in the development of CxP 70065
Computing Systems Requirements, with many stakeholders believing that
safety cases are at too low a maturity level to include as CxP
requirements. Thus they were included only in the form of guidelines
(for example, “G/L-31-004 Dependability Case. Each project should
develop and maintain a Dependability Case to show, at different stages
of a project's life cycle, how computing system dependability will be,
is being, and has been achieved.”). We suggest research on safety
cases to raise the maturity level through a “shadow application” in a
CxP safety-critical flight context. Such research is needed to:
(1) Show a concrete example of a safety case for a representative
safety-critical NASA system in which software is a major functional
component.
(2) Indicate the efficacy of a safety case for software-intensive
systems – the value stemming from a “product” oriented perspective
that a safety case offers, as a complement to the “process” oriented
perspective on development practices (e.g., ISO, CMMI).
(3) Reveal the level of effort it takes to develop and to review a
safety case - the longer term goal is to develop estimators of the
cost, effort and skills required for NASA’s use of safety cases for
software-intensive systems.
(4) Indicate the extent to which existing practices will already have
gathered the information from which a safety case can be assembled and
how to modify existing practices to fully support safety cases.
(5) Offer guidance on how to develop and review a safety case for
software-intensive systems - the longer term goal is to develop
courseware for this.
Relevant domain(s):
Safety-critical software-intensive systems
Project(s) that would use the proposed tool/solution:
Many - Constellation in particular. One of the outcomes of the
research should be a characterization of the kinds of NASA
applications to which Safety Cases are applicable and appropriate.
Current tool/solution in use and its shortcomings:
There are some graphical support tools available in Europe that may be
considered for a NASA pilot project. There are also some tutorial
materials available in Europe. However, the concept of safety cases,
and the applicability of these support materials to NASA systems, is
considered untested by CxP.
The shortcoming of NOT using safety cases is the lack of a
product-oriented perspective on whether and why a system fulfills its
safety requirements. In particular, it is difficult to evaluate a
range of dynamically developed product-oriented safety queries, such
as, "What are the likelihood and consequences of a worst case
single-even upset in the state estimation computation during initial
ascent stage?” that could arise during design or implementation
reviews. Another consequence of not producing safety cases is that it
is more difficult to produce arguments to justify or refute claims
that a proposed new system to ensure safety does in fact do so (and
not simply decrease overall safety by adding error-prone complexity).
Constraints that might influence the research work plan:
Access to relevant information.
Timeline that the proposed tool/solution must support:
It would be ideal if the work could be initiated in time to apply to
Constellation designs.
Language requirement:
Size or scale requirements:
A modest-sized example is preferable as the one to start with. Too
small (e.g., a “voting” algorithm) and it would fail to encompass the
full range of factors. Too large (e.g., the entire avionics system)
and it would be too large an effort to serve as a convenient pilot
study.
Required deliverables:
End products: a safety case for the system studied, a record of the
effort, skills, data needs, etc. that it took to develop that safety
case, lessons learned/guidance to help future developers of safety
cases. Perhaps a Safety Case Developers Guide or Tutorial; these might
be modeled closely after Euro counterparts, with the example replaced
by a NASA case study example.
Other useful information:
The discussion of safety cases within CxP was conducted in the context
of Level 2 requirements. The guidelines are contained in CxP 70065 -
Computing Systems Requirements. Thus if safety cases turn out to be a
valid and useful safety analysis tool for NASA, it can be expected
that they will be widely applicable at least within Constellation.
Topic:
4. Standards compliance
Need:
4.2. Standards compliance tools
-------------------------------
Tools that fulfill requirements for each process area and generate a
report for use by NASA Centers NPR 7123.1, 7150.2, 7120.5 and STD
8739.8 and 8719.13 that cover both software, including complex
electronics, and systems engineering.
Relevant domain(s):
All NASA and contractor software, complex electronics and system
engineering covered that should follow NPR 7123.1, 7150.2, and 7120.5
and STD 8739.8 and 8719.13, but that haven’t selected a tool yet.
Project(s) that would use the proposed tool/solution:
All NASA and contractor software, complex electronics and system
engineering projects constrained by NPR 7123.1, 7150.2, and 7120.5 and
STD 8739.8 and 8719.13
Current tool/solution in use and its shortcomings:
Constraints that might influence the research work plan:
Schedules. Changing NPRs. Changing personnel.
Timeline that the proposed tool/solution must support:
Timelines from Constellation and Space Shuttle schedules and the
impacts from that need to be considered.
Language requirement:
English
Size or scale requirements:
The product of this effort needs to be workable for small, medium, and
large scale efforts.
Required deliverables:
Evaluations and recommendation of tools.
Other useful information:
Topic:
4. Standards Compliance
Need:
4.3 Support for assessment of current implementation of NASA
requirements from NPR 7123.1, 7150.2, 7120.5 and STD 8739.8 and
8719.13.
---------------------------------------------------------------
NASA and contractor software and system engineering efforts should
follow NPR 7123.1, 7150.2, and 7120.5 and STD 8739.8 and 8719.13.
There is a need for information that supports decision-making at the
Agency level regarding updates/changes to the standards and
requirements.
Relevant domain(s):
Project(s) that would use the proposed tool/solution:
This effort is focused on supporting the Agency’s upcoming review of
requirements and standards, mining the lessons learned and proposing
possible guidance on usage as well as implementation.
Current tool/solution in use and its shortcomings:
Constraints that might influence the research work plan:
There is an expectation that the proposal would demonstrate how the
team will work together.
Timeline that the proposed tool/solution must support:
Delivery date of ‘final’ report - March 30, 2010
Language requirement:
Size or scale requirements:
Required deliverables:
*
Review & Summary of previous Center Gap Analyses
*
Current Gap Analyses for at least 3 Centers, preferably all of
them
*
A review of the Audit findings and any trends noted in compliance
and non-compliance
*
A report on any barriers to compliance and how they worked around
*
A report on how the NPRs and standards have altered work
activities and product performance as well as an evaluation of the
cost/benefit of the requirements/standards would be expected.
These reports can be combined or separate, preliminary, draft versions
are expected to be discussed and reviewed periodically.
Deliverables should include a planned monthly status exchange.
Other useful information:
Priority will be given to:
*
Proposals that involves collaboration from at least 3 centers,
*
proposals that include both manned and robotics focused centers,
*
Proposals with high FTE ratio (strong and appropriate civil
servant involvement),
*
Proposals whose planned deliverables demonstrate a balance of
qualitative and quantitative results
Topic:
5. Testing
==========
Need:
5.1. Random testing techniques.
-------------------------------
Better techniques for random testing of software (stochastic testing,
with feedback to bias operation & parameter choice); particularly,
both setting up random testing frameworks, and moving from pure random
testing towards verification and complete coverage
Relevant domain(s):
Testing
Project(s) that would use the proposed tool/solution:
File system testing for JPL missions and software projects would use
such an approach. Other modules amenable to automated testing
(possibly IPC, resource arbitration, etc.) would also likely benefit.
Current tool/solution in use and its shortcomings:
Currently, ad hoc one-time solutions are employed, where random
testing is used at all. Some tools exist for Java, but most flight
software (where this would be most critical) is not Java code.
Constraints that might influence the research work plan:
Area is known to be difficult; exhaustive testing is generally
impossible, and for rich properties evaluating "how well" a system has
been tested, or directing testing towards errors is known to be a very
hard problem. Effort of specification and automation for each
application is potentially large.
Timeline that the proposed tool/solution must support:
Mostly long-term, though upcoming missions would benefit before
software integration if possible.
Language requirement:
Tools applying to C/ C++ would be most useful for flight code.
Size or scale requirements:
Most applicable to relatively small, self-contained (10K lines or
less, known API) modules.
Required deliverables:
A framework for random testing would be most important, with a working
prototype being quite desirable. End product would be methodology and
tools.
Other useful information:
Topic:
5. Testing
Need:
5.2. Functional/integration testing tool/frameworks for Flex/Flash
based Rich Internet Applications.
------------------------------------------------------------------
Given the emergence of Web 2.0, developers are pushing the limits of
what browsers can do. The original intent of a web browser was to
deliver documents to end-users, not applications and thus, protocols
and standards to meet this need where designed as such.
Request-Response patterns have moved from full-up page refresh models
to incremental interactions similar to thick-client applications. As
users begin to demand more and more functionality delivered via web
browsers, new challenges are emerging for developers. To add further
complexity, there is a lack of commonality between different browser
vendors and browsers are being used in a manner in which they not
originally intended to do. Because of this, building browser-based
applications with Flex is becoming a popular option for Rich Internet
Application development.
Rich Internet Applications (RIA) are web applications that run in web
browsers but bypass the page refresh model just as AJAX does but
require a Flash runtime. Given the market penetration of the Flash
Player in market-share leading browsers, this is highly available
foundation to build solid RIA, especially in intranet applications
which are commonly deployed within NASA. The benefits of using Flex is
that a developer can write and test code for one platform, the flash
runtime, as opposed to a plethora of browsers/platforms which increase
complexity, implementation time and drive up cost.
As with most software development, testing applications is very
important to ensure software quality and user acceptance. For Flex
based applications, there are tools/frameworks readily available to do
unit testing, but there are limited options for doing integration and
functional testing. For AJAX based RIA applications, openqa.org
released an excellent open source project called Selenium. Selenium
allows QA engineers to test modules written with AJAX technologies.
Given the popularity of the Flex application development, a good open
source product to perform a similar function is lacking.
One COTS product that exists to do functional/application testing is
Mercury QuickTest Pro. This is a valuable tool but very expensive.
Also, this tool only works in Internet Explorer as it is implemented
as an ActiveX plug-in.
Another COTS product is iMacro from iOpus. This is another available
option that is far less expensive than Mercury QTP, but is not as
robust.
Because of the widespread adoption of Flex based RIA development and
the increasing importance of testing for applications, what is needed
is a quality integration/functional testing framework such as Selenium
for Flex RIA that is open-source and not tied to proprietary standards
and protocols.
Relevant domain(s):
Any Flex/Flash based RIA development effort within the agency.
Potentially the solution could address testing of Java and ActiveX
applets as well but this is not as critical.
Project(s) that would use the proposed tool/solution:
Any Constellation project doing Flex web application development.
Currently, there are efforts underway within Constellation that are
using Flex RIA approaches.
Current tool/solution in use and its shortcomings:
Available tools to test Flash/Flex based apps are COTS, and tend to be
very expensive such as Mercury QuickTest Pro. Lesser expensive tools,
such as iMacros tend to use non-robust techniques such as Image
Recognition and XY coordinates to locate GUI elements. Also, available
tools tend to be proprietary.
Constraints that might influence the research work plan:
Timeline that the proposed tool/solution must support:
None
Language requirement:
Flash, Flex, AIR
Size or scale requirements:
Typically small to medium sized client applications
Required deliverables:
A toolkit/framework similar to Selenium for recording macros to
perform functional testing of Flex/Flash based applications
Other useful information:
Topic:
5. Testing
Need:
5.3. Test coverage metrics
--------------------------
There is a need for better test coverage metrics, and guidance on
their application to NASA software.
Test coverage metrics play a prominent role in V&V and certification
of safety-critical software outside of NASA. The purpose of coverage
metrics is to indicate when sufficient testing has been performed and
where additional testing is needed. Currently, commercial avionics
software is developed and tested to the DO178B standard. This standard
mandates requirements-driven testing with MCDC test coverage for the
highest-criticality applications.
NASA needs test coverage metrics appropriate to its safety- and
mission-critical software. It is unlikely that the MCDC coverage
metric by itself is appropriate for NASA applications, for several
reasons that are discussed below. Therefore the need is twofold: a
better test coverage metric or metrics and guidance on their
application.
Relevant domain(s):
High assurance (Class A and B) software.
Project(s) that would use the proposed tool/solution:
Constellation and NASA safety- and mission-critical software. For
example, the Ares I Abort Fault Detection, Notification and Response
(AFDNR) Software system. Results could also influence the commercial
aviation standard.
Current tool/solution in use and its shortcomings:
There is no NASA standard or detailed guidance on test coverage
metrics comparable to that provided in the commercial aviation world
by DO178B and related documents. For example, the Constellation
Software Verification and Validation Plan lists several standard
coverage metrics but does not provide guidance in their application.
The DO178B coverage metric for Class A software, MCDC, is unlikely to
be appropriate for NASA applications. The amount of testing required
to attain the coverage mandated by this metric is onerously expensive,
and furthermore the survey by Kelly Hayhurst et al. showed widely
varying levels of satisfaction with its effectiveness in revealing
software bugs.
There is a growing understanding of the inadequacies of the MCDC
coverage metric itself. Like any metric, it is vulnerable to
deliberate approaches to thwart its intent (notably by designing the
program structure so as to minimize the testing required to attain the
level of coverage, but at increased risk of masking latent bugs). The
FAA provides guidance to mitigate this problem. More worrisome are
recently reported results showing that adoption of well-intentioned
program structures can also lead to this same phenomenon. Furthermore,
MCDC does not address coverage issues specific to reactive systems;
and it is unknown how it should be extended to other non-standard
forms of software (e.g., to model-based reasoning systems, in which
there is growing interest in Constellation and NASA).
Constraints that might influence the research work plan:
To the extent that the work plan included an effort to experiment with
proposed metrics on past NASA software developments, and especially if
there was a desire to retroactively estimate how adherence to the
proposed metric would compare with actual past practices, then the
following constraint would apply: Difficulty in obtaining historical
data including test plans and results, in order to evaluate the
effectiveness of proposed metrics.
The diversity of development processes, development platforms, and
software architectures will likely preclude the universal adequacy of
a single coverage metric.
Unconstrained development methods can defeat any coverage metric.
Thus, we expect that coverage metrics will impose constraints on
development methods.
Timeline that the proposed tool/solution must support:
For example, Ares AFDNR; related ground-based diagnostics systems
(ISHM).
Language requirement:
Generally no specific language requirement. The solutions (coverage
metrics and guidance on their application) should be language-neutral,
although specific tools for implementing the coverage metrics will be
language-specific. There may emerge a need for different metrics
depending on the class of language – e.g., graphical languages and
their accompanying code generation may demand a different coverage
metric to that required by traditional programming languages;
model-based reasoning systems may also represent a distinct class with
distinct metric needs.
Ideally, coverage metrics should also be able to take into account the
results of other verification techniques, such as static analysis, so
as to avoid duplication of effort where coverage has already been
demonstrated.
Size or scale requirements:
The coverage metrics must be applicable to real NASA or FAA
applications. The concern is that in the area of testing, coverage
metrics that demonstrate excellent results on “toy” or small-scale
applications may not scale to real applications. Thus there is a need
to demonstrate applicability to applications of at least 10KSLOC in
size, preferably more.
Required deliverables:
Clear, English-language definitions of new test coverage metrics;
Discussion of their range of applicability;
Justification of their use in place of, or together with, existing
coverage metrics;
Specific guidance on how to apply them--for example, through a
tutorial;
Discussion of development techniques that enable application of these
metrics; and factors that can defeat them;
Indication of tools, technologies and procedures that can implement
them;
What to do in the absence of a comprehensive tool solution.
Ultimately we will need tools that implement the coverage metrics for
C, C++ and any other languages expected to be used for developing
safety- and mission-critical software. However, we recognize that
these products may be more appropriate for commercial development
following delivery of the needed research results.
Other useful information:
Specific guidance on how to apply them--for example, through a
tutorial.
Topic:
6. Reliability estimation
=========================
Need:
6.1. Software reliability metrics and tool
------------------------------------------
Estimating reliability is extremely difficult. There isn't time to run
all the possible tests -- in many cases all inclusive testing would
take longer than the anticipated software lifetime. We need a way to
identify key tests. Simulating an operational scenario sometimes isn't
good enough, but it's the best we can do prior to a full-up more
expensive test round. Customers don't want to pay for the full-up
live-data operation tests to prove reliability; they want to depend on
the less expensive, faster simulation tests. The question is how to
prove a simulation of an operational environment is good enough. For
control center software, one technique is to record all live inputs
for control center systems during operations, so new systems using the
same data will have test cases. The recorded data includes good data
as well as dropouts, bad data points, and all the imperfections we
want to test. Sometimes the problem isn't in the test data but in the
way the system is used under operational conditions. The system may be
left idle for long periods or may have to reset and start over after
running multiple verification tests. Different operators may select
options faster/slower or in an order not previously tested. We need a
way to identify a test set that covers significant variations on
operator interactions. How do we identify and test variations in the
way people operate/setup/command a software system?
A tool to analyze a software architecture to develop a claims
-evidence diagram would be helpful. Everyone seems to want to see a
list of the project-specific evidence/artifacts required to say
assurance is complete. The lowest level of the evidence diagram should
be a set of required artifacts. The diagram provides a scope of
effort. This information would also help assurance personnel explain
what projects are buying and why.
Currently IEE-982 identifies a shopping list of hundreds of
reliability metrics – what is needed is a core set.
Relevant domain(s):
Software Assurance and Software Engineering
Project(s) that would use the proposed tool/solution:
Constellation (Orion and Ares)
With each new safety-critical or mission-critical software release,
reliability must be proven. NASA projects at multiple centers are
trying to better estimate reliability and better define test sets.
Software assurance personnel would likely make use of software
assurance claims-evidence diagrams throughout the project life cycle,
as a road map.
Current tool/solution in use and its shortcomings:
Running the systems for hours to calculate a Mean Time Between
Failures did not give a true indication of software performance.
Running a set of mission scenarios in a simulation environment also
failed to completely replicate an operational state. Tools to document
process have been tried at a high level, but they don't seem to
capture all the details. An automated or partially automated method of
checking software for possible combinations of operator interactions
could help if the possible interactions can be narrowed to a testable
set or if the testing can be automated.
We're lacking a claims-evidence diagram template developed to comply
with NASA standards.
Constraints that might influence the research work plan:
Creating an automated tool is dependent on first identifying a concept
and design. Near-term work should focus on a plan as the primary
delivery.
All the needed information should be available to create a
claims-evidence template.
Timeline that the proposed tool/solution must support:
Language requirement:
For any interactive tools (the assurance level tool or the claims
evidence diagram) a standard desktop application is needed.
An automated tool for defining test cases would most likely first
target C/C++ applications.
Size or scale requirements:
Both the assurance level and claims-evidence diagrams should address
the needs of small to large projects. The automated tool should target
software subsystems that allow operator interaction, taking into
account the effect on the rest of the software system.
Required deliverables:
A well-researched and tested method of identifying crucial variations
in operator interactions should be documented as the delivered
product. An automated tool for C/C++ code would be excellent (but not
expected).
A claims-evidence diagram template compliant with NASA standards that
can be tailored would be the delivered product. An interactive version
that facilitates project-specific modifications would be excellent
(but not expected).
Other useful information:
The identification of tests is not a small effort. Months of research
will most likely be required to come up with a workable plan.
Diagramming necessary components of the software assurance process
based on NASA requirements, standards, and guidelines will probably
require several weeks of effort. Producing example templates for
various projects will require significantly more effort, but will
provide a way to debug the template.
Topic:
7. Maintenance project assurance tools& techniques
==================================================
Need:
7.1. Tools & techniques for software maintenance
------------------------------------------------
Innovative tools and techniques for software maintenance. For example,
ways to easily determine the different modules of a system that will
be impacted by a proposed requirement, design, or source code change
and the extent of the impacts to those modules. Also related, ways to
easily determine the minimum selection of test cases and other
assurance techniques that need executed to have assurance for a system
modification. The tools and techniques developed need to integrate
with, extend or functionally replace developer tools and techniques
for requirements, design, code, and test artifact management since
maintenance personnel are going to use developer provided artifacts as
a starting point of the maintenance effort.
Relevant domain(s):
Deployed systems under maintenance--ground systems, software driven
satellite systems, etc.
Project(s) that would use the proposed tool/solution:
Any project undergoing maintenance, which includes any multi-year
satellite mission, human space flight, etc.
Current tool/solution in use and its shortcomings:
Current methods primarily use configuration control boards to review
changes and manual effort on the part of the maintainer to search for,
assess, and update relevant artifacts. Regression test efforts
typically involve a core regression test suite or a rerun of all
system tests. Automation of the manual processes and reduction or fine
tuning of the assurance (mostly test) activities is needed.
Constraints that might influence the research work plan:
The effort should make sure it minimizes human effort and error while
maintaining project artifacts for continued human comprehension and
use in future projects.
Timeline that the proposed tool/solution must support:
Language requirement:
Size or scale requirements:
Required deliverables:
The end product is likely a combination of tools and processes for
increasing assurance of systems under maintenance while decreasing
human effort for such systems.
Other useful information:
Topic:
8. Generic
==========
Need:
8.1. Rapid prototyping
----------------------
Rapid prototyping of assurance technologies to increase the
productivity of software development and case studies of the
application of assurance techniques and tools to detect defects early
in the life cycle.
Relevant domain(s):
Mission flight software assurance
Project(s) that would use the proposed tool/solution:
Multi-mission System Architectural Platform (MSAP)
Current tool/solution in use and its shortcomings:
Constraints that might influence the research work plan:
Timeline that the proposed tool/solution must support:
Language requirement:
Size or scale requirements:
Required deliverables:
Working prototype and "how-to" guidebooks
Other useful information:
Clearly defining why this research will help NASA to manage the
complexity of system design
Topic:
8. Generic
Need:
8.2. Delphi Knowledge Elicitation Method w/ Check List Analysis
---------------------------------------------------------------
Continue development into a verification tool. A Delphi knowledge
elicitation method was developed within NASA. The method requests
input from a group of domain experts. It has been applied on a limited
number of topics including batteries, valves and electric circuits.
Results can be used to generate checklists which engineers can use to
assist in identifying possible hazards.
Relevant domain(s):
Any engineering discipline
Project(s) that would use the proposed tool/solution:
Current tool/solution in use and its shortcomings:
Constraints that might influence the research work plan:
Timeline that the proposed tool/solution must support:
Language requirement:
Size or scale requirements:
Required deliverables:
Other useful information:
Topic:
8. Generic
Need:
8.3. Fault/Failure Tolerance Analysis
-------------------------------------
Fault Tolerance analysis is routinely done as part of safety analysis
but a systematic method of performing it and tools to implement the
method need to be developed. Internally developed process flow at Ames
has significantly assisted in identification of hazards/issues.
Refining this process flow method into a software tool that generates
usable data and generate reports that doesn’t need modification is
needed.
Relevant domain(s):
Fault/failure identification and analysis
Project(s) that would use the proposed tool/solution:
Current tool/solution in use and its shortcomings:
Constraints that might influence the research work plan:
Timeline that the proposed tool/solution must support:
Language requirement:
Size or scale requirements:
Required deliverables:
Other useful information:
Topic:
9. Autonomous Failure Detection
===============================
Need:
9.1. Autonomous Failure Detection, Isolation and Recovery/Integrated
systems health monitoring tools
--------------------------------------------------------------------
Autonomous Failure Detection, Isolation and Recovery/ Integrated
systems health monitoring tools
Relevant domain(s):
Project(s) that would use the proposed tool/solution:
Current tool/solution in use and its shortcomings:
Constraints that might influence the research work plan:
Timeline that the proposed tool/solution must support:
Language requirement:
Size or scale requirements:
Required deliverables:
Other useful information:
Topic:
10. Complex electronics
=======================
Need:
10.1. NASA standard on complex electronics
------------------------------------------
NASA Standard on Complex Electronics Development and Implementation
requirements for engineering and assurance.
Relevant domain(s):
SMA organization, Software Engineers and project manager
Project(s) that would use the proposed tool/solution:
The current Constellation program includes its projects and
sub-elements (Ex. Ares/CLV/CEV etc…)
Current tool/solution in use and its shortcomings:
Constraints that might influence the research work plan:
Timeline that the proposed tool/solution must support:
Language requirement:
Size or scale requirements:
Required deliverables:
Other useful information:
Topic:
10. Complex electronics
Need:
10.2. VV&A of complex electronics
---------------------------------
Including commercial software, including embedded software and
development environments; System on a Chip.
Relevant domain(s):
Project(s) that would use the proposed tool/solution:
Low Impact Docking System is currently using CE software assurance
research products. Other projects likely to use them include ISS to
CEV Communications Adapter (ICCA).
Current tool/solution in use and its shortcomings:
Constraints that might influence the research work plan:
Timeline that the proposed tool/solution must support:
Language requirement:
Size or scale requirements:
Required deliverables:
Other useful information:
Topic:
10. Complex electronics
Need:
10.3. Reconfigurable computing assurance
----------------------------------------
Reconfigurable computing is the upcoming trend. Research needs to be
done on what the best way is to assure the safety and quality of these
devices.
Relevant domain(s):
SMA organization, Software Engineers and project manager
Project(s) that would use the proposed tool/solution:
The current Constellation program includes its projects and
sub-elements (Ex. Ares/CLV/CEV etc…)
Current tool/solution in use and its shortcomings:
Constraints that might influence the research work plan:
Timeline that the proposed tool/solution must support:
Language requirement:
Size or scale requirements:
Required deliverables:
Other useful information:
Topic:
10. Complex electronics
Need:
10.4. Methodology for moving complex electronics from class D to class
A
----------------------------------------------------------------------
Much of the software and complex electronics is being developed as
class “D” for Constellation. A defined methodology must be created for
moving this “software” from class “D” to class “A”.
Relevant domain(s):
SMA organization, Software Engineers and project manager
Project(s) that would use the proposed tool/solution:
The current Constellation program includes its projects and
sub-elements (Ex. Ares/CLV/CEV etc…)
Current tool/solution in use and its shortcomings:
Constraints that might influence the research work plan:
Timeline that the proposed tool/solution must support:
Language requirement:
Size or scale requirements:
Required deliverables:
Other useful information:
Topic:
11. Metrics
===========
Need:
11.1. Reliability metrics
-------------------------
See 6.1. A core set of software reliability metrics
Relevant domain(s):
Project(s) that would use the proposed tool/solution:
Current tool/solution in use and its shortcomings:
Constraints that might influence the research work plan:
Timeline that the proposed tool/solution must support:
Language requirement:
Size or scale requirements:
Required deliverables:
Other useful information:
Topic:
11. Metrics
Need:
11.2. Test coverage metrics
---------------------------
See 5.3 Test coverage metrics
Relevant domain(s):
Project(s) that would use the proposed tool/solution:
Current tool/solution in use and its shortcomings:
Constraints that might influence the research work plan:
Timeline that the proposed tool/solution must support:
Language requirement:
Size or scale requirements:
Required deliverables:
Other useful information:
Topic:
11. Metrics
Need:
11.3. Metrics for complex electronics development
-------------------------------------------------
Develop metrics to measure the quality of complex electronic devices
as they are being developed.
Relevant domain(s):
SMA organization, Software Engineers and project manager
Project(s) that would use the proposed tool/solution:
The current Constellation program includes its projects and
sub-elements (Ex. Ares/CLV/CEV etc…)
Current tool/solution in use and its shortcomings:
Currently there are no standard metrics available.
Constraints that might influence the research work plan:
Timeline that the proposed tool/solution must support:
Short term need is for the high volume of contractor’s deliverable
documents. Long term is for Constellation operation.
Language requirement:
Size or scale requirements:
Required deliverables:
Other useful information:
Topic:
11. Metrics
Need:
11.4. UML quality metrics
-------------------------
See 3.2 Interoperability of frameworks and models, 3.3. Model-based
engineering, 3.4. State analysis and 3.5 VV&A of models and
simulations
Relevant domain(s):
Project(s) that would use the proposed tool/solution:
Current tool/solution in use and its shortcomings:
Constraints that might influence the research work plan:
Timeline that the proposed tool/solution must support:
Language requirement:
Size or scale requirements:
Required deliverables:
Other useful information:
Topic:
11 Metrics
Need:
11.5. Tool for software and software assurance metrics
------------------------------------------------------
See 12.2 Tool for process evaluation
Relevant domain(s):
Project(s) that would use the proposed tool/solution:
Current tool/solution in use and its shortcomings:
Constraints that might influence the research work plan:
Timeline that the proposed tool/solution must support:
Language requirement:
Size or scale requirements:
Required deliverables:
Other useful information:
Topic:
12. Process Improvement
=======================
Need:
12.1. CMMI in the small.
------------------------
Relevant domain(s):
Project(s) that would use the proposed tool/solution:
Current tool/solution in use and its shortcomings:
Constraints that might influence the research work plan:
Timeline that the proposed tool/solution must support:
Language requirement:
Size or scale requirements:
Required deliverables:
Other useful information:
Topic:
12. Process Improvement
Need:
12.2. Tool for process evaluation
---------------------------------
Tool for process evaluation based on set of criteria from CMMI or
other process improvement practices. The tool should also provide
method for data management allowing developers to easily record data
during development life cycles.
Relevant domain(s):
SMA organization, Software Engineers, Project Manager, software
community who employs CMMI.
Project(s) that would use the proposed tool/solution:
The current Constellation program includes its projects and
sub-elements (Ex. Ares/CLV/CEV etc…)
Current tool/solution in use and its shortcomings:
Constraints that might influence the research work plan:
Timeline that the proposed tool/solution must support:
Short term need is for the Constellation Program.
Language requirement:
Size or scale requirements:
Required deliverables:
Other useful information: