| |
Review of the All-H Analyzer (AHA)
February 4, 2005 | document ISRP/ISAB 2005-5
Background
At the Northwest Power and Conservation Council's January 21st,
2005 request, the ISRP and ISAB reviewed the "All-H Analyzer" (AHA)
model that Council staff is proposing to use as part of a larger exercise
in the Fish and Wildlife Program to establish draft numerical objectives
for anadromous fishes, including natural returns, hatchery escapement, and
harvest at the subbasin, province, and basin levels.
Subbasin Plans were recently found to be deficient in the integration
of hatchery and natural production with habitat, hydro, and harvest goals
for anadromous fishes.[1]
A goal of the use of the AHA model in planning is to facilitate
consideration of the balance of hatchery and natural production in
relation to habitat actions, out-of-subbasin harvest, and hydrosystem
constraints.
The Council asked five general and three supplemental review questions
concerning the AHA. To help the ISRP and ISAB answer these questions, lead
scientists from Puget Sound's Hatchery Scientific Review Group (HSRG)
presented the model to us, explained how the model was used in Puget Sound
for informing hatchery management, and described the development of the
fitness equation in the model. Although appreciated and informative, the
presentation raised many unanswered questions about the model's
structure and the mechanics of its use. In addition to the presentation,
the AHA model itself was made available, and several reviewers explored
its behavior. The ISRP&AB also reviewed the available background
documentation pertaining to the model, including HSRG technical papers and
some of the scientific literature that the fitness equation is based on,
in particular a recent paper by ISAB ex-officio member Mike Ford (Ford.
2000. Cons. Biol. 16: 815-825). It is important to note, however, that the
AHA model is still in development and much of the background material
needed for review and application is NOT available. In other words, this
ISRP/ISAB ?review? of the AHA is essentially a review of the intentions
or potential in the development and projected use of the model and not
a review of the reliability or accuracy of the model itself.
Before addressing the questions, the ISRP&AB emphasize that there
is a clear need for quantitative analysis, including disciplined use of
analytical and exploratory modeling, to improve fish and wildlife
management in the Columbia River Basin, particularly the integration of
natural and hatchery production with habitat actions at the subbasin level.
We strongly agree with the Council staff's observation that a major
problem within the Columbia Basin is the lack of clearly articulated
objectives integrated across the four Hs at the subbasin, province, and
basin levels. Without these objectives, it is difficult to prioritize
project implementation and monitoring activities. A key to developing
these objectives is a comprehensive, integrated analysis of habitat,
hatchery, hydrosystem and harvest actions. In our review of subbasin
plans, we specifically noted that most plans did not adequately describe
how hatchery programs would be integrated with existing natural
production, habitat improvements, and future rehabilitation activities.
Many plans also lacked stated measurable objectives for natural returns,
hatchery escapement, and harvest. An attractive feature of the AHA model
is that it is one of the first software products developed explicitly for
planning purposes in the region that attempts to incorporate some
representation of a relationship between hatchery and natural production
and fitness of the wild stock.
The vocabulary referring to the AHA model as a ?tool? (see the
wording of Question 4 specifically) merits special attention, because of
the unrealistic expectations that this labeling might convey. This
language derives from ?decision support tools,? developed generally as
?expert systems.? These are artificial intelligence products,
intended to capture existing hard scientific information and expert
opinion in a technical area and make these accessible, via a ?user
friendly? interface, to users who themselves are not expert in that
technical area. In practice, the goal of providing an interface that is
truly easy to use is seldom attained, so a cadre of technicians who are
adept with the interface often are needed to serve as facilitators between
the software product and its intended audience. But even so, the actual
deployment of the software in real applications is generally in the hands
of a user team that is not expert in the underlying scientific area
that is supposed to be supplied by the software itself. The quality of the
predictions (or recommendations) generated by the model will depend on:
the quality and quantity of the real data (measurements) input to it by
the user, the accuracy of the user-supplied input ?guesstimates?
(numbers not based formally on statistical evaluation of actual data), and
the hard scientific information and expert opinion encapsulated in the
software.
In the present instance ? predicting salmon productivity in the
presence of wild/hatchery interactions ? the available library of actual
data is very sparse, case-specific data that users can supply also will be
very sparse, the predictive power of the available pertinent hard science
is limited by data gaps, and there is not a consensus of expert opinion.
The ISAB has reviewed this state of the science in our supplementation
review.[2] For these
reasons, an expert system predicting salmon productivity in the presence
of wild/hatchery interactions will have unpredictable performance, and
probably low reliability. At best, such a system will offer a useful way
to organize assumptions and quantify the implications of adopting those
assumptions, provided the tool carefully documents its own assumptions,
as well as the user-supplied inputs, along with its output of predictions.
If the tool is adequately documented, and adequately documents inputs as
part of its output, the proper use of the tool is to generate
hypotheses that should be tested, rather than accepted at face
value. The ISRP and ISAB have explained this perspective in comments
on another expert system, EDT, during the course of the model synthesis
review.[3] The real
substantive benefits from use of a properly documented tool to generate
hypotheses will not accrue until the results are obtained from experiments
that are correctly designed to deliver the needed resolution. The design
and implementation of experiments of this sort, and on this scale, have
historically proven to be a very difficult challenge within the Columbia
River Basin. The ISAB has already commented, in its supplementation
review, on the insufficiency of the existing record of experimentation
with supplementation, despite the widespread institutional acknowledgement
that supplementation is experimental.
Review Summary
The AHA model attempts to provide a convenient interface to an
underlying model for evaluating the ?expected? consequences of a
hatchery program, given certain assumptions and quantitative relationships
(e.g., habitat capacity), but, unfortunately, those assumptions and
equations are incompletely documented. Also, some reasonable scoping of
the model's sensitivities is needed.
The ISRP and ISAB encourage the Council and region to pursue the effort
to set objectives and integrate across the Hs, but caution against using
the AHA model or any other unvalidated single model to generate specific
objectives, numerical or otherwise, or to propose recovery goals for
anadromous fish. As we understand the AHA model, it was designed to
evaluate how well a particular hatchery program in Puget Sound or Coastal
Washington, conforms to the HSRG's "integrated program"
criteria. It is essentially a deterministic model that provides an
estimate of the expected long-term equilibrium numbers and ratios of
hatchery and wild fish, given a certain set of incompletely documented
assumptions. It would be inappropriate to use this model to decide what
natural and hatchery production objectives and recovery criteria should be
in the first place. It is not a good idea to encourage people to apply an
undocumented model, where details of the structure or operations of the
model remain hidden from the user. Acceptance of the meaning of an unknown
process is more an act of faith than a sound application of science.
Recommendation. The AHA model should not be used to aid in the
development of draft numerical objectives for anadromous fishes,
including natural returns, hatchery escapement, and harvest at the
subbasin, province, and Columbia Basin levels until it is properly
documented and validated in a substantive review.
In terms of an overall approach to setting integrated objectives and
establishing priorities, the ISRP and ISAB recommend that two or more
modeling approaches be developed, and that a serious effort be dedicated
to validating the respective models against real data, diagnosing and
reconciling whatever differences emerge between models, and conducting
deliberate and rigorous experiments to resolve empirically the
uncertainties about parameter values or model form that prove to be
important to the predictions. We understand that the SHIRAZ model,
developed at the University of Washington, has similar objectives to the
AHA model as an integrated tool and was used in the Snohomish River Basin
for regional recovery planning. In the absence of a review of that model,
we do not know whether or not it is fully enough developed to allow
immediate application in the Columbia Basin.
Recommendation. Two or more models should be explored in the
process of developing tools for evaluation of numerical objectives for
anadromous fishes in relation to natural returns, hatchery escapement,
and harvest at the subbasin, province, and Columbia Basin levels.
There are some inherent differences between ecosystems in Puget Sound
and Coastal Washington and ecosystems in the Columbia River Basin. Some
additional components (e.g., more information about the hydrosystem) need
to be added to the AHA model before it is fully applicable in the Columbia
River Basin. Adult anadromous fishes need to be allowed to return over
multiple years, as do chinook and steelhead. The model needs to
incorporate different routes of passage and survival parameters through
the Columbia River Basin hydropower system. Allowance should be made for
variation in additional critical input parameters, such as productivity,
capacity, and harvest rate. Assigning single values to input parameters
fails to account for both uncertainty about the parameter and natural
variation in the parameter.
Recommendation. The AHA model should be tailored to meet unique
ecosystem conditions in the Columbia Basin.
Answers to Council Questions
Question 1) Does AHA provide a useful approach to developing
regional objectives by integrating hatchery programs and harvest goals
with subbasin plans?
It is a little misleading to call this model in its current form an
"All-4H analyzer". The hydro part is essentially missing.
Different routes of passage and survival parameters through the
hydrosystem (e.g., transportation or inriver with and without spill) are
not included. The habitat part consists only of a pair of capacity and
productivity parameters. Values for these must be generated outside of the
model. When applying AHA in a "4H" context, we assume there must
be some combination of use of AHA and EDT, for example.
The AHA model is intended to generate ranges and means for the numbers
of anadromous fish released from hatcheries, of returning natural fish, of
hatchery fish, and of harvested fish under constraints imposed by
limitations on the genetic interaction of hatchery and natural fish. The
ISRP and ISAB do not believe these numbers should be used as explicit
objectives for subbasin or province plans. When effects of the missing ?H,?
i.e., hydro are added, the basic fields (productivity, capacity, hatchery
production, harvest, fitness, etc.) needed to facilitate useful discussion
of all-H integration will be present in the tool, but the reliability of
the model predictions is unknown. Specifically, we cannot be confident in
the reliability of the numbers generated by this ?beta-test? version
of AHA, and we are uncertain about the sensitivity of AHA to assumptions
made in choosing values of its numerous input variables.
Some subbasin plans already specify numerical objectives for adult fish
returns. Once the AHA model is validated and its output compared with that
of other models, it may prove useful as an exploratory or discussion tool
that might lead to refinement or reassessment of fisheries objectives for
hatchery and naturally produced anadromous fish at the subbasin level.
More importantly, the model also could be used to suggest critical
experiments, which, if they were conducted properly, would quantify the
model's reliability (and possibly lead to improvements in modeling
capability and contribute to our scientific understanding of the factors
controlling salmon productivity and role of hatchery/wild interactions).
Questions 2 and 3) What cautions or caveats would you give for
interpreting the outputs of the AHA exercise? Do you have suggestions
for improving the approach?
The model is still in development and some components that would be
needed to effectively use and validate the tool in the Columbia River
Basin are not yet incorporated. The ISRP&AB offer several suggestions
to improve the AHA model. Documentation of the equations, and the data
which serve as a basis for these equations, is needed to properly review
the model. In addition, reviewers would need to meet with the modelers to
discuss various results from several pilot model runs.
Based on our immediate impressions, it should be possible to
incorporate the issues in the next three bullets in the AHA model
relatively quickly:
- Analyze species whose adults return over multiple years, e.g.,
chinook. Currently AHA is a discrete generation model. Most hatcheries
culture chinook, and it's not clear what effect this simplification
in AHA has on its output for chinook.
- Incorporate different routes and survival parameters through the
hydrosystem.
- Allow for variation in additional critical input parameters, such as
productivity, capacity, and harvest rate. Although the current version
of the model allows for variation in ocean survival based on a single
published dataset, other critical parameters are only considered as
fixed constants. Assigning single values to input parameters fails to
account for both uncertainty about the parameter and natural variation
in the parameter.
One possible improvement in the
model would be to allow the option of inputting parameters as the mean
and variance of an assumed distribution, with random selection of
parameter values for simulating natural variation in the system. For
example the model treats habitat productivity as a constant, whereas
it is known that productivity varies from year to year as the
environment varies, e.g., with climate events such as floods and
droughts. Using long-term habitat averages in the model exacerbates
the risk of missing the effect of short-term natural disturbances,
which could result in the undesirable situation of adding too many
hatchery fish when natural production is low. Harvest rates are not
likely to be constant either. The implications of this shortcoming are
significant. For example, we could use the AHA to create a scenario
employing reasonable estimates of capacity and productivity, in which
a small population would be stable indefinitely. If this abundance,
capacity, and productivity were employed in a Population Viability
Analysis model, however, the population would very likely go extinct
because of random variation in the demographics of small
populations.
The AHA model is undergoing a natural development from consideration of
a few relatively simple concepts to incorporation of more options to mimic
the complex reality of multiple natural systems. In the longer term, as
the model matures, it would be useful to develop the methods and ability
to:
- Incorporate the interaction of multiple species and their hatchery
programs within a subbasin or with straying from other subbasins.
Currently, the AHA model does not consider interactions among species
or the effect of hatchery fish straying from one subbasin to another.
Apparently, the HSRG encountered this difficulty in the Puget Sound
effort and developed principles to guide analyses on a case-by-case
basis.
- Aggregate objectives from subbasins to provinces and to the Columbia
River Basin as a whole.
- Incorporate metapopulation genetics that consider straying among
multiple populations that may differ in productivity and abundance.
Straying not only provides gene flow but also can have important
demographic impacts on component populations.
- Consider fitness changes due to human and natural causes, in
addition to incorporating variation in demographic parameters
discussed above. Fitness of hatchery and naturally spawning stocks is
assumed to be different, but constant for each type of population.
Human actions such as size selective harvest and habitat loss can
alter fitness of both hatchery and naturally spawning populations.
- Conduct a sensitivity analysis that provides documentation of
confidence in the data and parameter values used in the model. The
data requirements of the model are substantial. Most of the data
required by the model for specific populations may not be available at
this time. We are concerned that objectives and protocols would be
established based largely on multiple speculative parameter estimates
that are not grounded in concrete empirical data. This practice has
resulted in serious criticisms of EDT within the region. How
representative are the data used? Which parameters are the most
reliable, which most speculative, how does this uncertainty play out?
What are the confidence intervals of parameters input to and of
outputs from the model? For example, EDT outputs may be used for
estimates of productivity and capacity in the AHA model. During our
subbasin plan review, it was very apparent that the quality of and
subsequent confidence in EDT outputs were significantly affected by
the source and quality of the data available, the expertise of the
modelers, and the time and resources available to do the analysis. One
suggestion for capturing the users? confidence in data inputs would
be to add a sensitivity component to input fields that would shade the
field according to strength or degree of belief; e.g., red means
general opinion, yellow means data from nearby watersheds were used,
green means the model used empirical data from the area in
question.
- Conduct an analysis of the sensitivity of model results to various
input parameters; i.e., which input parameters are most influential
and which have little effect on model predictions.
- Test model results by comparison with real data for actual streams.
Can the AHA model replicate conditions close to where we are today?
Could alternative models fit the actual data equally well? Would these
alternative models lead to different predictions from the AHA if
applied to other scenarios after this common calibration? It is
our understanding that the HSRG carried out a calibration exploration
with the AHA model in their application, but the presentations and
materials we were provided did not include these exercises.
- Add a Ricker curve to the model to compare with the Beverton-Holt
curves currently used in the model, because Ricker curves include
certain density-dependent functions not present in the Beverton-Holt
curves.
A subset of ISRP&AB members spent some time exploring the model and
offer their observations on its use:
- The model can produce irrational results with certain combinations
of input parameters. In one use, the model generated cells with
"X" divided by zero, an "undefined" number. In the
summary output, this was recorded as years with large returns of fish
occurring a negative proportion of years.
- Using the model in its present form almost certainly would require
that one of its programmers be present. Although the simplicity of the
Excel spreadsheet approach permits easy access to using the program,
questions arise about the functions that underlie many of the cells
that can be manipulated by a user. The model would need to be used
with caution, and by individuals knowledgeable about the computation
of cell values. Regardless of how the model is used, we recommend that
the model equations be fully documented and provided to the user.
- Under the natural component tab, it appears that the egg-to-smolt
survival is adjusted by a formula as a function of the smolt-to-adult
survival to accommodate a fixed productivity that was set on the
population tab. There will be situations where the egg-to-smolt and
smolt-to-adult survivals will be positively correlated, i.e., one will
not compensate for the other. For example, in drought years, poor
conditions can exist for egg-to-smolt survival and for downstream
migration.
- There are numerous cells in the AHA model into which users can enter
numbers that don't seem to affect the output of the model. As an
example, in the natural and hatchery component tabs, changing the
number in the initial population cell doesn't seem to have an effect
on outputs on the population tab.
- Toggling the fitness function on and off did not change the output
very much. But altering the fitness function itself had a large effect
on the output. Additional time would be required to fully explore and
understand how these features lead to the observed changes in the
models predictions.
Question 4) Does the AHA model provide a useful approach to
incorporating assumptions about fitness loss?
With the model still in a ?beta? developmental state, and
undocumented, the ISRP and ISAB reviewers could not examine the
assumptions in detail with the presenters or in the supporting
documentation and literature. The model does not provide an exhaustive
menu of plausible alternatives, and dealing with the uncertainty about
assigned parameters is left to the user. The model user is required to
assign a difference to the optimum phenotype in a hatchery and in nature
and to identify at what life history stage this difference is manifest. At
this time, these assignments are based on professional judgment. There are
other considerations of genetic change, such as Goodman (In Press,
CJFAS) and Lynch and O?Hely (2001; this paper is cited in the model).
There needs to be more explanation of how the fitness equation is employed
throughout the population model; nevertheless, it is a laudable and
important development of the AHA model that fitness impacts of hatchery
fish on wild fish are explicitly considered in a planning effort. Any
model used for this exercise should include a fitness component.
Question 5) Do alternative tools exist that would be better?
We suggest that the Council consider that this should not be an
either/or choice. For numerous reasons we recommend that several
alternative models be developed and used for this exercise. When people
use a single model they may be tempted to place unwarranted faith in the
numerical results, and they may not scrutinize the model itself. Given the
current scientific uncertainty about some of the key relationships
considered by the AHA, it is almost inevitable that another model would
produce different results than the first model, and this would motivate
greater scrutiny of each model's predictions and assumptions. In
general, use of alternative models highlights differences in their outputs
and helps users understand the consequences of each model's predictions
and assumptions.
Models are useful, but practitioners should be aware of the assumptions
inherent in the models. It is dangerous to use models without this
awareness. We recommend that both proponents and skeptics be involved in
modeling exercises, and that the range of views as to veracity of the
results be documented
In terms of alternative tools, we are aware that Dr. Ray Hilborn at the
University of Washington has developed an integrated (multiple Hs) model,
called SHIRAZ, but we haven't studied it. There are, in the literature,
many alternative models that could serve as a basis for alternative tools,
or incorporated as options into a single tool.
Supplemental Questions
Questions 1 and 2: Did the initial applications of the AHA
model to the Yakima and Kalama Subbasins provide measurable objectives
for natural returns, hatchery escapement and harvest needs? Are these
measurable objectives scientifically sound? Are they of sufficient
quality to be adopted into the Council's program?
The Kalama and Yakima analyses are in draft form and aren't available
for our review. An evaluation of those analyses, when available, would
likely take more time than allotted for this review. Moreover, other
reviewer concerns identified above about the model would make such an
exercise unproductive at this time.
Based on our brief experience with the AHA model, we recommend that the
output from the AHA not serve as a standalone source for establishing
numerical hatchery production objectives. Output of these numbers is just
a first step in developing objectives, i.e., one element to consider.
There are other scientific considerations that are not currently
incorporated in the model, such as those described earlier. In addition,
we recommend that the Council look at results from at least two
alternative models; e.g., test the AHA model, in conjunction with another
model, on a subset of subbasins such as the Yakima and Kalama to see how
the model outputs compare.
Question 3. Is Mobrand the only firm that can provide this type
and quality of analysis?
See our comments on Question 5 above.
[1] ISRP and ISAB. 2004-13.
Scientific Review of Subbasin Plans for the Columbia River Basin Fish and
Wildlife Program. For the Columbia River Basin Indian Tribes, NOAA
Fisheries, and Northwest Power and Conservation Council. www.nwcouncil.org/library/isrp/isrpisab2004-13.htm
[2] ISAB 2003-3: Review of
Salmon and Steelhead Supplementation. www.nwcouncil.org/library/isab/isab2003-3.htm.
[3] ISAB. 2001. Model
Synthesis Report: An Analysis of Decision Support Tools Used in
Columbia River Basin Salmon Management (ISAB 2001-1). Northwest Power
Planning Council. www.nwcouncil.org/library/isab/isab2001-1.pdf
|
|
home