<<O>>  Difference Topic WhyUseBayesNetsArchival (r1.1 - 30 Nov 2005 - MarkCrowley)
Line: 1 to 1
Added:
>
>
META TOPICPARENT BenefitArchival

Why Use Bayes Nets?

Deployed Bayesian-Nets Systems in Routine Use

For information about deployed Bayesian-Nets Systems in routine use, check out Russ Greiner's page on applications of Bayesian Belief Nets.

Testimonials supporting Bayesian-Net

This page includes specific arguments which support the claim that systems based on Bayesian nets perform better than those based on X, for various technologies X. They correspond to the responses I received to a request for examples of the following situation:

We at CompanyV? needed to do task W. We tried BrandX? (eg, simple Rule-based Expert System, Neural Net, ...), but that didn't work, for reason R (eg, not expressive enough, not explainable, ...). So we switched to Bayes Nets, and this saved us Y millions of dollars ... and everyone lived happily ever after ...

Notes:

  • We are mis-using the term "Bayesian Nets" to refer to any structured system for encoding probabilistic information, including Influence Diagrams, ...
  • Each of the first three examples below correspond to a commercial system; and the final batch is a collection of anecdotes.
  • We are actively seeking other examples of this situation; please send them to
    greiner@scr.siemens.com
  • This list does NOT include technical arguments for using Bayesian nets, nor does it list arbitrary fielded systems that are based on Bayesian nets -- see Russ Greiner's page on applications of Bayesian Belief Nets.

See Bayesian Nets for yet other useful information about Bayesian nets.

  • I have annotated each entry with the source. My comments are indicated by {RG} *1. Pathology (Microsoft/KIC/...) * {David Heckerman (heckerma@microsoft.com)}

In 1983-4, Eric [Horvitz] and I tried to build a rule-based system for lymph-node pathology diagnosis. We managed to do so, but the expert complained that his certainty factors were unreliable. We then built an idiot Bayes system, assessing probabilities in the causal direction. The expert believed his assessments to be much more reliable, it took 1/5th the time to assess the numbers, and the recommendations of the system were much more accurate (Heckerman, 1988). When we moved from idiot Bayes to Bayes nets, we got more improvement in accuracy. The system, called Pathfinder, was commercialized as Intellipath. If we had stuck with a rule-based architecture, Intellipath would not be around today.

 @inproceedings{Heckerman88b, author="D.E. Heckerman", title="An empirical comparison of three inference methods", booktitle="Proceedings of the Fourth Workshop on Uncertainty in Artificial Intelligence, {\rm Minneapolis, MN}", publisher="Association for Uncertainty in Artificial Intelligence, Mountain View, CA", month="August", year=1988, pages="158--169", note="Also in Shachter, R., Levitt, T., Kanal, L., and Lemmer, J., editors, {\em Uncertainty in Artificial Intelligence 4,} pages 283--302. North-Holland, New York, 1990"} 
*2. Generator Expert Monitoring System (SRI/GE) * {John Mark Agosta (johnmark@sri.com)}

We had a project called GEMS, that began as an ES diagnostic system, and was pulled back from the brink of failure by conversion to BBNs. Ari Kornfeld wrote an paper that was published explaining why BBNs succeeded where rules didn't.

{Ari Kornfeld (Ari_Kornfeld@qm.sri.com) pointed to the paper, and continued:}

The bottom line was that rule-based systems simply didn't work for our situation. [...] The article includes an excellent anecdote in a side-box titled "A Happy Expert".

{RG}

 The article is @Article{CausalDiagram-Kornfeld, Author = "Ari Kornfeld", Title = "Causal diagrams: Clarifying Uncertainty", Journal = "AI Expert" Month = "November", year = 1991, Publisher = "Miller Freeman Publications"} 
Its side-box, on page 44, is especially relevant to the topic, as it states that the GEMS domain expert
 "constantly disagreed with our interpretations of his diagnostic descriptions and his manager started hinting they might pull him off the project entirely". 
After the development team switched to a causal diagram (Bayesian net), the new system's representation was sufficient to convert the expert into an enthusiastic champion of the project.

Btw, this prototype now appears to be used by GE; see

 @InProceedings{GEMS-TR, Author = "Mahesh A. Morjaia and F. John Rink and William D. Smith and Geoff Klempner and Clayton Burns and Jan Stein", Title = "Commercialization of {EPRI}'s Generator Expert Monitoring System (GEMS)", Booktitle ="Expert System Application for the Electric Power Industry", Publisher = "EPRI", year = 1993, Address = "Phoenix", Note = "Also: GE techreport GER-3790" } 
That TR suggests that GEMS will be available by the "second half of 1994"; but I haven't been able to track down more recent information.

{Mahesh Morjaria (morjaria@pcmail01.crd.ge.com)}

The key reason (in my opinion) why we have been successful in having our internal GE customers adopt this technology has been the ease of developing the diagnostic system. More specifically, the engineer's ease of expressing the causal knowledge for (electromechanical) types of equipment which can then be effectively used for the purpose of diagnosis.

*3. Mitre * {Chris Elsaesser (chris@azrael.mitre.org)}

 We at Company V needed to do task W. 
We at the MITRE Corporation needed to help the Navy (NSWC-DD) do some realtime weapons scheduling for ship self-defense (multiple, high speed [500+ kts] incoming, multiple hard + softkill (e.g., chaff) weapons of varying effectiveness at various ranges against various weapons. There is uncertainty in target type, and a "Pk" (probability of kill) curve (p wrt time to fire).
 We tried BrandX? (eg, simple Rule-based Expert System, Neural Net, ...), but that didn't work, for reason R (eg, not expressive enough, not explainable, ...). 
The Navy tried integer programming in C, and when there was more than one incoming the Sparc 20 normally worked for about 5 min. before having to be rebooted.

A chap in our Signal Processing center tried dynamic programming on a similar problem (tasking sensors) and got similar abysmal results.

(Surprise! Planning and reasoning under uncertainty are both still intractable!)

 So we switched to Bayes Nets, and this saved us Y millions of dollars ... and everyone lived happily ever after ... 
Scott [Musman] switched to transformational planning and Bayes nets and was able to handle multiple target, multiple weapon problems in under 2 sec. on a Sparc laptop (~Sparc 2) in Lisp with an interface in CLIM.

The Navy said we saved them $$, but part of that was in convincing them not to rehire the contractor who spent $5M of their bucks and got nowhere. The system is under consideration for insertion into the new ship self defense testbed, and for extensions to handle more temporal aspects (including uncertainty in arrival time of the attack).

{Scott Musman (musman@airborne.nrl.navy.mil) added:}

About the only thing worth adding is that there is no magic going on in what we did (are doing..). Belief nets provided an excellent way to encode our knowledge of how we expect certain self defense assets to perform against different threat types under a variety of conditions. Unfortunately evaluating belief networks is still NP hard, [...]

In a nutshell, the real business decision is more a testimonial to AI techniques as opposed to Bayesian Networks. As Chris described, other people applied mathematical optimization techniques to the problem, and somehow forgot that the issue was real-time. Our combined Bayesian/Transformational-Planning approach doesn't gain you a thing in terms of the final quality of solution produced if you decide to search the whole decision tree, but it does guarantee that you can have "a" solution when you need one.

*4. ...Knowledge Acquisition studies I... * {Edward J. Wright (EJWright@smtpgate.read.tasc.com)}

[We tried] to encode the "rules" for some simple photo-geology [within a public domain expert system shell]. But I could never get it to work. Later I found another expert system shell that included certainty factors. I could not get that to work either.

[I recently] used the same problem as a class project for a Bayesian network class. I will emphasize that the system I built is a very simple subset of the photo-geology problem, but it does provide answers that are always reasonable and usually correct. And I spent much less time building the Bayes net model then I did playing with the rule based systems.

I think the problem was too many interrelated variables and no way to deal with uncertain relationships, but this was no challenge for the Bayesian net.

*5. ...Knowledge Acquisition studies II... * {Eric Horvitz (horvitz@microsoft.com)}

[...] thought you might find reading this interesting, given your recent question on rationale for using influence diagrams and belief networks.

 @inproceedings{Henrion87a ,key="Henrion" ,author="Henrion, M. and Cooley, D.R." ,title="An experimental comparison of knowledge engineering for expert systems and for decision analysis" ,booktitle="Proceedings AAAI-87 Sixth National Conference on Artificial Intelligence, Seattle, WA" ,publisher="Morgan Kaufmann, San Mateo, CA" ,pages="471--476" ,month="July", year="1987"} 
[and] a discussion of efficiency of knowledge acquisition and modularity and its implications for maintaining knowledge bases in:
 @Article{Horvitz88b, Author="E.J. Horvitz and J.S. Breese and M. Henrion", Title="Decision Theory in Expert Systems and Artificial Intelligence", Journal="International Journal of Approximate Reasoning", volume="2", year="1988", pages="247--302", Note = "http://www.research.microsoft.com/research/dtg/horvitz/DT.HTM" } 
{RG}

As the title suggests, the first article compares the challenge of building an effective diagnostic system (for the diagnosis and treatment of root disorders of apple trees) within first a standard rule-based expert system framework (ES), and then an influence diagram (ID). There were several relevant findings:

  • The two systems produced had similar graphical structures, modulo the directions of the links.
  • While an ID can, in principle, require a great many numbers, here the expert had to specify only a few. (Most of the values degenerated to either 0 or 1, or were determined by other values.) Moreover, a sensitivity analysis suggests that even rough assessments were usually sufficient.
  • While it was easier to produce an initial ES than an ID (as the rule-based ES requires fewer numbers, etc), the ES requires more extensive testing, debugging and tuning, as the initial ES will cover only the combinations that the expert explicitly anticipated. This also means an ID can handle situations that the expert did expect, perhaps better than the initial expert could have. The second article is also superb. Among other things, it discusses the need for handling uncertainties, motivates the use of probabilities and decision theory, and reviews some early attempts. It also states that
"although recent research on the application of decision-science seems promising, for the most part only prototype systems have been demonstrated" 6. Anecdotes * *{RG: Below is a listing of various people's informal anecdotes on why Bayesian nets are, or at least should be, better.}

{"Knowledge Industries" homepage, http://www.kic.com/belnet.htm}

Comparison with Rule-Based Expert Systems:

[...], it is worth pointing out some differences between traditional expert systems and probability-based diagnosis systems. In an expert system, the knowledge engineer attempts to capture the reasoning process of that an expert uses during diagnosis. Probability-based systems, on the other hand, capture the expert's qualitative physical understanding of the system under diagnosis and use this knowledge to construct diagnostic models. While it is possible to capture diagnostic information directly in a probabilistic system, our experience shows that it is much faster and easier to assess causal models.

{Eric Horvitz (horvitz@microsoft.com)}

One of the strongest reasons is that uncertainty is indeed prevalent in the real world and it is critical to represent that uncertainty in at least a semi-coherent manner. Another reason is that Bayesian networks are a fundamentally more modular representation of uncertain knowledge than rule-based systems. This makes them easier to maintain, and to adapt them to different contexts, than there rule-based relatives.

[...]

The uncertain decision making context, and the modularity and ease of maintenance, make the Bayesian networks quite attractive.

[...]

Bayesian networks are very intuitive for nonexperts. We had a United Airlines engine person drawing and debugging these models within a few hours of exposure to them. The NASA Vista people needed to have a way to fuse together the implications of multiple sensors, some noisy, when we began the Vista project. There was no good way to take into consideration all of the sensors and their failure modes with a fault tree. {Bruce D'Ambrosio (dambrosi@chert.CS.ORST.EDU)}

I am currently doing some work with Litton Data Systems. The strong arguments there are primarily technological: improved performance, better communication with experts (model-based reps are modular and easy to work with).

{Stuart Russell (russell@CS.Berkeley.EDU)}

The commonsense case for BNs over eg NNs is

1) the expert can provide knowledge in the form of causal structure quite easily

2) the resulting network after training is understandable and extensible and provides probabilities on variables of interest, and can also be used easily with arbitrary missing data (in both training and use). * {RG}*

I was told that many of the papers at the recent

 NIPS'95 Workshop on Learning in Bayesian Networks and Other Graphical Models 
contained (often implicit) comparisions between Bayesian nets and neural nets, especially in terms of learning. See http://www.research.microsoft.com/research/nips95bn/

META TOPICMOVED MarkCrowley date="1133328095" from="Main.Resumehtm" to="Main.WhyUseBayesNetsArchival"
Revision -
Revision r1.1 - 30 Nov 2005 - 05:21 - MarkCrowley