Analogy – Archeology
When building a simulation model, collecting the information that I need can be a slow and frustrating process. The people I interview are not used to being asked about the small details of how things work. I think about the process as being an archeologist carefully extracting artifacts from the earth. My job is to recognize important findings and carefully capture each artifact in context (e.g. how is this bone arranged versus the others). When I treat the information that I am gathering like artifacts, I document the content and the context: time, place, participants, and subject.
This process has many advantages. I can keep the information from primary sources separate from my conclusions or inferences. I can support conclusions and inferences with corresponding sources. My potential biases as a modeler are separated from the primary source information allowing other simulation modelers to draw their own conclusions. The separation between participant source and consultant narrative is difficult to achieve using only meeting reports.
So what kinds of artifacts do I collect? I have found five useful types of “artifacts” that generally match up with the database structure for case study research as proposed by Robert Yin (2009). The first three artifact types are unaltered coming from a known source:
1) Published: Various documents and archival records that are generally available (with permission). Published artifacts can be externally published books or journal articles, but they can also be internally published procedure manuals or any other reference resources that are generally accepted as true. A sub-type of published artifacts are Annotations that excerpt or summarize a published artifact. Published information creates background context for a project.
2) Source Quote: Interviews, direct observations, documented verbal conversations, or electronic conversations from participants are all primary source quotes. This is the bulk of the detailed information that I collect when modeling a particular system. I find that source quotes can change as participants think about questions and/or gain a different understanding. For this reason, the context of a source quote is particularly important.
3) Other Media: Other forms of source information such as process flow charts, data tables, graphs, organizational charts, even the proverbial ‘back of a napkin’ can be included here. Sometimes this information is available in digital form and sometimes it is a physical item that requires a picture or scan to become part of a database. (If I was digging up dinosaur bones, they would be ‘other media’ type artifacts.) Context for this type of artifact is also important.
The last two artifact types are derived, either by me or by project participants:
4) Narrative: A narrative is any conclusion or inference that is derived from or based upon the validity of a prior source quote or narrative. Distinguishing between source and conclusion supports the ability to share primary/source information with others (who may draw different conclusions based on their experience). Collecting source and derived information using only one artifact type is much like sharing a research report that contains both the raw data and the conclusions that this research team determined. If simulation modeling is to be recognized as a scientific process, then the distinction between source and conclusion is required to independently duplicate results from the same set of source data.
5) Simulation: My simulation models evolve as client-ready versions are shared. Each time I share simulation results with project participants, I capture the model version as an artifact in the project database. This forms a record of what version was shared with whom and when and can be very helpful in explaining any evolution or changes in project understanding. Periodically, a participant will refer back to the model of a particular meeting. The simulation model artifact record gives me instant recall.
All of this is just a matter of process that requires some additional discipline. Getting to a working simulation model should not be over-burdened by information collecting techniques. But with the right database structure, and an easy to use interface, the added information collection process is more than compensated for by accurate and fast recall.
Yin, Robert K. (2009) “Case Study Research Design and Methods”, Applied Social Research Methods Series, Vol 5, 4th edition.