Cognitive NLP Research

Using AI Language Model Principles to Analyse Design Protocols

Morteza is "ready to rise to the formidable challenge of automating the coding and linking stages of protocol analysis and linkograph construction, which they rightly claim requires a lot of labor when done manually." Gabriella Goldschmidt, Linkography: Unfolding the Design Process. P141

Coding and linking concepts in design protocols is one of the most time-consuming and reseource-demanding stages of doing verbal protocol studies. A main challenge here is classifying the concepts based on the context and intention. One would think that, like in other domains, a problem is always a problem, and a solution is always a solution. In medicine, for example, one would rarely expect to see a disease as a medicine, or a medicine as a symptom. In design, however, the same concept could be considered a problem or a solution depending on the context. For instance, while designing a chair for exterior use, an industrial designer might suggest plastic as the material that could withstand the weather. Here, \textit{being plastic} is a solution in response to the need for the chair being used outside. Now imagine if in another project, the brief was asking for the chair to be made out of plastic. This time, \textit{being plastic} brings a collection of challenges and issues to the design space, i.e., it is part of the problem. This characteristic of design concepts and their dependency on the context and intention poses great challenge to any study, attempting to automatise the coding process. The recent rise of context-aware language transformers such as chatGPT, has created the possibility of doing so. However, it is not clear if the lingual context, i.e., the co-existence of words as used by context-aware language models is enough to identify problem and solution concepts.

To facilitate a comparison of multiple protocols, calculation of fractioned PS indices (e.g., 1/10 fractions) is suggested; thus, every protocol will be presented as a ten-point graph of PS index values (Jiang et al., 2012). For calculation of fractioned PS index, the results of the dynamic PS index may be used; but, the window length must be equal to the fraction of the protocol size, and the window must be shifted forward as much as its length at every step.

The dynamic PS index is a singular value to indicate the designer’s focus on problem and solution spaces across the design session. The value is normalised to the number of segments in the protocol\fraction and is comparable between different design protocols. Moreover, the PS index creates conceptual connections between design issues from FBS ontology and the idea of problem and solution spaces that is quite popular in design studies. This is an important connection which enables a comparison of the findings from protocol studies where the FBS coding scheme is used with the results of other non-FBS studies (Jiang et al., 2012).