Minimalism: "Other systems of the mind/brain have to be able to access expressions generated by the faculty of language, to "read" them and use them as "instructions" for thought and action... Language is an optimal solution to legibility conditions." -Chomsky 1998:8-9 ⇒ Our theory of grammar and its interaction with concepts should be as simple as possible Resolving the tension: two levels of representation (a simpler grammatical "map" and a more complex conceptual "territory")

⇒ Simple denotations

Good practices

• Taking morphology and syntax extremely seriously • Using good denotational hygiene

Back to event arguments

So what is the matter with event arguments?

Copley & Harley 2015, 2020 ms.: They're not the best map for the territory, not because they're arguments but because they're atomic, and because they're events. (What's the commonsense notion of an event? Maybe "change"; see Lombard 1979.)

• Hard to represent interaction between events • Hard to represent (a)telicity • Both of these ⇒ overly complex mapping to morphosyntax

We'll see that force arguments seem to give us a better "map" to the event-semantic "territory" than event arguments do. Do they give us a better mapping to morphosyntax as well?

2 About forces 2.1 What is a force?

• Intuitively, a force is an input of energy • It has an origin, a direction, and magnitude ⇒ vector representation • It has a Ceteris paribus condition for its result (≈ normality/stereotypicality/closed-world/efficacy)

• We can define abstract forces based on concrete forces Things the concept of force easily captures:

• Interaction or lack thereof (ceteris paribus/defeasibility/normality/stereotypicality/closed world/efficacy)

• Entrained (cotemporal) results

In thinking of forces, we need to distinguish change and energy.

Forces come for free

• Hume: Causally-derived regularities can be perceived but causal relations themselves cannot be, they come from the mind & Frykholm 1981& Frykholm , 1983: Force-dynamic information can be recovered from information about kinematics alone, difficult to ignore or obscure

A brief history of forces in semantics

Force dynamics in cognitive linguistics: Talmy 1973Talmy , 1976Talmy , 1981Talmy , 1988Talmy , 2000...;...;Shibatani 1973,... Croft 1990, 1991, 1998, 2012...;Langacker 1991..., Gärdenfors 2004, 2005, 2007...

Three components of Talmy's theory

3.1.1 Main organizing principle: Force opposition

• All force-dynamic meanings expressed in language necessarily involve an opposition between two forces that are in opposing directions • Each of these two forces is related to one of two entities (expressed or accommodated):

• The Agonist (usually the agent) is 'singled out for focal attention' • The Antagonist is considered only insofar as it impacts the Agonist • Does the Agonist's force overpower the Antagonist's force, or vice versa? (10) The ball kept rolling.

(

Intrapersonal forces: a way to understand exertion

Physical force manifestations of animate entities are understood to arise (in part at least) from their minds, rather than their physical properties alone Psyche split in two, one part that is "repose-oriented" and another that is "goal-oriented" try: Physical Agonist force is a result of exertion of the Agonist

Modality with forces: Talmy and Sweetser

(16)

The ball can't sail out of the ballpark with the new dome in place.

Bouletic/deontic may/ must

• "[W]anting . . . seems to be conceived in terms of a kind of psychological "pressure", "pushing", toward the realization of some act or state"; "psychosocial force" • Content of desire = direction, relative authority/ability = relative magnitudes • may: Agonist desires p, non-impingement by a potentially stronger Antagonist force • must: no or no particular desire for p on the part of the Agonist, stronger Antagonist force for p

Epistemic (Sweetser 1994(Sweetser , 1990) ) • Modals are "semantically ambiguous between our sociophysical understanding of force and some mapping of that understanding onto the domain of reasoning" • "Epistemic forces" cause a train of reasoning leading to a conclusion (17)

Mary must be at home: The strongest "epistemic force" points to Mary's being home Portner (2009) critique:

• 1. Not really a theory • 2.
Where is intensionality?

• 3. Don't they just reduce to logical relations?

My answers to the critique:

• 1. ok • 2. Intensionality should come from ceteris paribus property of forces; may also however need difference between e.g., desires and actions • 3. Logical relations are on this view essentially epistemic relations between propositions-the problem is in using them for root modality, not epistemic modality.

Causal chains: Croft and others

Croft (1991 et seq.) extended usefulness of force dynamics as an organizing principle for argument structure.

• Criteria for agent and patient roles can be causal, e.g. Dowty 1991, but change need not be implicated (Levin & Rappaport Hovav 1995): Dash hit the table. Force dynamics solves the problem -agents are energy sources and patients are recipients of the energy • Causal chains represented in structure: agents higher, patients lower, instruments in between • Direct/indirect distinction is important to language but is not represented if causation is only about dependencies (e.g. Lewis 1973)

Forces in formal theories

3.4.1 Can there be forces in formal theories?

Cog lx: meaning "cannot be reduced to an objective characterization of the situation described: equally important . . . is how the conceptualizer chooses to construe the situation and portray it for expressive purposes" (Langacker 1990) We must though stay "in touch with reality" (Johnson 1987; note similarity to map-territory question!)

Frege, Lewis, etc.: meaning is based on membership in a set and is about the world itself; propositional, digital, objective, so it's fruitless to try to understand meaning in terms of psychological and sociological phenomena or subjective ideas.

Are forces ok with the formal perspective? In generative approaches, the mapping between semantics and syntax is of intense interest. Is this a problem for formal forces?

No: cognitive structures as starting point from which to construct possible or plausible meanings, with work still to be done at the interface. Also, forces should simplify the interface!

Intensionality: possibilities and causation

Forces seem to have outcomes built into them (their direction). But these outcomes are not always realized.

Thus to formally represent forces we need some sort of caused (small) possibilia.

But wait, doesn't causation have to be represented with counterfactuals and possible worlds?

No.

Copley & Wolff 2014:

• Dependency theories of causation -causation is defined in terms of dependencies between (occurrences of) events. Includes logical, counterfactual, probability. • Production theories of causation -take causal concepts such as force and transmission of energy to be basic, defines causation in terms of these ⇒ Causation can be represented either with e.g. counterfactuals, or with force dynamics; both are powerful enough, and only one is needed 3.5 Formal force theories Forces are located vectors that have (i) an origin, (ii) a magnitude, and (iii) a direction A path is a continuous function from a time interval to the set of located force vectors. Roughly speaking, it is a sequence of positions at which forces (possibly zero) are exerted.

For any event e, path(e) is the path that corresponds to e, if defined.

An agent is the exerter of the force, a patient is the recipient of the force, an instrument is the part of the agent in forceful contact with the patient.

For any event e with path(e), there is a participant theme(e) that occupies subsequent positions of path(e) during the running time of e.

cause(e 1 , e 2 ): if path(e 1 , t i ) + tendency(e 1 (t i ) is collinear with path(e 2 ) while tendency(e 1 )(t i ) is not • v introduces events: e • force introduces forces: f • a force f is conceptualized as that force f of which e is an exertion: vP + force-denoting pP → e exert f • an "exertion" in this sense is best thought of as a notation of the idea that the event is the conduit through which the initiator is connected to the force a. Peter is the initiator of an event which is an exertion of a pulling force located on the carrot. b. Peter is the initiator of an event which is an exertion of a pulling force, where the carrot is the force recipient, and the goal of the pulling force is a region located out of the soil.

Van Lambalgen & Hamm 2005

• There are forces (and events)

• Two kinds of causation • Closed-world hypothesis • Next event • "Commonsense inertia"

• But: basic typological difference is governed by duration (25) An eventuality is a structure (f 1 , f 2 , e, f 3 ), where a. f 1 is a fluent which represents an activity, something which exerts a force b. f 2 is a parametrized fluent, representing a parametrized object or state, which is driven by the force f 1 c. e is the culminating event, representing a canonical goal d. f 2 is a fluent which represents the state of having achieved the goal Not viable given the widespread selectional distinctions between dynamic and non-dynamic verbs: progressive aspects, adverbials, choice of modal base depending on dynamicity of the complement . . .

Copley & Harley 2015

• Basic idea: Dynamic verbs convey that the application of a force results (either by launching or entrainment) in a (perhaps zero) change provided that nothing intervenes • Reify energy as force functions • "Flavors" of v (cf. • Forces (inputs of energy) arise in an initial situation • Forces (inputs of energy) are represented by functions from situations to situations

(26) [[ ]], U σ , U ϕ , U s , U f , net, init, f in is a causal structure iff: a. [[ ]] is an evaluation function b.
U σ is a non-empty set of conceptual situations c. U ϕ is a (possibly empty) set of conceptual forces (representing inputs of energy). d. U s is a non-empty set of linguistic situations such that ∀s ∈ U s , ∃σ : We define two linked sequences, one of situations and one of force functions:

[[s]] = σ e. U f is a (possibly empty) set of functions from U s to U s . f. net is a partial function from U s to U f such that ∀s ∈ U s , ∀f ∈ U f : net(s) := f iff [[f ]] is the net force of [[s]]. g. init(f ) := the inverse function of net (i.e., net(init(f )) = f ) h. f in(f ) := f (init(f ) (27)
(28) . . ., s -2 , s -1 , s 0 , s 1 , s 2 , . . . . . ., f -2 , f -1 , f 0 , f 1 , f 2 , . . . s -2 s -1 s 0 s 1 s 2 f -2 f -1 f 0 f 1 f 2 (29) a. Let f = net(s) iff [[f ]] is the net force of [[s]] b. f n = net(s n ) c. s n+1 = f (s n ) (30) a. init(f n ) = s n b. f in(f n ) = s n+1
Causal sequence, not temporal sequence. Causally-mediated temporal relation: effects do not begin before their causes.

• Stative predicates such as be in the room, know French are treated as predicates of situations, type st . • Dynamic predicates such as eat and stay are predicates of forces, type f t , aka ss, t .

• Active Voice, when present, introduces the Agent/Causer as the (main) source of the energy constituting the force. Kagan 2020 (Finnish):

(31) [[be -ing]] = λp f d λs . p(net(s)) (32) a. v become : λpλf . p(init(f )) & p(f in(f )) (cf Dowty's (1979) BECOME: ¬p(t 1 ) & p(t 2 )) b. v occur = λπλf.[∃f < f in(f ) : π(f )] c. v appear : λxλf.x is part of f in(f ) (presupposed: x is not part of init(f ) d. v stay : λpλf.p(f in(f )) (presupposed: p(init(f ))) (33) a. heat the soup (a little/to boiling). b. λf . [[soup hot]](init(f )) & [[soup hot]](f in(f )) (34) a. dance b. λf.[∃f < f in(f ) : [[dance]](f )] (
• Both telic and atelic predicates specify a result situation as well as a causing situation.

• The result situation need not be a state, though it can be.

• In the world, there can be an entrained and a launched result, though our description usually picks one of these. • Given a causing and a result situation, world/lexical knowledge tells us whether the result situation referred to in the description begins as the causing situation ends (launching), or begins as the causing situation begins, as a "cotemporal result" (entrainment).

Copley & Harley 2020 ms.

• Same basic idea: Dynamic verbs convey that the application of a force results (either by launching or entrainment) in a (perhaps zero) change provided that nothing intervenes • Reify energy as force functions, and change as degree intervals on scales The idea behind the degree approach is that the predicate makes reference to a scale, and telicity corresponds to the case when satisfaction of the predicate occurs on reaching the maximum endpoint of the scale. Atelicity, on the other hand, corresponds to the case when there is no endpoint specified, and the predicate is satisfied by any minimal amount of change that occurs.

(There is a) degree-interval spanning the degree of the heat of the soup as the force is applied and the degree of the heat of the soup in the resulting situation f in(f).

Incremental themes

But how do we distinguish creation vs. consumption? Let's call this difference the "slope", so creation is a positive slope and consumption is a negative slope; including also verbs of maintaining which have zero slope. See also Croft 2010, Verkuyl 2019. Claim: The slope can be recovered from the root meaning (what it means to say "eat(f)") and does not need to be represented in the denotation, so our structure is fine.

  Hume's definition of definition: "[T]he chief obstacle . . . to our improvement in the ... metaphysical sciences is the obscurity of the ideas, and ambiguity of the terms." -Enquiry Concerning Human Understanding 7.1.2/61 ⇒ Concepts should be broken down into combinations of more wellunderstood concepts ⇒ Complex denotations(4)

  , . . ., 2000) a. The ball was rolling along the green. b. The ball kept rolling along the green. (8) Copley & Harley 2015: keep + SC hard to do with event arguments a. Mary kept her door open. b. keep open = cause to stay open c. stay open = be open (complexity, difference in Aktionsart) d. stay open = cause to be open e. Solution: stay open: there is a force whose entrained (cotemporal) result is the door being open (9) CAUSE, ENABLE, PREVENT (Wolff & Song 2003, Wolff 2007)

  door: Subject(Mary, e 1 ) & a. [[v]] = λP λx λe ∃e 1 , e 2 [P(e 2 ) & v(e 1 ) & State(e 1 ) & e = e 1 → e 2 & Subject (x, e 1 )] b. [[V ]] = λP λx λe ∃e 1 , e 2 [P(e 2 ) & V(e 1 ) & Process(e 1 ) & e = (e 1 → e 2 ) & Subject (x, e 1 )]

  2010 and Goldschmidt & Zwarts 2016• on, Dutch op (support), aan (attachment/hanging) • attached to/*from, hanging *to/from • push/pull, squeeze/stretch: a different vector direction from Talmy'shits the nail into the wall.'

  . ∃p [ path(e) = p & auf(p, den-nagel) & schlagen(e) & agent(e) = Maria ] b. ∃e ∃e 1 ∃e 2 ∃p [ e = e 1 + e 2 & cause(e 1 , e 2 ) & schlagen(e 1 ) & patient(e 1 ) = theme(e 2 ) = den-nagel & path(e 2 ) = p & in(p, die wand) & agent(e 1 ) = Maria ] the carrot out of the soil.' Events are to V what forces are to P, or: V conceptualizes events and P conceptualizes forces. (22) a. verb phrase: VoiceP and vP b. prepositional phrase: pP (Svenonius 2003) and forceP

  Folli & Harley 2008) • Dual ontology: grammatical/linguistic/language map vs. conceptual/cognitive territory • This is familiar: it's [[ ]]. (Not familiar: using [[ ]] on variables)

  by evaluation function to -σ

  35) a. write a poem b. λf.[∃x : x is part of fin(f): [[a poem]](x)] (presupposed: x is not part of init(f )) (36) a. stay there b. λpλf.[[there]](f in(f )) (presupposed: [[there]](init(f )))

  • A single meaning for v Degrees and change: Kennedy & Levin 2007: triples S, R, δ where S is an (open or closed) set of degrees, R is an ordering (increasing for 'positive' adjectives like warm, decreasing for 'negative' adjectives like cool, and δ is the dimension of difference) (37) A typology of scale structures (Kennedy & McNally 2005) a. D (0,1) , R, δ (totally) open scale, e.g. hot/cold b. D [0,1) , R, δ lower closed scale, e.g. bent/straight c. D (0,1] , R, δ upper closed scale, e.g. certain/uncertain d. D [0,1] , R, δ (totally) closed scale, e.g. full/empty

  , wid-en, whit-en Hay, Kennedy, & Levin 1999: [INCREASE(ϕ)(x)(d)(e)] = 1 iff ϕ(x)(SPO(e)) + d = ϕ(x)(EPO(e)) (39) a. widen the road in/for an hour b. heat the soup in/for an hour Kennedy & Levin 2007: "we propose that the adjectival core of a degree achievement is a special kind of derived measure function that measures the degree to which an object changes along a scalar dimension as the result of participating in an event.sd )(f ) =: ιd : [p(init(f )) . . . . . . . . . . . , p(f in(f )) . . . . . . . . . . . ] (41) Unified dynamic v: [[v]] = λp sd λf . ∆(p)(f )

  v]](λyλs . hot(y)(s))([[the soup]]) . . . . . . . . . . . . . . . .. . . . . . . . . . meaning of √ hot = [[v]](λyλs . hot([[the soup]])(s)) . . . . . . . . . . . . . . . .. . . . . . FA = λp sd λf . ∆(p)(f )(λyλs . hot([[the soup]])(s)) . . . . . . . . . . . . . . . . . .. . . . (41) = λf . ∆(hot([[the soup]])(s) . . . . . . . . . . . . . . . .. . . . . )(f ) FA = λf . ιd : [hot([[the soup]])(init(f )) . . . . . . . . . . . . . . . . .. . . . . . . . . . . , hot([[the soup]])(f in(f )) . . . . . . . . . . . . . . . . .. . . . . . . . . . . ]

  Heim & Kratzer 1998): If a branching node α has as its daughters β and γ, and[[β]] and[[γ]] are both of type η,t then[[α]] = λx ∈ D η . [[β]](x) & [[γ]](x). This formula is of type η,t. (52)Predicate Restriction (proposed): If a branching node α has as its daughters β and γ, and[[β]] is of type η,t and [[γ]] is of type η,θ then [[α]] = λx η , [[β]](x) . [[γ]](x). This formula is of v the soup]] = λf, eat(f ) . ∆([[DegP ]])(f ) (41), (52), def. EXT = λf, eat(f ) . ∆(λs . EXT[[the soup]](s))(f ) def. EXT = λf, eat(f) . ιd : [EXT[[the soup]](init(f )) . . . . . . . . . . . . . . . . .. . . . . . . . . . . , EXT[[the soup]](f in(f )) . . . . . . . . . . . . . . . . . .. . . . . . . . . . ]

  = create a dance (entrained, atelic) b. Calve = create a calf (launching, telic) (57) Flavors of v for creation and consumption? (not proposed!) a. [[v creation ]] = λp sd λf, ∆(p)(f ) > 0 . ∆(p)(f ) b. [[v consumption ]] = λp sd λf, ∆(p)(f ) < 0 . ∆(p)(f )

  Change is not energy (see Croft 2012, 2015), but change and energy are closely related (Bohnemeyer & Swift 2006); change cannot happen without energy. But look at how the word "dynamic" is used. . .

	(6)	dynamic as 'characterized by energy':
		a. Comrie 1976: "With a dynamic situation, . . . the situation will only continue if it is
		continually subject to a new input of energy"
		b. Smith 1991: "The bounded nature of events can be derived by their dynamicity. Events
		require a constant input of energy."
		c. Bach 1986: 'dynamic' predicates are sit, lie, etc.
		d. Beavers 2011: "I assume that change can only be encoded in dynamic predicates. But
		which dynamic predicates involve changes . . .?"
	2.2 Forces are needed
	(5)	dynamic as 'characterized by change':
		a. Bohnemeyer & Swift 2006: "we propose the basic meaning of dynamicity is change"

b. Beavers 2008: dynamic predicates are those that "involve some "change" or potential change in one participant" c. Fábregas & Marín 2014: "dynamicity" refers to "(abstract) movement in some quality space" d. Maienborn 2007: definition of "dynamic predicates" excludes predicates such sleep and stay

•

  Wolff & Shepard (2013): Temporal gaps, direction influence causal impression ⇒ representation of causation must include time and direction ⇒ must not be mere regularities • Robles-De-La-Torre & Hayward (2001): force perception competes favorably with other kinds of perception • Runeson

•

  No reason formal theories can't talk about conceptual structures instead of the world(Hamm et. al 2006) • Three ways to represent "analogue" forces formally: break it down into minimal parts (e.g.

Zwarts & Winter 2000 vectors), make reference to them through basic relations such as Attach and Support (Zwarts 2010), or represent a simplified/abstract version of them (Copley & Harley 2015, 2020 ms)

3.4.2 The syntax-semantics interface

In cognitive lx, little interest in relating semantics to a formal, autonomous syntactic structure.

"Syntax is required only for the most subtle aspects of communication-pragmatic and semantic features are more fundamental for communication."

(Gärdenfors 2014)

  . atelic vs. variable telicity climb the mountain vs. dance vs. eat apples/the apple b. different paraphrases/flavors cause vs. become vs. do vs. ... c. intuitive causation vs. no intuitive causation heat the soup vs. go to the store d. intuitive degree scale vs. no intuitive degree scale heat the soup, eat the apple, go to the store vs. dance the polka e. delimiting/homomorphic object vs. non-delimiting/non-homomorphic object eat the apple, mow the lawn vs. heat the soup, push the cart f. change vs. no change put vs. stay Krifka 1998 e.g.: Certain predicates involve a mapping from parts of the event to parts of object; quantized object ⇒ telicity. b. Hay, Kennedy, and Levin 1999, Kennedy & McNally 2005, Kennedy & Levin 2008, e.g.: Certain predicates involve a mapping from parts of the event to degrees on a scale; endpoint ⇒ telicity. c. Filip & Rothstein 2005, Filip 2008, Beavers 2012: Both these mechanisms for telicity are needed. d. Hay, Kennedy, & Levin 1999, Kardos 2012: extent of an object can be seen as a degree scale

	4 A better "map" for a better mapping 4.1 Changes of state
	(42) Change-of-state structure: [ vP v [ SC (48) m(y)(s) ..... . . . . = the degree d such that y has the property m in s to the degree d √ root object]] (49) Change of state predicate (telic or atelic) vP f d λf . ) `````` a. telic vs(43) v SC Regarding telicity Is there causation in all telic verb phrases? a. (44) sd, f d λp sd λf . ∆(p)(f ) sd λs . [[ √ ]]([[DP ]])(s) . . . . . . . . . . . . . . .
		a. yes (subevents): Pustejovsky 1999, Higginbotham 2000, Folli 2000, Ramchand 2008...
		b. no (partial/maximal events): ter Meulen 1985, Bach 1986, Parsons 1990, Filip &
		Rothstein 2006, Filip 2008...
	(45)	Theories of the verbal syntax-semantics interface
		a. Levin and Rappoport Hovav 1998: canonical realization rules for classes of roots (e.g.
		externally caused change of state: [ [ x ACT ] CAUSE [ BECOME [ y <STATE> ] ]
		], that information is mapped to syntax
		b. Borer 2005: quantity head to get homomorphism facts
		c. Folli & Harley 2002, 2005: parallel verbal heads (flavors), v do (needs animate/Agent
		subject, takes incremental theme complement), v cause (merely needs Causer, takes
		state complement)
		d. Ramchand 2008: serial verbal heads (init, proc, res)
	(46)	Folli & Harley 2005
		a. The sea/the groom destroyed the beach.
		b. #The sea/the groom ate the beach.
		c. The sea ate away the beach.

type η, θ.

Predicate Modification can be derived from the special case of Predicate Restriction where θ = t, since λx .

EXT(y)(s) = the degree d such that y has the extent in s to the degree d

Conclusion

Better "map" (forces and degrees) ⇒ better mapping between morphosyntax and denotations (unified verbal head)


Katie L Hopkins
email: katie.hopkins@phe.gov.uk

Danie ` Le Meunier

Thierry Naas

Neil Woodford

Evaluation of the NG-Test CARBA 5 multiplex immunochromatographic assay for the detection of KPC, OXA-48-like, NDM, VIM and IMP carbapenemases

Staphylococcus aureus is frequently associated with biofilm-related infections in both inosocomial and community settings.

Sir, No single method currently commercialized for detecting acquired carbapenemases offers comprehensive coverage, but local testing is needed to support rapid detection and prompt action. Within the UK and globally most carbapenemase producers harbour one or more of the 'big 5' families: KPC, OXA-48-like, NDM, VIM and IMP. [START_REF]English Surveillance Programme for Antimicrobial Utilisation and Resistance (ESPAUR)[END_REF][START_REF] Logan | The epidemiology of carbapenem-resistant Enterobacteriaceae: the impact and evolution of a global menace[END_REF] Diagnostic laboratories should therefore consider tests that reliably detect at least four or, preferably, all five of these families. Here, we evaluated the NG-Test CARBA 5 immunochromatographic assay (NG Biotech, Guipry, France), [START_REF] Boutal | A multiplex lateral flow immunoassay for the rapid identification of NDM-, KPC-, IMP-and VIM-type and OXA-48-like carbapenemase-producing Enterobacteriaceae[END_REF] for detecting the 'big 5' carbapenemases in isolates referred to the UK's national reference laboratory.

The NG-Test CARBA 5 assay was evaluated using 197 previously characterized bacterial isolates including 177 confirmed carbapenemase producers and 20 carbapenem-resistant but carbapenemase-negative isolates (Table 1). Isolates represented the diversity of carbapenemase genes (but not the prevalence), and host organisms thereof, identified in the UK. One colony of overnight growth harvested from Columbia blood agar plates was tested according to the manufacturer's instructions. Time until appearance of one or more red lines in the test region of the cassette was recorded by comparison to a line in the control region, but with the final reading performed at 15 min as per the manufacturer's instructions. Any discrepancies between the results obtained with the NG-Test CARBA 5 and those obtained with in-house PCRs and/or WGS were investigated by retesting using the NG-Test CARBA 5 and in-house PCR. Where discrepancies still remained, the carbapenem inactivation method (CIM) was used to screen for carbapenemase activity. [START_REF] Van Der Zwaluw | The carbapenem inactivation method (CIM), a simple and low-cost alternative for the Carba NP test to assess phenotypic carbapenemase activity in Gram-negative rods[END_REF] The NG-Test CARBA 5 identified all KPC (n " 30), OXA-48-like (n " 31), NDM (n " 31) and VIM (n " 29) producers with no falsepositives. One NDM producer initially gave an additional falsepositive signal for VIM. On initial testing, IMP carbapenemases were detected in 12/17 isolates producing IMP-1 (n " 5), IMP-4 (n " 2), IMP-6 (n " 1), IMP-7 (n " 1), IMP-8 (n " 1) and IMP-29 (n " 2). Repeat testing by in-house PCR and NG-Test CARBA 5 confirmed 1/5 'failed' isolates was a false-negative. The four remaining isolates encoding variants most closely related to IMP-13 (n " 2) and IMP-14 (n " 2) were again negative by NG-Test CARBA 5 and carbapenemase activity was confirmed by CIM; therefore, these isolates were also considered false-negatives.

Among the 25 isolates producing two carbapenemases, discrepancies were observed between the NG-Test CARBA 5 and inhouse PCR for two isolates; both were thereafter considered as true-negatives owing to gene loss. One isolate initially gave an additional false-positive band for KPC. All 14 isolates producing carbapenemases belonging to families other than the 'big 5' and the 20 carbapenem-resistant isolates without carbapenemases were correctly identified as negative with no false-positives.

When calculating analytical sensitivity and specificity, we considered each NG-Test CARBA 5 assay to be five individual tests, meaning a total of 985 (197 % 5) tests were performed. After the first round of testing, i.e. including the false-positives but excluding the true-negatives due to gene loss, the overall sensitivity and specificity of the NG-Test CARBA 5 were 97.31% (95% CI 93.84%-99.12%) and 99.75% (95% CI 99.12%-99.97%), respectively. Although both sensitivity and specificity increased following repeat testing, the first NG-Test CARBA 5 assay results would be recorded in a frontline laboratory, with no reason to repeat.

The NG-Test CARBA 5 assay was easy to perform and set-up took 5 min per isolate with relatively little hands-on time. Although the final result was read at 15 min, positive results started to appear within 2-6 min.

All KPC, OXA-48-like, NDM and VIM gene variants identified in the UK to date could be detected with the NG-Test CARBA 5 assay and this study adds carbapenemase variants KPC-4/-23, OXA-245/-436/-484, NDM-2/-3 and IMP-4/-6/-7/-10/-29 to the range previously detected. [START_REF] Boutal | A multiplex lateral flow immunoassay for the rapid identification of NDM-, KPC-, IMP-and VIM-type and OXA-48-like carbapenemase-producing Enterobacteriaceae[END_REF] However, IMP-13-and IMP-14-like carbapenemases were not detected. Problems with detecting these IMP variants were also observed during an evaluation of the Cepheid Xpert V R Carba-R assay. [START_REF] Findlay | Evaluation of three commercial assays for rapid detection of genes encoding clinically relevant carbapenemases in cultured bacteria[END_REF] Sequence diversity within the IMP family and the even distribution of mutations throughout the sequence 6 make it difficult to design primer/probe sets that can detect all variants. However, IMP carbapenemases are reported to have two well-conserved amino acid regions, H2 (amino acids 124-130) and S6

V C Crown copyright 2018.
This article contains public sector information licensed under the Open Government Licence v3.0 (http://www.nationalarchives.gov.uk/ doc/open-government-licence/version/3/) Morganella morganii

1 NDM-1!GES-1 (1)
(amino acids 134-140), and these have been targeted in an immunochromatographic test reported to detect all IMP variants. [START_REF] Kitao | Development of an immunochromatographic assay for diagnosing the production of IMP-type metallob-lactamases that mediate carbapenem resistance in Pseudomonas[END_REF][START_REF] Notake | Influence of sublethal concentrations of vancomycin and quinupristin/ dalfopristin on the persistence of viable but non-culturable Staphylococcus aureus growing in biofilms Sonia Pasquaroli 1 , Barbara Citterio 2 , Gianmarco Mangiaterra 1 , Francesca Biavasco 1 and Carla Vignaroli[END_REF] This study highlights that coverage within families may vary between assays designed to detect the main carbapenemase families. Although IMP producers only account for 379/13682 (2.8%) carbapenemase-producing organisms confirmed by AMRHAI since 2000, they may be becoming more widespread, with 29 laboratories referring 167 isolates in 2017 compared with 20 laboratories referring 75 isolates in 2016 (K. L. Hopkins and N. Woodford, unpublished data). Laboratories should maintain a high level of suspicion regarding any isolate with a meropenem MIC above the EUCAST screening cut-off for carbapenemase-producing Enterobacteriaceae (MIC .0.12 mg/L) but negative in an assay marketed as covering the 'big 5' carbapenemases and refer any such isolates to a national reference laboratory for further investigation.

Table 1 .

 1 Details of clinical isolates with previously characterized mechanisms conferring resistance to carbapenems

		no carbapenemase	(n " 20)
	Carbapenemase status, including allele where known (number of isolates)	NDM VIM IMP combination other carbapenemase	(n " 31) (n " 29) (n " 17) (n " 25) (n " 14)
		OXA-48-like	(n " 31)
		KPC	(n " 30)
			Species	Klebsiella	pneumoniae

Downloaded from https://academic.oup.com/jac/article/73/12/3523/5091005 by Dawson France (CE DBCM SACLAY) user on 16 February 2021

Department of Life and Environmental Sciences, Polytechnic University of Marche, Ancona, Italy;

Department of Biomolecular Science, Biotechnology Section University of Urbino 'Carlo Bo', Urbino, Italy

Acknowledgements

We wish to thank NG Biotech for providing the NG-Test CARBA 5 cassettes and AMRHAI staff for their role in characterization of these isolates.

Funding

This study was carried out as part of our routine work.

Transparency declarations

PHE's AMRHAI Reference Unit has received financial support for conference attendance, lectures, research projects or contracted evaluations from numerous sources, including: Accelerate Diagnostics, Achaogen Inc., Allecra Therapeutics, Amplex, AstraZeneca UK Ltd, AusDiagnostics, Basilea Pharmaceutica, Becton Dickinson Diagnostics, bioMe ´rieux, Bio-Rad Laboratories, BSAC, Cepheid, Check-Points B.V., Cubist


domain: spi info

text:
Wei Xu
email: wei.xu@univ-eiffel.fr

Dominique Gruyer
email: dominique.gruyer@univ-eiffel.fr

Sio-Song Ieng
email: sio-song.ieng@univ-eiffel.fr

Generic Simulation Framework for Evaluation Process: Applied to AI-powered Visual Perception System in Autonomous Driving

Keywords: Evaluation process, Simulation framework, AI-powered systems, Autonomous driving

The rapid development of modern vehicle technology necessitates a robust and comprehensive evaluation framework for AI-powered systems in Autonomous Driving Systems (ADS). This paper introduces a generic simulation framework tailored to the evaluation process of AI-powered systems in ADS, with an emphasis on the visual perception system as a case study. The framework integrates different levels of evaluation that contain critical metrics, providing a comprehensive understanding of system performance and targeted evaluation and analysis of components, under challenging driving conditions. This information can be used to guide improvements to the system, such as selecting better AI algorithms, modifying the design of the system, or improving the environment in which it operates. For each component, the framework not only incorporates a unified definition and configuration but also establishes a communication mechanism, which contributes to effectively integrating different tools and platforms into the evaluation process.

The insights gleaned from the visual perception systems case study can be leveraged for adapting and applying the generic evaluation process to other AI-powered systems in ADS, aiming to promote the development of more reliable and safer systems.

I. INTRODUCTION

As the driving force of the current technological revolution, AI has been applied in many frontier fields. The modern automotive revolution has proven this claim. Over the last decade, vehicle information systems, embedded sensor technologies, and "smart" processing of data have been rapidly developed. Simultaneously, vehicles are increasingly equipped with AI-powered systems of automated mechanisms, which are designed to assist or replace human drivers and improve road user safety.

While AI-based systems have shown great promise in improving automated mobility, they also pose certain threats [START_REF] Goodfellow | Explaining and harnessing adversarial examples[END_REF], [START_REF] Dwivedi | Artificial intelligence (ai): Multidisciplinary perspectives on emerging challenges, opportunities, and agenda for research, practice and policy[END_REF]. Therefore, the evaluation and mitigation of potential threats in AI systems within ADS are critically important. While ADS testing involves both virtual and real approaches, relying solely on physical testing is impractical and timeconsuming due to the complexity of real-world scenarios.

Challenges arise due to factors such as complex traffic patterns, unpredictable pedestrian behavior, and diverse weather conditions, making it difficult to establish controlled experimental conditions for comprehensive evaluation. Consequently, the utilization of simulation tools and platforms has become indispensable.

Various simulators are available for virtual testing of ADS, including both proprietary tools and open-source alternatives. Proprietary simulators like CarCraft from Waymo [START_REF] Madrigal | Inside waymo's secret world for training self-driving cars[END_REF], developed by industry leaders, offer specialized features and capabilities that meet specific requirements. Conversely, open-source simulators such as CARLA [START_REF] Dosovitskiy | Carla: An open urban driving simulator[END_REF] and SVL simulator [START_REF] Rong | Lgsvl simulator: A high fidelity simulator for autonomous driving[END_REF], provide accessibility and facilitate collaboration within the community. Additionally, these simulators can be interconnected with ADS like AutoWare [START_REF] Kato | Autoware on board: Enabling autonomous vehicles with embedded systems[END_REF]. Other noteworthy simulators include SUMO [START_REF] Lopez | Microscopic Traffic Simulation using SUMO[END_REF] for traffic dynamics simulation, etc. For our case study, we utilized the Pro-SiVIC TM interconnecting with the ADS used in RTMaps TM [START_REF] Gruyer | Sivic and rtmaps, interconnected platforms for the conception and the evaluation of driving assistance systems[END_REF].

Researchers must develop specific procedures to target particular components and generate synthetic data when utilizing different simulation platforms. Consequently, there is a growing need for a generic design process, containing the generation of the test configuration, simulations execution, and also comprehensive evaluation of outcomes, to conduct thorough tests for evaluating AI-powered systems in ADS.

In this paper, we present a generic simulation framework for the evaluation process of AI-powered systems in ADS, as shown in Fig. 1. The main contributions can be summarized as follows:

• We conceptualized and implemented three integral components of the framework to demonstrate a generic evaluation process. • The framework contains different levels of evaluation and addresses corresponding metrics. • A case study is presented to illustrate the application of the framework to the visual perception system and provide experimental results containing different levels of evaluation.

The remainder of this paper is structured as follows. In section 2, the AI-powered systems and testing environments are introduced respectively. From sections 3 to 5, a detailed definition and description of the different components in the framework are given. section 6 shows the configuration of the experiments and corresponding results. Finally, section 7 is about conclusions and perspectives on future works.

A. Testing Environments

The environment within the framework typically includes digital models or simulations of real-world scenarios. It includes road networks, traffic conditions, weather conditions, pedestrians, vehicles, and other objects. These simulated environments aim to mimic the complexity and diversity of real-world driving situations, allowing researchers and developers to evaluate and test the performance of AIpowered systems in a controlled and repeatable manner. In our study case, we utilized a digital model derived from the digital twin constructed at the Satory test track, as depicted by Gruyer et al [START_REF] Gruyer | Development of full speed range acc with sivic, a virtual platform for adas prototyping, test and evaluation[END_REF].

B. AI-powered system

An AI-powered system in autonomous driving is a component that employs AI algorithms and techniques to carry out various functions critical for an automated vehicle in its operating environment. These functions include perception, localization, decision-making, path planning, control, and more. In the framework, a simulated sensor suite is needed to enable the vehicle to sense its surroundings, including cameras, LiDAR, radar, and others (as shown in Fig. 2). This comprehensive sensor setup aims to contribute to a more realistic and complex data source, allowing a more accurate and reliable evaluation of the AI-powered system. In the study case, our visual perception system is equipped with camera sensors built from Pro-SiVIC TM , which are already validated inside the digital twin of the Satory test track [START_REF] Gruyer | Modeling and validation of a new generic virtual optical sensor for adas prototyping[END_REF].

III. GENERATOR

The generator is crucial in building the framework, generating necessary configurations, and selecting algorithms for evaluation. As demonstrated in Algorithm 1, it is responsible for generating configurations of evaluation scenarios based on Operational Design Domain (ODD) and Object and Event Detection and Response (OEDR). It also selects candidates of AI algorithms for the framework according to specific objectives, then evaluates and validates them based on a representative real-world dataset. Moreover, the generator component generates the configuration of the ground truth for the executor based on the selected algorithms, ensuring the accuracy and reliability of the evaluation process.

A. Evaluation objectives

The evaluation objectives of an AI-powered system in ADS are derived from an analysis of the system and its operating environment. These objectives encompass multiple levels: 1) At the system level, the overall performance and quality of the AI system are evaluated in simulated environments; 2) The components/functionalities level focuses on evaluating specific functions and algorithms necessary to meet the expected functionalities of the system; 3) Additionally, the scenarios level evaluates the system's capabilities within a defined ODD, including safe driving in different scenarios under varying conditions like non-optimal weather, traffic, and lighting. Categorizing the evaluation objectives into these levels facilitates a comprehensive evaluation of the system's performance, safety, and areas for improvement, offering valuable insights into its capabilities and limitations.

B. Scenario Definition and Configuration

1) Scenario Definition: involves the conceptualization and specification of the fundamental elements:

• Scene contains the overall environment where the scene takes place, including return SC, GT C, Objs, As ▷ Return configurations, objectives, adverse conditions, and selected algorithms 8: end procedure weather or lighting, which can influence the behavior of dynamic elements.

• Event represents incidents or occurrences that unfold during the scenario. These events can be pre-defined or dynamically generated and contribute to the scenario's progression. They include stimuli, triggers, or changes in the environment or state change of other objects (outside ego), shaping the sequence of actions and reactions within the scenario. • Action pertains to the response or behavior exhibited by the ego object in the scenario. It demonstrates how the ego object in the scene reacts to events or encountered conditions. Actions may include acceleration, braking, or changes in the direction of the ego vehicle. • Criteria refers to the specific conditions or standards required for the simulation scenario to be deemed complete or successful. These criteria could include factors such as reaching a particular time limit, accomplishing predefined objectives, meeting specific performance metrics, satisfying safety requirements, or any other relevant measures that define the desired conclusion of the scenario.

2) Scenario Configuration: involves the implementation and customization of a scenario based on the definition. This process focuses on the detailed setup and arrangement of specific elements, conditions, and variables within the scenario. An effective scenario configuration should be done within the defined boundaries of ODD and OEDR.

ODD contains the specific operating conditions and environments within which ADS is intended to function safely and effectively. By considering the ODD in the scenario configuration, the scenarios accurately reflect the real-world conditions that the system is designed to encounter. This involves defining geographic boundaries, traffic conditions, and factors that influence the system's operational limits, thus ensuring the scenario's relevance and accuracy.

OEDR focuses on the system's ability to detect and respond to specific objects and events within its operational environment. When configuring scenarios, it is imperative to define the types of objects the system should detect. Furthermore, the scenario should include events that the system should recognize and respond to, such as sudden lane changes, emergency braking, or any other relevant mapping. By incorporating these elements, the scenario enables the evaluation and improvement of the system's perception and response capabilities.

By aligning scenario configuration with the ODD and OEDR, the resulting simulations accurately represent the operating boundary and allow for a comprehensive evaluation of ADS.

C. Algorithms selection

To select algorithm candidates for an AI-powered system, it is essential to establish the domain of AI first. In the case of a visual perception system, deep learning methods like Convolutional Neural Networks (CNNs) have demonstrated promising results and are commonly used for image detection and segmentation tasks.

Once the domain is determined, specific tasks should be extracted based on the system's objectives. After an extensive investigation of algorithms suitable for these tasks within the chosen AI domain, potential candidates can be identified. These candidates will undergo training using relevant datasets, and if possible, the models will be fine-tuned. Subsequently, the performance of the trained models will be validated to ensure they meet the necessary criteria for further consideration.

D. Ground truth selection

In real-world environments, ground truth can be generated through manual annotation or by using calibrated and accurate sensors or devices to capture the actual values of the variables being measured. For example, the BDD100k dataset [START_REF] Yu | Bdd100k: A diverse driving dataset for heterogeneous multitask learning[END_REF] has been widely used in visual perception research [START_REF] Wu | Yolop: You only look once for panoptic driving perception[END_REF], [START_REF] Pang | Quasi-dense similarity learning for multiple object tracking[END_REF] and provides ground truth labels for various tasks as shown in Fig. 3a. In the proposed framework, this type of ground truth is used for training and preliminary validation of the selected algorithm. In the simulation framework, the selection and configuration of the ground truth are based on the chosen algorithms and the characteristics of the environment. Ground truth data can be generated by using a physics engine to model the behavior of the vehicle and its interaction with the environment, Fig. 3b shows the different ground truth for visual perception tasks in Pro-SiVIC TM . The ground truth generated by the simulator will be collected by the executor and used for the final evaluation process.

IV. EXECUTOR A. Execution process

The executor is responsible for executing the different test cases on the integrated platform and tools, which are built by the output from the generator component of the framework, and also generating different types of results. The module must ensure that the system is executing properly and that the intermediate results are being generated correctly, and then passed back to the generator component as feedback. This process aims to refine the parameters inside the configuration of scenarios and adjust ODD or OEDR if needed.

If there are any issues or errors in the execution, it needs to be resolved before passing on the final results to the evaluator component. Once the execution is complete, the final results are passed to the evaluator component for assessment against the different types of evaluation metrics.

The process of executor is expressed by the Algorithm 2.

B. Integration with tools and platforms

Integrating the evaluation framework with appropriate tools and platforms is crucial for the executor component to effectively perform its tasks. Fig. 4 illustrates a case of integration of two interconnected software, RTMaps TM and Pro-SiVIC TM in our framework. By utilizing the capabilities of Pro-SiVIC TM to design realistic and complex virtual environments, developers can simulate various road, traffic, and weather conditions that their systems might encounter. Meanwhile, RTMaps TM provides a module-based environment to design different sub-systems for ADS, and also has real-time multisensor processing and data fusion capabilities to enable the effective management of sensor data. Besides, it also has the capability to record and replay the data from the observer, also providing an efficient means for the evaluation process.

As shown in Fig. 4, it is worth mentioning that the Data Distribution Service (DDS) as the communication mechanism is integrated within the evaluation framework, which offers an effective and interoperable Application Programming Interface (API) for seamless data sharing and communication among the various components. It contributes to enhancing the framework's overall performance, scalability, and effectiveness in the evaluation process.

V. EVALUATOR

The evaluator is responsible for evaluating the performance of the AI-powered systems. It applied the selected evaluation metrics to the output from the executor, and then hereby evaluate the results combining corresponding ground truth. The overall process can be abstracted as shown in Algorithm 3.

The metrics used in the framework are chosen based on the different levels inside the evaluation objectives of the system, such as component level, system level, and scenario level.

A. System evaluation

In order to evaluate the high-level quality of AI-powered system in ADS, such as a visual perception system, it is necessary to implement a full mobility service and propose relevant and representative scenarios involving an exhaustive set of conditions/configurations/situations allowing for quantification of the performances and the quality of the service. The metrics (a case of visual perception system) can refer to a set of specific Key Performance Indicators (KPIs):

• Risk specific: Longitudinal and lateral distance, Time to collision (TTC), Time Exposed Time-to-Collision (TET), Deceleration Rate to Avoid a Crash (DRAC), etc. • Task (detection/tracking) specific: Success rate, Loss, Distance, etc. • Time specific: Frequency, Time to detect/track, False alarm frequency.

B. Component evaluation

This level of evaluation focuses on the performance of individual algorithms or functions within the AI-powered system. The metrics are typically related to the functionalities of the system, as the metrics of the visual perception system shown in Table I.

C. Evaluation under complex scenarios

This level of evaluation involves testing the performance of an AI-powered system under various challenging conditions, such as adverse weather, low lighting, and unexpected obstacles. These complex scenarios can be difficult to replicate in real-world testing, which makes simulation tools and virtual environments more essential. The use of simulation allows for the creation of complex scenarios that can be repeatedly tested, analyzed, and modified to evaluate, analyze, and improve the performance of the system. One example we used in the experiment is Pro-SiVIC TM , which offers a range of adverse scenarios, as shown in Fig. 5.

The related metrics can vary depending on the specific application and system requirements. However, some common metrics for this level of evaluation include: • Robustness: This metric evaluates the ability of the system to perform consistently and accurately in various challenging and unforeseen situations, and is usually reflected in various performance metrics, such as accuracy, precision, recall, and F1-score, etc. Robustness can be measured by analyzing these performances in different scenarios and under different conditions, and also by assessing their ability to maintain performance levels over time.

• Reliability: This metric evaluates the system's ability to make reliable decisions in emergency situations or other adverse situations.

VI. EXPERIMENTS AND RESULT

A. System and Environment

In our experimental setup, we use a visual perception system based on a monocular camera. This system is implemented as a module within the RTMaps TM framework and is an extension of the driving system developed in the H2020 Trustonomy project [START_REF] Xu | Safe vehicle trajectory planning in an autonomous decision support framework for emergency situations[END_REF]. The camera-based perception system is specifically designed to provide essential functionalities such as object identification, localization, and tracking.

For the purpose of our simulations, the module-based ADS operates within the Pro-SiVIC TM rendering environment. To replicate real-world scenarios, we utilize the digital twin of the Satory test track within Pro-SiVIC TM . This virtual representation of the test track has undergone validation using an existing perception system equipped with a set of various sensor types and technologies.

B. Scenarios generation

To evaluate our framework's genericity, we developed a full bus stop service in RTMaps and the full environment with vehicles, sensors, infrastructures, and building in Pro-SiVIC TM . This set of models and modules is used for the evaluation of a visual perception system equipped with different AI models.

As shown in Fig. 6, the service and the situations that could be encountered are modeled with a set of 6 scenes. The continuous assembly of these scenes along a 3.4 km trajectory on the Satory's test track allows for the construction of a tree of possible scenarios. The vehicle trajectory includes 4 or 5 types of bus stations. The simulated environment comprises different types of bends and straight lanes. The ego vehicle is restricted to driving on its ego-lane, while other traffic objects can perform cut-in and cut-out maneuvers on different lanes.

The ODD is the following: Ego vehicle travels on the right lane of the predefined road at a maximum speed of 20 km/h, while maintaining a safe following distance and performing car-following maneuvers when a front vehicle is detected. The vehicle's primary objective is to ensure the safety of passengers and road users by avoiding collisions and adapting its speed to traffic conditions, visibility caused by different weather conditions, and the environment. Additionally, the vehicle stops at the bus station to allow passengers to board and alight. After the stop, the vehicle utilizes dedicated longitudinal and lateral profiles to return to the center of the right lane and resume driving.

C. Algorithms and models

To achieve the system functionalities, we have selected two renowned visual perception algorithms: YOLOv5 [START_REF] Jocher | YOLOv5 by Ultralytics[END_REF] and YOLOv8 [START_REF] Jocher | YOLO by Ultralytics[END_REF], along with their respective models: v5s and v8s. In addition to the object detection models, we have also incorporated two corresponding instance segmentation models (v5s-seg and v8s-seg), aiming to provide more detailed information about object boundaries within an image.

To further enhance the system's capabilities for large-scale diverse driving, we retrained and fine-tuned the models using the BDD100K dataset based on the previous COCO dataset [START_REF] Lin | Microsoft coco: Common objects in context[END_REF], the training phase is equipped with a 16-core Intel i9-12950HX 2.30 GHz CPU and NVIDIA-GTX A4500 GPU core graphics card.

D. Deployment and execution

After the training phase, we deployed the trained model into the software platform RTMaps TM (C++ support), in order to perform the model as a functioning part of the perception system and interconnect with the simulator. During the deployment, we used TensorRT C++ API to optimize the model for inference (with precision FP16), which is based on NVIDIA's Compute Unified Device Architecture (CUDA). The setup environment and tools contain CUDA 10.2, Py-Torch 1.10.1, cuDNN v8.2.1 corresponding to CUDA, Ten-sorRT 8.2, and Visual Studio 2017 (v141).

E. Evaluation and result

We proposed 3 levels of evaluation for this visual perception system equipped with different YOLO models.

1) Scenarios level: We have defined seven weather filters on the camera as shown in Fig. 7. The first scenario representing a clear sky is the reference. The remaining scenarios focus on simulating various weather conditions. We have categorized homogeneous fog into three levels using the Koschmieder law, ensuring an accurate representation. For rain, we have incorporated two filters. The first filter replicates a waterfall effect, while the second mimics raindrops on a camera lens or windshield. Similarly, we have created three different levels of intensity to simulate varying degrees of rain.

2) Component level: We curated seven distinct groups of datasets, each consisting of approximately 10,000 images for detection and instance segmentation, organized based on predefined weather conditions. For tracking, the datasets are 7 groups of real-time videos. These datasets included various types of simulated vehicles, such as cars and trucks, navigating the Satory test track. Different camera perspectives were incorporated, including front and rear views of the ego vehicle, as well as close and distant perspectives. The performed results were evaluated using a set of metrics. The evaluation values are presented in Fig. 7.

The results clearly demonstrate the influence of different weather conditions on the metrics. Adverse rain weather significantly impacts mAP, indicating difficulties in accurately detecting and localizing objects under heavy rain conditions. In contrast to the impact of adverse rain weather, varying levels of fog have a relatively lesser effect on the metrics. While there is still a decrease in the mAP with increasing fog levels, it is not as pronounced as the impact of rain. However, it is worth noting that there is a consistent increase in precision as the degree of fog increases. This suggests that fog has a discernible influence on improving precision by reducing background interference and minimizing the impact of distant objects.

Comparing the performance of v8s and v5s, it is observed that v8s generally achieves slightly higher metric values. However, when it comes to object tracking, v5s demonstrates a lower MOTP than v8s. This indicates that v5s exhibits higher precision in tracking object locations. The disparity in tracking performances may be attributed to the frequency, as v5s benefits from a higher update rate and more frequent predictions, leading to improved tracking precision.

3) System level: In comparison to the instance segmentation model, both v5s and v8s models demonstrate higher accuracy in providing boundary information, as reflected by mIOU. Hence, we have incorporated these two models into our system and evaluated their performance under bus stop scenarios. Specific KPIs were calculated based on the realtime results obtained from the system, as in Tab. II.

While v8s demonstrates higher detection and tracking capabilities than v5s, it also exhibits an increase in tracking time under adverse rain weather. This suggests that more objects may be detected due to the presence of raindrops Fig. 6. 6 scenes for the docking at a bus station on the lens, and YOLOv8 proves to be more sensitive in this scenario. On the other hand, under foggy conditions, both system frequencies are reduced, indicating that fewer remote objects are detected due to the degree of fog.

Tab. II also presents the evaluation results obtained using collision probability and risk of collision based on TTC [START_REF] Glaser | Maneuver-based trajectory planning for highly autonomous vehicles on real road with traffic and driver interaction[END_REF]. We calculated the mean value specifically for critical scenarios. Notably, the risk across all groups was found to be low primarily because our scenarios involve low speeds, resulting in a low likelihood of injury. However, the risk can still indicate heightened safety concerns in moderate and heavy rain weather conditions, similar to the collision probability.

VII. CONCLUSION

This paper presents a simulation framework designed to evaluate AI-powered systems in ADS. The framework is generic in nature, allowing for the utilization of various software and tools. In the case study presented, we used RTMaps TM and Pro-SiVIC TM , but these components can be substituted within the framework. The key strength of the framework lies in its ability to establish a unified evaluation process, and offer an effective and interoperable API for interconnecting between each component, as a result in facilitating the seamless integration of test case configuration, simulation execution, and evaluation library (different levels). This modular approach ensures the independence and collaboration of different components, enhancing the framework's flexibility and applicability.

Future work in this area could focus on two main aspects: 1) Firstly, the validation process could be integrated into the existing framework to assess AI-powered systems from different levels, including components, tools/modules, and the overall system. This would enhance the robustness and reliability of the evaluation process; 2) Secondly, expanding the scope of specific case studies would be valuable in approaching real-world scenarios. For instance, additional adverse conditions (non-homogeneous fog, cloud of dust, cloud of smoke, sun glare, optical blur) and specific physical effects (reflection on road surfaces and other objects) could be incorporated, as well as the use of intrinsic cameras parameters generated different adverse conditions (e.g., noise, DoF, Optical distortion, Auto exposure, glow). This would provide a more comprehensive evaluation of the system's performance under challenging and realistic conditions.

Fig. 1 .

 1 Fig. 1. Simulation Framework for evaluation of AI-powered systems in ADS

Fig. 2 .

 2 Fig. 2. Realistic sensor simulation provided by Pro-SiVIC TM

  (a) BDD (b) Pro-SiVIC TM

Fig. 3 .

 3 Fig. 3. Ground truth of visual perception from the real world and simulation

Fig. 4 .

 4 Fig. 4. Integration with tools and platforms

  (TNR), True Detection Rate (TDR), Accuracy, Precision, Recall, F-Measure, Receiver Operating Characteristic (ROC Curve), Detection Error Trade-off Curve (DET Curve), Precision-Recall Curve (PR Curve), Average-Precision (AP), mean Average Precision (mAP), etc. Segmentation Partitioning an image or video frame into regions and assigning semantic labels to each pixel or region Pixel Accuracy (PA), Class Pixel Accuracy (CPA), mean Pixel Accuracy (mPA), IoU, mean Intersection over Union (mIoU), etc. Tracking Following the movement and preserving the identity of an object or multiple objects over time in a video sequence Object Tracking Time delay, Identification switch(IDSW), Multiple ObjectTracking Accuracy (MOTA)[START_REF] Bernardin | Evaluating multiple object tracking performance: the clear mot metrics[END_REF], Multiple Object Tracking Precision (MOTP)[START_REF] Bernardin | Evaluating multiple object tracking performance: the clear mot metrics[END_REF], Higher Order Tracking Accuracy (HOTA)[START_REF] Luiten | Hota: A higher order metric for evaluating multi-object tracking[END_REF], etc.

Fig. 5 .

 5 Fig. 5. Adverse scenarios from Pro-SiVIC TM

Fig. 7 .

 7 Fig. 7. Result on different weather conditions at component level

  INSTANTIATE(S withAi in T i , E with SC , P ) ▷ Instantiate the system SI with algorithm Ai in test case T i and environment EI with scenario configuration SC on the integrated platform P

	Algorithm 2 EXECUTOR
	1: procedure EXECUTE( SC, As, GT C, ACs )	▷ Outputs from generator as inputs of executor
	2:	T s ← BUILD(As , ACs)	▷ Build the test cases with different algorithms As and adverse conditions ACs
	3:	for T i ∈ T s do
	4: SI , EI ← 5: T a T i ← T.GENERATE(S T i , ACi in T i) ▷ Generate the test action T a T i based on the environment state S T i and the adverse condition ACi in the test case T i
	6: 7: 8:	Sa T i ← S.GENERATE(Obs T i ) S T i , Obs T i ← E.UPDATE(T a T i , Sa T i ) end for	▷ Generate system action Sa T i based on the observation Obs T i ▷ Environment updates based on system actions Sa T i and test actions T a T i
	9:	GT s , Rs ← P.GENRATE(Obs , GT C) ▷ P records final results Rs and ground truth GT s (based on ground truth configuration GT C) from
		the observer Obs
	10:	return Rs, GT s	▷ Return final results and the ground truth
	11: end procedure
	Algorithm 3 EVALUATOR
	1: procedure EVALUATE ( Rs, GT s, Objs )	▷ Evaluates the final results with ground truth
	2:	LEs ← LEVEL(Objs)	▷ Define different levels of Evaluation LEs
	3:	M etrics ← Select(LEs)	▷ Select metrics for different evaluations
	4:	R metrics ← P rocess(Rs , GT s , M etrics)	▷ Calculate the result of metrics R metrics
	5:	return V isualize(R metrics ), Analyze(R metrics )	▷ Visualize and analyze the result of metrics R metrics
	6: end procedure

TABLE II EVALUATION

 II VALUE OF SYSTEM METRICS

			Clear Sky	Light Fog	Moderate Fog	Dense Fog	Light Rain	Moderate Rain	Heavy Rain
			v5s	v8s	v5s	v8s	v5s	v8s	v5s	v8s	v5s	v8s	v5s	v8s	v5s	v8s
	Perception	Success Rate (%)	62.7	64.4	55.1	65.0	48.5	51.9	45.3	36.9	49.7	62.8	37.7	53.5	12.6	31.9
	Specific KPIs	Mean Tracking Error (m)	2.30	1.91	2.25	1.72	1.67	1.48	1.31	1.44	2.49	2.33	2.14	2.43	3.51	2.68
		Detection time (ms)	20	36	21	37	19	36	19	36	19	36	19	36	20	37
	Time Specific KPIs	Tracking time (ms)	72	104	67	90	59	76	55	68	67	110	60	152	58	174
		System frequency (Hz)	16	11	17	13	17	14	18	16	16	11	17	8	18	8
		minimal TTC (s)	1.27	1.82	1.52	1.83	2.02	1.83	1.93	2.02	1.79	1.70	-	-	-	-
	Risk	Mean Collision Probability (%)	31.7	26.7	30.1	28.5	26.1	27.4	24.0	23.8	27.6	28.5	45.8	37.4	47.2	36.4
	Specific KPIs	Mean Risk (%)	3.77	2.75	3.23	2.76	3.03	2.75	2.85	2.78	2.90	2.78	8.51	4.84	8.93	5.04
		Crash	✘	✘	✘	✘	✘	✘	✘	✘	✘	✘	✔	✔	✔	✔

*This work was supported by the PRISSMA project and Horizon Europe AUGMENTED-CCAM project.


• Entrainment is easy to represent, bringing activities into alignment with causal treatments of accomplishments

• Force-dynamic causation is better than mere dependency (e.g. Lewis 1973) in accounting for direct/indirect distinctions

• We can represent goal-directed action: the actor has an intention/goal in s, and from s arises the actor's action to bring about a situation of which the goal holds

• The Davidsonian argument is retained (though it is no longer a commonsense event)

• There is a natural distinction between events and states (namely, energy) that accounts for verbs of maintaining 2 Two issues with Copley & Harley's force-theoretic framework

2.1 Dynamic verbs don't always describe an input of energy

(1) a. The ball rolled down the hill. b. The soup cooled. c. John let the children eat. d. Bill killed my plants by not remembering to water them.

Causal statives

(2) a. That wall supports the ceiling. b. That wall must support the ceiling. c. In case you want to know, that wall supports the ceiling.

(3) a. That curtain lets the light through. b. That curtain must let the light through. c. In case you want to know, that curtain lets the light through.

(4) a. The flowers decorate the room beautifully. b. The flowers must decorate the room beautifully. c. In case you want to know, the flowers decorate the room beautifully.

(5) a. #John decorates the room beautifully. b. John must decorate the room beautifully. c. In case you want to know, John decorates the room beautifully.

3 Causal models 3.1 What even is a causal model?

For our purposes, a causal model is a formal representation of the structure that causal relations give to our conceptual model of the world.

Basics (bedtime reading, sort of statisticky in parts): [START_REF] Pearl | The book of why: the new science of cause and effect[END_REF] Basics (mathy): [START_REF] Pearl | Causality: Models, reasoning and inference[END_REF]; [START_REF] Halpern | Axiomatizing causal reasoning[END_REF]; [START_REF] Halpern | Causes and explanations: A structural-model approach. part i: Causes[END_REF] . . . • There is a set V of variables that are the vertices (or nodes) of the graph.

In
• These are connected by a set of edges (or arrows) E.

• The edges are directed and represent the dependency of one value on another. For instance, A → B represents that the value of B is dependent in some way on the value of A .

• Absence of an edge between two variables means that the values are independent of each other.

( Variables with arrows pointing at them are endogenous variables.

• There is an asymmetry between exogenous variables and endogenous variables.

• Values of endogenous variables can be expressed by means of an equation over the variables that they depend on. When we do this, we can call the model a "structural equation model". ( 7) An example from [START_REF] Halpern | Causes and explanations: A structural-model approach. part i: Causes[END_REF] with truth values

F = 1 if There is fire is true, F = 0 if it is false. ("whether there is fire") M = 1 if The match is lit is true, M = 0 if it is false. ("whether the match is lit") L = 1 if There is lightning is true, L = 0 if it is false.
("whether there is lightning")

Function that takes L, M and yields value of

F : F = 1 if either L = 1 or M = 1. F L M

A formal setup

From [START_REF] Halpern | Causes and explanations: A structural-model approach. part i: Causes[END_REF]:

(8)
A signature S is a tuple (U, V, R) where U is a set of endogenous variables, V is a set of exogenous variables, and R associates with every variable Y ∈ U ∪ V a nonempty set R(Y) of possible values for Y .

(9) A causal model (or structural model ) is a tuple M = (S, F), where F associates with each variable X ∈ V a function denoted F X such that: (11) Taxonomy of basic 3-node structures:

F X = (× U ∈U R(U )) × (× Y ∈V-X R(Y )) → R(X) a. In words: F X is
Collider Chain Fork X Y Z X Y Z X Y Z

FAQs

How are the arrows to be read?

• The arrows are NOT to be read as CAUSE, leads to, material implication, or Talmian forces.

• The arrows ARE to be read as "the value of A affects the value of B ", "the value of A influences the value of B ", or "the value of B listens to the value of A ".

Wait, don't causal models have to use probabilities and Bayes' Theorem?

• No, they can use any value.

Which variable is THE cause?

• See Baglini & Bar-Asher Siegal 2020

How does one decide which variables to include?

• Short answer: Include everything that's relevant.

• Long answer: See next long answer.

How does one decide which arrows to include?

• Short answer: We have intuitions about where the causal relations are supposed to be.

• Long answer: We need to make sure that . . . . . . spurious dependencies do not get arrows (e.g. shoe size and reading ability)

. . . if A → B → C is in our model, the value of C only listens to the value of A through the value of B

. . . for any variable X in the structure, given X 's immediate causal ancestors, X is independent of its (other) non-descendants (Causal Markov condition)

. . .

Where do causal models come from?

The short answer is that they come from the field of statistics in the early 20th century, with a huge ramp up of formalization and popularity in the last 30 years or so. See https://plato.stanford. edu/entries/causal-models/. Pearl and Mackenzie (2018) also has some history.

Why use causal models?

Pearl and Mackenzie (2018) Chapter 1 (http://bayes.cs.ucla.edu/WHY/why-ch1.pdf) in particular gives some context:

(12)
Level 2: Intervention [START_REF] Pearl | Causality: Models, reasoning and inference[END_REF], see also [START_REF] Woodward | Sensitive and insensitive causation[END_REF]) do( X = x): Set the value of X to x and erase all arrows that point to X (13) Level 3: Counterfactuals a. [START_REF] Pearl | Causality: Models, reasoning and inference[END_REF] uses probabilities for these, e.g. in a case where X → Y is in the model, and we know that X = 1, we want to know the likelihood of Y = 1 if we (or something) had intervened to set X to 1. P ( Y X =0 | X = 1, Y = 1) b. [START_REF] Kaufmann | Causal premise semantics[END_REF]: No need for probabilities if we use both Kratzer's causal premise semantics along with causal models. The premise background corresponding to the ordering source determines whether the value of Y is necessary or sufficient, given its parents, avoiding determinism.

4 How are causal models related to... ) of the dependency. The tables associated with causal models are similar, except that all of the above are assumed to be due to causality (kind of a Level 1.5?). ( 14) Familiar truth tables:

And: Or: Material conditional:

P Q P ∧ Q 1 1 1 1 0 0 0 1 0 0 0 0 P Q P ∨ Q 1 1 1 1 0 1 0 1 1 0 0 0 P Q P ⇒ Q 1 1 1 1 0 0 0 1 1 0 0 1 (15)
Some relations and functions to think about: Force interactions are easy to represent as colliders. Two ways to figure out the value of a node that has incoming arrows:

Causal necessity Causal sufficiency A B 1 {(0),1} 0 0 A B 1 1 0 {0,(1)} Stimulatory influence (+): Inhibitory influence (-): A B 1 1 0 0 A B 1 0 0 1 A B 1
• Value of a node is determined from the values of all its influences (standard causal model)

• Each arrow represents a function and incoming arrows "compete" to determine the value of the node (more like force dynamics)

...intentions?

If one is doing an action intentionally, the value of the result listens to the value of the intention. So the causing node could be an intention.

...times/situations?

The variables can be relativized to times (Halpern and Pearl, 2005, 18):

X i1 , X i2 , X i3 ... (16)

...event arguments?

Statisticians who use causal models talk about X = x as an "event". We can represent a Davidsonian event as a node.

(17) a. John was buttering the toast in the kitchen. b. John was slowly buttering the toast. From there we have a choice: What do we need from the familiar Kratzer-style tools (atomic possible worlds, sets of atomic possible worlds, orderings on sets of atomic possible worlds)? And if we don't use them, do we need to use probabilities?

5 What can causal models do for semantics?

Many of the same things as the force-theoretic model:

• We get the benefits of explicitly talking about causal relations, even when there are multiple influences and/or effects • The Davidsonian argument is can be retained (though it is no longer a commonsense event)

• There is a natural distinction between events and states (namely, energy) Dynamic verbs don't always describe an input of energy, and that's fine:

  a function that determines the value of X given the values of all the other variables. b. Notation: The set of all ordered pairs (a, b), where a is an element of A and b is an element of B, is called the Cartesian product of A and B and is denoted by A × B. c. More notation: (× U ∈U R(U )) is the ordered n-tuple of all the values of all the variables in U. d. Exercise: How might we describe F X in words, using "the arrows that..."? (10) Example: if F X (Y, Z, U ) = Y + U , then if Y = 3 and U = 2, then X = 5 no matter how Z is set. (As shorthand, we can also write this equation as: X = Y + U ) The function F defines a set of these, called structural equations.

  a. Launching (for telicity): time of evaluation of influencing node < time of evaluation of affected node b. Entrainment (for atelicity): time of evaluation of influencing node = time of evaluation of affected node Another possibility: Variables have values, and these values are relativized to times. The valuation function (call it R) could be redefined to take a time argument in addition to the variable argument. So, R(X)(i) = x, or in a more semantics-friendly notation, R(X) = λi. [[p]](i).

  the hall, John was buttering the toast in the kitchen. b. *Quickly, John was slowly buttering the toast. Nominalizations (Data from Sichel 2010, Alexiadou et al 2013) (19) a. The scientists justified the evacuation. P → E P = whether ∃p : p(scientists)(i) E = whether occur(e justif ication-of -evacuation )(i ′ ) b. The hurricane justified the evacuation. P → J P = whether ∃p : p(hurricane)(i) J = whether justif ied adj (x evacuation )(i ′ ) c. The hurricane destroyed the marina. P → E P = whether ∃p : p(hurricane)(i) E = whether occur(e destruction-of -the-marina )(i ′ ) (20) a. The scientists' justification of the evacuation ιe . . . b. *The hurricane's justification of the evacuation No event argument to bind! c. The hurricane's destruction of the marina ιe . . . 4.6 ...possible worlds? Kaufmann (2013):

  ball rolled down the hill. b. The soup cooled. c. John let the children eat. d. Bill killed my plants by not remembering to water them. Causal statives: energy is detached from causation, no problem representing causation in the absence of energy Some more vague but still useful reasons to take up causal models: • More clarity around what the arrows are: they represent dependencies between values • Influences are much easier to individuate • More freedom with functions/relations (too much though?

  linguistic semantics: Work by Stefan Kaufmann in the early 2000s, and since about 2010 by Rebekah Baglini, Elitzur Bar-Asher Siegal, Cleo Condoravdi, Sven Lauer, Fabienne Martin, Prerna Nadathur, . . .

	• Causal structures are formally represented by means of a directed acyclic graph (DAG).

  Traditional truth tables have Level 1 information only: correlations of the values, and both the direction (what values depend on what other values?) and nature (which relation/function is it?

	4.1 ...truth tables?

•

  Ceteris paribus/closed-world condition: The existence of a causal relation doesn't entail the occurrence of the result, and no need to rule out irrelevant possibilities • Interaction is easy to represent • Entrainment is easy to represent, bringing activities into alignment with causal treatments of accomplishments • Better than mere dependency (e.g. Lewis 1973) in accounting for direct/indirect distinctions • We can represent goal-directed action (intention node)