Andrew Glennerster
email: a.glennerster@reading.ac.uk

Miles E Hansard

Andrew W Fitzgibbon

View-Based Approaches to Spatial Representation in Human Vision

In an immersive virtual environment, observers fail to notice the expansion of a room around them and consequently make gross errors when comparing the size of objects. This result is difficult to explain if the visual system continuously generates a 3-D model of the scene based on known baseline information from interocular separation or proprioception as the observer walks. An alternative is that observers use view-based methods to guide their actions and to represent the spatial layout of the scene. In this case, they may have an expectation of the images they will receive but be insensitive to the rate at which images arrive as they walk. We describe the way in which the eye movement strategy of animals simplifies motion processing if their goal is to move towards a desired image and discuss dorsal and ventral stream processing of moving images in that context. Although many questions about view-based approaches to scene representation remain unanswered, the solutions are likely to be highly relevant to understanding biological 3-D vision.

Is Optic Flow Used for 3-D Reconstruction in Human

Vision?

Optic flow, or motion parallax, provides animals with information about their own movement and the 3-D structure of the scene around them. Throughout evolution, motion is likely to have been more important for recovering information about scene structure than binocular stereopsis, which is predominantly used by hunting animals who are required to remain still. The discrimination of 3-D structure using motion parallax signals is known to be highly sensitive, almost as sensitive as for binocular stereopsis [START_REF] Rogers | Similarities between motion parallax and stereopsis in human depth perception[END_REF][START_REF] Bradshaw | The interaction of binocular disparity and motion parallax in the computation of depth[END_REF]. Motion parallax has also been shown to provide an estimate of the viewing distance and metric shape of objects, when combined with proprioceptive information about the distance the observer has travelled [START_REF] Bradshaw | The interaction of binocular disparity and motion parallax in deptermining perceived depth and perceived size[END_REF][START_REF] Bradshaw | The task-dependent use of binocular disparity and motion parallax information[END_REF]. An important and unsolved challenge, however, is to understand how this information is combined into a consistent representation as the observer moves through a scene. In computer vision, the problem is essentially solved. Photogrammetry is the process of using optic flow to recover information about the path of the camera, its internal parameters such as focal length and the 3-D structure of the scene [START_REF] Fitzgibbon | Automatic camera recovery for closed or open image sequences[END_REF][START_REF] Hartley | Multiple view geometry in computer vision[END_REF]. For most static scenes the process is robust, reliable and geometrically accurate. It does not suffer from the systematic spatial distortions that are known to afflict human observers in judgements of metric shape and distance [START_REF] Foley | Binocular distance perception[END_REF][START_REF] Gogel | A theory of phenomenal geometry and its applications[END_REF][START_REF] Johnston | Systematic distortions of shape from stereopsis[END_REF][START_REF] Tittle | A hierarchical analysis of alternative representations in the perception of 3-D structure from motion and stereopsis[END_REF][START_REF] Glennerster | Stereoscopic depth constancy depends on the subject's task[END_REF]. Most approaches to visual navigation in robotics are based on a partial reconstruction of the 3D environment, again using photogrammetric principles [START_REF] Basri | Visual homing: Surfing on the epipoles[END_REF][START_REF] Davison | Real-time simultaneous localisation and mapping with a single camera[END_REF][START_REF] Newman | SLAM-loop closing with visually salient features[END_REF].

It is questionable whether human vision is designed to achieve anything like photogrammetric reconstruction. Gibson [START_REF] Gibson | The ecological approach to visual perception[END_REF], for example, argued strongly against the idea that vision required explicit internal representation, but did not provide a clear description of an alternative [START_REF] Ullman | Against direct perception[END_REF]. Similar arguments are still popular [START_REF] O'regan | A sensori-motor account of vision and visual consciousness[END_REF] but are no more computationally precise. In fact, although there have been some attempts at generating non-metric representations for navigation in robots (see Section 3), there is as yet no well-developed rival to 3-D reconstruction as a model for representing the spatial layout of a scene, either in computer vision or models of human vision.

An Expanding Room

In our Virtual Reality Laboratory, we have developed a paradigm in which information about the distance of objects from vergence, motion parallax and proprioception conflict with the assumption of scene stability. Observers wear a head mounted display that has a wide-field of view and high resolution. In an immersive virtual environment, they make judgements about the relative size or distance of objects, allowing us to probe the representation of space generated by the visual system and to assess the way in which different cues are combined. Figure 1 illustrates one experiment [START_REF] Glennerster | Humans ignore motion and stereo cues in favour of a fictional stable world[END_REF]. As the observer moves from one side of the room to the other, the virtual room expands around him/her. Remarkably, almost all observers fail to notice any change in size of the room even though the room expands by a factor of 4. Even with feedback about the true size of objects in the room, they fail to learn to use veridical stereo and motion parallax cues appropriately [START_REF] Glennerster | Humans ignore motion and stereo cues in favour of a fictional stable world[END_REF][START_REF] Rauschecker | Stereo and motion parallax cues in human 3d vision: Can they vanish without trace[END_REF].

Our data suggest a process of scene representation that is very different from photogrammetry. For example, Figure 1b shows data for 5 human observers who were asked to judge the size of a cube seen on the right of the room (when the room is large) compared to a reference cube shown on the left of the room (when the room is small). The reference cube disappears when the observer walks across the room, so they must remember its size. Participants make a binary, forced choice judgement on each trial about the relative size of the reference and comparison cubes. Over many trials, it is possible to deduce the size of the comparison cube that they would match with the reference cube. As Figure 1b shows, this size varies with the viewing distance of the comparison cube: at 6m participants choose a match that is almost four times as large as the reference cube, i.e. almost equal to the expansion of the room, whereas at 1.5m they choose a more veridical match. Fig. 1. An expanding virtual room. a) As the observer walks from one side of the room to the other the virtual scene around them expands. The centre of expansion is mid-way between the eyes so the view from that point is unaffected. This stimulus separates purely visual cues to the 3-D structure of the scene from others, such as proprioception. b) Data from an experiment [START_REF] Glennerster | Humans ignore motion and stereo cues in favour of a fictional stable world[END_REF] in which participants matched the size of a cube visible when the room was large with one visible only when the room was small. The correct matched size ratio is 1 while a size ratio of 4 is predicted if participants judged the cube size relative to the room or texture elements. Data for 5 participants is shown (solid symbols) and for a simulation using Figure 1b also shows the predicted responses from a 3-D reconstruction package (Boujou [20]) supplied with the same images that a participant would see as they carried out the task. In this case, we provided one veridical baseline measure, which sets the scale of the reconstruction, so the height of the simulation data on the y-axis is not informative. The important point, though, is that there is no variation of predicted matching size with viewing distance for the 3-D reconstruction, unlike the pattern of the human data.

In some ways, it is not difficult to explain the human observers' behaviour. The fitted curve in Figure 1b illustrates the predictions of a cue combination model. Matches for a distant comparison object are close to that predicted by a texturebased strategy (judging the cube size relative to the bricks, for example), since here stereo and motion parallax cues to distance are less reliable and hence are given a lower weight [START_REF] Svarverud | Absolute and relative cues for location investigated using immersive virtual reality[END_REF]. At close distances, stereo and motion parallax are more reliable (they support lower depth discrimination thresholds when compared to texture cues) and hence have a greater effect on matched sizes [START_REF] Ernst | Humans integrate visual and haptic information in a statistically optimal fashion[END_REF][START_REF] Glennerster | Humans ignore motion and stereo cues in favour of a fictional stable world[END_REF][START_REF] Svarverud | Absolute and relative cues for location investigated using immersive virtual reality[END_REF]. However, combining cues in this way is very specific to the task. It does not imply that there should be related distortions in other tasks, for example those requiring the comparison of two simultaneously visible objects.

The difficulty with a task-specific explanation, such as the cue combination model in Figure 1b, is that it avoids the question of scene representation in the human visual system. It is all very well to say that the world can act as 'an outside memory', and be accessed by looking at the relevant parts of the scene when necessary [START_REF] O'regan | Solving the real mysteries of visual perception: The world as an outside memory[END_REF][START_REF] O'regan | A sensori-motor account of vision and visual consciousness[END_REF]. There must be a representation of some form to allow the observer to turn their head and eyes or walk to the right place to find the information. The representation may not be a 3-D reconstruction, but it must nevertheless be structured. There is as yet no satisfactory hypothesis about what that representation might be but the proposal in this paper is that it may have similarities to 'view-' or 'aspect-graphs', which are introduced in the next section.

Moving between Views

An alternative to 3-D reconstruction has been described in relation to navigation, in which a robot or animal stores views or 'snapshots' of a scene and records something about the motor output required to move between one view and the next, without integrating this information into a Cartesian map. The snapshots are the nodes in a 'view graph' and the movements are the edges [START_REF] Schölkopf | View-based cognitive mapping and path planning[END_REF][START_REF] Koenderink | The internal representation of solid shape with respect to vision[END_REF] (see Section 3). One possible explanation of people's perceptions in the expanding room is that, as they move across a room, observers generally have an expectation of the images that they will receive: the fact that they do not notice that they are in an expanding room implies that they are relatively insensitive to the rate at which those images arrive as they walk. This is illustrated in Figure 2a, where the grey plane represents a manifold of all the images that could be obtained by a person walking around the room. Each point on the plane represents one image and neighbouring points represent images visible from neighbouring vantage points. The dotted line shows, schematically, the set of images that a monocular observer might receive as they walk from the left to the right of the room. Potentially, that eye could receive exactly the same set of images whether the room was static or expanding: there is no way to tell from the monocular images alone. However, as Figure 2a shows, the rate at which new images arrive for each step the observer takes is different in the expanding room. On the left, where the room is small, new images arrive rapidly, whereas on the right they do so more slowly. Proprioceptive signals from the muscles provide information about the distance walked and these should be sufficient to tell observers about the size of the room. But in the expanding room, observers seem to disregard this information when it conflicts with the assumption that the room is stable, at least in determining their subjective experience of the room.

One attractive aspect of the view graph idea is its close similarity to known biological mechanisms. Moving from image to image or, more generally, from sensory state to sensory state, with an expectation of the sensory input that will be received as a consequence of each movement, is a familiar biological operation associated with well established circuitry, notably in the cerebellum [START_REF] Marr | A theory of cerebellar cortex[END_REF][START_REF] Albus | A theory of cerebellar function[END_REF][START_REF] Miall | Is the cerebellum a Smith predictor?[END_REF]. The same cannot be said of operations required for general 3-D coordinate transformations, particularly translations of the origin. There are no detailed suggestions about how such operations might be achieved in the brain.

On the other hand, Figure 2b illustrates operations that are known to take place in the brain and it looks very similar to Figure 2a. Now the grey plane illustrates a high dimensional space of potential sensory (and motivational) states.

Each black dot represents a potential sensory+motivational context that will lead to a particular motor output. The white dot represents the current sen-sory+motivational context, which is recognised as most similar to one of the stored contexts. The match leads to an action, whose consequence is a new sen-sory+motivational context, and so the movement progresses.

From this perspective, one can suggest a potential explanation for the fact that observers see the expanding room as static. Even though the proprioceptive signals are different in the two cases, the powerful visual feedback is exactly what is expected in a static room and hence, overall, the sensory context is sufficiently similar for it to be recognised as the expected context rather than an alternative one. In terms of Figure 2b, the path through sensory+motivational space as observers walk across the room is very similar, despite the different proprioceptive inputs, and hence their subjective experience is too.

It is not always true that the proprioceptive input is ignored. When observers move from a small room to a clearly separate large room their size judgements are much better than in the expanding room [START_REF] Glennerster | Humans ignore motion and stereo cues in favour of a fictional stable world[END_REF]. In that case, when confronted with a new room of unknown size, there is no strong expectation about the sensory feedback they will receive and so the proprioceptive information becomes decisive. Even in the expanding room, proprioceptive (and stereo) input contributes to some judgements while not affecting observers' subjective impression of the room size [START_REF] Glennerster | Humans ignore motion and stereo cues in favour of a fictional stable world[END_REF][START_REF] Rauschecker | Stereo and motion parallax cues in human 3d vision: Can they vanish without trace[END_REF][START_REF] Svarverud | Absolute and relative cues for location investigated using immersive virtual reality[END_REF]. Overall, the results in the expanding room suggest a representation that is looser, less explicit and more task-dependent than a 3-D reconstruction.

Biological Input and Output of Optic Flow Processing

For photogrammetry, the goal is clear: compute the 3-D structure of the scene and the path of the camera through it. Corresponding features are identified across frames, then the most likely 3-D scene, camera movement and camera intrinsic parameters (such as focal length) are computed given the image motion of the features. The camera motion is described in the same 3-D coordinate frame as the location of the points. The process can be written as:

x i j = P i X j (1)
where x gives the image coordinates of points in the input frames, X describes the 3-D locations of the points in the scene and P is the projection matrix of the camera. P includes the intrinsic parameters of the camera, which remain constant, and the extrinsic parameters (camera pose) which are the only things that change from frame to frame. Equation 1 applies to the j th point in the i th frame.

In computer vision applications, the camera is generally free to translate and rotate in any direction. This is not true in biology. The 6 degrees of freedom of camera motion are essentially reduced to 3. Animals maintain fixation on an object as they move and are obliged to do so for at least 150 ms (this the minimum saccadic latency) [START_REF] Carpenter | Movements of the eyes[END_REF]. Although the eye can rotate around 3 axes and the optic centre can translate freely in 3 dimensions, the rotation and translation of the eye are tightly coupled so that for each translation of the optic centre there is a compensatory rotation of the eye. Two dimensions of rotation maintain fixation while the third restricts the torsion of the eye with respect to the scene. This means that as a monocular observer moves through a static environment, maintaining fixation on an object, there is only one image that the eye can receive for each location of the optic centre.

A similar pattern of eye movements predominates throughout the animal kingdom and must clearly carry an evolutionary benefit. Land [START_REF] Land | Why animals move their eyes[END_REF] has documented eye movement strategies in many species. Animals fixate while moving, then they make a 'saccade' or rapid rotation of the eyes and fixate a new target as they continue to move. They do this even when their eyes are fixed within the head, as is the case in many insects. In a rare condition in which the eye muscles become paralysed, patients' eyes are fixed with respect to their head [START_REF] Gilchrist | Saccades without eye movements[END_REF]. In these cases, the head makes a rapid rotation between periods of fixation, so their visual diet is similar to that of a person with freely moving eyes. And when the eye and head are fixed with respect to the body, for example in a hover fly, the whole body makes saccades.

Why should animals do this? The constraint that eye rotation and translation are tightly linked does reduce the number of free parameters and so can help in the estimation of camera motion and scene reconstruction [START_REF] Aloimonos | Active vision[END_REF][START_REF] Bandopadhay | Egomotion perception using visual tracking[END_REF][START_REF] Sandini | Active tracking strategy for monocular depth inference over multiple frames[END_REF][START_REF] Daniilidis | Fixation simplifies 3D motion estimation[END_REF]. However, the rationale we describe here is different. If the goal is to navigate across a manifold of images, as we discussed in relation to Figure 2, then the restrictive pattern of eye movements that animals adopt makes sense. We illustrate this claim in the next sections by considering the role of the dorsal and ventral streams of visual processing during a simple set of movements. The cross marks the view that would be obtained if the viewer were at the location of the fixated object.

Dorsal Stream

Figure 3 shows a simple sequence of head and eye movements to illustrate a possible role of the dorsal stream in controlling head movements. In Figure 3a, the observer moves towards a fixated target. A fast, tight loop from retina to eye muscles keeps the eye fixating as the observer moves. This circuit can even bypass cortical motion processing [START_REF] Cohen | The nucleus of the optic tract: Its function in gaze stabilization and control of visual-vestibular interaction[END_REF]. The obligation to maintain fixation imposes a tight constraint on the type of image changes that can arise. The plane shown on the right in Figure 3a represents the set or manifold of images that can be obtained when the eye fixates a particular object. Each point on the manifold represents one image. As the observer approaches the fixated object, the current image (black dot) is shown moving towards the view that would be obtained if the observer were at the location of the object (shown by the cross). The retinal flow created by this movement is approximately radial expansion outwards from the fovea. There are many neurons in MSTd (the dorsal part of the medial superior temporal cortex, in the dorsal stream) that are sensitive to flow of this type [START_REF] Saito | Integration of direction signals of image motion in the superior temporal sulcus of the macaque monkey[END_REF][START_REF] Perrone | A model of self-motion estimation within primate extrastriate visual cortex[END_REF]. 3, each surface of images comprises the images that can be obtained by the observer while they fixate on a particular object -in this case a woman and a man.

Figure 3b illustrates in the same way a saccade followed by a lateral movement of the observer. Fixating a new object moves the current image to a new surface of images. The observer then moves laterally, staying the same distance from the fixation point. This produces a quite different pattern of retinal flow in which objects that are closer than the fixation point (shown as open symbols) move one way on the retina while more distant objects move in the opposite direction (closed symbols). The distinction between objects in front of and behind the fixation point can be signalled by disparity (crossed versus uncrossed) and indeed this seems to be a trick the visual system uses to disambiguate the flow. The same area that contains neurons responsive to expansion, MSTd, also contains neurons responsive to one direction of motion when the stimulus disparity is crossed and the opposite direction of motion when the stimulus disparity is uncrossed [START_REF] Roy | The role of disparity-sensitive cortical neurons in signalling the direction of self-motion[END_REF]. These neurons are ideally suited to signalling the type of flow that occurs when the observer moves his or her head laterally while fixating an object.

The two components of observer motion shown in Figure 3a and b can be detected independently. The neurons sensitive to forward motion can be used as a signal of progress towards the goal in Figure 3a with the neurons sensitive to lateral motion signalling error; the role of these types of neurons can then be reversed to control the lateral motion shown in Figure 3b. Greater calibration would be required to move on a path between these extremes, but the principle of using these two components remains [START_REF] Glennerster | Fixation could simplify, not complicate, the interpretation of retinal flow[END_REF] and the simplicity of the control strategy relies on the restriction that the observer fixates on a point as he or she moves. This is a quite different hypothesis about the role of dorsal stream neurons from the idea that they contribute to a computation of scene structure and observer heading in the same coordinate frame [START_REF] Perrone | A model of self-motion estimation within primate extrastriate visual cortex[END_REF].

Ventral Stream

The ventral stream of visual processing has a complementary role. Neurons in this part of the visual pathway provide, as far as possible, a constant signal despite the observer's head movements, indicating which object the observer is looking at (i.e. which surface the current image is on) in contrast to the dorsal stream which signals how the observer is moving in relation to the fixated object, independent of the identity of that object.

As illustrated in Figure 4, it is possible to show on the surface of images the 'receptive fields' of different types of ventral stream neurons. Rather than a receptive field on the retina, here we mean the set of images to which this neuron would respond. The dark grey patch on the left side of Figure 4 shows a hypothetical 'receptive field' for a size-independent cell [START_REF] Rolls | Size and contrast have only small effects on the responses to faces of neurons in the cortex of the superior temporal sulcus of the monkey[END_REF]: it would respond to an object from a range of viewing distances. The overlapping lighter grey patch shows the receptive field of a view-independent cell [START_REF] Booth | View-invariant representations of familiar objects by neurons in the inferior temporal cortex[END_REF], which responds to the same object seen from a range of different angles.

Combining both of these types of invariance, one could generate a receptive field that covered the entire surface of images. In other words the neuron would respond to the view of a particular object from a wide range of angles and distances. Neurons with this behaviour have been reported in the hippocampus of primates [START_REF] Georges-Francois | Spatial view cells in the primate hippocampus: allocentric view not head direction or eye position or place[END_REF]. The hippocampus, which lies at the end of the ventral stream, contains a large auto-association network [START_REF] Treves | Computational analysis of the role of the hippocampus in memory[END_REF] which has the effect of providing a constant output despite moderate changes in the input. Thus, there are mechanisms throughout the ventral stream that encourage a stable output to be maintained for the period of fixation. In terms of the surface of images, the argument here is that the ventral stream is designed to determine which surface of images contains the current image and the dorsal stream is designed to determine how the current image is moving across it.

So far, we have only considered the example of a short sequence of head and eye movements. In section 3, we discuss view graphs and how these can provide an extended spatial representation of observer location.

Representing Observer Location

A view graph (or aspect graph) representation consists of discrete reference views which form the nodes of the graph. In some simple examples, these have been the views at junctions in a maze [START_REF] Gillner | Navigation and acquisition of spatial knowledge in a virtual maze[END_REF], while in more realistic cases they have been the views at various locations in an open environment [START_REF] Franz | Biomimetic robot navigation[END_REF]. The edges of the graph are the movements that take the observer from one view to the next. A view graph is not a continuous representation of space (in contrast to the coordinate frame of X in Equation 1) but, when the observer is not at a location from which a reference view was taken, it is still possible to determine which of the reference views is most similar to the current view. This allows space to be divided up into regions, as the example below illustrates.

Figure 5 illustrates this principle for a 2-dimensional world. Suppose the room contained 3 objects as shown and that a 'view' consisted of a list of the angles subtended at the optic centre by each pair of objects. Figure 5a shows two reference locations, r 1 and r 2 , and the angles subtended between each of the three pairings of objects at a third, arbitrary location. The colours show a hypothetical division of the room into regions where the view is most like that at r 1 (green) and most like the view at r 2 (red). Figure 5b and c illustrate a specific example of this principle. There are now more than three objects, whose locations are shown by yellow triangles, but still two reference locations. The plots show the likelihoods that the view at each location is a noisy version of the view r 1 or r 2 , computed as follows. Scene features at positions (x p , y p ), p = 1, . . . , N, are imaged from viewpoint (x, y), and represented by the visual angles: ρ p,q (x, y) = ρ1,q(x, y), . . . , ρp,q(x, y), . . . , ρN,q(x, y) , q = 1, . . . , N,

(table,fruitbowl)
(

where ρ p,q is the angle between points (x p , y p ) and (x q , y q ). In order to avoid the use of a distinguished reference point, we compute all N 2 possible angles from each viewpoint. As well as general views ρ(x, y), we have R different reference views, ρ p,q (x1, y1), . . . , ρ p,q (xr, yr), . . . , ρ p,q (xR, yR) , q = 1, . . . , N,

taken from distinct locations (x r , y r ). Each reference view ρ(x r , y r ) is accompanied by a corresponding list of variances, σ 2 (x r , y r ):

ρ p,q (xr, yr) = ρ 1,q (x r , y r ), . . . , ρ p,q (x r , y r ), . . . , ρ N,q (x r , y r ) , q = 1, . . . , N, (4) σ 2 p,q (x r , y r ) = σ 2 1,q (x r , y r ), . . . , σ 2 p,q (x r , y r ), . . . , σ 2 N,q (x r , y r ) , q = 1, . . . , N.

(

In our simulations, we take σ 2 p,q = 1 in all cases. The fit of the r th reference view to the visual angles obtained at observer position (x, y) is defined as the squared difference between ρ(x, y) and ρ(x r , y r ), summed over all angles,

Er(x, y) = N q=1 N p=1 1 σ 2
p,q (xr, yr) ρp,q(x, y) -ρp,q(xr, yr) 2 .

(6)

We use E r to compute the likelihood of the current view ρ(x, y), under the hypothesis that the viewpoint coincides with that of model view ρ(x r , y r ). Specifically, we represent the likelihood as L ρ(x, y) | ρ(xr, yr) = e -Er (x,y) , [START_REF] Foley | Binocular distance perception[END_REF] which is proportional to the probability of the r th location-hypothesis:

P x = xr, y = yr | ρ(x, y) ∝ e -Er (x,y) . ( 8
)
The normalizing constant is obtained by integrating P over all viewpoints, (x, y).

The figures plot the maximum likelihood at each point (x, y), colour-coded by the reference location r for which L ρ(x, y) | ρ(x r , y r ) is maximum. If the visual system represents places using angles subtended between objects in a similar way, then fixating on an object as the observer moves is a sensible strategy. It allows changes in angles (at least, those with respect to the fixated object) to be monitored easily as the observer moves. Figure 6 illustrates the idea. If the optic centre, O, of the eye/camera is at the location marked by the blue cross and the fixation point is F , then the eccentricity of a point, P , imaged on peripheral retinal is the same as the angle (ρ) between that point and the fixation point subtended at the optic centre. If the observer maintains fixation on the same point, then change in eccentricity of the peripheral object signals change in the angle ρ, Figure 6c. This retinal flow, Δρ, is directly relevant to computing changes in likelihood that the current location is r 1 or r 2 , as Figures 6b andd illustrate.

The example in Figures 5 and6 is limited to control of movement between two reference locations. Figure 7 shows a larger set of reference locations chosen by a robot, equipped with an omnidirectional camera, at which it took 'snapshots' or reference views as it explored a real scene [START_REF] Franz | Learning view graphs for robot navigation[END_REF]. New snapshots were constrained to differ (above a criterion level) from those already stored. In general, views are connected as the nodes in a view graph, where the edges connect 'neighbouring' views. New edges were formed by the robot as it explored: as it left one reference location it compared its current view to all other 'snapshots' it had stored and then used a homing strategy [START_REF] Cartwright | Landmark learning in bees: experiments and models[END_REF][START_REF] Hong | Image-based homing[END_REF] to approach the reference location with the most similar view. The edges in the view graph contained no information about the direction or distance between two vertices, only that they were neighbours. This experiment illustrates how the idea of reference views can be extended over a large (potentially limitless) spatial range. Equally, of course, the resolution of the representation can be increased within a particular region of space by including more reference locations there. One can imagine many situations in which fine distinctions between different observer locations are important in one part of a scene but less important in others.

View graphs, then, provide a way of extending the ideas about control of observer location to a wider region of space. Section 2 dealt only with observer movement relative to the fixation point, and saccades. Now these can be seen as examples of moving over a small part of an infinitely extendible view graph. However, view graphs do not directly answer the question raised in section 1 about scene representation. For example, in Figure 7, how might the location of the objects (rather than the robot) be represented? The grey object appears in many views, including the snapshots from all the reference locations shown by filled symbols. Is there a sensible coordinate frame in which to unite all the information about an object's location, other than a 3-D world-based frame? Current implementations do not do so and it is not immediately obvious how it should be done. One might argue that in simple animals the distinction does not exist: for an ant to know where an object is located is much the same as knowing how to arrive at it. The same may be true for humans if an object is a long way away (like the Eiffel tower). In that case, the represented location of an object can be a node in the view graph. However, this only avoids the question of uniting estimates of object location across views, which is relevant at closer distances for actions such as reaching or pointing. Humans can point to an object that is not currently visible and can do so in a way that accounts for their rotations and translations since they last saw it. They can also direct their hand to an object that they have seen in peripheral vision but never fixated, even when they make a saccade before pointing at it (although they make systematic error in this case [START_REF] Henriques | Gazecentered remapping of remembered visual space in an open-loop pointing task[END_REF]). These capacities, however poorly executed, are evidence of a representation of object location that is maintained across head and eye movements. View graphs, or some similar view-based approach, must clearly be able to support similar behaviour if they are to become candidate models of human spatial representation.

Conclusion

We have argued that the human perception of space in an expanding virtual room may be understood in terms of a representation like a view graph, or a manifold of images, where the observer has an expectation of the images they will receive as they move across the room but are insensitive to the discrepancy between visual and proprioceptive feedback. It is clear that view graph models are currently inadequate in many ways but we have argued that an explanation along these lines is more likely to explain human perception in the expanding room than one based on 3-D reconstruction. It is clear from the expanding room and many other perceptual phenomena that the development and successful implementation of non-Cartesian, non-metric representations will be of great relevance to the challenge of understanding human 3-D vision.

  Fig.1. An expanding virtual room. a) As the observer walks from one side of the room to the other the virtual scene around them expands. The centre of expansion is mid-way between the eyes so the view from that point is unaffected. This stimulus separates purely visual cues to the 3-D structure of the scene from others, such as proprioception. b) Data from an experiment[START_REF] Glennerster | Humans ignore motion and stereo cues in favour of a fictional stable world[END_REF] in which participants matched the size of a cube visible when the room was large with one visible only when the room was small. The correct matched size ratio is 1 while a size ratio of 4 is predicted if participants judged the cube size relative to the room or texture elements. Data for 5 participants is shown (solid symbols) and for a simulation using a 3-D reconstruction package (open symbols, see text). The curve shows the predictions of a cue combination model. The dotted line shows the predicted responses if participants matched the retinal size of the cubes. Figure reproduced, with permission, from [18] c Elsevier.

Fig. 2 .

 2 Fig. 2. Possible images in an expanding room. a) The grey plane symbolises a manifold of views of the room shown in figure 1. Each point corresponds to a single view and neighbouring points correspond to views from neighbouring vantage points. The current view is indicated by the white dot. The dashed line indicates the set of views that a hypothetical observer receives as they walk across the room. The manifold of views for the expanding room is identical to that for a static room: the only difference is that when the observer is on the right hand side of the room, where it is large, views change at an unusually slow rate as the observer walks while on the left side of the room, where the room is small, the views change unusually fast. b) The grey plane now symbolises a manifold of potential sensory (or sensory+motivational) states with the current state indicated by the white dot. The state includes visual and proprioceptive input. The black dots indicate stored states. Both the current and the stored states can be described by points in the same high dimensional space.

Fig. 3 .

 3 Fig.3. Moving across a surface of images. a) An observer is shown in plan view moving towards a fixated object. The pattern of flow on the retina is illustrated and, on the right, the movement of the current image across a surface of images, similar to that shown in Figure2a. b) The observer makes a saccade to a new fixation point which is illustrated by the current image jumping to a new surface of images (upper surface). The observer then moves laterally while maintaining fixation on the same object. Objects with a crossed disparity (open symbols) move in one direction in the image while those with an uncrossed disparity (filled symbols) move in the opposite direction. The cross marks the view that would be obtained if the viewer were at the location of the fixated object.

Fig. 4 .

 4 Fig.4. Ventral stream neuron receptive fields. Neurons in the ventral stream respond selectively to certain objects or complex features while showing considerable invariance to size or viewing direction. The dark grey region illustrates a 'receptive field' of a size-invariant neuron, i.e. the set of images to which it might respond. The lighter grey 'receptive field' shows the set of images to which a view-invariant neuron might respond. As in Figure3, each surface of images comprises the images that can be obtained by the observer while they fixate on a particular object -in this case a woman and a man.

Fig. 5 .

 5 Fig. 5. Reference locations. a) Two reference locations r1 and r2 are shown and three objects, with the angle subtended between pairs of objects shown, e.g. ρ (table,fruitbowl) . The green and red areas indicate a possible division of the room into locations where the best answer to the question 'Where am I?' would be r1 and r2 respectively. b) Plan view of a scene in which the location of objects are marked by yellow triangles. Contour plots show the likelihoods that the observer is at the two candidate locations, colour coded according to which of the two likelihoods is greater. c) Same as b) but showing log likelihoods.

Fig. 6 .

 6 Fig. 6. Retinal flow and fixation. a) The observer's location is marked by the blue target and the fixated object is shown by a yellow triangle, F . Two other objects imaged on peripheral retina are shown, one of which (P ) is at an eccentricity of ρ. Movement of the observer causes a change in the angles between pairs of objects subtended at the optic centre (c and d). Because the observer maintains fixation as he or she moves, retinal flow provides a straight-forward record of the changes in these angles with respect to the fixation point. For example, as shown in c), the change in the angle ρ to ρ + Δρ results in flow, Δρ, at the point on the retina where the object P is imaged.The change in ρ (and other angles) with respect to the fixation point can be used to determine whether the current view is becoming less like that from reference location r1 and more like that from r2 (b and d).

Fig. 7 .

 7 Fig. 7. View graph of a scene. Circles show the locations of reference views (or 'snapshots'), which are the nodes in the view graph (adapted from a figure in Franz et al [47], with permission). The filled circles indicate the reference views that include an image of the grey object. It is not clear how best to define the location of objects in the view graph.

Acknowledgements

Funded by the Wellcome Trust. We are grateful to Bruce Cumming, Andrew Parker and Hanspeter Mallot for helpful discussions.


Ozgun Pinarer
email: opinarer@gsu.edu.tr

Framework for Heterogeneous Sensor Data Stream Management with BIM and WSN

Keywords:

This paper focuses on major challenge posed by merging WSN technology with BIM.

• A novel approach is proposed to design and manage a smart building environment.

• Sensor data is processed to reflect the real time monitoring on the 3D model.

INTRODUCTION

Use of smart technology has an important impact on enhancing the living/working environment. Designing a smart building requires an attentive interdisciplinary work. In a traditional building construction process, electrical, urban, environment, mechanical engineers, and an architect work in common to proceed the project. Besides, in the smart building construction process, computer engineers are also involved. With this interdisciplinary team, a new research domain has born: Building Information Modelling (BIM).

With the great improvement in the small size sensor technology and 3D modelling, BIM represents one of the most popular research areas from different domains. BIM provides a shared platform on which every participant may work on the 3D model at the same time without interrupting each other. Such working process provides a great working environment for agile teams. To achieve these benefits, IT-based knowledge and systems are highly necessary during the construction process of a smart building. Besides, reducing energy consumption of any environment requires a monitoring and management system which can be carried through by deploying wireless sensor devices and actuators. Deploying such equipment for managing the environment requires a highly developed framework built on suitable infrastructure. Managing wireless sensor devices and communication protocols are well-known topics in the literature. However, merging WSN technology with BIM, integration of such IoT based devices require a novel approach to merge various independent systems about designing and managing a smart building environment [START_REF] Isikdag | Building Information Models: An Introduction[END_REF].

The rest of the paper is organized as follows: Requirement and importance of such integration is presented in Section 2. Existing studies and more relevant research are given in Section 3. Section 4 presents the approach and system architecture. Besides, it gives details about the proposed approach, and defines all the components, tools and services integrated into the framework. Built environment and testbed are described in Section 5. Finally, conclusion and discussion are given in Section 6.

IMPORTANCE OF BUILDING INFORMATION MODELLING IN CONSTRUCTION

Intelligence and sustainability are two concepts that complement each other in this domain. Besides, high energy consumption of buildings is our major challenge to solve. It is proved that a significant part of the energy sources is consumed by the traditional buildings. For efficient use of resources on earth, many studies and research are performed on traditional buildings for energy harvesting and applying energy saving methods, and a new dimension has been brought to construction standards on behalf of sustainable: Building Management and Monitoring Systems via BIM. Such smart building management systems aim to increase the living standards of the people living in the buildings and to increase the living standards, as well as to improve energy monitoring and energy consumption. Since earth resources will end one day, there is always a need for smart systems that care about energy resources and try to minimize consumption as much as possible in order to leave a livable world for future generations. Figure 1 presents the awareness and the usage of BIM in real life. Due to the energy concern, in many developed countries, using such technologies in the building construction process becomes compulsory.

Figure 1. Awareness and usage of BIM [2]

LITERATURE REVIEW

As mentioned, BIM is an interdisciplinary research area in which various researchers from different domains work to get the benefits of BIM and to integrate BIM workflow to their internal work processes. Detailed literature review of BIM and its usage in different domains are deeply presented in [START_REF] Corekci | Designing a novel smart home monitoring systems with the integration of bim and wsn[END_REF]. As indicated there, BIM is used not only to manage smart building environments but also, it is commonly used in early stages of construction environments. Such environments can be very critical in terms of workers' health. BIM is highly important to prevent from industrial accidents.

Table 1 presents the enabled technologies and most popular platforms in this research domain. Current studies in the literature suggest local and limited solutions. Besides, most of the proposed approaches are prototypes and are not suitable for automation management systems. It is quite hard to adopt these approaches to real life scenarios. Moreover, in these studies, data management of multiple sensor nodes and actuators are not addressed, and these are limited to predefined building practices. As a concrete solution, in this study, we present a complete architecture and a well-developed framework for a smart building environment that handles the data from its acquisition till its usage in third party applications running on our framework. As indicated in several studies, existing BIM applications and implementation still require more experiences and concrete solutions about data sharing policies, insufficient collaboration, and integration.

These studies are specialized solutions and application-based approaches however a proposed solution should be generic and be able to fit to different environments. Moreover, the BIM concept, existing studies on IoT and BIM integration commonly intent energy saving and finding energy harvesting solutions [START_REF] Wu | A visual and persuasive energy conservation system based on BIM and IoT technology[END_REF]. Not only IoT or smart technologies but Industry 4.0 approaches are also involved in this domain. Concept of Industry 4.0 focuses on the design stage of construction in terms of using renewable and energy-efficient materials or technologies. Intention of Industry 4.0 in BIM is to increase and improve the productivity level, safety, and quality of the construction [START_REF] Maskuriy | Industry 4.0 for the construction industry-how ready is the industry?[END_REF]. VFEA -dynamic and holistic virtual assessment of the energy performance of buildings

Usage of BIM and WSN in cultural heritage

BIM is also preferred in protecting the cultural heritage of every country. As there are very few researchers and research teams working in this field, it appears to be a highly open field for development. Within the scope of cultural heritage, historical buildings and interior decorations are evaluated within this scope. For a concrete living example, due to the fire in January 2013 in the historical palace building at the Galatasaray University Ortaköy campus, many historical paintings and wall decorations inside the building have disappeared. In addition, the 675-year-old Notre Dame Cathedral in Paris, France, was heavily damaged by fire in April 2019. Such structures are structures that should be passed down from generation to generation as the common heritage of humanity and should be preserved to the finest detail. These developments and research are referred to as Historical Building Information Modeling (HBIM).

In this sense, BIM offers a digital solution to prevent such structures from being lost. BIM, by definition, is the technology to digitize the 3D detailed model of the media. In this way, detailed information of each object in the environment is recorded. As BIM can provide detailed 3D models of buildings within the scope of cultural heritage, it is a very important resource in repair/reconstruction etc. after accident cases such as fire.

Lee et al. proposes a new metadata to facilitate the digitization of cultural heritage structures in their work [START_REF] Lee | Context-aware risk management for architectural heritage using historic building information modeling and virtual reality[END_REF]. With the new data structure presented, it is thought that real time calculations can be made on the risk management of the building in the virtual environment. Logothetis et al. discusses the use and evolution of building information modeling in the field of cultural heritage [START_REF] Logothetis | Building information modelling for cultural heritage: a review[END_REF].

PROPOSED APPROACH

In this study, a novel framework that benefits from building model and wireless sensor nodes for a smart building environment is presented. As mentioned before, to manage such a pervasive environment like a smart building environment, various components should be merged to achieve a full automated management system.

System Design

Our proposed declarative architecture is composed of three principal layers as illustrated in Figure 2.

Besides, on the same figure, sub layers of the developed framework are also introduced. This framework is developed not only for a smart building environment but can be easily adapted to any automation system for a pervasive environment willing to be monitored and managed.

Figure 2. Generic architecture of the proposed approach

Application Layer: This layer is responsible for the connection between the user and the proposed framework. Requests from the user are managed in this layer and transmitted by converting them to the lower layers. Likewise, information from the lower layers is extracted and served to the relevant users or parts. Visualization and the reporting issues are processed in this layer. This layer stands for the graphical user interface of the framework that interacts with the users.

Automation Management Layer: This layer is the core of our approach and the developed framework.

Each critical action takes place on this layer. This layer has a precision role that connects the upper and lower layers of the declarative architecture. Since this layer manages a huge part of the job, it consists of several sub layers. Main objectives of these sub layers are the storage, data processing and the subscription. These layers are responsible for storing the data, streaming, processing and subscription model between the framework and the physical layer.

Besides, this layer also serves/streams the requested data to outer platforms or systems. This layer interacts with many other outer components such as 3D building and physical models. These outer components are explained in the detailed presentation of the framework.

Physical Layer: This layer is the lowest layer of the architecture and represents the physical environment. Wired/wireless physical sensor and actuator nodes deployed in the environment to be monitored/managed.

On this generic architecture, user requirements are processed as a top-down process. Besides, the data processing and serving to users by the integration with BIM principals follow a bottom-up process.

System Components

In this study, a novel framework is proposed to manage a smart building environment. Introduced framework serves not only building management but also provides a platform on which further machine learning or big data research can be performed. Figure 3 introduces the detailed architecture and the components of the system that compose the framework and other layers.

Sensor Data Acquisition

Physical layer of our architecture consists of Iot driven wireless sensor devices. Several sensor nodes provide physical measures from the environment and send these sensor data to the lowest layer of the framework. Here, sensor data acquisition is achieved through MQTT protocol. In a basic network architecture, devices are communicating via HTTP. Here, instead of using request/response model, MQTT protocol uses a publish/subscribe architecture which is event-driven and enables messages to be pushed to clients. MQTT Broker is the central communication point. To receive the sensor data, a subscription should be done. Once a subscription to a sensor node is made, the sensor node publishes its sensor data. MQTT receives the sensor data and serves this data if there exists a client that is subscribed to that published data.

Each NodeMcu connects online based on the connection rules for establishing a connection to MQTT Broker. Due to the assignment of a static IP address to the microcontroller, the gateway, the subnet, the DNS, IP address and SSID with its password should be described in Arduino programming (on Arduino IDE).

On the MQTT Broker side, a configuration should be set. There exists a configuration block that is responsible for internet connection and start-up command to the sensors to make a robust and stable connection during the streaming.

Stream Management via Apache Kafka

A basic smart system is based on the acquired data and management of the stream data. Hence, such systems require a stream management tool to process the data flow and forward it to the right clients. In this study, Apache Kafka is preferred for the sensor data stream management. At this layer of the architecture, we focus on modeling of data streams. A tool is required to both transport these streams of data to all the systems (as a pipeline) and applications that need them to build real time applications on top of them. Thus, Apache Kafka is a suitable solution to manage our sensor data streams.

Such stream management tools adopt the MQTT, subscribe/publish model. Here, sensor devices are the producers that produce and publish data, upper layers are our consumers that are subscribed to these streams. Besides, a topic is needed to be created. A topic can be considered as a category or feed name in which acquired data are published (inserted). The main purpose of existence of topics is that there always exist multiple subscribers for the same stream. A broker handles the consumers, producers, and the topics.

After receiving the sensor data on the MQTT Broker, a python script is executed to save this stream in a csv file. Still, sensor data should be transferred to Kafka. To get there, firstly Zookeeper is used. It is a centralized service for maintaining configuration information before distributing to other services. It provides highly reliable data registry, serialization for encoding data according to specific rules. Zookeeper ensures that the data streams run consistently. Once Zookeeper is launched, Kafka is started to run.

Kafka's main role is the real time data feeds, collecting big data and performing real time analysis on that data. It adopts a producer/consumer model. Hence, a creation of a topic is mandatory, and the producer should be defined. In our case, the producers are the sensor nodes. They provide a sensor data stream to our framework. While the bottom-up process, sensor source is hidden in the framework and from the application point of view, they are only service providers, and the applications benefit these services. On the other hand, parts that benefit these services are the consumers of the system.

To establish a proper producer and consumer connection, it is mandatory to create a topic. Once the topic is created, the pipeline is ready for use and Kafka sends the csv file through this pipeline to Apache Flume. Flume is a service for collecting, aggregating, and moving large amounts of data efficiently. The flume configuration file defines the rules and the paths for the messages coming through the pipelines. Via Flume, csv files are transmitted to HDFS for storage.

HDFS and Required Tools

HDFS is a Hadoop distributed file system used to store unstructured and semi-structured data. HDFS adopts a master/slave architecture model. The master side is called NameNode and manages the file system namespace operations like opening, closing, and renaming files and directories and determines the mapping of blocks to DataNodes. Besides, slaves are called DataNodes and are responsible for serving read and write requests from the file system's clients.

The data coming from the Apache Flume is unstructured. Data is kept unstructured on purpose, so that developed frameworks may handle heterogeneous data formats from various service providers (sensor nodes). To manage unstructured data in HDFS, several tools should be initialized.

Once the csv file is reached to the HDFS, it should be put in a format suitable for analysis (map) and performing mathematical operations. Here, MapReduce is required for processing and generating big data sets with a parallel, distributed algorithm approach. The mapper processes the data and creates several small tuples of data. The reduce phase produces a new set of output, which will be stored in the HDFS.

To be able to read the data in HDFS, another tool is also required: Hadoop Common. It provides the systems to read the data stored under the Hadoop file system. Besides, it contains libraries and utilities needed by other Hadoop modules as well.

Here, the data is successfully formatted and stored on HDFS and prepared to be read. However, data is the main resource of the system and to manage this resource, YARN is also required. YARN manages computing resources in clusters. It contains the parameters of Node Manager (framework agent, responsible for containers, monitoring their resource usage), Resource Manager (accepts job submissions from users, schedules the jobs and allocates resources to them), Application Manager (created for each application to negotiate for resources and work with the Node Manager).

HIVE

Data stored in HDFS is unstructured. So, to be able to analyze and structure it, the Hive can be used. Apache Hive is a data warehouse system for synthesizing and analyzing data and for querying large data systems in the open-source Hadoop platform. It converts SQL queries into MapReduce tasks for easy execution and processing of extremely large volumes of data.

The three important features for which Hive is deployed are data summary, data analysis, and data query. The query language, exclusively supported by Hive, is HiveQL. This language translates SQL queries into MapReduce tasks to deploy them on Hadoop. HiveQL also supports MapReduce scripts which can be connected to queries. Hive increases schema design flexibility as well as data serialization and deserialization. The data stored in HDFS is sent to Hive and via Hive and HiveQL, data can be processed, analyzed and can be used for other applications such as machine learning algorithms etc. Figure 4 shows the HiveQL "SELECT * FROM data.dissertion ORDER BY datatime DESC LIMIT 10" and the result of the given query.

BIM-WSN Integration via Revit Dynamo

The major novelty of the developed framework that differs from the existing studies in the literature is the BIM-WSN integration. Proposed framework provides not only sensor data management to process and to serve other applications, but also integration with the 3D model. The main benefit of such integration is to monitor the environment on the 3D model in real time.

For the BIM-WSN integration, a programming tool of Autodesk is used: Revit Dynamo. Revit Dynamo is an extension of Autodesk that uses Revit API for visualized programming. Simply it provides a building information workflow. In Revit Dynamo, each node performs a specific task. The nodes have inputs and outputs. The outputs of one node are connected to the inputs of another using 'wires'. The program or 'graph' circulates from node to node through the network of wires. The result is a graphic representation of the steps necessary to achieve the final design.

The main advantage of using Dynamo and creating scripts on Dynamo is to automate the repetitive tasks such as fetching sensor data from MQTT Broker and refresh the 3D model in real time. Besides, it provides combining data from different categories (from different products from MQTT's point of view).

An illustration of the Revit Dynamo for our experiment environment is given in Figure 5. Revit Dynamo finds the NodeMcu ports and starts listening to them. As presented above, in our physical layer, three different sensor types are used. Based on the acquired data from these sensors, a real time monitoring system on 3D is established and the data communication is achieved through the Revit Dynamo. Figure 6 presents the real time temperature stream monitoring and the modification of the color of the parts according to the values read from the ports. It is seen that the network halts in various local minimums when the curve obtained from the training with artificial neural networks only is studied. Eventually, it is understood that the error does not show any declining trend by accepting the local minimum obtained at around 1200 th epochs and by terminating the training process. When the curve of artificial neural network optimised by genetic algorithm is studied, it is seen that errors are instantly noticed, and a faster training is obtained. Finally, when the curve of artificial neural network that was trained by firefly algorithm is compared to the others, it is observed that both it passes the minimum levels fast and obtains a more optimum value by reaching the lowest MSE value.

ENVIRONMENT SETUP

For the environment, Galatasaray University building is chosen, and wireless sensor devices are deployed into offices.

Model Transformation -2D to 3D

In traditional building construction, the model of the building is designed in two dimensions commonly via AutoCad©. To enable the developed framework presented in this study, 2D models should be transformed into 3D models. For 3D design, in this study, Autodesk Revit is preferred. Autodesk Revit is Building Information Modeling (BIM) software offering a multi-disciplinary and collaborative approach to design and construction projects. Here, for testing our testbed built in our university building, 2D models of our engineering department building are obtained and as the first step of this study, it is transformed into 3D models. Figure 7 presents this transformation from 2D to 3D on the platform Autodesk Revit.

Physical devices

In this study, for the data acquisition, a sensor node is built using 3 sensors: DHT11 for temperature and humidity measurements, BH1750 for luminosity measurement and finally MQ-2 for various gas detection and measurement. All these sensors are connected to a NodeMcu ESP8266. A sample node with connected sensors is presented in Figure 8. As seen from the figure, these devices are small and light weight factors. Deployment such devices in suitable locations of the building provide a concrete environmental monitoring system. Specifications and the data acquisition parameters are indicated in Table 2.

Physical representation of 3D model

As described in the architecture representation, acquired data is directed to three different parts of the system: upper layers in the same framework for the storage and to serve to applications, Revit Dynamo to monitor environment from the 3D model in real time and finally a concrete model printed in 3D printer. This model is enriched with RGB leds for each room represented in 3D model. This lightning mechanism represents the environment status based on the acquired data. A single NodeMcu (without branching any sensor) fetches the sensor data from MQTT Broker and processes it. Based on the sensor data processing, relevant alarm or status indications are performed on the model.

Data Visualization

A graphical user interface is implemented for the user interaction. Figure 9 presents the GUI of the system. In Figure 9a, last acquired physical measures from the chosen location are presented at top. It is always possible to set the threshold values for each one. Based on the threshold values and the measured value, a warning or a critical alarm can be produced. A small sign on each measure indicates if the physical measure is between the suitable interval or a warning/alarm is produced. It is also possible to observe the previously acquired data as indicated in Figure 9b with the paging system. For the moment, sensor data is gathering successfully, the real dataset is being constructed and the framework executes and analyzes data in real time. In the future works, we are planning to integrate the machine learning approaches based on the acquired data. Various data analysis and machine learning methodologies can be applied to improve the current system.

CONFLICTS OF INTEREST

No conflict of interest was declared by the author.

Figure 3 .

 3 Figure 3. Detailed architecture of the proposed approach

Figure 4 .

 4 Figure 4. Hive and HiveQL

Figure 5 .Figure 6 .

 56 Figure 5. Connecting sensor data stream to Revit Dynamo

Figure 7 .

 7 Figure 7. University building floor -Transformation process from 2D to 3D on Autodesk Revit platform

Figure 8 .

 8 Figure 8. Sensor devices and NodeMcu with its motor shield

  (a) Last values (b) Passed Values Figure 9. Data Visualization of selected location

Table 1 .

 1 Enabled platforms for BIM

	Platform	Research	Proposed Solution
		Riaz et al. 2017 [6]	CosMos -Monitoring for construction environment
		Arslan et al. 2014 [7] Monitoring for construction environment
	Autodesk RevitArchitec	Wang et al. 2013 [8] RBIVS -Visualisation of university campus
	ture	Woo et al. 2014 [9]	Analysis system to facilitate decision-making and economic modernization of aging commercial buildings
		Hu et al. 2014 [10]	Optimise sensor device location
	OpenGL	Lee et al. 2012 11]	Monitoring for tower crane operators
	IFC Model
	Editor +	Liu et al. 2009 [12]	BIM Sensor Modeler create integrated sensor models
	Sensor ML
	IFC Model
	Editor +	Wu et al. 2014 [13]
	CityGML

Table 2 .

 2 Specifications and Data Acquisition Parameters

	Feature	DHT11	BH1750	MQ-2
	Size	22 x 20.5 x 1.6mm	21 x 16 x 3.3mm	35 x 22 x 23mm
	Operating Voltage	3.5V to 5V	2.4V-3.6V	5V
	Operating current	0.3mA	0.12mA	40mA
	Modality	Temperature (C) and humidity (%)	light intensity (lux)	LPG, Alcohol, C3H8, H2, CO and CH4 (ppm)
	Measurement Range	0 C to 50 C -20% to 90%	1 -65535 lux	200 -10000 ppm
	Sampling rate (Hz)	0.2	0.2	0.2
	Transmission rate (Hz)	0.2	0.2	0.2
	Reach to Sink	Single hop	Single hop	Single hop
	Output	Serial data	Serial data	Serial data
	Resolution	16-bit	16-bit	16-bit
	Accuracy	±1%	±20%	±10%
	Avg. Delay (ms)	25	25	150

CONCLUSION AND DISCUSSION

In the recent years, smart building environments and management systems became one of the trend topics in computer science. Various analysis realized by researchers from different domains prove that traditional buildings are primary consumers of a significant portion of energy resources.

In this study, we highlight the major challenge posed by merging WSN technology with BIM, integration of IoT based devices requires a novel approach to merge various independent systems about designing and managing a smart building environment. Here we focus on integrating into smart building environments and connected systems. Based on the declarative principles of PEMS, we present a sustainable declarative surveillance architecture and a novel framework adopted to that architecture for allowing the processing of massive raw data from sensors, assisted by a digital model of the building.

The most popular studies in this domain are listed with their usage and supported features in Table 3. These studies include several dependencies and propose limited solutions. They are dedicated to specific applications and domains such as tower crane operations, energy saving lighting systems etc. However, the automation management systems in practice need more generic and convenient solutions in real life scenarios. On the other hand, as indicated on the last line of the table, our proposed framework provides a real time indoor and outdoor environmental monitoring with full integration with the digital model of the building. Not only a visualization mechanism but also an alarm system is also covered by this solution. To manage a smart building environment, we propose a framework and BIM-WSN integration. In this study, for the testbed environment university building is chosen and wireless sensor devices are deployed into offices. Firstly, 2D model of the relevant environment is transformed into 3D to obtain a numerical model of the environment. Besides, a highly complex framework is developed to manage the sensor data streams and makes the streams available for diverse applications. As a result, sensor data is acquired by the sensor device and is transferred to the framework. Here, sensor data is passed by several pipelines between tools and services and meanwhile it is processed to reflect the real time monitoring on the 3D model (virtual and printed one). Concurrently, data is stored in a big data platform in order to serve to other applications such as machine learning applications, data science studies etc.