 prsnt img-s rnring tht llows r viwpoint wlkthroughs o urn sns using just  w photogrphs s input. Commril pplitions suh s Googl trtviw, Bing ps t. us ruimntry orms o img-s rnring or urn visuliztion; mor sophistit pprohs us th ull D mol o th sn s input. As th qulity o D mol grs, rnring rtits r osrv whih rstilly ru th utility o suh pplitions. In this thsis, w propos img-s pproximtions to ompnst or th lk o urt D gomtry. In th irst pproh, w us is-ontinuous img wrping gui y qusi-ns pth mps whih improvs visul qulity ompr to prvious mthos tht rly on txturing D mols. is pproh involvs  smll gr o mnul intrvntion to mrk olusion ounris in th input imgs.  uil upon this in th son p-proh y vloping  ompltly utomti solution tht is pl o hnling mor omplx sns.

 ovrsgmnt input imgs into suprpixls n wrp thm inpnntly using sprs pth.  introu pth synthsis to rt pproximt pth in poorly ronstrut rgions o th img n us this with our img wrps or gnrting high qulity rsults.  ompr our rsults to mny rnt lgorithms n show tht our pproh xtns vry wll to r viwpoint nvigtion.

 lso prorm prptul nlysis o iffrnt img-s rnring rtits in sprt usr stuis unr ontroll onitions.  us vision sin to invstigt prsptiv istortions pro-u whn  singl img is projt on  plnr gomtry n viw rom novl viwpoints.  us th xprimntl t to vlop  quntittiv rmwork or priting th lvl o prsptiv istortions s  untion o ptur n viwing prmtrs. In nothr stuy, w ompr rtits us y smooth trnsitions (lning imgs) with rupt trnsitions (popping) n vlop gui-lins or slting th il troff unr iffrnt ptur n rnring snrios.  us guilins rom ths stuis to motivt th sign o our img-s rnring systms sri ov.

 monstrt n pplition o our pproh or ognitiv thrpy.  rt th irst virtul rlity pplition tht uss img-s rnring inst o tritionl omputr grphis. is rstilly rus th ost o moling D sns or virtul rlity whil prouing highly rlisti wlkthroughs. vrll, w liv our work is  signiint stp towrs r viwpoint img-s rnring sign on soun prptully-s ountions.

Acknowledgments

First o ll, I thnk th Frnh govrnmnt or th allocation de recherche (rsrh llowship) tht inn most my otorl stuis.

I m most thnkul to my visor, Gorg Drttkis, or his thorough involvmnt n ptin. His inril zl or rsrh n  nr suprnturl ility o ulilling multipl lins t th sm tim hs n  sour o motivtion. H push m towrs gols whih sm unrsonl t th tim, ut ltr orm th kon o this thsis. or on thn not, h ws th rson tht kpt omplny t y.

I m grtul to vi mmoorthi or hosting m t th nivrsity o Cliorni Brkly, n to Fro Durn n ylvin ris or suprvising m uring my sty t sshustts Institut o hnology. ough w i not ollort, Arin Boussu h n importnt inlun on my work. I ttn his thsis n s n uin mmr, n sin thn I hv striv, with limit suss, to mth th qulity o rsrh h onut uring his otorl stuis.

I ow mny thnks to ll my o-uthors, spilly lg orkin-Hornung or hr guin uring th ours o my min projts, n tr ngorp or his guin on two othr projts. I h th or-tun o working with n inrily pl group o stunts n rsrhrs -Jonthn gn-Klly, ylvin Duhn, Ars Lg, Bruno Glrn, irr-vs Lffont, Emmnull Chpouli, hi Gurhouh, tn opov, Christin ihrt, Crls Bosh, Jorg Lopz-orno, Evnthi Dimr n ylvin Lvr. I must thnk ylvin ris or rptly rviwing rs o my pprs uring ritil lins. I lso thnk rtin Eismnn, ihl Gosl n r ollys or thir vi on stting up om-prisons with prvious work. I thnk ll th mmrs o th jury: Christin olt, rus gnor, Fro Durn, Jn on n Frééri rioso or spning prious tim on my issrttion.

Finlly, I ttriut som o th shortomings o this thsis to my mily, rins, ltmts n la belle France who gv m nough rsons to sp work vry now n thn.

Résumé

ous présntons un pproh  rnu à s 'imgs (BI) qui prmt, à prtir  photos,  nv-igur lirmnt t générr s points  vus qulonqus ns s sèns urins. Ls pprohs préénts épnnt s moèls D t onnnt liu à s rnus  qulité réuit v uoup 'rtts. Dns tt thès, nous proposons un pproximtions sés sur l'img pour ompnsr l mnqu  préision  l géométri D. ous utilisons un wrp 'img guié pr s rts  proonur qusi-nss qui onnnt liu à uoup moins 'rtts. En s snt sur tt p-proh, nous vons évloppé un métho ntièrmnt utomtiqu prmttnt  tritr ls sèns omplxs. ous sur-sgmntnt ls imgs ' ntrés n suprpixls qui limitnt ls olusions sur ls ors s ojts. ous introuisons l synthès  proonur pour rér un pproximtion  tt proonur ml ronstruit ns rtins régions t lulons ls wrps sur ls suprpixls pour synthétisr l résultt inl. ous omprons nos résultts à  nomruss pprohs rénts. ous vons nlysé ls rtéts  l'BI 'un point  vu prpti n omprnt ls rtéts générés pr l mélng  plusiurs imgs v ux s trnsitions tmporlls rusqus. Ctt étu nous  prmis ' élorr un méthoologi pour l séltion 'un on ompromis ntr ls ux. ous vons égl-mnt nlysé ls istorsions prsptivs t nous vons évloppé un moèl quntitti prmttnt  préir ls istorsions prçus n ontion s prmètrs  ptur t  visulistion. Comm ppli-tion, nous vons mis n oeuvr un systèm  rélité virtull qui utilis l'BI à l pl  l'inogrphi tritionnls.

vi

Chapter Introduction

Computr grphis tritionlly hs our stgs: () gomtri moling, () optionl nimtion, () mtril/lighting sign n () rnring. vr th lst thr to our s, omputr grphis hs volv immnsly n n now hiv ultr-sophistit spil ffts. n o th most importnt gols in omputr grphis hs n to gnrt rsults tht r inistinguishl rom rlity. Looking t th stt o th rt in gomtri moling, omputr nimtion n glol illumintion, it is ir to sy tht lmost ll nturl n mn-m phnomn n  simult with stonishing lvls o rlism, givn nough tim n rsours.

In fft, th pross o moling n rnring involvs muh mnul s wll s omputtionl ffort. Computtionl omplxity o h o ths stgs n  n importnt tor, ut on o th most signiint ottlnks is mnul moling n txturing whih n  vry xpnsiv n tim onsuming. For xmpl, suppos n pplition rquirs showing  gnri urn town squr. It will tk  smll group o rtists ys to mol vry til o  typil squr -th gomtry, mtril proprtis, txturs, lighting t. using stt o th rt ommril sowr. An intrsting trntiv woul  to quir n xisting sn n us th t to utomt som or most o th moling pross. As isuss in Chptr , rpi vns hv n m in th il o D ronstrution. Howvr, prt pixl urt ronstrutions r still vry hr to otin, mor so i th sn is vry omplx.

In th sn o prt gomtri ronstrution o th sn, it will  thortilly impossil to o-tin physilly orrt rnring o th sn rom novl viwpoints. ror, our trgt is to gnrt plausible rsults, to whih n w invstigt th ollowing qustions:

• How n w gnrt plausible r viwpoint nvigtion using img-s rnring?

• How n w in th notion o plausibility in th ontxt o img-s rnring?

 ormr is the lssil qustion tht img-s rnring rsrh hs ttmpt to nswr. ur work vns th stt o th rt y vloping nw thniqus or img-s rnring in our sp-ii ontxt, whih w xplin in th ollowing stions. n o th most importnt tors tht istin-guishs us rom prvious work is tht w irtly pl ourslvs in th ontxt o free viewpoint navigation, whil most prvious pprohs hv trgt viw intrpoltion whr th trgt is to gnrt viws tht intrpolt th positions n orinttions o two input viwpoint.

. Context

Bing  t-rivn pproh, img-s rnring is snsitiv to input t n ploymnt on-itions. ror, it is importnt to lrly intiy th us s or our systms.  trgt strt-si (su)urn imgry ptur using hnhl or vhil mount mrs (s Figur . ). ypilly, suh imgry ontins rhittur, vgttion, popl, vhils t.  only hnl stti sns, so w ssum tht thr r no moving lmnts suh s popl or trffi in th photogrphs.  nsity o ptur is typilly to mtrs twn iffrnt viwpoints.  intn to rnr ths sns rom viwpoints orint roughly in th sm irtion s th input viwpoint, within ∘ o th vr-g orinttions o ptur mrs. ur gol is to llow th usr to nvigt in  zon o to mtrs ur ntrl prolm sttmnt is to vlop nw img-s pprohs tht monstrt prviously unttmpt lvls o sophistition -r viw nvigtion, whil ssuming sprsr rsours in th orm o input imgs, or mor omplx sns whr prprossing stgs, nmly multi-viw stro, r not xpt to giv ompltly urt rsults.  istinguish ourslvs rom prvious work y pursuing  hrr st o hllngs:

• urn sns in thir ull omplxity,

• miniml pturs with wi isplmnts o up to -mtrs twn input viws,

• omplx isolusion ffts u to irrgulr silhoutts o multipl orgroun ojts,

• plusil olusion n prllx in spit o poor qulity D ronstrution in mny rgions, n

• r viwpoint nvigtion.

 sonry prolm is to quntiy img-s rnring rtits. r r lmost no mtris or quntiying rnring qulity o iffrnt img-s rnring pprohs. is is us o th shr numr o tors tht fft th inl rsult, .g. sn omplxity, numr o input imgs, sim-ult viw positions t.  trgt prptul nlysis o rnring rtits; to whih n w vlop xprimntl stups, stimuli n protools or prinipl stuis n monstrt th utility o suh stuis to img-s rnring stups.

.

Main intuitions

. . Min intuitions

 min intuition is to tivly utiliz all th D points otin rom multi-viw stro, vn th smllst lustrs on poorly ronstrut ojts.  lint iffrnt pth lyrs o th sn using silhoutts n us th D points in h rgion sprtly. ypilly, pth t ny pixl llows prvious pprohs to rprojt th pixl into  novl viwpoint.  ssum tht vry w pixls in n img rgion hv pth; w rprojt th ntir img rgion into th novl viwpoint y using  shapepreserving warp. ur wrps r gui y th smll numr o pixls whih hv pth n rgult y img-s D onstrints whih sk to minimiz th ovrll istortion in th inl rsult. is high lvl i is hiv y iffrnt mns in Chptrs n .

Anothr importnt i tht ontriuts hvily to th suss o our pprohs is tht w n-or silhoutts in n img-s mnnr. rvious pprohs pn upon silhoutts ing p-propritly rprsnt in th ronstrut gomtry Eismnn et al., . ln itting pprohs Gllup et al.,  improv this y using grph uts to rinor img gs into th itt plns.

s grph uts n  thought o s joint optimiztion on img gs n D plns whih n l-wys us plns to l into rronous rgions i no gomtry is vill ovr  signiint rgion or simply u to numril issus. Intuitivly, oupling silhoutts rom gomtry stimtion will l-wys prorm ttr in trms o urt silhoutts.  omput img silhoutts in  prpross n us thm to isolt th shp prsrving wrp o iffrnt rgions, rsulting in high qulity olusions n prllx ffts.

vrll, w monstrt tht our modus operandi o ormulting th whol prolm in trms o onstrints tht trgt plusil img synthsis is highly fftiv t ompnsting or rrors in ron-strution.

For prptul nlysis, w numrt som o th most importnt rtits n prorm usr stu-is tht llow us to orrlt priv visul qulity with sn n rnring prmtrs.  most importnt insight in th sign o prptul xprimnts is to isolt th rtits using simplii stups tht llow prinipl nlysis, whil rmining suffiintly los to tul img-s rnring stups tht r intrsting rom n pplition s wll s rsrh prsptiv. ur stups llow us to ontrol th gr o rtits using  smll numr o prmtrs in th stimuli; usr stuis unr ths onitions giv  irt rltionship twn rnring prmtrs n priv qulity.

 strt with th stuy o prsptiv istortions, whih r inhrnt in ny mtho tht rprojts n img ptur rom on viwpoint into nothr viwpoint.  thn stuy th mor omplx s o visul rtits rt y lning ontnt rom multipl input imgs to synthsiz ny pixl o th novl viw.  us th guilins rom ths stuis using iffrnt mthoologis in h o Chptrs n .

. Contributions

. Overview

 rst o th thsis is orgniz s ollows:

• Chptr givs  isussion on prvious work in omputr grphis, vision, gomtry n pr-ption tht is rlvnt to th thniqus sri in this thsis.

• Chptr prsnts  novl img-s rnring pproh s on vritionl img wrps tht is pl o hnling som o th most omplit tst ss ttmpt.

http://www. dapp.com/catch

• Chptr uils upon th prvious pproh y vloping  lol vritionl wrp s on im-g ovrsgmnttion. is is mong th vry irst thniqus to prsnt r viwpoint intrtiv nvigtion using img-s rnring.

• Chptr sris prptul nlysis o visul rtits ssoit with img-s rnring systms.

• Chptr sris n virtul rlity stup using img-s rnring. ough still in rly stgs o vlopmnt, this is th vry irst systm o its kin.

• Chptr summrizs th rsults o this thsis n proposs immit nxt stps s wll s long trm rsrh vnus.

Chapter Previous Work

Img-s rnring hs n n tiv r o rsrh sin its inption in th orm o img Givn th ov, w s tht our work rws rom multipl rsrh omins -stro ronstrution, img-s rnring, img wrping n img sgmnttion. In this hptr, w isuss th stt o th rt in h o ths omins. Retargeting using variational warping nipulting th spt rtio o imgs hngs th spt rtio o th ontnt n ls to ormtions. Img rtrgting pprohs try to prsrv th spt rtio o slint img ontnt whil shiing most o th ormtion to rgions whih r ithr unim-portnt or mk it hr to priv th istortions. wrps sk to minimiz n nrgy untion tht wrps ll pixls o n input using  muh sprsr pixl to pixl mpping. is mthmtil rmwork is lso th sis o img wrping or othr pplitions s isuss low.

. D reconstruction

D image warping

. Discussion

A mjority o img-s rnring pprohs r rstrit to stuio pturs, smll slins n/or viw intrpoltion hjn et al., . s pprohs r not irtly pplil to our trgt. sults or omplx orgroun ojts o not improv proportionlly. Aing mor imgs rus th isprity, mking it sir or stro ronstrution pprohs to mth img turs. Howvr, this os not hlp th s o txtur-poor surs or usy txturs suh s vgttion us ommonly us mtris suh s normliz ross orrltion rmin miguous in suh situtions irrsptiv o th isprity.

 prsnt  nw pproh whih rsss ths limittions. ur ntrl i is to ompnst or inorrt or inomplt gomtri inormtion y introuing silhoutt prsrving vritionl wrping.  ous on sns ontining hr to ronstrut omplx orgroun gomtry within th ontxt o wi slin img-s rnring. ur min ontriutions n  summriz s ollows:

• th rprsnttion onsisting o sprs pth onstrints n silhoutt gs, whih nls shp-prsrving vritionl wrping or wi-slin snrios,

• th introution o silhouette-aware wrping in whih "lsti" gs sor istortions us y (is)olusions whil pth isontinuitis r prsrv, n

• n ffiint rnring lgorithm with  goo tr-off twn lurring n olor isontinuitis. ur pproh grtly rus rtits ompr to th st omintion o stt o th rt th-niqus, whil ovroming th limittions isuss ov (s rsults Figur . n . ). In prtiulr w trt sns with hr-to-ronstrut ojts n viwing pths whih o not intrpolt th input mrs.  rquir only  smll numr o imgs, rsulting in  lightwight ptur pross.

. Overview

 input to our mtho is  st o imgs lirt using nvly et al.,  n  D point lou gnrt using Furukw n on, .  point lou n  projt into th input imgs to giv pth vlus t rtin pixls, whih w rr to s depth samples. ur pproh hs thr min stps (s Figur . ):

Preprocessing ur pproh irst slts silhoutts roun orgroun ojts (trs, rs, t.) or h input img (tion . . ) in  usr ssist shion. s silhoutts r us to orrtly hn-l pth isontinuitis.  son stp imts th st o pth smpls to  sprs uniormly istriut st. is stp lso ills in poorly ronstrut rgions using pth rom nighoring points (tion . . ).  rsulting pth smpls srv s onstrints or our img wrp. .

Silhouette

Shape-preserving warp using D constraints

Givn  novl viw, xprss y  mr projtion mtrix C N , our gol is to wrp th input imgs I 1 , I 2 , … , I k so tht thy mth th tul sn s it woul hv ppr in tht viw s ithully s possil.  thn us th wrp imgs in th rnring piplin (s tion . ).

Dnot y C i th mr projtion mtrix o input img I i . I w knw th mpping U i rom vry

pixl x ∈ I i to th orrsponing D point p, i.. C i (p) = C i (U i (x)) = x,
{(v 2 -v 1 ) , R 90 ⋅ (v 2 -v 1 )} tth to th tringl is shown in grn.
th shp o th img ontnt.  hnl olusions y xpliitly moling th sir wrp hvior long silhoutts, s will  sri in tion . . . ogthr, th positionl onstrints n wrp hvior priors in n nrgy untionl whih w minimiz to wrp th input imgs.

Setup In orr to omput th vritionl wrp, w isrtiz th img omin y ovrlying  tri-ngl msh on img I i . Eh  o th wrp msh hs or mor pth smpls.  not th input wrp msh vrtis y v n thir wrp positions y ṽ , whih r th unknowns in th wrp optimiztion.  in th vritionl wrp s  linr optimiztion; th ull wrp n  omput y solving or wrp vrtx positions ṽ n rnring th wrp msh with th originl txtur oorints.

Reprojection energy ll tht th pth smpl prprossing stp rom tion . gv  sprs st i o uniormly istriut pth smpls or h input img. For h pth smpl D[x], w lot th tringl T o th wrp msh tht ontins th pth smpl. Dnot th vrtis o T y

(v 1 , v 2 , v 3
) n lt th ryntri oorints o th lotion o th pth smpl t pixl x in tringl T  ( , , ):

x = ⋅ v 1 + ⋅ v 2 + ⋅ v 3 ( . )
 rprojtion nrgy msurs th istn twn th wrp position o th pth smpl n th il rprojt lotion using th novl viw mtrix C N :

E p [x] = ‖ ⋅ ṽ 1 + ⋅ ṽ 2 + ⋅ ṽ 3 -C N ⋅ C -1 I i ⋅ D[x]‖ 2 ( . ) whr C -1 I i
is th k-projtion mtrix o img I i .

Shape preserving energy

(v 1 , v 2 , v 3 ) n tth  lol orthogonl rm to it: {(v 2 -v 1 ) , R 90 ⋅ (v 2 -v 1 )}, whr
a = (v 3 -v 1 ) T ⋅ (v 2 -v 1 ) ‖v 2 -v 1 ‖ , b = (v 3 -v 1 ) T ⋅ R 90 ⋅ (v 2 -v 1 ) ‖v 2 -v 1 ‖ ( . )
s lol oorints n  us to xprss h vrtx o th tringl s  linr sum o th sis vtors o th lol rm:

v 1 = v 1 + 0 ⋅ (v 2 -v 1 ) + 0 ⋅ R 90 ⋅ (v 2 -v 1 ) , v 2 = v 1 + 1 ⋅ (v 2 -v 1 ) + 0 ⋅ R 90 ⋅ (v 2 -v 1 ) , v 3 = v 1 + a ⋅ (v 2 -v 1 ) + b ⋅ R 90 ⋅ (v 2 -v 1 )
As th tringl unrgos  similrity trnsorm, th lol oorint rm rmins orthogonl n lol oorints (a, b), omput rom initil positions, rmin th sm.  inl vrtx position ṽ 3 n  xprss s  untion o lol vrtx positions n inl position o othr two vrtis ṽ 1 n ṽ 2 .  similrity nrgy trm n thus  xprss s:

E s [T] = ‖ ṽ 3 -( ṽ 1 + a ⋅ ( ṽ 2 -ṽ 1 ) + b ⋅ R 90 ⋅ ( ṽ 2 -ṽ 1 ) ) ‖ 2 ( . )
. .

Silhouette-aware warp

 nrgis sri so r r smooth n shp-prsrving vrywhr. Howvr, th wrp shoul hv isontinuitis in th viinity o silhoutts us o th pth isontinuitis. hn onsiring  smll nighorhoo roun  silhoutt g, th wrp my hv  isontinuity prpniulr to th g (to mimi (is)olusion) whil rmining shp-prsrving in th tngnt irtion.  mol this hvior y onptully insrting  nrrow n highly lsti n prlll to th silhou-tt tht is llow to sor hvy istortion u to isontinuity (s Figur . ()).  shp o th silhoutt itsl, on th othr hn, is prsrv y ing  urv-similrity nrgy trm sri low, thus voiing istortion o orgroun ojts. In orr to proprly isrtiz th img omin o prsrv th shp o th silhoutt itsl, w rquir th silhoutt urv to lolly unrgo  shp-prsrving trnsormtion.  nrgy ormultion is similr to Eqution . , ut it is in on th silhoutt urv this tim, inst o  D rgion. Consir thr onsutiv vrtis lying on th urv, inx w.l.o.g. s e = (v 0 , v 1 , v 2 ).  ll suh  squn o two urv gs n edgelet (s Figur . ()). A similrity trnsormtion o th glt e mns tht th ngl θ twn th two gs, s wll s th lngth rtio ‖v 0 -v 1 ‖/‖v 2 -v 1 ‖, rmins th sm.  n thror writ th urv similrity nrgy trm s: s ns r ltr ill using txtur rom  iffrnt img in th inl rsult (xplin in tion . ). us, our img wrp is roust to wi-slin (is)olusion.

() v 0 v 2 v 1 elastic triangles ()
E b [e] = ‖( ṽ 0 -ṽ 1 ) -( ‖v 0 -v 1 ‖ ‖v 2 -v 1 ‖ ) ⋅ R θ ⋅ ( ṽ 2 -ṽ 1 )‖ 2 ( . ) whr R θ is th 2 × 2 rottion mtrix tht rotts th g (v 2 -v 1 ) onto (v 0 -v 1 )

. . Total warp energy

 optiml wrp minimizs th wight sum o th rprojtion, similrity n silhoutt nrgis rom Equtions . , . n . ):

E i = w p ∑ ∀x∈ i E p [x] + w s ∑ ∀T∈ i E s [T] + w b ∑ ∀e∈ℰ i E b [e] ( .
P ng ( Ĩ i , x) = ros ( ⟨c n -p n (x), c i -p n (x)⟩ ) P ov ( Ĩ i , x) = { ∞ i x lis outsi th wrp msh, 0 othrwis P e ( Ĩ i , x) = { ∞ i x lis insi th lsti n, 0 othrwis ( . )
 inl pnlty is givn y

P( Ĩ i , x) = P ng ( Ĩ i , x) + P ov ( Ĩ i , x) + P e ( Ĩ i , x) ( . )
 us prptully-s guilins rom Chptr tht xssiv lning us y unstrutur lumigrph wights n  ojtionl.  st rsults r otin whn only two imgs r ln or h pixl in th novl viw (s tion . . ). In orr to ln only th two most suitl imgs t h pixl, w rt two txturs ℛ 0 n ℛ 1 , whr ℛ 0 is ompos o pixls hving th lowst pnltis rom ll wrp imgs Ĩ i n ℛ 1 th son lowst. Contiguous loks o pixls look up rom th sm wrp img r shown in iffrnt olors or ℛ 0 n ℛ 1 in Figur . .  omput lning wights rom pnltis n stor thm in th lph hnnls o ℛ 0 n ℛ 1 , rsptivly:

w ℛ 0 (x) = 1 -Ψ ⋅ ( P(ℛ 0 , x) P(ℛ 1 , x) ) , whr Ψ ∈ [0, 1] w ℛ 1 (x) = 1 -Ψ ⋅ ( P(ℛ 1 , x) P(ℛ 1 , x) ) = 1 -Ψ ( . )
 txturs ℛ 0 n ℛ 1 r lph ln to giv th txtur ℛ.  gin us th huristi o minimizing lning rom tion . . n introu n xtr tor Ψ tht mpliis th iffrn in th pnltis o ℛ 0 n ℛ 1 , thry ruing th gr o lning. For xmpl, Ψ = 1 woul us w ℛ 1 to lwys rmin 0. I th rtio o pnltis P(ℛ 0 ):P(ℛ 1 ) is : , stting Ψ = 0.87 woul us th rtio o wights w ℛ 0 :w ℛ 1 to om . : . is grtly rus th ontriution o ℛ 1 vrywhr xpt whr its pnlty is vry los to tht o ℛ 0 . In ll xmpls prsnt hr, w hv us Ψ = 0.87, whih givs goo rsults in ll our xmpls. ur gol is to synthsiz plusil pth or  suffiint numr o points in h trgt suprpixl.

 o this y intiying  st o source superpixels, whih r sptilly los n shoul illy long to th sm ojt in th sn s tht o th trgt suprpixl. In ition, our gol is to hv  ully utomti lgorithm whih rquirs no sn pnnt prmtr tuning. Dpth vlus within trgt suprpixls r synthsiz using n intrpoltion pproh s on th istriution o pths in th sour suprpixls.

. . Computing similar superpixels

 irst omput  st o "most similr" suprpixls or h trgt suprpixl. suprpixls ill in  prvious itrtion r us to  pth to rmining suprpixls in th nxt itrtion.  lgorithm stops whn no mor suprpixls n  ssign pth smpls. I no pixls o  prtiulr sn ojt wr originlly ronstrut, th suprpixls o suh n ojt will in sour suprpixls rom othr ojts n th inl pth histogrm is most likly to rmin unrlil.

C(S T i -→ S j ) = | |-1 ∑ t=1 d(ℋ [ (t)], ℋ [ (t + 1)]) ( . ) C (S T i → S j ) = min ∈Γ[S T i →S j ] C(S T i -→ S j ) ( . ) whr Γ[S T i → S j ] is th st o ll pths rom trgt suprpixl S T i to S j , is on suh pth o lngth | | suh tht (0) = S T i n (| |) = S j , C(S i -→ S j ) is th ost o pth , n d(⋅, ⋅) is th χ 2 istn
 isr suprpixls with multipl split pks n mrk thm s hols (s Figur . ()). ot tht w oul inorport sptil istn n L histogrm istn in  singl mtri y wighing thm ppropritly, ut this woul involv tuning th wights rully or h tst -pning on img ontnt, ojt shps, t.

. . Interpolating depth samples

 now intrpolt pth smpls rom th source superpixels [S T i ].  rt th omin his-togrm o pth smpls rom ll sour suprpixls.  thn rt th joint proility istriution o pth smpls y normlizing th histogrm in siz y th totl r unr th histogrm. is givs th pproximt proility nsity untion (DF) o pth smpls. sing th DF s intrpoltion wights utomtilly ttnuts th fft o noisy pth smpls.  intrpolt th invrs o pth vlus, s pth is invrsly proportionl to isprity Gosl et al.,

.  inl invrs pth t pixl x o S T i is givn y:

1 D[x] = ∑ S k ∈ [S T i ] ∑ y∈ [S k ] P(D[y]) ⋅ ‖x -y‖ -2 ⋅ D -1 [y] ∑ S k ∈ [S T i ] ∑ y∈ [S k ] P(D[y]) ⋅ ‖x -y‖ -2 ( . )
  -pth smpls t rnom pixls in S T i .  rsult or th xmpl in Figur . () is shown in Figur . .  got similr rsults or -pth smpls; highr numrs inrs th siz o th wrp optimiztion.

Furukw n on  , lik ny multi-viw stro pproh, o not ronstrut sky rgions.

 intiy suh rgions using th pproh sri in Appnix B n ssign thm th prntil pth o th img or pplying th ov pth synthsis. is is n optionl stp rquir i thr r signiint sky rgions.

.

Local warping of superpixels with depth samples

Dpth smpls rom multi-viw stro n  noisy, spilly nr silhoutts. In ition, our synth-siz pth is only plausible rthr thn photoonsistnt or urt. Consquntly, irt rprojtion o suprpixls using ths pth smpls, .g., using th io sh t strutur Chn et al., ,

x = ⋅ v 1 + ⋅ v 2 + ⋅ v 3 ( . )
 rprojtion nrgy msurs th istn twn th wrp position o th pth smpl n th rprojt lotion using th novl viw mtrix C N :

E p [x] = ‖ ⋅ ṽ 1 + ⋅ ṽ 2 + ⋅ ṽ 3 -C N ⋅ C -1 I i ⋅ D[x]‖ 2 ( . ) whr C -1 I i
is th kprojtion mtrix o img I i .

Shape-preserving energy

For h tringl o th wrp msh with vrtis (v 1 , v 2 , v 3 ), this nrgy trm msurs its shp istortion r th wrp. Illy th tringl only unrgos  similrity trns-ormtion, rsulting in  null nrgy vlu.  similrity nrgy is otin y xprssing on vrtx o th tringl s  linr omintion o th othr two s in Eqution . :

E s [T] = ‖ ṽ 3 -( ṽ 1 + a ⋅ ( ṽ 2 -ṽ 1 ) + b ⋅ R 90 ⋅ ( ṽ 2 -ṽ 1 ) ) ‖ 2 , ( . )
whr a n b r th sm s Eqution . :

a = (v 3 -v 1 ) T ⋅ (v 2 -v 1 ) ‖v 2 -v 1 ‖ , b = (v 3 -v 1 ) T ⋅ R 90 ⋅ (v 2 -v 1 ) ‖v 2 -v 1 ‖ . ( . )
Hr, R 90 is ∘ rottion.  ovrll nrgy untion or th suprpixl wrp is givn y

E[S k ] = w p ∑ ∀x∈ (S k ) E p [x] + w s ∑ ∀T∈ (S k ) E s [T], (

. Rendering

nring is hiv in thr psss. In th irst pss, w slt n wrp our input mrs losst to th novl mr. xt, w ln th rsulting wrp suprpixl imgs to synthsiz th novl viw.

. . endering A inl hol-illing pss omplts th rnring lgorithm. Blending weights For h nit, w omput th pnlty n hn th lning wight. Con-sir pixl x o th novl viw with ntr o projtion c n n  wrp input img Ĩ i whos ntr o projtion is c i . Lt p n (x)  th point whr th ry shot rom c n through pixl x intrsts th sn g-omtry. is point n  omput simply y kprojting th min pth o th suprpixl using invrs mr projtion mtris o th img I i .  us th ngl twn (c n -p n (x)) n (c i -p n (x)) s n ngl pnlty (s Figur . ).  lso in  il-o-viw pnlty tht hks whthr th pixl lis in  isolu hol rgion o th wrp img whih ppr s lk rgions in Figur . (l).

. . Pass  Camera selection and warping

P ng ( Ĩ i , x) = ros ( ⟨c n -p n (x), c i -p n (x)⟩ ) P ov ( Ĩ i , x) = { ∞ i x is in  isolu hol 0 othrwis ( . ) (

. Results

 prsnt th rsults o our pproh on  wi vrity o tsts, inluing sns ptur y our-slvs n y othrs.  hool tst is rom iroso hotosynth. ChplHill n ChplHill http://photosynth.net/view.aspx?cid=aaeb ecf-cfef-c0 -be -bc ae f c0 . Contribution For th stuy o prsptiv istortions in tion . , th ontriution o th thsis is in th orm o:

• prt o stimuli gnrtion or th xprimnts (tion . . ),

• vlition stuy (tion . . ), n

• pplitions to img-s rnring (tion . . ).

 or thory s on vision sin, xprimnt sign (tion . . ) n sttistil nlysis (tion . . ) r yon th sop o th thsis, pls rr to ngorp et al.,  or tils. s r xplin to provi ontxt.

For th stuy in tion . , th ontriution o this thsis is in th orm o:

• prt o onptul sign o th xprimnts (tions . . n . . ), n

• gnrtion o rl worl img-s rnring stimuli or oth xprimnts,

• inl inrns rom xprimntl t (tion . . ).

 xprimnt usr intr whr multipl stimuli r shown to th prtiipnt (Figur . ) n sttistil nlysis o xprimntl t is yon th sop o thsis, pls rr to ngorp et al.,

 or tils.

.

Perception of perspective distortions

. . Experiment design

 onut two psyhophysil xprimnts to trmin how th img istortions in typil strt-lvl img-s rnring pplitions r priv:

. An ngl-mthing xprimnt tht tst th pritions o th sn n xtn rtinl hy-pothsis y sking th prtiipnts to trmin th priv ngl o ornrs.  rsults llow us to msur th rltiv inlun o th two hypothss on ngl prption.

. A rting xprimnt tht trmin th sujtiv ptility o ngl istortions.  rsults llow us to trmin whih priv ngl istortions r ptl.

Stimuli

. . Experiment  Hinge angle matching

In th irst xprimnt, w trmin how  ∘ ornr is priv r it unrgos istortion u

. . Experiment  Angle rating

In th son xprimnt, w stuy th ptility o vrious ngl istortions.  sk prtiipnts to init how ptl  givn ornr ws s  simultion o  ∘ ornr. rtiipnts wr shown th sm imgs s in th prvious xprimnt in rnom orr. y rt how los th init ornr in h img look to  right ngl on  -point sl whr to orrspon to "prt", "los nough", "kin o ", "not rlly", n "no wy!". rtiipnts ntr h rting using  numril kyp n onirm th ntry y prssing "Entr" (s Figur . n vio ).

Results

. . Inferences Predictive model for perspective distortions

ur xtn rtinl hypothsis (s Eqution . ) givs th priv ngl ζ rt s  untion o n-triity n simultion ngls, whrs th sn hypothsis givs th priv ngls s th tru ngl i.., ∘ , in ll ss. In prti, th tul priv ngl is lwys twn th two hypothss (s Figur . ).  it th xprimntl t to n nlyti untion tht intrpolts th two hypothss n givs th priv ngl s ollows:

ζ priv = ζ rt ⋅ g(d) + 90 ∘ ⋅ ( 1 -g(d) ) ( . )

. . Validation of experimental results

 prsnt  prototyp intr or strt-lvl img-s rnring whih w us to visuliz th qulity o  simult viw pth, n prorm  stuy to vlit our pritiv mol.

Chpter . Evlution of Imge-sed endering using ereptul tudies

Visualization of predicted ratings ur implmnttion rs  st o mrs lirt using stru- visil?" (s vio ). y hos  vlu on th sm iv-point sl (s tion . . ) or  totl o nin pths (thr pths or thr sns).  prsnt th thr vios o h tst on  singl pg, n instrut prtiipnts to just thir rltiv rtings twn th thr vios. A totl o pr-tiipnts prorm th stuy on thir own omputr srns.  summriz th rsults in Figur . , whih plots osrv rtings s  untion o prit rtings sprtly or th thr sns.  orr-ltion twn prit n osrv rtings is mort (r > 0.5) or th irst two sns n strong (r > 0.8) or th thir. us, th pritions r rsonly goo spit th mny iffrns twn th xprimnts us to gnrt th pritions (stti sns with wll ontroll onitions) n this usr stuy (unstrutur ynmi sns). Capture guidelines  lso us our pritiv mol to provi ptur nsity guilins. Chpter . Evlution of Imge-sed endering using ereptul tudies n th othr hn, ruing th gr o lning twn multipl imgs ls to pronoun trnsitions twn imgs s th novl mr is trnslt or rott.  rr to ths isturing tm-porl isontinuitis s popping rtits. ost img-s rnring pprohs rsolv this troff y slting lning wights tht giv th st rsults or th tsts ing tst without ny intuition out th gnrlizility o th lning shm. thr pprohs ypss this issu y voiing lning ltogthr; thy omposit pixls rom multipl imgs using grph ut hjn et al.,  to synthsiz th novl viw.  rsults r imprssiv ut ths pprohs r r rom rl tim n r unsuitl or our gol o img-s rnring s n intrtiv visuliztion tool. in lning nnot  voi, it is importnt to vlop  prinipl stuy o lning rtits.

. . Applications to street-level image-based rendering

In this stion, w vlop psyhophysil xprimnts to stuy th troff twn ghosting n popping rtits.  rt  numr o img-s rnr stimuli using iffrnt ptur n rn-ring prmtrs n show ths to prtiipnts s ompr to groun truth i.., vio o th sm sn. uh omprisons with groun truth hv n shown or or still imgs Fitzgion et al.,

 ut our work is th irst to pproh th prolm rom  prptul point o viw; prvious work hs ous on ompring img sttistis whih n  inomplt us th rltionship twn img sttistis n priv rnring qulity is not nssrily wll in.

In th isussion o th rsults o our stuy, w provi guilins to ilitt optiml ptur, s wll s motivt lgorithmi hois us in img-s rnring systms.

. . Experiment overview

 stuy img-s rnring in two o its most ommon orms: unstrutur lumigrph Buhlr et al.,  n ross-ing inh et al., .

Unstructured lumigraph ny urrnt img-s rnring systms us D gomtry o th sn n txtur it using pixls rom multipl input imgs ln with pproprit pr-pixl wights.

I th D gomtry is not prt, ghosting rtits r osrv i multipl imgs r ln. In on-trst, rupt tmporl popping rtits r osrv i  singl img is us to synthsiz ny output pixl.

 two prmtrs tht ontrol th svrity o rtits or  givn lvl o gomtri ronstrution, r th coverage twn input imgs, n th number of images blended, t ny givn pixl. Covrg is  wy to msur ptur img nsity, n thus ins th totl numr o imgs us to gnrt th inl rsult.  in ovrg in  nonil ronto-prlll viwing onition, s th numr o imgs ovring  givn point on th plnr proxy on vrg. Low ovrg uss slow popping with inrqunt ut long jumps;  ns st o input imgs uss fast popping with rqunt ut short jumps.

Blning mor imgs pr-pixl (.g. , or mor) inrss ghosting rtits; low ovrg uss wll-sprt ghosts whil high ovrg ls to smllr isplmnt twn th ghosts.  onut two psyhophysil xprimnts to ormlly stuy th troff twn ghosting n popping or th ov two img-s rnring stups:

Cross-fading

.  irst xprimnt ompr ghosting n popping rtits in unstrutur lumigrph using simpl plnr gomtry s  untion o th ovrg n numr o imgs ln.

.  son xprimnt stui th sm troff in ross-ing, n urthr ompr this to unstrutur lumigrph with iffrnt prmtrs.

Dtil isussion o th sttistil nlysis o xprimntl rsults is yon th sop o this thsis, pls rr to ngorp et al.,  or mor tils.

. . Experiment  Artifact analysis in Unstructured Lumigraph

 purpos o this xprimnt is to msur how ghosting n popping rtits fft th priv qulity o unstrutur lumigrph o rl çs. From th isussion in prvious stion, it is lr tht ghosting n popping rtits r th rsult o iffrnt lvls o lning twn input imgs.

 spii qustions w sk to nswr r s ollows:

• nr whih onitions o th rtits om ojtionl?

• hih typ o rtit is wors?

• ht is th optiml isply strtgy whn thr r rstritions on th numr o imgs tht n  ptur or stor?

Chpter . Evlution of Imge-sed endering using ereptul tudies (s vio ).

Results

In wht ollows, w rport visul qulity lvls s prntgs pning upon th position o th slir s mrk y th prtiipnts.  rport iffrns in visul qulity lvls s prntg points (pp). Intuitivly, w woul xpt  monotoni progrssion o qulity s w inrs th numr o imgs us ovrll.  ky qustion is how this is fft y popping n lning rtits.

Popping artifacts  top rows o Figur . rr to popping, sin only  singl img is ing us t ny givn pixl. For this s, th ovrll visul qulity pprs to pn on th svrity o th rtits whih vris rom sn to sn. is pnn on th sn is rvl y linr rgrssion o th qulity s  untion o ovrg. r is  signiint prrn or str popping in th Cornr sn (signiintly positiv slop o . pp pr pproximt ouling o ovrg). A mor surprising outom is th prrn or slower popping in th own Hll sn (signiintly ngtiv slop o -. pp pr pproximt ouling o ovrg). is rsult is o intrst sin it mns tht it is not nssrily vntgous to us  lrgr numr o imgs.

Ghosting artifacts

In ontrst, or ghosting rtits (s Figur . , ottom rows), linr rgrssion onirms our xpttion tht th ovrll visul qulity improvs s th ovrg grows (signiintly positiv slop o . pp pr pproximt ouling o ovrg). ith  sprsr st o input imgs, th imgs ln r urthr rom th output mr position on vrg n thror rsult in lrgr tur mislignmnt whn projt onto th plnr gomtri proxy.

 might xpt tht lning mor imgs togthr t vry pixl improvs pprn y smoothing out trnsitions. Intrstingly, howvr, w in tht priv visul qulity tns to improv whn wr imgs r ln pr pixl.  vrg priv qulity inrss y . pp rom to imgs http://www.youtube.com/watch?v=akaWUe0mum ln pr pixl. hn gomtry is not suffiintly urt, lning wr imgs t ny givn pixl rus lurring or th numr n sptil xtnt o ghost imgs.

Comparison between artifacts

It is intrsting to stuy whthr thr is  lr iffrn in qulity twn popping (using img pr pixl) or lning imgs pr pixl.  in tht th rltiv unplsntnss o popping n lning rtits pns on th prrn or st or slow popping in th sn. Howvr, in oth sns thr is  rossovr point; popping is prrr or low ovrg, n lning imgs whih rsults in limit ghosting, is prrr or high ovrg. Blning imgs rsults in mor pronoun ghosting whih onsistntly rnks lss prrl to th othr two options.

. . Experiment  Artifact analysis in Cross Fading

 gol o this xprimnt is to rss th ollowing qustions:  gnrt th stimuli or rl sns using th sm prour, with rl photogrphs rpling snpshots o th synthti sn.  groun truth ws gnrt y rnring  squn o imgs or th synthti s. For th rl sn, w i not ror  groun truth.

Experimental procedure For th ross-ing stimuli, w tst thr iffrnt urtions o ross-ing.  rt th stimuli with ,  n  o th pth unr ross-ing or oth rtiiil n rl stimuli. For th unstrutur lumigrph stimuli, w vri th ovrg n numr o imgs ln pr-pixl th sm wy s in Exprimnt , th only iffrn ing w i not tst th s or lning imgs pr-pixl us th rsults o th prvious xprimnt init tht inrsing th numr o imgs rom to i not improv visul qulity in ny s (s Figur . ).

In s o th synthti sns, prtiipnts wr prsnt with th stimuli n sk to "rt how (-1.6, 0, 2.4) (1.6, 0, 2.4) (-1.6, 0, 0) (1.6, 0, 0) z (-1.6, 2.4, 0)

(1.6, 2.4, 2.4) (-1.6, 2.4, 2.4) up vector orientation vector

R i ⋅ v = R r ⋅ R pr ⋅ v ⇒ R pr = R T r ⋅ R i ( . )
ot tht th invrs o  rottion mtrix is th sm s its trnspos. xt, ssum tht th input mr's ntr o projtion is c i n th rrn position is c r . Applying th rottion n trnsltion w gt:

R pr ⋅ c i + t pr = c r ⇒ t pr = c r -R pr ⋅ c i ⇒ t pr = c r -R T i ⋅ R r ⋅ c i (

. . Head tracking

 pproh in Chptr is vlop or sktop pplitions whr th usr nvigts using  mous.

In rnr trgt s th IB rsult with pth tst nl. is utomtilly pls th ojts t th orrt pth in th sn giving orrt (is)olusion ffts. us, our  systm n  lnly onvrt into n ugmnt rlity systm whr th most o th sn is rnr using IB n synthti ojts r rnr using th tritionl grphis piplin.

. Results

 now prsnt th rsults o our img-s rnring systm in th immrsiv sp in Figur .

n . .  ovrll rm-rt o th implmnttion is F; highr rsolution o th immrsiv vroming ths iffiultis will urthr inrs th utility o suh systms or linil prours. In In omprison, th mnul silhoutt nnottion us in Chptr took lss thn sons or h img, whih is ttr thn th options sri ov. Hn, inst o th ov options, w us mnul silhoutts in Chptr n thn utomt it using img ovrsgmnttion in Chptr .

  is si intuition, oupl with th vnt o igitl mrs l to th onption o img-s pprohs hum et al.,  whr hnhl mrs srv s th quisition vi. Img-s pprohs n  ivi into two lsss: thos tht ttmpt to synthsiz nw imgs rom th sm viwpoint ut with iffrnt pprn, n thos whih ttmpt to hng th viwpoint whil kping th sm pprn.  ormr r known s relighting pprohs whil th lttr r known s imagebased rendering pprohs. is thsis ouss on img-s rnring; th or prolm sttmnt is to "ptur  sn in  w photogrphs n visuliz it rom  novl viwpoint in  rgion roun th input viwpoints". Conptully, it woul  possil to otin physilly orrt rnrings only i th sn n  r-onstrut in ull til rom th input photogrphs. o this n, pprn mnipultion pprohs rquir th illumintion n mtril proprtis o th sn Dv, ; Lffont et al.,  n viwpoint mnipultion thniqus rquir gomtri ronstrution o th sn Buhlr et al., .

  roun ny o th input viwpoints. Exmpls o novl mr pths n  sn in th rsult stions o Chptrs n , on suh xmpl novl viw is shown in Fig. . . is is n inrsingly rlvnt ontxt sin th sussul ploymnt o virtul tourism nvly et al.,  n strt-si visuliztion Kop et al.,  s ommril systms lik iroso hotosynth, Googl trtviw t. Currntly ths systms r rstrit to trnsition twn viwpoints;th isply rsult is  rom on img to th nxt.  ition o free viewpoint img-s rnring to ths systms will llow th usr to hv  muh mor powrul immrsiv xprin.

Figure . :

 . Figure . : Exmpls o urn imgry us or img-s rnring.  top row shows on input imgs n ottom row shows th top viw o th sn with plholrs or input viwpoints.  novl viwpoint, shown in r, lis r rom th pth intrpolting th input viwpoints, whih w rr to s free viewpoint.  input viwpoints losst to th novl viwpoint r shown in lu.

Figure . :

 . Figure . : Ltst vrsion o Bing ps suggsts  ir mount o D inormtion is vill or omplt itis.

Figure . :

 . Figure . : ulti-viw stro. Dpth mps or input img in xtrt using Gosl et al.,  n Furukw n on, . onstrut rgions r shown in lk whil unronstrut rgions r shown in whit. ot th unvn istriution o pth smpls n omplt lk thro in som rgions.

Figure . :

 . Figure . : ur ronstrution. () Input img, () oisson sur ronstrution Kzhn et al.,  on D point lou xtrt rom Furukw n on, .  qulity o D mols is r rom prt or omplx sns whih mnists s rtits in img-s rnring.

Figure . :

 . Figure . : iwis-plnr ronstrution inh et al., . () Input img, () xtrt plns, n () rprojtion o gomtry in  novl viw. Clrly, th orgroun trtor gts mrg with th kgroun ç n rsults in rnring rtits.

Figure . :

 . Figure . : Img intrpoltion without D gomtry hjn et al., . Intrpolt img (mil) gn-rt rom input imgs (l n right). ot tht th slin twn th input img is o th orr o just  w pixls, whih is on o th most importnt limittions o suh mthos.

Figure . :

 . Figure . : Joint stro n img-s rnring. () tuio ptur stup us in itnik et al.  ; itnik n Kng  . () Input imgs r shown on l n right n intrpolt viw shown in th mil. s pprohs r limit to viw intrpoltion ovr irly smll slins.

Figure . :

 . Figure . : ritionl wrps Gl et al., or rtrgting.  input img (uppr l) is wrp to prou th rsult (mil).  slin msk (lowr l) is us to impos onstrints on th vrtis o  rgulr wrp msh, whih gt orm to th inl unrlying wrp msh show on th right.

Figure . :

 . Figure . : D img wrping Liu et al., .  input img (l) is wrp into iffrnt novl viws (mil n right). op row: wrp imgs showing th unrlying wrp msh n th D points in r. Bottom row: inl img r ropping th wrp rsult.

  Figure . : ()  us -input photogrphs n multi-viw stro to rt  ns D point lou. () ith our usr-ssist prprossing, th usr signts importnt silhoutts n w ru th D point lou to ∼ , pth smpls pr img. ()  silhoutts n pth smpls gui  silhouette-aware warp, ppli to imgs t h rm. () ur rnrr gnrts  high-qulity inl img whih hnls hr ss suh s trs n othr orgroun ojts.

  Figure . : () ilhoutts mrk t pth isontinuitis, () originl st o pth smpls otin rom th , ronstrut D points (shown s lk pixls), n () uniormly istriut pth smpls s-lt y our pproh. ot th rgions with no originl pth smpls r lso ill with nw smpls.

Figure . :

 . Figure . : rp msh tringl or n r  similr trnsorm.  lol oorint rm {(v 2 -v 1 ) , R 90 ⋅ (v 2 -v 1 )} tth to th tringl is shown in grn.

  R 90 is  ountrlokwis rottion y grs. Assum tht v 1 is th origin o th lol rm; v 2 n thn  xprss simply s (1, 0) in th lol oorint systm s shown in Figur . , n v 3 s (a, b) givn y:

Figure

  Figure . : () Constrin onorml tringultion us s wrp msh with onstrin silhoutt polylins shown in r, n () two prlll onstrin polylins (shown in r)  or th silhoutt polylin. Any thr onsutiv vrtis on ithr o ths polylins orm n glt.  tringls wg twn ths gs orm th lsti n.

Figure . :

 . Figure . : op row: An input img wrp to iffrnt novl viws without ny silhoutt hnling. Bottom row: m img wrp to sm viws using our silhoutt-wr isontinuous wrp.  lsti n tht sors ll th istortion is shown in r.

  Figure . : nring piplin. () rp imgs, () omposit txturs ℛ 0 n ℛ 1 . ixls rom sm wrp img r shown in sm olor. () ℛ gnrt y lning rom ℛ 0 n ℛ 1 . () oisson synthsis output ℛ ′ using ℛ s Dirihlt onstrints n grint rom ℛ 0 .

Figure . :

 . Figure . : Comprison o our rsult (l) with prvious pprohs (right) on Cstl- , trt-, Aqurium-tsts (top to ottom).

Figure . :

 . Figure . : Comprison o our rsult (l) with prvious pprohs (right) on r-, llowhous-tsts (top to ottom).

   st o imgs o  givn sn, tkn rom iffrnt viwpoints. Ar D ronstrution, w us Ahnt et al.,  to ovrsgmnt h input img, n ffiint lgorithm tht givs supr-pixls o pproximtly qul siz n with rgulr shps (s Figur . ()), unlik Flznszwl n Huttnlohr,  whih givs suprpixls o highly irrgulr shps n sizs u to lk to sptil omptnss.  not th st o ll suprpixls in n img y = {S i } i∈{0…n-1} .  projt th ronstrut D points into th img, suh tht th pth t pixl x is not y D[x] (shown in Figur . ()).  st o pth smpls insi h suprpixl is thus [S i ] = {x ∈ S i | D[x] > 0}.  istinguish two lsss o suprpixls: thos ontining lss thn 0.5% ronstrut pixls, whih w ll target superpixels (shown in grn in Figur . ()) n ll othrs whih w onsir to hv rlil pth.

Figure . :

 . Figure . : () Input img, () suprpixl ovrsgmnttion, () projt pth smpls, n () target superpixels mrk in grn.  suprpixls mrk in orng oul not  ssign pth rlily y our pth synthsis stp (tion . . ). s r mrk s hols.

  Figure . : op: trgt suprpixl in yllow n th source superpixels [S T i ] in lu. Bottom: orrsponing pth histogrms o [S T i ]. Dpth histogrm or th irst hs  singl pk initing rlil pth. plit pks in th son init tht source superpixels hv pth rom  iffrnt sn ojts. is is tru or th sour suprpixls t th tr silhoutt whih ontins D points rom th wll hin th tr (s Figur . (l)).

Figure . :

 . Figure . : ur pth synthsis s smpls with plusil pth (right) vlus to poorly ronstrut rgions shown in th l igur (n Figur . ()).

Figure . :

 . Figure . : L: uprpixl sgmnttion showing suprpixls t multipl pths s wll s pth smpls on-tin insi h suprpixl (shown s whit ots). il:  rgulr gri whih is us s wrp msh, ovrli ovr h suprpixl. ight: rp suprpixls n gri or  novl viw. rping h suprpixls inpn-ntly prsrvs ll silhoutts. ot how kgroun suprpixls sli unr orgroun.

Figure . :

 . Figure . : rp suprpixl imgs (l our) n inl rsult r lning (right).

Figure . :

 . Figure . : Angl θ us or omputing th pnlty P ng ( Ĩ i , x)

Figure . :

 . Figure . : A singl rm n orrsponing top viw o th sn or som o th tsts. From top to ottom, nivrsity, usum , itorHugo , itorHugo , Aqurium-, trt-, Commr, hool, ChplHill n ChplHill tsts.  top viw shows th input mrs in yllow, novl mr in r n th imgs slt or gnrting th novl viw in lu.

Figure . :

 . Figure . : riginl ronstrut points or on o th imgs rom som o our tsts. ough rhittur is wll ronstrut, rgions with vgttion or othr orgroun ojts r vry poorly ronstrut. ur pproh is pl o gnrting plusil rnrings vn or suh rgions.

Figure . :

 . Figure . : iw intrpoltion omprison or th llowhous-n usum tsts. Eismnn et al.,  pns on  D mol n thus shows signiint ghosting. In rgions with vry poor pth (s Fig-ur . ), our mtho is l to rt plusil rsults whil Gosl et al.,  rts  smr point lou. ilhoutt-wr wrp givs rsults similr to ours r . hours o mnul intrvntion to mrk urt sil-houtts n /orrt pth smpls, howvr som istortions r still visil whih om muh mor pronoun wy rom viw-intrpoltion pth (s Figur . ).

Figure . :

 . Figure . : Fr viwpoint nvigtion omprison or th llowhous-n usum tsts. ur mtho prous plusil rsults vn or viwpoints quit r rom th input imgs. In ontrst, th rtits o Eis-mnn et al.,  r lrly visil.  istortions inurr y th glol wrp in silhoutt-wr wrp r vn mor pronoun, spit . hours o mnul intrvntion.

Figure . :

 . Figure . : ilhoutt-wr wrp (l) isintgrts i ny pth smpls is behind th novl mr s shown in top viw (right). is prvnts th usr rom wlking "into" th sn. ur lol wrp os not suffr rom this limittion (mil).

Figure . :

 . Figure . : Exmpls o two o ommon rtits in img-s rnring, nmly ghosting (l) n prsptiv istortion (right).

   rt synthti D sns n rnr thm rom known mr positions.  thn rt  singl pln to pproximt th D sn n txtur it with th prviously rnr imgs using projtiv txtur mpping. is simults th typil worklow whr sns r pproximt y simpl D gomtry whih is txtur using photogrphs. ur stimuli inlu thr çs, h with prpniulr lonis t thr iffrnt protrusions, rnr rom our iffrnt ntriitis with rspt to th ornr.  lonis or othr ornrs o th D sn r snt on th D pln; w r intrst in th prption o ths ngls. ynthti stimuli giv us ull ontrol ovr th sn; w vlit our rsult our rsults on rl tsts in tion . . . Experimental procedure  us our iffrnt isply sizs or our xprimnts: " tlvision srn, " sktop monitor, " i n . " ihon. is is to tst th fft o isply siz on n-gl prption. In oth xprimnts, ll stimuli -iv simultion ngls, our ntriity ngls, n thr çs, h with thr lony pths, wr prsnt twi. In ition, thr prtiulr stimuli wr prsnt ight tims to llow us to ssss th onsistny o rsponss.  rpt th pross with iffrnt srns rsulting in × × stimuli, ounting th our srns n two xprimnts. orr o stimuli prsnt ws rnomiz. In th ginning, prtiipnts wr givn xtnsiv in-

Figure . :

 . Figure . : Exprimnt . () Exprimntl stup: w rpt th xprimnt on srns o iffrnt sizs -lrg siz , omputr monitor, i n ihon, () Hing vi us y prtiipnts to spiy priv ngl, n ()  prtiipnt prorming th xprimnt.

Figure . :

 . Figure . : rnshots o xprimnt on ihon (l) n tlvision (right).

Figure . :

 . Figure .: riv ngl pritions y th xtn rtinl hypothsis (), ompr to th ngl-mthing rsults or iffrnt ç pths (-) n vrg ovr ll pths (). Error rs init th twn-prtiipnt grmnt.  ott lin t ∘ rprsnts th sn hypothsis.

   us intrpolt mins vll,  to summriz th rtings t umult rom iffrnt prtiipnts ovr iffrnt çs n isply vis (s Figur . ). Clrly, th most un-ptl stimuli r in th lowr l n uppr right ornrs o ths plots, whih orrspon to lrg simultion n ntriity ngls o th sm sign.  most ptl stimuli r in th mil o th plot whr th simultion n ntriity ngls r smll in mgnitu -n th uppr l n http://vimeo.com/ Chpter . Evlution of Imge-sed endering using ereptul tudies

Figure . :

 . Figure . : ting rsults showing intrpolt mins ross ll prtiipnts: () ross ll ç pths, n (-) rtings or iffrnt ç pths.

  Figure . : imult viws long  nvigtion pth (s inst).  ht mp o prit rtings o Figur .is us s th olor shm to visuliz prit qulity long th pth.  viw shown hr is prit to  hvily istort, s init y th r lin-o-sight in th inst (ll "" in inst).  ptur mr is init y "C", ç y th lk lin n th lony ornr y th lk ot on th lk lin in inst.

Figure . :

 . Figure . : srv rtings (olor lins) or th thr sns ompr to prit rtings (ott lins) givn y our pritiv mol or prsptiv istortions (s tion . . ).

Figure . :

 . Figure . : Intrtiv nvigtion tool. () hn th usr trnslts, th inst shows prit rtings or ll mr positions kping th orinttion ix. () hn th usr turns, th inst shows rtings or ll possil mr orinttions kping th position ix.  pplition rstrits th usr nvigtion to rgions with -ptl prit qulity. Cptur mr is init y "C", simultion mr y "", ç y  lk lin n ornr y  lk ot on th lk lin in th insts.

  Figure . : Cptur nsity guilins. Inrsing th numr o ptur mrs inus lrgr rs o pt-l istortion (shown s lu).  osrv mpirilly tht  slin o on-ourth th istn o th ptur mr rom th ç rsults in lrgly ptl rtings vrywhr.  ptur mr position is shown in lk n th ç is init y th lk lin on top.

  Anothr pproh or synthsizing novl viws in th ontxt o viw intrpoltion is ross-ing inh et al., ; Kmlmhr-hlizrmn et al., . is pprs to  th s or ommril img-s rnring systms lik Googl trtviw. Hr th sm two input imgs r ln t h pixl n th lning wight o h img is sm or ll th pixls o th novl viw. nstrutur lumigrph is  gnrliztion o ross-ing us it n slt  iffrnt st o input imgs, h with its own lning wight, to synthsiz h pixl o th novl viw inpnntly. It is thror pl o r viwpoint img-s rnring whil ross-ing is rstrit to viw intrpoltion. Howvr, ross-ing my  vntgous in rtin ss whr it prsnts  iffrnt troff twn ghosting n popping rtits. hil trnsitioning rom img A to B, w n us only A or  rtion o th pth, thn ross- twn A n B or nothr rtion n thn us B or th rst o th pth.  urtion o th ross- trmins th svrity o th two rtits. Cross-ing or th ntir urtion o th pth rsults in mximum ghosting with vry smooth tmporl ffts whil ross-ing or  vry smll urtion (.g. < 5%) rsults in  visil rupt tmporl trnsition.

Figure . :

 . Figure . : Cornr sn (l) n own Hll sn (right) us or xprimnt .  ptur mr viwpoints (shown in grn) in th thr rows monstrt th thr ptur nsitis or ovrg us in th xprimnt.

Figure . :

 . Figure . : Exprimntl intr or th visul qulity rting xprimnt or Cornr sn (l) n own Hll sn (right).

Figure . :

 . Figure . : Intr or th ross-ing xprimnt.  prtiipnt is prsnt irst with (-) on o thr img-s rnring rsult n th rrn, n thn with () ll thr rnrings n th rrn.

Figure . :

 . Figure . : Immrsiv sp stup. () iw o th immrsiv sp long with imnsions n th oorint systm us or omputing th viwing rustum rom ny h position H insi th immrsiv sp. () ho-togrph o th immrsiv sp with img-s rnring rsult on th ront srn.  hir n rpt r us in th minisn rpy xprimnt s shown in Figur . .

Figure . :

 . Figure . : L: visuliztion o ptur mrs or on o th tsts. uh  ptur llows nvigtion in  signiintly lrg portion o th sn ( m× m shown hr s top viw).  ronstrut sn n th immrsiv sp (shown in orng) n  in totlly iffrnt oorint systms. ight: w rgistr th immrsiv sp (shown in orng) n th ronstrut sn y ligning on o th input mr positions with  rrn point in th immrsiv sp.

  Figure . : op viw o immrsiv sp to show th rust or l n right y.  molviw mtris n rust or l n right y r omput using th h positions c H,l n c H,r , whih r omput y prturing th h position rturn y th h trkr in l n right irtions prlll to th srn.

Figure . :

 . Figure . : nrings gnrt y our  systm. s srnshots r gnrt y mirroring th  rn-rings on  sktop monitor.  inst is not isply in th  rnrings. It is shown hr to illustrt th h position in th D sn uring intrtiv nvigtion.

Figure . :

 . Figure . : hotogrphs o our  systm running in th immrsiv sp.

Figure . :

 . Figure . : minisn rpy xprimnts. L: iffrnt tsts us s known n unknown lnmrks o th ity. il: usr immrs in our img-s  systm. rtiipnts r st to ommot possil moility rstritions. ight: ur  systm nhn with ingr gsturs n synthti ojts.

  Figure . :() Immrsiv sp top viw: orinttion vtors o mr us to isply ront, l n srns r shown in r. r-slting input imgs using ths orthogonl orinttions ls to inonsistnis t srn ounris us iffrnt imgs r slt or iffrnt srns. () Immrsiv sp si viw: th orin-ttion or ottom srn isply mr (r) is lmost orthogonl to th ptur mrs orinttions (grn).  sn loor is ptur t grzing ngls n wrping ths suprpixls ovr  lrg ngl twn ptur mrs n ottom srn mr ls to rtits.

  photo olltions n  o muh utility or ugmnting prsonl photogrphs, th omin st n  us or vry high qulity rsults. ur mphsis on sul pturs using only hnhl mrs is inspir y th ultimt gol o using ritrrily unstrutur intrnt photo olltions. Apprn mnipultion thniqus r inispnsl or this gol sin photogrphs r xpt to hv vrying pprns.

High quality image-based rendering

   min ontriution o this thsis is in th orm o img-s rnring lgorithms sign or urn nvironmnts tht r pl o prouing high qulity rsults in th sn o urt D ronstrution. It is wll-known tht multi-viw stro mthos Gosl et al., ; Furukw n on,  n prou imprssiv rsults or rhittur ut thir prormn grs on luttr sns. ur pproh is in lin with th rnt trn o img-s rnring systms tht us D point lous prou y multi-viw stro irtly Gosl et al., ; inh et al., ; Kop et al.,  rthr thn xpting  D msh whih n  xtrmly hr to otin or luttr sns, spilly urn imgry ontining vgttion, vhils, rhittur t.

Free viewpoint navigation is is th irst rsrh work in img-s rnring to tivly pro-pos r viwpoint nvigtion whr th novl or simult mr is llow to nvigt quit r rom th input viwpoints in th sn. Almost ll prvious pprohs, th rlist to th vry ltst, hv only rss viw intrpoltion Chn n illims, ; Fitzgion et al., ; itnik n Kng, ; hjn et al., ; Gosl et al., ; tih et al., ; inh et al., ; Kop et al., . is is n importnt issu us r nvigtion xposs th tru vntg o suh systms y llowing  sn ptur with just  w photogrphs to  visuliz in rih tils in  vrity o pplitions, on o thm ing h-trk virtul rlity systms, n rly prototyp o whih is lso monstrt in Chptr . Perceptual analysis  lssi vlution mtho or img-s rnring hs lwys n viwr opinion. om pprohs us img sttistis Fitzgion et al., ; howvr suh mtris r suit-l whn th pproh is xpt to prou physilly-orrt rsults. ost img-s rnring systms trgt plausible or good looking rsults, with som pprohs using non-photorlisti ffts Gosl et al., .  only wy to vlut suh systms is y mns o prptul stuis. is thsis proposs prptul nlysis o visul rtits whr simpl rnring n input t stups r us to isolt th rtits.  t rom th stuis llows us to orrlt th svrity o rtits with sn or rnring prmtrs. . Current and potential applications  ltst vrsions o ommril prouts lik Bing ps show mssiv ronstrut urn rs (s Figur . ). is gr o D inormtion is suffiint or rly xprimnts with our img-s rnring pprohs. It is lr rom Figur . tht th D inormtion is suffiint to rnr lrg struturs vry wll ut tils suh s trs t. r rprsnt s lurry los.  pprohs rom . . verview

  thy r sign or los ojts tht hv n photogrph rom ll sis. y n to  initiliz with  ouning ox n o not giv goo rsults or unoun sns suh s

	urn imgry.
		 ov limittion hs inspir lrg sl multi-viw stro systms. Gosl et al. 	 prsnt
	 multi-viw stro pproh or lrg ommunity photo olltions rom th intrnt. ollys et al.
		 prsnt  rl tim pproh or vry lrg sns ptur using vhil mount mrs sim-
	ilr to Googl trtviw. th-s multi-viw stro Furukw n on,	 mths tur
	points twn input imgs, stimts thir pths n uss ths to stimt pths o nighoring
	pths. Extnsions o Furukw n on,	 hv n us to ronstrut ity lvl ronstru-
	tions in Agrwl et al.,	,	 rom hunrs o thousns o photogrphs. thr pprohs
	Multi-view stereo  vlopmnt o lrg sl strutur-rom-motion nvly et al., prohs, known s surface reconstruction, n  lssii into two typs -ronstrution rom un- hs
	provi  stl solution to th long stning prolm o utomti mrkr-lss mr lirtion orgniz point lous n thniqus tht us unrlying strutur in point lou t. rominnt
	or lrg unorr imgry. is vlopmnt hs grtly vn multi-viw stro rsrh llow-xmpls o th ormr inlu Hopp et al., , oving lst squrs Lvin, , oint st sur-
	ing rsrhrs to xprimnt with  wi vrity o tsts i.. outoors, inoors, ommunity photo s Alx et al.,  n oisson sur ronstrution Kzhn et al., . s pprohs

Dspit imprssiv vns in rnt yrs, stt o th rt D ronstrution n still giv inurt rsults whih n prou rnring rtits whn us in xisting img-s rnring rmworks Buhlr et al., ; Eismnn et al., ; inh et al., ; Gosl et al., .  isuss rnt vns in multi-viw stro n thir limittions in this stion. tro ronstrution hs n on o th most tiv omins o rsrh in omputr vision. itz et al.   prsnt  omprhnsiv ovrviw o th rly work in multi-viw stro ronstrution. ost o rly rsrh in D ronstrution ous on isolt ojts suh s sttus. nvly et al.   opn th oors to ronstrution o vst opn urn sns. Hr w rviw only th rnt vlopmnts whih r rlvnt to img-s rnring. olltions t. n not just sprt ojts. Gosl et al.   us pln-swp stro or stimting pth mps or h input viw. inh et al.   us  volumtri grph ut to stimt ull D gomtry o n ojt. hil th rsults o ll ths pprohs r omplling, thir most importnt . . D reonstrution limittion is tht suh s Ltut et al., ; Hip et al.,  stimt pth o  lrg numr o intrst points ol-low y Dluny tringultion; th inl D mol is gnrt y omputing n insi-outsi ut on th ttrhr rsulting rom th Dluny tringultion.  rsults r nsr thn Furukw n on,  n sm losst to thos provi y th ommril solution Autosk D .  us Furukw n on,  to rovr  D point lou rom input imgs; th hoi o multi-viw stro lgorithm is not ritil n n  rpl y Gosl et al., ; Hip et al.,  or ommril prouts lik Autosk D or Aut D mrt DCptur . hil signiintly iffrnt in implmnttion, ths multi-viw stro pprohs r irly similr in prinipl. y mth img turs twn imgs in unorgniz photo olltions n stimt thir pth. y thn us this pth to initiliz pth stimtion or nighoring img pths.  rsult o ths multi-viw stro pprohs is thus irly similr. As shown in Figur . , ths p-prohs giv vry goo rsults or rgulr struturs lik çs. Howvr, th qulity is muh wors or txtur-poor rgions, usy txturs n irrgulr gomtry t.  istriution o ronstrut points or pth smpls is highly irrgulr n vry sprs in som rgions.  loliztion o silhou-tts is lso inurt on mny sn ojts. Surface extraction from D point clouds  D point lous omput y multi-viw stro hv to  onvrt into polygon mshs in orr to  rnr s ontinuous soli ojts. s p-ronstrut  watertight mesh whih is not pproprit or open urn sns. Howvr, this prolm http://www. dapp.com/catch http://www.acute d.com/smart dcapture/ () Input img () Kzhn et al., 

  intrpoltion; th rsults r us in th mo Who Cares . s pprohs r powrul n roust ut only hnl smll slins twn imgs. hjn Ghosting n mislignmnt r lrly visil whn th proxy is not urt s shown in ths xmpls.

	thy ompnst or inurt gomtry y using  vry lrg numr o input imgs. Amint point
	lous Gosl et al.,	 is  viw intrpoltion pproh tht uss  non-photorlisti rnring
	styl in poorly ronstrut rgions. inh et al. 	 prsnt img-s rnring or rltiv
	surs y ronstruting two pth lyrs or rltions, whih is urthr improv in th orm o
	 grint-omin pproh Kop et al.,	. Both o ths pprohs omput pixl-ns pth
	mps to intrpolt twn imgs. Bht et al. 	 lso us  grint omin pproh to trnsr
	() Buhlr et al., tils rom  smll numr o high rsolution photogrphs to  lrg numr o low inition vio 
	rms. y us D gomtry in th orm o sprs st o D ronstrution to rgistr th photogrphs
	n vio rms. thr thniqus hv n vlop to rowsing vio rhivs Blln et al.,	;
	ompkin et al.,	; ths pprohs trnsition twn vio strms using D ronstrution o
	th sn.  gol is to provi  smooth trnsitions rom on vio strm to nothr rthr thn
	plusil novl viws.
	() Eismnn et al., Discussion  vlopmnt o th ov pprohs whih us D gomtry hs ntwin img-
	s rnring with multi-viw stro. s pprohs whn omin with st possil ron-
	strution o th input sns using ithr o Gosl et al.,	; ollys et al.,	; Furukw n
	on,	 rprsnt stt o th rt in img-s rnring systms.  us this omintion to
	show omprisons in Chptrs n .  inility o D ronstrution to prou prt pth  to work with img sgmnttion whih provis silhoutts. Fitzgion et al.   us img-s priors to prsrv strutur in synthsiz viws in  omputtionlly xpnsiv optimiztion. Hornung n Kolt   xtrt viw-pnnt pth mps irtly rom input imgs n mrg thm in rl tim using  min iltr on th G. y r sign to intrpolt stro pirs or strutur stu-mps or urt silhoutts or urt D mshs omin with th inility o rnring piplins plnopti moling tht uss D gomtry o th sn in th orm o  D pln. s pprohs to sussully ompnst or ths rrors mns tht vn th most sophistit img-s rn-us  rstilly smllr numr o input imgs thn prvious pprohs suh s light ils Lvoy n Hnrhn, . ith th vlopmnt o multi-viw stro Furukw n on, ring systm woul ll short o our trgt o rnring th omplx urn sns using s w imgs , uto-mtilly ronstrut point lous n D mols hv rpl mnully mol proxis Dv s possil in  r viwpoint wlkthrough snrio.
	io pturs using ptur rigs s shown in Figur . . s pprohs xploit th prior knowlg et al.,  or singl D plns Higl et al., .
	tht two mrs r nighors, n trt multi-viw tsts in  pirwis shion. y sm to  tightly rstrit to smll slin stuio pturs n viw intrpoltion. y r lrgly s on nstrutur lumigrph Buhlr et al.,  is  gnrliz img-s rnring rmwork. It omputs th olor o trgt pixls y kprojting thm on to th D gomtry n rprojt-. Image warping
	stro ronstrution, n n  xpt to suffr rom th sm prolms s multi-viw stro (s ing into th input viws. Contriution rom multipl input imgs is ln using wights omput A mjority o img-s rnring pprohs Buhlr et al., ; Eismnn et al., ; inh et al.,
	tion . ) i us or unorgniz wi slin urn imgry. using rltiv istns twn ntrs o projtion o mrs.  min vntgs r tht it l- us D gomtry s th only onstrint or rprojting n input img to  novl viwpoint. 
	lows input imgs to  tkn in n ritrry mnnr n oul xtn to r viwpoint wlkthroughs. sk to ompnst or insuffiint D y using img-s pproximtions. o this n, w wrp Image-based rendering using D geometry ost morn pprohs us xpliitly omput D is pproh in works vry wll givn prt gomtry ssuming no olusions. Floting txturs input imgs to novl viwpoints using D gomtry s  so onstrint whih is rgult y othr gomtry in orr to hnl lrg slins twn input imgs n r viwpoint img s rn-ring.  lumigrph Gortlr et al., Eismnn et al.,  introu olusion hnling n  orrtion pss s on optil low tht l-D onstrints whih sk to prvnt istortions in th inl rsult.  mthmtil tools w us r  ws th irst pproh to suggst us o ors D gomtry ompr to plnopti moling illn n Bishop, lvits lning rtits u to inurt gomtry.  min limittion is tht th olusion hnling inspir y img wrping pplitions whih llow usrs to orm n input img in  vrity o wys.  n light ils Lvoy n Hnrhn, . iw pnnt txtur mpping Dv et al., ssums urt silhoutts in th D proxy n optil low orrtion is limit to mislignmnt o  hllng or ths pplitions is to gnrt ontnt-wr wrps, mnipulting iffrnt prts  uss  usr-rt mol o simpl rhittur n projtiv txturing to prou omplling wlkthroughs. Higl et al.  up to -pixls. Huswisnr et al.   prsnt  rl tim visul hull omputtion or ynmi o th input img in iffrnt wys without introuing visil isontinuitis, ormtions or othr  prsnt sns; thir img-s rnring pproh is similr to Buhlr et al., , pt or tim volv-rtits, synthsizing plusil imgs whih ppr just s onsistnt s photogrphs. is hs l to
	http://graphics.tu-bs.de/projects/whocares/ ing D gomtry. In othr rlvnt work, Alig et al.   vrity o pplitions suh s rtrgting, viw morphing, img intrpoltion t. , prsnt wlkthroughs o inoor sns;

et al.   rport  -pixl mximum slin. Bing lrgly olivious to D gomtry, thy hv iffiulty hnling olusions n r lso stritly rstrit to viw intrpoltion. s r ritil rstritions in our ontxt; hnling wi slins n olusions r two most importnt prolms. As  rsult, ths pprohs prov lrgly insuffiint in our snrio. Joint reconstruction and image-based rendering ny pprohs stimt pth/isprity -twn img pirs with th sol purpos o img-s rnring. itnik et al.   omput is-pritis twn pirs o imgs using rkov rnom il priors n intrpolt thm using  ly-r rprsnttion uilt on top o isprity mps. is is pt in itnik n Kng,  Figure . : Img-s rnring using D gomtry.  lmost imgs r rsults tkn irtly rom th rsptiv pprs. il n right imgs r rsults on othr tsts gnrt using our implmnttion o Buhlr et al.,  (s tion . or tils) n uthors' implmnttion o Eismnn et al., .

. . ereption of imge-sed rendering rtifts

	whr n is th numr o pixls in th img. is quikly oms vry slow or vn mium siz
	imgs (roun mgpixls). usqunt work on ovrsgmnttion n n lik,	; Flzn-
	szwl n Huttnlohr,	; ori,	; li n otto,	; Lvinshtin et al.,	 improv
	th omputtionl ffiiny n llow or mor usr ontrol ovr siz n shp o suprpixls. tt o
	th rt ovrsgmnttion LIC Ahnt et al.,	 n sgmnt lrg imgs in  w sons whil
	prouing rgulrly shp suprpixls s shown in Figur . . A til omprison o ovrsgmnt-
	tion lgorithms n  oun in Ahnt et al.,	.  us LIC or ovrsgmnttion in Chptr .
	() It is importnt to not tht suprpixls orm th sis o mny img lssiition n rognition () () () () lgorithms us thy ptur ll prominnt silhoutts in n img.  us suprpixls or th sm
	Figure . : Comprison o iffrnt ovrsgmnttion lgorithms () Flznszwl n Huttnlohr, ori, , () li n otto, , () Lvinshtin et al., , n () Ahnt et al., rson. thr pplitions o ovrsgmnttion inlu viw intrpoltion itnik n Kng, . tih et al., , pth stimtion Cigl et al., , improving ronstrution o mn-m stru-, () ;
	turs ičušík n Košká, Discussion  ov isussion shows tht vritionl wrps r pl o gnrting plusil  t. s pprohs irtly stimt pth or suprpixls n n  onsir piwis plnr ronstrution pprohs whr suprpixls provi th piwis wrp imgs using  sprs mpping twn input img pixls n trgt img pixl. In Chptrs plnr rgions. n , w monstrt shape-preserving image warping s on vritionl wrping, tht prsrv In ontrst, w us suprpixls or linting rgions with unrlil or poor pth n using sp-img silhoutts, provi olusion hnling n r roust ovr wi-slins twn input n il prossing or suh rgions (s Chptr ). s unronstrut rgions woul rmin s suh trgt viwpoints. vn i w us itnik n Kng, ; ičušík n Košká, ; thy il to ronstrut us
	. o rsons tht hol or ny D pth stimtion pproh -lk o txtur, stohsti txtur, omplx Image segmentation gomtry, insuffiint imgs t.
	In ontrst to th D wrping onstrints in ll o th ov thniqus, D img n o th importnt prolms in img-s rnring is olusion hnling n silhoutts. As wrps r gui y sprs/qusi-ns D gomtry n morph n input img to nothr viwpoint.  motivtion oms rom D shp mnipultion Igrshi et al., ; hr et al., isuss prviously, D ronstrution os not lwys prou prt rsults t silhoutts. n th  whr othr hn, purly img-s pprohs r muh ttr t xtrting silhoutts.  stt o th th usr nimts  D skth y pulling  smll numr o hnls n shp ormtions r mini-rt in img sgmnttion Hoim et al., ; ir et al., ; Arlz et al.,  n lssiy  miz y rigi/onorml/similr/ffin onstrints. D img wrping rpls th usr hnls with vry wi vrity o ojts. Howvr, ll img lssiition pprohs r s on mhin lrning D ronstrut points or pth smpls whih n  rprojt into ritrry viwpoints. Liu et al.  n nnot  xpt to sgmnt ll ojts prtly spilly i n img hs multipl prominnt  us this i or D mr stiliztion y wrping h vio rm rom th originl viw-ojts. is n l to unr-sgmnttion n prts o ojts n  miss-lssii. Appnix A point to  viwpoint on  stiliz mr trjtory (s Figur . ). y o not hnl olusions, howvr this is not  mjor prolm in this ontxt us vio rms r wrp to viwpoints in omprs img lssiition pprohs n shows ilur ss.
	los viinity. Oversegmentation Img ovrsgmnttion is th pross o iviing th img into hunrs o
	 most importnt ingrint o D img wrping is rgulriztion in th orm o rigi Igrshi smll rgions o homognous img ontnt ll superpixels. uprpixls ptur all th silhoutts
	et al., whil lso prouing  lrg numr o runnt ounris. In ontrst, mhin lrning s , ffin hr et al.,  or similrity Liu et al., ,  onstrints. s p-
	prohs monstrt tht  vry sprs st o guiding constraints, .g. usr hnls Igrshi et al., pprohs n on miss silhoutts in som rgions.  runnt ounris prou y ovr-
	or D points Liu et al., sgmnttion r unsirl, yt nign.  gurnt tht ll silhoutts r ptur y suprpixls , n  sussully ompnst y ths rgulriztion onstrints whih
	msk privl ormtions. Howvr, olusion hnling n wrping ovr lrgr slins r still is n importnt vntg.
	opn prolms.  sminl work on suprpixls hi n lik,	 us sptrl nlysis o n n × n mtrix,

.

Perception of image-based rendering artifacts ur trgt is to synthsiz plausible novl viws o vry omplx sns using s w photogrphs s pos-

  omprtiv stuy o iffrnt rtits prsnt in this thsis.  vntg o impliit stuis ovr xpliit psyhophysil stuis is tht th rsults r not is y th ntur o qustions or tsks prorm y prtiipnts. Howvr, th min rwk is th low signl-to-nois rtio in th t ror y th EEG, s not n prtilly llvit in ust et al., .  rltionship twn inrns rwn rom xpliit n impliit stuis is lso unlr spilly whn th two r ivrgnt. o th st o our knowlg, thr r no prptul stuis tht invstigt ghosting n popping rtits with rspt to h othr in th ontxt o img-s rnring. Exssiv lning ls to ghosting rtits ut rts smooth trnsitions twn viwpoints whil th ontrry givs risp imgs ut popping rtits in trnsitions.  stuy o ths opposing rtits is ritil us

	tion o D shp. Consir th pitur o  slnt rtngl. gwik 	 ormults th priv
	slnt o th rtngl using vnishing points; mor spiilly, using th ngl twn  lin rom th
	viwr to th vnishing point n  lin rom th viwr to th rtngl. hn  pitur is viw
	rom th ntr o projtion, popl r quit urt t rovring th D gomtry o th originl
	sn, inluing th slnts o surs in tht sn mith n mith,	; Coopr et al.,	. I th
	viwr's y is offst rom th ntr o projtion, prsptiv-s us no longr spiy th origi-
	nl D sn; inst, thy spiy  iffrnt, istort sn. ur trgt is to quntiy this fft s 
	untion o ptur n viwing prmtrs.
	Rendering artifacts Almost ll img-s rnring pprohs umult visul ontnt rom
	multipl input imgs s th novl viwpoint trnsitions ross th sn. islignmnt twn in-
	put imgs whn thy r rprojt into th novl viwpoint, us y inurt orrsponns
	twn th imgs, rsults in rnring rtits. A mjority o pprohs ln multipl imgs,
	whih n l to ghosting rtits whil voiing lning n l to tmporl isontinuitis known
	s "popping" rtits. om pprohs us prptul stuis to tt rnring rtits or slt
	rnring prmtrs to mitigt th rtits. orvn n 'ullivn 	 prsnt prptully-
	motivt omprssion thniqus or th lrg mounts o img t rquir or lumigrphs. Brgr
	et al. 	 tt ghosting rtits in imgs using img gs; howvr, thy o not nlyz th -
	tors tht l to ghosting rtits. hwrz n tmmingr 	 prsnt  prptully-motivt
	pritor or popping rtits or gnrl omputr grphis pplitions. y o not ompr pop-
	ping rtits to othr ltrntivs suh s lning.  most losly rlt prptul stuy on img-
	s rnring thniqus is th stuy o ovrll visul qulity o pnormi trnsitions orvn n
	'ullivn,	. y onlu tht th mgnitu o th pth isontinuity t silhoutts is  ky
	sil.  vill img t is typilly muh sprsr thn wht woul  rquir to gnrt prt tor in visul qulity. is work ws n importnt irst stp towrs th gol o unrstning th pr-
	novl viws. is hmprs th lvl o plusiility us o inorrt prllx, prsptiv istortions, ption o rnring rtits. ompkin et al.   ompr ross-ing ffts using iffrnt orms
	sptil rnring rtits lik ghosting n tmporl rtits lik popping. o D rprsnttions o th sn -ull D gomtry, D point lou, singl pln, D orrsponns,
	r hs n rnt intrst in stuying prption or imgs gnrt using img-s rn-no gomtry or rupt hngs with no ross-ing t ll. y onlu tht using ull D gomtry
	ring.  gol o th mjority o ths pprohs is to mk "prptully-optiml" lgorithmi -is y r th st solution. ur stuy n  onsir orthogonl to thir work us w ix th D
	isions tht hlp msk wy rtits, somtims ompni y prptul stuis to onirm lgo-gomtry o th sn n vry th rnring prmtrs tht ontrol th gr o ghosting or popping
	rithmi hois. (s tion . ), whil ompkin et al.,	 ompr iffrnt orms o D gomtry or gnrting
	trnsitions whil kping th rnring prmtrs ix.  ov pprohs stuy "xpliit visul
	Perspective distortions  most ommon prolm with rprojting n img ptur rom on prosss" whr th prtiipnts r xpliitly sk to jug th qulity o th stimuli y prorming
	viwpoint into nothr viwpoint is prsptiv istortion. rsptiv is n importnt u tht hlps
	trmin th D lyout o  sn up to  sl tor gwik,	. ision sin hs stui pr-
	sptiv in th ontxt o pitur prption ousing on how prsptiv istortions fft th prp-

tsks or nswring qustions. ust et al.   stuy "impliit visul pross" whr prtiipnts' rspons to stimuli is msur y n EltroEnphloGrph (EEG). ust et al.   show tht iffrnt rnring rtits invok iffrnt rsponss rom th rin. is osrvtion justiis th n or th most img-s rnring pprohs prsnt  troff twn ghosting n popping rtits y twking prmtrs until th rsult "pprs goo".

  Hole illing  pth smpls rtin r imtion r spltt on th img. I  winow o n×n pixls os not ontin ny spltt pth smpls, w mrk this winow s  hol.    smpl whih projts insi this winow t th pth o nrst nighor smpl.  hoos n th sm s th splt siz us rlir (s Figur . ()). It is importnt to not tht th nwly  pth smpls r gnrt on  pr-img sis n r not photoonsistnt. y o not ugmnt th ronstrution; thy simply provi onstrints or stilizing th img wrp sri in tion . . Silhouette depth samples o voi mixing orgroun n kgroun smpls on ithr si o th silhoutts, w onsrvtivly rmov ll xisting smpls within  smll istn o silhoutts n rpl thm with smpls using th pth rom thir rsptiv si. is nsurs tht th silhoutts

).  splt siz is not ritil s long s it is not too smll; w us × in our xprimnts. lrly sprt smpls with iffrnt pths.  osrv this os not ompromis wrp ury us suh rgions r too smll to ontin signiint pth grints.

Manual outlier removal is optionl stp is usul whn thr r mny smpls with inorrt pths. ur intr shows smpls with olor o pths, whih mks it sy or th usr to intiy suh outlirs (s Figur . ()). y n  rmov y  simpl slt-n-lt' oprtion t ny stg o th pross.

In our xmpls, D ronstrution prou , -, pth smpls or h img. sing th pross sri ov, w rtin -smpls pr img.  osrv tht -K smpls i not improv th wrp qulity n lss thn smpls l to wrping rtits.  optiml num-r o smpls tully pns on sir output img rsolution. A highr sir lvl-o-til woul rquir mor onstrints or th wrp, hn mor smpls.  ntir pross, inluing usr intrtion (i n), took out minuts or  tst o imgs.

  o minimiz th istortion us y th wrp, th wrp must  lolly shp-prsrving.  thror us  similrity nrgy trm, suh tht th trnsormtion o h msh tringl is s los s possil to  similrity trnsormtion. Anlogous nrgy trms wr us in Liu

	et al.,	; hng et al.,	; ng et al.,	. Consir  msh tringl T with vrtis

)

  Hr, i is th st o ll pth smpls, i is th st o ll wrp msh tringls n ℰ i is th st o glts o img I i .  us th rprojtion E p n silhoutt nrgy E b s strong onstrints whih gui th optimiztion n th shp prsrving trm E s s  wk rgulrizr to prvnt istortions on ll tringls outsi th lsti n. us, w st w p = w

	omput th inl onstrin onorml Dluny tringultion using C	inu,	. Finlly, th
	wrp mshs r rt y rnring th wrp mshs using originl vrtx positions s txtur
	oorints.
	.	Rendering
	o synthsiz  novl viw, w irst slt  st o our imgs whih n  us or ll pixls in th inl
	img.  osrv tht thr to our input imgs r suffiint to synthsiz  novl viw.  wrp
	ths imgs to th novl viw using our wrp ormultion rom tion . .  thn omput lning
	wights o th ontriution rom h wrp img t h pixl in  pixl shr n rtin th st
	two nits.  inlly ln th nits using wights whih r vry similr to unstrutur
	lumigrph rnring Buhlr et al.,	, xpt tht w us pr-pixl lning. Buhlr et al. 	

b = 2, w s = 0 (or lsti n tringls) n w s = 1 or tringls outsi th lsti n. Evry input img I i hs its own wrp msh n n ssoit linr systm E i ; w us pn to wrp multipl imgs on prlll ors.

 nrgy E i is qurti in th unknown wrp vrtx positions ṽ ; w thror it hs  uniqu minimum tht is oun y solving th sprs linr qution ∇E i = 0.  us th irt sprs Cholsky solvr  olo, . ot tht th systm mtrix os not hng or novl viw prmtrs viws sin only th right-hn si o th linr systm hngs.  thror promput th mtrix toriztion n only prorm k-sustitutions t runtim.

In our xprimnts, w oun n initil × wrp msh to  suffiint or × pixl output rm rsolution. is smpling is lolly rin to insrt th silhoutt gs, s sri ov.  omput lning wights t wrp msh vrtis only n us stnr pnGL ilinr intrpoltion to otin lning wights or h pixl whil w omput th lning wights or h pixl irtly in  pixl shr; this givs ttr rsults s ompr to th originl pproh.

  . ot tht oth txturs ℛ 0 n ℛ 1 ontin ontiguous pths o pixls look up rom th sm input imgs.  grint insi ny o ths pths is th sm s th grint o th originl img, whih hlps rtin risp til.  or th grint o th inl img ℛ ′ to 0 or ll pixls lying on pth ounris ℬ 0 o ℛ 0 , whih mounts to smooth ompltion o thos rs.  synthsiz th inl output y solving th oisson qution:∇ 2 ℛ ′ = iv sujt to ∇ℛ ′ | ℬ 0 = 0 ( . ) initiliz th oisson synthsis with th ln rsult ℛ n prorm  w multigri Joi itr-tions. is hlps smoothn pth ounris n limints ghosting gs us th grint mp is rt rom txtur ℛ 0 . is n  sn in th inst o Figur . (right). ur pproh thus prrs smooth sptio-tmporl trnsitions ovr lning mor imgs.

		 with  orgroun ojt
	(trtor) in vry wi slin imgs. iwis-plnr ronstrution inh et al.,	 givs un-
	ptl rtits on th trtor. Aqurium-hs multipl orgroun ojts t iffrnt pths,
	whih r known to  iffiult to hnl hjn et al.,	. trt-n r-show our rsults
	or gnrl urn sns with vhils n trs. r-tst h mny inorrtly ronstrut
	points on th tr, whih wr mnully rmov.  prsn o vgttion mks D ronstrution
	n sur xtrtion vry iffiult; mnully moling suh sns is lso vry iffiult n tious.

Poisson synthesis  ln txtur ℛ my hv sptil isontinuitis.  pth ounris or sms o ℛ 0 n ℛ 1 my rmin visil, spilly th lsti n rgions whr th txtur origints rom n img with sustntilly iffrnt viw. o rt  ttr inl output img, w us oisson synthsis to inpint th sm rs mor grully.  rt th grint mp rom ℛ 0 , n its . . esults ivrgn mp iv . Results o provi ir omprisons, w omin stt o th D ronstrution Furukw n on, ; Kzhn et al.,  with th st st o thniqus vill or r viwpoint wi slin img-s rnring.  us unstrutur lumigrph Buhlr et al.,  rnring with pr-pixl lning, in ontrst to vrtx lning us in th originl mtho.  urthr  visiility hking lgorithm Eismnn et al.,  to giv olusion hnling.  us this hyri pproh or omprisons with our rsults in Figur . n . (s lso vio ). Givn th lk o urt gomtry or or-groun ojts, prvious pprohs hv ghosting rtits n inorrt olusion hnling. isiility hking os not llvit olusion rtits us th proxy us or rting visiility mps is r-ronous.  prsnt th rsult o our pproh on hllnging tsts whih nnot  ronstrut u-rtly. Cstl- is  stnr multi-viw stro tst trh et al.,  slins in our tsts vry rom pixls (  o img hight) in Aqurium-; pixls ( ) in trt-to pixls ( ) in Cstl- . ith miniml usr input, our pproh gnrts sults vio: http://vimeo.com/ 0

  to iffrnt sns (s Figur . ), inluing on rom iroso hotosynth n two rom ollys et al., .  monstrt intrtiv nvigtion sssions or ll sns, whih show tht our pproh pushs th limits o img-s rnring to r viwpoint nvigtion n omplx urn imgry. At th sm tim, our pproh is lso mor sll n omputtionlly  ovrsgmnttion n projt pth llow us to intiy poorly ronstrut suprpixls in h img. Dpth synthsis ills in poorly ronstrut suprpixls using pth rom "similr" suprpixls o th img.  rt  grph strutur with suprpixls s nos n in  rul trvrsl o th grph whih llows us to intiy st mthing suprpixls in trms o olor n sptil proximity.  kp th thr st mthing suprpixls n intrpolt th pth rom ths suprpixls to   smll st o nw pth vlus into th originl poorly ronstrut suprpixl. Local shape preserving warp uprpixls now ontin ronstrut pth rom multi-viw stro or plausible synthsiz pth.  pth smpls my  inurt or noisy or not photoonsistnt; rprojting thm will l to visil rtits in rnring. o llow plusil novl viws, w prorm  local shp-prsrving wrp on h suprpixl iniviully, in ontrst to Liu et al.,  n th silhoutt-wr wrp (Chptr ) whih wrp th ntir img. uprpixls orrspon to wll-in rgions o homognous olor ontnt, n thus giv goo rsults with our lol shp-prsrving wrp. Rendering nring is hiv with  thr-pss lning lgorithm.  irst slt our input mrs losst to th novl mr, n wrp ths imgs to th trgt viw.  our wrp imgs r thn ln, with wights spii y mr orinttion ut lso th rliility o pth inormtion in h wrp suprpixl. Finlly, w ill hols with oisson synthsis érz et al., .

	Storage ur mtho rquirs th storg o	-	pth smpls pr img. In ontrst, til
	proxis n  quit lrg or omplx sns. (s l . ). is is n importnt onsirtion or ity-
	sl pplitions.
	. lightr thn  vrity o prvious pprohs inluing Eismnn et al., Limitations Chapter wrp (Chptr ).	 n silhoutt-wr
	 most importnt limittion o th pproh is tht it rquirs mnully mrk silhoutts. ur
	xprimnts with sgmnttion pprohs tin n Hrt,  (s Appnix A) show tht ths pprohs r not irtly pplil prtly us thy r ; H n uill, ; Arlz et al., Depth Synthesis and Local Warps for . Overview
	nvr ths pprohs only provi highly irrgulr ontours.  urt n prtly us w n silhoutt polylins to insrt into th wrp msh whil Preprocessing ur input is  smll st o -imgs tkn rom multipl viwpoints.  prpro-Plausible Image-based Navigation ss th input t using off th shl omputr vision pprohs.  irst lirt th mrs using
	nvly et al.,	 n ronstrut th sn using multi-viw stro Furukw n on,	.
	Dtset Cstl- trt-wr wrp.  improv th rnring lgorithm o silhoutt-wr wrp to urthr ru ghosting Input imges roxy size ur pproh . B . B . B . B r-. B . B Aqurium-. B rtits.  min ontriutions r irstly,  pth synthsis lgorithm whih provis pth smpls . B llowhous-. B in poorly ronstrut rgions, n sonly,  lol shp-prsrving wrp n rnring lgorithm s st mths r gnrlly not immit sptil nighors. us, our pth synthsis is pl . B tht uss th synthsiz pth n ovrsgmnttion to gnrt plusil novl viws. o prorming non-lol intrpoltion tht prsrvs pth isontinuitis provi y th suprpixl
	rprsnttion.

muh improv novl viws rom  r viwpoint mr pth vn or poorly ronstrut tsts.

Performance  tst our mtho on n Intl on ( . Ghz) running inows with n  uro G. tting up th wrp msh n toring th linr systm or h input img with  tks -sons. At run tim, wrping input imgs on prlll ors tks -ms.  ovrll rm rt is -F or × siz rnr trgts.

le . : torg or th proxy us y D mol s pprohs Buhlr et al.,  ompr to th storg rquirmnts o our pproh. roxy sizs r mshs (vrtis/s) without normls/olors/txtur oorints in ACII .obj ormt.

 wrp th ull img s  linr systm whih is rl tim ut th systm is rthr ig n slow to toriz t th strt o th pplition. Also, th onorml Dluny tringultion n  numrilly unstl t th juntion o multipl silhoutts.  glol wrp lso rsults in istortions whn th novl mr is mov signiintly wy rom th input mrs (s Figur . ), rstriting th r viwpoint nvigtion zon.  tringultion is lso iffiult to stup or vry thin orgroun ojts lik rilings. Lstly, th glol wrp ssums tht th omplt sn is in ront o th viwing position us projtion o ny pth smpl in  viwpoint tht is in ront o itsl givs unpritl rsults, using th rprojtion nrgy (Eqution . ) n hn th omplt wrp to xplo.  glol wrp os not llow th systm to ignor suh ss, hn th viwpoint n nvr "wlk into" th sn, this is shown in Figur . .  prsnt  iffrnt pproh rssing ths limittions in th ollowing hptr.

 silhoutt-wr wrp pproh sri in Chptr n gnrt plusil novl viws rom sprs irrgulr pth mps. It lso rsss th ll importnt issus o silhoutts n olusion hn-ling y  omintion o img wrping n mnul silhoutt sltion. Howvr, this mnul stp is th most importnt limittion. In ition, this pproh involvs  ull img wrp whih involvs solving  lrg linr systm tht n  slow n numrilly unstl t tims.  glol wrp lso r-strits th r viwpoint pilitis us it prous xggrt istortions whn th novl mr is mov signiintly wy rom th input mrs. onthlss, th silhoutt-wr wrp provs tht shp-prsrving wrps r vry fftiv t hn-ling sprs pth mps. In this hptr, w uil upon this insight n sign  ompltly utomt lightwight pproh s on img ovrsgmnttion n lol shp-prsrving vritionl wrp.

 irst ovrsgmnt

Ahnt et al.,  th input imgs, rting suprpixls o homognous olor ontnt whih prsrv pth isontinuitis.  thn introu depth synthesis or poorly ron-strut rgions y uiling  grph on th suprpixl sgmnttion. uprpixls llow our lgorithm to oth intiy rgions rquiring pth synthsis n to in pproprit pth xmplrs.  thn pply  lol shp-prsrving wrp on th suprpixls whih rprous ll th vntgs o th silhoutt-It is importnt to not tht th gol o our pth synthsis is not to prou photoonsistnt pth.  gol is to prou plausible pth n us it within th shp-prsrving wrp to prou plusil, though not physilly urt novl viws, vn whn th usr is r rom th input mrs.  hv ppli our pproh  projt th D point lou into input imgs to otin  st o projt pth smpls in h im-g.  thn ovrsgmnt Ahnt et al.,  ll th input imgs rting suprpixls tht lint rgions o homognous olor ontnt n prsrv pth isontinuitis. ur img-s rnring pproh is inpnnt o th hoi o ronstrution n sgmnttion pprohs; w hoos th stt o th rt or ths tsks. ur pproh hs two min stps: pth synthsis n local shp prsrving wrp, ollow y  thr pss rnring lgorithm. Depth synthesis  ky motivtion or this stp is tht vn r using th st ronstrution, thr n  signiint rgions with no pth. ost piwis plnr stro inh et al.,  n img-s rnring lgorithms Gosl et al.,  ignor suh rgions ompltly. Inst o isring suh rgions, w synthsiz plausible pth suitl or img-s rnring, whih is not nssrily photoonsistnt.  pth synthsis os not ugmnt th D ronstrution us th nw pth smpls r not lwys photoonsistnt. y srvs s pproximtions suitl or plusil img-s rnring . . Depth synthesis within  rgulriz rmwork lik our shp prsrving wrp.  prsnt n xtnsiv st o xmpl sns, ll ontining hllnging rgions whih stt o th rt multi-viw stro ronstruts poorly. ur lgorithm llows plusil nvigtion or suh sns.  lso ompr to th two most rlvnt rnt img-s rnring lgorithms Eismnn et al., ; Gosl et al.,  n th silhoutt-wr wrp (Chptr ). ur pproh iminishs mny o th rtits o ths mthos n provis vry onvining nvigtion xprins.

  Among mny similrity mtris or msuring th ffinity o irrgulr img rgions, Grunmnn et al.   hv sussully us χ 2 istn twn L histogrms o suprpixls in orr to msur olor similrity. thr mt-ris lik sum o squr iffrns (D) r lss suitl or irrgulr shps n sizs o suprpixls. suring vrg olor o  suprpixl prorm wors thn L histogrm istn. ror,  onvrt th img into L sp n rt sprt histogrms or h suprpixl with ins in h o L, A n B xs.  ontnt th histogrms to giv  D sriptor ℋ [S i ] or h supr-pixl S i ∈ .  omput th nrst nighors o h trgt suprpixl rom ll suprpixls lry ontining pth smpls using th histogrm sriptors sp with χ 2 istn mtri. is givs  st o "most similr" suprpixls [S i ].  kp th most similr suprpixls, shown in yllow in nighoring suprpixls n long to vry iffrnt ojts or r off rgions o th sm ojt in rih urn sns. is n our us o txtur-lss rhittur, stohsti txtur (.g., trs, hgs) or txtur rptition (.g., winows) s shown in Figur . ().  rin [S i ] y slting th sptilly losst suprpixls. Howvr, th irrgulr n highly non-onvx shps o suprpixls mk Eulin istn twn suprpixls vry miguous. orovr, th siz o th sptil nighorhoo is lso miguous us o th vrying sizs o suprpixls.  rsolv th ov miguity using  grph trvrsl lgorithm.  rt  D suprpixl grph y ing gs twn ny two suprpixls whih shr  ommon ounry (s Figur . )().  ssign th g lngth twn two suprpixls s th hng in olor.  us th χ 2 istn -twn L histogrms to msur hng in olor ontnt. us, two suprpixls with vry similr olor ontnt will hv  short g, s nnott in yn or two suprpixls on th wll in Figur . (). im-ilrly, ny two suprpixls on iffrnt sn ojts r likly to hv iffrnt olor ontnt n thus  high g lngth, s nnott in r in Figur . ().  omput th shortst pth twn target superpixel S T i n h source superpixel S j ∈ [S T i ], whih nots th pth twn th two suprpixls involving th lst hng in olor long th pth. is pth is omput y minimizing th pth ost C ovr ll possil pths rom S T i to S j .

	. .	Shortest walk algorithm

Figur . () or th trgt suprpixl shown in r.  ssum tht ny signiint ojt woul  roun  o img r, quivlnt to -suprpixls.  xprimnt sussully withmost similr suprpixls; highr numrs nlssly inrs omputtion. s

  Conclusions mks our pproh suitl or intrtiv wlkthrough pplitions suh s h trk virtul rlity systm, n rly prototyp o whih is lso monstrt in Chptr .

	Chapter
	Evaluation of Image-based Rendering using
	Perceptual Studies
	suh systms.
	rptul stuis r muh mor usul or vluting plusil rsults. Evn though img-s
	rnring rsrh hs vn y lps n ouns ovr th lst , prptul nlysis hs lgg
	hin. syhophysil xprimnt sign is non-trivil in this s: th fft o iffrnt vrils on
	th inl rsult s wll s thir rltiv importn r unlr. orovr, thr xists  wi rry o
	 hv prsnt  r viwpoint img-s rnring lgorithm sign or urn nvironmnts, pt mthoologis or onuting usr stuis; slting th pproprit on is irly hr.
	pl o prouing high qulity rsults in th sn o urt D ronstrution.  ompnst In this hptr, w ttmpt to nlyz rtits using prptul stuis.  stuy th most uiquitous
	or th lk o urt D ronstrution or hllnging ss y synthsizing plusil pth, whih rtits in isolt sttings or two ss: prsptiv istortions, us whn imgs ptur rom

is not nssrily photoonsistnt. ur shp-prsrving wrp n rnring piplin us th synth-siz pth to prou high qulity novl viws.  ompr our rsults to our rnt img-s rnring mthos n monstrt tht our pproh xtns vry wll to r viwpoint nvigtion. is untittiv nlysis o img-s rnring rsults is vry iffiult us o th shr numr o vrils involv: sn omplxity, qulity o D gomtry, viwpoints, olor/txtur ontnt o sn ojts, siz o sn, ptur nsity, ptur stup t. rvious work us leave-one-out tsts Fitzgion et al.,  whr th img sttistis o  simult novl viw r ompr to  rl photogrph rom th sm viwpoint. uh quntittiv tsts r rlvnt in vry smll slin viw intrpoltion itnik et al., ; Fitzgion et al., ; hjn et al.,  whr th pproh is xpt to prou physilly orrt rsults. Howvr, most morn img-s rnring systms, lik ours, trgt plausible or good looking rsults us th input t is r too sprs to simult physilly orrt rsults. orovr, iffrnt kins o rtits r osrv simultnously in img-s rnring rsults. untittiv vlution suh s lv-on-out tsts o not giv ny insight into th rltiv svrity o iffrnt typs o rtits; this knowlg hs  strong ring on th sign o on viwpoint r projt on to D gomtry rom nothr viwpoint, n ghosting rtits us y lning pixls rom multipl photogrphs to synthsiz  novl viw. o hiv mningul rsults in suh prptul stuis, w n to simpliy th s unr o-

  zons o nvigtion or i th optiml ptur strtgy tht n  xpt to giv  rquir lvl o priv qulity.In this stion, w vis xprimnts tht invstigt th prption o prsptiv istortions.  uil th prmis o our xprimnts upon wll-stlish vision sin litrtur whih xplins th prption o piturs. ision sin hs long stui th prption o pintings or photogrphs tht r ptur or pint rom  rtin viwpoint n viw rom iffrnt viwpoints, typilly in n xhiition or gllry. A ky insight in this work is tht this is vry similr to th s o img-s rnring whr photogrphs ptur rom on viwpoint r rprojt into nothr viwpoint. As xplin in th ollowing prgrphs, vision sin hypothss r not irtly pplil in our on-txt spit th strong intuitiv nlogy.  thror xtn th hypothss rom pitur prption n sign two psyhophysil xprimnts using this thory.  prorm sttistil nlysis o xpr-imntl t to vlop  quntittiv pritiv mol or prsptiv istortions n its pplitions in th ontxt o strt-lvl img-s rnring.Goals ur irst trgt is to stuy how wll our xtn rtinl hypothsis n prit priv ngls in img-s rnring snrios. only, w wnt to quntiy th troff twn th rtinl n sn hypothss n xprss th rltionship s  untion o ptur n/or simultion prmtrs.Lstly, w wnt to vlop n nlyti mol tht prits th lvl o prsptiv istortions givn  st o ptur n simultion prmtrs. o this n, w sign n prorm prptul xprimnts to msur priv ngls n ompr thm to th xtn rtinl hypothss.  thn it xprimn-tl t to n nlyti untion tht prits th lvl o prsptiv istortions n vriy this mol y mns o  vlition stuy. Finlly, w monstrt prtil pplitions o our nlysis, whih togthr with th vlition stuy, is th min ontriution o this thsis in this stuy.

						z
					1 s θ
		θ	2	θ e 1		θ s 2
					Simulation camera 1	Simulation camera 2
		Capture camera			Capture camera	x
	pnst or inorrt viwing position, so th prptul outom is muh losr to th originl D
	Img-s rnring systms rprojt photogrphs ptur rom on viwpoint into  novl viw-point.  photogrph pturs th prsptiv o th sn only rom th originl viwpoint; rprojt-sn thn itt y th istort rtinl img.  rtinl hypothsis, on th othr hn, stts tht viwrs o not ompnst or inorrt position; th prptul outom is itt y th istort ζ rt = f ( θ s , θ e) ( . )
	ing it into  novl viwpoint prous prsptiv rrors pning upon th D gomtry, ptur n rtinl img. hn viwrs r l or right o th ntr o projtion n viw th pitur n its rm  rivtion o th untion f is yon th sop o th thsis, pls rr to ngorp et al.  
	viwing prmtrs. rsptiv istortions r lwys prsnt in ll img-s rnring systms. with oth ys, thy ompnst or thir inorrt viwing position, initing tht sn hypothsis or tils. Entriity ngl θ e with rspt to  prtiulr ornr on th ç is th ngl twn
	s rtits r somtims nign n rly notil, whil lswhr thy n  ojtionl. might  ominnt osinski et al., ; ishwnth et al., . In othr situtions, whn th slnt th ç norml n th lin joining th ptur mr with th ornr (s Figur . ). imultion
	o voi showing prsptiv istortions, img-s rnring pplitions tn to rstrit viwing o  pitur ojt is nrly prpniulr to th pitur sur or whn th viwrs r too los to ngl θ s is in s th ngl twn th ç norml n lin joining th simultion mr n
	or too r rom th pitur, thy o not ompnst or th inu img istortions n thror
	priv D strutur inorrtly Ams,	; Lumsn,	; oorović,	; Bnks et al.,	;
	Coopr et al.,	.

positions los to ptur viwpoints. Howvr, this is on in  rthr -ho mnnr us o th lk o prinipl unrstning o prsptiv rrors. A quntittiv mol tht orrlts th prption o prsptiv istortions with ptur/isply prmtrs n llow pplitions to slt pt-l  til isussion o vision sin thory, xtnsion o vision sin hypothss n sttistil nlysis o xprimntl t is yon th sop o this thsis; pls rr to ngorp et al.   or tils.  introu th xprimntl stup n ous on th rsults, thir vlition n pplitions in img-s rnring. Perception of pictures hn  pitur is viw rom th sm position s th ntr o projtion o th virtul mr tht "photogrph" th D sn, th img orm on th rtin o th viwr, known s rtinl img, is orrt. I th position o th viwr is iffrnt rom th ntr o projtion, th rtinl img is istort: its prsptiv us suh s vnishing points r iffrnt rom thos o th originl photogrph.  prption o this rtinl img is xplin y two ompting hypothss in vision sin litrtur: th scene hypothesis n th retinal hypothesis.  sn hypothsis stts tht viwrs om-e Figure . : Entriity n simultion ngls.  ngls orrspon to th ornr mrk with orrsponing olor in th photogrphs. il n right: th imgs shown s simult viw r rnr y rprojting th input img in th simult viwpoint. Extended retinal hypothesis  originl rtinl hypothsis only pplis to th prption o pi-turs. In th ontxt o img-s rnring,  ç is irst photogrph rom  rtin position. is img is thn projt onto simpl D proxy gomtry, whih in turn is viw rom  novl m-r position n thn projt on to  D isply vi.  rtinl hypothsis os not xplin suh "photogrph o photogrph" ss. Howvr, it is sil to riv th priv ngl using th sm prinipls s th rtinl hypothsis: projtiv gomtry n vnishing points.  xtn th rtinl hy-pothsis to th ontxt o img-s rnring suh tht it rlts th priv ngl ζ rt , simultion ngl θ s n ntriity ngl θ e . th ornr. Clrly, simultion ngl inrss s th simultion mr is mov urthr wy rom th ptur mr position.  ssum tht ll photogrphs r ptur y mrs tht r ronto-prlll to th ç, thror, th ç norml is oinint with th ptur mr orinttion.

  th pth.  prtiipnts wr sk to "rt th visul qulity o th pproximtion with rspt to th rrn" using  ontinuous slir (s Figur . ). is provis  irt msur o qulity. Eh o th × stimuli is rpt tims in rnom orr, in sprt loks or oth sns Avrg visul qulity rtings or Exprimnt , rnging rom th worst qulity ( , lk) to th st ( , whit). Highr vlus mns th squn look ttr, i.., wr rtits. op row orrspons to popping rtits whil th othr rows orrspon to ghosting rtits.

				Corner scene	Town Hall scene		Average
	images blended	1 2 3	39 29 22	49 46 30	60 67 61	52 41 23	41 68 60	39 79 79	45 35 23	45 57 45	49 73 70
			lo	me	hi	lo	me	hi	lo	me	hi
	Experimental procedure  prmtrs w vry or th pproximt rnrings r ( ) ovrg, n ( ) numr o imgs ln or ny givn pixl. For ovrg, w us low (lo), mium (me) n high (hi) vlus orrsponing to pproximtly , n imgs ovring ny point o th sn gomtry.  n , n (own Hll) or (Cornr) input imgs to hiv ths vlus o ovrg (s Figur . ). For th numr o imgs ln pr pixl, w us vlus o , n , s ommonly us or this lss o img-s rnring thniqus Buhlr et al., ; Eismnn et al., .  prtiipnts wr prsnt with  pir o vios: n img-s rnring n th orrspon-ing vio rrn.  vios ply in  loop o pproximtly sons with th mr moving to coverage coverage n ro long coverage Figure . :

Chpter . Evlution of Imge-sed endering using ereptul tudies th

  Avrg visul qulity rtings or Exprimnt using rtiiil n rl stimuli, rnging rom th worst qulity ( , lk) to th st ( , whit).  l igur ll "L" shows th rsults o unstrutur lumigrph n th right igur ll "CF" shows ross-ing rsults.  top row o unstrutur lumigrph rsult orrspons to popping rtits (using img pr pixl), n th ottom row orrspons to ghosting rtits (lning imgs pr pixl). muh th rtits othr thm" y justing  ontinuous slir s in th prvious xprimnt. r-tiipnts wr irst prsnt with th intii rrn stimulus in th ntr o th srn, with on itionl stimulus orrsponing to on o lning, popping or ross-ing. s wr prsnt in rnomiz orr, to th l, right n low th rrn (s Figur . ). Blning n popping in  givn tril us th sm totl numr o imgs.  prtiipnt rt h stimulus with rspt to th rrn. Ar th thr stimuli hv n rt, th prtiipnt ws prsnt with ll thr stimuli, Validation of unstructured lumigraph results  sign o th xprimnt llows us to rvisit th qustion o whthr popping or lning rtits r prrl in unstrutur lumigrph. In ontrst to th Cornr sn in prvious xprimnt, slow popping is prrr (signiintly ngtiv slop o -. pp pr ouling o ovrg). s rsults lso onirm th rsults rom th prvious xprimnt tht priv qulity o ghosting rtits in unstrutur lumigrph improvs with highr ovrg (s Figur . ). rsults or rtiiil sns show  highr rltiv qulity o ross-ing ompr to unstru-tur lumigrph.  liv this is most likly us y th lk o til n omplxity in th rtiiil ç n y th high ury o its gomtri proxy n mr positions. In rl sns, mislign-mnt twn th two input imgs t th r ns o  ç s us y ross-ing r typilly lrgr n mor notil thn twn onsutiv input imgs s us or unstrutur lumigrph. r-or w hypothsiz tht th rsults with rl stimuli r mor suitl s  sis or guilins whih will gnrliz to othr rl worl sns. now summriz th rsults o th xprimnts n prsnt guilins or ptur n isply.ConclusionIt is lr tht hvy ghosting rtits r lwys priv s unptl vn i thy l to smooth tmporl trnsitions. srvrs sm to prr risp imgs vn i thr r prllx rrors or tmporl jumps. is unmntl guilin is th sis o our lning strtgy in oth th img-s rnring pprohs o Chptrs n . Both th lning strtgis us t most imgs t h pixl n vn thn vor on nit hvily, rsulting in miniml ghosting. Although our rsults r vry rlvnt to urrnt ommril systms suh s Googl trtviw, thy nnot vlut th qulity o stt o th rt img-s rnring pprohs suh s thos prsnt in Chptrs n . A ull lg prptul vlution rmwork must xtn to sophistit img-s rnring lgorithms n tk into ount th omplxity o th sn n qulity o ronstrut D gomtry. syhophysil xprimnt sign or suh omplx srrios n  irly non-trivil. For suh ss, it my  simplr to stuy impliit visul pross y msuring rin rspons irtly ust et al., . uh stuis n  o muh utility, provi th rltionship twn onsious visul ognitiv prosss n impliit prosss is wll unrstoo.  nlyz iffrnt kins o istortions n giv guilins or signing systms tht mximiz pr-iv qulity. ur nlysis o lning n ssoit rtits rvls tht pplitions shoul voi lning xssivly.  vlop this guilin in  simpl img-s rnring systm, ut this is  powrul rsult whih pplis to mor sophistit systms suh s thos in Chptrs n . r-sptiv istortions r th most importnt rtits us thy will lwys  prsnt. Howvr, our stuis show tht th humn visul systm is quit tolrnt towrs suh istortions whih lvs  lot o sop or img-s rnring pprohs to xprimnt with lgorithms tht n work with sprs pth mps vn though thy inur istortions to som xtnt.  xprimnt with shp prsrving wrps in this spirit; thr is promis or vn ttr rsults in this irtion o rsrh.

	ULR singl img to synthsiz h pixl unr low ovrg i.. slow popping, or () lning imgs ULR
	coverage 43 38 me () Artiiil stimuli 38 46 hi crossfading length 67 50 37 10% 40% 100% CF pr-pixl unr ns ovrg. images blended 60 17 lo 1 2 Ghosting artifacts versus perspective distortions  t ollt rom ross-ing xprimnt coverage images 76 64 60 1 CF blended 32 65 78 lo me hi 2 crossfading length 57 42 19 10% 40% 100% () l stimuli Figure . : slirs n th rrn, n my just th rltiv rtings (s vio ).  rpt th sm xpri-mnt or th rl sns ut without th rrn stimulus whih mounts to no-rrn omprtiv stuy twn ross-ing n lning. Results Figur . () summrizs th visul qulity or th ross-ing xprimnt with rtiiil stimuli, vrg ovr prtiipnts. hort ross-ing is givn th highst qulity rting ovrll, whil longr ross-ing riv vry low rtings, monstrting  prrn or shortr ross-ing (sig-niintly ngtiv slop o -. pp pr  inrs in ross-ing lngth). hort ross-ing rsults in strongr prllx rtits towrs th mil o th pth, ut lss prolong ghosting rtits ur-ing th trnsition. is suggsts tht th prllx istortions r lss ojtionl thn th lning rtits in ths stimuli. Figur . () summrizs th visul qulity or th ross-ing xprimnt with th rl sn, vrg ovr prtiipnts. is onirms th trns within h thniqu. r is gin  lr pr-rn or shortr ross-ing (signiintly ngtiv slop o -. pp pr  inrs in ross-ing . . Guidelines for current image-based rendering systems Unstructured lumigraph guidelines ghosting versus popping  rsults rom th unstrutur lumigrph xprimnt (s Figur . ) show  systmti rnking o popping n ghosting. hn ovrg is low, popping is lrly prrl to ghosting rtits. is sms to  th s us synthsiz viws r s risp s th input imgs n th rtits r visil only t trnsitions, pr-snting  plusil img or longr urtion. Dns pturs l to mor rqunt popping whih is priv s wors thn slow popping with sprsr pturs. Ghosting uss slint img ontnt suh s txt, rhitturl turs t. to om illgil or unrognizl, whih is priv s mor is-turing ompr to sun trnsitions.  st ovrll rsult is hiv whn ovrg is high n imgs r ln pr pixl. is givs th st troff; howvr it might  imprtil us ns pturs n rsult in prohiitiv quisition n storg osts. Cross fading guidelines ur xprimnt inits n intrsting wy to improv img-s nv-igtion pplitions s on ross-ing, suh s Googl trtviw whih urrntly ppr to us  thniqu kin to long ross-ing. ur rsults show tht y swithing to shortr ross-ing priv qulity woul  nhn, spit prsptiv istortions. with oth rtiiil n rl stimuli suggst tht ross-ing o short trnsition is prrr. In th short ross- onition, prsptiv istortions om ut towrs th mil o th pth; spit this, th onition is rnk s highst qulity mong ross-ing stimuli. is inirtly inits tht pr-sptiv istortions r mor tolrl thn ghosting rtits us y long trnsition ross-s.  shp-prsrving wrp in Chptrs n xploits this osrvtion.  wrp prous prsptiv rrors, howvr, th ovrll priv qulity is high us oth th pprohs minimiz th pr-ptully mor importnt ghosting rtits.  shp-prsrving wrp us in oth th img-s rnring pprohs is gin motivt y th t tht prsptiv istortions r lss notil thn ghosting rtits.  shp-prsrving wrp srvs s  D pproximtion or th tru ffin trnsormtion n thus ls to prsptiv rrors, ut thy r hrly privl, vn mor so or th lol wrps in Chptr us th prsptiv rror is loliz to muh smllr rgions. . Discussion  min limittion o oth th prptul stuis is tht thy nlyz  simpl orm o img-s rnring rstrit to xis-lign gomtry typil o çs. In tion . , w nlyz th troff twn ghosting n popping rtits n giv guilins or rsolving it, ut our guilins o not -ount or th omplxity o th sn. ur nlysis is lso rstrit to unstrutur lumigrph Buhlr et al.,  n ross-ing inh et al.,  whil morn img-s rnring pprohs r mor sophistit. Agin, our nlysis o prsptiv istortions in tion . is limit to vry simpl Conclusion Chapter Virtual Reality using Image-based Rendering irtul rlity () uss omputr grphis to immrs usrs in virtul nvironmnts.  pplitions prou li siz rnrings o virtul nvironmnts roun th usr rthr thn on  omputr srn s . r is littl rsrh on th or rnring lgorithm or virtul nvironmnts. is is rthr surprising -us  rquirs lrg n possily nimt virtul nvironmnts  rnr t intrtiv rts in stro. ror, urrnt  pplition us vry trivil hong shing or txtur mpping. It is hr to us vn rough pproximtions to glol illumintion, .g. srn sp mint olusion, simply us th rm rt rops signiintly on lrg stro isplys.  us o tritionl grphis poss two min prolms or  systms. Firstly, lrg virtul sns hv to  mol mnully whih n  tim onsuming n xpnsiv. only, lrg multi-srn stro rnring lns  powrul sns o immrsion ut th rnring qulity is on poor.  prsnt th irst img-s rnr (IB) immrsiv systm tht llvits th ov prolms. IB systms us photogrphs: this mks pturing lrg nvironmnts irly trivil ompr to mnul D mol-ing. or on thn not,  pplitions try to mol rl sns, IB n   vry onvnint tool or sn gomtry. . suh ss. At th sm tim, IB givs photorlisti rsults whih n potntilly inrs th sns lngth). r is  slight prrn or slow popping in th s o unstrutur lumigrph (signiintly Cross fading versus unstructured lumigraph  omprison twn ross-ing n unstru-ngtiv slop o -. pp pr pproximt ouling o ovrg) n or lning with nsr ovrg. o immrsion in  pplitions.
	tur lumigrph on rl sns (s Figur . ()) show tht th lttr givs ttr qulity y () using

http://www.youtube.com/watch?v=akaWUe0mum is ommon in  typil omputr grphis pplition. is is gnrlly hiv y spiliz hrwr suh s multi-srn projtion systms, h mount isplys t. is is n xiting r o rsrh with  lot o potntil or gming, trining simultions n hlth pplitions suh s ognitiv thrpy.

 ulk o rsrh in  ouss on hrwr -multi-srn projtion systms, stro mhnism, trking n hptis. ost o th sowr rlt rsrh ouss on humn-omputr intrtion whr th trgt is to vlop mor intuitiv mhnisms or intrting with th virtul nviron-mnt using th sm or nhn hrwr .g. virtul nvigtion intrs Cirio et al., hil  lrg numr o IB pprohs r vill or xprimnttion in  systms, it is sy to s tht pprohs rstrit to viw intrpoltion itnik et al., ; Gosl et al.,  n non-

  isplying th virtul nvironmnt.  prolms with using ll srns is isuss ltr in tion . .  siz o th srn is . m× . m with  isply rsolution o × pixls.  projtors us pssiv Init stro glsss , whih r trk using th A trking systm . n solut oorint systm ix to th immrsiv sp s shown in Figur . .  thror stimt  × rottion mtrix R pr n trnsltion vtor t pr twn th ronstrut sn n immrsiv sp.  nt trnsormtion twn th two oorint systms is givn y In orr to stimt th rottion n trnsltion, w slt ny input mr n lign its ntr o projtion n rottion mtrix with  ix rrn position n rottion mtrix in th immrsiv sp rsptivly. Lt th slt input mr's rottion mtrix  R i n th rrn rottion mtrix in th immrsiv sp  R r .  projtion o ny D point in th rottion mtrix o th input mr must  th sm s th projtion o th sm point r pplying th rottion R r in th rrn rottion mtrix. is invrin givs th ollowing qution:

			T pr = (	R pr t pr	)	( . )
	.	Capture and dataset preparation
	 ptur th sn using photogrphs s shown in Figur . .  igur shows th sn ptur
	using lss thn	photogrphs. is xmpliis th utility o IB in suh systms; lrg sns suh
	s th on shown n rquir  lot o mn hours or D moling. Cpturing th whol sn using
	photogrphs rquir just minuts in this s.
	Ar ptur, w prpross th photogrphs similr to Chptr y running D ronstrution
	nvly et al.,	; Furukw n on,	, ovrsgmnttion Ahnt et al.,	 n pth
	synthsis in poorly ronstrut rgions (s tion . ).
	. .	Registration of D scene and immersive space
	 sl o th D sn n pos o input mrs r trmin in n ritrry oorint systm t
	y D ronstrution s shown in Figur . .  irst omput th sl tor twn th tul sn
	n its ronstrut vrsion s th rtio o th tul istn twn two known points o th sn
	n th istn twn thir D ronstrut vrsions. is ngts ny sl vritions twn rl
	worl imnsions n ronstrut imnsions.

 D sn hs to  isply in th immrsiv sp whr th isply mr mtris r http://infitec.net/index.php/home/glasses/infitec-premium-glasses http://www.ar-tracking.com onstrut using

  th immrsiv sp stup, th novl mr position c H is provi y  h trking vi in rl tim.  ptur D sn is lry rgistr with th immrsiv sp suh tht on o th input mrs is lign with  known rrn point in th immrsiv sp. H with th our ornrs o th srn Cruz-ir et al., . n w pr-slt th input imgs, w wrp thm using th ovrll projtion mtrix givn y P f,H ⋅ M f,H .  ln th wrp imgs s sri in tion . .  ov pproh smlssly hnls h movmnt within th immrsiv sp. o llow long rng nvigtion, w us th wand vi whih hs  joystik n  orinttion snsor.  hrwr rturns th urrnt orinttion o th wn n movmnt o th joystik.  intrprt th joystik s positiv or ngtiv trnsltion in th irtion givn y th orinttion o th wn. is givs n ovrll trnsormtion T or th position o th usr in th virtul worl whih n  hiv y trnsorming th h position y T. Howvr, rll tht th h position is upt synhronously y th h trkr. ror, inst o trnsorming th h position y T, w trnsorm th sn n input mrs y T -1 with quivlnt fft. o trnsorm th sn, w pply T -1 on h D pth smpl us or IB.  trnsorm th input mrs y trnsorming th ntr o projtion s wll s th molviw or xtrinsi mtrix.  xplin this rivtion in Appnix C. hil our systm provis vry rlisti imgs or r nvigtion, lik ny img-s mtho it is rstrit to rprsnting ontnt whih tully xists in som o th input imgs. hn th usr lvs this rgion, visul rtits ppr. o voi this, w limit nvigtion to th zon whr rtits r vry smll. is still lvs suffiint room or r nvigtion y tul movmnt within th immrsiv or long rng trnsltion using th wn.

	. . For stro rnring, w rnr two rms into th pnGL l n right uffrs, h omput Stereo rendering Navigation . . Rendering synthetic objects with IBR Cpturing n moling  rl sn n provi  vry rlisti D nvironmnt. Howvr, it my not ontin ll th sir sn lmnts rquir or   pplition. is is spilly tru or trining or simultion pplitions whr it my  ritil to hv itionl ojts or intrtion (s Fig-ur . (right) or xmpl). uh ojts hv to  mol or ptur sprtly n  to th virtul nvironmnt. o llow this, w moiy th IB piplin o Chptr to  othr ojts. ll tht w stor th min pth o h suprpixl s mtt whil wrping it to th novl viw (s tion . . ). During th son pss or lning, w rprojt this min pth or h nit suprpixl to th novl viw using th ollowing: d ,novl = C ,novl ⋅ C -1 input ⋅ d ( . ) whr d is th min pth o th suprpixl, C input n C novl r th projtion mtris o th input n novl mr rsptivly. hil lning th nit suprpixls, w writ th rprojt pth inpnntly using th sm img-s rnring piplin. For h h position c . . o th highst wigh nit s th inl pth o th pixl.

 virtul mr us to isply th ront srn hs orinttion (0, 1, 0) n up vtor (0, 0, 1) s shown in Figur . . s vtors rmin th sm irrsptiv o th position n orinttion o th h. us, in pnGL trminology, th molviw mtrix is givn y:

M f,H = gluLookAt (c H , c H + (0, 1, 0), (0, 0, 1)) ( . )

whr th susript f nots th ront srn.  slt input imgs whos molviw mtris r most similr to tht o th novl mr. us, to synthsiz th inl img, w slt th input imgs whos ntr o projtion n orinttion r losst to c H n (0, 1, 0) rsptivly, in th oorint systm o th immrsiv sp.  omput th prsptiv mtrix P ,H using th stnr pproh o joining th h position c H , w rt two h positions c H,l n c H,r sprt y introulr istn o -m suh tht th lin joining c H,l n c H,r is prlll to th srn (s Figur . ). At h rm, w omput n IB rsult orrsponing to h o c H,l n c H,r n isply thm using qu-uffr stro.

Hving rnr th ull sn using th img-s piplin, w thn rnr synthti ojts.

s ojts r typilly rt in D moling sowr suh s Autosk y or Blnr t.  thror us hong shing with txtur mpping to rnr ths ojts.  rnr thm in th sm . . esults

  Experiment  prtiipnts wr prsnt with iffrnt stimuli or minuts n sk to gnrt s mny mmoris s possil rlt to th nvironmnt, using short sntns.  vrl rponss wr ror n nlyz y  sph linguist who lssii thm s onious or vgu rolltions. h s is xpt to init th utility o th our sttings or minisn rpy.  prorm th xprimnt on lrly prtiipnts n not ptints sin this projt is still in its rly stgs n th immrsiv sp is not n uthoriz linil lortory. In t, ing  novl thnology, it is mor likly to ngg th prtiipnts thn tritionl pprohs. n o th grt hllngs or  is whthr iffrnt thnologis mintin th usr's motivtion whn onront thm with  rptitiv sris o trining hllngs.  intrtivity n rlism sm to improv motivtion n nggmnt. s rsults show tht img-s thniqus offr grt promis or , n or  in gnrl. ur systm hs numrous vntgs ovr tritionl D ssts us in .  t tht only  w sul photogrphs r rquir to rt  sn tht n  us or  is n vntg with vry signiint onsquns.  lvl o rlism otin y th imgry, spit som rsiul rtits, is t lst s goo s tht prou t grt ost with mnul moling. is is n xprimntl stup with mny thnil hllngs, th most importnt ing  multi-srn IB solution (s tion . ).

	 only ount onious rolltions or our purpos.  iffrnt stimuli inlu () img-s 
	rnring o  known ity lnmrk, () img-s  rnring o unknown ity lnmrk, () stti
	photogrph o  milir lnmrk n () gry img whih rprsnts no visul stimulus.  numr
	o rolltions in
	.	Application Reminiscence Therapy
	 monstrt th  prototyp y using it in  minisn rpy () xprimnt. In minis-
	n rpy, ptints r prsnt with milir nvironmnts, .g. thir nighorhoos, prominnt
	lnmrks o thir ity t. ritionl  omplishs this y mns o photogrphs.  n  us
	or th sm purpos y moling th nvironmnts n llowing th ptint to intrt with th n-
	vironmnts virtully. is is s on rnt work tht show th utility o  or mmory trtmnts
	Brooks n os,	; Gonnu et al.,	. Howvr, rting  rlisti D mols o  ptint's -
	milir nvironmnts using tritionl mnul moling is r too xpnsiv oth in tim n rsours.
	 hypothsiz tht th rlisti rnrings using IB r qully or mor powrul thn photogrphs
	or synthti sns whil lso mking it muh sir to ptur suh nvironmnts.  vlop x-
	primnts whr milir n unmilir lnmrks o th ity r prsnt to lrly prtiipnts,
	thir mmory rspons msur y mns o  qustionnir n ompr to th s whr still
	photogrphs r us s stimulus. om photogrphs o th xprimnts n  sn in Figur . .

is xprimnt inluing th sign, nlysis n inrns r yon th sop o th thsis.  giv  ri ovrviw o th xprimnt n rsults to monstrt th utility o our  systm. ls rr to Chpouli et al.,

 or mor tils.

. . Applition: eminisene erpy

Results  most importnt rsult o th stuy is tht th numr o rolltions ws highst or img-s  rnrings o  milir lnmrk ollow y stti photogrph o milir lnmrk.

is inits tht rlisti immrsiv visuliztion o D nvironmnts surpsss stti photogrphs or triggr onsious rolltions o utoiogrphil mmory, whih is th min purpos o . ur systm n  sily pt to nvironmnts milir to th ptint; this hols prtiulr promis rom  linil prsptiv, sin it mks   vil linil prour.  rsponss to our qustionnirs init th ility o IB to onvy  sns o "ing thr".  lso got onlusiv vin tht im-mrsion in milir lnmrks o th ity gnrt mor mmoris, whih inits tht th lvl o rlism in our systm in mks  virtul nvironmnt mor rognizl.

Bsis, our stuis onirm th ptility o sophistit  thnology y lrly prtiipnts.

 rsponss to th qustionnirs init tht th thnologil stup is wll tolrt y th prti-ipnts.

  trms o spii ontriutions, w liv tht th introution o isontinuous vritionl wrp in Chptr rks nw groun. In prtiulr, w hv monstrt how to introu ontnt prsrving isontinuitis in  smooth vritionl wrp. is will hopully hv pplitions in othr img rlt tsks.  pth synthsis in Chptr is th irst pproh whih synthsizs plusil pth in  non-lol mnnr; it is pl o ining plusil pth rom rltivly istnt img rgions whil prvious pprohs hv only tri lol pth propgtion ng et al., ; Gosl et al., .  hv lso vlop  shp-prsrving wrp tht nsurs istortion r img synthsis vn though th guiing onstrints in th orm o pth smpls r somtims pproximt.  wrp ppli inpnntly to h suprpixl prsrvs th struturs within suprpixls, ross suprpixls s wll s ross tim (uring n intrtiv wlkthrough). o glol onstrints r nssry to nsur sptil or tmporl ohrn. imilr pprohs hv n us or or wrping imgs Liu et al., , ; our work is th irst to monstrt ths wrps ppli to lol rgions inst o th ull img, n to omin thm with olusion hnling, prllx ffts n viw synthsis y lning multipl wrp imgs. orovr, our solutions r lso roust or situtions whr input imgs hv to  wrp ovr signiint slins o nrly mtrs.  hv vlut ommon rtits rlvnt to img-s rnring using prptul nlysis. ur stuis rvl tht sptil ghosting rtits r mor ojtionl ompr to spori tmporl isontinuitis, whih inits tht osrvrs prr risp rnrings rthr thn smooth trnsitions.  vlop  quntittiv mol tht orrlts th priv lvl o prsptiv istortions with p-tur n rnring prmtrs, n lso show tht osrvrs r mor tolrnt towrs prsptiv istortions s ompr to ghosting rtits. Lstly, this thsis is th irst rsrh work in img-s rnring to trgt r viwpoint nv-igtion; prvious work hs lrgly ous on viw intrpoltion.  liv tht utur img-s rnring pprohs, in n ttmpt to improv upon th rsults o this thsis, will onsir this n inispnsl gol.  rsults o this thsis r ing us in th E I projt EE or vloping virtul rlity pplitions or minisn rpy (s Chp. ). In this ontxt, th nwly uilt Centres Mémoire de Ressources et de Recherche (mmory ntr) in i hs instll   stup whih will us our systm in  linil stting.  rsults lso inspir th E I projt C-LA whih trgts img-s rnring s  tool or ontnt gnrtion or lightwight gms.  ompnis involv in this projt r prsu tht improv vrsions o our solutions will rsult in signiint svings to thir proution osts.  rnring spt o th Frnh projt A EALI is lso s on our is. Bsis, mjor inustril plyrs hv xprss intrst in vlution o thniqus prsnt in this thsis.  limittions o our img-s rnring pprohs provi nw irtions or rsrh. ur shp prsrving wrp ssums lrgly ronto-prlll pth whih rsults in rtits on surs ptur t grzing ngls. is oul  rsolv y ormulting th wrp so s to ount or lol pth vrition. Computr nimtion n gomtry mnipultion litrtur my provi insights. Currnt pprohs, inluing ours, hv prolms hnling vry thin ojts suh s lmp posts, grtings, rilings t. is is lso th s or non-lmrtin surs suh s rltions, trnsprn-is t., lthough rltions hv n hnl in rstrit sttings inh et al., . igniint lgorithmi hngs woul  rquir to provi roust solutions to ths ss.  us si oisson synthsis to ill hols in th inl rsult, thr is n or mor sophistit lgorithms or this purpos. Inpinting Criminisi et al., , omin with rnt lrtion th-niqus .g., thth Brns et al., , oul provi  sis or suh  solution. ur pth synthsis n lning pprohs n nit rom ojt osgmntion Kowl et al.,  whih simultnously sgmnt ll imgs o  multi-viw tst whil lso uiling  orrspon-n grph o sgmnts ross iffrnt imgs. ur pth synthsis uss  grph s pproh inspir y Gosi mthos in omputr grph-is Criminisi et al., . Altrntivly, F s ormultions itnik n Kng,  might im-prov th rsults. An importnt hllng in pth synthsis is to ismigut twn suprpixls s on visul ontnt.  ompr olor histogrms whih n  prolmti in s o shows or strutur rptitiv txturs suh s hkror pttrn on  ç. or roust rsults n  otin y omputing th intrinsi omposition o th img Boussu et al., ; Lffont et al.,  n ompring th lo. Although rl tim, our pproh is still mor omputtionlly mning thn simpl projtiv txtur mpping o piwis-plnr ronstrutions inh et al., ; Gllup et al.,  whih n  us vry fftivly or simplr sns. Hyri pprohs n  vlop whih us plnr r-onstrution whrvr possil n swith to mor omplx pprohs suh s ours or non-plnr or irrgulr rgions. is might lso llvit som o th limittions o our pproh suh s th prolm with surs ptur t grzing ngls. ost morn img-s rnring systms r rl tim, howvr thy onsum  lrg mount o C mmory or D mols or point lous n G mmory or txturs. s rquirmnts mk it iffiult to port ths pplitions on moil vis. Fortuntly, multi-viw sns ontin  lot o runnt t us th sm visul ontnt is prsnt in  lrg numr o imgs. Dpning upon th mmory ugt, this runny n  xploit to omprss multi-viw sns into txtur tlss. ur prptul stuis o visul rtits (Chptr ) r limit to simpl img-s rnring stups on simpl sns.  immit nxt stp is to gnrliz ths stuis to mor omplx sttings. In our xprimnts, w hv shown n inirt omprison twn prsptiv istortions n ghosting rtits; it woul  intrsting to quntiy this omprison.

	. .	Research impact and deployment
	.	Future work

http://vimeo.com/

http://www.verveconsortium.eu/ http://www.cmrr-nice.fr/ http://cordis.europa.eu/fp /ict/creativity/creativity-projects-fp _en.html https://project.inria.fr/semapolis/

Appendix B Depth Synthesis for Sky Regions

 pth synthsis (s Chptr . ) s plusil pth to rgions whih hv vry w or no pth smpls. In this stion, w sri th spil s o pth synthsis or img whih hv signiint sky rgions, whih ws n or nivrsity n ChplHill tsts in our xprimnts. ur pth synthsis pproh n synthsiz pth vlus on ojts whih hv some though sprs pth smpls. Lrg rgions o sky typilly hv no pth smpls t ll.  intiy suh sky rgions in th img using  grph-ut.  ssum tht th imgs r ptur upright n sky pixls r los to th top orr.  rt  grph with ll th pixls o th img s nos n  gs twn jnt pixls.  ll osts or th grph ut r givn in th ollowing tl. 

Appendix C

Transformation of Camera Matrices to

Immersive Space

In tion . , w trnsorm th ntir D sn to th oorint systm o th immrsiv sp. o this n, w pply  rigi trnsormtion to th whol sn.  trnsormtion o ll D ronstrut points is trivil. In this stion, w sri th trnsormtion o th xtrinsi or molviw mtris o th input mrs to th immrsiv sp.

Assum th ntir sn inluing th input mrs, hs to  trnsorm y th mtrix M whih ompriss  uniorm sl s, rottion R M n trnsltion T M . Any D point x in homognous oori-nts n  trnsorm y pplying M.

Clrly, M is invrtil us th sl, rottion n trnsltion r invrtil.

Consir n input mr with originl prsptiv mtrix (or rustum) F, rottion mtrix R n ntr o projtion v.  mr xtrinsi or molviw mtrix is givn y:

 projtion o ny point x in this mr is givn y

Appendix C. rnsformtion of Cmer Mtries to Immersive pe hil trnsorming th ntir sn, th mr's rottion mtrix n ntr o projtion hng ut th rustum rmins th sm us it is n intrinsi proprty o th mr. Lt th nw mr position n rottion mtrix  v n R rsptivly.  projtion o  sn point x using th originl mr shoul  th sm (up to onstnt tor) s tht o th trnsorm point x using th trnsorm mr.

is givs th ollowing qutions

olving ths two, w gt th rottion mtrix n ntr o projtion o th trnsorm mr.