Marc Arnaudon
email: arnaudon@math.univ-poitiers.fr

Anton Thalmaier
email: anton.thalmaier@uni.lu

Li-Yau Type Gradient Estimates and Harnack Inequalities by Stochastic Analysis

Keywords: 2000 Mathematics Subject Classification. 58J65, 60H30 Heat equation, Ricci curvature, Li-Yau inequality, Harnack inequality, gradient bound, Brownian motion

In this paper we use methods from Stochastic Analysis to establish Li-Yau type estimates for positive solutions of the heat equation. In particular, we want to emphasize that Stochastic Analysis provides natural tools to derive local estimates in the sense that the gradient bound at given point depends only on universal constants and the geometry of the Riemannian manifold locally about this point.

§1. Introduction

The effect of curvature on the behaviour of the heat flow on a Riemannian manifold is a classical problem. Ricci curvature manifests itself most directly in gradient formulas for solutions of heat equation.

Gradient estimates for positive solutions of the heat equation serve as infinitesimal versions of Harnack inequalities: by integrating along curves on the manifold local gradient estimates may be turned into local Harnack type inequalities.

Solutions to the heat equation

(1.1) ∂ ∂t u = 1 2 ∆u
on a Riemannian manifold M are well understood in probabilistic terms. For instance, if u = u(x, t) denotes the minimal solution to (1.1), then a straightforward calculation using Itô's calculus leads to the stochastic representation of u as

(1.2) u(x, t) = E 1 {t<ζ(x)} f (X t (x))
where f = u(•, 0) is the initial condition in (1.1), X . (x) denotes a Brownian motion on M , starting from x at time 0, and ζ(x) its lifetime.

It is a remarkable fact that exact stochastic representation formulas for the derivative of solutions to the heat equation can be given, expressing the gradient ∇u of u in terms of Ricci curvature.

The following typical example for such a Bismut type derivative formula is taken from [START_REF] Thalmaier | Gradient estimates for harmonic functions on regular domains in Riemannian manifolds[END_REF].

(1.3) ∇u(•, T ) x , v = -E u(X τ (x), T -τ ) τ 0 Q s ls , dB s ,
where:

(1) X ≡ X . (x) is a Brownian motion on M , starting at x, and τ = τ (x) ∧ T where

τ (x) = inf{t > 0 : X t (x) ∈ D}
is the first exit time from D; the stochastic integral is taken with respect to the Brownian motion B in T x M , related to X by the Stratonovich equation dB t = // -1 t δX t , where // t : T x M → T Xt M denotes the stochastic parallel transport along X.

(2) The process Q takes values in the group of linear automorphisms of T x M and is defined by the pathwise covariant ordinary differential equation,

dQ t = -1 2 Ric // t (Q t ) dt, Q 0 = id TxM ,
where

Ric // t = // -1 t • Ric ♯ Xt • // t (a linear transformation of T x M ), and Ric ♯ z u, w = Ric z (u, w) for any u, w ∈ T z M , z ∈ M .
(3) Finally, ℓ t may be any adapted finite energy process taking values in

T x M such that ℓ 0 = v, ℓ τ = 0 and τ 0 | lt | 2 dt 1/2 ∈ L 1+ε for some ε > 0.
Formula (1.3) is valid for any solution u of the heat equation and does not require positivity of u.

Remark 1.2. Formula (1.3) is easily adapted to more specific situations, for instance:

(i) Let u = u(x, t) be a solution of the heat equation

(1.1) on D × [0, T ] such that u| t=0 = f and u(•, t)|∂D = f |∂D. Then ∇u(•, T ) x , v = -E f (X τ (x)) τ 0 Q s ls , dB s , τ = τ (x) ∧ T . (ii) Let u = u(x, t) be a solution of the heat equation (1.1) on D × [0, T ] such that u| t=0 = f and u(•, t)|∂D = 0. Then ∇u(•, T ) x , v = -E f (X T (x)) 1 {T <τ (x)} τ (x)∧T 0 Q s ls , dB s .
Such formulas are interesting by several means. For instance, on a complete Riemannian manifold M , starting from the minimal solution to the heat equation

(1.4) u(x, T ) = E 1 {T <ζ(x)} f (X T (x))
with bounded initial conditions u(•, 0) = f , since for arbitrarily small T > 0 Brownian motion explores the whole manifold M , we observe that the global structure of M enters in formula (1.4); lower Ricci bounds may fail and thus "Brownian motion may travel arbitrarily fast". Nevertheless, looking at the formula for the gradient ∇u(•, T ) x and taking into account that

u(X τ (x), T -τ ) = E Fτ 1 {T <ζ(x)} f (X T (x)) ,
we see that Eq. (1.3) reads as

∇u(•, T ) x , v = -E 1 {T <ζ(x)} f (X T (x)) τ 0 Q s ls , dB s
where τ = τ (x) ∧ T and τ (x) the first exit time of X . (x) from an arbitrarily small chosen neighbourhood of x. In other words, as far as the gradient at (x, T ) is concerned, Ricci curvature of M matters only locally about the point x.

No derivative of the heat equation appears in the right-hand side of Eq. (1.4), thus gradient formulas are well suitable for estimates, see [START_REF] Thalmaier | Gradient estimates for harmonic functions on regular domains in Riemannian manifolds[END_REF]. Such inequalities easily allow to bound ∇u in terms of some uniform norm of u.

For positive solutions of the heat equation however one wants to do better: typically one seeks for pointwise estimates which allow (modulo additional terms if necessary) to control ∇u(•, T ) x by u(x, T ).

Theorem 1.3 (Classical Li-Yau estimate [START_REF] Li | On the parabolic kernel of the Schrödinger operator[END_REF]). Let M be complete Riemannian manifold and assume that Ric ≥ -k where k ≥ 0. Let u be a strictly positive solution of

∂ ∂t u = 1 2 ∆u on M × R + and let a > 1. Then (1.5) |∇u| u 2 (x, T ) -a ∆u u (x, T ) ≤ c(n, a) k + 1 T .
If Ric ≥ 0, i.e. k = 0, then the choice a = 1 is possible.

It is a surprising fact which has been noticed by many people that no straightforward way to pass from Bismut type derivative formulas to Li-Yau type gradient estimates seems to exist.

Li-Yau type inequalities for positive solutions u of the heat equation aim at estimating ∇ log u rather than ∇u. This adds an interesting non-linearity to the problem which is better to deal with in terms of submartingales and Bismut type inequalities than in terms of martingales which are the underlying concept for Bismut formulas. This point of view has been worked out in [START_REF] Arnaudon | Gradient estimates for positive harmonic functions by stochastic analysis[END_REF] for local estimates in the elliptic case of positive harmonic functions. Such estimates in global form, i.e., for positive harmonic functions on Riemannian manifolds, are due to S.T. Yau [START_REF] Yau | Harmonic functions on complete Riemannian manifolds[END_REF]; local versions have been established by Cheng and Yau [START_REF] Cheng | Differential equations on Riemannian manifolds and their geometric applications[END_REF].

In this paper we pursue the approach via submartingales to study the parabolic case. Even if it is meanwhile quite standard to obtain Li-Yau type estimates in global form via analytic methods, local versions require often completely new arguments [START_REF] Bakry | A logarithmic Sobolev form of the Li-Yau parabolic inequality[END_REF][START_REF] Ni | The entropy formula for linear heat equation[END_REF][START_REF] Souplet | Sharp gradient estimate and Yau's Liouville theorem for the heat equation on noncompact manifolds[END_REF][START_REF] Qi | Some gradient estimates for the heat equation on domains and for an equation by Perelman[END_REF].

In this paper we derive various submartingales which lead to the wanted estimates in a surprisingly simple way. §2. Basic formulas related to positive solutions of the heat equation and some elementary submartingales

The following formulas for solutions of the heat equation on a Riemannian manifold depend on the fact that the solutions are strictly positive.

Lemma 2.1. Let M be a Riemannian manifold (not necessarily complete) and let u = u(x, t) be a positive solution of the heat equation

(2.1) ∂ ∂t u = 1 2 ∆u on M × [0, T ].
Then the following equalities hold:

1 2 ∆ -∂ t (log u) = - 1 2 |∇u| 2 u 2 , (2.2) 1 2 ∆ -∂ t (u log u) = 1 2 |∇u| 2 u , (2.3) 1 2 ∆ -∂ t |∇u| 2 u = 1 u Hess u - ∇u ⊗ ∇u u 2 + Ric(∇u, ∇u) u . (2.4)
Proof. All three equalities are easily checked by direct calculation.

Q.E.D.

Eq. (2.4) in Lemma 2.1 gives raise to some inequalities frequently used in the sequel and crucial for our approach. Most of our results are based on the following observation.

Corollary 2.2. Let M be a Riemannian manifold of dimension n (not necessarily complete) and let u = u(x, t) be a positive solution of the heat equation (2.1). Then we have:

(2.5) 1 2 ∆ -∂ t |∇u| 2 u ≥ 1 nu ∆u - |∇u| 2 u 2 + Ric(∇u, ∇u) u .
If Ric ≥ -k on M for some k ≥ 0, then

(2.6) 1 2 ∆ -∂ t |∇u| 2 u ≥ 1 nu ∆u - |∇u| 2 u 2 -k |∇u| 2 u ,
and in particular,

(2.7) 1 2 ∆ -∂ t |∇u| 2 u ≥ -k |∇u| 2 u .
Proof. This is a direct consequence of Eq. (2.4). Q.E.D.

Lemma 2.3. Let M be a Riemannian manifold (not necessarily complete) and let u(x, t) = P t f (x) be a positive solution of the heat equation

(2.8) ∂ ∂t u = 1 2 ∆u on M × [0, T ].
For any Brownian motion X on M , the process

m 1 t := |∇P T -t f | 2 P T -t f (X t ) exp - t 0 Ric(X r ) dr , (2.9)
where Ric(x) denotes the smallest eigenvalue of the Ricci curvature at the point x, is a local submartingale (up to its natural lifetime). Furthermore

m 2 t := log P T -t f (X t ) + 1 2 t 0 ∇P T -s f P T -s f 2 (X s ) ds (2.10)
m 3 t := P T -t f log P T -t f (X t ) - 1 2 t 0 |∇P T -s f | 2 P T -s f (X s ) ds (2.11)

are local martingales (up to their respective lifetimes).

Proof. The first claim is a consequence of (2.5); the second part comes from Eqs. (2.2) and (2.3).

Q.E.D.

Lemma 2.4. Let M be a Riemannian manifold and u(x, t) = P t f (x) be a positive solution of the heat equation (2.8) on M × [0, T ]. If Ric ≥ -k for some k ≥ 0, then for any Brownian motion X on M , the process

(2.12) N t := T -t 2(1 + k(T -t)) |∇P T -t f | 2 P T -t f (X t )+ P T -t f log P T -t f (X t )
is a local submartingale (up to its lifetime).

Proof. The proof follows from Itô's formula using inequality (2.7), along with Eq. (2.3).

Q.E.D. §3. Global gradient estimates

In this section we explain how submartingales related to positive solutions of the heat equation can be turned into gradient estimates. The resulting estimates of this section are classical inequalities; our focus lies on the stochastic approach.

The main problem in the subsequent sections will then be to use methods of Stochastic Analysis to derive localized versions of the bounds. The following gradient estimates follow immediately from Lemma 2.4. Theorem 3.1 (Entropy estimate). Let u(x, t) = P t f (x) be a positive solution of the heat equation on a compact manifold M . Assume that Ric ≥ -k for some k ≥ 0. Then

(3.1) ∇P T f P T f 2 (x) ≤ 2 1 T + k P T f P T f (x) log f P T f (x) (x).
Proof. Indeed if M is compact, then the local submartingale N t in (2.12) is a true submartingale. Let

h(t) = T -t 2(1 + k(T -t))
.

Then, evaluating E[N 0 ] ≤ E[N T ], we obtain h(0) |∇P T f | 2 P T f (x) + P T f (x) log P T f (x) ≤ P T (f log f )(x),
or in other words,

|∇P T f | 2 P T f (x) ≤ 1 h(0) P T f log f P T f (x) (x).
Dividing through P T f (x) completes the proof. Q.E.D.

Corollary 3.2. Keeping notation and assumptions of Theorem 3.1 we observe that, for any δ > 0,

|∇P T f (x)| ≤ 1 2δ 1 T + k P T f (x) + δ P T (f log f ) (x) -P T f (x) log P T f (x) . (3.2)
Proof. Indeed with h(0) = T /(2 + 2kT ), we conclude from (3.1) that

|∇P T f | P T f (x) ≤ 1 2δ h(0) 2δ P T f P T f (x) log f P T f (x) (x) ≤ 1 2δ 1 T + k + δ P T f P T f (x) log f P T f (x) (x).
Q.E.D.

Corollary 3.3 (Hamilton [START_REF] Hamilton | A matrix Harnack estimate for the heat equation[END_REF]). Let M be a compact Riemannian manifold such that Ric ≥ -k throughout M for some k ≥ 0. Suppose that u(x, t) is a positive solution of the heat equation (2.8) on M × [0, T ], and let A := sup M×[0,T ] u. Then

(3.3) |∇u| 2 u 2 (x, T ) ≤ 2 1 T + k log A u(x, T ) .
In particular, if Ric ≥ 0 then

(3.4) |∇u| u (x, T ) ≤ 1 T 1/2 2 log A u(x, T ) .
Proof. The proof of (3.3) is an application of Theorem 3.1, along with the observation that P T f * (x) = 1 when f is normalized as f * := f /P T f (x) for fixed x.

Q.E.D.

Remark 3.4. In the proofs above compactness of the manifold has only been used to assure that the local submartingale (2.12) is a true submartingale. The results extend to bounded positive solution of the heat equation on complete manifolds with lower Ricci bounds. Indeed, u(x, t) = P t f (x) may be assumed to be bounded away from 0 (otherwise one may first pass to u * := u + ε and let ε > 0 tend to 0 in the obtained estimate). The term (T -t) |∇P T -t f | 2 may be bounded by Bismut's formula, see [START_REF] Thalmaier | Gradient estimates for harmonic functions on regular domains in Riemannian manifolds[END_REF].

We now turn to the classical Li-Yau estimate (1.5). Let M be a given complete Riemannian manifold of dimension n such that Ric ≥ -k for some k ≥ 0. Suppose that u is a positive solution to the heat equation (2.8) on M × [0, T ]. Starting from (2.6) in Corollary 2.2 we have

(3.5) 1 2 ∆ -∂ t |∇u| 2 u ≥ 1 nu ∆u - |∇u| 2 u 2 -k |∇u| 2 u ,
and thus

(3.6) 1 2 ∆ -∂ t |∇u| 2 u ≥ 1 nu q 2 -k q -k ∆u,
where

q := |∇u| 2 u -∆u = |∇u| 2 u -2 ∂ t u.
Fixing x ∈ M , let X = X(x) be a Brownian motion on M starting at x. Our first goal is to investigate the process

h t |∇u| 2 u (X t , T -t) t≥0
where h t = ℓ 2 t for some adapted continuous real-valued process ℓ t with absolutely continuous paths such that ℓ 0 = 1 and ℓ T = 0.

Let q t = q(X t , T -t), u t = u(X t , T -t) and (∆u) t = ∆u(X t , T -t). Using (3.6) we find (modulo differentials of local martingales)

d h t |∇u| 2 u (X t , T -t) (3.7) ≥ ḣt q t + h t nu t q 2 t -k q t h t + ḣt -k h t (∆u) t dt. (3.8)
Minimizing the term

h t nu t q 2 t + ( ḣt -k h t )q t
as a quadratic function of q t , we find

h t nu t q 2 t + ( ḣt -k h t )q t ≥ - ( ḣt -k h t ) 2 4h t nu t .
Thus, integrating (3.7) from 0 to T and taking expectations, we obtain (3.9)

|∇u| 2 u (x, T ) ≤ E T 0 n ( ḣt -k h t ) 2 4h t u t + (kh t -ḣt )(∆u) t dt
Theorem 3.5 (Li-Yau inequality for Ric ≥ 0). Let M be a complete Riemannian manifold of dimension n such that Ric ≥ 0. Let u = u(x, t) be a positive bounded solution of the heat equation

(2.8) on M × [0, T ]. Then, for each x ∈ D, (3.10) |∇u| 2 u 2 (x, T ) - ∆u u (x, T ) ≤ n E T 0 | lt | 2 u(X t (x), T -t) u(x, T ) dt
where (ℓ t ) is an adapted continuous real-valued process ℓ t with absolutely continuous paths such that ℓ 0 = 1 and ℓ T = 0.

In particular, with the choice ℓ t := (T -t)/T , we obtain

|∇u| 2 u 2 (x, T ) - ∆u u (x, T ) ≤ n T .
which is the classical estimate of Li-Yau.

Proof. By (3.7) we have

(3.11) |∇u| 2 u (x, T ) ≤ E T 0 n | ḣt | 2 4h t u t -ḣt (∆u) t dt.
First note that ∆u is a solution of the heat equation as well, hence (∆u) t a local martingale. In particular,

d h t (∆u) t m = ḣt (∆u) t dt,
where m = stands for equality modulo differentials of local martingales, and hence

E T 0 ḣt (∆u) t dt = -h 0 (∆u) 0 = -∆u(x, T ).
Thus (3.11) shows that

|∇u| 2 u 2 (x, T ) - ∆u u (x, T ) ≤ n E T 0 l2 t u t u 0 dt = n E T 0 l2 t dt • u T u 0 .
Q.E.D.

Remarks 3.6. (i) Using (3.9), Theorem 3.5 is easily extended to the case of a lower Ricci bound. For local versions of the Li-Yau's estimate one could try to modify the process ℓ t in such a way that ℓ t already vanishes as soon as the Brownian motion X t reaches the boundary of D. We shall not pursue this approach here, but rather adopt an even simpler argument in the next section which leads to the local estimate.

(ii) Of particular interest are localized versions of the entropy estimates (3.1) and (3.2). Such estimates lead to Harnack inequalities and heat kernel bounds, valid on arbitrary manifolds without bounded geometry, see [START_REF] Arnaudon | Harnack inequality and heat kernel estimates on manifolds with curvature unbounded below[END_REF][START_REF] Arnaudon | Gradient estimate and Harnack inequality on non-compact Riemannian manifolds[END_REF]. Results in this direction will be worked out elsewhere [START_REF] Arnaudon | Gradient-entropy estimates on Riemannian manifolds[END_REF]. §4. Local Li-Yau type inequalities Our main task of the remaining sections will be to localize the arguments of Section 3 to cover local solutions of the heat equation on bounded domains. We start with the Li-Yau estimate.

Y t = C 1 (T -t) -1 + C 2 ϕ -2 (X t (x)) + αk,
where C 1 , C 2 > 0 are constants which will be specified later.

Let h t be the solution of

(4.3) ḣt = -h t Y t , h 0 = 1;
in other words,

h t = exp - t 0 C 1 T -r + C 2 ϕ 2 (X r (x))
+ αk dt .

Then, letting

q t = |∇u| 2 u -∆u (X t , T -t),
we find (modulo differentials of local martingales)

(4.4) d(h t q t ) ≥ h t q 2 t nu t + ḣt -kh t q t -kh t (∆u) t dt.
Thus letting (4.5)

S t := h t q t + nu t ḣt ≡ h t (q t -nu t Y t ), t ∈ [0, T ∧ τ (x)[,
we get (modulo differentials of local martingales)

dS t = d(h t q t ) -nu t d(h t Y t ) + n d[u, ḣ] t ≥ h t q 2 t nu t + ḣt -kh t q t -kh t (∆u) t dt + nu t h t Y 2 t dt -nu t h t C 1 (T -t) -2 + C 2 c ϕ (X t ) ϕ -4 (X t ) dt -nC 2 h t d[u, ϕ -2 (X(x))] t
where the bracket [•, •] stands for quadratic covariation on the space of continuous semimartingales and where

(4.6) c ϕ (x) = 3|∇ϕ| 2 -ϕ∆ϕ (x).
On the other hand, denoting again

u t = u(T -t, •)(X t ), ∇u t = ∇u(T -t, •)(X t ) (4.7) and ∆u t = ∆u(T -t, •)(X t ), (4.8)
we have for any α > 0,

-nC 2 h t d[u, ϕ -2 (X t )] t = nC 2 h t 2ϕ -3 (X t ) ∇u t , ∇ϕ(X t ) dt ≥ -2nC 2 h t ϕ -3 (X t )|∇u t | |∇ϕ(X t )| dt = -2nC 2 h t ϕ -1 (X t ) (αnu t ) -1/2 |∇u t | × (αnu t ) 1/2 (ϕ -2 |∇ϕ|)(X t ) dt ≥ -α -1 C 2 h t ϕ -2 (X t )(q t + ∆u t ) -C 2 αn 2 u t h t ϕ -4 |∇ϕ| 2 (X t ) dt ≥ -α -1 h t (Y t -αk)(q t + ∆u t ) -C 2 αn 2 u t h t ϕ -4 |∇ϕ| 2 (X t ) dt = α -1 ḣt + kh t (q t + ∆u t ) -C 2 αn 2 u t h t ϕ -4 |∇ϕ| 2 (X t ) dt.
Hence letting, for α > 0, (4.9) S α,t = h t q t -α -1 ∆u t + nu t ḣt we get Minimizing the first line on the right hand side, we find

dS α,t ≥ h t q 2 t nu t + 1 + α -1 ḣt q t dt + h t Y 2 t nu t dt -nu t h t C 1 (T -t) -2 + C 2 C ϕ,α,n ϕ -4 (X t ) dt
dS α,t ≥ h t Y 2 t nu t 4 -(1 + α -1 ) 2 4 dt -nu t h t C 1 (T -t) -2 + C 2 C ϕ,α,n ϕ -4 (X t ) dt.
Putting things together, we arrive at the following result.

Lemma 4.1. We keep notation and assumptions from above. Assume that α > 1 and let

(4.11) C 1 ≥ C 1 (α) := 4 4 -(1 + α -1 ) 2 and C 2 ≥ C 2 (ϕ, α, n) := 4 4 -(1 + α -1 ) 2 sup D (3 + αn)|∇ϕ| 2 -ϕ∆ϕ . (4.12) Then S α,t is a local submartingale on [0, T ∧ τ (x)[. Proof. Using (4.13) Y 2 t ≥ C 2 1 (T -t) -2 + C 2 2 ϕ -4 (X t ),
we get

dS α,t ≥ h t nu t C 1 4 -(1 + α -1 ) 2 4 C 1 -1 (T -t) -2 dt + h t nu t C 2 4 -(1 + α -1 ) 2 4 C 2 -C ϕ,α,n ϕ -4 (X t ) dt.
Thus, under condition (4.11) and (4.12), the right-hand side is nonnegative.

Q.E.D. For any α ∈ ]1, ∞[, we have

(4.14) |∇u 0 | 2 u 2 0 -(1 + α -1 ) ∆u 0 u 0 ≤ n C 1 (α) T + n C 2 (ϕ, α, n) ϕ 2 (x) + nαk,
where k is a lower Ricci bound on the domain D, and where C 1 (α) and C 2 (ϕ, α, n) are specified in (4.11), resp. (4.12). The function ϕ is as in Assumption 1.

Recall that u 0 = u(•, T ) and ∆u 0 = ∆u(•, T ) according to (4.7), resp. (4.8)

Proof. Let C 1 = C 1 (α) and C 2 = C 2 (ϕ, α, n). Consider S α,t = h t q t -α -1 ∆u t -nu t Y t .
We assume that S α,0 > 0 and let τ α be the first hitting time of 0 by S α,t . Then clearly τ α < T ∧ τ (x), since

q t -α -1 ∆u t -nu t Y t converges to -∞ as t tends to T ∧ τ (x). Let K := max D×[0,T ]
(q -α -1 ∆u) and m = min

D×[0,T ] u.
Then, for t ∈ [0, τ α ], we have S t ≥ 0, which implies Y t ≤ K/nm. From this we easily prove that on [0, τ α ], the process S α,t is a submartingale with bounded local characteristics. As a consequence, we have

S α,0 ≤ E[S α,τα ].
Since S α,τα ≡ 0, this contradicts the assumption S α,0 > 0. Hence we must have S α,0 ≤ 0, which is the desired inequality. Q.E.D.

Remark 4.3. In the case of a global solution of the heat equation on a compact manifold, we can take ϕ as a constant and then C 2 (α, ϕ, n) = 0. If moreover k = 0, then one can take C 2 = 0 and C 1 = 1, and S t is a local submartingale. This recovers one more time the usual Li-Yau estimate

(4.15) |∇u 0 | 2 u 2 0 - ∆u 0 u 0 ≤ n T .
A similar reasoning applies for global solutions on complete Riemannian manifolds with a lower Ricci curvature bound. §5. Li-Yau inequality with lower order term

We keep Assumption 1 and 2 of Section 4 as standing assumptions for the rest of the paper and study now the process (5.1)

S ′ t = q t -nu t Z t
where

Z t = C 1 (T -t) -1 + C 2 ϕ -2 (X t ) + C 3 with constants C 1 , C 2 , C 3 > 0 to be specified later. Let (5.2) C ϕ = sup D 3|∇ϕ| 2 -ϕ∆ϕ .
Then we have

dS ′ t ≥ q 2 t nu t -k |∇u t | 2 u t -nu t C 1 (T -t) -2 + C 2 C ϕ ϕ -4 (X t ) dt -C 2 nd [ϕ -2 (X), u] t ≥ q 2 t nu t -nu t C 1 (T -t) 2 + C 2 C ϕ ϕ 4 (X t ) + k n |∇u| u 2 D×[0,T ] dt -2C 2 nu t |∇u| u D×[0,T ] ϕ∇ϕ 2 D ϕ -4 (X t ) dt ≥ q 2 t nu t -nu t C 1 (T -t) 2 + C 2 ϕ 4 (X t ) × C ϕ + 2 |∇u| u D×[0,T ] ϕ∇ϕ D dt -nu t k n |∇u| u 2 D×[0,T ]
dt.

Lemma 5.1. Let

C 1 = 1, C 2 = C ϕ + 2 |∇u| u D×[0,T ]
ϕ∇ϕ ∞ and

C 3 = k n |∇u| u D×[0,T ]
.

(5.3)

Then on {S ′ t ≥ 0}, the process S ′ t has nonnegative drift.

Proof. We have

Z 2 t ≥ C 2 1 (T -t) -2 + C 2 2 ϕ -4 (X t ) + C 2 3 .
Consequently, under condition (5.3),

dS ′ t ≥ 1 nu t q 2 t -(nu t Z t ) 2 dt = S ′ t nu t (q t + nu t Z t ) dt
and the right-hand side is nonnegative on

{S ′ t ≥ 0}. Q.E.D.
Similarly to Theorem 4.2, we obtain the following result.

|∇u 0 | 2 u 2 0 - ∆u 0 u 0 ≤ n T + nC ϕ ϕ 2 (x) + √ nk + 2n ϕ∇ϕ D ϕ 2 (x) |∇u| u D×[0,T ]
.

§6. Local gradient estimates of Hamilton type

We keep Assumptions 1 and 2 of Section 4 and study now the process (6.1)

S t = |∇u t | 2u t -u t (1 -log u t ) 2 Z t ,
where

Z t = C 1 (T -t) + C 2 ϕ 2 (X t ) + C 3 for some constants C 1 , C 2 , C 3 > 0.
Assume that 0 < u ≤ e -3 (this assumption will be removed in Theorem 6.1 through replacing u by e -3 u/ u D×[0,T ] ). Let c ϕ (x) again be given by (4.6). Then, denoting g(t, x) = |∇u| 2 u (t, x) and g t = g(T -t, X t ), and using the fact that ∇ u(1log u) 2 = (log 2 u -1)∇u and

d u t (1 -log u t ) 2 m = g t log u t dt,
we get

dS t ≥ -k 1 2 g t dt -u t (1 -log u t ) 2 C 1 (T -t) 2 + C 2 c ϕ (X t )ϕ -4 (X t ) dt -2 log u t 1 2 g t Z t dt -2C 2 (1 -log u t )(1 + log u t )ϕ -3 (X t ) ∇u t , ∇ϕ(X t ) dt. Now from u ≤ e -3 we get -2 log u t ≥ 3(1 -log u t )/2.
This together with |1 + log u t | ≤ 1log u t yields (modulo differentials of local martingales)

dS t ≥ (1 -log u t ) 3 2 Z t -k 1 2 g t -u t (1 -log u t ) C 1 (T -t) 2 + C 2 c ϕ (X t )ϕ -4 (X t ) dt -C 2 (1 -log u t ) 2 √ 2 ϕ -2 (X t ) |∇ϕ(X t )| √ u t ϕ -1 (X t ) g t 2 dt ≥ (1 -log u t ) 3 2 Z t -k 1 2 g t -u t (1 -log u t ) C 1 (T -t) 2 + C 2 c ϕ (X t )ϕ -4 (X t ) dt -(1 -log u t ) 2 u t 4C 2 ϕ -4 (X t ) |∇ϕ(X t )| 2 + 1 4 g t C 2 ϕ -2 (X t ) dt ≥ (1 -log u t ) (Z t -k) 1 2 g t -u t (1 -log u t ) 2 × C 1 (T -t) 2 + C 2 c ϕ (X t ) + 4|∇ϕ(X t )| 2 ϕ -4 (X t ) dt. Letting (6.2) C 1 = 1, C 2 = sup D c ϕ + 4|∇ϕ| 2 and C 3 = k, we get dS t ≥ (1 -log u t ) (Z t -k) 1 2 g t -u t (1 -log u t ) 2 (Z t -k) 2 dt ≥ (1 -log u t )(Z t -k) 1 2 g t -u t (1 -log u t ) 2 Z t dt.
This proves that S t has nonnegative drift on {S t ≥ 0}. On the other hand, S t converges to -∞ as t → T ∧ τ (x).

Similarly to Theorem 4.2, we obtain the following result.

∇u 0 u 0 2 ≤ 2 1 T + sup D 7|∇ϕ| 2 -ϕ∆ϕ ϕ 2 (x) + k 4 + log u D×[0,T ] u 0 2
where ϕ is as above. §7. Explicit upper bounds

The estimates in Theorems 4.2, 5.2 and 6.1 have been given in terms of a function ϕ ∈ C 2 ( D) such that ϕ > 0 in D and ϕ|∂D = 0. To specify the constants an explicit choice for ϕ has to be done.

We fix x ∈ D and let δ x = ρ(x, ∂D) where ρ denotes the Riemannian distance. We replace D by the ball B = B(x, δ x ) and consider on B (7.1) ϕ(y) = cos πρ(x, y) 2δ x .

Clearly ϕ(x) = 1, ϕ is nonnegative and bounded by 1, and ϕ vanishes on ∂B.

It is proven in [START_REF] Thalmaier | Gradient estimates for harmonic functions on regular domains in Riemannian manifolds[END_REF] that

(7.2) dϕ -2 (X t ) ≤ 1 2 ∆(ϕ -2 )(X t ) dt
where by convention ∆ϕ -2 = 0 at points where ϕ -2 is not differentiable. Moreover, since the time spent by X t on the cut-locus of x is a.s. zero, the differential of the brackets [ϕ(X t ), u t ] may be taken as 0 at points where ϕ -2 is not differentiable. As a consequence, all estimates in Theorems 4.2, 5.2 and 6.1 remain valid with ϕ defined by (7.1).

We are now going to derive explicit expressions for the constants. To this end we observe that ∇ϕ B ≤ π 2δ x .

From [START_REF] Thalmaier | Gradient estimates for harmonic functions on regular domains in Riemannian manifolds[END_REF], we get

-∆ϕ ≤ π k(n -1) 2δ x + π 2 n 4δ 2
x which gives for any β > 0,

-∆ϕ ≤ π 2 (1 + β)n 4δ 2 x + k 4β .
This yields

C ϕ ≤ π 2 [(1 + β)n + 3] 4δ 2 x + k 4β , C ϕ,α,n ≤ π 2 [(1 + β + α)n + 3] 4δ 2 x + k 4β , sup D 7|∇ϕ| 2 -ϕ∆ϕ ≤ π 2 [(1 + β)n + 7] 4δ 2 x + k 4β .
Finally we replace α by a = 1 + α -1 to obtain from Theorems 4.2, 5.2 and 6.1 the following explicit upper bounds.

Theorem 1 . 1 (

 11 Stochastic representation of the gradient). Let D be a relatively compact open domain in a complete Riemannian manifold M , and let u = u(x, t) be a solution of the heat equation (1.1) on D × [0, T ] which is continuous on D × [0, T ]. Then, for any v ∈ T x M and x ∈ D,

Assumption 1 .Assumption 2 .

 12 Let M be a complete Riemannian manifold of dimension n, and let u = u(x, t) be a solution of the heat equation (2.8) on D×[0, T ] where D ⊂ M is a relatively compact open subset of M with nonempty smooth boundary. Assume that u is positive and continuous on D × [0, T ]. Furthermore let (4.1) k := inf Ric x (v, v) : v ∈ T x M, |v| = 1, x ∈ D be a lower bound for the Ricci curvature on the domain D. Finally let ϕ ∈ C 2 ( D) with ϕ > 0 in D and ϕ|∂D = 0. Let x ∈ D and X(x) be a Brownian motion on M starting at x at time 0. Denote by τ (x) its first exit time of D. Now, for t ∈ [0, T ∧ τ (x)[, consider the process (4.2)

where ( 4 . 10 )( 3 +

 4103 C ϕ,α,n = sup D c ϕ + αn|∇ϕ| 2 = sup D αn)|∇ϕ| 2 -ϕ∆ϕ .

Theorem 4 . 2 (

 42 Li-Yau inequality; local version). We keep the assumptions from above. Let u = u(x, t) be a solution of the heat equation on D × [0, T ] which is positive and continuous on D × [0, T ].

Theorem 5 . 2 (

 52 Local Li-Yau inequality with lower order term). We keep the notation from above, as well as Assumption 1 and 2 from Section 4. Let u = u(x, t) be a solution of the heat equation on D × [0, T ] which is positive and continuous on D × [0, T ]. Then

Theorem 6 . 1 (

 61 Local Li-Yau inequality of Hamilton type). We keep the assumptions as above. Assume that u is a solution of the heat equation on D × [0, T ] which is positive and continuous on D × [0, T ]. Then

Theorem 7 . 1 (where

 71 Local Li-Yau inequalities with explicit constants). Let u be a solution of the heat equation on D × [0, T ] where D is a relatively compact open subset of a Riemannian manifold M . Assume that u is positive and continuous on D × [0, T ]. Furthermore let k be a lower bound for the Ricci curvature on the domain D. Fix x ∈ D and let a ∈ ]1, 2[. For any β > 0 we have n denotes the dimension of M and δ x the Riemannian distance of x to the boundary of D. Recall that u 0 = u(•, T ) and ∆u 0 = ∆u(•, T ).


Yana Hasson

Gül Varol

Cordelia Schmid

Ivan Laptev

Towards Unconstrained Joint Hand-Object Reconstruction From RGB Videos

Our work aims to obtain 3D reconstruction of hands and manipulated objects from monocular videos. Reconstructing hand-object manipulations holds a great potential for robotics and learning from human demonstrations. The supervised learning approach to this problem, however, requires 3D supervision and remains limited to constrained laboratory settings and simulators for which 3D ground truth is available. In this paper we first propose a learningfree fitting approach for hand-object reconstruction which can seamlessly handle two-hand object interactions. Our method relies on cues obtained with common methods for object detection, hand pose estimation and instance segmentation. We quantitatively evaluate our approach and show that it can be applied to datasets with varying levels of difficulty for which training data is unavailable.

Introduction

Joint reconstruction of hand-object interactions in the context of object manipulation has a broad range of practical applications in robotics and augmented reality. Yet, reliable and generic models for recovering 3D hand-object configurations from images do not exist to date. Learningbased approaches typically do not generalize beyond their specific training domains. Furthermore, obtaining precise 3D annotations for supervision is tedious and requires additional sensors. Existing datasets rely on depth data [START_REF] Chao | DexYCB: A benchmark for capturing hand grasping of objects[END_REF][START_REF] Hampali | HOnnotate: A method for 3D annotation of hand and object poses[END_REF], visible marker equipment [START_REF] Garcia-Hernando | First-person hand action benchmark with RGB-D videos and 3D hand pose annotations[END_REF][START_REF] Pham | Hand-object contact force estimation from markerless visual tracking[END_REF] or multiple views [START_REF] Brahmbhatt | ContactPose: A dataset of grasps with object contact and hand pose[END_REF][START_REF] Moon | InterHand2.6M: A dataset and baseline for 3D interacting hand pose estimation from a single RGB image[END_REF][START_REF] Simon | Hand keypoint detection in single images using multiview bootstrapping[END_REF][START_REF] Zimmermann | FreiHAND: A dataset for markerless capture of hand pose and shape from single RGB images[END_REF]. Moreover, such datasets typically feature a limited number of objects [START_REF] Mueller | Real-time hand tracking under occlusion from an egocentric RGB-D sensor[END_REF][START_REF] Sridhar | Real-time joint tracking of a hand manipulating an object from RGB-D input[END_REF][START_REF] Tzionas | Capturing hands in action using discriminative salient points and physics simulation[END_REF][START_REF] Tzionas | 3D object reconstruction from hand-object interactions[END_REF] while still requiring manual annotations [START_REF] Chao | DexYCB: A benchmark for capturing hand grasping of objects[END_REF] or post-processing [START_REF] Simon | Hand keypoint detection in single images using multiview bootstrapping[END_REF][START_REF] Zimmermann | FreiHAND: A dataset for markerless capture of hand pose and shape from single RGB images[END_REF]. Synthetic rendering techniques can be used to create labeled datasets with a larger number of objects, however, the resulting images lack realism in terms of appearance and grasp configurations [START_REF] Hasson | Learning joint reconstruction of hands and manipulated objects[END_REF][START_REF] Mueller | GANerated hands for real-time 3D hand tracking from monocular RGB[END_REF][START_REF] Zimmermann | Learning to estimate 3D hand pose from single RGB images[END_REF].

Developing an RGB-only method to retrieve 3D handobject configurations for diverse objects would enable scal-Figure 1: Hand-object reconstructions: We present an optimization-based method to fit 3D parametric hand model and a known object model to 2D estimates. Our method is generic and can be applied across a variety of recent video datasets presenting hand-object interactions [START_REF] Damen | Scaling egocentric vision: The epic-kitchens dataset[END_REF][START_REF] Hampali | HOnnotate: A method for 3D annotation of hand and object poses[END_REF][START_REF] Lomonaco | Core50: a new dataset and benchmark for continuous object recognition[END_REF]. Our method can also handle reconstruction of images with left and right hands.

ing up the datasets, and help the field move towards in-thewild scenes. In this work, we argue for an optimizationbased approach for its robustness across domains. Recent progress in 2D detection of objects and 3D pose estimation of isolated hands makes it possible to obtain a good initialization when fitting 3D hand-object poses to these estimates. Nevertheless, this is still very challenging due to depth ambiguities, occlusions, noisy 2D estimates and can result in physically implausible configurations. In our analysis, we show improvements over independent fits when considering a joint hand-object fitting framework with several interaction constraints. We take inspiration from 1 PHOSA [START_REF] Zhang | Perceiving 3D human-object spatial arrangements from a single image in the wild[END_REF], a recent work showing promising results for fitting 3D human bodies and objects, relying on object segmentation and human pose estimation models.

Several approaches propose to explore the complementarity between fitting and learning approaches for human body shape estimation. Fitting for learning has been explored for reconstructing human bodies, either semi-automatically [START_REF] Lassner | Unite the people: Closing the loop between 3d and 2d human representations[END_REF] or via learning with fitting in-theloop [START_REF] Kolotouros | Learning to reconstruct 3D human pose and shape via model-fitting in the loop[END_REF]. Recently, applying fitting on sign language videos has shown benefits for training isolated hand reconstruction models [START_REF] Kulon | Weakly-supervised mesh-convolutional hand reconstruction in the wild[END_REF]. [START_REF] Liu | Semi-supervised 3d hand-object poses estimation with interactions in time[END_REF] shows that hand pseudolabelling and automatic filtering using spatial and temporal consistency allows to improve the hand pose estimation branch of a model trained for joint hand-object pose estimation. Different from these works, we fit hand and object meshes which represent a more complex scene. [START_REF] Hampali | HOnnotate: A method for 3D annotation of hand and object poses[END_REF] presents a similar approach by fitting hand-object configurations to RGB-D videos to later use in training; however, the reliance on a depth sensor significantly reduces its applicability to unconstrained images.

In this work, we propose a method which reconstructs the hand and manipulated object from short RGB video clips, assuming an approximate model for the target object is available. We investigate the strengths and limitations of our approach in challenging scenarios and present results on challenging scenes which are not currently handled by learning methods for joint hand-object pose estimation. Our code is publicly available at https://github.com/ hassony2/homan.

Our contributions are the following: (i) We propose a fitting-based approach for hand-object reconstruction from a video clip; (ii) We present a detailed quantitative evaluation analyzing different components of our optimization method and compare to learning-based models [START_REF] Hasson | Leveraging photometric consistency over time for sparsely supervised hand-object reconstruction[END_REF] on a standard benchmark [START_REF] Hampali | HOnnotate: A method for 3D annotation of hand and object poses[END_REF]; (iii) We demonstrate qualitatively the capabilities of our framework to generalize on unconstrained videos and two-hand reconstructions.

Related Work

We briefly review relevant literature on hand-object reconstruction, works that employ temporal constraints and hand-object datasets. Hand-object reconstruction. 3D pose estimation for hands [START_REF] Iqbal | Hand pose estimation via latent 2.5d heatmap regression[END_REF][START_REF] Kulon | Weakly-supervised mesh-convolutional hand reconstruction in the wild[END_REF][START_REF] Mueller | GANerated hands for real-time 3D hand tracking from monocular RGB[END_REF][START_REF] Simon | Hand keypoint detection in single images using multiview bootstrapping[END_REF][START_REF] Zimmermann | Learning to estimate 3D hand pose from single RGB images[END_REF] and objects [START_REF] Groueix | AtlasNet: A papier-mâché approach to learning 3D surface generation[END_REF][START_REF] Kato | Neural 3D mesh renderer[END_REF][START_REF] Labbe | CosyPose: Consistent multi-view multi-object 6D pose estimation[END_REF][START_REF] Li | Deepim: Deep iterative matching for 6d pose estimation[END_REF][START_REF] Mescheder | Occupancy networks: Learning 3D reconstruction in function space[END_REF][START_REF] Park | DeepSDF: Learning continuous signed distance functions for shape representation[END_REF][START_REF] Wang | Pixel2Mesh: Generating 3D mesh models from single RGB images[END_REF][START_REF] Xiang | PoseCNN: A convolutional neural network for 6d object pose estimation in cluttered scenes[END_REF] have often been tackled in isolation [START_REF] Lepetit | Recent advances in 3d object and hand pose estimation[END_REF]. Joint reconstruction of hands and objects has recently received increased attention [START_REF] Cao | Reconstructing hand-object interactions in the wild[END_REF][START_REF] Doosti | Hope-net: A graph-based model for hand-object pose estimation[END_REF][START_REF] Hampali | HOnnotate: A method for 3D annotation of hand and object poses[END_REF][START_REF] Hasson | Leveraging photometric consistency over time for sparsely supervised hand-object reconstruction[END_REF][START_REF] Hasson | Learning joint reconstruction of hands and manipulated objects[END_REF][START_REF] Bugra Tekin | H+O: Unified egocentric recognition of 3D hand-object poses and interactions[END_REF]. Hasson et al. [START_REF] Hasson | Learning joint reconstruction of hands and manipulated objects[END_REF] introduces an end-to-end model to regress MANO [START_REF] Romero | Embodied hands: Modeling and capturing hands and bodies together[END_REF] hand parameters jointly with object mesh vertices deformed from a sphere. Works of [START_REF] Hasson | Leveraging photometric consistency over time for sparsely supervised hand-object reconstruction[END_REF][START_REF] Bugra Tekin | H+O: Unified egocentric recognition of 3D hand-object poses and interactions[END_REF] assume a known object model and regress a 6DOF object pose instead. Other methods focus on the grasp synthesis [START_REF] Corona | GanHand: Predicting human grasp affordances in multi-object scenes[END_REF][START_REF] Karunratanakul | Grasping field: Learning implicit representations for human grasps[END_REF][START_REF] Taheri | itrios Tzionas. GRAB: A dataset of whole-body human grasping of objects[END_REF], or contact modeling [START_REF] Brahmbhatt | ContactDB: Analyzing and predicting grasp contact via thermal imaging[END_REF][START_REF] Brahmbhatt | ContactPose: A dataset of grasps with object contact and hand pose[END_REF] given a 3D object.

Interaction constraints are imposed to avoid collisions and to encourage contact points. To achieve this, competing attraction and repulsion terms are employed in [START_REF] Corona | GanHand: Predicting human grasp affordances in multi-object scenes[END_REF][START_REF] Hasson | Learning joint reconstruction of hands and manipulated objects[END_REF]. Collision penalization is implemented either with approximate shape primitives [START_REF] Bogo | Keep it SMPL: Automatic estimation of 3D human pose and shape from a single image[END_REF][START_REF] Kyriazis | Scalable 3D tracking of multiple interacting objects[END_REF][START_REF] Oikonomidis | Argyros. Full DOF tracking of a hand interacting with an object by modeling occlusions and physical constraints[END_REF] or triangle meshes [START_REF] Ballan | Motion capture of hands in action using discriminative salient points[END_REF][START_REF] Hampali | HOnnotate: A method for 3D annotation of hand and object poses[END_REF][START_REF] Hassan | Resolving 3D human pose ambiguities with 3D scene constraints[END_REF][START_REF] Hasson | Learning joint reconstruction of hands and manipulated objects[END_REF][START_REF] Jiang | Coherent reconstruction of multiple humans from a single image[END_REF][START_REF] Kyriazis | Physically plausible 3D scene tracking:the single actor hypothesis[END_REF][START_REF] Pavlakos | Expressive body capture: 3D hands, face, and body from a single image[END_REF][START_REF] Tzionas | Capturing hands in action using discriminative salient points and physics simulation[END_REF][START_REF] Zhang | Perceiving 3D human-object spatial arrangements from a single image in the wild[END_REF]. Similarly, in this work we impose error terms in our joint hand-object fitting to favor physically plausible interactions.

Recent work on monocular hand-object reconstruction mostly adopts learning-based CNN models [START_REF] Hampali | HOnnotate: A method for 3D annotation of hand and object poses[END_REF][START_REF] Hasson | Leveraging photometric consistency over time for sparsely supervised hand-object reconstruction[END_REF][START_REF] Hasson | Learning joint reconstruction of hands and manipulated objects[END_REF][START_REF] Kokic | Learning task-oriented grasping from human activity datasets[END_REF][START_REF] Lepetit | Recent advances in 3d object and hand pose estimation[END_REF][START_REF] Liu | Semi-supervised 3d hand-object poses estimation with interactions in time[END_REF][START_REF] Bugra Tekin | H+O: Unified egocentric recognition of 3D hand-object poses and interactions[END_REF]. Such methods obtain promising results but remain limited to constrained datasets and lack generalization. Moreover, learning-based methods are typically limited to the predefined number of hands and objects. In contrast, our method can reconstruct both single and two-hand interactions detected in the image. Concurrent work of [START_REF] Cao | Reconstructing hand-object interactions in the wild[END_REF] extends the optimization-based body-object reconstruction method PHOSA [START_REF] Zhang | Perceiving 3D human-object spatial arrangements from a single image in the wild[END_REF] to perform hand-object fitting. While our method shares similar optimization components with [START_REF] Cao | Reconstructing hand-object interactions in the wild[END_REF], it differs by leveraging video data. Temporal constraints. In case of video inputs, temporal constraints have been used for body motion estimation in the context of neural networks [START_REF] Rayat | Exploiting temporal information for 3D human pose estimation[END_REF][START_REF] Kanazawa | Learning 3D human dynamics from video[END_REF], or optimization [START_REF] Arnab | Exploiting temporal context for 3D human pose estimation in the wild[END_REF][START_REF] Peng | Sfv: Reinforcement learning of physical skills from videos[END_REF]. For hands, [START_REF] Cai | Exploiting spatial-temporal relationships for 3D pose estimation via graph convolutional networks[END_REF] proposes a graph convolutional approach to learn temporal dependencies. Hampali et al. [START_REF] Hampali | HOnnotate: A method for 3D annotation of hand and object poses[END_REF] make use of a temporal consistency term when fitting hand-object configurations to RGB-D data. We explore a similar term to obtain temporally smooth fits to RGB data and initialize the optimization from the previous frame's fit. [START_REF] Liu | Semi-supervised 3d hand-object poses estimation with interactions in time[END_REF] propose to filter noisy hand reconstructions by detecting implausible or inconsistent hand poses and jittering predictions over consecutive frames. 3D label acquisition for hand-object datasets. Recent efforts aim to scale up data collection for 2D hand-object interactions [START_REF] Shan | Understanding human hands in contact at internet scale[END_REF]. Due to the difficulty of annotating 3D in images, several approaches for automatic label acquisition have been proposed. [START_REF] Hasson | Learning joint reconstruction of hands and manipulated objects[END_REF] introduces ObMan, a synthetic dataset obtained by rendering MANO hand model [START_REF] Romero | Embodied hands: Modeling and capturing hands and bodies together[END_REF] automatically grasping ShapeNet objects [START_REF] Angel | ShapeNet: An information-rich 3D model repository[END_REF]. The dataset of [START_REF] Garcia-Hernando | First-person hand action benchmark with RGB-D videos and 3D hand pose annotations[END_REF] utilizes visible magnetic sensors while [START_REF] Pham | Hand-object contact force estimation from markerless visual tracking[END_REF] rely on force sensors which limit the range of posible grasps. [START_REF] Hampali | HOnnotate: A method for 3D annotation of hand and object poses[END_REF] presents the HO-3D dataset where labels result from automatic fits to RGB-D data. [START_REF] Brahmbhatt | ContactPose: A dataset of grasps with object contact and hand pose[END_REF] collects a dataset for 3D printed object models in a restricted capture setup. Very recently, [START_REF] Chao | DexYCB: A benchmark for capturing hand grasping of objects[END_REF] and [START_REF] Kwon | H2o: Two hands manipulating objects for first person interaction recognition[END_REF] propose to collect annotations using hybrid methods to collect reliable labels. In this work, we use the HO-3D benchmark for quantitative evaluations. We also show qualitative results for videos in-the-wild where 3D ground truth is not available.

Automatic labelling of 3D hands and objects

We first describe the optimization-based fitting procedure, consisting of estimating 2D detections (Section 3.1), Our method takes a video of hand-object manipulation as input. We assume that an exact or approximate object model representing the manipulated object is provided and use the ground-truth camera intrinsic parameters when available.

Obtaining 2D hand-object evidence

2D hand and object detection. For each video, the first step is to detect initial 2D bounding boxes. We use a recent hand and manipulated object detector [START_REF] Shan | Understanding human hands in contact at internet scale[END_REF] to extract object and hand bounding boxes in each frame. The hands are predicted with left or right side labels. When the predicted boxes do not match the known properties of the dataset in terms of object presence, hand number or hand sides, we discard the detections for the given frame, and recover detections through tracking in a subsequent step. Tracking. We apply an off-the-shelf 2D bounding box tracker [START_REF] Muron | [END_REF] which relies on Kalman filtering [START_REF] Labbe | Kalman and bayesian filters in python[END_REF] to extract hand and object tracks from the noisy per-frame detections. This step allows to recover missed or discarded detections, and produces hand and object bounding box candidates for the full video clip.

For the Epic-Kitchens dataset, the real number of visible hands is unknown. We automatically select video clips for which at least one object and one hand track extend over more then 20 consecutive frames after tracking. Segmentation. The key image evidence we rely on for fitting is 2D segmentation. We extract instance masks Mobj for each tracked object detection using the instance segmentation head of the PointRend [START_REF] Kirillov | PointRend: Image segmentation as rendering[END_REF]. Similar to PHOSA [START_REF] Zhang | Perceiving 3D human-object spatial arrangements from a single image in the wild[END_REF], we use a model pretrained on the COCO [START_REF] Lin | Microsoft coco: Common objects in context[END_REF] dataset. However, while [START_REF] Zhang | Perceiving 3D human-object spatial arrangements from a single image in the wild[END_REF] fits objects among the COCO categories, our target everyday objects are often not present among the COCO classes. For each object detection, we use the mask associated to the highest class activation of the PointRend instance classifier. We observe this class-agnostic approach to perform well in most cases. To account for hand occlu-sions, we extract the COCO masks associated to the person class Mhand for the tracked hand boxes, see the Section B of the supplemental material for additional details.

Independent pose initialization

Hand initialization. We employ the recent publicly available hand pose estimator FrankMocap [START_REF] Rong | FrankMocap: Fast monocular 3D hand and body motion capture by regression and integration[END_REF] to estimate the initial hand articulated poses, as well as the hand location and scale in pixel space. We recover an estimated depth using the world scale of the hand and the exact intrinsic camera parameters when available. When the exact camera intrinsic parameters are unknown, we approximate the focal length given the specifications of the camera, and assume the central point is at the center of the pixel image. Object initialization. We use the 2D object segmentation to initialize the object pose for the 3D model associated to the target video clip. To obtain pose candidates for the first video frame, we sample random rotations uniformly in SO(3) and use the radius of the instance bounding box to estimate the 3D center of the provided mesh in the first frame. We update the object depth minimizing the difference between the diagonals of the projected model's tight bounding box and the predicted detection in the least squared sense. We then compute a translation update in the camera plane, minimizing the difference between the centers of the projected object model and the detection given a fixed depth. We repeat these two steps until convergence to recover an estimate of the object translation. We further optimize the object pose using differentiable rendering. We optimize the object pose with a pixel squared loss on the difference between the differentiably rendered object mask and the PointRend [START_REF] Kirillov | PointRend: Image segmentation as rendering[END_REF] object segmentation following PHOSA [START_REF] Zhang | Perceiving 3D human-object spatial arrangements from a single image in the wild[END_REF]. As for PHOSA, our object loss is handocclusion aware: only pixels which are not associated with the person label are taken into account. For each subsequent frame, we use the object pose from the previous frame as initialization and further refine it with differentiable rendering as described above. This process results in as many candidate motion initializations as there are candidate object poses. In practice, the number of candidate initializations is empirically set to 50.

We select the object motion candidate for which the average IoU score between the rendered mask and the target occlusion mask is highest.

Joint fitting

Independent hand-object fits are often inaccurate and do not take into account interaction-based constraints. We refine the initial hand-object poses leveraging both coarse and fine-grained manipulation priors.

Optimized parameters. The goal of our fitting is to find the optimal hand and object pose parameters for a sequence of T consecutive frames. For each frame, we optimize 3D translations D hand , D obj , 3D global rotations R hand , R obj as well as θ hand pose parameters. Additionally, we optimize a shared hand scale s hand .

We optimize the articulated MANO [START_REF] Romero | Embodied hands: Modeling and capturing hands and bodies together[END_REF] model in the latent pose space θ. Given the θ pose parameters, the MANO model differentiably outputs 3D hand vertex coordinates centered on the middle metacarpophalangeal joint V c hand = MANO(θ). Following [START_REF] Taheri | itrios Tzionas. GRAB: A dataset of whole-body human grasping of objects[END_REF], we use a pose latent subspace of size 16.

We optimize the hands and object rotation R hand , R obj using the 6D continuous rotation representation [START_REF] Zhou | On the continuity of rotation representations in neural networks[END_REF] and optimize the 3D translation D hand , D obj in metric camera space. When we use approximate object meshes, we additionally optimize a scalar scaling parameter s obj which allows the object's size to vary. We also allow hand vertices to scale by a factor s hand which is shared across the T frames. The hand vertices in camera coordinates V 3d hand are estimated as following:

V 3d hand = s hand (R hand V c hand ) + D hand . (1)
The object vertices V 3d obj are estimated as a rigid transformation of canonically oriented model vertices V c obj :

V 3d obj = s obj (R obj V c obj ) + D obj . (2)
Next, we describe the individual error terms that we minimize during fitting.

Object silhouette matching (L obj ). We use a differentiable renderer [START_REF] Kato | Neural 3D mesh renderer[END_REF] to render the object mask M obj and compare it to the reference segmentation mask Mobj . This error term is occlusion-aware as in [START_REF] Zhang | Perceiving 3D human-object spatial arrangements from a single image in the wild[END_REF]. No penalization occurs for the object silhouette being rendered in pixel regions Mhand where hand occlusions occur. We write this error as:

L obj = ||(1 -Mhand ) • (M obj -Mobj )|| 2 2 (3)
Projected hand vertices (L v2d ). We constrain the hand position by penalizing projected vertex offsets from the initial vertex pixel locations V hand 2d predicted by FrankMocap [START_REF] Rong | FrankMocap: Fast monocular 3D hand and body motion capture by regression and integration[END_REF].

To compute the current 2D vertex locations, we project the MANO [START_REF] Romero | Embodied hands: Modeling and capturing hands and bodies together[END_REF] vertices V 3D hand to the pixel plane using the camera projection operation Π. This error is written as:

L v2d = ||Π(V 3D hand ) -V2d hand || 2 2 (4)
Hand regularization (L pca ). Given that we optimize the articulated hand pose, we regularize the optimized hand pose. As in [START_REF] Boukhayma | 3D hand shape and pose from images in the wild[END_REF][START_REF] Hasson | Learning joint reconstruction of hands and manipulated objects[END_REF][START_REF] Rong | FrankMocap: Fast monocular 3D hand and body motion capture by regression and integration[END_REF], we apply a mean square error term to the PCA hand components L pca = ||θ|| 2 2 to bias the estimated hand poses towards statistically plausible configurations. Scale (L scale ). Similarly to PHOSA [START_REF] Zhang | Perceiving 3D human-object spatial arrangements from a single image in the wild[END_REF], when we allow the elements in the scene to scale, we penalize deviations from category-level average dimensions. Smoothness (L smooth ). We further leverage a simple smoothness prior over the T sampled frames

L smooth = T -1 t=1 ||V 3D t+1 -V 3D t || 2 2
which encourages minimal 3D vertex variances across neighboring frames for both hands and objects as in [START_REF] Hampali | HOnnotate: A method for 3D annotation of hand and object poses[END_REF]. Coarse interaction (L centroid ). Following [START_REF] Zhang | Perceiving 3D human-object spatial arrangements from a single image in the wild[END_REF], we penalize the squared distance between hand and object centroids when the predicted hand and object boxes overlap to encode a coarse interaction prior. As we assume the object scale to be provided, this error only impacts the rigid hand pose, effectively attracting the hand towards the interacted object. In case of multiple hands, all overlapping hand-object pairs of meshes are considered. Collision (L col ). We rely on a recent collision penalization term introduced to enforce non-interpenetration constraints between multiple persons in the context of body mesh estimation [START_REF] Jiang | Coherent reconstruction of multiple humans from a single image[END_REF]. The collision error L col is computed for each pair k, l of estimated meshes. We compute

L k,l col = i Φ k (V i l ).
Where Φ k is the negative truncated signed distance function (SDF) associated to the mesh k, Φ k (V) = max(0, -SDF (V)).

L col = k,l L k,l col (5)
This formulation allows to handle any number of visible hands and objects in the scene. Local contacts. Hands interact with objects by establishing surface contacts without interpenetration. We experiment with the hand-object heuristic introduced by [START_REF] Hasson | Learning joint reconstruction of hands and manipulated objects[END_REF]. We repurpose this loss which has been introduced in a learning framework to our optimization setup. This additional term encourages the contacts to occur at the surface of the object by penalizing the distance between hand vertices the closest object vertex for hand vertices in the object's vicinity. We refer to the supplemental material Section C for additional details.

The final objective L is composed of a weighted sum of the previously described terms, where the weights are empirically set to balance the contributions of each error term.

L = λ obj L obj + λ v2d L v2d +λ pca L pca + λ scale L scale + λ smooth L smooth +λ centroid L centroid + λ local L local + λ col L col (6)
While [START_REF] Zhang | Perceiving 3D human-object spatial arrangements from a single image in the wild[END_REF] adapt the weights for their optimization for each object categories, we fix the weight parameters empirically and keep them constant across all experiments. We refer to Section C of the supplemental material for exact weight values and additional implementation details.

Experiments

We first define the evaluation metrics (Section 4.1) and the datasets (Section 4.2) used in our experiments. Then, we provide an ablation to measure the contribution of each of our optimization objective terms (Section 4.3). We investigate the sensitivity of our approach to the quality of the 2D estimates (Section 4.4). Next, we compare our approach to the state of the art (Section 4.5). Finally, we provide qualitative results for in-the-wild examples (Section 4.6).

Metrics

The structured output for hand-object reconstruction is difficult to evaluate with a single metric. We therefore rely on multiple evaluation measures. Object metrics. We evaluate our object pose estimates by computing the average vertex distance. Common objects such as bottles and plates often present plane and revolution symmetries. To account for point matching ambiguities, we further report the standard pose estimation average closest point distance (add-s) [START_REF] Xiang | PoseCNN: A convolutional neural network for 6d object pose estimation in cluttered scenes[END_REF]. Hand metrics. We follow the standard hand pose estimation protocols [START_REF] Hampali | HOnnotate: A method for 3D annotation of hand and object poses[END_REF][START_REF] Zimmermann | FreiHAND: A dataset for markerless capture of hand pose and shape from single RGB images[END_REF] and report the procrustes-aligned hand vertex error and F-scores. We compare the hand joint predictions using average distances after scale and translation alignment. When investigating the results of the joint fitting, we additionally report the average hand vertex distances without alignment. Interaction metrics. Penetration depth (mm): We report the maximum penetration depth between the hand and the object following previous work on hand-object interactions [START_REF] Brahmbhatt | ContactPose: A dataset of grasps with object contact and hand pose[END_REF][START_REF] Hasson | Leveraging photometric consistency over time for sparsely supervised hand-object reconstruction[END_REF][START_REF] Hasson | Learning joint reconstruction of hands and manipulated objects[END_REF][START_REF] Karunratanakul | Grasping field: Learning implicit representations for human grasps[END_REF]. Contact (%): We also report the contact percentage following [START_REF] Karunratanakul | Grasping field: Learning implicit representations for human grasps[END_REF]. When ground truth contact binary labels are available, we report contact accuracy, as we further detail in the supplemental material Section C .

Datasets

HO-3D [START_REF] Hampali | HOnnotate: A method for 3D annotation of hand and object poses[END_REF] is the reference dataset which provides accurate hand-object annotations during interaction for markerless RGB images. The users manipulate 10 objects from the YCB [START_REF] Berk C ¸alli | The ycb object and model set: Towards common benchmarks for manipulation research[END_REF] dataset, for which the CAD models are provided. The ground-truth annotations are obtained by fitting the hand and object models to RGB-D evidence which assumes limited hand motion. We present results on the standard test set composed of 13 videos with a total of 11525 frames depicting single-hand object manipulations. Core50 [START_REF] Lomonaco | Core50: a new dataset and benchmark for continuous object recognition[END_REF] contains short sequences of unannotated images of hands manipulating 50 object instances from 10 everyday object categories such as cups, light bulbs and phones. We manually associate 26 objects with approximately matching 3D object models from the ShapeNet dataset [START_REF] Angel | ShapeNet: An information-rich 3D model repository[END_REF]. We further annotate hands being left or right for each of the 11 video sequences available for each object, resulting in 286 video clips and 86k frames. Epic Kitchens [START_REF] Damen | Scaling egocentric vision: The epic-kitchens dataset[END_REF] is an unscripted dataset which has been collected without imposing constraints or equipment beyond a head-mounted camera. In contrast to existing datasets for which 3D information is available, it therefore presents natural hand-object interactions. This dataset is however densely annotated with action labels which include the category of the object of interest. We focus on a subset of common object categories: cups, plates, cans, phones and bottles which are involved in a total of 3456 action video clips. For each object category, we associate an object model from the CAD ShapeNet [START_REF] Angel | ShapeNet: An information-rich 3D model repository[END_REF] database. We assume that the target manipulated object is the one with the longest track in the associated action clip.

Contribution of error terms in fitting

As explained in Section 3, our method introduces several error terms which determine the final reconstruction quality. We evaluate the contribution of the main error terms on the HO-3D benchmark [START_REF] Hampali | HOnnotate: A method for 3D annotation of hand and object poses[END_REF] in Table 1. We validate that our joint reconstruction outperforms the naive composition baseline, which is obtained by separately fitting the object to the occlusion-aware object mask and the hand using the hand-specific terms L v2d and L pca . When fitted independently, the scale-depth ambiguity prevents an accurate estimate of the hand distance. As we use the ground-truth object model, the 3D object pose can be estimated without ambiguity using the camera intrinsic parameters. Joint fitting improves the 3D pose estimates using both smoothness and interaction priors. We observe that the coarse interaction prior is critical towards improving the absolute hand pose. When removing this error term, the hand pose error increases two-fold from 8.9 to 17.1cm. We observe that the temporal smoothness term, while simple, provides a strong improvement to both the hand and object pose estimates. Leveraging information across neighboring video frames reduces the errors by 25% and 38% for the hand and object, respectively. While the local interaction and collision penalization terms only marginally change the hand and object reconstruction scores, their impact can be quan-Figure 3: Effect of error terms: Qualitative analysis showing the effects of the various error terms for the hand-object reconstruction accuracy on the HO-3D dataset. We highlight visual evidence of local corrections attributed to the local interaction [START_REF] Hasson | Learning joint reconstruction of hands and manipulated objects[END_REF] and collision [START_REF] Jiang | Coherent reconstruction of multiple humans from a single image[END_REF] terms. titatively observed in the interaction metrics. The collision penalization terms reduce the average penetration depth by a large factor (10.2mm vs 2.4mm). The local interaction term [START_REF] Hasson | Learning joint reconstruction of hands and manipulated objects[END_REF] reduces both the interpenetration depth and the contact percentage, which is defined as either exact surface contact or interpenetration between the hand and the object. Qualitatively, we observe that this term produces local corrections in the vicinity of estimated contact points. Including all error terms results in more plausible grasps, which we illustrate in Figure 3 qualitatively.

While ground truth 3D poses are hard to annotate for generic videos with hand-object manipulations, interaction metrics such as penetration depth can be directly computed from the predicted reconstructions. As the Core50 [START_REF] Lomonaco | Core50: a new dataset and benchmark for continuous object recognition[END_REF] dataset presents only videos in which the object is actively manipulated by the hand, we can additionally report contact accuracy as proxies to evaluate the quality of the reconstructed grasps. We report these two metrics on the Core50 dataset in Table 2 and confirm the benefit from our joint fitting approach. Using joint fitting significantly increases the contact accuracy from 7.3% to 89.5%, while only increasing the average penetration depth by 0.6mm.

Sensitivity to estimated 2D evidence

Our method makes use of generic models for detection and mask estimation and would directly benefit from more Figure 4: Sensitivity to 2D detections: Dependence of our 3D reconstruction on the accuracy of the 2D evidence by running our method with ground truth (GT) hand and object detections and ground truth object masks for the HO-3D dataset [START_REF] Hampali | HOnnotate: A method for 3D annotation of hand and object poses[END_REF]. accurate detection, 2D hand pose estimation and instance segmentation models. The reliance of our method on 2D cues therefore allows it to benefit from additional efforts in 2D image annotation which is simpler compared to 3D annotation in practice. We investigate the dependence of our method on the quality of the available 2D evidence. To investigate the expected improvements our method could gain from stronger object detections, instead of using noisy detections, we use the hand and object ground truth bounding boxes provided for the HO-3D test set. We observe in Figure 4 the improvements we obtain from using the groundtruth detections. Both hands and objects benefit from more accurate detections, improving by 2cm when compared to the tracking-by-detection estimates. To investigate the errors which come from using noisy approximate instance masks, we render the object and hand ground truth masks and use them to guide our optimization. By relying on 2D information, our approach suffers from limitations such as depth ambiguities which can result from fitting to image segmentation masks. Object asymmetries which rely on color information can also be hard to resolved during fitting. We observe that using ground truth hand and object masks allows to further decrease the 3D pose errors. We note that the object error decreases to 2cm and below 1cm when comparing distances to closest points. When the object model is available, our joint fitting method produces highly accurate object poses in the presence of accurate 2D evidence.

State-of-the-art comparison

Recent efforts for joint hand-object pose estimation in camera space [START_REF] Hasson | Leveraging photometric consistency over time for sparsely supervised hand-object reconstruction[END_REF][START_REF] Hasson | Learning joint reconstruction of hands and manipulated objects[END_REF][START_REF] Bugra Tekin | H+O: Unified egocentric recognition of 3D hand-object poses and interactions[END_REF] have focused on direct bottom-up regression of 3D poses. We compare the performance of our fitting approach to recent learning-based methods for joint hand-object reconstruction [START_REF] Hasson | Leveraging photometric consistency over time for sparsely supervised hand-object reconstruction[END_REF][START_REF] Liu | Semi-supervised 3d hand-object poses estimation with interactions in time[END_REF] in Table 3.

While these learnt methods produce more accurate hand predictions, their object predictions are instance specific. As a direct consequence, the methods do not generalize to Table 3: State-of-the-art-comparison: We compare the hand performance of the single-view pose estimation baselines. Note that the reported results for [START_REF] Hampali | HOnnotate: A method for 3D annotation of hand and object poses[END_REF] output hand meshes only. Hasson 2020 [START_REF] Hasson | Leveraging photometric consistency over time for sparsely supervised hand-object reconstruction[END_REF], Liu 2021 [START_REF] Liu | Semi-supervised 3d hand-object poses estimation with interactions in time[END_REF] and our method predict the hand-object meshes jointly. All methods are trained only on the real images from the HO-3D training split and evaluated on the official test split through an online submission 1 . The hand mesh error is reported after procrustes alignment.

new objects at test time. In contrast, our generic fitting method performs equally well across the seen and unseen objects of the HO-3D test split, see Table 4.

In-the-wild 3D hand-object pose estimation

We test the limits of our approach and showcase the strength of our method by comparing qualitatively to a model trained for joint hand-object reconstruction [START_REF] Hasson | Learning joint reconstruction of hands and manipulated objects[END_REF]. Their model estimates the shape of the object by deforming a sphere and therefore does not depend on known ob- Table 4: Unseen objects: Vertex errors (cm) for estimated hand and object meshes. Compared to [START_REF] Hasson | Leveraging photometric consistency over time for sparsely supervised hand-object reconstruction[END_REF], our method performs similarly across seen and unseen objects. ject models. However, given this object topology restriction, this method has limited expressivity. It can only capture a subset of all object shapes which excludes everyday objects such as mugs or cups, see Fig. 6. In comparison, while our method makes stronger assumptions by relying on an approximate object model, it is applicable to any everyday objects for which an approximate mesh can be retrieved without further limitations. Additionally, while the manipulation reconstruction from [START_REF] Hasson | Learning joint reconstruction of hands and manipulated objects[END_REF] estimates the grasp relative to the root joint of the hand, our method outputs image-aligned predictions.

We further show that our method can be applied to the challenging Epic-Kitchens dataset [START_REF] Damen | Scaling egocentric vision: The epic-kitchens dataset[END_REF] which presents natural manipulation of common objects, see Fig. 5. Note that our objective [START_REF] Brahmbhatt | ContactPose: A dataset of grasps with object contact and hand pose[END_REF] is not restricted to a single hand-object pair and naturally generalizes to multiple hands and objects. To handle scenes with two hands in [START_REF] Damen | Scaling egocentric vision: The epic-kitchens dataset[END_REF] we optimize [START_REF] Brahmbhatt | ContactPose: A dataset of grasps with object contact and hand pose[END_REF] with pairwise losses defined for both detected hands and the detected object. We show results of two-hand manipulations which represent the majority of examples in the target dataset. While we observe cases of depth ambiguity, especially with almost plannar objects such as plates, we show that our method can recover plausible reconstructions for several object categories across a variety of hand poses.

Figure 5: In-the-wild reconstructions: Our results on natural hand-object manipulations of the Epic-Kitchens dataset [START_REF] Damen | Scaling egocentric vision: The epic-kitchens dataset[END_REF]. We present several success and failures of our method on the challenging Epic-Kitchens dataset. We highlight typical failure modes for our method, in particular, object orientation errors resulting from depth ambiguity. We observe that our fitting method recovers plausible interactions across different object categories and hand-object configurations.

Conclusions

We present a robust approach for fitting 3D hand-object configurations to monocular RGB videos. Our method builds on estimates from neural network models for detection, object segmentation and 3D hand pose estimation trained with full supervision. Due to the lack of supervision at similar scale for 3D hand-object interactions, we opt for a fitting-based approach and demonstrate advantages on several datasets. A key limitation of current methods that estimate 6DOF object pose is the reliance on known object models. Future work will consider automatic object recovery to fully automate the hand-object reconstruction process.

Figure A.2: Reconstructions obtained from independent composition and joint fitting on HO-3D [START_REF] Hampali | HOnnotate: A method for 3D annotation of hand and object poses[END_REF] dataset. We observe that enforcing hand-object interaction constraints allows our method to recover plausible grasps and significantly improves upon the naive independent composition baseline.

Detections

Hand IoU Object IoU Ground Truth boxes 0.63 0.74 Predicted boxes [START_REF] Shan | Understanding human hands in contact at internet scale[END_REF] 0.61 0.66 meshes. As the hand ground truth meshes are unavailable for the test split, we report results on the HO-3D train split. As described in the method section of the main paper, we select the mask produced for the most confident class prediction for the object and the Person class for the hand. To investigate the segmentation error which is due to bounding box errors, we report IoU scores for the PointRend segmentation head both using the tracked and the ground truth detections in Table . A.2. As expected, segmentation masks are more precise when using ground truth object boxes, especially for objects. Hence, our method would benefit from more robust and accurate detections. We illustrate success and failures for 2D mask extraction in

C. Implementation details.

We describe details of our fitting implementation. Our code and manually collected annotations will be made available upon publication.

Optimization procedure. Two of our error terms, the collision and fine-grained interactions act locally and require a reasonable initial configuration to provide meaningful gradients. We therefore obtain the final hand-object reconstructions by performing two consecutive steps. We first recover a coarse pose for the hand and the object using the data and coarse interaction and regularization terms (L obj , L v2d , L pca , L scale , L smooth , L centroid ). We then refine the optimized parameters using the full set of error terms. Both steps are run for 200 optimization steps using the Adam [START_REF] Diederick | Adam: A method for stochastic optimization[END_REF] optimizer. We use different learning rates for various optimized parameters in order to distribute the updates between articulated and rigid pose updates. We use a fixed learning rate of 0.1 for the PCA components of the MANO [START_REF] Romero | Embodied hands: Modeling and capturing hands and bodies together[END_REF] articulated hand model and the continuous rotation parameters [START_REF] Zhou | On the continuity of rotation representations in neural networks[END_REF]. For the translation and scale updates we use a learning rate of 0.01.

Error terms weighting. We use the same weights to balance the various error terms across all reconstructed datasets. We empirically fix the weights to the following values. Data weights: λ obj : 1, λ v2d : 50 Regularization weights: λ pca : 0.004, λ scale : 0.001, λ smooth : 2000 Interaction weights: λ centroid : 1, λ local : 1, λ col : 0.001 Mesh preprocessing. In order to reason about collisions and compute penetration depths, we preprocess the meshes to make them watertight. We further uniformly downscale the mesh to speed-up the rendering which is required to compute the object silhouette loss L obj . We compute watertight meshes for all ShapeNet models [START_REF] Angel | ShapeNet: An information-rich 3D model repository[END_REF] using Manifold-Plus [START_REF] Huang | Manifoldplus: A robust and scalable watertight manifold surface generation method for triangle soups[END_REF]. We rely on ACVD [START_REF] Valette | Generic remeshing of 3d triangular meshes with metricdependent discrete voronoi diagrams[END_REF] to uniformly resample the object meshes, and set the target number of vertices to 1000 which we observe empirically to provide a satisfactory trade-off mesh approximation and rendering speed.

Collision loss. Unlike PHOSA [START_REF] Zhang | Perceiving 3D human-object spatial arrangements from a single image in the wild[END_REF] which penalizes collisions using a local loss [START_REF] Ballan | Motion capture of hands in action using discriminative salient points[END_REF] which acts at the mesh surfaces, we use an implementation which relies on Signed Distance Fields (SDF) [START_REF] Jiang | Coherent reconstruction of multiple humans from a single image[END_REF]. We follow the original implementation, using a grid size of 32 for the signed distance field.

Local interaction heuristic. We use the loss term introduced by [START_REF] Hasson | Learning joint reconstruction of hands and manipulated objects[END_REF], which showed improved qualitative results when used in the context of fully-supervised learning. We accelerate computations by relying on the SDF computation from [START_REF] Jiang | Coherent reconstruction of multiple humans from a single image[END_REF] to perform collision checking. We use the same parameters as in the original implementation for the attractive and repulsive terms, 1cm and 2cm respectively. Contact accuracy. We measure contact following [START_REF] Karunratanakul | Grasping field: Learning implicit representations for human grasps[END_REF], where contact is defined as negative hand vertex values in the object SDF.

Runtime. As we rely on learnt models for detection, segmentation and initial hand pose estimation, pre-processing 10 frames of dimension 640*480 takes less than a second on a Tesla V100 Nvidia GPU. The initial object fitting stage for 50 random object pose initialization takes around 10 second per frame. The joint fitting runs at 14 iterations per iteration in absence of collision and contact terms. With the finegrained interaction constraints, the iteration time increases up to 4 iterations per second. In total, complete fitting of a 10 frame video clip takes between 2 and 3 minutes.

Figure 2 :

 2 Figure 2: Joint hand-object fitting: We independently initialize the hand and object poses based on 2D detections and segmentations. We refine this configuration with interaction-based constraints to obtain our final joint fitting.

Figure 6 :

 6 Figure 6: Comparison with[START_REF] Hasson | Learning joint reconstruction of hands and manipulated objects[END_REF]: Qualitative comparison of our fits to the ObMan-trained model[START_REF] Hasson | Learning joint reconstruction of hands and manipulated objects[END_REF] estimations on the Core50 dataset.While our model requires an approximate mesh to be provided, it generalizes to objects of arbitrary topology.

  Figure A.1.

Table 1 :

 1 Contribution of error terms: We show benefits of the joint modeling for hand-object interactions by the increased reconstruction accuracy when compared to independent hand and object composition on the HO-3D[START_REF] Hampali | HOnnotate: A method for 3D annotation of hand and object poses[END_REF] dataset. Our smoothness and interaction terms impose additional constraints which improve the final hand-object pose reconstructions.

				Hand		Object		Interaction
				vertex mean	mepe	vertex mean	add-s pen. depth contact
			distance (cm)↓ aligned (cm) ↓ distance (cm) ↓ (cm) ↓	(mm) ↓	%
	indep. composition		26.2	5.2	12.1	7.7	3.2	25.8
	joint fitting			8.6	5.4	8.0	3.8	2.8	77.5
	w/out local interactions	8.5	5.4	8.0	3.8	2.4	72.3
	w/out collision		8.9	5.4	8.0	3.8	10.2	80.5
	w/out coarse interaction	17.1	5.3	8.1	3.8	1.9	59.4
	w/out smoothness		11.4	5.6	12.8	8.3	3.0	79.1
	Dataset Contact Accuracy (%) ↑ Pen. Depth (mm) ↓
	independent	joint	independent joint
	Core50	7.3	89.5	0.6	1.2

Table 2 :

 2 Results on Core50: Interaction errors for handobject fits obtained on the Core50 dataset. We observe significantly improved contact accuracy with joint fitting over independent fits at the expense of a minor cost of a 0.6mm increase in penetration.

		mesh	F-score	F-score	generalizes to
	Method	error ↓ @5mm ↑ F@15mm ↑ unseen objects
	Liu 2021 [41]	0.95	0.96	0.53
	Hampali 2020 [16]	1.06	0.51	0.94
	Hasson 2020 [18]	1.14	0.42	0.93
	Joint fitting	1.47	0.39	0.88

Table A .

 A

2: Segmentation accuracy: Hand and object segmentation IoU for the PointRend mask estimates on the HO-3D dataset

[START_REF] Hampali | HOnnotate: A method for 3D annotation of hand and object poses[END_REF]

.

https://competitions.codalab.org/competitions/22485

Acknowledgments. This work was funded in part by the MSR-Inria joint lab, the French government under management of Agence Nationale de la Recherche as part of the "Investissements d'avenir" program, reference ANR19-P3IA-0001 (PRAIRIE 3IA Institute), by Louis Vuitton ENS Chair on Artificial Intelligence, as well as gifts from Google and Adobe. This work was granted access to the HPC resources of IDRIS under the allocation 2020-AD011012004R1 made by GENCI.

APPENDIX

We first present additional qualitative results for the HO-3D [START_REF] Hampali | HOnnotate: A method for 3D annotation of hand and object poses[END_REF] dataset (Section A). Next, we analyze the 2D detection and segmentation performance of the general-purpose estimators on the HO-3D [START_REF] Hampali | HOnnotate: A method for 3D annotation of hand and object poses[END_REF] dataset (Section B). Finally, we provide additional implementation details for our joint hand-object optimization (Section C).

A. Qualitative results for HO-3D dataset

In Figure A.2, we compare qualitatively on the HO-3D test set our joint fitting method with independent composition, which results from fitting the object and the hand without interaction constraints independently on each frame. We observe that the joint fitting allows to recover improved configurations across various test objects and consistently improves upon independent composition.

B. 2D evidence accuracy.

B.1. Evaluation of hand and object detection

We assess the performance of our procedure to recover the detections and segmentation masks for the hands and manipulated objects.

We use the 100 Days of Hands (100DOH) hand-object detector [START_REF] Shan | Understanding human hands in contact at internet scale[END_REF] to predict hand and object bounding boxes. Following [START_REF] Shan | Understanding human hands in contact at internet scale[END_REF], we evaluate hand detection accuracy by reporting average precision (AP) for predicted hand and object boxes and set the intersection over union (IoU) threshold to 0.5 on the HO-3D [START_REF] Hampali | HOnnotate: A method for 3D annotation of hand and object poses[END_REF] train split. To compute ground truth detections, we compute the tight bounding box around pixel coordinates of the projected vertices for the objects and the hand. For the hands, we post-process the detections to recover square boxes. In Table . A.1 we report the AP for hands and objects before and after tracking. We observe that tracking [START_REF] Muron | [END_REF] effectively allows us to recover valid detections. The use of tracking improves AP by 24% and 12% for hands and objects respectively.

B.2. Evaluation of hand and object segmentation

We evaluate the accuracy of our instance mask estimation procedure. We report the IoU between the PointRend [START_REF] Kirillov | PointRend: Image segmentation as rendering[END_REF] outputs and the ground truth segmentation masks, which we obtain by rendering the ground truth