Chenxi Li

Boyuan Zhang

Yongqiang Duan

Yang Li

Zuochang Ye

Weifeng Liu

Dingwen Tao

Zhou Jin

MASC: A Memory-Efficient Adjoint Sensitivity Analysis through Compression Using Novel Spatiotemporal Prediction

Adjoint sensitivity analysis is critical in modern integrated circuit design and verification, but its computational intensity grows significantly with the size of the circuit, the number of objective functions, and the accumulation of time points. This growth can impede its wider application. The intimate link between the forward integration in transient analysis and the reverse integration in adjoint sensitivity analysis allows for the retention of Jacobian matrices from transient analysis, thereby speeding up sensitivity analysis. However, Jacobian matrices across multiple timesteps are often so large that they cannot be stored in memory during the forward integration process, necessitating disk storage and incurring significant I/O overhead. To address this, we develop a memoryefficient sensitivity analysis method that utilizes data compression to minimize memory overhead during simulation and enhance analysis efficiency. Our compression method can efficiently compress the sparse tensor that contains the Jacobian matrices over time by exploiting the spatiotemporal characteristics of the data and circuit attributes. It also introduces a shared-indices technique, a cutting-edge spatiotemporal prediction model, and robust residual encoding. We evaluate our compression method on 7 datasets from real-world simulations and demonstrate that it can reduce the memory requirements for storing Jacobian matrices by more than 16× on average, which is significantly more efficient than other state-of-the-art compression techniques.

INTRODUCTION

In contemporary integrated circuit design, transient sensitivity analysis plays a vital role in various domains, including circuit optimization [START_REF] Agrawal | Circuit optimization using scale based sensitivities[END_REF], performance modeling [START_REF] Stevanovic | Quadratic backward propagation of variance for nonlinear statistical circuit modeling[END_REF], and yield estimation [START_REF] Gu | An efficient, fully nonlinear, variability-aware non-monte-carlo yield estimation procedure with applications to sram cells and ring oscillators[END_REF]. The working performance of a circuit is influenced by numerous critical parameters, such as transistor dimensions, parasitic resistances, capacitances, and more. Sensitivity analysis is a valuable tool for examining the impact of these factors on system output. However, the conventional direct method falls short when dealing with a large number of parameters [START_REF] Sagan | Transient adjoint dae sensitivities: a complete, rigorous, and numerically accurate formulation[END_REF]; as a result, the adjoint method has become the standard in modern circuit simulations.

In large-scale circuit simulations, transient adjoint sensitivity analysis often results in significant computational overhead. This arises from the discretization of the dynamic system through numerical integration methods, which typically require solving large-scale differential equations at numerous time points -a process that can be quite time-consuming. This challenge is particularly prevalent in modern integrated circuit design. A major portion of the time spent in the adjoint method is devoted to computing Jacobian matrices [START_REF] Hu | Adjoint transient sensitivity analysis for objective functions associated to many time points[END_REF]. Therefore, any approach that enables the adjoint method to bypass these computations could lead to considerable time savings.

Fortunately, the adjoint system and the original system usually share the same Jacobian matrices. By storing and reusing these matrices instead of recomputing them, significant reductions in simulation time can be achieved.

Nonetheless, efforts to speed up sensitivity analysis by retaining Jacobians during transient analysis result in excessive memory overhead. Consequently, an alternative approach is to save this data on disk storage. However, this introduces frequent file I/O operations and data retrieval processes, leading to considerable time overhead. Such an approach significantly diminishes the simulator's efficiency while also placing substantial demands on disk storage.

To address the aforementioned issues, the introduction of inmemory compression is crucial. However, existing compression methods do not adequately exploit the intrinsic characteristics of data during the simulation process, resulting in suboptimal compression ratios and speeds. It is noteworthy that these matrices form a tensor along the time dimension, characterized by the following features: Firstly, each matrix is stored in a sparse storage format, with row indices and column indices represented as integer arrays, and non-zero values as floating-point arrays. Secondly, due to the brief step lengths in numerical integration, there is a significant correlation between temporally adjacent matrices, essentially making these matrices a form of time series data. Furthermore, these Jacobians are intimately linked to the circuit's structure.

To this end, we propose a novel lossless compression method that employs spatiotemporal prediction, effectively enhancing compression ratios for Jacobians across timesteps. This algorithm skillfully utilizes the spatiotemporal attributes of Jacobian matrices, enabling efficient compression of the sparse tensor formed by these matrices. As a result, it significantly reduces memory overhead in simulations without sacrificing accuracy. To our knowledge, this represents the first instance of a data compression-based approach to adjoint sensitivity analysis. It also constitutes the first thorough exploration of sparse Jacobian matrix compression in the context of SPICE simulations. Our contributions are summarized as follows:

• We conduct a comprehensive analysis of data characteristics in circuit simulation and investigate different floating-point compression methods from various fields. • We develop a lossless floating-point compression method based on a novel spatiotemporal prediction model for Jacobian matrices. We adeptly employ the shared indices technique to address the storage and computational overhead associated with the Jacobian matrix indices. • We evaluate our compression method on 7 real-world datasets, showcasing its superior performance compared with other leading compression methods.

• We further employ the proposed compression algorithm into the sensitivity analysis in SPICE simulator, and demonstrate memory-efficient acceleration with end-to-end comparison.

The rest of this paper is organized as follows. Section 2 provides an outline of the research background, and Section 3 discusses the motivation. In Section 4, we present our compression method, followed by the MASC design in Section 5. The evaluation results are detailed in Section 6. We conclude the paper in Section 7.

BACKGROUND 2.1 ADJOINT SENSITIVITY ANALYSIS

For sensitivity analysis, the circuit we are interested in can be represented by a set of Differential Algebraic Equations (DAE):

𝒈(𝒙, 𝑡, 𝒑) = 𝑑 𝑑𝑡 𝒒(𝒙, 𝒑) + 𝒇 (𝒙, 𝒑) + 𝒃 (𝑡, 𝒑) = 0, (1)
where 𝒙 ∈ R 𝐷 is the state vector, 𝑡 is time, 𝒑 ∈ R 𝑀 is the parameter vector with respect to which we need sensitivities, 𝒒 ∈ R 𝐷 and 𝒇 ∈ R 𝐷 respectively represent dynamic and static elements, 𝒃 ∈ R 𝐷 is the input. It is common to discretize the DAE system using numerical integration methods such as backward Euler, and then employ the Newton-Raphson iteration to solve at each timestep. The objective function is typically a function of the state vector ì 𝑥 at one or multiple time points:

𝑶 = 𝜻 (𝒙 0 , 𝒙 1 , • • • , 𝒙 𝑛 ).
(2) Then the sensitivity is the derivative of the objective function with respect to the system parameters p:

𝑑𝑶 𝑑𝒑 = 𝑛 ∑ 𝑖=0 𝑑𝑶 𝑑𝒙 𝑖 𝜕𝒙 𝜕𝒑 𝑖 . ( 3
)
Adjoint method is shown as:

𝑑𝑂 𝑑𝑝 = 𝑛 ∑ 𝑖=0 𝑛 ∑ 𝑘=𝑖 ( d𝑶 d𝒙 ) 𝑘 ( 𝜕𝒈 𝜕𝒙 ) -1 𝑘 𝑖 𝑗=𝑘 -1 ( 1
ℎ 𝑡 𝜕𝒒 𝜕𝒙 ) 𝑗 ( 𝜕𝒈 𝜕𝒙 ) -1 𝑗 ( 𝜕𝝋 𝜕𝒑 ) 𝑖 . ( 4
)
It can be considered as a backward propagation of transient analysis, with ℎ 𝑡 as the step size for numerical integration and 𝝋 representing terms related to the system parameter 𝒑:

𝜕𝝋 𝜕𝒑 𝑖+1 = 1 ℎ 𝑡 [( 𝜕𝒒 𝜕𝒑 ) 𝑖+1 -( 𝜕𝒒 𝜕𝒑 ) 𝑖 ] + 𝜕𝒇 𝜕𝒑 𝑖+1 + 𝜕𝒃 𝜕𝒑 𝑖+1 (5)
Based on (4), it is evident that during the adjoint process, repetitive computations of Jacobian matrices at multiple time points are required, leading to a significant additional time overhead.

FLOATING-POINT DATA COMPRESSION

Jacobian matrices, which are usually sparse, are often stored in Compressed Sparse Row (CSR) format, which utilizes integer indices and floating-point non-zero values. To further reduce the size of these CSR sparse matrices, compression techniques can be applied to both the non-zero floating-point values and the integer indices. General-purpose lossless compression can be divided into two main categories: (1) Dictionary-based compression, which aims to reduce spatial redundancy in the input data stream. Examples include LZ77 and LZW . LZ77 is designed to efficiently encode locally repeated sequences, while LZW assembles a lexicon of recurring data strings, allowing for the substitution of long repeated patterns with shorter dictionary indices. ( 2) Entropy-based compression, which utilizes the entropy/frequency information of the input for encoding. This category includes methods such as Asymmetric Numeral Systems (ANS) encoding [START_REF] Duda | Asymmetric numeral systems: entropy coding combining speed of huffman coding with compression rate of arithmetic coding[END_REF] and Huffman encoding . ANS efficiently compresses data by assigning shorter codes to more frequent symbols and longer codes to less frequent ones, while Huffman encoding uses a different encoding strategy which is building a Huffman tree based on the frequency to generate the prefix code. While Huffman encoding is simpler and quite effective for many applications, ANS encoding offers higher efficiency and adaptability at the cost of increased complexity. Several significant endeavors, including FPZIP [START_REF] Lindstrom | Fast and efficient compression of floating-point data[END_REF], NDZIP [START_REF] Knorr | ndzip: A high-throughput parallel lossless compressor for scientific data[END_REF], and ZFP [START_REF] Lindstrom | Fixed-rate compressed floating-point arrays[END_REF], have proposed new general-purpose lossless compressors specifically for floating-point data. Each algorithm employs distinct methodologies to achieve decorrelation of floating-point data, which is a prerequisite step before the application of various encoding techniques, including the aforementioned ones or alternative, more straightforward encoding methods. However, these compressors do not consider the specific characteristics of the data such as Jacobian matrices during decorrelation.

MOTIVATION 3.1 TIME-CONSUMING COMPUTATION

Table 1 presents a time comparison between transient analysis and ajoint sensitivity analysis for circuits of varying scales. This comparison is made under different numbers of objective functions, given parameters, and at various time points. Due to the impact of these factors, the time overhead associated with sensitivity analysis is substantial, often being several times or even tens of times that of transient simulation, which is unacceptable. Furthermore, in adjoint sensitivity analysis, the computation of the Jacobian matrix is notably time-consuming. For example, as shown in Table 1, the time spent on calculating the Jacobian matrix accounts for about 50%, and in some cases, it can go as high as about 65%. If the calculation of the Jacobian matrix can be avoided in the sensitivity analysis process, the efficiency of the sensitivity analysis can be significantly enhanced.

C H IP _ 0 1 C H IP _ 0 2 C H IP _ 0 3 C H IP _ 0 4 C H IP _ 0 5 C H IP _ 0 6 C H IP _ 0 7 C H IP _ 0 8 C H IP _ 0 9 r a m 2 k s m u lt 2 0 R C _ 0 1 R C _

Spatiotemporal Prediction

HIGH MEMORY COST

By reusing Jacobian from transient analysis, we can significantly accelerate the sensitivity analysis process. However, this strategy leads to considerable memory overhead. As illustrated in Figure 1, in adjoint sensitivity analysis, the storage overhead for Jacobians can surpass several hundred gigabytes as the scale of the circuit increases. Consequently, developing a compression method that offers a high compression ratio for Jacobians is essential to effectively reduce the memory burden during the simulation process.

Furthermore, to enhance the accuracy of simulation results, it is preferable to employ efficient lossless compression during the simulation process rather than lossy compression. The use of lossy compression could lead to significant cumulative errors.

SPATIOTEMPORAL PREDICTION BASED COMPRESSION FRAMEWORK

To minimize memory overhead during simulation, we propose an innovative approach for effectively compressing sparse Jacobians. The complete workflow of our compression method is depicted in Figure 2. We employ sparse storage formats, such as CSR, to store the Jacobian matrix. Considering that the values in the index array are integers and the non-zero element array comprises floatingpoint numbers, we discuss their compression separately.

SHARED INDICES

First, let us briefly discuss the compression and storage of indices. By observing the stamping process of the Jacobian matrix, we can notice that the structure of all matrices is identical. Therefore, we have ingeniously employed a technique called "Shared Indices" to effectively reduce the computational and storage overhead of the index arrays. Since all Jacobian matrices share a common set of row and column index, we place them in heap memory to ensure a longer lifecycle. If further compressing index arrays is desired, delta encoding can first be utilized to convert large integers into smaller ones, and then variable-length encoding can be applied to achieve efficient compression of indices.

SPATIOTEMPORAL PREDICTION MODEL

The primary challenge in compressing Jacobians lies in the efficient compression of non-zero element arrays. In this section, we introduce an efficient lossless floating-point compression method based on a novel hybrid prediction model to address this challenge.

Temporal Prediction Model. Due to the significant temporal correlations present in Jacobian matrices, our predictor employs the temporal prediction model to effectively capture and address these temporal dependencies.

In time-series compression, the typical approach is to compute the predictive value using historical data points [START_REF] Liakos | Chimp: efficient lossless floating point compression for time series databases[END_REF]. We expand this concept to the spatial dimension by employing a neighboring time point matrix to predict the current matrix 𝑴 𝑡 , where M𝑡 = 𝑴 𝑡 +1 . However, using multiple adjacent matrices to predict the current matrix is not feasible, as storing the historical information of several matrices would result in a substantial increase in memory overhead. Therefore, during the forward integration process, we store the Jacobian matrix at time 𝑡 𝑛 for use at time point 𝑡 𝑛+1 , relying solely on Jacobian matrices from adjacent time points for predictive modeling. Subsequent difference computation can eliminate the data redundancy introduced by linear elements.

Spatial Prediction Model. In the spatial dimension, we establish the prediction model, termed the "stamping-based prediction model", aligning with the Jacobian matrix construction methodolog, i.e. MNA, as illustrated in Figure 3(a). The Jacobian matrix is essentially an aggregate of the stamps from all the elements in the circuit [START_REF] Roychowdhury | Numerical Simulation and Modelling of Electronic and Biochemical Systems[END_REF]. A typical matrix stamp can be characterized by the set {𝑆 (𝑖, 𝑖), 𝑆 (𝑖, 𝑗), 𝑆 ( 𝑗, 𝑖), 𝑆 ( 𝑗, 𝑗)}. The four elements in this set are derived from the same element's contribution to the matrix, demonstrating a significant interdependence among them. For example, in linear resistors and capacitors, there exists a relationship such that 𝑆 (𝑖, 𝑖) = 𝑆 ( 𝑗, 𝑗) = -𝑆 (𝑖, 𝑗) = -𝑆 ( 𝑗, 𝑖).

Similarly, we examine four non-zero elements at corresponding positions in the matrix, forming a pattern described by the set {𝑉 (𝑖, 𝑖), 𝑉 (𝑖, 𝑗), 𝑉 ( 𝑗, 𝑖), 𝑉 ( 𝑗, 𝑗)}. One of the values in this set can be predicted using the other values. The design of our prediction model is structured as follows:

         V𝑝 (𝑖, 𝑗) = 𝑒𝑣𝑎𝑙 ( V ( 𝑗, 𝑖), -V (𝑖, 𝑖), -V ( 𝑗, 𝑗)) V𝑝 ( 𝑗, 𝑖) = 𝑒𝑣𝑎𝑙 (-V (𝑖, 𝑖), -V ( 𝑗, 𝑗)) V𝑝 ( 𝑗, 𝑗) = V (𝑖, 𝑖) (6)
Due to the unique nature of the values in the matrix stamp of different elements, we do not employ the Lorenz predictor scheme, in contrast to the approaches used in NDZIP and FPZIP. Instead, we independently evaluate the similarity between the true value and the other values in the matrix stamp. We use an evaluation function to select the value that is closest to the true value as the prediction result for this model. Notably, the non-zero elements on the main diagonal of the Jacobian matrix in simulations often have sign bits opposite to those in other regions. To capitalize on this, we invert the sign bits of the elements on the main diagonal, leading to a higher compression ratio during the encoding phase. Figure 3(b)(c)(d) outlines the workflow of our predictor. A more detailed explanation will be provided in the next section.

Markov predictor. To minimize the time overhead involved in selecting the best-fit prediction model, we innovatively employ Markov models. This approach is based on our observation of regularity in the selection of prediction models, due to strong spatiotemporal dependencies in the data. Specifically, we record the predictor's decision history during the best-fit phase in a historical table, estimate probabilities using frequency, and then utilize a Markov predictor to directly make predictions for the choice of prediction models (as shown in Figure 4). This design not only reduces the compression time overhead but also significantly lessens the extra space overhead incurred by recording the selection of prediction models, though there may be a slight decrease in prediction accuracy (see detail in Table 3).

EFFICIENT RESIDUAL CODING

Figure 5 shows the encoding process for non-zero elements. The compressor reads the actual value from the front of the compression queue and the predicted value from the hybrid predictor, then performs an XOR operation between them to generate the residual. Subsequently, the encoder encodes these residuals. Since the actual and predicted values are usually very close, the residuals often contain numerous leading and trailing zeros. Hence, effectively encoding these zero bits can result in a significant compression ratio for the floating-point data.

We perform a statistical analysis of the residual characteristics across multiple datasets and developed an efficient residual encoding scheme, as shown in Figure 5(a). Figure 5(b) displays the distribution of leading zeros in the residuals, indicating that approximately 60% of the residuals comprise 64 consecutive zero bits, this also robustly validates the effectiveness of our spatiotemporal prediction model. Consequently, we utilize just 1 bit to represent this scenario. Given that fewer than 2% of the residuals have fewer than 8 leading zeros, we use 3 bits for encoding the number of leading zeros, treating it as 0 if the count of leading zero bits is between 0 and 7. Moreover, if the meaningful bits of the current residual are within the range of the previous residual's meaningful bits, the encoding of leading zeros can be shared. Once the encoding process is completed, these bits are concatenated and stored in a buffer.

5:

if 𝑉 𝑖 ∈ 𝐿 then 6:

V (𝐿)
𝑖 ← 𝐿𝑎𝑠𝑡𝑉 𝑎𝑙𝑢𝑒.

7:

end if 8:

𝑠𝑜𝑙 = 𝑎𝑟𝑔 𝑚𝑖𝑛 𝑃 ={ (𝑇 ),(𝑆 ),(𝐿) } ( | V 𝑃 𝑖 -𝑉 𝑖 | ) 9:
if 𝑉 𝑖 ∈ 𝐷 then 10:

Use 1 bit to record the bestfit prediction model.

11:

else 12:

Use 2 bits to record the bestfit prediction model.

13:

end if

14:

else if 𝑠𝑜𝑙 = 𝑀𝑎𝑟𝑘𝑜𝑣_𝑠𝑜𝑙 then 15:

𝜋 * ← 𝑀𝑎𝑟𝑘𝑜𝑣𝑃𝑟𝑒𝑑𝑖𝑐𝑡𝑜𝑟 , 𝑠𝑜𝑙 = 𝜋 * .

16:

end if 17:

𝑅𝑒𝑠 𝑖 = V 𝑠𝑜𝑙 𝑖
⊕ 𝑉 𝑖 , encode the residual,write it into the buffer.

18: end for Overview of our proposed compression method. Finally, our complete spatiotemporal compression method is presented in Algorithm 1. Specifically, we compress 𝑴 𝑡 at the 𝑡 + 1 time point during the transient simulation process, using 𝑴 𝑡 +1 as the prediction matrix. Due to the strong symmetry of the Jacobian matrix, we divide the original matrix into three regions: the strictly upper triangular region, the strictly lower triangular region, and the diagonal region. The non-zero elements in these regions correspond to three sets 𝑈 , 𝐿, and 𝐷, respectively. Following the matrix stamp-based prediction model, we sequentially compress the floating-point numbers in these sets. To minimize bit wastage in encoding prediction results in set 𝐿 and enhance prediction accuracy, we employ a last value predictor [START_REF] Sazeides | The predictability of data values[END_REF]. This predictor selects the neighboring non-zero element in the same row as the matrix's true value for the predicted value. Ultimately, it chooses the prediction closest to the true value from multiple prediction models as the final prediction value.

ACCELERATING ADJOINT SENSITIVITY SIMULATION USING COMPRESSION

Integrating the SPICE simulation process with efficient data compression techniques substantially improves the efficiency of sensitivity analysis and adeptly tackles the issue of memory overhead. It is important to note that in scenarios with ample computational resources, the proposed design offers potential for parallel execution.

The complete simulation workflow is depicted in Algorithm 2. In the transient analysis, data compression and storage are implemented. Initially, shared indices are obtained during the calculation of the circuit's DC operating point. During transient simulation, the Jacobian matrix at 𝑡 𝑛 is stored to 𝑡 𝑛+1 , and compression of 𝑴 𝑛 occurs at 𝑡 𝑛+1 . This ensures that during the reverse integration process, data decompression is conducted in a reverse manner. Data decompression and memory release are performed during sensitivity analysis. Assuming the objective function is a function of the system's final state, we decompress 𝑴 𝑛-1 at 𝑡 𝑛 for subsequent adjoint sensitivity analysis at 𝑡 𝑛-1 . At this point, 𝑴 𝑛 can be released, thereby reducing memory overhead. Adjoint sensitivity analysis at 𝑡𝑛 .

12:

Decompress 𝑴 𝑛-1 using 𝑴𝑛 ; free memory for 𝑴𝑛 . 13: end while

EXPERIMENTAL EVALUATION

In this section, we present our evaluation results. We start by describing our experimental setup. Next, we examine the compression ratio and time of our proposed compression method. We then demonstrate the effectiveness of our prediction models and conclude by showcasing the overall end-to-end performance of MASC in sensitivity analysis simulations with integrated compression. Datasets. We evaluate the performance of our compression on seven datasets, as shown in Table 2, all generated from actual simulations. Specifically, #CirElem and #Steps represent the number of circuit elements and discrete time points, respectively; 𝑆 𝐶𝑆𝑅 denotes the space required to store all Jacobian matrices in CSR format; and 𝑆 𝑁 𝑍 indicates the space occupied by all non-zero elements in the matrices, which is the primary focus of our compression evaluation.

Baselines. We compare the performance of our proposed compressor with that of the general compressor GZIP , two advanced floating-point compressors (FPZIP [START_REF] Lindstrom | Fast and efficient compression of floating-point data[END_REF] and NDZIP [START_REF] Knorr | ndzip: A high-throughput parallel lossless compressor for scientific data[END_REF]), and the lossy compressor SpiceMate [START_REF] Li | Efficient and accuracy-ensured waveform compression for transient circuit simulation[END_REF] from the EDA domain.

COMPRESSION PERFORMANCE

First, we evaluate the performance of our proposed compression method and compare it with the aforementioned baselines. Table 3 demonstrates that our spatiotemporal compression significantly surpasses other compression methods, achieving a compression ratio 4.95× higher than SpiceMate and 2.24× higher than FPZIP. This superior performance can be attributed to the effective exploitation of the spatiotemporal characteristics of sparse tensors.

In terms of compression time, the average compression time of our spatiotemporal compression method is about 25% of that of GZIP, although it is slower than FPZIP. This slower speed is due to the matrix partitioning step in our spatiotemporal compression method. Despite this, the compression efficiency is remarkably high. Overall, the performance of our spatiotemporal compression method satisfactorily meets the demands of simulation processes.

EFFECTIVENESS OF PREDICTION MODELS

We then evaluate the effectiveness of three prediction models, as illustrated in Figure 6. Specifically, the temporal prediction model was selected with a frequency exceeding 60% in certain datasets, indicating that temporal dependencies are more significant than spatial dependencies in these datasets. In datasets exhibiting regularity in the spatial dimension, the matrix stamping-based prediction model was chosen with a frequency close to 60%, confirming the effectiveness of this model. Concurrently, the last value prediction, introduced to improve prediction accuracy, was selected with a frequency of around 10%, validating its necessity in the model.

END-TO-END PERFORMANCE OF MASC

Finally, we assess the end-to-end performance of our proposed simulator, MASC, and compare it with Xyce [START_REF] Verley | Xyce: Open source simulation for largescale circuits[END_REF]. As Figure 7 illustrates, MASC achieves approximately a 50% reduction in time overhead for sensitivity analysis compared to the original simulation Xyce. This improvement is primarily due to MASC's in-memory compression scheme, which efficiently circumvents the significant time overhead involved in repeatedly computing the Jacobian matrix during the reverse process. When comparing MASC to simulations that directly save Jacobian matrices to disk rather than using in-memory compression, MASC demonstrates 6× end-to-end performance improvement. This notable performance boost is attributed to two factors: (1) our compression method leads to over an 80% reduction in the overhead associated with storing Jacobian matrices on disk, and (2) the speed of our parallel compressor is considerably high (∼2.3 GB/s), far surpassing the SSD's bandwidth (∼0.5 GB/s), which ensures minimal additional time overhead in simulations. Note that the speed of our parallel (de)compression increases rapidly with the number of threads, peaking at around 16 threads. At this point, the parallel speed is more than 8× of the serial speed.

CONCLUSION

In this paper, we seamlessly integrate data compression into simulation processes, utilizing Jacobian matrices stored during forward integration to expedite the reverse adjoint sensitivity solving process. Concurrently, we develop a novel lossless compression method that significantly reduces memory overhead during the simulation. The time overhead of our MASC is approximately half of that with Xyce. Furthermore, our compression method exhibits remarkable enhancements compared to general compression methods and state-of-the-art floating-point compression methods, effectively minimizing memory overhead during the simulation process.

Figure 2 :

 2 Figure 2: Overall framework of sparse Jacobian matrices compression technology.

Figure 3 :

 3 Figure 3: Matrix stamp and proposed spatiotemporal prediction model.

Figure 4 :Figure 5 :

 45 Figure 4: Matrix stamp and proposed spatiotemporal prediction model.

Algorithm 1 :

 1 Compressing a Jacobian Matrix 𝑴 𝑡 . Input: 𝑴𝑡 and 𝑴 𝑡 +1 Output: Compressed 𝑴𝑡 1: Partition 𝑴𝑡 , obtain 𝑈 , 𝐿, and 𝐷, initiate compression task for these three sequentially, involving 𝑁 floating-point values. 2: for (𝑉 𝑖 , 𝑖 = 0, 1 • • • 𝑁 -1) do 3: if 𝑠𝑜𝑙 = 𝑏𝑒𝑠𝑡 𝑓 𝑖𝑡 _𝑠𝑜𝑙 then

Algorithm 2 : 1 :: else 6 : 9 :

 2169 Proposed MASC sensitivity simulation. Input: Input netlist Output: Sensitivity analysis results 𝑑𝑶 𝑑𝒙 Parse netlist and begin transient simulation. 2: while 𝑡𝑛 ≤ 𝑡 𝑓 do 3: if 𝑡𝑛 = 𝑡 0 then 4: Compute DC operating points; get shared indices; store 𝑴 0 . 5Transient analysis at 𝑡𝑛 ; compress 𝑴 𝑛-1 using 𝑴𝑛 ; store 𝑴𝑛 . End transient simulation and begin sensitivity simulation. 10: while 𝑡𝑛 ≥ 𝑡 0 do 11:

Figure 7 :

 7 Figure 7: End-to-end simulation time reduction of MASC compared to baselines.

Table 1 :

 1 Time comparison between transient analysis and sensitivity analysis and the proportion of Jacobian matrix calculation time in sensitivity analysis.

	Circuits Type #Elem #Param #Obj #Steps Tran (s) Sens (s) 𝑇𝑆𝑒𝑛𝑠/𝑇𝑇𝑟𝑎𝑛 𝑇𝐽𝑎𝑐/𝑇𝑆𝑒𝑛𝑠
	CHIP_01 BJT 40221	126	8	3738	144.6	595.6	4.1	63.3%
	CHIP_02 BJT 58946	183	12	5524	242.4	1969.3	8.1	62.4%
	CHIP_03 BJT 100445	351	21	2837	258.8	3279.1	12.7	57.9%
	CHIP_04 BJT 132097	398	27	1278	358.5	2449.0	6.8	57.4%
	CHIP_05 BJT 165669	421	32	682	89.3	1668.3	18.7	61.6%
	CHIP_06 BJT 162440	409	30	552	73.4	148.1	20.2	57.0%
	CHIP_07 BJT 226583	512	38	2625	431.5	9413.9	21.8	58.3%
	CHIP_08 BJT 289669	602	40	3521	801.6 16032.7	20.0	59.4%
	CHIP_09 BJT 316556	728	48	6678	1855.9 38887.5	21.0	60.4%
	ram2k MOS 14059	109	12	2330	172.6	438.4	2.5	65.3%
	smult20 MOS 46075	254	52	6276	867.7	5107.5	5.9	50.6%
	RC_01	RC 117843	324	42	5204	55.9	1959.5	35.1	46.1%
	RC_02	RC 153338	387	52	1106	14.7	668.3	45.4	45.9%

Row Index Shared Column Index Non-Zero Elements Arrays t Matrix For Prediction Matrix Under Compression Compressed Matrix Tensor Compression Queue Buffer Compressed 𝑴 𝟏 t Adaptive Hybrid Predictor Encoder Stack Temporal Matrix stamp Last value A. Best-Fit B. Markov 𝑉 𝑉 Model Selector Heap Residual NZ Array IEEE 754

								Bits
	𝒕 𝒏+𝟏
	𝒕 𝒏
	𝒕 𝒏-𝟏	𝑀 𝑡	𝑀 𝑡+1
	Infor
			0 20 40 60 80 100 120 140 160 180 200 220 Memory-cost (GB)	2.9 9.2	50.7	28.4 17.6 15.4	59.6	151.4	223.8	9.0	19.3 24.8 6.7
										0 2

Figure 1: Memory cost for storing Jacobians.

Compressed 𝑴 𝟎 Compressed 𝑴 𝟐 Shared

Table 2 :

 2 Detail of our test datasets.

	Dataset	#CirElem	#Steps	𝑆 𝐶𝑆𝑅 (GB)	𝑆 𝑁 𝑍 (GB)	CR (gzip)	𝑇𝑐𝑜𝑚𝑝 (gzip)
	add20	5091	42799	9.09	5.80	2.52	164.60s
	smult20	46075	9048	22.99	14.37	5.85	211.17s
	mem_plus	217431	13124	18.38	11.56	13.04	114.64s
	MOS_T5	902631	8680	40.13	24.32	2.00	736.68s
	MOS_T7	175487	25280	163.68	99.20	1.81	3076.71s
	MOS_T8	494077	12740	177.48	111.62	1.97	3366.62s
	MOS_T10	224398	20592	208.08	129.24	1.98	3827.76s
	6.1 EXPERIMENTAL SETUP
	Platform and software. Experiments are conducted on a Linux
	server with an AMD EPYC 7702 CPU at 2.0 GHz, 512 GB of RAM,
	and a Micron 5100 SSD, running Ubuntu 20.04. We implement our
	proposed compressor in C++ and develop its parallel version using
	OpenMP. It is available at [13]. We implement MASC based on Xyce
	[14], a high-performance, SPICE-compatible circuit simulator.

Table 3 :

 3 Comparison of compression ratio, compression time (in seconds), and decompression time (in seconds) for different compression methods. Statistics analysis of selection rate for three prediction models.

			FPZIP [7]				NDZIP [8]			SpiceMate [15]		MASC w/o Markov		MASC w/ Markov
	Dataset
		CR	𝑇𝑐𝑜𝑚𝑝	𝑇 𝑑𝑒𝑐𝑜𝑚𝑝	CR	𝑇𝑐𝑜𝑚𝑝	𝑇 𝑑𝑒𝑐𝑜𝑚𝑝	CR	𝑇𝑐𝑜𝑚𝑝	𝑇 𝑑𝑒𝑐𝑜𝑚𝑝	CR	𝑇𝑐𝑜𝑚𝑝	𝑇 𝑑𝑒𝑐𝑜𝑚𝑝	CR	𝑇𝑐𝑜𝑚𝑝	𝑇 𝑑𝑒𝑐𝑜𝑚𝑝
	add20	4.92	34.02	39.18	1.05	179.33	176.31	2.24	113.07	60.79	11.90	44.54	29.93	10.59	40.09	27.24
	smult20	4.66	86.97	95.85	1.04	215.22	211.95	5.01	127.17	67.56	11.07	105.96	74.46	10.04	87.83	70.74
	mem_plus	5.31	57.94	66.24	1.01	168.35	163.47	8.68	86.72	56.48	11.51	83.70	61.54	10.27	65.40	59.52
	MOS_T5	14.53	86.16	107.92	1.10	341.78	342.76	1.96	422.36	91.60	35.32	161.40	142.80	32.32	124.29	135.16
	MOS_T7	9.78	407.06	484.42	1.14	1454.23	1443.15	1.88	1740.08	444.01	20.65	739.97	594.45	19.62	643.77	563.20
	MOS_T8	4.80	646.84	726.27	1.06	1603.85	1505.99	1.97	1998.04	574.19	9.91	880.75	676.13	9.02	702.21	631.28
	MOS_T10	8.46	542.52	624.24	1.12	1845.00	1848.24	2.00	2103.76	497.17	17.19	967.68	820.08	16.16	872.85	796.54
	Average	7.49	265.93	306.30	1.07	829.68	826.64	3.39	941.60	255.97	16.79	426.28	342.77	15.43	362.34	326.24
							7HPSRUDO3UHGLFWLRQ
	6HOHFWLRQ3HUFHQWDJH						0DWUL[6WDPS3UHGLFWLRQ /DVW9DOXH3UHGLFWLRQ
		DGG	PHPBSOXV	&LUFXLWV	VPXOW	YRWHU
	smult20 mem_plus MOS_T5 MOS_T7 MOS_T8 MOS_T10 Circuits 49.1 58.2 62.0 60.4 63.4 55.7 84.8 84.5 85.3 86.7 82.6 87.2 Versus Original Simulation Figure 6: add20 40 50 60 70 80 90 100 63.7 Time Reduction(%) 84.1 Versus Simulation Saving Jacobians to Disk

ACKNOWLEDGMENTS

Corresponding authors: Zhou Jin and Dingwen Tao. This work was supported by the NSFC under Grant No. 62204265, 62234010, the National Key R&D Program of China under Grant No. 2022YFB4400400, and U.S. NSF under Grant No. 2312673, 2311876, 2247080, and 2303064.


David Rodriguez
email: mbelen.rodriguez@iisgm.com

Belén Rodríguez-Sánchez

Hugo López Fernández

Applications of MALDI- TOF Mass-Spectrometry in Clinical Microbiology Target Audience

Course Objectives

This course aims to provide participants with the tools to do their best with their MALDI instruments: methodology for new MALDI applications and analytical tools for spectra analysis. The attendees will have the opportunity to interact with leading researchers in the field of MALDI-TOF massspectrometry applications in clinical microbiology and discuss recent breakthroughs and new applications. The workshop will include theoretical presentations and practical training, and will represent an excellent chance for networking between clinical microbiologists, basic researchers, MALDI experts, bioinformaticians and others.

Online Education Course

Online Access

Links for online access will be provided to registrants prior to the commencement of the course.

The programme is specified in Central European Summer Time (CEST)

MALDI-TOF


C Weinreich

L De Oliveira

A Houdard

G Nader

Real-Time Neural Materials using Block-Compressed Features

Keywords:

Figure 1: PBR material texture set is decoded in real-time from our Block Compressed neural features (BCf) resulting in an image that is visually sharper than standard BC textures of similar resolution.

Introduction

The continuous challenge of real-time rendering systems is to improve the graphics quality while reducing both the memory † Equal Contribution, order determined by coin toss cost and the evaluation time. Recent progress in graphics hardware and rendering algorithms has gotten us closer to this goal. For instance, hardware accelerated ray-tracing [Fen22] facilitates real-time dynamic global illumination and drastically improves the quality of shadows, reflections and refractions [WNK22, OLK * 21, DNSD22]. Besides that, micropolygon-based data struc-tures [MMT23, KSW21] allows the rendering of extremely detailed and dense scenes. In terms of visual aesthetics, the Physically Based Rendering (PBR) framework has been the standard in real-time applications for well over a decade [MHH * 12,MHM * 13, HMB * 20]. In this framework, a material is composed of multiple texture layers (such as albedo, normals, metalness, etc.) with each layer serving a distinct role in accurately representing various aspects of the material's visual characteristics. Enhancing the visual quality of materials typically involves the process of either layering multiple elements [HdR23] or increasing the texture resolution, which affects their computational cost and memory footprint, respectively. Consequently, rendering hyperrealistic materials in real-time applications, such as video games, remains somewhat restricted.

Recent work has shown the potential of neural networks to model and represent material properties [SRRW21, ZZW * 21, FWH * 22, XWH * 23]. This neural approach aims at replacing traditional PBR textures with a collection of learned latent features, also known as neural textures [TZN19], alongside a neural network, usually a Multi-Layer Perceptron (MLP). In this context, the network plays a crucial role in decoding the learned information and reconstructing the original material. While neural approaches have proven successful in accelerating the rendering of complex appearances [ZRW * 23] and compressing high-resolution PBR materials [VSW * 23], its integration in consumer oriented real-time applications such as video games is not straightforward. The trained latent features have a relatively big memory footprint as they are often stored in GPU memory using an uncompressed format. Recent work by Vaidyanathan et al. [VSW * 23] proposed to overcome this issue by reducing the resolution and heavily quantize the values of these features. However, to compensate for the loss in resolution and precision, the associated decoder network is rather large and thus computationally intensive which leads slow evaluation time. As a result, achieving real-time performance remains challenging, and depends on specific hardware extensions that are supported on only the most recent high-end hardware. The method we present addresses these challenges. Our goal is to design a neural material model that can be integrated into a rendering pipeline and achieve real-time performance without having to rely on any specific hardware acceleration.

To do so, we present a framework capable of (1) learning the material information at any point and scale and (2) leveraging hardware-based Block Compression (BC) format to store the learned neural features. Our features can thus be decoded continuously at any scale in uv space, making it possible to reconstruct the material information with one sample per pixel and achieve smooth transition between various scales. This removes the need for any subsequent filtering step. Furthermore, encoding our neural features as regular block compressed textures reduces their memory footprint enabling us to increase their resolution. This, in turn, directly influences the complexity of the decoder network, making it considerably simpler and significantly reduces computational time.

The paper is organized as follows. Section 2 quickly details existing material representation and compression methods. We then present the technical aspects of our neural material learning frame-work and block based features in sections 3 and 4 respectively. In section 5 we describe how to practically use our framework to learn a standard PBR material and integrate it in a real-time rendering pipeline. Finally, we present our results in section 6 and conclude by discussing some limitations and future work in section 7.

Related Work

This section provides an overview of the various methods to represent and store materials that are relevant to our work. We first focus on the traditional texture based approaches (sec. 2.1) then highlight the more recent field of neural representation methods (sec. 2.2).

Texture Materials

The material properties of a three-dimensional object can be thought of as a multi-dimensional signal, which associates every point (u, v) on its two-dimensional surface with the parameter space of its appearance model. In a real-time rendering context, this is primarily done via texture mapping [START_REF] Akenine-Mller T | Real-Time Rendering[END_REF] where the material properties are discretely stored in a collection of textures with dimensions h × w × c. Here, h and w denote the spatial resolution of the textures, and c represents the total number of channels across all texture layers in the set. Increasing the visual fidelity of materials primarily relies on either adding more layers [HMB * 20] or increasing their resolution which has led to a significant rise in memory requirements. To address this issue, textures are stored in GPU memory in a compressed format. Due to the object's arbitrary position and orientation, material information are sampled from the textures at random locations. It is therefore necessary for the texture's compression format to allow for random-access and filtering so that the information can be decompressed at any point in uv space when needed in real-time. This makes standard image compression techniques, such as JPEG [Wal92, AVAB * 19], as well as newer neural based ones [BMS * 18, CSTK20] not suitable for this application as they require unpacking the entire image before being able to access pixel information.

For real-time rendering, block compression methods [CDF * 86, INH99] have long been the standard for compressing material textures. There are seven variants of the BC format (BC1-BC7) supported in DirectX [D3D], each designed for specific types of image data. All BC formats divide the image into block of 4x4 pixels and operate under the assumption that the colors in each block exhibits minimal variation and are evenly distributed along one or more line segments within the RGB color space. This means that each block can be represented by a very small color palette. In this context, the information in each block is encoded by storing the two endpoints of each segment and indexing each pixel according to its position in the RGB space (fig. 2). Reconstructing the pixel value is simply done by blending the two endpoints proportionally to the index value. The more recent BC6 and BC7 formats are the ones using more than one segment per block. These formats, designed for floating points and RGBA data respectively, improve the compression quality by separating the pixels inside a block into several groups, each having its own set of endpoints. The groups are chosen from a set of predefined partitions. In this case, each block stores more than one set of endpoints as well as the partition number. However, BC formats can only compress textures submitted to COMPUTER GRAPHICS Forum (2/2024).

BC Block Uncompressed Block

Figure 2: Block Compression algorithms encode a block of 4 × 4 pixels with a set of endpoints forming one or multiple line segments and index each pixel based on it's projected position in the RGB space. In the case where two or more line segments are stored, the pixels are separated into groups according to a pre-defined partition P.

with up to four channels which is not suited for high dimensional materials. To overcome this, the material data is separated into several textures that are compressed independently. ASTC [NLP * 12] is another popular block-based texture compression technique. It is more flexible than BC as it supports various block dimensions including non-square blocks and can even handle 3D textures.

Neural Materials

Over the last few years, advancement in neural rendering [TTM * 22] have shown that it is possible to represent a digital signal, such as a material M, with a neural network fη by minimizing over its weights η the following quantity:

∑ i, j ∥ fη(u i , v j ) -M i, j ∥ 2 , (1)
where M i, j is the pixel value of the target material and (u i , v j ) are their corresponding local coordinates. This is an overfitting problem where the weights of the network are optimized such that the network's output matches the target image at a given pixel. Thus, the capability of the network is key to perfectly recover the original image. For instance, simple coordinate based networks [MST * 20] are not capable of dealing with high frequency details. To improve the network's reconstruction capacity, positional encoding [TSM * 20, CRSL22] where the input (u, v) coordinates is encoded as a vector generated from a periodic function, is usually employed. However, this requires a large network to accurately reconstruct the original image which makes it not suitable for real-time evaluation. Using trainable spatial features [MESK22, CXW * 23, CLS * 23, SP23], i.e., neural textures, drastically reduces the size of the network and improves the reconstruction quality. In this setting, the input coordinates are used to sample in the neural textures using bilinear interpolation and the resulting feature vector is given as input to the neural network.

The idea of pairing a set of discrete spatial features and a neural network have proven to be popular for material representation. For instance, Rainer et al. [RJGW19] uses an encoder-decoder architecture to compress a large set of Bidirectional Texture Function layers. The encoder is trained to generate a latent code for each texel. This code is then used by the decoder, in conjunction with a light and view direction to output a single RGB value. Kuznetsov et al. [KMX * 21] combined a pyramid of neural textures with a fully connected network to learn the material properties at different scales. Their model is capable of rendering materials with intricate parallax effects on an infinite plane but fails to generalize to curved surfaces. This was later done in [KWM * 22]. By adding curvature and transparency as part of the network's input and output respectively. More recently, Zeltner et al. [ZRW * 23] have used a set of hierarchical textures and two MLPs to bake complex filmquality appearance. Here, the first network learns the material's reflectance and the seconds produces importance-sampled directions. This model is about three times faster than standard node-based multi-layers materials when integrated into a path-tracing pipeline.

Despite this progress, these methods often overlook the issue of storage size. In practice, learned neural features are stored in an uncompressed format which can be quite large to practically use in a real-time environment, especially when considering the memory capacity of mainstream hardware. Vaidyanathan et al [VSW * 23] addressed this issue and demonstrated how neural material representations can be more efficient than standard texture compression techniques at storing PBR material information. To do so, they reduce the resolution of the neural features as much as possible and aggressively quantize their values. This made it possible to compress PBR textures at very low bit-rates, up to 0.2 bits per-pixel per-channel (bppc). However, integrating this neural material decompression in current rendering pipeline and achieving real-time performance requires access to the latest high-end hardware. Moreover, since the model from [VSW * 23] only learns the material information at fixed coordinates, it is essential to couple the neural decompression with a filtering operation [FWSP23] in order to minimize flickering and aliasing artefacts. This introduces additional computational overhead as it entails decompressing multiple samples per pixel.

In the following, we introduce a novel neural material representation using Block-Compressed features (BCf) specifically designed to be integrated into a traditional real-time rendering pipeline at minimal computational overhead.

Neural Material Framework

A PBR material M is constituted of a set of properties. These properties are represented at a coordinate (u, v) by a vector of dimension c. In a standard 3D application, this data is stored in a discrete fashion through a texture of size w × h × c and mapped onto a 3D object. At render time, this information is sampled from the textures as follows:

F(M)(u, v, s) ∈ R c , ( 2
)
where F is a filtering operation and s is a scale value. The filtering is essential here. It accounts for the misalignment and the difference in area between the screen pixel and its corresponding texels, caused by the 3D object's arbitrary distance and orientation. Our neural material model aims at replacing the material textures with a set of neural features containing abstract information and a decoder network. The role of the decoder here is to reconstruct the material data at a given point and scale using the learned features. Figure 3 gives an overview of our neural material framework.

Let {T θ0 , . . . , T θn } be a set of neural features of size submitted to COMPUTER GRAPHICS Forum (2/2024). (w 0 , h 0 , d 0 ) , . . . , (wn, hn, dn) with trainable parameters {θ 0 , . . . , θn}. And let fη be a fully connected neural network of input size ∑ n i=1 d i and output size c with trainable parameters η. To reconstruct the material information at a point (u, v) with respect to a scale s, we sample each of the features, concatenate the resulting values and pass it to the neural network as follow:

G θ,η (u, v, s) = fη T θ0 (u, v, s) , . . . , T θn (u, v, s) . (3)
In this framework, learning the material M boils down to optimizing θ and η such that:

θ, η = arg min θ,η G θ,η -F(M) 2 . ( 4
)
This allows us to train a model such that it simulates a given filtering operation. In practice, we use a batched stochastic gradient descent to perform this optimization where the gradient is computed on a batch of random values of (u, v, s). This leads us to minimize the following loss:

∑ (u,v,s)∈B G θ,η (u, v, s) -F(M)(u, v, s) 2 .
(5)

Once trained, we export the learned features as textures and store the model's weights in a binary buffer. At render time, the neural textures and the weights are used to reconstruct the material information. To make neural materials more appealing, it is important to reduce their memory footprint and to make their memory size match the traditional material texture. There are two possible strategies here. The first one consists in using low bitrate and low resolution neural features [VSW * 23]. However, this approach leads to a more complex decoder architecture, making the material reconstruction more computationally expensive. The second aims at storing the neural features more efficiently by using a compression algorithm. We chose the latter and propose to store them in the BC6 format as it is designed to handle floating point data and allows for random-access. Additionally, these compressed features can be handled like any traditional texture, which considerably simplifies the neural material's decoding at rendertime. However, naively compressing the learned features at the end of the training will lead to artifacts that severely impact the visual quality of the reconstructed material (fig. 4). In order to tackle this issue, we propose a specific block compressed neural feature parameterization that is compatible with the BC6 format.

Block Based Neural Features

In this section, we detail our block-based neural features. The goal here is to be able to store the trained neural features as BC6 textures without affecting the visual quality and use them in a real-time ensubmitted to COMPUTER GRAPHICS Forum (2/2024).

vironment. To do so, we structure the features in blocks of 4 × 4 and define the parameters on a per-block basis (sec. 4.1). Then, we design a forward pass that emulate the BC6 decompression (sec. 4.2) and takes advantages of hardware texture samplers (sec. 4.3). This makes it possible to directly export the neural features as BC6 textures without the need for a subsequent compression step and use them into real-time environment with a minimal computational overhead.

Block parameterization

In the BC6 setting, an image is stored by encoding the endpoints value for each 4 × 4 block and indexing each pixel according to its distance to the corresponding line segment (fig. 2). The dual partition mode of BC6 divides the pixels inside each block into two regions allowing for two lines segments per block.

We design the neural features to mimic this behaviour. Given a feature layer T θi of size w × h × 3, we force its values within a block of size 4 × 4 to lie on two lines segments. This is done by design, we structure our block-based neural features by modeling the parameter set θ i such that each block of size 4 × 4 is modeled with

     l 1 = {e 1 , e 2 } ⊂ R 3 l 2 = {e 3 , e 4 } ⊂ R 3 x = {x 1 , . . . , x 16 } ⊂ [0, 1] , (6)
where l 1 and l 2 are two sets of endpoints for the first and second line segments respectively and x pixel index represented as the relative position of each one of the sixteen pixels on the corresponding line segment. To determine which pixel belongs to which segment, we attach to each block an integer, k ∈ {0, . . . , 31}, linking to a binary mask of the corresponding partition. We refer the reader to the DirectX BC documentation [D3D] for all the technical details regarding the pre-defined partitions.

Reconstructing the material information (eq. 3) requires sampling each neural feature T θi at position (u, v) and scale s. This consists of performing the BC6 decompression and then filtering the corresponding value according to a certain strategy (fig. 5). In order to train our block-based model, it is necessary to backpropagate through this decompression. Additionally, it is important to choose a filtering strategy that does not incur additional computation overhead. The latter is crucial to be able to integrate our neural material model in rendering pipelines.

Trainable BC6 decompression

In the two partition mode, a BC6 block stores two sets of quantized endpoints, the pixel indices and a partition ID. The decompression operation uses this data to recover back the original information by mixing the endpoints proportionally with the index values. According to the BC6 standard specification [D3D], this operation is non-linear and is done in three steps. First, the endpoints of each block are unquantized as follow : Where a = 31 /64 for the unsigned BC6 mode, a = 31 /32 for the signed BC6 mode and b is the number of bits used to quantize the endpoints. This essentially maps the endpoint values into a valid range. The second step consists in interpolating the endpoints linearly:

e i = a2 16 e i + 2 15 2 b , i = 1, . . . ,
y = P k ⊙ (e 1 + 2 q
x(e 2e 1 )) + ¬P k ⊙ (e 3 + 2 q x(e 4e 3 )), (8) where P k is the binary mask associated with partition k and q is the number of bits used to quantize the pixel indices. This will give a value y ∈ [-31743, 31743] and whose bits are finally re-intrepreted as a half precision number [iee19]. This cast is a non-linear transformation that can be simulated with the following operation

w = 2 h(y)-14 y 1024 -h(y) , (9)
where h(y) = max (⌊ (y-1) /1024⌋ -1, 0).

Therefore, to simulate the hardware BC6 decompression for each pixel within a block, we unquantize and mix the endpoints using eq. ( 7) and eq. ( 8), and finally simulate the bit re-interpretation operation with eq. ( 9) resulting in the final value w. All these operations being almost-everywhere differentiable, it allow us to backpropagate through the BC6 decompression, when the partitions are fixed.

Hardware compliant filtering

In a real-time rendering context, sampling a texture at a particular scale is often done via mipmapping. This technique consists in explicitly storing in memory a version of the texture at different scales, also known as mip, and performing trilinear filtering. This involves doing a texture lookup and bilinear filtering on the two closest mipmap levels (one higher and one lower), and then linearly interpolating the results. In general, this approach stabilizes sampling performance as it fixes the number of processed pixels for each query independently of the scale values.

Since our goal is to integrate the neural material model in a realtime environment, we propose to rely on trilinear interpolation to filter the neural features T θi . This makes it possible to exploit hardware accelerated texture filtering to sample the corresponding neural textures during rendering. To do so, we consider that each feature layer T θi is composed of a pyramid with S i block-based mips, Figure 6: When sampling a feature at a scale level s, the block compressed features are filtered using a trilinear interpolation. This requires sampling the two closest mip levels ⌊s⌋ and ⌊s⌋ + 1 using a bilinear interpolation than linearly mixing the results.

T θi = {T 0 θi , . . . , T Si θi }, of decreasing sizes. Each block-based mip has independent parameters that will be adjusted during the training process. In this setting, the scale parameter s ∈ [0, S i ] refers to the level at which the features T θi are sampled. This is done as follows:

T θi (u, v, s) = λT ⌊s⌋+1 (u, v) + (1 -λ)T ⌊s⌋ (u, v), (10)
where, λ = s -⌊s⌋ and T k θi (u, v) is the bilinear interpolation from the k th block based mip at coordinates (u, v) (fig. 6).

Implementation details

In this section, we present all the practical details related to the encoding and decoding of PBR material information with our block based neural features.

Material encoding

Model structure In our implementation, we use 4 sets of neural block-based features {T θ0 , T θ1 , T θ2 , T θ3 } with size w i = h i = 2 ni . As described in section 4.3, each T θ0 consists of a pyramid with several block-based mips of decreasing power of two sizes. We use a simple MLP with RELU activations as our decoder network and adjust the size of the output according to the endcoded material.

Partition selection strategy As mentioned in section 4.2, we can backpropagate through the simulated BC6 decompression only if the partition IDs for each block are fixed. In order to maximise the reconstruction quality, it is necessary to select the most optimal partition for each block. However, choosing the one with the lowest error during training is not straightforward. Indeed, the optimal values of l 1 , l 2 and x i might vary significantly depending on which partition k is considered. This means every-time the partition changes, these parameters need to re-adapt to the new set of pixels resulting in training instability. To overcome this issue, one approach would be to learn in parallel all the parameters for each possible partition for each block. This is unpractical and would increase both the memory and time needed to learn the material model. Instead of that, we propose to consider the partitions as hyper-parameters, and thus fix them before the training. Since randomly setting their values would possibly lead to non-optimal partition attribution, we therefore propose to learn the material model in two steps. First, we perform a quick training using a set of features with unconstrained parameters, i.e., move freely in 3D space and not constrained on a line segment. Then, we initialize the block-based features by compressing the unconstrained features with the BC6 algorithm. This provides us both a partition selection strategy and a relevant initialization of the parameters within each block. In our experiments, we train the model for 5k iterations with unconstrained parameters, then use the result to initialise the block based features. While this does not guarantee the selection of optimal partitions, we found that it improves the reconstruction quality and consistency of the results.

Reference material sampling

In our experiments, we consider reference materials with S mipmaps and sample them at random during training by doing a bicubic filtering on the closest two mips then linearly interpolating the results. More precisely, we process batches of 512x512 uniformly sampled uv-grids, for each batch we also sample a continuous scale parameter s uniformly in [0, S] where S is the number of mipmap. Both the reference material and the neural material (sec. 4.3) are then sampled at (u, v, s) and the loss is computed as the mean squared error between them. For more details on the training parameters, we refer to the results in section 6.

Real-time Decoding

We export each trained feature layer as a mipmapped BC6 texture. The block-based structure of these features simplifies this operation as it only involves the quantization and encoding of the endpoints and pixel indices with 6 bits and 3 bits respectively. The network's weights are stored directly in a binary buffer as fp16 values since their size is negligible.

Multiple methods can be employed to integrate our model into a real-time renderer. Our implementation is rather straightforward and simply consists in loading the neural textures and weights in GPU memory and proceed as follows (fig 3). First, we use the partial derivatives of the input pixel's uv coordinates to compute the sampling scale s i for each neural texture T θi

s i = log 2 (max (| ∂uv /∂u| , | ∂uv /∂v|)) + b i , (11)
where b i = log 2 (max( hi /h, wi /w)) is a bias value that depends on the neural texture's and render resolutions, respectively w i × h i and w × h. Then, we sample each of the neural textures using the GPU's hardware filtering capabilities and pass the resulting values to the decoding shader. Within this shader, the sampled values processed through a sequence of functions that perform vector-matrix multiplications with the model's weights and the return the material information. When rendering a scene with multiple neural materials, we store all the different networks weights in a single buffer and use a material id value as an offset to access a specific model's weights. Finally, the pixel is shaded.

Results

Experimental setup

Evaluation dataset We gather a dataset of 19 materials from polyhaven.com, 12 of which are accompanied with a 3D model. All of submitted to COMPUTER GRAPHICS Forum (2/2024). the material information consist of at least 9 channels that are stored in a set of three textures : albedo, normals, and arm. The albedo layer has 3 channels and contains the diffuse color information. The normals layer has 3 channels and contains surface normals with respect to a local tangent frame. In practice, our model learns the x and y components since the z value can be reconstructed. The arm layer has 3 channels and contains the ambient occlusion, surface roughness and metalness parameters. 8 of the materials in the dataset contains an additional displacement channel that is stored in its own texture. Note that our model is not restricted to learn only this specific material representation but can handle materials with arbitrary number of layers.

Model configuration. In our experiments, our models consisted of four mipmapped square shaped block compressed features. Table 1 details the resolution of each of the feature layers along with the corresponding number of mips. Our decoder network consist of a small MLP with 12 inputs and one hidden layer of dimension 16. The output of the network depends on the number of channels in the material texture set. For the rest of this paper, we will refer to our model by the name of its block compressed features configuration and the size of its network. For instance BCf-0.5K refers to the model with a resolution of T θ0 equal to 512.

Training parameters. We use the Adam stochastic gradient descent algorithm [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF] with an exponential decay learning rate scheduler. We set a different learning rate for both the block compressed features and the MLP in order to balance the gradient backpropagation. We train the models for a total of 205k iterations in two phases. In the first phase we train our model with unconstrained features for 5k iterations using learning rates of 5 × 10 -2 and 10 -3 for the features and the MLP respectively, and a decay parameter γ = 0.9995. The second phase starts by initializing the block compressed features from unconstrained ones as described in section 5.1. Then we train the model for 200k iterations using a learning rate of 10 -2 for the features and 10 -3 for the MLP with decay parameter γ = 0.9999. For reference, our BCf-1K requires about 140 minutes to be trained for the 205k iteration on an NVIDIA RTX2070 with our implementation using the PyTorch library [PGM * 19]. Note that we observed that the model already outputs decent results after 10k steps, in approximately 15 minutes.

Compared methods. We compare our method with standard BC compression, ASTC [NLP * 12] and our implementation of Vaidyanathan et al.'s NTC [VSW * 23]. Both BC and ASTC compressed textures were generated using NVIDIA's texture tools exporter [NVI] with a compression quality set to the highest possible setting. For BC compression, we used the BC5 format to store the (a) psnr: 34.9 db (b) psnr: 34.7 db normal layer and the BC1 for the rest for rest of the texture set (the albedo, armand displacement textures when available). For ASTC, we used a 12 × 12 blocks as it matches the size of our BCf-1K configuration. In the context of our experiments, where the reference material resolution is 2048 × 2048, the highest resolution of NTC's features grid of set to 512 for NTC0.2 and NTC0.5, and 1024 for NTC1.0.

Considered metrics.

Quantifying the visual quality of an image is still an open problem as no metric can effectively align with human perception. This is especially the case when the types of distortions introduced by the compression methods are different. For instance, traditional BC and ASTC methods tends to have more blocky artefacts while neural methods such as ours and NTC tend to exhibit color shift and feature bleeding. For this reason we mainly rely on the PSNR value as it is directly linked to the loss we are optimizing. In this sense, the PSNR can be seen more as a proxy for the method's capacity to approximate the reference data and not as a measure of visual quality. We also include SSIM [WBSS04] and FLIP [ANA * 20] values as they are supposed to be more in line with human perception. We compute our metrics for each mip level and aggregate the values as described by Vaidyanathan et al. in [VSW * 23].

Reconstruction quality

Block-compressed features. Visual Performance. We trained all our model configurations against a reference 2K material and compared them to the methods mentioned in section 6.1. The results are gathered in table 2 and illustrated in figure 8. In these experiments, the texture set is reconstructed using a regular uv grid aligning with the center of pixels and the metrics are evaluated by comparing the result with the reference 2K material. Our model outperforms standard BC and ASTC textures as it does not exhibit blocky artifacts and is capable of reconstructing sharper visuals with less memory (see fig. 9). It is not surprising to see that the resolution of the neural features has a direct impact on the capacity of the model to reconstruct the material. The higher the resolution, the better the reconstruction. However, our model's performance does not scale equally with the increase in resolution and can output a slightly smoother result when compared with NTC [VSW * 23] (see fig. 9). This is due to our minimalist neural architecture that relies on a very small MLP with one hidden layer of size 16 and an input of size 12 to reconstruct a very complex signal which is particularly well suited for real-time performance. While naively increasing the size of the hidden layers does improve the quality, it causes the inference time to increase in a quadratic manner. We refer to the supplementary material for additional experiments with larger networks and other configurations.

The main advantage of our approach with respect to NTC, is its capacity to output filtered material information. This is particularly important since, in a 3D environment, material texture sets are almost never perfectly aligned with the screen's pixel. The misalignment between the 3D object and the viewport, along with changes in distance, results in the material being sampled at random points within the uv domain and across continuous scales. NTC cannot handle this in a straightforward manner since it is trained by considering a fixed uv grid and lod values. For instance, in fig. 10 compare our method to NTC by evaluating the result while constantly changing the lod value. It shows that NTC's performance is not stable when opting to reconstruct the material with 1 sample per pixel (spp) and requires to be paired with a filtering operator. In this case, a bilinear or trilinear filtering is not suited for real-time rendering as it requires decoding multiple samples per pixel. While temporal stochastic filtering can mitigate this issue, its reconstruction quality is dependant on several factors (jittering pattern, noise distribution, accumulation policy, etc) and does result in a smoother reconstruction in motion (fig. 11). Our approach, on the other hand does not require any filtering and is capable to reconstruct the material texture with 1spp. Moreover, the values outputted by our decoder remains stable across time (fig. 11) and mip levels (fig. 10) and can even handle extreme magnifications (fig. 12) . This is not surprising since our model is trained to emulate a continuous filtering operation. In this case our network plays the role of a decoder and a filter at the same time.

Decoding performance

In order to better illustrate the computational overhead of our neural material model, we rendered a textured full-screen quad at 1080p, 1440p and 2160p (4K). We then compared the total rendering time of our BCf-1K with rendering times obtained using the same setup but with standard BC textures. The results are presented in table 3. They clearly show that our method is well suited for real-time environments as the computational overhead on a 4K resolution is only 0.6ms for a full 4K screen. The small computational overhead is a direct consequence of our block compressed neural features. Ex- porting these features as BC6 textures makes their memory footprint small which makes it possible to increase their resolution. This allows us to rely on a very small and fast decoder network to reconstruct the material. More importantly, it enables the use of hardware texture filtering operations to sample the features at render-time which minimizes the overhead.

Quantization vs. Compression

As mentioned in section 3, there are two possible strategies that can be adopted when it comes to storing the learned neural features with fewer bits. We can either quantize their values or maintain them at a high bitrate and compress them. Both approaches are not straightforward to implement. Simply reducing the number of bits downgrades quality. This can be mitigated by increasing features's dimension but requires reducing the resolution to maintain manageable memory usage. For instance, it is possible to match the size of our BCf-1K profile with features of 4 channels quantized with two bits, or by dividing their resolution by two and increasing the number of channels to 8 and quantizing with 4 bits. Figure 13 plots the PSNR values of these configuration for the ukulele model and shows that, when pairing the features with a small network, it is more beneficial to have high bitrate features that are compressed than high dimensional low bitrate values. On average, at equal resolution, neural features with 3 channels stored according to be BC6H format performs similarly to ones with 5 channels and quantized with 4 bits resulting in a memory reduction of 40% (fig. 14). Increasing the number of channels and increasing the resolution without increasing the complexity of the network is not a viable strategy. The resolution of the features has a major impact on the reconstruction quality that cannot be compensated by increasing the number of channels and more importantly the larger input will overwhelm the small network and impact the result in a negative way. A larger network and more complex architecture are, thus, needed to fully take advantage of the higher number of input features and compensate for the lower resolution and bitrate as demonstrated by

Impact of BC partition selection

By design, the introduced BC6 constraints restrain the model capability compared to unconstrained features. The more the partition shape is aligned with the content of the data in a 4 × 4 block, the better the compression. Thus randomly selecting the partitions or forcing a static one is not optimal and will lead to lower reconstruction performance. Figure 15 shows that by initializing the partitions from pre-trained unconstrained neural features, as discussed in section 5.1, yields better result in the long run. In general, the warmup stage is not required to be long. We found that usually 5K iterations are sufficient for the shape of the unconstrained features to stabilise resulting in more suitable partitions. On our hardware, this takes up to 3 minutes.

Limitations and Future Work

In this section we discuss some of the limitations of our neural material model and offer avenues for future work.

Reconstruction artifacts Our reconstructed material can experience layer bleeding and color shift. Since our model is exploiting correlation between the channels and projecting the data into a smaller dimensional space, we believe these artifacts occur when there is small correlation or their absolute values are too divergent.

In addition, the use of a decoding network consisting of only one hidden layer makes our model struggle with reconstructing and separating the data from the material layer. Even increasing the network size is hardly an option considering the potential overhead. It should be possible to overcome these issues by designing a decoder network that separates the reconstruction of uncorrelated layers. In addition to that, we could learn a tone mapping operator to map each output to the corresponding values. The challenge here, is to keep the decoding network small in order to maintain real-time performance, so we leave this to future work.

Higher quality reconstruction Depending on the profile, our model may struggle to learn high-frequency details which results in a slightly over-smooth reconstruction. This is a direct consequence of having neural features with only 3 channels and a very small decoder network, which prevent the effective use of positional encoding that could have contributed additional high frequency information as shown in [VSW * 23]. Increasing the number of channels is key here. However, it is not straightforward in the context our BC neural features as the BC6H format is only capable of compressing RGB images. One way to overcome this is to group the high dimensional features into sets of 3 and process them independently. In this context, our BCf-2k++ configuration can be seen as one where we have three neural layers where the first one has 6 channels.

Reference 2k

42.7MB

BCf-1K Simulating more complex filtering Even with simple MLP decoders, our architecture is able to emulate simple texture filtering operations such as bilinear and bicubic filtering. We believe that this idea can be explored further by training models to emulate more elaborate filtering operations. For instance it would be possi- ble to sparsely sample the neural textures in an anisotropic manner, i.e., along the major ellipse axis, and train them against a high density anisotropic filtered ground truth. This would make it possible to emulate high quality anisotropics filtering with few samples.

Conclusion

This work introduces a novel block-compressed feature layer along with a continuous neural material encoding framework. This allows us to design an encoding and a decoding method that fits well within the real-time rendering pipeline constraints. We demonstrate the ability of our method to compress PBR materials efficiently and decompress them in real-time in a shader on consumer-level hardware. By taking into account memory and time constraints, and by taking advantage of existing hardware operations, we make neural textures usable at large scale in a real-time rendering pipeline, such as a video game engine. We hope that our work could serve as a starting point for future work and open the door for more complex use cases, such as learning directional materials or more complex filtering.

Figure 3 :

 3 Figure 3: Overview of our neural material framework. (a) The neural features T θi and the MLP fη are fitted through backpropagation to match the filtered material F(M). (b) After training, the neural features T θi are exported as mipmapped texture sets that can be sampled by the engine and the weights η of the MLP are exported as a binary buffer. A shader is used to perform the MLP inference after trilinearly sampling the neural texture, outputting the filtered material G θ,η (u, v, s) ≈ F(M)(u, v, s). Finally, The renderer can perform the shading step as usual.

  Figure 4: Neural material reconstructed from (a) raw unconstrained neural features and (b) compressed ones. The naive compression of the neural features will lead to artifacts severely affecting the visual quality.

Figure 5 :

 5 Figure5: Evaluating T θi at position (u, v) and scale s is done by performing a BC6 decompression then filtering the result with respect to the given scale. Our simulated BC6 decoding makes it possible to backpropagate through this operation and train the perblock neural features.

Figure 7 :

 7 Figure 7: Our training process comprises three stages: A warmup stage (blue) with unconstrained neural features. The main training stage (yellow) with Block-based features initialised from the result of the previous stage. Finally, a finetune stage (green) the BC features are quantized. The results at the end of the second stage (a) and third stage (b) are visually identical.

  Figure 7 shows the evolution of the PSNR value throughout the entire training. We start by training a set of unconstrained embedding and use them to initialise the BC submitted to COMPUTER GRAPHICS Forum (2/2024).

Figure 9 :

 9 Figure 9: Close-up crop of the Ukulele's material texture set. Comparison between ASTC, BC, our BCf-1K, NTC0.5 methods and the reference 2K material.

Figure 10 :

 10 Figure 10: Evolution of the PSNR reconstruction metric over the scales on the Ukulele's material.

Figure 11 :Figure 12 :

 1112 Figure11: Close-up crop of the reconstruction of the Ukulele's material when subject to a circular motion. We also track the values of each channel throughout time at point P. NTC is not stable when it is evaluated center of the pixel and requires some filtering which smoothen the result. Our approach does not require any filtering and output values that are stable temporally.

Figure 13 :

 13 Figure 13: The reconstruction performance of the Ukulele's material using neural features with various number of channels and quantization rate versus Block Compressed ones. The quantization rate is written in white.

Figure 14 :

 14 Figure 14: Texture set reconstruction obtained with neural features of various configurations. Our BCf-1K configuration matches up visually with neural features of dimension 5 and quantized on 4 bits while being 40% smaller in size.

Figure 15 :

 15 Figure15: Initializing the partitions of the neural BC feature from a set of unconstrained features yields better result in the long run. After 50k iterations, our initialisation heuristic is ahead by 1 PSNR unit compared to a randomized initialisation.

Table 1 :

 1 our Neural block-compressed feature configuration.

		T θ0		T θ1		T θ2		T θ3
		res	mips	res	mips	res mips	res mips
	BCf-0.5k	512	8	256	7	128	6	64	5
	BCf-1k	1024	9	512	8	256	7	128	6
	BCf-2k	2048	10	1024	9	512	8	256	7
	BCf-2k++ 2048	10	2048	10	512	8	256	7

Table 2 :

 2 Evaluation of the reconstruction metrics (PSNR, SSIM and FLIP) on the whole dataset. For PSNR, we averaged the mean-squared error over all the mips (as proposed in [VSW* 23]) and all the models then computed the PSNR. at the 5000th iteration as stated in section 5.1. It is expected that the PSNR value drops at this point since the network's input undergoes a change due to the alteration of the embeddings. The BC constrained model is trained for 200K iterations, after which we quantize the trained end-points and indices. Following this, we freeze the values of the BC features and proceed with training for 1000 more iterations to fine-tune the network. Although quantizing the trained features leads to a slight reduction in PSNR value, its visual impact is negligible. The material information reconstruction from the raw unquantized fp16 block compressed features (fig.7 (a)) and from the exported BC6 block features (fig.7 (b)) are visually indistinguishable.

	BC	ASTC12

features

  , weFigure 8: Reconstruction PSNR versus size in megabytes (MB) for all the methods presented in table 2.

33.88 dB 34.41 dB 33.81 dB 34.58 dB 33.04 dB 34.85 dB 35.09 dB 32.86 dB 31.51 dB 32.41 dB 31.62 dB 30.27 dB

  submitted to COMPUTER GRAPHICS Forum (2/2024).

	ASTC 12x12	BC 1024	BCf-1k	NTC 0.5	Reference 2k
	1.89MB	2.67 MB	1.77MB	2.26MB	32MB
	Diffuse
	Normal
	ARM

Table 3 :

 3 Average time to render a textured full-screen quad using an NVIDIA RTX2070.

		BCf-1K BC Textures total time neural overhead
	1080p	0.376 ms	0.555 ms	0.179 ms
	1440p	0.614 ms	0.924 ms	0.310 ms
	2160p	1.684 ms	2.324 ms	0.640 ms

submitted to COMPUTER GRAPHICS Forum (2/2024).

Acknowledgements

We thank Dr. Heqi Lu for all the discussions and advice and Arnaud Schoentgen for proofreading the paper. We also thank all the anonymous reviewers for their insightful comments.