-------------------------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  
                                                   Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg     Self CUDA   Self CUDA %    CUDA total  CUDA time avg    # of Calls  
-------------------------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  
void flash::flash_fwd_kernel<Flash_fwd_kernel_traits...         0.00%       0.000us         0.00%       0.000us       0.000us     479.425us       100.00%     479.425us     119.856us             4  
                                          ProfilerStep*         1.69%      47.721us        15.11%     426.888us     142.296us       0.000us         0.00%     358.144us     119.381us             3  
                                       flash_kernel_fwd         2.47%      69.860us        13.42%     379.167us     126.389us       0.000us         0.00%     358.144us     119.381us             3  
                                          FlashAttnFunc         2.81%      79.424us        10.95%     309.307us     103.102us       0.000us         0.00%     358.144us     119.381us             3  
         _flash_attn2_cuda_8a730d9::_flash_attn_forward         3.59%     101.322us         7.87%     222.453us      74.151us       0.000us         0.00%     358.144us     119.381us             3  
                         _flash_attn2_cuda_8a730d9::fwd         1.34%      37.810us         4.29%     121.131us      40.377us     358.144us        74.70%     358.144us     119.381us             3  
                                       flash_kernel_fwd         0.00%       0.000us         0.00%       0.000us       0.000us     358.144us        74.70%     358.144us     119.381us             3  
                                 cudaDeviceGetAttribute         0.07%       2.040us         0.07%       2.040us       0.227us       0.000us         0.00%       0.000us       0.000us             9  
                                       aten::empty_like         0.50%      14.091us         1.02%      28.841us       9.614us       0.000us         0.00%       0.000us       0.000us             3  
                                    aten::empty_strided         0.52%      14.750us         0.52%      14.750us       4.917us       0.000us         0.00%       0.000us       0.000us             3  
                                            aten::empty         0.71%      20.020us         0.71%      20.020us       2.224us       0.000us         0.00%       0.000us       0.000us             9  
                                   cudaFuncSetAttribute         0.15%       4.120us         0.15%       4.120us       1.373us       0.000us         0.00%       0.000us       0.000us             3  
                                       cudaLaunchKernel         1.00%      28.300us         1.00%      28.300us       9.433us       0.000us         0.00%       0.000us       0.000us             3  
                                            aten::alias         0.26%       7.430us         0.26%       7.430us       2.477us       0.000us         0.00%       0.000us       0.000us             3  
                                  cudaDeviceSynchronize        84.89%       2.398ms        84.89%       2.398ms       2.398ms       0.000us         0.00%       0.000us       0.000us             1  
-------------------------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  
Self CPU time total: 2.825ms
Self CUDA time total: 479.425us