-------------------------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  
                                                   Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg     Self CUDA   Self CUDA %    CUDA total  CUDA time avg    # of Calls  
-------------------------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  
                                       flash_kernel_fwd         0.00%       0.000us         0.00%       0.000us       0.000us     248.286us       101.31%     248.286us      82.762us             3  
                                          ProfilerStep*         1.56%      39.280us        22.73%     573.121us     191.040us       0.000us         0.00%     245.085us      81.695us             3  
                                       flash_kernel_fwd         2.88%      72.583us        21.18%     533.841us     177.947us       0.000us         0.00%     245.085us      81.695us             3  
                                          FlashAttnFunc         2.75%      69.291us        18.30%     461.258us     153.753us       0.000us         0.00%     245.085us      81.695us             3  
         _flash_attn3_cuda_8a730d9::_flash_attn_forward         6.85%     172.593us        15.55%     391.967us     130.656us       0.000us         0.00%     245.085us      81.695us             3  
                         _flash_attn3_cuda_8a730d9::fwd         2.34%      58.961us         6.92%     174.493us      58.164us     239.678us        97.79%     245.085us      81.695us             3  
void cutlass::device_kernel<flash::enable_sm90<flash...         0.00%       0.000us         0.00%       0.000us       0.000us     239.678us        97.79%     239.678us      79.893us             3  
                                            aten::zero_         0.45%      11.330us         2.29%      57.660us      19.220us       0.000us         0.00%       5.407us       1.802us             3  
                                            aten::fill_         0.58%      14.610us         1.84%      46.330us      15.443us       5.407us         2.21%       5.407us       1.802us             3  
void at::native::vectorized_elementwise_kernel<4, at...         0.00%       0.000us         0.00%       0.000us       0.000us       5.407us         2.21%       5.407us       1.802us             3  
                                        aten::new_empty         0.47%      11.750us         1.54%      38.821us       4.313us       0.000us         0.00%       0.000us       0.000us             9  
                                            aten::empty         1.23%      30.951us         1.23%      30.951us       2.063us       0.000us         0.00%       0.000us       0.000us            15  
                                       cudaLaunchKernel         1.80%      45.441us         1.80%      45.441us       7.573us       0.000us         0.00%       0.000us       0.000us             6  
                                cudaGetDriverEntryPoint         0.13%       3.360us         0.13%       3.360us       0.187us       0.000us         0.00%       0.000us       0.000us            18  
                                   cudaFuncSetAttribute         0.08%       1.970us         0.08%       1.970us       0.657us       0.000us         0.00%       0.000us       0.000us             3  
-------------------------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  
Self CPU time total: 2.521ms
Self CUDA time total: 245.085us