To change the number of call traces that can be stored in the SN10k, you will need to edit the tbuctwriter application parameters.
On the left hand menu, navigate to the following:
From the "Application Configuration List", select "tbuctwriter".
Now select the "Application Parameters" tab and you will see the option for "Maximum calls".
The default is 10k and the maximum number of stored calls is 100k. RAM usage will be used more the higher this number is but it should not affect performance.