The "stability map" is the Bootstrap Ratio result, which is randomized for each loop. Therefore, you cannot expect exactly same result. If you could upload your data to somewhere that I can have access, I can take a look for you.

The computation for Non-Rotated Task PLS is pretty straightforward. We calculate crossblock first. The crossblock is the projection of contrast onto mean-centered datamat. For example, if you try: dat=rand(2,6), x=[-1 1]*dat, y=[1 -1]*dat, you can find that x & y is just a sign switch.

By the way, this computation algorithm has never changed. i.e. The most updated version has the same result as any of the previous versions. Because we have hundreds of previous versions, we cannot support each of them. Therefore, only the most updated version is supported. If you upload your data, I download it, and I can only look into it under the most updated version.

Hi Jimmy and Randy,

It only happens when I change the order, but not when I try to reproduce the stability maps under the same order repeatedly. In this latter case the maps are almost always identical. 

Thanks for pointing out I should look at the bootstrap details. When the order is changed, the Maximum ratios are almost the same with a sign change, but one threshold (3.15) looks much more normal than the other (1.03). So I guess if I want to look at anything, i should look at the one with the more reasonable threshold.

Regarding the number of iteration, since my data size is usually small and compution time is not a concern, i go much higher than 100/200 (500/1000). But the same problem arised when I use 100/200. Regarding the similarity between the extracted LVs, I compared the wave_amplitude matrices of the two analyses and they only differ by their signs.

I put a copy of my raw data and analysis files in Dropbox.There are 16 subjects each taking part in A and B. The raw data contains 70 electrodes x 1330 time points (from -1200ms to 4000ms; 256hz) exported from EEGLAB but I am only examining [-200 1060ms] on 66 scalp electrodes sans the EOGs and zeroed nose reference. Number of iterations for permutation and bootstrapping are 100 and 200, respectively, with 95%CI. I generated the results using non-rotated task PLS by fitting either contrast [1 -1] (A-B_ERPresults) or [-1 1] (B-A). I also included a document containing the Salience maps and Bootstrap thresholds/ratios of A-B and B-A analyses.

https://dl.dropboxusercontent.com/u/26412142/PLS/probeC_for_upload.rar

I totally understand the concern with the version. But if the algorithm has not been changed, i think there must be something I'm not doing right...

On the other hand, it would be great if you could also advise me on non-rotated PLS when two groups are involved. In some of my other data I tried to supply a contrast like [1 1 -1 -1] hoping to capture any between-group differences (the first two weights corresponded to Group1 and the latter two Group2). If I look at the contrast GUI they appear to be treated as two separate within-group contrasts [1 1] and [-1 -1]. Is this the way to compare between groups? Or should I do further analyses on the brain scores (e.g., extract them and compare them with parametric/permutation tests) if I want to conclude that the two groups differ in the given LV? Pointers to relevant manual/ materials would already be very useful.

Regards,

Eric

Thanks Jimmy. So everytime when i run an analysis, I should look at the boostrap distribution.

But then which set of stable salience should I intepret? I notice that the thresholds in the toolbox is at 95% positive tail of the bootstrap distribution. Does it sound right if I:

1) for each LV of interests, I look at the bootstrap distribution and then either

2a) set the threshold of the negative threshold at the 2.5th percentile and positive threshold at 97.5th percentile, using both tails of the bootstrap result with respect to the sign of the salience, OR

2b) depending on the skewness, I take either the 5th or 95th percentile, whichever gives a more stringent criterion (assuming this is a valid conversative option although beating the purpose of having an empirical distribution, and I opt for convservative decisions)?

Thanks again for the prompt help! 

Regards,

Eric

wrt to skewness, for this dataset, it is not surprising, as your primary effect is that session 1 tends to have more positive amplitudes than session2 (this is reflected in the positive saliences from your first analysis).

From an interpretation point of view... using the default  "rule of thumb" threshold for the bootstrap ratio, most of your stable results are all late (>~500ms), the P2 effect comes in using a bootstrap threshold of ~2.5, which is a reasonable threshold (although note that PLS is detecting the rate to/from the peak, not the absolute value at the peak).

nancy

The main reason for the bootstrap ratio rather than the exact confidence interval is that in order to calculate the exact confidence interval, you need to store the distribution of weights (saliences) for each element to create an empirical distribution and then for each derive the upper and lower bounds of the confidence interval.  Storing that is a challenge as the "brain" side of the analysis grows.   We opted for calculating the standard deviation of the weights distribution using the computational formula for the standard deviation, which allows you to store a single vector that is the running tally of sums and sums of squares.

If you compare the two criteria (i.e., ratio vs confidence interval), they are quite close in terms of what  weights are stable.  When they differ, it is often at borderline cases where an estimate is just at threshold.  The ratios are valid proxies when the distribution of the parameter estimate is Gaussian - by central limit theorem they should be close, but there are cases where there may be a skew (e.g, correlation coefficients).  In severe cases, the confidence intervals based on percentiles will be more accurate.  This is why the confidence intervals are provided for the behavior/seed correlations in behavior PLS.

For additional clarity, in case it comes up, the skewness I am talking about here is for the distribution of the weight for a given electrode&time point (or voxel) across all bootstrap iterations and *not* the distribution of bootstrap ratios across all electrodes/voxels.  In the latter case, as evidenced in your own data Eric, the distribution can be skewed if there is a large effect.

Hi Eric..

One of the interesting features of using the bootstrap approach to identify the reliability of the saliences is that we can identify "small" saliences that are reliable.  These small effects might be missed by traditional parametric statistics.  You can see an example of this in Fig1 in the 2004 review paper.

I think you are still confusing skewness with the direction of the effect - the bootstrap distribution of a negative effect (A<B) is assumed to be gaussian (not skewed); the same is true for the bootstrap distribution of a positive effect (A>B). 

The "95th" percentile we provide is just a first-guess estimate of a threshold you might want to set for your data - reliability is reliability, so the direction is less important...  From the 2004 paper - we often find that it is easier to grasp the concept of reliability if it is couched in terms of the more-familiar p-value... threshold of 2.57, which has an approximate two-tailed probability of 0.01 assuming a unit normal distribution. You can also think of it in terms of a z-score.  In your example, a threshold of 1.44 would be a very liberal threshold..

nancy