Introduction
We learned about correlations back when we were still working on descriptive statistics. If you remember, they use the statistic Pearson's r, which represents the amount that two variables (continuous) vary together. This statistic ranges from -1 to 1, with r values closer to 1 or -1 representing stronger relationships. The sign dictates the direction of the relationship. In the plots below, we can see examples of each of these relationship types.
v1<-c(1,2,3,4,5)
v2<-c(6,7,8,9,10)
v3<-c(5,4,3,2,1)
v4<-c(1,1,1,1,1)
fake<-data.frame(v1,v2,v3,v4)
par(mfrow=c(2,2))
plot(fake$v1, fake$v2, main="Positive Relationship")
plot(fake$v1, fake$v3, main="Negative Relationship")
plot(fake$v1, fake$v4, main="No Relationship")
We can also examine the actual correlations between each of these.
positive<-cor(fake$v1, fake$v2)
negative<-cor(fake$v1, fake$v3)
print(paste0("Positive Correlation = ", positive,"; ", "Negative Correlation = ", negative, "; ", "No Correlation = 0"))
[1] "Positive Correlation = 1; Negative Correlation = -1; No Correlation = 0"
So, in this example, we can see that there is a perfect positive correlation, a perfect negative correlation, and a perfect lack of correlation. In these cases, it's pretty easy to tell that the relationship (when we're looking at covariation, we're looking at relationship rather than difference) is very likely real. However, what if the Pearson's r was .35? While that looks "moderate" by the standards we used when we first talked about correlation, as we saw with ANOVA and t-tests, effects sizes are not the same as statistical significance. So, what gives? How are we to know if a relationship is real?
Enter the Correlation Test
The good news is that it's really easy to test for bi-variate relationships between continuous variables using (basically) what we've been doing all along. Additionally, we use essentially the same steps as we use to test for difference (in terms of setting hypotheses, etc.), so it's pretty much what you're used to seeing at this point. However, it serves as the basis for understanding regression, which is fundamental to the types of analysis we will be engaging in for the remainder of the semester. So it is really important to get this stuff down.
Correlation Test in a Nutshell
When we saw above that the Pearson's correlation coefficients (r) for the charts above were 1, -1, and 0, (respectively), what does that actually mean? Well, in short, what we're doing is pretty straightforward. If, as a variable (usually denoted "x") increases by a unit (whatever unit it's measured in), the other variable (usually denoted "y") also increases by a unit (or more - again, in whatever units its measured in) each time, then there will be a perfect relationship. In other words, we're not measuring the units, we're measuring the movement across the axes as we increase values.
The good news is that makes correlation tests very easy to understand. If the y variable increases with the x variable's increase at a higher than expected rate, we can say the relationship is significant. Fortunately, because we have understood that most everything can be reduced to a normal distribution, we can measure the expectation of co-variation using an approximation of the curve.
Let's use a "real world" example to better understand the correlattion test. As usual, we visualize the data first. Here I'm using the mtcars dataset. In the code, you'll notice I use a formula to calculate the line (and plot it using abline()). We will talk quite a bit more about this when we talk about regression. For now, just think of it as a slope-intercept model (if you remember high school - y = m(x) + b), or don't worry about it as we will cover it in detail later.
From the above plot, it's pretty clear that there is a negative relationship. We can confirm this by calculating the Pearson's r - as we have done for awhile now!
cor(mtcars$wt, mtcars$mpg)
[1] -0.8676594
Based on the above, the relationship is clearly negative and strong - matching what we expect from our chart. However, is it real? To determine that, we use a new function: cor.test().
cor.test(mtcars$wt, mtcars$mpg)
Pearson's product-moment correlation
data: mtcars$wt and mtcars$mpg
t = -9.559, df = 30, p-value = 1.294e-10
alternative hypothesis: true correlation is not equal to 0
95 percent confidence interval:
-0.9338264 -0.7440872
sample estimates:
cor
-0.8676594
Now, a couple of things to notice. First, other than including ".test" the R code is exactly the same. Why? Because in order to calculate the test, we first need to calculate exactly the same Person's r that we calculated before! Second, we can see that there is additional output that we don't get with the cor() function - and much of it looks familiar! That's beause you've seen it before. We use a t-test to determine whether the relationship between variables is real! The same t-test we used before!
You might be thinking, "I thought t-tests were for difference between groups, but here we're looking at relationships, so what gives?" The t-test for correlation coefficients is actually testing for a difference - but one that implies relationship if the difference exists. Specifically, the t-test here is testing to see if the correlation coefficient (Person's r) is significantly different from 0. The null hypothesis, in this case, is that Pearson's r is = 0. We can reject if the t-test tells us that it is either > or < 0 - with an alpha of .05.
With that info, you can read the output above. We have a t-value of -9.599 (df = 30), and p < .05. This means (as usual) we can reject the null hypothesis and that there is a real relationship between the variables. Specifically, as weight increases, miles-per-gallon decreases. The 95% confidence interval is the range which we can be 95% sure that the "true Pearson's r" falls within. The bottom number, under "cor", is the Pearson's r value we're used to getting from cor(), as you can see below.
cor(mtcars$wt, mtcars$mpg)
[1] -0.8676594
Effects sizes
While Pearson's r itself functions as an inexact measure of effects size, we have a more specific, and useful measure in r-squared (also called R-squared, though technically that's reserved for regression, but don't worry about it). It is calculated by simply squaring the Pearson's r.
cor(mtcars$wt, mtcars$mpg)^2
[1] 0.7528328
The code above is relying on cor() to get r-squared. We can also get it from cor.test() directly, with a small amount of additional code.
mpg_wt$estimate^2
cor
0.7528328
What the r-squared value tells us is that the x-variable, in this case weight, explains 75% of the variation in the y-variable, mpg. This is much more intuitive, as if we can explain 75% of something with a single variable, we're doing pretty good! For context, imagine we could explain 75% of crime with something like depression. If we gave everyone depression meds, we would solve 75% of crime!
Real World Example
I'm going to use some data that I found from Toronto's Open Data portal. Specifically, I'm using neighborhood crime rates from: https://open.toronto.ca/dataset/neighbourhood-crime-rates/. The data consists of a .csv file with 140 rows, 1 per neighborhood. Below you can see the first few rows of the dataset.
From the above, you can see that there are a number of crime-related variables at the interval-ratio level of measurement (continous). I'm particularly interested in the relationship of crime types across neighborhoods. Specifically, I want to know if homocide and robbery are related. While I could compare like-years (e.g. 2016 to 2016), I see that there is an "average" column for each crime time, and given the relatively low number of homocide each year, it is likely better to use the average column for each.
First, I want to visualize my relationship.
Examining the plot above, it looks like there is a positive relationship between homocides and roberies within neighborhoods. To test this, I need to first establish my hypotheses:
H0: \(r = 0\) is the null hypothesis (Pearson's r is = 0)
H1: \(r \neq 0\) is the research hypothesis (Person's r not equal to 0)
Having done that, as with t-tetstts, I can "roll" the rest of the steps togetther and test using cor.test().
cor.test(toronto$Homicide_AVG, toronto$Robbery_AVG)
Pearson's product-moment correlation
data: toronto$Homicide_AVG and toronto$Robbery_AVG
t = 11.199, df = 138, p-value < 2.2e-16
alternative hypothesis: true correlation is not equal to 0
95 percent confidence interval:
0.5918602 0.7679971
sample estimates:
cor
0.6900103
Examining the above output, it is clear that there is a statistically significant relationship between homocide and robberies within neighborhoods in Toronto (t = -11.199(138), p < .05). This relationship appears to be fairly strong (Pearson's r = .69) and positive - nighborhoods tend to have high rates of one if they have high rates of the other. The r-squared value is .48, indicating that about 48% of the variation in homocide is explained by robberies within the neighborhoods.
We can further visualize this relationship by examining the same plot as above, with some additional information included.
#install.packages("ggpubr")
library(ggpubr)
ggscatter(toronto, x = "Robbery_AVG", y = "Homicide_AVG",
add = "reg.line", conf.int = TRUE,
cor.coef = TRUE, cor.method = "pearson",
xlab = "Robberies", ylab = "Homocide")
In the above plot, we can see the relationship plotted with a line. Additionally, the Pearson's r and the p-value are also plotted. The gray band around the line is the 95% confidence interval - the area in which we can be sure the true correlation resides.
Conclusion
Hopefully, its clear how the above can be helpful if you're looking for a relationship - rather than a difference - between two variables. Correlation tests give us ways to examine these relationships so we know that they are real. Additionally, because they rely on what we've done before, they are generally easy to understand. However, it is essential to make sure you know what's going on here, as correlation tests form the basis for the next major technique we're going to examine - and will examine for the remainder of the semester - regression.
LS0tCnRpdGxlOiAiQ29ycmVsYXRpb24gVGVzdHMiCm91dHB1dDogaHRtbF9ub3RlYm9vawotLS0KCiMgSW50cm9kdWN0aW9uCgpXZSBsZWFybmVkIGFib3V0IGNvcnJlbGF0aW9ucyBiYWNrIHdoZW4gd2Ugd2VyZSBzdGlsbCB3b3JraW5nIG9uIGRlc2NyaXB0aXZlIHN0YXRpc3RpY3MuIElmIHlvdSByZW1lbWJlciwgdGhleSB1c2UgdGhlIHN0YXRpc3RpYyBQZWFyc29uJ3Mgciwgd2hpY2ggcmVwcmVzZW50cyB0aGUgYW1vdW50IHRoYXQgdHdvIHZhcmlhYmxlcyAoY29udGludW91cykgdmFyeSB0b2dldGhlci4gVGhpcyBzdGF0aXN0aWMgcmFuZ2VzIGZyb20gLTEgdG8gMSwgd2l0aCByIHZhbHVlcyBjbG9zZXIgdG8gMSBvciAtMSByZXByZXNlbnRpbmcgc3Ryb25nZXIgcmVsYXRpb25zaGlwcy4gVGhlIHNpZ24gZGljdGF0ZXMgdGhlIGRpcmVjdGlvbiBvZiB0aGUgcmVsYXRpb25zaGlwLiBJbiB0aGUgcGxvdHMgYmVsb3csIHdlIGNhbiBzZWUgZXhhbXBsZXMgb2YgZWFjaCBvZiB0aGVzZSByZWxhdGlvbnNoaXAgdHlwZXMuCgpgYGB7cn0KdjE8LWMoMSwyLDMsNCw1KQp2MjwtYyg2LDcsOCw5LDEwKQp2MzwtYyg1LDQsMywyLDEpCnY0PC1jKDEsMSwxLDEsMSkKCmZha2U8LWRhdGEuZnJhbWUodjEsdjIsdjMsdjQpCgpwYXIobWZyb3c9YygyLDIpKQpwbG90KGZha2UkdjEsIGZha2UkdjIsIG1haW49IlBvc2l0aXZlIFJlbGF0aW9uc2hpcCIpCnBsb3QoZmFrZSR2MSwgZmFrZSR2MywgbWFpbj0iTmVnYXRpdmUgUmVsYXRpb25zaGlwIikKcGxvdChmYWtlJHYxLCBmYWtlJHY0LCBtYWluPSJObyBSZWxhdGlvbnNoaXAiKQpgYGAKCgpXZSBjYW4gYWxzbyBleGFtaW5lIHRoZSBhY3R1YWwgY29ycmVsYXRpb25zIGJldHdlZW4gZWFjaCBvZiB0aGVzZS5eW1NlZSBjb2RlIGZvciB0aGUgYWN0dWFsIHRlc3RzLl0KCmBgYHtyfQpwb3NpdGl2ZTwtY29yKGZha2UkdjEsIGZha2UkdjIpICMganVzdCB0aGUgY29ycmVsYXRpb24gY29lZmZpY2llbnQgLSBub3QgdGhlIHRlc3QKbmVnYXRpdmU8LWNvcihmYWtlJHYxLCBmYWtlJHYzKSAjIGp1c3QgdGhlIGNvcnJlbGF0aW9uIGNvZWZmaWNpZW50IC0gbm90IHRoZSB0ZXN0CnByaW50KHBhc3RlMCgiUG9zaXRpdmUgQ29ycmVsYXRpb24gPSAiLCBwb3NpdGl2ZSwiOyAiLCAiTmVnYXRpdmUgQ29ycmVsYXRpb24gPSAiLCBuZWdhdGl2ZSwgIjsgIiwgIk5vIENvcnJlbGF0aW9uID0gMCIpKSAjIHVzaW5nIGEgZmFuY3kgcHJpbnQgZnVuY3Rpb24gdG8gbWFrZSBpdCBsb29rIG5pY2UuCmBgYAoKU28sIGluIHRoaXMgZXhhbXBsZSwgd2UgY2FuIHNlZSB0aGF0IHRoZXJlIGlzIGEgX3BlcmZlY3RfIHBvc2l0aXZlIGNvcnJlbGF0aW9uLCBhIF9wZXJmZWN0XyBuZWdhdGl2ZSBjb3JyZWxhdGlvbiwgYW5kIGEgX3BlcmZlY3RfIGxhY2sgb2YgY29ycmVsYXRpb24uIEluIHRoZXNlIGNhc2VzLCBpdCdzIHByZXR0eSBlYXN5IHRvIHRlbGwgdGhhdCB0aGUgcmVsYXRpb25zaGlwICh3aGVuIHdlJ3JlIGxvb2tpbmcgYXQgY292YXJpYXRpb24sIHdlJ3JlIGxvb2tpbmcgYXQgcmVsYXRpb25zaGlwIHJhdGhlciB0aGFuIGRpZmZlcmVuY2UpIGlzIHZlcnkgbGlrZWx5IF9yZWFsXy4gSG93ZXZlciwgd2hhdCBpZiB0aGUgUGVhcnNvbidzIHIgd2FzIC4zNT8gV2hpbGUgdGhhdCBsb29rcyAibW9kZXJhdGUiIGJ5IHRoZSBzdGFuZGFyZHMgd2UgdXNlZCB3aGVuIHdlIGZpcnN0IHRhbGtlZCBhYm91dCBjb3JyZWxhdGlvbiwgYXMgd2Ugc2F3IHdpdGggQU5PVkEgYW5kIF90Xy10ZXN0cywgZWZmZWN0cyBzaXplcyBhcmUgbm90IHRoZSBzYW1lIGFzIHN0YXRpc3RpY2FsIHNpZ25pZmljYW5jZS4gU28sIHdoYXQgZ2l2ZXM/IEhvdyBhcmUgd2UgdG8ga25vdyBpZiBhIF9yZWxhdGlvbnNoaXBfIGlzIHJlYWw/CgojIEVudGVyIHRoZSBDb3JyZWxhdGlvbiBfVGVzdF8KClRoZSBnb29kIG5ld3MgaXMgdGhhdCBpdCdzIHJlYWxseSBlYXN5IHRvIHRlc3QgZm9yIGJpLXZhcmlhdGUgcmVsYXRpb25zaGlwcyBiZXR3ZWVuIGNvbnRpbnVvdXMgdmFyaWFibGVzIHVzaW5nIChiYXNpY2FsbHkpIHdoYXQgd2UndmUgYmVlbiBkb2luZyBhbGwgYWxvbmcuIEFkZGl0aW9uYWxseSwgd2UgdXNlIGVzc2VudGlhbGx5IHRoZSBzYW1lIHN0ZXBzIGFzIHdlIHVzZSB0byB0ZXN0IGZvciBkaWZmZXJlbmNlIChpbiB0ZXJtcyBvZiBzZXR0aW5nIGh5cG90aGVzZXMsIGV0Yy4pLCBzbyBpdCdzIHByZXR0eSBtdWNoIHdoYXQgeW91J3JlIHVzZWQgdG8gc2VlaW5nIGF0IHRoaXMgcG9pbnQuIEhvd2V2ZXIsIGl0IHNlcnZlcyBhcyB0aGUgYmFzaXMgZm9yIHVuZGVyc3RhbmRpbmcgcmVncmVzc2lvbiwgd2hpY2ggaXMgKl9mdW5kYW1lbnRhbF8qIHRvIHRoZSB0eXBlcyBvZiBhbmFseXNpcyB3ZSB3aWxsIGJlIGVuZ2FnaW5nIGluIGZvciB0aGUgcmVtYWluZGVyIG9mIHRoZSBzZW1lc3Rlci4gU28gaXQgaXMgX3JlYWxseV8gaW1wb3J0YW50IHRvIGdldCB0aGlzIHN0dWZmIGRvd24uCgojIyBDb3JyZWxhdGlvbiBUZXN0IGluIGEgTnV0c2hlbGwKCldoZW4gd2Ugc2F3IGFib3ZlIHRoYXQgdGhlIFBlYXJzb24ncyBjb3JyZWxhdGlvbiBjb2VmZmljaWVudHMgKHIpIGZvciB0aGUgY2hhcnRzIGFib3ZlIHdlcmUgMSwgLTEsIGFuZCAwLCAocmVzcGVjdGl2ZWx5KSwgd2hhdCBkb2VzIHRoYXQgYWN0dWFsbHkgbWVhbj8gV2VsbCwgaW4gc2hvcnQsIHdoYXQgd2UncmUgZG9pbmcgaXMgcHJldHR5IHN0cmFpZ2h0Zm9yd2FyZC4gSWYsIGFzIGEgdmFyaWFibGUgKHVzdWFsbHkgZGVub3RlZCAieCIpIGluY3JlYXNlcyBieSBhIHVuaXQgKHdoYXRldmVyIHVuaXQgaXQncyBtZWFzdXJlZCBpbiksIHRoZSBvdGhlciB2YXJpYWJsZSAodXN1YWxseSBkZW5vdGVkICJ5IikgYWxzbyBpbmNyZWFzZXMgYnkgYSB1bml0IChvciBtb3JlIC0gYWdhaW4sIGluIHdoYXRldmVyIHVuaXRzIGl0cyBtZWFzdXJlZCBpbikgX2VhY2ggdGltZV8sIHRoZW4gdGhlcmUgd2lsbCBiZSBhIHBlcmZlY3QgcmVsYXRpb25zaGlwLiBJbiBvdGhlciB3b3Jkcywgd2UncmUgbm90IG1lYXN1cmluZyB0aGUgdW5pdHMsIHdlJ3JlIG1lYXN1cmluZyB0aGUgbW92ZW1lbnQgYWNyb3NzIHRoZSBheGVzIGFzIHdlIGluY3JlYXNlIHZhbHVlcy4gCgpUaGUgZ29vZCBuZXdzIGlzIHRoYXQgbWFrZXMgY29ycmVsYXRpb24gdGVzdHMgdmVyeSBlYXN5IHRvIHVuZGVyc3RhbmQuIElmIHRoZSB5IHZhcmlhYmxlIGluY3JlYXNlcyB3aXRoIHRoZSB4IHZhcmlhYmxlJ3MgaW5jcmVhc2UgYXQgYSBoaWdoZXIgdGhhbiBleHBlY3RlZCByYXRlLCB3ZSBjYW4gc2F5IHRoZSByZWxhdGlvbnNoaXAgaXMgc2lnbmlmaWNhbnQuIEZvcnR1bmF0ZWx5LCBiZWNhdXNlIHdlIGhhdmUgdW5kZXJzdG9vZCB0aGF0IG1vc3QgZXZlcnl0aGluZyBjYW4gYmUgcmVkdWNlZCB0byBhIG5vcm1hbCBkaXN0cmlidXRpb24sIHdlIGNhbiBtZWFzdXJlIHRoZSBleHBlY3RhdGlvbiBvZiBjby12YXJpYXRpb24gdXNpbmcgYW4gYXBwcm94aW1hdGlvbiBvZiB0aGUgY3VydmUuIAoKTGV0J3MgdXNlIGEgInJlYWwgd29ybGQiIGV4YW1wbGUgdG8gYmV0dGVyIHVuZGVyc3RhbmQgdGhlIGNvcnJlbGF0dGlvbiB0ZXN0LiBBcyB1c3VhbCwgd2UgdmlzdWFsaXplIHRoZSBkYXRhIGZpcnN0LiBIZXJlIEknbSB1c2luZyB0aGUgbXRjYXJzIGRhdGFzZXQuIEluIHRoZSBjb2RlLCB5b3UnbGwgbm90aWNlIEkgdXNlIGEgZm9ybXVsYSB0byBjYWxjdWxhdGUgdGhlIGxpbmUgKGFuZCBwbG90IGl0IHVzaW5nIGFibGluZSgpKS4gV2Ugd2lsbCB0YWxrIHF1aXRlIGEgYml0IG1vcmUgYWJvdXQgdGhpcyB3aGVuIHdlIHRhbGsgYWJvdXQgcmVncmVzc2lvbi4gRm9yIG5vdywganVzdCB0aGluayBvZiBpdCBhcyBhIHNsb3BlLWludGVyY2VwdCBtb2RlbCAoaWYgeW91IHJlbWVtYmVyIGhpZ2ggc2Nob29sIC0geSA9IG0oeCkgKyBiKSwgb3IgZG9uJ3Qgd29ycnkgYWJvdXQgaXQgYXMgd2Ugd2lsbCBjb3ZlciBpdCBpbiBkZXRhaWwgbGF0ZXIuCgpgYGB7cn0Kc2xvcGU8LWxtKG10Y2FycyRtcGd+bXRjYXJzJHd0KSAjIENhbGN1bGF0ZSBhIGxpbmVhciBtb2RlbCAoZmFuY3ksIGZhbmN5KSB0byBnZXQgc2xvcGUgYW5kIGludGVyY2VwdApwbG90KG10Y2FycyR3dCwgbXRjYXJzJG1wZykgIyBQbG90IHRoZSByZWxhdGlvbnNoaXAKYWJsaW5lKHNsb3BlJGNvZWZmaWNpZW50c1sxXSwgc2xvcGUkY29lZmZpY2llbnRzWzJdKSAjIHBsb3QgdGhlIGxpbmUgZnJvbSB0aGUgc2xvcGUgYW5kIGludGVyY2VwdCBjYWxjdWFsdGVkIGFib3ZlCgpgYGAKCkZyb20gdGhlIGFib3ZlIHBsb3QsIGl0J3MgcHJldHR5IGNsZWFyIHRoYXQgdGhlcmUgaXMgYSBuZWdhdGl2ZSByZWxhdGlvbnNoaXAuIFdlIGNhbiBjb25maXJtIHRoaXMgYnkgY2FsY3VsYXRpbmcgdGhlIFBlYXJzb24ncyByIC0gYXMgd2UgaGF2ZSBkb25lIGZvciBhd2hpbGUgbm93ISAKCmBgYHtyfQpjb3IobXRjYXJzJHd0LCBtdGNhcnMkbXBnKQpgYGAKCkJhc2VkIG9uIHRoZSBhYm92ZSwgdGhlIHJlbGF0aW9uc2hpcCBpcyBjbGVhcmx5IG5lZ2F0aXZlIGFuZCBzdHJvbmcgLSBtYXRjaGluZyB3aGF0IHdlIGV4cGVjdCBmcm9tIG91ciBjaGFydC4gSG93ZXZlciwgaXMgaXQgX3JlYWxfPyBUbyBkZXRlcm1pbmUgdGhhdCwgd2UgdXNlIGEgbmV3IGZ1bmN0aW9uOiBjb3IudGVzdCgpLgoKYGBge3J9CmNvci50ZXN0KG10Y2FycyR3dCwgbXRjYXJzJG1wZykKYGBgCgpOb3csIGEgY291cGxlIG9mIHRoaW5ncyB0byBub3RpY2UuIEZpcnN0LCBvdGhlciB0aGFuIGluY2x1ZGluZyAiLnRlc3QiIHRoZSBSIGNvZGUgaXMgZXhhY3RseSB0aGUgc2FtZS4gV2h5PyBCZWNhdXNlIGluIG9yZGVyIHRvIGNhbGN1bGF0ZSB0aGUgdGVzdCwgd2UgZmlyc3QgbmVlZCB0byBjYWxjdWxhdGUgZXhhY3RseSB0aGUgc2FtZSBQZXJzb24ncyByIHRoYXQgd2UgY2FsY3VsYXRlZCBiZWZvcmUhIFNlY29uZCwgd2UgY2FuIHNlZSB0aGF0IHRoZXJlIGlzIGFkZGl0aW9uYWwgb3V0cHV0IHRoYXQgd2UgZG9uJ3QgZ2V0IHdpdGggdGhlIGNvcigpIGZ1bmN0aW9uIC0gYW5kIG11Y2ggb2YgaXQgbG9va3MgZmFtaWxpYXIhIFRoYXQncyBiZWF1c2UgeW91J3ZlIHNlZW4gaXQgYmVmb3JlLiBXZSB1c2UgYSBfdF8tdGVzdCB0byBkZXRlcm1pbmUgd2hldGhlciB0aGUgcmVsYXRpb25zaGlwIGJldHdlZW4gdmFyaWFibGVzIGlzIHJlYWwhIFRoZSBzYW1lIF90Xy10ZXN0IHdlIHVzZWQgYmVmb3JlIQoKWW91IG1pZ2h0IGJlIHRoaW5raW5nLCAiSSB0aG91Z2h0IHQtdGVzdHMgd2VyZSBmb3IgZGlmZmVyZW5jZSBiZXR3ZWVuIGdyb3VwcywgYnV0IGhlcmUgd2UncmUgbG9va2luZyBhdCByZWxhdGlvbnNoaXBzLCBzbyB3aGF0IGdpdmVzPyIgVGhlIHQtdGVzdCBmb3IgY29ycmVsYXRpb24gY29lZmZpY2llbnRzIF9pc18gYWN0dWFsbHkgdGVzdGluZyBmb3IgYSBkaWZmZXJlbmNlIC0gYnV0IG9uZSB0aGF0IGltcGxpZXMgcmVsYXRpb25zaGlwIGlmIHRoZSBkaWZmZXJlbmNlIGV4aXN0cy4gU3BlY2lmaWNhbGx5LCB0aGUgdC10ZXN0IGhlcmUgaXMgdGVzdGluZyB0byBzZWUgaWYgdGhlIGNvcnJlbGF0aW9uIGNvZWZmaWNpZW50IChQZXJzb24ncyByKSBpcyBzaWduaWZpY2FudGx5IGRpZmZlcmVudCBmcm9tIDAuIFRoZSBudWxsIGh5cG90aGVzaXMsIGluIHRoaXMgY2FzZSwgaXMgdGhhdCBQZWFyc29uJ3MgciBpcyA9IDAuIFdlIGNhbiByZWplY3QgaWYgdGhlIHQtdGVzdCB0ZWxscyB1cyB0aGF0IGl0IGlzIGVpdGhlciA+IG9yIDwgMCAtIHdpdGggYW4gYWxwaGEgb2YgLjA1LiAKCldpdGggdGhhdCBpbmZvLCB5b3UgY2FuIHJlYWQgdGhlIG91dHB1dCBhYm92ZS4gV2UgaGF2ZSBhIHQtdmFsdWUgb2YgLTkuNTk5IChkZiA9IDMwKSwgYW5kIHAgPCAuMDUuIFRoaXMgbWVhbnMgKGFzIHVzdWFsKSB3ZSBjYW4gcmVqZWN0IHRoZSBudWxsIGh5cG90aGVzaXMgYW5kIHRoYXQgdGhlcmUgaXMgYSByZWFsIF9yZWxhdGlvbnNoaXBfIGJldHdlZW4gdGhlIHZhcmlhYmxlcy4gU3BlY2lmaWNhbGx5LCBhcyB3ZWlnaHQgaW5jcmVhc2VzLCBtaWxlcy1wZXItZ2FsbG9uIGRlY3JlYXNlcy4gVGhlIDk1JSBjb25maWRlbmNlIGludGVydmFsIGlzIHRoZSByYW5nZSB3aGljaCB3ZSBjYW4gYmUgOTUlIHN1cmUgdGhhdCB0aGUgInRydWUgUGVhcnNvbidzIHIiIGZhbGxzIHdpdGhpbi4gVGhlIGJvdHRvbSBudW1iZXIsIHVuZGVyICJjb3IiLCBpcyB0aGUgUGVhcnNvbidzIHIgdmFsdWUgd2UncmUgdXNlZCB0byBnZXR0aW5nIGZyb20gY29yKCksIGFzIHlvdSBjYW4gc2VlIGJlbG93LgoKYGBge3J9CmNvcihtdGNhcnMkd3QsIG10Y2FycyRtcGcpCmBgYAoKIyMjIEVmZmVjdHMgc2l6ZXMKCldoaWxlIFBlYXJzb24ncyByIGl0c2VsZiBmdW5jdGlvbnMgYXMgYW4gaW5leGFjdCBtZWFzdXJlIG9mIGVmZmVjdHMgc2l6ZSwgd2UgaGF2ZSBhIG1vcmUgc3BlY2lmaWMsIGFuZCB1c2VmdWwgbWVhc3VyZSBpbiByLXNxdWFyZWQgKGFsc28gY2FsbGVkIFItc3F1YXJlZCwgdGhvdWdoIHRlY2huaWNhbGx5IHRoYXQncyByZXNlcnZlZCBmb3IgcmVncmVzc2lvbiwgYnV0IGRvbid0IHdvcnJ5IGFib3V0IGl0KS4gSXQgaXMgY2FsY3VsYXRlZCBieSBzaW1wbHkgc3F1YXJpbmcgdGhlIFBlYXJzb24ncyByLiAKCmBgYHtyfQpjb3IobXRjYXJzJHd0LCBtdGNhcnMkbXBnKV4yCmBgYAoKVGhlIGNvZGUgYWJvdmUgaXMgcmVseWluZyBvbiBjb3IoKSB0byBnZXQgci1zcXVhcmVkLiBXZSBjYW4gYWxzbyBnZXQgaXQgZnJvbSBjb3IudGVzdCgpIGRpcmVjdGx5LCB3aXRoIGEgc21hbGwgYW1vdW50IG9mIGFkZGl0aW9uYWwgY29kZS4KCmBgYHtyfQptcGdfd3Q8LWNvci50ZXN0KG10Y2FycyR3dCwgbXRjYXJzJG1wZykgIyBUaGlzIGNyZWF0ZXMgYSAic3BlY2lhbCIgb2JqZWN0IHdpdGggbXVsdGlwbGUgcGFydHMKbXBnX3d0JGVzdGltYXRlXjIgIyB3ZSBjYW4gYWNjZXNzIGl0IHVzaW5nICQganVzdCBsaWtlIGEgZGF0YWZyYW1lIG9iamVjdCAtIGluIHRoaXMgY2FzZSB3ZSB3YW50ICJlc3RpbWF0ZSIKYGBgCgpXaGF0IHRoZSByLXNxdWFyZWQgdmFsdWUgdGVsbHMgdXMgaXMgdGhhdCB0aGUgeC12YXJpYWJsZSwgaW4gdGhpcyBjYXNlIHdlaWdodCwgZXhwbGFpbnMgNzUlIG9mIHRoZSB2YXJpYXRpb24gaW4gdGhlIHktdmFyaWFibGUsIG1wZy4gVGhpcyBpcyBtdWNoIG1vcmUgaW50dWl0aXZlLCBhcyBpZiB3ZSBjYW4gZXhwbGFpbiA3NSUgb2Ygc29tZXRoaW5nIHdpdGggYSBzaW5nbGUgdmFyaWFibGUsIHdlJ3JlIGRvaW5nIHByZXR0eSBnb29kISBGb3IgY29udGV4dCwgaW1hZ2luZSB3ZSBjb3VsZCBleHBsYWluIDc1JSBvZiBjcmltZSB3aXRoIHNvbWV0aGluZyBsaWtlIGRlcHJlc3Npb24uIElmIHdlIGdhdmUgZXZlcnlvbmUgZGVwcmVzc2lvbiBtZWRzLCB3ZSB3b3VsZCBzb2x2ZSA3NSUgb2YgY3JpbWUhCgojIFJlYWwgV29ybGQgRXhhbXBsZQoKSSdtIGdvaW5nIHRvIHVzZSBzb21lIGRhdGEgdGhhdCBJIGZvdW5kIGZyb20gVG9yb250bydzIE9wZW4gRGF0YSBwb3J0YWwuIFNwZWNpZmljYWxseSwgSSdtIHVzaW5nIG5laWdoYm9yaG9vZCBjcmltZSByYXRlcyBmcm9tOiBodHRwczovL29wZW4udG9yb250by5jYS9kYXRhc2V0L25laWdoYm91cmhvb2QtY3JpbWUtcmF0ZXMvLiBUaGUgZGF0YSBjb25zaXN0cyBvZiBhIC5jc3YgZmlsZSB3aXRoIDE0MCByb3dzLCAxIHBlciBuZWlnaGJvcmhvb2QuIEJlbG93IHlvdSBjYW4gc2VlIHRoZSBmaXJzdCBmZXcgcm93cyBvZiB0aGUgZGF0YXNldC4KCmBgYHtyfQp0b3JvbnRvPC1yZWFkLmNzdigiaHR0cHM6Ly9ja2FuMC5jZi5vcGVuZGF0YS5pbnRlci5wcm9kLXRvcm9udG8uY2EvZG93bmxvYWRfcmVzb3VyY2UvM2Q1NTZmYzItZGRhYi00YWEwLTk3ZTEtMjI3NzA3NTgwZWM2P2Zvcm1hdD1jc3YmcHJvamVjdGlvbj00MzI2IikgIyBub3RpY2UgSSBqdXN0IHVzZWQgdGhlIHdlYi1hZGRyZXNzIGZvciB0aGUgQ1NWIGZpbGUsIHNhdmluZyBhIHN0ZXAKCmhlYWQodG9yb250bykKCmBgYAoKRnJvbSB0aGUgYWJvdmUsIHlvdSBjYW4gc2VlIHRoYXQgdGhlcmUgYXJlIGEgbnVtYmVyIG9mIGNyaW1lLXJlbGF0ZWQgdmFyaWFibGVzIGF0IHRoZSBpbnRlcnZhbC1yYXRpbyBsZXZlbCBvZiBtZWFzdXJlbWVudCAoY29udGlub3VzKS4gSSdtIHBhcnRpY3VsYXJseSBpbnRlcmVzdGVkIGluIHRoZSByZWxhdGlvbnNoaXAgb2YgY3JpbWUgdHlwZXMgYWNyb3NzIG5laWdoYm9yaG9vZHMuIFNwZWNpZmljYWxseSwgSSB3YW50IHRvIGtub3cgaWYgaG9tb2NpZGUgYW5kIHJvYmJlcnkgYXJlIHJlbGF0ZWQuIFdoaWxlIEkgY291bGQgY29tcGFyZSBsaWtlLXllYXJzIChlLmcuIDIwMTYgdG8gMjAxNiksIEkgc2VlIHRoYXQgdGhlcmUgaXMgYW4gImF2ZXJhZ2UiIGNvbHVtbiBmb3IgZWFjaCBjcmltZSB0aW1lLCBhbmQgZ2l2ZW4gdGhlIHJlbGF0aXZlbHkgbG93IG51bWJlciBvZiBob21vY2lkZSBlYWNoIHllYXIsIGl0IGlzIGxpa2VseSBiZXR0ZXIgdG8gdXNlIHRoZSBhdmVyYWdlIGNvbHVtbiBmb3IgZWFjaC4KCkZpcnN0LCBJIHdhbnQgdG8gdmlzdWFsaXplIG15IHJlbGF0aW9uc2hpcC4KCmBgYHtyfQpwbG90KHRvcm9udG8kSG9taWNpZGVfQVZHLCB0b3JvbnRvJFJvYmJlcnlfQVZHLCB4bGFiID0gIkhvbW9jaWRlcyIsIHlsYWI9IlJvYmJlcmllcyIpCmBgYAoKRXhhbWluaW5nIHRoZSBwbG90IGFib3ZlLCBpdCBsb29rcyBsaWtlIHRoZXJlIGlzIGEgcG9zaXRpdmUgcmVsYXRpb25zaGlwIGJldHdlZW4gaG9tb2NpZGVzIGFuZCByb2JlcmllcyB3aXRoaW4gbmVpZ2hib3Job29kcy4gVG8gdGVzdCB0aGlzLCBJIG5lZWQgdG8gZmlyc3QgZXN0YWJsaXNoIG15IGh5cG90aGVzZXM6CgoqKkh+MH4qKjogJHIgPSAwJCBpcyB0aGUgbnVsbCBoeXBvdGhlc2lzIChQZWFyc29uJ3MgciBpcyA9IDApCgoqKkh+MX4qKjogJHIgXG5lcSAwJCBpcyB0aGUgcmVzZWFyY2ggaHlwb3RoZXNpcyAoUGVyc29uJ3MgciBub3QgZXF1YWwgdG8gMCkKCkhhdmluZyBkb25lIHRoYXQsIGFzIHdpdGggdC10ZXRzdHRzLCBJIGNhbiAicm9sbCIgdGhlIHJlc3Qgb2YgdGhlIHN0ZXBzIHRvZ2V0dGhlciBhbmQgdGVzdCB1c2luZyBjb3IudGVzdCgpLgoKYGBge3J9CmNvci50ZXN0KHRvcm9udG8kSG9taWNpZGVfQVZHLCB0b3JvbnRvJFJvYmJlcnlfQVZHKQpgYGAKCkV4YW1pbmluZyB0aGUgYWJvdmUgb3V0cHV0LCBpdCBpcyBjbGVhciB0aGF0IHRoZXJlIGlzIGEgc3RhdGlzdGljYWxseSBzaWduaWZpY2FudCByZWxhdGlvbnNoaXAgYmV0d2VlbiBob21vY2lkZSBhbmQgcm9iYmVyaWVzIHdpdGhpbiBuZWlnaGJvcmhvb2RzIGluIFRvcm9udG8gKHQgPSAtMTEuMTk5KDEzOCksIHAgPCAuMDUpLiBUaGlzIHJlbGF0aW9uc2hpcCBhcHBlYXJzIHRvIGJlIGZhaXJseSBzdHJvbmcgKFBlYXJzb24ncyByID0gLjY5KSBhbmQgcG9zaXRpdmUgLSBuaWdoYm9yaG9vZHMgdGVuZCB0byBoYXZlIGhpZ2ggcmF0ZXMgb2Ygb25lIGlmIHRoZXkgaGF2ZSBoaWdoIHJhdGVzIG9mIHRoZSBvdGhlci4gVGhlIHItc3F1YXJlZCB2YWx1ZSBpcyAuNDgsIGluZGljYXRpbmcgdGhhdCBhYm91dCA0OCUgb2YgdGhlIHZhcmlhdGlvbiBpbiBob21vY2lkZSBpcyBleHBsYWluZWQgYnkgcm9iYmVyaWVzIHdpdGhpbiB0aGUgbmVpZ2hib3Job29kcy4KCmBgYHtyLCBpbmNsdWRlID0gRkFMU0V9CmhvbV9yb2I8LWNvci50ZXN0KHRvcm9udG8kSG9taWNpZGVfQVZHLCB0b3JvbnRvJFJvYmJlcnlfQVZHKQpob21fcm9iJGVzdGltYXRlXjIKYGBgCgpXZSBjYW4gZnVydGhlciB2aXN1YWxpemUgdGhpcyByZWxhdGlvbnNoaXAgYnkgZXhhbWluaW5nIHRoZSBzYW1lIHBsb3QgYXMgYWJvdmUsIHdpdGggc29tZSBhZGRpdGlvbmFsIGluZm9ybWF0aW9uIGluY2x1ZGVkLgoKYGBge3J9CiNpbnN0YWxsLnBhY2thZ2VzKCJnZ3B1YnIiKQpsaWJyYXJ5KGdncHVicikKZ2dzY2F0dGVyKHRvcm9udG8sIHggPSAiUm9iYmVyeV9BVkciLCB5ID0gIkhvbWljaWRlX0FWRyIsIAogICAgICAgICAgYWRkID0gInJlZy5saW5lIiwgY29uZi5pbnQgPSBUUlVFLCAKICAgICAgICAgIGNvci5jb2VmID0gVFJVRSwgY29yLm1ldGhvZCA9ICJwZWFyc29uIiwKICAgICAgICAgIHhsYWIgPSAiUm9iYmVyaWVzIiwgeWxhYiA9ICJIb21vY2lkZSIpCmBgYAoKSW4gdGhlIGFib3ZlIHBsb3QsIHdlIGNhbiBzZWUgdGhlIHJlbGF0aW9uc2hpcCBwbG90dGVkIHdpdGggYSBsaW5lLiBBZGRpdGlvbmFsbHksIHRoZSBQZWFyc29uJ3MgciBhbmQgdGhlIHAtdmFsdWUgYXJlIGFsc28gcGxvdHRlZC4gVGhlIGdyYXkgYmFuZCBhcm91bmQgdGhlIGxpbmUgaXMgdGhlIDk1JSBjb25maWRlbmNlIGludGVydmFsIC0gdGhlIGFyZWEgaW4gd2hpY2ggd2UgY2FuIGJlIHN1cmUgdGhlIHRydWUgY29ycmVsYXRpb24gcmVzaWRlcy4KCiMgQ29uY2x1c2lvbgpIb3BlZnVsbHksIGl0cyBjbGVhciBob3cgdGhlIGFib3ZlIGNhbiBiZSBoZWxwZnVsIGlmIHlvdSdyZSBsb29raW5nIGZvciBhIHJlbGF0aW9uc2hpcCAtIHJhdGhlciB0aGFuIGEgZGlmZmVyZW5jZSAtIGJldHdlZW4gdHdvIHZhcmlhYmxlcy4gQ29ycmVsYXRpb24gdGVzdHMgZ2l2ZSB1cyB3YXlzIHRvIGV4YW1pbmUgdGhlc2UgcmVsYXRpb25zaGlwcyBzbyB3ZSBrbm93IHRoYXQgdGhleSBhcmUgX3JlYWxfLiBBZGRpdGlvbmFsbHksIGJlY2F1c2UgdGhleSByZWx5IG9uIHdoYXQgd2UndmUgZG9uZSBiZWZvcmUsIHRoZXkgYXJlIGdlbmVyYWxseSBlYXN5IHRvIHVuZGVyc3RhbmQuIEhvd2V2ZXIsIGl0IGlzIGVzc2VudGlhbCB0byBtYWtlIHN1cmUgeW91IGtub3cgd2hhdCdzIGdvaW5nIG9uIGhlcmUsIGFzIGNvcnJlbGF0aW9uIHRlc3RzIGZvcm0gdGhlIGJhc2lzIGZvciB0aGUgbmV4dCBtYWpvciB0ZWNobmlxdWUgd2UncmUgZ29pbmcgdG8gZXhhbWluZSAtIGFuZCB3aWxsIGV4YW1pbmUgZm9yIHRoZSByZW1haW5kZXIgb2YgdGhlIHNlbWVzdGVyIC0gcmVncmVzc2lvbi4=
Comments
Post a Comment