Ask Your Neighbors, They Know Us