Prediction Errors

We assume the train and test sets have a balanced sample of difficult-to-predict-examples for a given model and performance metric.

Is there evidence that examples in the train set and test set are equally challenging to predict?

1. Are there examples in the train set or the test set that are always predicted incorrectly?
2. Are there domain segments of the train set or the test set that the are harder to predict than other segments?
3. Are there some classes that are more challenging to predict than others on the train or test sets (classification only)?
4. Are there deciles of the target variable that are more challenging to predict than others on the train or test sets (regression only)?