In a large mixing bowl combine sour cream, cream cheese, cheddar cheese, salt, and pepper. Stir until well combined. Add potatoes and 3/4 of chopped bacon. Mix until well combined.
Pour into a greased (6"x10") baking dish. Cover with aluminum foil and bake for 60-75 minutes, until potatoes are fork-tender. Remove from the oven. Stir. Sprinkle Colby Jack cheese over top and bake for 15 more minutes uncovered.
Sprinkle casserole with reserved bacon and scallions.