"Once a colony have been founded outside your culture-borders, that colony-square is automatically harvested, once road-connected to a city. That mobile worker-unit have in effect been automatically transformed into a rooted 1 pop colony-workforce. So you dont lose anything. Yes, your city-population has decreased 1 pop, but that 1 pop have just emigrated to a colony - you still rule and exploit them.
Now, once your culture-borders had swallowed up that colony, the former colony resource-tile is still harvested (because its still road-connected). Nothing have changed. There is no "lost" 1 pop to regain - its a mirage. The only thing that happened is that the colony-graphics have dissapeared on the map, because allowing them visible would clutter up the map too much later then more special resources/luxuries is uncovered by new techs.

Infact: you can regard all road-connected special resource/luxury squares within your culture-borders as "invisibe colonies" (or suburbs, if your prefer that) - still alive and kicking. In terms of under-the-hood game-mechanics, they still are. Wheter visible (and later invisible) colonies are reflected on an indevidual city pop-points level, is unimportant. The important thing is that they are reflected on an overal empire pop-points level.

In other words: