c++ – How to store a vector of smart pointers, except some of them are owned by another object?

I’m making a basic platformer game. I have a Game class as well as Level class. The game object holds a pointer to the current Level object. A level currently has a std::vector of GameObject raw pointers, and normally it handles the destruction of the GameObjects. However, some of them are Player objects (a derived class from GameObject). These should not be deleted when the level is, as they are owned by the Game.

My current Level destructor looks like this:

Level::~Level() {
    for (GameObject* obj : gameObjects) {
        if (!obj->livesPastLevel()) {
            delete obj;

I want to convert from raw pointers to smart pointers. Changing the Game‘s pointer to a level into a unique_ptr is easy. However, I’m having trouble thinking of a good way to handle the Level’s gameObject* vector.

I can’t just use unique_ptrs because then the Player would get deleted when the Level is destroyed. I know the normal solution would just be to use a shared_ptr, but these seems like overkill. The majority of the pointers will just have one owner (the level) and it would be needlessly slow to reference count those with shared_ptr.

Is there any good way to use a vector<unique_ptr> except a few of them are actually owned by a different object? Like setting unique_ptr.autoCleanup=false. Or is there another pointer type I should use to handle that?

Or perhaps the overhead of shared_ptr is just something I have to deal with. It’s also possible that I am structuring this whole resource management incorrectly.

So how can I use smart pointers when most (but not all) would be usable as unique_ptr?