python – 403 Request had insufficient authentication issues while accessing Secrets on GCP within a container

I am trying to access a secret on GCP Secrets and I get the following error :

in get_total_results "api_key": get_credentials("somekey").get("somekey within key"), File
 "/helper.py", line 153, in get_credentials response = client.access_secret_version(request={"name": resource_name})
 File "/usr/local/lib/python3.8/site-packages/google/cloud/secretmanager_v1/services/secret_manager_service/client.py", 
line 1136, in access_secret_version response = rpc(request, retry=retry, timeout=timeout, metadata=metadata,) 
File "/usr/local/lib/python3.8/site-packages/google/api_core/gapic_v1/method.py", line 145, in __call__ 
return wrapped_func(*args, **kwargs) File "/usr/local/lib/python3.8/site-packages/google/api_core/retry.py", line 285, in retry_wrapped_func return retry_target( File "/usr/local/lib/python3.8/site-packages/google/api_core/retry.py", 
line 188, in retry_target return target() File "/usr/local/lib/python3.8/site-packages/google/api_core/grpc_helpers.py", 
line 69, in error_remapped_callable six.raise_from(exceptions.from_grpc_error(exc), exc) File "<string>", 
line 3, in raise_from google.api_core.exceptions.PermissionDenied: 
403 Request had insufficient authentication scopes.

The code is fairly simple:-

def get_credentials(secret_id):
    project_id = os.environ.get("PROJECT_ID")
    resource_name = f"projects/{project_id}/secrets/{secret_id}/versions/1"

    client = secretmanager.SecretManagerServiceClient()
    response = client.access_secret_version(request={"name": resource_name})

    secret_string = response.payload.data.decode("UTF-8")
    secret_dict = json.loads(secret_string)
    return secret_dict

So, what I have is a cloud function, which is deployed using Triggers, and uses a service account which has the Owner role.

The cloud function triggers a Kubernete Work Job and creates a container, which downloads a repo inside the container and executes it.

Dockerfile is:

FROM gcr.io/project/repo:latest
FROM python:3.8-slim-buster
COPY . /some_dir
WORKDIR /some_dir
COPY --from=0 ./repo /a_repo
RUN pip install -r requirements.txt & pip install -r a_repo/requirements.txt 
ENTRYPOINT ("python3" , "main.py")