Hello, I'm Maneshwar, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems. Star us to help devs discover the project, give it a try, and share your feedback to help improve the product.

Here is a classic microservices interview question.

"One of your services should only accept calls from other internal services. How do you secure it?"

Most people answer in one line.

"Put it in a private network, no public IP."

That answer is correct.

It is also about 25% of the answer, and the other 75% is where the interview is actually won.

So let's take the question apart the way an attacker would, one layer at a time.

The setup: a payments service with a secret

Make it concrete.

A user places an order. The order service calls the payments service to charge the card.

That is the only legitimate way payments is ever used.

No user should talk to it directly. Nothing on the internet should even know it exists.

So the goal is not "make payments secure".

The goal is narrower and much more useful: make payments answer exactly one caller, and prove who that caller is.

Hold on to that sentence, because every layer below is an attempt to make it true.

Layer 1: no public address

Start with the thing everyone says.

In Kubernetes, a Service defaults to type ClusterIP, which gets a virtual IP that is only routable inside the cluster.

So you do not give payments a LoadBalancer or a NodePort, and you make sure no Ingress rule points at it.

From the outside, the service simply does not exist. There is nothing to scan and nothing to brute force.

That is a genuinely good first move, and it kills the laziest class of attack, which is "somebody found the URL".

Now look at what it does not do.

Inside the cluster, the network is flat by default.

Every pod can open a connection to every other pod, and there is no door between them.

Your marketing website runs in the same cluster. It has a CMS, three plugins nobody updated, and a contact form written during a hackathon.

An attacker finds one bug in it.

Congratulations, they are now inside your private network, and payments will happily answer them, because from payments' point of view the request came from a neighbour.

"Private" is a statement about strangers.

It says nothing about tenants.

Layer 2: a firewall for pods

The fix for a flat network is to stop it being flat.

Kubernetes has NetworkPolicy for exactly this. Think of it as a firewall rule that follows pods around.

The pattern is two policies. First close everything, then name exactly who gets in:

# 1. the namespace starts closed
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
  namespace: shop
spec:
  podSelector: {}
  policyTypes: [Ingress]
---
# 2. payments accepts order pods, on its own port, and nothing else
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: payments-from-order
  namespace: shop
spec:
  podSelector:
    matchLabels: { app: payments }
  policyTypes: [Ingress]
  ingress:
    - from:
        - podSelector:
            matchLabels: { app: order }
      ports:
        - protocol: TCP
          port: 8080

Now the compromised marketing pod can still open a socket, but its packets are dropped on the way in.

One trap here, and it is a nasty one.

A NetworkPolicy is only enforced if your network plugin enforces it.

The Kubernetes docs are blunt about this: if the plugin does not implement policies, creating the object has no effect.

Calico and Cilium enforce them. Some setups do not, or need a switch flipped on.

So you can write a lovely allow list, kubectl apply it, get a calm green "created", and be protected by absolutely nothing.

Test that it actually drops traffic. We will do that at the end.

The blind spot

Even when it is enforced, a network policy has a fundamental limit.

It decides based on where a packet came from, not who actually sent it.

It is checking the address on the envelope, and says nothing about the person who wrote the letter.

In practice that breaks down in three ways:

The label is the identity, and labels are cheap.
Anyone who can create a pod with app: order in that namespace walks straight through the allow list. A policy trusts the label, and nothing verifies the label.

It is blind above the network layer.
It knows "TCP, port 8080". It has no idea whether the request is POST /charge or DELETE /everything.

It does not encrypt anything.
Inside the cluster, the bytes are readable by anyone who can sniff the node's traffic.

Pod IPs also get recycled constantly, so an address that was the order service a minute ago can belong to a different pod now.

That is why a good policy matches on labels instead of raw IPs.

But the deeper problem is the one above: there is no proof attached to the claim.

A network policy is great at shrinking the blast radius.

It is not great at proving identity.

For that, you need ID cards.

Layer 3: mutual TLS

Regular TLS is what your browser does. The server shows a certificate, and you check it.

In mutual TLS, the client shows one too, and the server checks it before any data moves.

So when order calls payments:

• order proves "I am the order service"
• payments proves "I am the payments service"
• both are verified against the same trusted authority, and only then does the request go through

An attacker sitting inside the network can reach the port. They cannot forge a certificate signed by your authority, so the handshake fails and no request is ever read.

That one change moves you from "trust the address" to "trust the proof".

Who hands out the certificates?

This is the practical question that makes most people wave their hands.

Every service needs its own certificate, issued by one authority that you run inside the cluster.

The identity inside it is usually a SPIFFE style name, and in a mesh it is derived from the pod's Kubernetes ServiceAccount, something like spiffe://cluster.local/ns/shop/sa/order.

Two details matter more than they look:

The certificates are short-lived.
Hours, not years. A stolen cert is a very temporary problem, and nobody has a spreadsheet of expiry dates to forget about.

The identity is the ServiceAccount.
If every deployment in your cluster runs as the default ServiceAccount, they all share one identity, and your fancy mTLS can no longer tell them apart. Give each service its own.

You do not write any of this

Nobody hand-rolls certificate rotation inside every service.

You use a service mesh, such as Istio or Linkerd.

The mesh injects a small proxy, a sidecar, next to every pod. That proxy fetches the certificate, presents it, verifies the peer's, rotates it before it expires, and encrypts the traffic.

Your application keeps speaking plain HTTP to localhost. It never sees a certificate.

(Sidecars are not the only shape. Istio also has an ambient mode that moves the proxying into a per-node component, so the idea survives, the extra container goes away.)

Here is one call, start to finish, as the time-ordered version of that picture:

sequenceDiagram
    participant CA as Cluster CA
    participant OP as order sidecar
    participant PP as payments sidecar
    participant P as payments app

    OP->>CA: prove who I am (ServiceAccount token)
    CA-->>OP: short-lived cert for sa/order
    PP->>CA: prove who I am (ServiceAccount token)
    CA-->>PP: short-lived cert for sa/payments
    Note over OP,PP: later, the order app makes a call
    OP->>PP: TLS handshake, here is my cert
    PP-->>OP: and here is mine
    Note over OP,PP: both verify against the CA root
    OP->>PP: POST /charge, encrypted
    PP->>P: plain request on localhost
    P-->>OP: 200, back through both proxies

The gotcha that gets teams

Installing a mesh does not mean you are enforcing mTLS.

Istio's default mode is PERMISSIVE, which accepts both mTLS and plain text, so that you can migrate without breaking things.

That is a sensible default for rollout.

It is also a state in which an attacker's plain curl still works, while your dashboard cheerfully says "mesh enabled".

You have to flip payments to STRICT on purpose. The config for that is in the next section.

Layer 4: authenticated is not the same as allowed

Now every caller has a verified identity.

So the next question is the one most people forget: should this caller be allowed to do this thing?

The order service may call payments.

It has no business calling the refund service, because refunds belong to the support service.

Here is why this matters. Say the marketing site also runs in the mesh. It has a sidecar, so it has a perfectly valid certificate.

mTLS says: "yes, that really is the marketing service."

It does not say: "and the marketing service may charge cards."

That second sentence is an authorization policy, and the same mesh enforces it:

apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: default
  namespace: shop
spec:
  mtls:
    mode: STRICT          # the default is PERMISSIVE, which still accepts plaintext
---
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: payments-allow-order
  namespace: shop
spec:
  selector:
    matchLabels:
      app: payments
  action: ALLOW
  rules:
    - from:
        - source:
            principals: ["cluster.local/ns/shop/sa/order"]
      to:
        - operation:
            methods: ["POST"]
            paths: ["/charge"]

Read the rules block out loud.

"The identity sa/order may POST to /charge on payments."

Once an ALLOW policy exists for a workload, anything that does not match a rule is denied. You get default-deny for free, at the level of identity and URL rather than IP and port.

The proxy next to payments checks this on every single request.

This is least privilege, applied between services instead of between people.

If someone steals the order service's identity, they can charge cards. They still cannot issue refunds, because that was never in its rules.

Linkerd has the same idea under different names, a Server plus an AuthorizationPolicy that binds identities to routes. The shape transfers, the YAML does not.

Proving it, because "created" is not "enforced"

Every one of these layers fails differently, and that is a feature.

Run one request from a pod you are not supposed to trust, and the way it fails tells you which gate caught it:

kubectl exec deploy/marketing -- \
  curl -sS -m 3 -X POST http://payments.shop:8080/charge

# NetworkPolicy dropped it   -> curl: (28) Connection timed out after 3001 milliseconds
# mTLS STRICT refused it     -> curl: (56) Recv failure: Connection reset by peer
# AuthorizationPolicy said no -> RBAC: access denied   (HTTP 403)

Those are the usual signatures. The exact text shifts a little between versions and tools, but the character is stable: a hang, a reset, a polite 403.

flowchart TD
    A[a call to payments fails] --> B{what does the<br/>client actually see?}
    B -->|name does not resolve,<br/>or no route at all| C[ClusterIP: you are outside the cluster]
    B -->|hangs, then times out| D[NetworkPolicy dropped the packets]
    B -->|connection reset| E[mTLS: no valid cert,<br/>or plaintext sent to a STRICT peer]
    B -->|HTTP 403 RBAC access denied| F[AuthorizationPolicy: valid identity,<br/>not allowed to do this]
    B -->|HTTP 200| G[all four gates said yes]

    classDef decision fill:#f4d35e,stroke:#b8991f,color:#1a1a1a
    classDef start fill:#e9ecef,stroke:#6c757d,color:#1a1a1a
    classDef block fill:#ff6b6b,stroke:#c94a4a,color:#1a1a1a
    classDef ok fill:#5ee6c8,stroke:#1f9c86,color:#1a1a1a

    class B decision
    class A start
    class C,D,E,F block
    class G ok

One honest caveat on that chart.

A timeout is ambiguous. "No route" and "dropped by policy" can both look like a hang from the client side.

That is why you test from inside a pod that should be blocked, and from a pod that should be allowed. Both directions. If the blocked one succeeds, your network plugin is not enforcing anything.

Also note the marketing pod, if it is in the mesh, will sail past gate three and be stopped at gate four. A request that dies at the last gate is not a failure of the earlier ones, it is the reason the last one exists.

Putting it together

Here are the four gates in one picture:

Each one stops a different mistake:

• ClusterIP stops the internet. It sees: no route.
• NetworkPolicy stops other pods. It sees: a timeout.
• mTLS stops pods with no identity. It sees: a reset.
• AuthorizationPolicy stops the wrong service. It sees: a 403.

The attacker has to beat all four.

You only need one of them to hold.

That is the actual interview answer: defense in depth, where no single layer has to be perfect, because each is covering for a hole in the one before it.

What is still left (the part that gets you the senior grade)

Four gates is a strong answer. It is not the end of the story, and it helps to say so.

mTLS authenticates services, not users.
If the order service is compromised, or has an SSRF bug, it can ask payments to charge any card, and every gate above will wave it through, because it really is the order service.

This is the classic confused deputy problem.

The usual answer is to carry the end user's identity downstream, for example a signed token that payments validates with something like Istio's RequestAuthentication, plus sanity checks such as "does this order id belong to this user".

Stolen credentials still work until they expire.
Short-lived certs shrink that window. They do not remove it.

You can do all of this without a mesh.
Cloud providers have their own service-to-service identity, such as IAM based auth on AWS or Google Cloud. SPIFFE and its runtime, SPIRE, work outside Kubernetes. A mesh is the convenient way to get the same ideas, it is not the only way.

The mesh is now critical infrastructure.
If your certificate authority goes down for long enough, certs expire and services stop trusting each other. Monitor it like you monitor your database.

The 30 second version

If you get asked this in an interview, here is the whole thing in one breath:

1. Give the service a ClusterIP and no public address, so the internet cannot reach it.
2. Add a default-deny NetworkPolicy with an allow list, so only the right pods can connect.
3. Turn on mutual TLS, in STRICT mode, so callers prove who they are with short-lived certificates.
4. Add an authorization policy, so even a verified caller can only do what it is supposed to do.
5. Then mention that none of this authenticates the end user, and you will have said something most candidates do not.

"Put it in a private network" was never wrong.

It was just the first gate of four.

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production secure and reliable without slowing you down.