Whether you use qui or que (as relative pronouns) depends on two rules:
1) Use qui if you are referring to the subject or if there is a preposition directly preceding it.
2) Use que referring to the direct object (COD).
Examples are:
C'est le gars qui m'a parlé hier. -- That's the guy who spoke to me yesterday.
C'est le gars à qui j'ai parlé hier. -- That's the guy with whom I spoke yesterday.
C'est le gars que j'ai rencontré hier. -- That's the guy (whom) I met yesterday.