RTP
RTP or Real-time Transport Protocol is a protocol for streaming media (including VoIP and video teleconferencing) over the Internet. RTCP is used alongside this protocol to give feedback on the quality of the connection and RTSP changes streaming aspects of the connection.
For example, RTP might stream a video, RTCP might show dropped frames, and RTSP might pause the video playback.
RTP is often layered on top of UDP because UDP is quicker than TCP.
VLC has built-in support for RTP as a server (streaming output). As a client, VLC uses the LiveMedia library.
RTP/AVP
RTP is generic and is extensible through profiles. RTP/AVP or RTP Audio Video Profile is a profile for RTP specific for live multimedia streams. RTP/AVP requires a specified payload format to establish which codecs will be used. Each has a dedicated RFC, e.g. RFC 6184 for H.264 payload format.
