En tête technical corner
Mediastreamer2 is a powerful lightweight streaming engine for voice/video telephony applications.
This open source library is responsible for receiving and sending all multimedia streams in Linphone, including voice/video capture, encoding and decoding, and rendering.
Mediastreamer2 in Linphone architecture
- Audio codecs: opus, speex, g711, g729, gsm, iLBC, AMR, AMR-WB, g722, SILK, iSAC, BV16, Codec2
- Video codecs: VP8, H.264 and H.265 with resolutions up to 1080P, MPEG4
- Hardware accelerated H.264 codec for Mac OSX / iOS (VideoToolbox) and Android (MediaCodec)
- Send and receive RTP/RTCP streams
- Audio conferencing API
- Features an innovative jitter buffer algorithm, which can quickly adapt to network conditions with a lot of jitter and improve control of the audio latency
- Adaptive bit rate control algorithm: congestion control and estimation of available bandwidth, in order to optimise the quality of audio and video
- ICE (RFC 5245), STUN and TURN (RFC 5766) for optimised NAT traversal, allowing peer-to-peer audio and video connections whenever possible
- SRTP, zRTP and SRTP-DTLS voice and video encryption
- RTP/AVPF support: RTCP control messages for video error recovery: PLI, SLI, RPSI, FIR
- TMMBR support (Temporary Maximum Media Stream Bit Rate Request)
- Acoustic echo cancellation using echo cancellers from libspeexdsp, webrtc AECm or AEC
- Capture and playback from various platform-dependent sound architectures (ALSA, PulseAudio, AudioUnit, AudioQueue, WaveApi, WASAPI, Android AudioTrack/AudioRecord, Android OpenSLES)
- Supports any webcam, based on the OS's camera API: QuickTime, DirectShow, Video4Linux, Android camera
- Play and record to or from raw, wav, or mkv (matroska) files
- Optimised rendering of YUV pictures, using OpenGL, DrawDib, X11/Xv
- Dual tone generation
- Custom tone detector
- Audio parametric equaliser
- Volume control, automatic gain control
- Can use plugins to add new codecs, for new sound input/output backend etc.
- GNU/Linux: x86, x86-64, ARM v5 to v7, arm64 ; Debian 8/9, Centos 7
- Windows Desktop: x86 (works also on x86_64), Windows 7 and later
- macOS: x86_64 ; 10.11 and later.
- GNU/Linux embedded: Linphone-daemon or Liblinphone are good candidates to provide the software stack for a hardware phone or hardware communication system.
- Apple iOS 9 to 12 (ARM v7, ARM 64)
- Google Android 4.1 to 8.1 (ARM v7-v8, x86)
- Windows 10 UWP : mobile and desktop (ARM v7)
Design and principles
Each processing entity is contained within an MSFilter object. MSFilter(s) have inputs and/or outputs that can be used to connect from and to other MSFilters.
A trivial example can illustrate this:
- MSRtpRecv is an MSFilter that can receive RTP packets from the network, unpacketise them and post them on its only output.
- MSSpeexDec is an MSFilter that takes everything on its input assuming these are Speex encoded packets, decodes them and puts the result on its output.
- MSFileRec is an MSFilter that takes everything on its input and writes it to a wav file (assuming the input is 16-bit linear pcm).
MSFilters can be connected together to form a filter chain. If we connect the three examples above, we obtain a processing chain that receives RTP packets, decodes them and write the uncompressed results into a wav file.
MSRtpRecv --> MSSpeexDec --> MSFileRec
The execution of the media processing is scheduled by a MSTicker object, a thread that wakes up every 10 ms to process data in all the MSFilter chains it manages. Several MSTickers can be used simultaneously. For example, one for audio filters, one for video filters, or one on each processor of the machine on which it is running.
- RTP: A Transport Protocol for Real-Time Applications, RFC 3550
- 5. RTP Data Transfer Protocol
- 6.4 Sender and Receiver Reports
- 6.5 SDES: Source Description RTCP Packet
- 6.6 BYE: Goodbye RTCP Packet
- 6.7 APP: Application-Defined RTCP Packet
- RTP Profile for Audio and Video Conference with Minimal Control, RFC 3551
- Real-time Transport Control Protocol (RTCP)-Based Feedback (RTP/AVPF), RFC 4585
- Temporary Maximum Media Stream Bit Rate Request and Notification, section 3.5.4. of RFC 5104
- Symmetric RTP / RTP Control Protocol (RTCP), RFC 4961
- Session Traversal Utilities for NAT (STUN), RFC 5389 (Basic procedures)
- Secure Real Time Transport Protocol (SRTP, RFC 3711)
- ZRTP, RFC 6189
- ICE, RFC 5245 & RFC 6336
- TURN, RFC 5766
- RTP Payload for Text conversation, RFC 4103
- rtcp-mux, RFC 5761
- SRTP-DTSL, RFC 5763
- RTP Payload Format for the Opus Speech and Audio Codec, RFC 7587
- RTP Payload Format for the Speex Codec, RFC 5574
- Real-Time Transport Protocol (RTP) Payload Format and File Storage Format for the Adaptive Multi-Rate (AMR) and Adaptive Multi-Rate Wideband (AMR-WB) Audio Codecs
- only RTP octet-align=1 mode, without interleaving, crc, single channel
- RTP Payload for DTMF Digits, Telephony Tones, and Telephony Signals, RFC 4733
- RTP Payload Format for Named Telephone Events
- RTP Payload Format and File Storage Format for SILK Speech and Audio Codec
- RTP Payload Format for BroadVoice Speech Codecs, RFC 4298
- XML Schema for Media Control, RFC 5168
- Sending and processing of picture fast update command in SIP INFO
- RTP Payload Format for MPEG-4 Audio/Visual Streams, RFC 3016
- 3. RTP Packetization of MPEG-4 Visual bitstream
- RTP Payload Format for ITU-T Rec. H.263 Video, RFC 4629
- RTP Payload Format for H.264 Video, RFC 3984
- RTP Payload Format for VP8 Video
Mediastreamer2 is dual licensed. It can be licensed and distributed:
- Under a GNU GPLv2 license, for free (open source)
- Under a proprietary license, for use in closed source applications. Contact Belledonne Communications for costs and other service information.
Source code can be downloaded from our GitLab.
Mediastreamer2 releases can be downloaded here.
You can also use Git to retrieve the latest source code (recommended for developers):
git clone firstname.lastname@example.org:BC/public/mediastreamer2.git