Tag Archives: Multicast

Handle Discontinuity and PAT/PMT change in Android Multicast Playback

In previous post it briefly introduces how to implement multicast playback in Android.
In real world, multicast may get discontinuities or PAT/PMT changes, which will cause the playback intermittent or failure if it’s not correctly handled.

In this post, it will introduces how to handle discontinuity and PAT/PMT changes.

Background

In NuPlayer::Decoder::fetchInputData(), it handles three types of DISCONTINUITY:

  • DISCONTINUITY_TIME
  • DISCONTINUITY_AUDIO_FORMAT
  • DISCONTINUITY_VIDEO_FORMAT

Originally, they are used to handle adaptive cases in HLS, e.g. switch from low-resolution stream to high-resolution stream. They’re perfect for multicast stream’s discontinuity and PAT/PMT change as well.

Handle Discontinuity

In ATSParser::Stream::parse(), it already checks the ContinuityCounter in TS streams. We can just queue a time discontinuity unit, then NuPlayer::Decoder will handle the case, reset the playback and re-sync A/V.

if (mExpectedContinuityCounter >= 0
    && (unsigned)mExpectedContinuityCounter != continuity_counter) {
  ALOGI("discontinuity on stream pid 0x%04x", mElementaryPID);
  ...
  signalDiscontinuity(DISCONTINUITY_TIME, NULL);
  ...
}

In ATSParser::Program::convertPTSToTimestamp(), there is a special flag for calculating time stamp from PTS.

if (!(mParser->mFlags & TS_TIMESTAMPS_ARE_ABSOLUTE)) {
  if (!mFirstPTSValid) {
    mFirstPTSValid = true;
    mFirstPTS = PTS;
    PTS = 0;
  } else if (PTS < mFirstPTS) {
    PTS = 0;
  } else {
    PTS -= mFirstPTS;
  }
}
...

If TS_TIMESTAMPS_ARE_ABSOLUTE is not set, it uses the first valid PTS as the base time stamp, and assumes the PTS is increasing all the time. This is not the case for multicast live streaming. So we need to set this flag to let it calculate time stamp by absolute value of PTS.

  // In MulticastSession::onConnect()
  mTSParser = new ATSParser(ATSParser::TS_TIMESTAMPS_ARE_ABSOLUTE);

With the changes, multicast discontinuity is correctly handled and it works fine when multicast stream is looped.

Handle PAT/PMT change

In ATSParser implementation, it just parses PAT/PMT and store the program table and the A/V streams. If PAT/PMT is changed, it just update the program table and streams, and requires the “user” of ATSParser to handle the change, which is quite complicated.
In the other hand, when PAT/PMT is changed in multicast, we know the stream is changed and thus can just throw away the previous program table. So why not delete the previous ATSParser and create a new one to handle the new stream? It makes things much easier.
The changes:

  • ATSParser
    1. Add an observer that should receive the event of PAT/PMT change
    2. Save PAT/PMT crc32 and detect the change; If it’s changed, call onPatPmtChange() and notify the observer
  • MulticastSession
    1. Receive parser’s event and call onPatPmtChanged() for PAT/PMT changed event
    2. Reset ATSParser by delete and create new one
    3. Queue FORMAT change DISCONTINUITY for both audio and video
    4. In dequeueAccessUnit(), check the discontinuity queue and return DISCONTINUITY for audio/video stream.

With the changes, ATSParser is reset when PAT/PMT changes, as if it plays a new stream, which is exactly the expected behavior.

Share

Multicast playback in Android

This article is to briefly introduce how to add multicast playback support in Android platform based on AOSP 5.0.0_r2 tag.

Background

  • NuPlayer is the default media player in Android L. It’s part of libmediaplayerservice located in frameworks/av/media/libmediaplayerservice
  • libstagefright is a library used by NuPlayer to provide a lot of media related functions, e.g. media download, parse and extract. Located in frameworks/av/media/libstagefright
  • chromium_org is a library used by Browser to provide all webview related functions. <video> tag in Webview also uses this library to render its content. Located in external/chromium_org
  • Gallery2 is a default system app that handles media playback intents. Located in packages/apps/Gallery

The above components will be changed to add multicast support.

Implementations

Precondition

Multicast as a kernel option, shall be enabled in kernel build.
Note: this option is enabled by default on Nexus Tablets, but disabled on Nexus Phones.
So you probably need to compile your custom kernel if you plan to test multicast on your phone.

  CONFIG_IP_MULTICAST=y
NuPlayer
  • MediaPlayerFactory::IFactory::scoreFactory() checks the URI scheme to decide which player shall be used to play a certain media.
    In NuPlayerFactory::scoreFactory(), multicast’s score shall be added to be chosen for playing multicast stream.
    if (!strncasecmp("http://", url, 7)
        || !strncasecmp("https://", url, 8)
        || !strncasecmp("file://", url, 7)) {
      … // return kOurScore
    }
    if (!strncasecmp("rtsp://", url, 7)) {
      return kOurScore;
    }
    if (!strncasecmp("udp://", url, 6)
        || !strncasecmp("igmp://", url, 7)) {
      return kOurScore;
    }
  • NuPlayer checks the URI scheme and creates different DataSource to receive media data. MulticastSource is added to handle multicast stream.
    In NuPlayer::setDataSourceAsync(), multicast shall be supported as below
	if (IsHTTPLiveURL(url)) {
	  … // new HTTPLiveSource
	} else if (!strncasecmp(url, "rtsp://", 7)) {
	  … // new RTSPSource
	}
	…
	else if (!strncasecmp(url, "udp://", 6)
	         || !strncasecmp("igmp://", url, 7)) {
	  source = new MulticastSource(notify, httpService, url, headers);
	}
	else {
	  … // Generic source
	}
  • MulticastSource inherits NuPlayer::Source and provide related functions. Basically it just handles events and pass all functions to MulticastSession, which is implemented in libstagefright.
libstagefright
  • Multicast socket related code shall be implemented to provide the socket interface.
  • MulticastSession uses the socket to receive media data, uses existing ATSParser to parse the TS stream, and extract so-called accessUnit to feed the decoder. It could be put at frameworks/av/media/libstagefright/multicast
    This class handles the main part of multicast playback.
  • MulticastDataSource is created by DataSource and used by MediaExtractor to extract the meta data of the media, e.g. whether it has video or audio, what the video resolution is, etc.
    This class is used in <video> tag playback, that webview will retrieve the media element’s metadata.
    Note: DataSource expects the media as static content, so it uses readAt(offset) to read the data from the source. But multicast is a stream, so MulticastDataSource shall be implemented carefully to provide the correct data with offset to DataSource.
  • DataSource shall be updated to create MulticastDataSource for multicast scheme.
	if (!strncasecmp("file://", uri, 7)) {
	  … // new FileSource
	} else if (!strncasecmp("http://", uri, 7)
	           || !strncasecmp("https://", uri, 8)
	           || isWidevine) {
	  … // new Http or Cache source
	} else if (!strncasecmp("data:", uri, 5)) {
	  … // new DataURISource
	} else if (!strncasecmp("igmp://", uri, 7)
	           || !strncasecmp("udp://", uri, 6)) {
	  source = new MulticastDataSource(uri);
	} else {
	  … // new FileSource
	}
chromium_org

By default, WebMediaPlayerAndroid does not support multicast scheme, and thus it treats the URI as a file and fails to load the media. WebMediaPlayerAndroid::load() shall be updated to support multicast scheme.

  if (url_.scheme() == "udp" || url_.scheme() == "igmp") {
    DVLOG(1) << "Do not load multicast";
    UpdateReadyState(WebMediaPlayer::ReadyStateHaveNothing);
    DidLoadMediaInfo(MediaInfoLoader::kOk, GURL(url), GURL(""), false);
    return;
  }
Gallery2

By default, Gallery2 does not handle multicast, and thus if an application sends a VIEW intent with multicast scheme, no application handles the intent and exception throws. AndroidManifest.xml shall be updated to support multicast scheme, so that it handles such VIEW intent.

            <intent-filter>
                <action android:name="android.intent.action.VIEW" />
                <category android:name="android.intent.category.DEFAULT" />
                <category android:name="android.intent.category.BROWSABLE" />
                <data android:scheme="udp" />
                <data android:scheme="igmp" />
                <data android:mimeType="video/*" />
                <data android:mimeType="audio/*" />
             </intent-filter>

Conclusion

The above is a simplified introduction of the changes needed to support multicast playback in AOSP.
Google will probably not officially add such support, but some fork may be interested in multicast. This could be a start of the work.

Share