Android platform RTMP streaming or lightweight RTSP service (camera or same screen) data access type summary before encoding

Android platform RTMP streaming or lightweight RTSP service (camera or same screen) data access type summary before encoding

Many developers always feel that the interface is not enough when doing RTMP streaming or lightweight RTSP services (camera or same screen) on the Android platform. Take the Daniel Live SDK as an example ( Github ). Let s summarize, what we usually need to support What are the types of audio and video data before encoding:

1. The data interface of the Android camera before and after the camera through OnPreviewFrame() callback:

@Override public void onPreviewFrame(byte[] data, Camera camera) { frameCount++; if (frameCount% 3000 == 0) { Log.i("OnPre", "gc+"); System.gc(); Log.i("OnPre", "gc-"); } if (data == null) { Parameters params = camera.getParameters(); Size size = params.getPreviewSize(); int bufferSize = (((size.width | 0x1f) + 1) * size.height * ImageFormat.getBitsPerPixel(params.getPreviewFormat()))/8; camera.addCallbackBuffer(new byte[bufferSize]); } else { if (isRTSPPublisherRunning || isPushingRtmp || isRecording || isPushingRtsp) { libPublisher.SmartPublisherOnCaptureVideoData(publisherHandle, data, data.length, currentCameraType, currentOrigentation); } camera.addCallbackBuffer(data); } } Copy code

 Corresponding interface definition:

/**     * Set live video data(no encoded data).     *     * @param cameraType: CAMERA_FACING_BACK with 0, CAMERA_FACING_FRONT with 1     *      * @param curOrg:          * PORTRAIT = 1;//Vertical screen          * LANDSCAPE = 2;//The situation where the home button is on the right side of the horizontal screen          * LANDSCAPE_LEFT_HOME_KEY = 3;//The home button on the left side of the horizontal screen     *     * @return {0} if successful     */     public native int SmartPublisherOnCaptureVideoData(long handle, byte[] data, int len, int cameraType, int curOrg); Copy code

2. Some customized devices only support YV12 data:

    /**      * YV12 data interface      *      * @param data: YV12 data      *      * @param width: image width      *      * @param height: image height      *      * @param y_stride: y-stride      *      * @param v_stride: v-face step length      *      * @param u_stride: u face step length      *      * rotation_degree: rotate clockwise, must be 0, 90, 180, 270      *      * @return {0} if successful      */     public native int SmartPublisherOnYV12Data(long handle, byte[] data, int width, int height, int y_stride, int v_stride, int u_stride, int rotation_degree); Copy code

3. Support NV21 data interface:

The nv21 data interface, in addition to the regular camera data access, the data from some customized cameras is flipped, and this interface also supports it.

    /**      * NV21 data interface      *      * @param data: nv21 data      *      * @param len: data length      *      * @param width: image width      *      * @param height: image height      *      * @param y_stride: y-stride      *      * @param uv_stride: uv face step length      *      * rotation_degree: rotate clockwise, must be 0, 90, 180, 270      *      * @return {0} if successful      */     public native int SmartPublisherOnNV21Data(long handle, byte[] data, int len, int width, int height, int y_stride, int uv_stride, int rotation_degree);     /**      * NV21 data interface      *      * @param data: nv21 data      *      * @param len: data length      *      * @param width: image width      *      * @param height: image height      *      * @param y_stride: y-stride      *      * @param uv_stride: uv face step length      *      * rotation_degree: rotate clockwise, must be 0, 90, 180, 270      *      * @param is_vertical_flip: Whether to flip vertically, 0 does not flip, 1 flip      *      * @param is_horizontal_flip: Whether to flip horizontally, 0 does not flip, 1 flip      *      * @return {0} if successful      */     public native int SmartPublisherOnNV21DataV2(long handle, byte[] data, int len, int width, int height, int y_stride, int uv_stride, int rotation_degree,                                                  int is_vertical_flip, int is_horizontal_flip); Copy code

4. Support YUV data access:

    /**     * Set live video data(no encoded data).     *     * @param data: I420 data     *      * @param len: I420 data length     *      * @param yStride: y stride     *      * @param uStride: u stride     *      * @param vStride: v stride     *     * @return {0} if successful     */     public native int SmartPublisherOnCaptureVideoI420Data ( long handle,   byte [] data, int len, int yStride, int uStride, int vStride) ; copy code

5. Support RGBA data access (support data access after cropping, mainly used in the same screen scene):

    /**     * Set live video data(no encoded data).     *     * @param data: RGBA data     *      * @param rowStride: stride information     *      * @param width: width     *      * @param height: height     *     * @return {0} if successful     */     public native int SmartPublisherOnCaptureVideoRGBAData ( long handle, ByteBuffer data, int rowStride, int width, int height) ;     /**      * Deliver the cropped RGBA data      *      * @param data: RGBA data      *      * @param rowStride: stride information      *      * @param width: width      *      * @param height: height      *      * @param clipedLeft: left; clipedTop: top; clipedwidth: width after cropping; clipedHeight: height after cropping; ensure that the width and height after passing down are even      *      * @return {0} if successful      */     public native int SmartPublisherOnCaptureVideoClipedRGBAData ( long handle, ByteBuffer data, int rowStride, int width, int height, int clipedLeft, int clipedTop, int clipedWidth, int clipedHeight) ;     /**      * Set live video data(no encoded data).      *      * @param data: ABGR flip vertical (vertical flip) data      *      * @param rowStride: stride information      *      * @param width: width      *      * @param height: height      *      * @return {0} if successful      */     public native int SmartPublisherOnCaptureVideoABGRFlipVerticalData ( long handle, ByteBuffer data, int rowStride, int width, int height) ; Copy code

6. Support RGB565 data access (mainly used in the same screen scene):

    /**      * Set live video data(no encoded data).      *      * @param data: RGB565 data      *      * @param row_stride: stride information      *      * @param width: width      *      * @param height: height      *      * @return {0} if successful      */     public native int SmartPublisherOnCaptureVideoRGB565Data ( long handle,ByteBuffer data, int row_stride, int width, int height) ; Copy code

7. Support camera data access (mainly used for camera2 interface docking):     

/*     * The interface provided specifically for the android.graphics.ImageFormat.YUV_420_888 format of android.media.Image     *     * @param width: must be a multiple of 8     *     * @param height: must be a multiple of 8     *     * @param crop_left: Cut the horizontal coordinate of the upper left corner, generally filled according to android.media.Image.getCropRect()     *     * @param crop_top: Cut the vertical coordinate of the upper left corner, generally filled according to android.media.Image.getCropRect()     *     * @param crop_width: must be a multiple of 8, fill in 0 will ignore this parameter, generally fill according to android.media.Image.getCropRect()     *     * @param crop_height: must be a multiple of 8, fill in 0 will ignore this parameter, generally fill according to android.media.Image.getCropRect()     *     * @param y_plane corresponds to android.media.Image.Plane[0].getBuffer()     *     * @param y_row_stride corresponds to android.media.Image.Plane[0].getRowStride()     *     * @param u_plane corresponds to android.media.Image.Plane[1].getBuffer()     *     * @param v_plane corresponds to android.media.Image.Plane[2].getBuffer()     *     * @param uv_row_stride corresponds to android.media.Image.Plane[1].getRowStride()     *     * @param uv_pixel_stride corresponds to android.media.Image.Plane[1].getPixelStride()     *     * @param rotation_degree: clockwise rotation, must be 0, 90, 180, 270     *     * @param is_vertical_flip: Whether to flip vertically, 0 does not flip, 1 flip     *     * @param is_horizontal_flip: Whether to flip horizontally, 0 does not flip, 1 flip     *     * @param scale_width: zoom width, must be a multiple of 8, 0 does not zoom     *     * @param scale_height: zoom height, must be a multiple of 8, 0 does not zoom     *     * @param scale_filter_mode: zoom quality, the range must be [1,3], pass 0 to use the default speed     *     * @return {0} if successful     */     public native int SmartPublisherOnImageYUV420888 ( long handle, int width, int height,                                                       int crop_left, int crop_top, int crop_width, int crop_height,                                                      ByteBuffer y_plane, int y_row_stride,                                                      ByteBuffer u_plane, ByteBuffer v_plane, int uv_row_stride, int uv_pixel_stride,                                                       int rotation_degree, int is_vertical_flip, int is_horizontal_flip,                                                       int scale_width, int scale_height, int scale_filter_mode) ; copy code

8. Support PCM data access:  

    /**      * Pass PCM audio data to SDK, audio data is passed in every 10ms      *       *   @param pcmdata: pcm data, need to use ByteBuffer.allocateDirect allocation, ByteBuffer.isDirect() is true.      *   @param size: pcm data size      *   @param sample_rate: sampling rate, currently only supports {44100, 8000, 16000, 24000, 32000, 48000}, 44100 is recommended      *   @param channel: channel, the current channel supports single channel (1) and dual channel (2), single channel (1) is recommended      *   @param per_channel_sample_number: Please pass in sample_rate/100      */     public native int SmartPublisherOnPCMData ( long handle, ByteBuffer pcmdata, int size, int sample_rate, int channel, int per_channel_sample_number) ;     /**      * Pass PCM audio data to SDK, audio data is passed in every 10ms      *      *   @param pcmdata: pcm data, need to use ByteBuffer.allocateDirect allocation, ByteBuffer.isDirect() is true.      *   @param offset: pcmdata offset      *   @param size: pcm data size      *   @param sample_rate: sampling rate, currently only supports {44100, 8000, 16000, 24000, 32000, 48000}, 44100 is recommended      *   @param channel: channel, the current channel supports single channel (1) and dual channel (2), single channel (1) is recommended      *   @param per_channel_sample_number: Please pass in sample_rate/100      */     public native int SmartPublisherOnPCMDataV2 ( long handle, ByteBuffer pcmdata, int offset, int size, int sample_rate, int channel, int per_channel_sample_number) ;     /**      * Pass PCM audio data to SDK, audio data is passed in every 10ms      *      *   @param pcm_short_array: pcm data, short is native endian order      *   @param offset: array offset      *   @param len: number of array items      *   @param sample_rate: sampling rate, currently only supports {44100, 8000, 16000, 24000, 32000, 48000}, 44100 is recommended      *   @param channel: channel, the current channel supports single channel (1) and dual channel (2), single channel (1) is recommended      *   @param per_channel_sample_number: Please pass in sample_rate/100      */     public native int SmartPublisherOnPCMShortArray ( long handle, short [] pcm_short_array, int offset, int len, int sample_rate, int channel, int per_channel_sample_number) ; Copy code

9. Support remote PCM data access and PCM data access after mixing (mainly used for one-to-one interaction):    

/**      * Set far end pcm data      *       * @param pcmdata: 16bit pcm data      * @param sampleRate: audio sample rate      * @param channel: auido channel      * @param per_channel_sample_number: per channel sample numbers      * @param is_low_latency: if with 0, it is not low_latency, if with 1, it is low_latency      * @return {0} if successful      */     public native int SmartPublisherOnFarEndPCMData ( long handle, ByteBuffer pcmdata, int sampleRate, int channel, int per_channel_sample_number, int is_low_latency) ;     /**      * Pass the PCM mixed audio data to the SDK, the audio data is passed in once every 10ms      *      *   @param stream_index: currently only 1 can be passed, other errors will be returned      *   @param pcm_data: pcm data, need to use ByteBuffer.allocateDirect allocation, ByteBuffer.isDirect() is true.      *   @param offset: pcmdata offset      *   @param size: pcm data size      *   @param sample_rate: sample rate, currently only supports {44100, 8000, 16000, 24000, 32000, 48000}      *   @param channels: channels, the current channel supports single channel (1) and dual channel (2)      *   @param per_channel_sample_number: Please pass in sample_rate/100      */     public native int SmartPublisherOnMixPCMData ( long handle, int stream_index, ByteBuffer pcm_data, int offset, int size, int sample_rate, int channels, int per_channel_sample_number) ;     /**      * Pass the PCM mixed audio data to the SDK, the audio data is passed in once every 10ms      *      *   @param stream_index: currently only 1 can be passed, other errors will be returned      *   @param pcm_short_array: pcm data, short is native endian order      *   @param offset: array offset      *   @param len: number of array items      *   @param sample_rate: sample rate, currently only supports {44100, 8000, 16000, 24000, 32000, 48000}      *   @param channels: channels, the current channel supports single channel (1) and dual channel (2)      *   @param per_channel_sample_number: Please pass in sample_rate/100      */     public native int SmartPublisherOnMixPCMShortArray ( long handle, int stream_index, short [] pcm_short_array, int offset, int len, int sample_rate, int channels, int per_channel_sample_number) ; copy code

summary:

The above is just the data interface before encoding. If a product wants to be universal enough, it needs to be connected too much. What do you think?