How to realize screen recording and live broadcast on Android

How to realize screen recording and live broadcast on Android

When many developers are working on the same screen in the smart classroom or in the meeting, they often encounter various problems based on the Andriod platform to collect screens and encode them. The following is a share of some technical considerations in our development process, right As a starter:

Protocol selection, data source and processing

1. In the intranet environment, is multicast or RTMP?

Answer: This question has been asked by countless developers. For this reason, I wrote a separate blog argument: blog.csdn.net/renhui1112/..., if you are interested, you can refer to it. In short, RTMP is RTMP. If it is really the same screen without concurrency bottleneck in the intranet environment, you can start the built-in RTSP service (using unicast), and then it is a good solution to pull streams from other terminals.

2. How to set or scale the push resolution?

Answer: Generally speaking, many Android devices, especially high-resolution screens, the original width and height of the video obtained are very large. If the original resolution is pushed, the encoding and uplink pressure is large. Therefore, it is generally recommended to scale appropriately, such as width and height scaling. To 2/3, it is generally recommended to zoom in equal proportions. In addition, it is recommended that the zoom width and height be aligned to 16 bytes.

Not much nonsense, the example code:

private void createScreenEnvironment() { sreenWindowWidth = mWindowManager.getDefaultDisplay().getWidth(); screenWindowHeight = mWindowManager.getDefaultDisplay().getHeight(); Log.i(TAG, "screenWindowWidth: "+ sreenWindowWidth + ",screenWindowHeight:" + screenWindowHeight); if (sreenWindowWidth> 800) { if (screenResolution == SCREEN_RESOLUTION_STANDARD) { scale_rate = SCALE_RATE_HALF; sreenWindowWidth = align(sreenWindowWidth/2, 16); screenWindowHeight = align(screenWindowHeight/2, 16); } else if(screenResolution == SCREEN_RESOLUTION_LOW) { scale_rate = SCALE_RATE_TWO_FIFTHS; sreenWindowWidth = align(sreenWindowWidth * 2/5, 16); } } Log.i(TAG, "After adjust mWindowWidth: "+ sreenWindowWidth + ", mWindowHeight:" + screenWindowHeight); int pf = mWindowManager.getDefaultDisplay().getPixelFormat(); Log.i(TAG, "display format:" + pf); DisplayMetrics displayMetrics = new DisplayMetrics(); mWindowManager.getDefaultDisplay().getMetrics(displayMetrics); mScreenDensity = displayMetrics.densityDpi; mImageReader = ImageReader.newInstance(sreenWindowWidth, screenWindowHeight, 0x1, 6); mMediaProjectionManager = (MediaProjectionManager) getSystemService(Context.MEDIA_PROJECTION_SERVICE); } Copy code

3. Automatic adaptation of horizontal and vertical screens

Answer: Because the width and height of the captured screen are different in the horizontal and vertical screen state, if the horizontal and vertical screens are switched, at this time, you need to consider the adaptation of the horizontal and vertical screens. Make sure that, for example, in the vertical screen state, when switching to the horizontal screen, push and pull both ends of the stream. It can be automatically adapted, the horizontal and vertical screens are automatically adapted, the encoder needs to be restarted, and the streaming end needs to be able to automatically adapt to changes in width and height, and automatically play.

4. Certain frame compensation strategy

Answer: Many people don t understand why it is necessary to add frames. In fact, when the screen is collected, if the screen is not moving, there will be no data going on. At this time, it is better to save the last frame of data and set a certain Make sure that the frame interval is too large to cause the playback end to fail to receive data for a few seconds. Of course, if the server can cache GOPs, this problem can be solved.

5. Abnormal network handling, event callback mechanism

Answer: If you are using RTMP, network jitter or other network abnormalities, you need a good reconnection mechanism and status feedback mechanism.

class EventHandeV2 implements NTSmartEventCallbackV2 { @Override public void onNTSmartEventCallbackV2 ( long handle, int id, long param1, long param2, String param3, String param4, Object param5) { Log.i(TAG, "EventHandeV2: handle=" + handle + "id:" + id); String publisher_event = "" ; switch (id) { case NTSmartEventID.EVENT_DANIULIVE_ERC_PUBLISHER_STARTED: publisher_event = "Start..." ; break ; case NTSmartEventID.EVENT_DANIULIVE_ERC_PUBLISHER_CONNECTING: publisher_event = "Connecting..." ; break ; case NTSmartEventID.EVENT_DANIULIVE_ERC_PUBLISHER_CONNECTION_FAILED: publisher_event = "Connection failed.." ; break ; case NTSmartEventID.EVENT_DANIULIVE_ERC_PUBLISHER_CONNECTED: publisher_event = "Connected successfully.." ; break ; case NTSmartEventID.EVENT_DANIULIVE_ERC_PUBLISHER_DISCONNECTED: publisher_event = "The connection is broken.." ; break ; case NTSmartEventID.EVENT_DANIULIVE_ERC_PUBLISHER_STOP: publisher_event = "Close..." ; break ; case NTSmartEventID.EVENT_DANIULIVE_ERC_PUBLISHER_RECORDER_START_NEW_FILE: publisher_event = "Start a new recording file: " + param3; break ; case NTSmartEventID.EVENT_DANIULIVE_ERC_PUBLISHER_ONE_RECORDER_FILE_FINISHED: publisher_event = "A video file has been generated: " + param3; break ; case NTSmartEventID.EVENT_DANIULIVE_ERC_PUBLISHER_SEND_DELAY: publisher_event = "Send delay: " + param1 + " Number of frames:" + param2; break ; case NTSmartEventID.EVENT_DANIULIVE_ERC_PUBLISHER_CAPTURE_IMAGE: publisher_event = "Snapshot: " + param1 + " Path: " + param3; if (param1 == 0 ) { publisher_event = publisher_event + "The snapshot is successfully taken.." ; } else { publisher_event = publisher_event + "Failed to take snapshot.." ; } break ; case NTSmartEventID.EVENT_DANIULIVE_ERC_PUBLISHER_RTSP_URL: publisher_event = "RTSP service URL: " + param3; break ; case NTSmartEventID.EVENT_DANIULIVE_ERC_PUSH_RTSP_SERVER_RESPONSE_STATUS_CODE: publisher_event = "RTSP status code received, codeID: " + param1 + ", RTSP URL:" + param3; break ; case NTSmartEventID.EVENT_DANIULIVE_ERC_PUSH_RTSP_SERVER_NOT_SUPPORT: publisher_event = "The server does not support RTSP push, the pushed RTSP URL: " + param3; break ; } String str = "Current callback status:" + publisher_event; Log.i(TAG, str); Message message = new Message(); message.what = PUBLISHER_EVENT_MSG; message.obj = publisher_event; handler.sendMessage(message); } } Copy code

6. Part of the screen data collection

Answer: In many scenes we have encountered, 3/4 of the area will be taken out for delivery to the students in the classroom side, and 1/4 of the area will be used for some instructions and other operations. At this time, you need to consider the screen area. Cut, the interface can be designed as follows:

/** * Deliver the cropped RGBA data * * @param data: RGBA data * * @param rowStride: stride information * * @param width: width * * @param height: height * * @param clipedLeft: left; clipedTop: top; clipedwidth: width after cropping; clipedHeight: height after cropping; ensure that the width and height after passing down are even numbers * * @return {0} if successful */ public native int SmartPublisherOnCaptureVideoClipedRGBAData ( long handle, ByteBuffer data, int rowStride, int width, int height, int clipedLeft, int clipedTop, int clipedWidth, int clipedHeight) ; copy the code
//The actual cropping ratio can be adjusted as appropriate int left = 100 ; int cliped_left = 0 ; int top = 0 ; int cliped_top = 0 ; int cliped_width = width_; int cliped_height = height_; if (scale_rate == SCALE_RATE_HALF) { cliped_left = left/2 ; cliped_top = top/2 ; //After the width is cropped, display 3/4 ratio cliped_width = (width_ * 3 )/4 ; //The height is not cropped cliped_height = height_; } else if (scale_rate == SCALE_RATE_TWO_FIFTHS) { * = left cliped_left 2/. 5 ; cliped_top = top * 2/5 ; //After the width is cropped, display 3/4 ratio cliped_width = (width_ * 3 )/4 ; //The height is not cropped cliped_height = height_; } if (cliped_width% 2 != 0 ) { cliped_width = cliped_width + 1 ; } if (cliped_height% 2 != 0 ) { cliped_height = cliped_height + 1 ; } if ((cliped_left + cliped_width)> width_) { Log.e(TAG, "invalid cliped region settings, cliped_left:" + cliped_left + "cliped_width:" + cliped_width + " width:" + width_); return ; } if ((cliped_top + cliped_height)> height_) { Log.e(TAG, "invalid cliped region settings, cliped_top: " + cliped_top + " cliped_height:" + cliped_height + "height:" + height_); return ; } //Log.i(TAG, "clipLeft:" + cliped_left + "clipTop:" + cliped_top + "clipWidth:" + cliped_width + "clipHeight:" + cliped_height); libPublisher.SmartPublisherOnCaptureVideoClipedRGBAData(publisherHandle, last_buffer, row_stride_, width_, height_, cliped_left, cliped_top, cliped_width, cliped_height ); Copy code

7. Text, image watermark

Answer: In many scenarios, people on the same screen will display the company logo and certain text information on the push end. At this time, you need to consider the text and image watermark issues. For details, please refer to the following interface settings:

/** * Set Text water-mark * * @param fontSize: it should be "MEDIUM", "SMALL", "BIG" * * @param waterPostion: it should be "TOPLEFT", "TOPRIGHT", "BOTTOMLEFT", "BOTTOMRIGHT". * * @param xPading, yPading: the distance of the original picture. * * <pre> The interface is only used for setting font water-mark when publishing stream. </pre> * * @return {0} if successful */ public native int SmartPublisherSetTextWatermark ( long handle, String waterText, int isAppendTime, int fontSize, int waterPostion, int xPading, int yPading) ; /** * Set Text water-mark font file name (set text watermark font path) * * @param fontFileName: font full file name, eg:/system/fonts/DroidSansFallback.ttf * * @return {0} if successful */ public native int SmartPublisherSetTextWatermarkFontFileName ( long handle, String fontFileName) ; /** * Set picture water-mark (set png picture watermark) * * @param picPath: the picture working path, eg:/sdcard/logo.png * * @param waterPostion: it should be "TOPLEFT", "TOPRIGHT", "BOTTOMLEFT", "BOTTOMRIGHT". * * @param picWidth, picHeight: picture width & height * * @param xPading, yPading: the distance of the original picture. * * <pre> The interface is only used for setting picture(logo) water-mark when publishing stream, with "*.png" format </pre> * * @return {0} if successful */ public native int SmartPublisherSetPictureWatermark ( long handle, String picPath, int waterPostion, int picWidth, int picHeight, int xPading, int yPading) ; Copy code

How to send screen data via RTMP or RTSP protocol

There are many open source options for RTMP or RTSP solutions, but most of them are demo-level. If it is productized, self (filling) research (pit) requires a lot of effort. The following uses Daniel Live SDK as an example to introduce how to connect to data. Many developers mentioned why there are so many RTMP push interfaces on the Android platform of the Daniel Live SDK? Unlike some open source or commercial RTMP pushes, only a few interfaces are simple and clear. Not much nonsense, you can refer to Github if you are interested  :

1. Initialize the Publisher interface and return the push instance handle:

/** * Open publisher (start push instance) * * @param ctx: get by this.getApplicationContext() * * @param audio_opt: * if 0: do not push audio * if 1: Push pre-encoding audio (PCM) * if 2: Push encoded audio (aac/pcma/pcmu/speex). * * @param video_opt: * if 0: Do not push video * if 1: Push the video before encoding (YUV420SP/YUV420P/RGBA/ARGB) * if 2: Push encoded video (H.264) * * @param width: capture width; height: capture height. * * <pre>This function must be called firstly.</pre> * * @return the handle of publisher instance */ public native long SmartPublisherOpen (Object ctx, int audio_opt, int video_opt, int width, int height) ; copy code

2. Related Event event callbacks, such as network status, real-time snapshot, recording status, etc. callbacks:

/** * Set callback event * * @param callbackv2: callback function * * @return {0} if successful */ public native int SetSmartPublisherEventCallbackV2 ( long handle, NTSmartEventCallbackV2 callbackv2) ; copy code
class EventHandeV2 implements NTSmartEventCallbackV2 { @Override public void onNTSmartEventCallbackV2 ( long handle, int id, long param1, long param2, String param3, String param4, Object param5) { Log.i(TAG, "EventHandeV2: handle=" + handle + "id:" + id); String publisher_event = "" ; switch (id) { case NTSmartEventID.EVENT_DANIULIVE_ERC_PUBLISHER_STARTED: publisher_event = "Start..." ; break ; case NTSmartEventID.EVENT_DANIULIVE_ERC_PUBLISHER_CONNECTING: publisher_event = "Connecting..." ; break ; case NTSmartEventID.EVENT_DANIULIVE_ERC_PUBLISHER_CONNECTION_FAILED: publisher_event = "Connection failed.." ; break ; case NTSmartEventID.EVENT_DANIULIVE_ERC_PUBLISHER_CONNECTED: publisher_event = "Connected successfully.." ; break ; case NTSmartEventID.EVENT_DANIULIVE_ERC_PUBLISHER_DISCONNECTED: publisher_event = "The connection is broken.." ; break ; case NTSmartEventID.EVENT_DANIULIVE_ERC_PUBLISHER_STOP: publisher_event = "Close..." ; break ; case NTSmartEventID.EVENT_DANIULIVE_ERC_PUBLISHER_RECORDER_START_NEW_FILE: publisher_event = "Start a new recording file: " + param3; break ; case NTSmartEventID.EVENT_DANIULIVE_ERC_PUBLISHER_ONE_RECORDER_FILE_FINISHED: publisher_event = "A video file has been generated: " + param3; break ; case NTSmartEventID.EVENT_DANIULIVE_ERC_PUBLISHER_SEND_DELAY: publisher_event = "Send delay: " + param1 + " Number of frames:" + param2; break ; case NTSmartEventID.EVENT_DANIULIVE_ERC_PUBLISHER_CAPTURE_IMAGE: publisher_event = "Snapshot: " + param1 + " Path: " + param3; if (param1 == 0 ) { publisher_event = publisher_event + "The snapshot is successfully taken.." ; } else { publisher_event = publisher_event + "Failed to take snapshot.." ; } break ; case NTSmartEventID.EVENT_DANIULIVE_ERC_PUBLISHER_RTSP_URL: publisher_event = "RTSP service URL: " + param3; break ; case NTSmartEventID.EVENT_DANIULIVE_ERC_PUSH_RTSP_SERVER_RESPONSE_STATUS_CODE: publisher_event = "RTSP status code received, codeID: " + param1 + ", RTSP URL:" + param3; break ; case NTSmartEventID.EVENT_DANIULIVE_ERC_PUSH_RTSP_SERVER_NOT_SUPPORT: publisher_event = "The server does not support RTSP push, the pushed RTSP URL: " + param3; break ; } String str = "Current callback status:" + publisher_event; Log.i(TAG, str); Message message = new Message(); message.what = PUBLISHER_EVENT_MSG; message.obj = publisher_event; handler.sendMessage(message); } } Copy code

3. Detect/set soft and hard code:

/** * Set Video H.264 HW Encoder, if support HW encoder, it will return 0 (Set H.264 hard coding) * * @param kbps: the kbps of different resolution(25 fps). * * @return {0} if successful */ public native int SetSmartPublisherVideoHWEncoder ( long handle, int kbps) ; /** * Set Video H.265(hevc) hardware encoder, if support H.265(hevc) hardware encoder, it will return 0 (set H.265 hard code) * * @param kbps: the kbps of different resolution(25 fps). * * @return {0} if successful */ public native int SetSmartPublisherVideoHevcHWEncoder ( long handle, int kbps) ; copy code

4. Set text watermark and PNG image watermark:

/** * Set Text water-mark * * @param fontSize: it should be "MEDIUM", "SMALL", "BIG" * * @param waterPostion: it should be "TOPLEFT", "TOPRIGHT", "BOTTOMLEFT", "BOTTOMRIGHT". * * @param xPading, yPading: the distance of the original picture. * * <pre> The interface is only used for setting font water-mark when publishing stream. </pre> * * @return {0} if successful */ public native int SmartPublisherSetTextWatermark ( long handle, String waterText, int isAppendTime, int fontSize, int waterPostion, int xPading, int yPading) ; /** * Set Text water-mark font file name (set text watermark font path) * * @param fontFileName: font full file name, eg:/system/fonts/DroidSansFallback.ttf * * @return {0} if successful */ public native int SmartPublisherSetTextWatermarkFontFileName ( long handle, String fontFileName) ; /** * Set picture water-mark (set png picture watermark) * * @param picPath: the picture working path, eg:/sdcard/logo.png * * @param waterPostion: it should be "TOPLEFT", "TOPRIGHT", "BOTTOMLEFT", "BOTTOMRIGHT". * * @param picWidth, picHeight: picture width & height * * @param xPading, yPading: the distance of the original picture. * * <pre> The interface is only used for setting picture(logo) water-mark when publishing stream, with "*.png" format </pre> * * @return {0} if successful */ public native int SmartPublisherSetPictureWatermark ( long handle, String picPath, int waterPostion, int picWidth, int picHeight, int xPading, int yPading) ; Copy code

5. Soft coding variable bit rate setting (such as camera capture and coding, if the equipment performance allows, you can consider soft coding variable bit rate):

/** * Set software encode vbr mode (soft encoding variable bit rate). * * <pre>please set before SmartPublisherStart while after SmartPublisherOpen.</pre> * * is_enable_vbr: if 0: NOT enable vbr mode, 1: enable vbr * * video_quality: vbr video quality, range with (1,50), default 23 * * vbr_max_kbitrate: vbr max encode bit-rate(kbps) * * @return {0} if successful */ public native int SmartPublisherSetSwVBRMode ( long handle, int is_enable_vbr, int video_quality, int vbr_max_kbitrate) ; Copy code

6. Basic settings such as frame rate, soft coding rate, key frame interval, coding speed, soft coding profile, etc.:

/** * Set gop interval (set I frame interval) * * <pre>please set before SmartPublisherStart while after SmartPublisherOpen.</pre> * * gopInterval: encode I frame interval, the value always> 0 * * @return {0} if successful */ public native int SmartPublisherSetGopInterval ( long handle, int gopInterval) ; /** * Set software encode video bit-rate * * <pre>please set before SmartPublisherStart while after SmartPublisherOpen.</pre> * * avgBitRate: average encode bit-rate(kbps) * * maxBitRate: max encode bit-rate(kbps) * * @return {0} if successful */ public native int SmartPublisherSetSWVideoBitRate ( long handle, int avgBitRate, int maxBitRate) ; /** * Set fps (set frame rate) * * <pre>please set before SmartPublisherStart while after SmartPublisherOpen.</pre> * * fps: the fps of video, range with (1,25). * * @return {0} if successful */ public native int SmartPublisherSetFPS ( long handle, int fps) ; /** * Set software video encoder profile. * * <pre>please set before SmartPublisherStart while after SmartPublisherOpen.</pre> * * profile: the software video encoder profile, range with (1,3). * * 1: baseline profile * 2: main profile * 3: high profile * * @return {0} if successful */ public native int SmartPublisherSetSWVideoEncoderProfile ( long handle, int profile) ; /** * Set software video encoder speed * * <pre>please set before SmartPublisherStart while after SmartPublisherOpen.</pre> * * @param speed: range with(1, 6), the default speed is 6. * * if with 1, CPU is lowest. * if with 6, CPU is highest. * * @return {0} if successful */ public native int SmartPublisherSetSWVideoEncoderSpeed ( long handle, int speed) ; copy code

7. Set audio encoding type, such as AAC encoding, Speex encoding, and set parameters such as AAC encoding rate:

/** * Set audio encoder type (set audio encoding type) * * @param type: if with 1:AAC, if with 2: SPEEX * * @return {0} if successful */ public native int SmartPublisherSetAudioCodecType ( long handle, int type) ; /** * Set audio encoder bit-rate (set audio encoding bit rate), currently only valid for AAC encoding * * @param kbit_rate: bit rate (unit is kbps), if it is 0, the default bit rate will be used, which must be greater than or equal to 0 * * @return {0} if successful */ public native int SmartPublisherSetAudioBitRate ( long handle, int kbit_rate) ; /** * Set speex encoder quality (set speex encoding quality) * * @param quality: range with (0, 10), default value is 8 * * @return {0} if successful */ public native int SmartPublisherSetSpeexEncoderQuality ( long handle, int quality) ; copy the code

8. Audio processing, such as automatic gain control, noise suppression:

/** * Set Audio Noise Suppression (set audio noise suppression) * * @param isNS: if with 1:suppress, if with 0: does not suppress * * @return {0} if successful */ public native int SmartPublisherSetNoiseSuppression ( long handle, int isNS) ; /** * Set Audio AGC (set audio automatic gain control) * * @param isAGC: if with 1:AGC, if with 0: does not AGC * * @return {0} if successful */ public native int SmartPublisherSetAGC ( long handle, int isAGC) ; copy code

9. Audio mixing interface settings:

/** * Set up audio mixing, currently supports two audio audio mixing * * @param is_mix: 1 mixes, 0 does not mix, default does not mix * * @return {0} if successful */ public native int SmartPublisherSetAudioMix ( long handle, int is_mix) ; copy the code

10. Real-time mute setting:

/** * Set mute or not during publish stream * * @param isMute: if with 1:mute, if with 0: does not mute * * @return {0} if successful */ public native int SmartPublisherSetMute ( long handle, int isMute) ; copy code

11. Volume adjustment:

/** * Set the input volume, this interface is generally not recommended to be called, it may be used in some special situations, generally it is not recommended to amplify the volume * * @param index: usually 0 and 1, if there is no mixing, only use 0, if there is mixing, 0,1 set the volume respectively * * @param volume: volume, the default is 1.0, the range is [0.0, 5.0], set to 0 mute, 1 volume unchanged * * @return {0} if successful */ public native int SmartPublisherSetInputAudioVolume(long handle, int index, float volume); Copy code

12. Front camera mirroring:

/** * Set mirror (set the front camera mirroring) * * @param isMirror: if with 1:mirror mode, if with 0: normal mode * * Please note when with "mirror mode", the publisher and player with the same echo direction * * @return {0} if successful */ public native int SmartPublisherSetMirror ( long handle, int isMirror) ; copy code

13. Recording related configuration, such as recording only audio or video, the maximum size of a single file, the recording storage directory, start/pause/stop recording:

/** * Audio recording switch, the purpose is to control the recording more fine-grained, generally do not need to call this interface, this interface use scenarios such as pushing audio and video at the same time, but only want to record video, you can call this interface to turn off audio recording * * @param is_recoder: 0: do not recorder; 1: recorder; sdk default is 1 * * @return {0} if successful */ public native int SmartPublisherSetRecorderAudio ( long handle, int is_recoder) ; /** * Video recording switch, the purpose is to control the recording more fine-grained, generally do not need to call this interface, this interface use scenarios such as pushing audio and video at the same time, but only want to record audio, you can call this interface to turn off video recording * * @param is_recoder: 0: do not recorder; 1: recorder; sdk default is 1 * * @return {0} if successful */ public native int SmartPublisherSetRecorderVideo ( long handle, int is_recoder) ; /** * Create file directory (create video storage directory) * * @param path, Eg:/sdcard/daniulive/rec * * <pre> The interface is only used for recording the stream data to local side. </pre> * * @return {0} if successful */ public native int SmartPublisherCreateFileDirectory (String path) ; /** * Set recorder directory (set record storage directory) * * @param path: the directory of recorder file. * * <pre> NOTE: make sure the path should be existed, or else the setting failed. </pre> * * @return {0} if successful */ public native int SmartPublisherSetRecorderDirectory ( long handle, String path) ; /** * Set the size of every recorded file (set the size of a single recording file, if it exceeds the maximum file size, it will automatically switch to the next file for recording) * * @param size: (MB), (5M~500M), if not in this range, set default size with 200MB. * * @return {0} if successful */ public native int SmartPublisherSetRecorderFileMaxSize ( long handle, int size) ; /** * Start recorder (start recording) * * @return {0} if successful */ public native int SmartPublisherStartRecorder ( long handle) ; /** * Pause recorder (pause/resume recording) * * is_pause: 1 means pause, 0 means resume recording, input other values will fail the call * * @return {0} if successful */ public native int SmartPublisherPauseRecorder ( long handle, int is_pause) ; /** * Stop recorder (stop recording) * * @return {0} if successful */ public native int SmartPublisherStopRecorder ( long handle) ; copy the code

14. Real-time snapshot:

/** * Set if needs to save image during publishing stream (set whether to enable snapshots) * * @param is_save_image: if with 1, it will save current image via the interface of SmartPlayerSaveImage(), if with 0: does not it * * @return {0} if successful */ public native int SmartPublisherSaveImageFlag ( long handle, int is_save_image) ; /** * Save current image during publishing stream (real-time snapshot) * * @param imageName: image name, which including fully path, "/sdcard/daniuliveimage/daniu.png", etc. * * @return {0} if successful */ public native int SmartPublisherSaveCurImage ( long handle, String imageName) ; copy the code

15. Set the push RTMP URL:

/** * Set rtmp publish stream url (set RTMP url for push) * * @param url: rtmp publish url. * * @return {0} if successful */ public native int SmartPublisherSetURL ( long handle, String url) ; copy the code

16. The data interface of the Android camera before and after the camera through OnPreviewFrame() callback:

@Override public void onPreviewFrame(byte[] data, Camera camera) { frameCount++; if (frameCount% 3000 == 0) { Log.i("OnPre", "gc+"); System.gc(); Log.i("OnPre", "gc-"); } if (data == null) { Parameters params = camera.getParameters(); Size size = params.getPreviewSize(); int bufferSize = (((size.width | 0x1f) + 1) * size.height * ImageFormat.getBitsPerPixel(params.getPreviewFormat()))/8; camera.addCallbackBuffer(new byte[bufferSize]); } else { if (isRTSPPublisherRunning || isPushingRtmp || isRecording || isPushingRtsp) { libPublisher.SmartPublisherOnCaptureVideoData(publisherHandle, data, data.length, currentCameraType, currentOrigentation); } camera.addCallbackBuffer(data); } } Copy code

 Corresponding interface definition:

/**     * Set live video data(no encoded data).     *     * @param cameraType: CAMERA_FACING_BACK with 0, CAMERA_FACING_FRONT with 1     *      * @param curOrg:          * PORTRAIT = 1;//Vertical screen          * LANDSCAPE = 2;//The situation where the home button is on the right side of the horizontal screen          * LANDSCAPE_LEFT_HOME_KEY = 3;//The home button on the left side of the horizontal screen     *     * @return {0} if successful     */     public native int SmartPublisherOnCaptureVideoData(long handle, byte[] data, int len, int cameraType, int curOrg); Copy code

17. Some customized devices only support YV12 data:

    /**      * YV12 data interface      *      * @param data: YV12 data      *      * @param width: image width      *      * @param height: image height      *      * @param y_stride: y-stride      *      * @param v_stride: v-face step length      *      * @param u_stride: u face step length      *      * rotation_degree: rotate clockwise, must be 0, 90, 180, 270      *      * @return {0} if successful      */     public native int SmartPublisherOnYV12Data(long handle, byte[] data, int width, int height, int y_stride, int v_stride, int u_stride, int rotation_degree); Copy code

18. Support NV21 data interface:

The nv21 data interface, in addition to the regular camera data access, the data from some customized cameras is flipped, and this interface also supports it.

    /**      * NV21 data interface      *      * @param data: nv21 data      *      * @param len: data length      *      * @param width: image width      *      * @param height: image height      *      * @param y_stride: y-stride      *      * @param uv_stride: uv face step length      *      * rotation_degree: rotate clockwise, must be 0, 90, 180, 270      *      * @return {0} if successful      */     public native int SmartPublisherOnNV21Data(long handle, byte[] data, int len, int width, int height, int y_stride, int uv_stride, int rotation_degree);     /**      * NV21 data interface      *      * @param data: nv21 data      *      * @param len: data length      *      * @param width: image width      *      * @param height: image height      *      * @param y_stride: y-stride      *      * @param uv_stride: uv face step length      *      * rotation_degree: rotate clockwise, must be 0, 90, 180, 270      *      * @param is_vertical_flip: Whether to flip vertically, 0 does not flip, 1 flip      *      * @param is_horizontal_flip: Whether to flip horizontally, 0 does not flip, 1 flip      *      * @return {0} if successful      */     public native int SmartPublisherOnNV21DataV2(long handle, byte[] data, int len, int width, int height, int y_stride, int uv_stride, int rotation_degree,                                                  int is_vertical_flip, int is_horizontal_flip); Copy code

19. Support YUV data access:

    /**     * Set live video data(no encoded data).     *     * @param data: I420 data     *      * @param len: I420 data length     *      * @param yStride: y stride     *      * @param uStride: u stride     *      * @param vStride: v stride     *     * @return {0} if successful     */     public native int SmartPublisherOnCaptureVideoI420Data ( long handle,   byte [] data, int len, int yStride, int uStride, int vStride) ; copy code

20. Support RGBA data access (support data access after cropping, mainly used in the same screen scene):

    /**     * Set live video data(no encoded data).     *     * @param data: RGBA data     *      * @param rowStride: stride information     *      * @param width: width     *      * @param height: height     *     * @return {0} if successful     */     public native int SmartPublisherOnCaptureVideoRGBAData ( long handle, ByteBuffer data, int rowStride, int width, int height) ;     /**      * Deliver the cropped RGBA data      *      * @param data: RGBA data      *      * @param rowStride: stride information      *      * @param width: width      *      * @param height: height      *      * @param clipedLeft: left; clipedTop: top; clipedwidth: width after cropping; clipedHeight: height after cropping; ensure that the width and height after passing down are even numbers      *      * @return {0} if successful      */     public native int SmartPublisherOnCaptureVideoClipedRGBAData ( long handle, ByteBuffer data, int rowStride, int width, int height, int clipedLeft, int clipedTop, int clipedWidth, int clipedHeight) ;     /**      * Set live video data(no encoded data).      *      * @param data: ABGR flip vertical (vertical flip) data      *      * @param rowStride: stride information      *      * @param width: width      *      * @param height: height      *      * @return {0} if successful      */     public native int SmartPublisherOnCaptureVideoABGRFlipVerticalData ( long handle, ByteBuffer data, int rowStride, int width, int height) ; Copy code

21. Support RGB565 data access (mainly used in the same screen scene):

    /**      * Set live video data(no encoded data).      *      * @param data: RGB565 data      *      * @param row_stride: stride information      *      * @param width: width      *      * @param height: height      *      * @return {0} if successful      */     public native int SmartPublisherOnCaptureVideoRGB565Data ( long handle,ByteBuffer data, int row_stride, int width, int height) ; Copy code

22. Support camera data access (mainly used for camera2 interface docking):     

/*     * The interface provided specifically for the android.graphics.ImageFormat.YUV_420_888 format of android.media.Image     *     * @param width: must be a multiple of 8     *     * @param height: must be a multiple of 8     *     * @param crop_left: Cut the horizontal coordinate of the upper left corner, generally filled according to android.media.Image.getCropRect()     *     * @param crop_top: Cut the vertical coordinate of the upper left corner, generally filled according to android.media.Image.getCropRect()     *     * @param crop_width: must be a multiple of 8, fill in 0 will ignore this parameter, generally fill according to android.media.Image.getCropRect()     *     * @param crop_height: must be a multiple of 8, fill in 0 will ignore this parameter, generally fill according to android.media.Image.getCropRect()     *     * @param y_plane corresponds to android.media.Image.Plane[0].getBuffer()     *     * @param y_row_stride corresponds to android.media.Image.Plane[0].getRowStride()     *     * @param u_plane corresponds to android.media.Image.Plane[1].getBuffer()     *     * @param v_plane corresponds to android.media.Image.Plane[2].getBuffer()     *     * @param uv_row_stride corresponds to android.media.Image.Plane[1].getRowStride()     *     * @param uv_pixel_stride corresponds to android.media.Image.Plane[1].getPixelStride()     *     * @param rotation_degree: clockwise rotation, must be 0, 90, 180, 270     *     * @param is_vertical_flip: Whether to flip vertically, 0 does not flip, 1 flip     *     * @param is_horizontal_flip: Whether to flip horizontally, 0 does not flip, 1 flip     *     * @param scale_width: zoom width, must be a multiple of 8, 0 does not zoom     *     * @param scale_height: zoom height, must be a multiple of 8, 0 does not zoom     *     * @param scale_filter_mode: zoom quality, the range must be [1,3], pass 0 to use the default speed     *     * @return {0} if successful     */     public native int SmartPublisherOnImageYUV420888 ( long handle, int width, int height,                                                       int crop_left, int crop_top, int crop_width, int crop_height,                                                      ByteBuffer y_plane, int y_row_stride,                                                      ByteBuffer u_plane, ByteBuffer v_plane, int uv_row_stride, int uv_pixel_stride,                                                       int rotation_degree, int is_vertical_flip, int is_horizontal_flip,                                                       int scale_width, int scale_height, int scale_filter_mode) ; copy code

23. Support PCM data access:  

    /**      * Pass PCM audio data to SDK, audio data is passed in every 10ms      *       *   @param pcmdata: pcm data, need to use ByteBuffer.allocateDirect allocation, ByteBuffer.isDirect() is true.      *   @param size: pcm data size      *   @param sample_rate: sampling rate, currently only supports {44100, 8000, 16000, 24000, 32000, 48000}, 44100 is recommended      *   @param channel: channel, the current channel supports single channel (1) and dual channel (2), single channel (1) is recommended      *   @param per_channel_sample_number: Please pass in sample_rate/100      */     public native int SmartPublisherOnPCMData ( long handle, ByteBuffer pcmdata, int size, int sample_rate, int channel, int per_channel_sample_number) ;     /**      * Pass PCM audio data to SDK, audio data is passed in every 10ms      *      *   @param pcmdata: pcm data, need to use ByteBuffer.allocateDirect allocation, ByteBuffer.isDirect() is true.      *   @param offset: pcmdata offset      *   @param size: pcm data size      *   @param sample_rate: sampling rate, currently only supports {44100, 8000, 16000, 24000, 32000, 48000}, 44100 is recommended      *   @param channel: channel, the current channel supports single channel (1) and dual channel (2), single channel (1) is recommended      *   @param per_channel_sample_number: Please pass in sample_rate/100      */     public native int SmartPublisherOnPCMDataV2 ( long handle, ByteBuffer pcmdata, int offset, int size, int sample_rate, int channel, int per_channel_sample_number) ;     /**      * Pass PCM audio data to SDK, audio data is passed in every 10ms      *      *   @param pcm_short_array: pcm data, short is native endian order      *   @param offset: array offset      *   @param len: number of array items      *   @param sample_rate: sampling rate, currently only supports {44100, 8000, 16000, 24000, 32000, 48000}, 44100 is recommended      *   @param channel: channel, the current channel supports single channel (1) and dual channel (2), single channel (1) is recommended      *   @param per_channel_sample_number: Please pass in sample_rate/100      */     public native int SmartPublisherOnPCMShortArray ( long handle, short [] pcm_short_array, int offset, int len, int sample_rate, int channel, int per_channel_sample_number) ; Copy code

24. Support remote PCM data access and PCM data access after mixing (mainly used for one-to-one interaction):    

/**      * Set far end pcm data      *       * @param pcmdata: 16bit pcm data      * @param sampleRate: audio sample rate      * @param channel: auido channel      * @param per_channel_sample_number: per channel sample numbers      * @param is_low_latency: if with 0, it is not low_latency, if with 1, it is low_latency      * @return {0} if successful      */     public native int SmartPublisherOnFarEndPCMData ( long handle, ByteBuffer pcmdata, int sampleRate, int channel, int per_channel_sample_number, int is_low_latency) ;     /**      * Pass the PCM mixed audio data to the SDK, the audio data is passed in once every 10ms      *      *   @param stream_index: currently only 1 can be passed, other errors will be returned      *   @param pcm_data: pcm data, need to use ByteBuffer.allocateDirect allocation, ByteBuffer.isDirect() is true.      *   @param offset: pcmdata offset      *   @param size: pcm data size      *   @param sample_rate: sample rate, currently only supports {44100, 8000, 16000, 24000, 32000, 48000}      *   @param channels: channels, the current channel supports single channel (1) and dual channel (2)      *   @param per_channel_sample_number: Please pass in sample_rate/100      */     public native int SmartPublisherOnMixPCMData ( long handle, int stream_index, ByteBuffer pcm_data, int offset, int size, int sample_rate, int channels, int per_channel_sample_number) ;     /**      * Pass the PCM mixed audio data to the SDK, the audio data is passed in once every 10ms      *      *   @param stream_index: currently only 1 can be passed, other errors will be returned      *   @param pcm_short_array: pcm data, short is native endian order      *   @param offset: array offset      *   @param len: number of array items      *   @param sample_rate: sample rate, currently only supports {44100, 8000, 16000, 24000, 32000, 48000}      *   @param channels: channels, the current channel supports single channel (1) and dual channel (2)      *   @param per_channel_sample_number: Please pass in sample_rate/100      */     public native int SmartPublisherOnMixPCMShortArray ( long handle, int stream_index, short [] pcm_short_array, int offset, int len, int sample_rate, int channels, int per_channel_sample_number) ; copy code

25. H264 extended SEI message:

/** * Set the size of the sending queue. To ensure real-time performance, the default size is 3, and a number greater than 0 must be set * * @param max_size: the maximum length of the queue * * @param reserve: reserved field * * NOTE: 1. If the data exceeds the queue size, the header data will be lost; 2. Please call this interface before StartPublisher * * @return {0} if successful */ public native int SmartPublisherSetPostUserDataQueueMaxSize ( long handle, int max_size, int reserve) ; /** * To clear the user data queue, it may be used in some situations, for example, there are 4 messages in the sending queue and waiting to be sent, and you want to send the latest message quickly, you can clear the queued messages first, and then call PostUserXXX * * @return {0} if successful */ public native int SmartPublisherClearPostUserDataQueue ( long handle) ; /** * Send binary data * * NOTE: * 1. The current data size is limited to 256 bytes, too large may affect the video transmission, if you have special needs and need to increase the limit, please contact us * 2. If the accumulated data exceeds the set queue size, the previous header data will be discarded * 3. Data must be sent after calling StartPublisher * * @param data: binary data * * @param size: data size * * @param reserve: reserved field * * @return {0} if successful */ public native int SmartPublisherPostUserData ( long handle, byte [] data, int size, int reserve) ; /** * Send utf8 string * * NOTE: * 1. The length of the string cannot exceed 256. If it is too large, it may affect the video transmission. If you have special needs and need to increase the limit, please contact us * 2. If the accumulated data exceeds the set queue size, the previous header data will be discarded * 3. Data must be sent after calling StartPublisher * * @param utf8_str: utf8 string * * @param reserve: reserved field * * @return {0} if successful */ public native int SmartPublisherPostUserUTF8StringData ( long handle, String utf8_str, int reserve) ; copy code

26. Close the push instance:

/** * To close the push instance, you must call the close interface to release resources when it is over * * @return {0} if successful */ public native int SmartPublisherClose ( long handle) ; copy the code