Client Module Architecture

Deep dive into the Android client module responsible for UI, decoding, and user input handling

Module Overview

The client module (app/) is an Android application that provides the user interface and handles video/audio decoding, user input collection, and ADB communication.

Package: org.client.scrcpy
Location: app/src/main/java/org/client/scrcpy/

Core Components

MainActivity

The main activity manages the application lifecycle and coordinates between UI and streaming service.

File: MainActivity.java (837 lines)

Key Responsibilities

Connection Management

Handles server connection setup, including:

• Reading user preferences (resolution, bitrate, delay)
• Validating server address
• Deploying server JAR via ADB
• Establishing socket connection

From MainActivity.java:738-791:

private void connectScrcpyServer(String serverAdr) {
    String[] serverInfo = Util.getServerHostAndPort(serverAdr);
    String serverHost = serverInfo[0];
    int serverPort = Integer.parseInt(serverInfo[1]);
    
    // Extract and write server JAR
    InputStream inputStream = assetManager.open("scrcpy-server.jar");
    byte[] buffer = new byte[inputStream.available()];
    inputStream.read(buffer);
    
    FileOutputStream outputStream = new FileOutputStream(
        new File(context.getExternalFilesDir("scrcpy"), 
        "scrcpy-server.jar"));
    outputStream.write(buffer);
    
    // Deploy via ADB and start streaming
    sendCommands.SendAdbCommands(context, serverHost, 
        serverPort, localForwardPort, videoBitrate, maxSize);
}

Lifecycle Management

Manages activity states with service binding:

private final ServiceConnection serviceConnection = new ServiceConnection() {
    @Override
    public void onServiceConnected(ComponentName name, IBinder binder) {
        scrcpy = ((Scrcpy.MyServiceBinder) binder).getService();
        scrcpy.setServiceCallbacks(MainActivity.this);
        serviceBound = true;
        scrcpy.start(surface, serverAddress, 
            screenHeight, screenWidth, delayControl);
    }
};

Key lifecycle methods:

• onCreate(): Initialize UI and restore state
• onPause(): Pause streaming and optionally disconnect
• onResume(): Resume streaming or reconnect
• onSaveInstanceState(): Persist configuration

Display and Touch Handling

Calculates aspect ratio and sets up touch event handling (MainActivity.java:362-433):

public void set_display_nd_touch() {
    int[] rem_res = scrcpy.get_remote_device_resolution();
    int remote_device_height = rem_res[1];
    int remote_device_width = rem_res[0];
    float remote_aspect_ratio = (float) remote_device_height / remote_device_width;
    
    // Calculate padding for aspect ratio matching
    // ...
    
    // Set touch listener
    surfaceView.setOnTouchListener((view, event) -> 
        scrcpy.touchevent(event, landscape, 
            surfaceView.getWidth(), surfaceView.getHeight()));
}

Rotation Handling

Supports dynamic orientation changes:

@Override
public void loadNewRotation() {
    unbindService(serviceConnection);
    landscape = !landscape;
    swapDimensions(); // Swap width and height
    
    if (landscape) {
        setRequestedOrientation(
            ActivityInfo.SCREEN_ORIENTATION_SENSOR_LANDSCAPE);
    } else {
        setRequestedOrientation(
            ActivityInfo.SCREEN_ORIENTATION_SENSOR_PORTRAIT);
    }
}

Scrcpy Service

Background service that manages the streaming connection and coordinates decoders.

File: Scrcpy.java (486 lines)

Architecture

Key Methods

Connection Loop (Scrcpy.java:342-470):

private void loop(DataInputStream in, DataOutputStream out, int delay) {
    while (LetServiceRunning.get()) {
        // Send queued events
        byte[] sendevent = event.poll();
        if (sendevent != null) {
            out.write(sendevent, 0, sendevent.length);
        }
        
        // Read incoming packets
        if (in.available() > 0) {
            byte[] packetSize = new byte[4];
            in.readFully(packetSize, 0, 4);
            int size = ByteUtils.bytesToInt(packetSize);
            
            byte[] packet = new byte[size];
            in.readFully(packet, 0, size);
            
            if (MediaPacket.Type.getType(packet[0]) == VIDEO) {
                VideoPacket videoPacket = VideoPacket.readHead(packet);
                videoDecoder.decodeSample(packet, offset, size, 
                    timestamp, flags);
            } else if (MediaPacket.Type.getType(packet[0]) == AUDIO) {
                AudioPacket audioPacket = AudioPacket.readHead(packet);
                audioDecoder.decodeSample(packet, offset, size, 
                    timestamp, flags);
            }
        }
    }
}

Touch Event Processing (Scrcpy.java:136-182):

public boolean touchevent(MotionEvent event, boolean landscape, 
                          int displayW, int displayH) {
    // Calculate scaling between display and remote device
    float remoteW = landscape ? 
        Math.max(remote_dev_resolution[0], remote_dev_resolution[1]) :
        Math.min(remote_dev_resolution[0], remote_dev_resolution[1]);
    float remoteH = landscape ? 
        Math.min(remote_dev_resolution[0], remote_dev_resolution[1]) :
        Math.max(remote_dev_resolution[0], remote_dev_resolution[1]);
    
    switch (event.getAction()) {
        case MotionEvent.ACTION_MOVE:
            // Handle all pointers for multi-touch
            for (int i = 0; i < event.getPointerCount(); i++) {
                int pointerId = event.getPointerId(i);
                int x = (int) (event.getX(i) * realW / displayW);
                int y = (int) (event.getY(i) * realH / displayH);
                sendTouchEvent(action, buttonState, x, y, pointerId);
            }
            break;
        // ... handle other actions
    }
}

VideoDecoder

Decodes H.264 video stream using Android’s MediaCodec API.

File: decoder/VideoDecoder.java (131 lines)

Implementation Details

Configuration

public void configure(Surface surface, int width, int height, 
                      ByteBuffer csd0, ByteBuffer csd1) {
    MediaFormat format = MediaFormat.createVideoFormat(
        "video/avc", width, height);
    format.setByteBuffer("csd-0", csd0); // SPS
    format.setByteBuffer("csd-1", csd1); // PPS
    
    mCodec = MediaCodec.createDecoderByType("video/avc");
    mCodec.configure(format, surface, null, 0);
    mCodec.start();
}

AudioDecoder

Decodes AAC audio stream and plays back using AudioTrack.

File: decoder/AudioDecoder.java (172 lines)

Key Features

• Format: AAC (audio/mp4a-latm)
• Sample Rate: 48 kHz
• Channels: Stereo (2 channels)
• Bit Rate: 128 kbps

Playback Pipeline

private void configure(byte[] data) {
    MediaFormat format = MediaFormat.createAudioFormat(
        MIMETYPE_AUDIO_AAC, SAMPLE_RATE, 2);
    format.setInteger(MediaFormat.KEY_BIT_RATE, 128000);
    format.setByteBuffer("csd-0", ByteBuffer.wrap(data));
    
    mCodec = MediaCodec.createDecoderByType(MIMETYPE_AUDIO_AAC);
    mCodec.configure(format, null, null, 0);
    mCodec.start();
    
    // Initialize AudioTrack
    initAudioTrack();
    audioTrack.play();
}

// Worker thread reads decoded PCM and writes to AudioTrack
ByteBuffer outputBuffer = mCodec.getOutputBuffer(index);
byte[] data = new byte[info.size];
outputBuffer.get(data);
audioTrack.write(data, 0, info.size);

SendCommands

Handles ADB command execution for server deployment.

File: SendCommands.java (117 lines)

Server Deployment Process

1. Connect to Device

   App.adbCmd("connect", ip + ":" + port);
   
2. Push Server JAR

   App.adbCmd("-s", ip + ":" + port, "push", 
       serverJarPath, "/data/local/tmp/scrcpy-server.jar");
   
3. Setup Port Forwarding

   App.adbCmd("-s", ip + ":" + port, "forward", 
       "tcp:" + serverport, "tcp:7007");
   
4. Launch Server

   String[] commands = new String[]{
       "-s", ip + ":" + port,
       "shell",
       "CLASSPATH=/data/local/tmp/scrcpy-server.jar",
       "app_process",
       "/",
       "org.server.scrcpy.Server",
       "/" + localip,
       Long.toString(size),
       Long.toString(bitrate) + ";"
   };
   App.adbCmd(commands);
   

Data Models

Packet Structures

VideoPacket and AudioPacket share a common base structure:

public class MediaPacket {
    public enum Type {
        VIDEO(0x00), AUDIO(0x01);
    }
}

public class VideoPacket {
    public enum Flag {
        CONFIG,      // SPS/PPS configuration
        KEY_FRAME,   // I-frame
        FRAME,       // P-frame or B-frame
        END          // End of stream
    }
    
    Type type;
    Flag flag;
    long presentationTimeStamp;
    byte[] data;
}

Threading Model

Main Thread

• UI rendering
• Activity lifecycle
• User input collection

Service Thread

• Socket I/O loop
• Packet routing
• Event queue management

VideoDecoder Worker

• MediaCodec input feeding
• Output buffer management
• Surface rendering

AudioDecoder Worker

• MediaCodec audio decoding
• AudioTrack playback
• PCM buffer management

Error Handling

The client implements retry logic and error recovery:

// Connection retry in MainActivity
private void connectExitExt(boolean userDisconnect) {
    if (!userDisconnect) {
        errorCount += 1;
        if (errorCount >= 3) {
            // Restart ADB server after 3 failures
            App.startAdbServer();
        }
    }
    
    if (headlessMode && !userDisconnect) {
        // Show reconnection dialog
        Dialog.displayDialog(this, 
            getString(R.string.connect_faild),
            getString(R.string.connect_faild_ask), 
            () -> connectScrcpyServer(serverAddress), 
            () -> finishAndRemoveTask());
    }
}

Performance Optimizations

Frame Dropping

The service implements intelligent frame dropping based on delay threshold:

if (System.currentTimeMillis() - 
    (lastVideoOffset + (videoPacket.presentationTimeStamp / 1000)) < delay) {
    videoDecoder.decodeSample(...); // Decode frame
} else {
    videoPassCount++; // Skip frame
}

Multi-touch Support

Supports multiple simultaneous touch points with pointer ID tracking (Scrcpy.java:161-170):

for (int i = 0; i < event.getPointerCount(); i++) {
    int currentPointerId = event.getPointerId(i);
    int x = (int) event.getX(i);
    int y = (int) event.getY(i);
    sendTouchEvent(action, buttonState, x, y, currentPointerId);
}

Related Documentation

Server Module

Learn about screen capture and encoding

Architecture Overview

Understand the complete system design