> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/zwc456baby/ScrcpyForAndroid/llms.txt
> Use this file to discover all available pages before exploring further.

# Client Module Architecture

> Deep dive into the Android client module responsible for UI, decoding, and user input handling

## Module Overview

The client module (`app/`) is an Android application that provides the user interface and handles video/audio decoding, user input collection, and ADB communication.

**Package**: `org.client.scrcpy`\
**Location**: `app/src/main/java/org/client/scrcpy/`

## Core Components

### MainActivity

The main activity manages the application lifecycle and coordinates between UI and streaming service.

**File**: `MainActivity.java` (837 lines)

#### Key Responsibilities

<AccordionGroup>
  <Accordion title="Connection Management">
    Handles server connection setup, including:

    * Reading user preferences (resolution, bitrate, delay)
    * Validating server address
    * Deploying server JAR via ADB
    * Establishing socket connection

    From `MainActivity.java:738-791`:

    ```java theme={null}
    private void connectScrcpyServer(String serverAdr) {
        String[] serverInfo = Util.getServerHostAndPort(serverAdr);
        String serverHost = serverInfo[0];
        int serverPort = Integer.parseInt(serverInfo[1]);
        
        // Extract and write server JAR
        InputStream inputStream = assetManager.open("scrcpy-server.jar");
        byte[] buffer = new byte[inputStream.available()];
        inputStream.read(buffer);
        
        FileOutputStream outputStream = new FileOutputStream(
            new File(context.getExternalFilesDir("scrcpy"), 
            "scrcpy-server.jar"));
        outputStream.write(buffer);
        
        // Deploy via ADB and start streaming
        sendCommands.SendAdbCommands(context, serverHost, 
            serverPort, localForwardPort, videoBitrate, maxSize);
    }
    ```
  </Accordion>

  <Accordion title="Lifecycle Management">
    Manages activity states with service binding:

    ```java theme={null}
    private final ServiceConnection serviceConnection = new ServiceConnection() {
        @Override
        public void onServiceConnected(ComponentName name, IBinder binder) {
            scrcpy = ((Scrcpy.MyServiceBinder) binder).getService();
            scrcpy.setServiceCallbacks(MainActivity.this);
            serviceBound = true;
            scrcpy.start(surface, serverAddress, 
                screenHeight, screenWidth, delayControl);
        }
    };
    ```

    Key lifecycle methods:

    * `onCreate()`: Initialize UI and restore state
    * `onPause()`: Pause streaming and optionally disconnect
    * `onResume()`: Resume streaming or reconnect
    * `onSaveInstanceState()`: Persist configuration
  </Accordion>

  <Accordion title="Display and Touch Handling">
    Calculates aspect ratio and sets up touch event handling (`MainActivity.java:362-433`):

    ```java theme={null}
    public void set_display_nd_touch() {
        int[] rem_res = scrcpy.get_remote_device_resolution();
        int remote_device_height = rem_res[1];
        int remote_device_width = rem_res[0];
        float remote_aspect_ratio = (float) remote_device_height / remote_device_width;
        
        // Calculate padding for aspect ratio matching
        // ...
        
        // Set touch listener
        surfaceView.setOnTouchListener((view, event) -> 
            scrcpy.touchevent(event, landscape, 
                surfaceView.getWidth(), surfaceView.getHeight()));
    }
    ```
  </Accordion>

  <Accordion title="Rotation Handling">
    Supports dynamic orientation changes:

    ```java theme={null}
    @Override
    public void loadNewRotation() {
        unbindService(serviceConnection);
        landscape = !landscape;
        swapDimensions(); // Swap width and height
        
        if (landscape) {
            setRequestedOrientation(
                ActivityInfo.SCREEN_ORIENTATION_SENSOR_LANDSCAPE);
        } else {
            setRequestedOrientation(
                ActivityInfo.SCREEN_ORIENTATION_SENSOR_PORTRAIT);
        }
    }
    ```
  </Accordion>
</AccordionGroup>

### Scrcpy Service

Background service that manages the streaming connection and coordinates decoders.

**File**: `Scrcpy.java` (486 lines)

#### Architecture

```mermaid theme={null}
graph LR
    A[MainActivity] -->|Binds| B[Scrcpy Service]
    B -->|Creates| C[VideoDecoder]
    B -->|Creates| D[AudioDecoder]
    B -->|Socket| E[Server]
    E -->|Video Stream| C
    E -->|Audio Stream| D
    C -->|Renders| F[Surface]
    D -->|Plays| G[AudioTrack]
    A -->|Touch Events| B
    B -->|Send Events| E
```

#### Key Methods

**Connection Loop** (`Scrcpy.java:342-470`):

```java theme={null}
private void loop(DataInputStream in, DataOutputStream out, int delay) {
    while (LetServiceRunning.get()) {
        // Send queued events
        byte[] sendevent = event.poll();
        if (sendevent != null) {
            out.write(sendevent, 0, sendevent.length);
        }
        
        // Read incoming packets
        if (in.available() > 0) {
            byte[] packetSize = new byte[4];
            in.readFully(packetSize, 0, 4);
            int size = ByteUtils.bytesToInt(packetSize);
            
            byte[] packet = new byte[size];
            in.readFully(packet, 0, size);
            
            if (MediaPacket.Type.getType(packet[0]) == VIDEO) {
                VideoPacket videoPacket = VideoPacket.readHead(packet);
                videoDecoder.decodeSample(packet, offset, size, 
                    timestamp, flags);
            } else if (MediaPacket.Type.getType(packet[0]) == AUDIO) {
                AudioPacket audioPacket = AudioPacket.readHead(packet);
                audioDecoder.decodeSample(packet, offset, size, 
                    timestamp, flags);
            }
        }
    }
}
```

**Touch Event Processing** (`Scrcpy.java:136-182`):

```java theme={null}
public boolean touchevent(MotionEvent event, boolean landscape, 
                          int displayW, int displayH) {
    // Calculate scaling between display and remote device
    float remoteW = landscape ? 
        Math.max(remote_dev_resolution[0], remote_dev_resolution[1]) :
        Math.min(remote_dev_resolution[0], remote_dev_resolution[1]);
    float remoteH = landscape ? 
        Math.min(remote_dev_resolution[0], remote_dev_resolution[1]) :
        Math.max(remote_dev_resolution[0], remote_dev_resolution[1]);
    
    switch (event.getAction()) {
        case MotionEvent.ACTION_MOVE:
            // Handle all pointers for multi-touch
            for (int i = 0; i < event.getPointerCount(); i++) {
                int pointerId = event.getPointerId(i);
                int x = (int) (event.getX(i) * realW / displayW);
                int y = (int) (event.getY(i) * realH / displayH);
                sendTouchEvent(action, buttonState, x, y, pointerId);
            }
            break;
        // ... handle other actions
    }
}
```

### VideoDecoder

Decodes H.264 video stream using Android's MediaCodec API.

**File**: `decoder/VideoDecoder.java` (131 lines)

#### Implementation Details

<CodeGroup>
  ```java Configuration theme={null}
  public void configure(Surface surface, int width, int height, 
                        ByteBuffer csd0, ByteBuffer csd1) {
      MediaFormat format = MediaFormat.createVideoFormat(
          "video/avc", width, height);
      format.setByteBuffer("csd-0", csd0); // SPS
      format.setByteBuffer("csd-1", csd1); // PPS
      
      mCodec = MediaCodec.createDecoderByType("video/avc");
      mCodec.configure(format, surface, null, 0);
      mCodec.start();
  }
  ```

  ```java Decoding theme={null}
  public void decodeSample(byte[] data, int offset, int size, 
                           long presentationTimeUs, int flags) {
      int index = mCodec.dequeueInputBuffer(-1);
      if (index >= 0) {
          ByteBuffer buffer = mCodec.getInputBuffer(index);
          buffer.put(data, offset, size);
          mCodec.queueInputBuffer(index, 0, size, 
              presentationTimeUs, flags);
      }
  }
  ```

  ```java Rendering theme={null}
  @Override
  public void run() {
      MediaCodec.BufferInfo info = new MediaCodec.BufferInfo();
      while (mIsRunning.get()) {
          if (mIsConfigured.get()) {
              int index = mCodec.dequeueOutputBuffer(info, 0);
              if (index >= 0) {
                  // Render frame to Surface
                  mCodec.releaseOutputBuffer(index, true);
              }
          }
      }
  }
  ```
</CodeGroup>

### AudioDecoder

Decodes AAC audio stream and plays back using AudioTrack.

**File**: `decoder/AudioDecoder.java` (172 lines)

#### Key Features

* **Format**: AAC (audio/mp4a-latm)
* **Sample Rate**: 48 kHz
* **Channels**: Stereo (2 channels)
* **Bit Rate**: 128 kbps

#### Playback Pipeline

```java theme={null}
private void configure(byte[] data) {
    MediaFormat format = MediaFormat.createAudioFormat(
        MIMETYPE_AUDIO_AAC, SAMPLE_RATE, 2);
    format.setInteger(MediaFormat.KEY_BIT_RATE, 128000);
    format.setByteBuffer("csd-0", ByteBuffer.wrap(data));
    
    mCodec = MediaCodec.createDecoderByType(MIMETYPE_AUDIO_AAC);
    mCodec.configure(format, null, null, 0);
    mCodec.start();
    
    // Initialize AudioTrack
    initAudioTrack();
    audioTrack.play();
}

// Worker thread reads decoded PCM and writes to AudioTrack
ByteBuffer outputBuffer = mCodec.getOutputBuffer(index);
byte[] data = new byte[info.size];
outputBuffer.get(data);
audioTrack.write(data, 0, info.size);
```

### SendCommands

Handles ADB command execution for server deployment.

**File**: `SendCommands.java` (117 lines)

#### Server Deployment Process

<Steps>
  <Step title="Connect to Device">
    ```java theme={null}
    App.adbCmd("connect", ip + ":" + port);
    ```
  </Step>

  <Step title="Push Server JAR">
    ```java theme={null}
    App.adbCmd("-s", ip + ":" + port, "push", 
        serverJarPath, "/data/local/tmp/scrcpy-server.jar");
    ```
  </Step>

  <Step title="Setup Port Forwarding">
    ```java theme={null}
    App.adbCmd("-s", ip + ":" + port, "forward", 
        "tcp:" + serverport, "tcp:7007");
    ```
  </Step>

  <Step title="Launch Server">
    ```java theme={null}
    String[] commands = new String[]{
        "-s", ip + ":" + port,
        "shell",
        "CLASSPATH=/data/local/tmp/scrcpy-server.jar",
        "app_process",
        "/",
        "org.server.scrcpy.Server",
        "/" + localip,
        Long.toString(size),
        Long.toString(bitrate) + ";"
    };
    App.adbCmd(commands);
    ```
  </Step>
</Steps>

## Data Models

### Packet Structures

**VideoPacket** and **AudioPacket** share a common base structure:

```java theme={null}
public class MediaPacket {
    public enum Type {
        VIDEO(0x00), AUDIO(0x01);
    }
}

public class VideoPacket {
    public enum Flag {
        CONFIG,      // SPS/PPS configuration
        KEY_FRAME,   // I-frame
        FRAME,       // P-frame or B-frame
        END          // End of stream
    }
    
    Type type;
    Flag flag;
    long presentationTimeStamp;
    byte[] data;
}
```

## Threading Model

<CardGroup cols={2}>
  <Card title="Main Thread" icon="mobile">
    * UI rendering
    * Activity lifecycle
    * User input collection
  </Card>

  <Card title="Service Thread" icon="server">
    * Socket I/O loop
    * Packet routing
    * Event queue management
  </Card>

  <Card title="VideoDecoder Worker" icon="video">
    * MediaCodec input feeding
    * Output buffer management
    * Surface rendering
  </Card>

  <Card title="AudioDecoder Worker" icon="volume">
    * MediaCodec audio decoding
    * AudioTrack playback
    * PCM buffer management
  </Card>
</CardGroup>

## Error Handling

The client implements retry logic and error recovery:

```java theme={null}
// Connection retry in MainActivity
private void connectExitExt(boolean userDisconnect) {
    if (!userDisconnect) {
        errorCount += 1;
        if (errorCount >= 3) {
            // Restart ADB server after 3 failures
            App.startAdbServer();
        }
    }
    
    if (headlessMode && !userDisconnect) {
        // Show reconnection dialog
        Dialog.displayDialog(this, 
            getString(R.string.connect_faild),
            getString(R.string.connect_faild_ask), 
            () -> connectScrcpyServer(serverAddress), 
            () -> finishAndRemoveTask());
    }
}
```

## Performance Optimizations

### Frame Dropping

The service implements intelligent frame dropping based on delay threshold:

```java theme={null}
if (System.currentTimeMillis() - 
    (lastVideoOffset + (videoPacket.presentationTimeStamp / 1000)) < delay) {
    videoDecoder.decodeSample(...); // Decode frame
} else {
    videoPassCount++; // Skip frame
}
```

### Multi-touch Support

Supports multiple simultaneous touch points with pointer ID tracking (`Scrcpy.java:161-170`):

```java theme={null}
for (int i = 0; i < event.getPointerCount(); i++) {
    int currentPointerId = event.getPointerId(i);
    int x = (int) event.getX(i);
    int y = (int) event.getY(i);
    sendTouchEvent(action, buttonState, x, y, currentPointerId);
}
```

## Related Documentation

<CardGroup cols={2}>
  <Card title="Server Module" icon="server" href="/development/server-module">
    Learn about screen capture and encoding
  </Card>

  <Card title="Architecture Overview" icon="sitemap" href="/development/overview">
    Understand the complete system design
  </Card>
</CardGroup>