Audio Decoder

Hardware-accelerated audio decoder using Android MediaCodec for AAC audio streams

Overview

The AudioDecoder class provides hardware-accelerated AAC audio decoding using Android’s MediaCodec API and plays the decoded PCM audio through AudioTrack. It runs in a separate thread for continuous audio playback.

Package: org.client.scrcpy.decoder

Audio Codec: audio/mp4a-latm (AAC-LC)

Sample Rate: 48000 Hz

Channels: 2 (Stereo)

Encoding: PCM 16-bit

Initialization

start()

Starts the decoder worker thread.

public void start()

Example:

AudioDecoder audioDecoder = new AudioDecoder();
audioDecoder.start();

stop()

Stops the decoder, AudioTrack, and releases resources.

public void stop()

Example:

@Override
protected void onDestroy() {
    if (audioDecoder != null) {
        audioDecoder.stop();
    }
    super.onDestroy();
}

Configuration

configure()

Configures the MediaCodec decoder with AAC codec-specific data.

public void configure(byte[] data)

data · byte[] · required

Codec-specific data (csd-0) containing AAC configuration

Example:

// Receive CONFIG packet
AudioPacket configPacket = AudioPacket.readHead(packet);
if (configPacket.flag == AudioPacket.Flag.CONFIG) {
    byte[] configData = new byte[dataLength];
    System.arraycopy(packet, AudioPacket.getHeadLen(), 
                    configData, 0, dataLength);
    audioDecoder.configure(configData);
}

Internal Implementation:

// From source (AudioDecoder.java:90-109)
MediaFormat format = MediaFormat.createAudioFormat(
    MIMETYPE_AUDIO_AAC, // "audio/mp4a-latm"
    SAMPLE_RATE,        // 48000
    2                   // stereo channels
);

// Set bit rate
format.setInteger(MediaFormat.KEY_BIT_RATE, 128000);

// Set codec-specific data
format.setByteBuffer("csd-0", ByteBuffer.wrap(data));

mCodec = MediaCodec.createDecoderByType(MIMETYPE_AUDIO_AAC);
mCodec.configure(format, null, null, 0);
mCodec.start();

// Initialize and start AudioTrack
initAudioTrack();
audioTrack.play();

Calling configure() while already configured will stop the current decoder and AudioTrack, then create new instances. This allows dynamic reconfiguration.

Decoding

decodeSample()

Queues an encoded audio sample for decoding and playback.

public void decodeSample(byte[] data, int offset, int size, 
                         long presentationTimeUs, int flags)

data · byte[] · required

Byte array containing the encoded audio frame

offset · int · required

Starting position in the data array

size · int · required

Number of bytes to decode

presentationTimeUs · long · required

Presentation timestamp in microseconds

flags · int · required

MediaCodec flags

Example:

AudioPacket audioPacket = AudioPacket.readHead(packet);
if (audioPacket.flag == AudioPacket.Flag.FRAME || 
    audioPacket.flag == AudioPacket.Flag.KEY_FRAME) {
    
    audioDecoder.decodeSample(
        packet,
        AudioPacket.getHeadLen(),
        packet.length - AudioPacket.getHeadLen(),
        audioPacket.presentationTimeStamp,
        audioPacket.flag.getFlag()
    );
}

Frame Processing:

// From source (AudioDecoder.java:116-129)
int index = mCodec.dequeueInputBuffer(-1);
if (index >= 0) {
    ByteBuffer buffer;
    
    if (Build.VERSION.SDK_INT < Build.VERSION_CODES.LOLLIPOP) {
        buffer = mCodec.getInputBuffers()[index];
        buffer.clear();
    } else {
        buffer = mCodec.getInputBuffer(index);
    }
    
    if (buffer != null) {
        buffer.put(data, offset, size);
        mCodec.queueInputBuffer(index, 0, size, presentationTimeUs, flags);
    }
}

AudioTrack Integration

The decoder automatically manages an AudioTrack instance for audio playback.

AudioTrack Configuration

// From source (AudioDecoder.java:28-32)
private void initAudioTrack() {
    int bufferSizeInBytes = AudioTrack.getMinBufferSize(
        SAMPLE_RATE,                      // 48000 Hz
        AudioFormat.CHANNEL_OUT_STEREO,   // 2 channels
        AudioFormat.ENCODING_PCM_16BIT    // 16-bit PCM
    );
    
    audioTrack = new AudioTrack(
        AudioManager.STREAM_MUSIC,
        SAMPLE_RATE,
        AudioFormat.CHANNEL_OUT_STEREO,
        AudioFormat.ENCODING_PCM_16BIT,
        bufferSizeInBytes,
        AudioTrack.MODE_STREAM
    );
}

Audio Playback Loop

// From source (AudioDecoder.java:140-157)
int index = mCodec.dequeueOutputBuffer(info, 0);
if (index >= 0) {
    if ((info.flags & MediaCodec.BUFFER_FLAG_END_OF_STREAM) 
        == MediaCodec.BUFFER_FLAG_END_OF_STREAM) {
        break;
    }
    
    // Read decoded PCM data
    ByteBuffer outputBuffer = mCodec.getOutputBuffer(index);
    if (outputBuffer != null) {
        byte[] data = new byte[info.size];
        outputBuffer.get(data);
        outputBuffer.clear();
        
        // Write to AudioTrack for playback
        audioTrack.write(data, 0, info.size);
    }
    
    // Release output buffer
    mCodec.releaseOutputBuffer(index, true);
}

AudioPacket Structure

Audio data is transmitted using the AudioPacket structure.

Packet Format

Header (10 bytes):

Byte 0:    Type (1 = AUDIO)
Byte 1:    Flag (frame type)
Bytes 2-9: Presentation timestamp (long)

Total packet structure:

[Type][Flag][Timestamp][Data...]
  1B    1B      8B       Variable

Flag Types

FRAME · byte

Value: 0 - Regular audio frame

KEY_FRAME · byte

Value: 1 - Key audio frame

CONFIG · byte

Value: 2 - Configuration data (codec-specific)

END · byte

Value: 4 - End of audio stream

Reading Packets

// Read packet header only
AudioPacket audioPacket = AudioPacket.readHead(packet);

// Access packet properties
MediaPacket.Type type = audioPacket.type;              // AUDIO
AudioPacket.Flag flag = audioPacket.flag;             // FRAME, CONFIG, etc.
long timestamp = audioPacket.presentationTimeStamp;    // microseconds

// Extract data from full packet
int dataLength = packet.length - AudioPacket.getHeadLen();
byte[] audioData = new byte[dataLength];
System.arraycopy(packet, AudioPacket.getHeadLen(), audioData, 0, dataLength);

Usage Example

Complete example of setting up and using the audio decoder:

public class AudioPlayer {
    private AudioDecoder audioDecoder;
    
    public void startAudioPlayback() {
        // 1. Initialize decoder
        audioDecoder = new AudioDecoder();
        audioDecoder.start();
        
        // 2. Wait for configuration packet
        byte[] packet = receiveAudioPacket();
        AudioPacket configPacket = AudioPacket.readHead(packet);
        
        if (configPacket.flag == AudioPacket.Flag.CONFIG) {
            // Extract configuration data
            int dataLength = packet.length - AudioPacket.getHeadLen();
            byte[] configData = new byte[dataLength];
            System.arraycopy(packet, AudioPacket.getHeadLen(), 
                           configData, 0, dataLength);
            
            // Configure decoder
            audioDecoder.configure(configData);
        }
        
        // 3. Decode and play audio frames
        while (isPlaying) {
            packet = receiveAudioPacket();
            AudioPacket audioPacket = AudioPacket.readHead(packet);
            
            if (audioPacket.flag == AudioPacket.Flag.FRAME || 
                audioPacket.flag == AudioPacket.Flag.KEY_FRAME) {
                
                audioDecoder.decodeSample(
                    packet,
                    AudioPacket.getHeadLen(),
                    packet.length - AudioPacket.getHeadLen(),
                    audioPacket.presentationTimeStamp,
                    audioPacket.flag.getFlag()
                );
            }
            
            if (audioPacket.flag == AudioPacket.Flag.END) {
                break;
            }
        }
        
        // 4. Cleanup
        audioDecoder.stop();
    }
}

Threading Model

The AudioDecoder uses an internal Worker thread:

// From source (AudioDecoder.java:134-167)
@Override
public void run() {
    try {
        MediaCodec.BufferInfo info = new MediaCodec.BufferInfo();
        while (mIsRunning.get()) {
            if (mIsConfigured.get()) {
                int index = mCodec.dequeueOutputBuffer(info, 0);
                if (index >= 0) {
                    // Check for end of stream
                    if ((info.flags & MediaCodec.BUFFER_FLAG_END_OF_STREAM) 
                        == MediaCodec.BUFFER_FLAG_END_OF_STREAM) {
                        break;
                    }
                    
                    // Decode and play PCM audio
                    ByteBuffer outputBuffer = mCodec.getOutputBuffer(index);
                    if (outputBuffer != null) {
                        byte[] data = new byte[info.size];
                        outputBuffer.get(data);
                        outputBuffer.clear();
                        audioTrack.write(data, 0, info.size);
                    }
                    
                    mCodec.releaseOutputBuffer(index, true);
                }
            } else {
                // Wait for configuration
                Thread.sleep(5);
            }
        }
    } catch (IllegalStateException e) {
        // Handle codec errors
    }
}

Audio Specifications

MIMETYPE_AUDIO_AAC · String

Value: "audio/mp4a-latm" - AAC Low Complexity profile

SAMPLE_RATE · int

Value: 48000 - 48 kHz sampling rate

CHANNELS · int

Value: 2 - Stereo output

BIT_RATE · int

Value: 128000 - 128 kbps encoding bitrate

PCM_ENCODING · int

Value: AudioFormat.ENCODING_PCM_16BIT - 16-bit PCM output

Performance Considerations

The decoder uses hardware acceleration and real-time audio playback. Ensure that:

• The device supports AAC hardware decoding
• Audio focus is properly managed
• The AudioTrack stream type is appropriate for your use case
• Frame processing keeps up with the sample rate to avoid buffer underruns

Key Points:

• Decoded PCM data is written directly to AudioTrack
• Input buffer dequeue timeout is set to -1 (wait indefinitely)
• Output buffer dequeue timeout is 0 (non-blocking)
• AudioTrack uses MODE_STREAM for continuous playback
• Minimum buffer size is calculated automatically

Common Issues

Buffer Underruns

If you experience audio stuttering:

// Increase AudioTrack buffer size
int minBufferSize = AudioTrack.getMinBufferSize(...);
int bufferSize = minBufferSize * 2; // Double the minimum size

audioTrack = new AudioTrack(
    AudioManager.STREAM_MUSIC,
    SAMPLE_RATE,
    AudioFormat.CHANNEL_OUT_STEREO,
    AudioFormat.ENCODING_PCM_16BIT,
    bufferSize, // Larger buffer
    AudioTrack.MODE_STREAM
);

Audio Focus

Manage audio focus properly:

AudioManager audioManager = (AudioManager) getSystemService(Context.AUDIO_SERVICE);
int result = audioManager.requestAudioFocus(
    focusChangeListener,
    AudioManager.STREAM_MUSIC,
    AudioManager.AUDIOFOCUS_GAIN
);

if (result == AudioManager.AUDIOFOCUS_REQUEST_GRANTED) {
    audioDecoder.start();
}

See Also

• Video Decoder - Video stream decoding
• Scrcpy Service - Main service integration
• Event Controller - Input event handling